• Home
  • Product Ideas
  • How to use is_new() function with lastUpdatedTime from FDM type in transformation

How to use is_new() function with lastUpdatedTime from FDM type in transformation

Related products: Data Modeling

Hi,

 

When the data is in CDF RAW then we can use is_new function with lastUpdatedTime  in transformation and this works fine but how can we use that function for data coming from FDM model. 

Let’s say, from CDF RAW we are populating the data in FDM domain model and we are considering only incremental data with is_new function in transformations.

This is the domain model (space: dm):

type Entity @view(space: "dm", version: "1"){

name:String

}

 

This is transformation query to populate above Entity type:

select
  distinct concat('wellbore:',WellUWI)  as externalId,
  WellName as name,
 FROM
  `db1`.`wellbores`
where is_new(
        "db1:wellbores",
        lastUpdatedTime
      )

This is good and we can see only incremental data from CDF RAW in the domain model. 

 

Now this is the solution model (sol_model):

type WELLBORE{

name:String 

}

This is transformation query to populate above WELLBORE type ( I can directly map the name of WELLBORE of solution model to ENTITY of domain model but for this example I am ignoring it to understand is_new() function):

select
  entity_table.externalId as externalId,

  entity_table.name as name
 FROM
  cdf_nodes('dm','Entity','1') as entity_table
where is_new(
        "dm:entity",
        entity_table.lastUpdatedTime
      )

Here, I can’t use lastUpdatedTime because even though it is internal attribute of Entity type, it is not possible to get it in the result of cdf_nodes . The cdf_nodes doesn’t give us internal attributes. 

How can we make sure, the solution model transformations only consider the incremental data coming from domain model?  Am I missing anything here?

 

 

 

@Niranjan Madhukar Karvekar This is not currently supported, but on the short term roadmap scheduled for October release.

-Jason

Thanks @Jason Dressel for the update. 

New→Planned for development

Hi, update on this ticket: We originally planned this for the October release, but after looking closer at possible solutions this week we have decided to go for an implementation that is slightly more complex, but that we believe will provide a better experience and performance for the users. The timeline is therefore a bit more unceratin, but we’re working actively on this.

@Jørgen Lund  Will this be available somewhere in November? The reason is to understand if we should go for a workaround in the meantime or should be wait for the solution from your team?

@Aditya Kotiyal Thanks for the additonal context, and apologies for the late reply. We’re looking into the implementation details, which brings a follow-up question to be able to answer your question: How frequently are you planning to execute the Transformations that will leverage this functionality once it’s available? Initially, there will be a difference implementation for jobs running more frequently than every 3 days (e.g. hourly), and jobs running less frequently (e.g. weekly). 

@Jørgen Lund Thanks for your response.

I will let @Niranjan Madhukar Karvekar  comment on the technicalities but let’ say if the feature becomes available in the fullness of time, the idea will be to automate it. So that every time new data arrives in the base model, the transformations from base to solution model is auto triggered.

I believe if it is about running a cron job than for us it will be more frequent and to mimic as and when data comes, it can be hourly, but will let @Niranjan Madhukar Karvekar  comment.

@Jørgen Lund Thanks for your response. We have scheduled our current transformations to run hourly from domain model to solution model. As we are ingesting more data in domain model , the transformations from domain to solution has been taking time as it always considers all the data. The scheduled transformation from RAW to domain model is working fine as it only considers new data.  

 

Hi @Niranjan Madhukar Karvekar @Aditya Kotiyal. Great, thanks for the additional information. The team will prioritize this feature, meaning you do not have to go for a workaround as we will provide the proper solution soon. I will keep you updated, and let you know when you can try it out.

Hi @Jørgen Lund , Do you have any update of this feature? Can we try in any CDF project?

Terms & ConditionsCookie Settings