Hi,
When the data is in CDF RAW then we can use is_new function with lastUpdatedTime in transformation and this works fine but how can we use that function for data coming from FDM model.
Let’s say, from CDF RAW we are populating the data in FDM domain model and we are considering only incremental data with is_new function in transformations.
This is the domain model (space: dm):
type Entity @view(space: "dm", version: "1"){
name:String
}
This is transformation query to populate above Entity type:
select
distinct concat('wellbore:',WellUWI) as externalId,
WellName as name,
FROM
`db1`.`wellbores`
where is_new(
"db1:wellbores",
lastUpdatedTime
)
This is good and we can see only incremental data from CDF RAW in the domain model.
Now this is the solution model (sol_model):
type WELLBORE{
name:String
}
This is transformation query to populate above WELLBORE type ( I can directly map the name of WELLBORE of solution model to ENTITY of domain model but for this example I am ignoring it to understand is_new() function):
select
entity_table.externalId as externalId,
entity_table.name as name
FROM
cdf_nodes('dm','Entity','1') as entity_table
where is_new(
"dm:entity",
entity_table.lastUpdatedTime
)
Here, I can’t use lastUpdatedTime because even though it is internal attribute of Entity type, it is not possible to get it in the result of cdf_nodes . The cdf_nodes doesn’t give us internal attributes.
How can we make sure, the solution model transformations only consider the incremental data coming from domain model? Am I missing anything here?
-Jason
Thanks@Jason Dressel for the update.
Hi, update on this ticket: We originally planned this for the October release, but after looking closer at possible solutions this week we have decided to go for an implementation that is slightly more complex, but that we believe will provide a better experience and performance for the users. The timeline is therefore a bit more unceratin, but we’re working actively on this.
I will let@Niranjan Madhukar Karvekar comment on the technicalities but let’ say if the feature becomes available in the fullness of time, the idea will be to automate it. So that every time new data arrives in the base model, the transformations from base to solution model is auto triggered.
I believe if it is about running a cron job than for us it will be more frequent and to mimic as and when data comes, it can be hourly, but will let@Niranjan Madhukar Karvekar comment.
Hi@Niranjan Madhukar Karvekar @Aditya Kotiyal . Great, thanks for the additional information. The team will prioritize this feature, meaning you do not have to go for a workaround as we will provide the proper solution soon. I will keep you updated, and let you know when you can try it out.
Hi@Jørgen Lund , Do you have any update of this feature? Can we try in any CDF project?