Upsert functionality in Rest APIs of Cognite

@Anita Hæhre , Can you please tag Celanese to this Hub Item? Also add/tag @rsiddha  for all Celanese relevant Hub items in the future for my visibility and tracking purposes. Thank you.

Hi @rsiddha Celanese-tag added and will tag you moving forward :)

@rsiddha FYI

@Anita Hæhre  can we get a response on this from product?

Hi @Harsha / @ibrahim.alsyed and thank you for the input.

Upsert will be included in the new Enhanced Data Modeling services and, but the state of Upsert for the more "traditional" Coginite resource APIs - Assets, Events, Labels, Relationships, etc - is under consideration right now as we haven't fully decided what we will be doing with those APIs as we roll out and iterate on Data Modeling after GA in April. 

We have been asked for upsert support through other channels, but with the focus on Data Modeling we have elected to not prioritize that work at this time. 

I have captured your product idea and we will use this as part of our consideration for when and how we choose to implement "Update or Insert" functionality as we decide how we will continue to provide these APIs.

@Thomas Sjølshagen Thank you for your response on this item. This has been evaluated further on our end and the requested functionality is considered critical requirement (and a significant blocker) for our multi-site scaleup. Please let us know what does it take from us to make the “Must Have’s” described in this item, a critical priority on development side for Cognite and be added to an upcoming sprint (with assigned timeline). 

I beleive we need to drive urgency on this for FDM.

Not quite following you @ibrahim.alsyed?

Enhanced Data Modeling (aka FDM) has upsert support today and is expected to GA with upsert support included.

Beyond that; The work to get FDM to GA in the April release consumes our available storage resources, and more than 95% of the team that is needed to deliver upsert for other resources like Assets, Events or Relationships are heads down with the FDM deliverable. They will likely stay engaged with that until sometime after the GA release (potentially for as long as 4-8 weeks post GA). 
 

Possibly too much “making the sausage” info above, but wanted to share in the interest of transparency.

@Thomas Sjølshagen any decision made on this request. we are seeing a high volume of data interfaces and they can be reduced by half if we can have an upsert API 

@Hunter Beck we need this item priority escalated on Cognite end and incorporation of this into roadmap and timeline for implementation on this functionality, as this limitation has come across several times over the past year. If you would like us to setup a call to discuss the details, I would be glad to setup a call with our Ingestion team members. Thanks

@Harsha could you explain what you mean by “we are seeing a high volume of data interfaces and they can be reduced by half”?

As @Thomas Sjølshagen mentioned above in the thread, this capability is already present in our new Data Modeling service. Ultimately, this will be the primary place for data like assets and events. I know Celanese still has a blend of classic and new data models, but you have much of your data in FDM or are moving that direction to my understanding. Therefore, I would expect this to not be a significant problem no (even though it may have been 6 - 12 months ago). As you know @rsiddha, we have many additional features we should add to FDM which I believe you would agree should take priority to what will become a legacy data model soon. 

we still need this for classical assets, events, and time series