Skip to main content
Solved

CDF Architecture - Data Persistence on CDF

V
Committed
Forum|alt.badge.img

Is the data persist both on staging and the contextualized layer(destination layer after transformation as per target data model)? Or only on staging layer? If only on the staging layer, is the data pulled from staging layer during the API calls and presented as per the target model?

 

Regards

Vikash

Best answer by Fredrik Holm

Hi Vikash. 
The data available through the data model is residing in containers, and is not connected to the staging area. When a transformation is run the data is read from the staging area, transformed into types according to the data model and written to the containers used by the data model. This allows you to mix and match data, and also allows you to update the data in the containers directly using REST API or GraphQL mutations. You may read on up the details of the persistence layer of the data model here: https://developer.cognite.com/dm/concepts/containers_views 

Additional details of transformation concepts can be found here: https://docs.cognite.com/cdf/integration/concepts/transformation/ 

I hope this clarifies the concepts a bit. 

View original
    Anita Hæhre
    Galo Santiago Segura Melo
    Andre Alves
    4 people like this
    • Anita Hæhre
      Anita Hæhre
    • Galo Santiago Segura Melo
      Galo Santiago Segura Melo
    • Andre Alves
      Andre Alves
    • André Luiz Ribeiro de Araujo
      André Luiz Ribeiro de Araujo
    Did this topic help you find an answer to your question?

    3 replies

    F
    Practitioner
    Forum|alt.badge.img
    • Fredrik Holm
    • Practitioner
    • 9 replies
    • Answer
    • August 17, 2023

    Hi Vikash. 
    The data available through the data model is residing in containers, and is not connected to the staging area. When a transformation is run the data is read from the staging area, transformed into types according to the data model and written to the containers used by the data model. This allows you to mix and match data, and also allows you to update the data in the containers directly using REST API or GraphQL mutations. You may read on up the details of the persistence layer of the data model here: https://developer.cognite.com/dm/concepts/containers_views 

    Additional details of transformation concepts can be found here: https://docs.cognite.com/cdf/integration/concepts/transformation/ 

    I hope this clarifies the concepts a bit. 

    V
    1 person likes this
    • V
      vsanwaria
    V
    Committed
    Forum|alt.badge.img
    • vsanwaria
    • Author
    • Committed
    • 5 replies
    • August 17, 2023

    Thanks @Fredrik Holm . As we discussed in our session today, getting the architectural document for under the hood processing of data on different layers will be quite useful for creating a better and transparent view of data processing.

     

    Cheers!

    Vikash

    V
    Committed
    Forum|alt.badge.img
    • vsanwaria
    • Author
    • Committed
    • 5 replies
    • August 17, 2023

    @Fredrik Holm : Summary of my understanding:

    • Only way a container can be created is, when the transformation is run on the staging layer or thru REST API/GraphQL mutation
    • Each container will have an associated version (data model).

    There were discussions around changing the data model, it means the schema of the container for a given version is tightly coupled with the schema/data-model which was latest during the time of that respective container creation.

    • Does all the containers get created for each new versions of data model?
    • For older data models associated with older version, will the container even get created for all the new data coming to staging layer?

    Regards

    Vikash

     

    Reply


    Terms & ConditionsCookie Settings

    Cookie Policy

    We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

     
    Cookie Settings