Developer Community

Hello, I have two questions:When using cdf_nodes in transformation, does query optimizer take advantage of the container indexes? What we notice is it is not the case.  How can we run broadcast joins in transformation. Hints does seem to be working: https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-hints.htmlThank you!

WITH archived_wells AS ( SELECT DISTINCT ac.WellId FROM cdf_nodes( 'sp_dm_dap_knowledge_graph', 'Activity', '0.71' ) ac JOIN contextualization.dap_wellview_wellid_space_mapping map ON ac.WellId = map.key WHERE map.Archived = TRUE AND ac.Archived = FALSE)SELECT s.externalId, s.space, s.WellId, true as ArchivedFROM cdf_nodes( 'sp_dm_dap_knowledge_graph', 'WellState', '0.71' ) sJOIN archived_wells aw ON s.WellId = aw.WellIdWHERE s.Archived = FALSE AND EXISTS ( SELECT 1 FROM archived_wells )WellState has 20M rows of data. To improve SQL performance, I want to use a CTE to get the IDs of archived wells and then update WellState (archived and WellId are indexed). However, I found that even when using EXISTS to check that the CTE result is empty, cdf_nodes still loads instances. In this case, shouldn’t the result just be empty?Additionally, when I replaced the CTE with a subquery,

Hi everyone,As we prepare for the General Availability (GA) release of Records in CDF, we're implementing important changes based on learnings from the Private Beta. These changes will take effect on November 3rd, 2025.What's ChangingStream Limits per CDF Project:Active streams: reduced from 10 to 3 Soft-deleted streams: reduced from 100 to 30Stream Templates: We're streamlining from 6 private beta-phase templates to 3 templates:ImmutableTestStream - for experimentation only BasicArchive - for perpetual data storage (immutable) BasicLiveData - for production usage (mutable)The following beta templates will no longer be available for new stream creation:ImmutableDataStaging ImmutableNormalizedData ImmutableArchive MutableTestStream MutableLiveDataIf you're using ImmutableTestStream, note that the template capacity has been significantly reduced. The maximum total records decreased from 5B to 50M, and the maximum total data volume decreased from 500GB to 50GB.Existing streams will retain

Hi all, looking for guidance on two issues we hit with CDF related to timeseries visibility and datamodel queries.403 error leaking timeseries externalIds:when a user without access requests a timeseries resource, a 403 error is returned as expected, but the error body contains the externalIds of the timeseries. Those externalIds are sensitive and should not be exposed to users who don’t have access.Is it expected behavior that externalIds are included in 403 error messages? If not, is there a setting / configuration / log-level that controls whether identifiers are returned in errors? Any recommended mitigation or planned fix? Datamodel query returns datapoints from unauthorized timeseriesSetup: Created dataset_1 and dataset_2 Created two timeseries, TS_A in dataset_1 and TS_B in dataset_2 Attached both TS_A and TS_B to a datamodel Granted the user only timeseries:READ scope:dataset_2 (so they should see TS_B only) Observed behavior: querying the datamodel returns datapoints for T

Hi, Data models names are limited to 37 chars today. Is this a hard limit? Is there a way to extend this so we can have more possibilities for internal naming conventions? Thank you!

Hello, We are using cdf toolkit to deploy our data models. We have a relatively a big data models that uses 478 views. Each time we deploy a breaking change our data model architecture (well centric) obliges us to redeploy a new version for all views. We notice a huge deployment time at the dependencies resolution step: 2025-08-19T09:34:54+00:00 WARNING [LOW]: Failed to create 478 views: One or moreviews do not exist: 'sp_dm_mud_brines_for_ops_real_wview:Activity/0.10'.. Attempting to recover...The overall deployment time takes 35 mins, this way bigger than our standard CI/CD job duration where we try to keep it under 10min at worst. Is there a way to optimize this please?  Thank you!    

I created an asset hierarchy with the Cogntie Core Data Model, then I created a custom data model through neat and I need to migrate the assets from the core data model to the custom extended data model, I tried creating a transformation but got the following error: failed with status 403: Properties [assetHierarchy_path, assetHierarchy_path_last_updated_time, assetHierarchy_root] are maintained by DMS and cannot be modified by end users. is there a way I can migrate/update the core data model assets to our extended data model?

Hi,Context:We currently organize our data in CDF using a per-country partitioning strategy, where each country has its own space.This approach was chosen primarily to restrict data access by country in a fine-grained manner.On top of that, we expose data models grouped by Business Object, such as Well Architecture, Cost Model, etc.Each Business Object model aggregates several data objects under a common business theme, which also allows us to control access by business domain in addition to country-based access.We are now planning to integrate a large amount of historical data, which will likely increase our model size by around 4x.These historical datasets are rarely queried, but we want to make sure their addition does not degrade performance for operational data — both in query latency and data ingestion throughput.We are evaluating two potential strategies: Keep everything in the same spaces, adding an indexed attribute (e.g., is_legacy = true) to distinguish legacy records. Crea

While I try to insert a Stream (using the http post as shown in the Getting Started guide), I am getting the next error:cognite.client.exceptions.CogniteAPIError: Project 2982735218914002 not enabled for Industrial Log Analytics | code: 400 | X-Request-ID: 66573df8-87ce-9b37-a837-180aee796a6c | cluster: westeurope-1 | project: slb-mmv-ops-dev This detail isn’t mentioned anywhere on the documentation. I cannot find how to change that. Can you enable ILA for https://delfi-dev.fusion.cognite.com/slb-mmv-ops-dev/?

I am stumbling through the Beta documentation. When I try to run the http post to create a Stream (as described here), I get this error:The feature can only be used with [alpha, beta] headersI found documentation that seemed relevant, but it seems the format of the header is this cdf-version: 20230101-beta https://api-docs.cognite.com/20230101/#section/API-versions/Beta-versionsHow do I know what date should I put into the header to gain access to the Streams beta feature?

Hi,I have been updating the beta documentation a bit since the last edition linked in the invitation. This applies to both the planned standard Cognite documentation, and the developer API documentation. All of the documents are available using a direct link to our document rendering services (linked below), and should be updated as we privately deploy new information.Note that all of these documents are works in progress with ongoing updates, so please forgive any typos, omissions, and other errors at this stage: Streams API documentation Records API documentation Capabilities for CDF Records Updated Data Modeling concepts page Concepts page for CDF Records (and Streams) Example high level use case (alarms) for CDF RecordsPlease do not share these documents.

Hi team,  do you have 2-3 articles on how to get started, like a simple quick start example as a starting point to be acquainted with the service ?Thanks !FYI ​@Marwen TALEB 

Is there a way to delete the relationship between an instance and a container? Let’s say I have a view Foo in one data model and another view FooExtension in another data model. FooExtension implements Foo.I have an instance of externalId “FOO”, space: “FOO-DAT” that have data stored in both views/containers.If it was no longer necessary to store data in container FooExtension is there a way to just delete this relationship in a way that `hasData(FooExtension)` from the DMS Query doesn’t find this instance anymore without affecting the data stored in Foo and the instance itself? Since instance count is limited, it’s becoming more common in our project modeling things having the mindset of shared instance when they mean the same thing across data models but with different formats to present the data on different Apps. But sometimes we don’t want to have the data in the app anymore and virtual deletion (boolean flags) is not always an option specially when the data volume is big. 

Hello, We are a heavy users of Grafana CDF datasource. Are you planning to support alerting anytime soon?  Thanks !

HiI am testing out event based triggers to see if its a good fit for a use case I am working on. When deploying the trigger the workflow is triggered many times until it has processed the whole input query, however as very many workflow executions are started at the same time, some of them will fail as the limit for running workflows has been reached.How is this handled by the trigger, will the failing workflow runs be retried? If not, is there a setting I can use to limit the amount of concurrent runs?I think this would be good both for distributing the workflow runs out and avoid the failures and also reduce the risk of failing function calls because of high load on the API.Thank you!Sebastian

Is there a way to configure a Hosted HiveMQ Extractor to extract the data into CogniteTimeSeries and not the traditional Time Series?

As illustrated in our documentation, the CDF Records feature uses a Data Modeling container as the schema definition for record data. I.e. you have to create a space and container first, before you start loading Records to a Stream. (Note: It is possible to use multiple containers together to define the schema for a single record. This may make sense in the context of, for instance, a Work Order record).However, at the moment when I write this, using the container based schema represents a somewhat confusing “limitation” when it comes to the size of a container vs the size of a record (number of properties). The way we have implemented this capability in CDF Records at the moment - using the DM containers - comes with a side-effect: The number of properties you can have for a single container is, as of right now, the same as it is in your Data Modeling service.The limits we're documenting in terms of properties for CDF Records are linked to the properties containing data within a singl

When running the following lines of code I get an error with Pydantic:from cognite.neat import NeatSessionneat = NeatSession(client)The error is as follows:---------------------------------------------------------------------------ImportError Traceback (most recent call last)Cell In[13], line 1----> 1 from cognite.neat import NeatSession 2 neat = NeatSession(client)File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\cognite\neat\__init__.py:4 1 from cognite.neat.core._utils.auth import get_cognite_client 3 from ._version import __version__----> 4 from .session import NeatSession 6 __all__ = ["NeatSession", "__version__", "get_cognite_client"]File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\cognite\neat\session\__init__.py:1----> 1 from ._base import NeatSession 3 __all__ = ["NeatSession"]File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\cognite\neat\session\_base.py:9 7 from

Hello,  We recently migrating the way of deployment of Streamlit App in our cdf toolkit project. Before we used to deploy our streamlit app by creating a file in a dedicated dataset. Now we use out of box way of deploying Streamlit Apps provided by cdf toolkit  (this was not available before).  When changing to this way, we notice that the app is only deployed using “--include streamlit” option.i.e: when not using the “--include streamlit” cdf toolkit the deploy summary is the following:    however, we notice it is a previous version that has been deployed!  when using “--include streamlit”: summary is the following And the App latest version is deployed as expected. Toolkit Version used is '0.6.20'. Are we missing something? Could you please take a look at this?  Thanks!      

When creating a Cognite Function you specify the dataset it belongs to, as an example lets say D2 LCI, then the files of the function is stored as files in that dataset. However, as D2 LCI is a dataset for file storage for engineering documents these function files cause noise, not much but still. If you were to list out all files for the dataset these “internal” CDF files would simply show as regular files and would have to be filtered out.Now, as we (AkerBP) are moving away from the asset-centric and over to data modelling this might not be an issue given how data models and spaces are. But in the future when datasets are a thing of the past, how will function and the like work, where will those files be stored. Storing the file used for compute with the data they compute might be a simple implementation, but not ideal as it does cause some noise. Note: this does not only apply to Cognite Functions, but to all CDF features where “internal” files are stored alongside the regular data.

Datasets have an “Access Control” page where you can see which groups have access. Is there a similar way to quickly identify which groups have access to Data Models / Spaces?If not, can this be implemented as a feature?

Hello,I’m looking to compute the standard deviation of a timeseries on the fly with synthetic timeseries.I expected to use this pseudocode formula : sqrt(avg(pow(TS{externalid}-avg(TS{externalid}),2))) with endpoint :client.time_series.data.synthetic.query(    expressions=expression,    start="2w-ago",    end="now")Unfortunately, avg expect at least 2 inputs, I try to switch to aggregate feature but I found it available only for timeseries, not synthetic timeseries.expression = '''sqrt( avg( pow( ts{ID} - ts{ID, aggregate="average", granularity="14d"}, 2 ) ))'''Do you have any tips or workaround to compute this value when “start” value changes ? Dont hesitate to explain I'm open to any opportunity to calculate this metric using another method.Thanks in advance,Pierre edit : I find this function in additionnal library : Rolling standard deviation of data points time delta — indsl 8.7.0 documentation but i’m looking for a answer without additionn

I need help getting attached error(Error: no gl) in chart page

Hello,  We are heavely relying on transformations to transform our data from Raw (staging) service to Data Models. We notice that it has very high latency compared to a “simple” Spark job. As per our discussion with our Solution Architect we understood that the bottleneck is the Raw service. Any plans to improve this in the future? Thank you!

Hello,I am working with the Cognite Data Modeling API to validate edges in a large data set. Our edges do not have a dedicated "edge view" defined in the model. When I attempt to use the instances.query() API to fetch edges filtered by their type, I encounter errors or unexpected behavior indicating that selected properties do not match the view schema.Specifically, filtering or selecting on properties such as "externalId" fails because the queried view does not explicitly contain those properties, as the edges exist only as instances without a separate view.Is there a recommended approach or best practice for querying and validating edges in scenarios where no dedicated edge view is defined? Should we rely exclusively on the instances.list() API with filters, or is there a way to construct valid queries for edges in the current model setup?Any guidance on how to effectively query and validate edges in such cases, especially for large-scale data, would be appreciated.Thank you!