Developer Community

All,Now that you’ve had some time to “play with” the Records feature, we’re looking to identify what the consensus is when it comes to the available aggregations for Records. Do we have a useful mix of aggregations available in the API today? Which of the aggregates do you use most often? What is your experience with the API documentation for aggregate API endpoint so far? Have you attempted to use the movingFunction aggregate? What problem did you use it to solve for your  use case? Was the information in the documentation sufficient and useful for you? If you haven’t used the aggregate, please help us understand why not? Have you attempted to use the timeHistogram aggregate? Was the information in the documentation sufficient and useful for you? If you haven’t used the aggregate, please help us understand why not? Is there a use case involving aggregates that you have not been able to address in Records? Please describe the use case? How important is the “missing” aggregate for

Had a chance to experiment with streams and records and it seems to be working well.  Couple of questions based on what I found:Are any of the filtering options that are available for general data modeling queries but not available with records likely to be supported in future?  Specifically, I am thinking of queries like Fetch me all of the records in the last week where the `asset` property is below `Facility-ABC` (i.e. it contains `Facility-ABC` in asset.path).  I think this would require us to use the nested filter unless the total number of assets that were below Facility-ABC was small enough to pass them directly into the filter.  I can imagine that this kind of filter is more difficult to implement and a more expensive operation, but I think it could be valuable. Is there a reason why containers rather than views must be used when creating/querying records?  It seems like a view with a specific version is nothing more than a set of containers with a (possibly incomplete) list of

All,We’re readying a release of the OPC UA Extractor with support for the Records API and are looking for help with testing data flow of OPC UA events/alarms to CDF Records. The extractor creates containers for several different OPC UA event types and will be able to store data in either mutable or immutable streams (as specified in the configuration file for the extractor)Please reach out - feel free to email me at thomas.sjolshagen@cognite.com and let me know if you’re interested in testing the new extractor).Also, keep in mind that we have support for Records as a target and source in the Hosted extractors for CDF, so let me know here if you’ve got questions about how to use the hosted extractor for the purpose of onboarding relevant data to Records.

Hello,I am testing streams and records in one of our projects, but I am getting “The user does not have any ILA streams capabilities.” error.I am also not able to add the capabilities for Streams and Records to my user group as I cannot see them in the capability type list 

I have the following question regarding Containers,  Spaces and Instances. Let's say I have a Container named Well in the space A and all its instances are created in space B. When I delete container Well in space A and recreate it (because I want to change a column name for example): Does instance in space B are deleted as well? If they are not deleted, If I rerun the transformation to populate the new container again in space B with the new schema (same external_id) what state should I expect my instances to be in ? A merge of old and new schema? What is the best practice ? To delete container in its space and its instances in all spaces also? Thank you!  

This might be a basic question, but I often find myself uncertain about when to use CogniteDescribable in data modeling within Cognite Data Fusion.For example, let's say we have domain tables in our data model, such as MaterialType, which contains data from SAP and defines the types of materials we work with (e.g., code and description). In this case, would it be appropriate to implement CogniteDescribable to reuse common properties like name and description?What are the advantages of doing so? Does using CogniteDescribable provide any specific benefits—such as improved performance through indexing or partitioning—when used it containers?In summary, should I always use CogniteDescribable in these types of scenarios, or are there cases where it's better not to?

Hi,We have just rolled out “mutability” support for the Records API service. Mutability is the ability to change a record once it has initially been written to the Records API service.Enabling mutability for a stream requires using the settings.template.name key in the payload of the creation request for a stream. I.e. submitting a POST operation to the /streams endpoint, with - for example - the following body:items: [ externalId: "a-mutable-stream-1", settings: { template: { name: "MutableTestStream" } }] There are two supported “mutable”stream settings templates: “MutableTestStream”, and “MutableLiveData”.To update or create a record in this stream, you must use the newly introduced upsert endpoint in the Records API and specify the required identifiers of the previously ingested record you’re wanting to update.Over the next couple of weeks, we would love it if you could spend some time familiarizing yourself with mutable streams, and test record updates to he

We intend to effect 3 breaking changes to the Records API over the next couple of weeks of the Private Beta program. These changes may require updates of your test procedures. /streams API “settings” attribute will be required Summary: Modifying the Cognite Streams API.From when: After July 15, 2025Description:The Streams API provides a broad spectrum of functionalities, but it's important to understand that these capabilities are not mutually exclusive; enhancing one often means adjusting another. For instance, if a stream is designed for permanent data storage, it will offer unlimited record retention but a lower maximum ingestion rate. Conversely, streams built for temporary data staging will support significantly bigger ingestion rate but only for a brief retention period. Similarly, you'll choose between mutable streams (allowing record changes) and immutable streams (optimized for high volume and speed).Because of these crucial distinctions, it's essential for users to be fully a

Hello, We are considering storing a relational database table with more than 100 columns as Records.Since the maximum number of properties per container is 100, we need to define a single table across two or more containers.In this case, I would like to know how to write a query that allows us to freely filter records that have properties defined across multiple containers.Our goal is to query the Records data in a way similar to executing SQL on the original RDB table with over 100 columns.Based on my testing, the behavior was as follows, which is not what I intended:When using only properties from container A in the filter or sort: only records of properties from container A are retrieved. When using properties from both container A and container B in the filter or sort: an empty JSONL is returned.If this cannot be resolved with a single query, I would appreciate advice on what kind of post-processing could help achieve the desired result.

i want to take export of one table to another table but facing this datatype error .also tried with multiple datatype (like string , float and Bigint ) still same issue is coming.  

Hello,  We are using: Toolkit Version '0.5.20' We are having a file schedules.Function.yaml: - name: Run every 10 minutes to do some work functionExternalId: fn_name description: Run every 10 minutes to do some work cronExpression: "*/10 * * * *" data: extractionPipeline: "test"When deploying we have this error trace: Traceback (most recent call last): File "venv/bin/cdf-tk", line 8, in <module> sys.exit(app()) File "venv/lib/python3.10/site-packages/cognite_toolkit/_cdf.py", line 116, in app _app() File "venv/lib/python3.10/site-packages/typer/main.py", line 340, in __call__ raise e File "venv/lib/python3.10/site-packages/typer/main.py", line 323, in __call__ return get_command(self)(*args, **kwargs) File "venv/lib/python3.10/site-packages/click/core.py", line 1161, in __call__ return self.main(*args, **kwargs) File "venv/lib/python3.10/site-packages/typer/core.py", line 740, in main return _main( File "venv/lib/python3.10/site-packages/typer/core

Hi,We have uploaded a 3D .nwd model using the Fusion UI and noticed that the underlying processed 3D files (e.g., .glb, .reveal, .zip) are stored as file resources and are visible in the Data Explorer under the Files resource type.I couldn’t find any documentation or guideline on how to assign security categories to hide or restrict access to these files for end users. I also tried using dataset-based access control, but I’m looking for a better or recommended way to handle this.Appreciate your guidance or suggestions.

The python extractor utils requires the config to be stored as a local file there is not way to pass it directly as a string/yaml object. This makes it quite cumbersome to use as not all our deployments have access to local file storage for config files.Is there a way to add support for the config to be passed as an environment variable? Reading config from env example:  

I am getting following error when ingesting files from SharePoint site to CDF. send_request_body.failed exception=LocalProtocolError('Too much data for declared Content-Length')Anything I need to specify in .yaml file to allow content-length? Also how do I trace which file is causing the problem. I changed Logger to DEBUG level but still don’t see the file name info.Thanks,Zuber

Hello, We envision to implement a versioning based on minor and major version of our data models. So a data model version will be presented with two digits: <Major>.<Minor><Major>: incremented when we do a breaking change that is not backward compatible with the previous one and we know that it will break our consumers queries. Exp: delete an attribute in a view in the data model<Minor>: incremented when we do a change that is backward compatible and consumer queries will not break. Exp: add a new field. Is there a way to get a graphql url that maps always to the latest minor version of a given data model? This will avoid our data model consumers to redeploy their application every time we increment our minor version.  Thank you!

When deploying a function with the SDK you can specify the source code “folder” as a parameter. I have used this to deploy the functions with my own python module which is just a directory with some py files. I want to migrate to using the toolkit, but I am not sure how to deploy with the source code folder I want as I do not see a way to specify it in the yaml file. Can anyone help with this? I have multiple Cognite Functions that I want to deploy that all access the same source code I have written.Thank you!Sebastian

I see there is a fork for ToolJet in the CogniteData git repo.  I’m curious to know if Cognite has done anything with this branch and if so, how it’s being used within the product. Thanks!

We are trying to implement automatic monitoring over our transformations in CDF. We are doing this by running a transformation and a monitoring function in the same workflow, where the monitoring function is dependent on the transformation being finished.Then we retrieve the sum of updated and created events like this: def _created_and_updated_sum(self, transformation_external_id: str) -> int: """ Get the sum of the number of created and updated events in the last transformation job. If there are no finished jobs, return 0 instead. """ transformation = self.cdf_client.transformations.retrieve( external_id = transformation_external_id, ) if transformation is None: raise ValueError(f"Transformation {transformation_external_id} not found") if transformation.last_finished_job is None: DUMMY = 0 self.logger.info(f"Transformation {transformation_external_id} has no finished jobs.

ContextQuerying views with a large number of instances (>1 million), we frequently encounter query timeout issues. This has become a critical bottleneck affecting application performance and user experience. To mitigate this, we introduced on the app layer a pre-query caching strategy:Before sending a query to Cognite, we aggregate the number of instance spaces for a given view using the endpoint /models/instances/aggregate. This result is stored in a cache layer. When a query is initiated, we check if the user included a space filter. If not, we append the known relevant spaces from the cache to the query filter. This approach has significantly reduced timeouts across our applications. However, it introduces new challenges:One request per view is still needed to fetch associated spaces. Cache invalidation must be managed periodically, especially as user capabilities may change. This workaround does not help with timeouts in the CDF UI or Infield tools, where we cannot control the q

HiI am looking into using the toolkit more actively for deploying resources to CDF. One question that was raised when researching how to use the toolkit is what kind of validations actually happens when doing a dry run for deploying data modeling resources. I do not really have any specific issue I want answered, but rather want to learn more about the tool so prepare for a lot of questions from my notes:)Does it test that the configuration of views and containers work together?Does a successful dry run mean that I can be sure that the deployment will always work?Are there anything I need to consider even after getting a successful dry run?What kind of responses do I get if the dry run finds that something is wrong? Do I get any hints about how to fix an issue?Does it consider what is already deployed into the CDF environment?Will it tell me about any issues that can happen with new breaking changes? Appreciate all kinds of insights and experiences around this topic :)Sebastian

Hello people,I’m testing CDM and I want to extend a current model type, I may do it wrong, but my idea is. I’m creating a new type called MyModel, that will be pretty much and CogniteActivity + new fields.But I don’t know what I’m doing wrong.type MyModel @import ( dataModel: { externalId: "CogniteActivity", space: "cdf_cdm", version: "v1" } ){ FieldA: String, FieldB: String}I got some guidance from this link https://docs.cognite.com/cdf/dm/dm_guides/dm_extending_core_data_modelWhen I try to save, I got this error{   "title": "Error: could not update data model",   "message": "An error has occured. Data model was not published.",   "extra": "[Line: 40] Data model 'DataModelRef(space=cdf_cdm, externalId=CogniteActivity, version=v1)' cannot be found" }Thanks in advance

On behalf of CelaneseHi.I would like to report a potential issue with cursor-based pagination when sorting in descending order and request clarification on the undocumented forceCursorsDespitePerformanceHazard flag.We are executing a query against the view OEEEvent, which is defined with a cursorable BTREE index on the startDateTime property, as shown in the simplified schema below: type OEEEvent@container(  indexes: [    {      fields: ["startDateTime"]      identifier: "idx_oeev_startDateTime"      indexType: BTREE      cursorable: true    }  ]) {  startDateTime: Timestamp  endDateTime: Timestamp  totalDuration: Float} We are querying with the following time range filter: "startDateTime": {  "gte": "2024-12-31T00:00:00.000-03:00",  "lt":  "2025-05-01T00:00:00.000-03:00"} The result set includes 840 records.When applying a page size of 500 and using the following ascending sort configuration: {  "property": [    "INO-COR-ALL-DML",    "OEEEvent/472bdbcb9cd6f1",    "startDateTime"  ], 

Hi,Will there be any major changes to the API and Python SDK when moving from an asset centric data model to a flexible data model? Do we have to expect to refactor our backend using the current python sdk once the flexible data model is operationalized?Best regards,Vetle.

I am able to Delete data using python code using Primary key of the table in CDF staging/RAW. But I need help with deleting data based on where condition for columns other than primary key. I am following the below documentation for deleting based on primary key. Data Ingestion — cognite-sdk 7.74.5 documentation  Delete rows from table:>>> from cognite.client import CogniteClient>>> client = CogniteClient()>>> keys_to_delete = ["k1", "k2", "k3"]>>> client.raw.rows.delete("db1", "table1", keys_to_delete)

I’m trying to update the is_string value of Time Series from false to true using CDF python class ognite.client.data_classes.time_series.TimeSeriesUpdate. But I’m unable to update it. The value in CDF remains false even though I updated it. Is this an expected behavior of Time Series Update?