Developer Community

Hello,  We are heavely relying on transformations to transform our data from Raw (staging) service to Data Models. We notice that it has very high latency compared to a “simple” Spark job. As per our discussion with our Solution Architect we understood that the bottleneck is the Raw service. Any plans to improve this in the future? Thank you!

Hello,I am working with the Cognite Data Modeling API to validate edges in a large data set. Our edges do not have a dedicated "edge view" defined in the model. When I attempt to use the instances.query() API to fetch edges filtered by their type, I encounter errors or unexpected behavior indicating that selected properties do not match the view schema.Specifically, filtering or selecting on properties such as "externalId" fails because the queried view does not explicitly contain those properties, as the edges exist only as instances without a separate view.Is there a recommended approach or best practice for querying and validating edges in scenarios where no dedicated edge view is defined? Should we rely exclusively on the instances.list() API with filters, or is there a way to construct valid queries for edges in the current model setup?Any guidance on how to effectively query and validate edges in such cases, especially for large-scale data, would be appreciated.Thank you!

Hi Cognite Team,I'm building a React dashboard using Cognite Data Fusion for my portfolio and would like to know if there are any demo projects available for working with CDF data.

We’re using the client.documents.previews.download_page_as_png_bytes() function in the Python SDK to retrieve png previews of P&IDs in order to display them in a webapp (Streamlit).For some P&IDs, the dimensions of the returned png are distorted. The image is in the correct orientation but squashed to fit within a portrait envelope. The affected P&IDs are displayed correctly in both CDF Data Management and when downloaded and opened in a pdf viewer.Has anyone had this issue before and, if so, did you find a way to handle it? 

Something is wrong with this overview. The fields CREATED/STARTED/FINISHED does not seem to correspond with what is shown in the graph, and the DURATION.Can someone explain what the STARTED field actually is, the graph and the timestamps don’t line up. Does it count the start of the transformation when it start writing or when it start reading (only one of these is sensible).The duration should be the time between the transformation was triggered and it was completed. From the perspective of the user the transformation has started when it triggers, how long it takes before it actually begins should be part of this duration. Measuring both intervals could be interesting, but the one that is currently there is not really useful for us. 

HiWe are testing out a new deletion transformation for a view in data modeling. The transformation looks something like this:select externalIdfrom cdf_data_models(...)where project in ("ProjectA","ProjectB")This transformation always times out and returns a “Graph query time out” error message. However, when changing the transformation to this the transformation just runs fine:select externalIdfrom cdf_data_models(...)where project = "ProjectA"unionselect externalIdfrom cdf_data_models(...)where project = "ProjectB"For context, the project column here is indexed and the view has around 10 properties and about 1.7 million rows. The transformations all query the same view.We are therefore wondering how the filtering in transformations handles filtering in data modeling and why these transformations are performing different. Learning more about this would benefit us greatly when working with transformations and data modeling in the future :)Thanks in advance!Sebastian

Hi When testing some new functionality in a Cognite Function I got this error message: “The size of the data field is too large.”The data input was from the output of another function ran previously in a workflow.I am looking through the documentation to find what the limits are, but can not find anything.For my use case I can decrease the input size and run more batches, but it would be good to know what the size limits are for input data into Cognite Functions. Also, are there any other size limits I should be aware of?Thank you!Sebastian

All,Now that you’ve had some time to “play with” the Records feature, we’re looking to identify what the consensus is when it comes to the available aggregations for Records. Do we have a useful mix of aggregations available in the API today? Which of the aggregates do you use most often? What is your experience with the API documentation for aggregate API endpoint so far? Have you attempted to use the movingFunction aggregate? What problem did you use it to solve for your  use case? Was the information in the documentation sufficient and useful for you? If you haven’t used the aggregate, please help us understand why not? Have you attempted to use the timeHistogram aggregate? Was the information in the documentation sufficient and useful for you? If you haven’t used the aggregate, please help us understand why not? Is there a use case involving aggregates that you have not been able to address in Records? Please describe the use case? How important is the “missing” aggregate for

Had a chance to experiment with streams and records and it seems to be working well.  Couple of questions based on what I found:Are any of the filtering options that are available for general data modeling queries but not available with records likely to be supported in future?  Specifically, I am thinking of queries like Fetch me all of the records in the last week where the `asset` property is below `Facility-ABC` (i.e. it contains `Facility-ABC` in asset.path).  I think this would require us to use the nested filter unless the total number of assets that were below Facility-ABC was small enough to pass them directly into the filter.  I can imagine that this kind of filter is more difficult to implement and a more expensive operation, but I think it could be valuable. Is there a reason why containers rather than views must be used when creating/querying records?  It seems like a view with a specific version is nothing more than a set of containers with a (possibly incomplete) list of

All,We’re readying a release of the OPC UA Extractor with support for the Records API and are looking for help with testing data flow of OPC UA events/alarms to CDF Records. The extractor creates containers for several different OPC UA event types and will be able to store data in either mutable or immutable streams (as specified in the configuration file for the extractor)Please reach out - feel free to email me at thomas.sjolshagen@cognite.com and let me know if you’re interested in testing the new extractor).Also, keep in mind that we have support for Records as a target and source in the Hosted extractors for CDF, so let me know here if you’ve got questions about how to use the hosted extractor for the purpose of onboarding relevant data to Records.

Hello,I am testing streams and records in one of our projects, but I am getting “The user does not have any ILA streams capabilities.” error.I am also not able to add the capabilities for Streams and Records to my user group as I cannot see them in the capability type list 

I have the following question regarding Containers,  Spaces and Instances. Let's say I have a Container named Well in the space A and all its instances are created in space B. When I delete container Well in space A and recreate it (because I want to change a column name for example): Does instance in space B are deleted as well? If they are not deleted, If I rerun the transformation to populate the new container again in space B with the new schema (same external_id) what state should I expect my instances to be in ? A merge of old and new schema? What is the best practice ? To delete container in its space and its instances in all spaces also? Thank you!  

This might be a basic question, but I often find myself uncertain about when to use CogniteDescribable in data modeling within Cognite Data Fusion.For example, let's say we have domain tables in our data model, such as MaterialType, which contains data from SAP and defines the types of materials we work with (e.g., code and description). In this case, would it be appropriate to implement CogniteDescribable to reuse common properties like name and description?What are the advantages of doing so? Does using CogniteDescribable provide any specific benefits—such as improved performance through indexing or partitioning—when used it containers?In summary, should I always use CogniteDescribable in these types of scenarios, or are there cases where it's better not to?

Hi,We have just rolled out “mutability” support for the Records API service. Mutability is the ability to change a record once it has initially been written to the Records API service.Enabling mutability for a stream requires using the settings.template.name key in the payload of the creation request for a stream. I.e. submitting a POST operation to the /streams endpoint, with - for example - the following body:items: [ externalId: "a-mutable-stream-1", settings: { template: { name: "MutableTestStream" } }] There are two supported “mutable”stream settings templates: “MutableTestStream”, and “MutableLiveData”.To update or create a record in this stream, you must use the newly introduced upsert endpoint in the Records API and specify the required identifiers of the previously ingested record you’re wanting to update.Over the next couple of weeks, we would love it if you could spend some time familiarizing yourself with mutable streams, and test record updates to he

We intend to effect 3 breaking changes to the Records API over the next couple of weeks of the Private Beta program. These changes may require updates of your test procedures. /streams API “settings” attribute will be required Summary: Modifying the Cognite Streams API.From when: After July 15, 2025Description:The Streams API provides a broad spectrum of functionalities, but it's important to understand that these capabilities are not mutually exclusive; enhancing one often means adjusting another. For instance, if a stream is designed for permanent data storage, it will offer unlimited record retention but a lower maximum ingestion rate. Conversely, streams built for temporary data staging will support significantly bigger ingestion rate but only for a brief retention period. Similarly, you'll choose between mutable streams (allowing record changes) and immutable streams (optimized for high volume and speed).Because of these crucial distinctions, it's essential for users to be fully a

Hello, We are considering storing a relational database table with more than 100 columns as Records.Since the maximum number of properties per container is 100, we need to define a single table across two or more containers.In this case, I would like to know how to write a query that allows us to freely filter records that have properties defined across multiple containers.Our goal is to query the Records data in a way similar to executing SQL on the original RDB table with over 100 columns.Based on my testing, the behavior was as follows, which is not what I intended:When using only properties from container A in the filter or sort: only records of properties from container A are retrieved. When using properties from both container A and container B in the filter or sort: an empty JSONL is returned.If this cannot be resolved with a single query, I would appreciate advice on what kind of post-processing could help achieve the desired result.

i want to take export of one table to another table but facing this datatype error .also tried with multiple datatype (like string , float and Bigint ) still same issue is coming.  

Hello,  We are using: Toolkit Version '0.5.20' We are having a file schedules.Function.yaml: - name: Run every 10 minutes to do some work functionExternalId: fn_name description: Run every 10 minutes to do some work cronExpression: "*/10 * * * *" data: extractionPipeline: "test"When deploying we have this error trace: Traceback (most recent call last): File "venv/bin/cdf-tk", line 8, in <module> sys.exit(app()) File "venv/lib/python3.10/site-packages/cognite_toolkit/_cdf.py", line 116, in app _app() File "venv/lib/python3.10/site-packages/typer/main.py", line 340, in __call__ raise e File "venv/lib/python3.10/site-packages/typer/main.py", line 323, in __call__ return get_command(self)(*args, **kwargs) File "venv/lib/python3.10/site-packages/click/core.py", line 1161, in __call__ return self.main(*args, **kwargs) File "venv/lib/python3.10/site-packages/typer/core.py", line 740, in main return _main( File "venv/lib/python3.10/site-packages/typer/core

Hi,We have uploaded a 3D .nwd model using the Fusion UI and noticed that the underlying processed 3D files (e.g., .glb, .reveal, .zip) are stored as file resources and are visible in the Data Explorer under the Files resource type.I couldn’t find any documentation or guideline on how to assign security categories to hide or restrict access to these files for end users. I also tried using dataset-based access control, but I’m looking for a better or recommended way to handle this.Appreciate your guidance or suggestions.

The python extractor utils requires the config to be stored as a local file there is not way to pass it directly as a string/yaml object. This makes it quite cumbersome to use as not all our deployments have access to local file storage for config files.Is there a way to add support for the config to be passed as an environment variable? Reading config from env example:  

I am getting following error when ingesting files from SharePoint site to CDF. send_request_body.failed exception=LocalProtocolError('Too much data for declared Content-Length')Anything I need to specify in .yaml file to allow content-length? Also how do I trace which file is causing the problem. I changed Logger to DEBUG level but still don’t see the file name info.Thanks,Zuber

Hello, We envision to implement a versioning based on minor and major version of our data models. So a data model version will be presented with two digits: <Major>.<Minor><Major>: incremented when we do a breaking change that is not backward compatible with the previous one and we know that it will break our consumers queries. Exp: delete an attribute in a view in the data model<Minor>: incremented when we do a change that is backward compatible and consumer queries will not break. Exp: add a new field. Is there a way to get a graphql url that maps always to the latest minor version of a given data model? This will avoid our data model consumers to redeploy their application every time we increment our minor version.  Thank you!

When deploying a function with the SDK you can specify the source code “folder” as a parameter. I have used this to deploy the functions with my own python module which is just a directory with some py files. I want to migrate to using the toolkit, but I am not sure how to deploy with the source code folder I want as I do not see a way to specify it in the yaml file. Can anyone help with this? I have multiple Cognite Functions that I want to deploy that all access the same source code I have written.Thank you!Sebastian

I see there is a fork for ToolJet in the CogniteData git repo.  I’m curious to know if Cognite has done anything with this branch and if so, how it’s being used within the product. Thanks!