Community Discussions

Hi: I’m doing DATA ENGINEER BASICS - INTEGRATE course.Trying to execute Notebook 1, I cannot pass through the cell getting the authentication with Azure, namely:creds = authenticate_azure()Any help?P.S. I’m using Google Colab  

I’ve been introduced to CDF in the last 2 weeks, very interesting and learning through. Just wondering if CDF has been used for real time monitoring use cases, vs. predictive/BI type use cases. For example, how soon can the alerts be triggered from the time a new abnormal measurement is detected. I imagine that it depends on a lot of factors  such as network latency, source extractors, transform functions & schedules, Is there a threshold (say 1hr response time) beyond which CDF can be used.

Hi Team, I am facing some issues while using the above pipeline.yaml  file setup to run the pipeline, Could you please help me doing the devops setup so that i can understand the process.I will set up a call , please let me know your available timings. Regards,Nidhi N G

Learn 360 images in Cognite Reveal.

The cognite replicator fails to replicate asset data and the behaviour is inconsistent. It worked yesterday and not working today. Note that all resource types like time series, events and datapoints are replicated consistently. I am getting below error while replicating the asset data.Please see attached screenshot for the successful run of the asset replication yesterdayTraceback (most recent call last):  File "/Users/j.subhash.parandekar/oid-replicator/oid_replicator/cognite_replicate.py", line 84, in <module>    assets.replicate(SOURCE_CLIENT, DEST_CLIENT, config=cognite_config,  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/cognite/replicator/assets.py", line 324, in replicate    src_dst_ids_assets = create_hierarchy(                         ^^^^^^^^^^^^^^^^^  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/cognite/replicator/assets.py", line 220, in create_hierarchy    updated_assets = replication

When adding a node instance with Float properties, it always gives an error “The value of property 'xxx' must be a 64 bit floating point number” if the value in the request body is 0 or negative number, but it works fine if the value is positive.Could you help take a look?

Hi Team, I am facing some issues while using the above pipeline.yaml  file setup which is suggeted by cdf team before, i am getting validation error, even after passing all mandatory fields.Could you please help me doing the devops setup so that i can understand the process.Please set up a call to work on this issue as it is impacting deliverables. Regards,Nidhi N G

I am trying to make a comparison between time series data from CDF and from Pi (accessed with the Seeq Python API). Some questions arise:​​​​​How do I ensure that the DateTimeIndex-s that are returned are identical? Why does CDF return DateTimeIndex with dates before my start value?Examples:Pulling a raw time series from Pi via Seeq looks like this:start = datetime(2021, 1, 1)end = datetime(2021, 1, 2)spy.pull(items, start=start, end=end, grid=None, quiet=True)And the CDF analog looks like this:start = datetime(2021, 1, 1)end = datetime(2021, 1, 2)cognite.datapoints.retrieve_dataframe( external_id=external_id, start=start, end=end, granularity=None)The resulting DateTimeIndex-es are not identical:From Pi via Seeq:DatetimeIndex(['2021-01-01 00:00:00+01:00', '2021-01-01 00:00:05+01:00', '2021-01-01 00:00:10+01:00', '2021-01-01 00:00:15+01:00', '2021-01-01 00:00:20+01:00', '2021-01-01 00:00:25+01:00', '2021-01-01 00:00:30+01:00', '2021

Hi Team, Please help with the current setup or process  of azure devops  to deploy Cognite (CDF) Function(API) to CDF project.

We are currently exploring workflow development on top of CDF python SDK. The client secret is used in the workflow code in order to connect with CDF project instance. This client secret is bound to expire after predefined interval. It will incur production downtime and manual intervention in order to update the secret.Can we have or do we have some concept similar to refresh tokens in CDF Python SDK.

Can we have function chaining feature in Cognite functions where on sucess/failure status of a function we initiate another function.

Learn Cognite API in java script SDK.

Hi all,what's the best way to write a condition in the transformation query, that checks for presence of certain key in asset metadata?Example asset metadata:"KEY": "VALUE"pseudocode:select * from _cdf.assets A where ‘KEY’ in A.metadata.keys I’ve been provided with following solutiion by Cognite support, with a note to ask here. Is there a better/other solution?select * from _cdf.assets where metadata['KEY'] is not null;

Hi, I see this error in extractor log, I hope even if this happens , it will be retried and there will not be the data loss. Please correct if I am wrong or if there are ways to change any configuration in extractor to avoid this.  2023-03-07 05:42:04.233 UTC [WARNING ] QueryExecutor_3 - Too many concurrent requests in pod | code: 503

In this post we’ll share with you our thoughts on key areas to focus on to empower you as Domain Experts - frontline workers - to efficiently utilize data and analytics in day-to-day operations. Let us know in the comments what you think!  The direction for Cognite Data FusionCognite Data Fusion (CDF) is an Industrial Data platform, and part of our mission is to ensure that we provide a “batteries included” experience to quickly realize value across our target industries Manufacturing, Energy and Power and Renewables.We want CDF to be the go-to tool for not only Data Scientist and deeply technical roles, but also the Domain Experts that know first hand what the operational challenges are and what the optimization potential is. Over the last couple of months, and in our upcoming releases, you will see a focus on enabling you as a Domain Expert to solve industrial data problems through low-code user interfaces - without assistance from “a coder”.  What sort of industrial data problems ar

It seems I can create an edge with any type. In the code example below, the externalId of the “type” is set to “xxx” when calling POST Add edges, which should not exist. However, the edge was added successfully. Furthermore, I could not see any edges were created in the SimulationModel table, even the “externalId” was set to “SimulatonModelSimulator”.  { "instanceType": "edge", "space": "XiaofengTest", "externalId": "externalId:edge1", "type": { "space": "XiaofengTest", "externalId": "xxx" }, "startNode": { "space": "XiaofengTest", "externalId": "externalId:simulator1" }, "endNode": { "space": "XiaofengTest", "externalId": "externalId:simmodel1" } } 

I’m trying to transform some time-series data (manufacturing pump pressures) into Asset Hierarchy/Numeric Data.The SQL is not writing to the timestamp column and not seeing errors in Preview.I’ve tried MANY different SQL commands to convert the DateTimeStamp string column to timestamp column/datatype and none have worked. CSV files are attached if anyone wants to try it. Added query in attachments. Help is greatly appreciated! Transform setup: Here is some of my data being edited in Excel and saved as CSV then imported into CDF Raw. TagID Tagname Value DateTimeStamp FCE1-HYD-PMP-3 FCE1-HYD-PMP-3 NULL 3/2/2023 3:41 FCE1-HYD-PMP-3 FCE1-HYD-PMP-3 28.54443 3/2/2023 5:51 FCE1-HYD-PMP-3 FCE1-HYD-PMP-3 35.68053 3/2/2023 5:51 FCE1-HYD-PMP-3 FCE1-HYD-PMP-3 42.81664 3/2/2023 5:51 FCE1-HYD-PMP-3 FCE1-HYD-PMP-3 49.95274 3/2/2023 5:51  RAW     

Why are algorithms and data-driven models only available for the few domain experts who also are fluent in advanced software coding? We certainly don’t believe this should be the case.We are making them available to the rest of the world — particularly to non-coding domain experts.For many years  industrial data scientists have been building smart algorithms to solve  complex industrial problems. They are now available in a no-code drag-and-drop intuitive interface. Liberate your data and empower domain experts the tools to drive impact every day.Cognite Charts includes several data science toolboxes that provide subject matter experts (SMEs) out-of-the-box algorithms to process and manipulate data, conduct root cause analysis (RCA) and develop solutions without having to code.The toolboxes cover basic operations, statistical methods, data transformation, and advanced models. They work out-of-the-box with Cognite Charts, and we will continuously add new algorithms, features, and functi

It seems the link to download the demo data is not working in https://docs.cognite.com/cdf/data_modeling/upload_raw_dm. Once clicked the “the demo data” link, got an error message “This site can’t be reached”. Seems the dropbox link is broken. https://www.dropbox.com/s/o5o9eaewb9j57kt/FDM_Movie_data.zip?dl=1 Could you help have a check?

User sessions are managed via your IDP. Access token lifetime can vary from 60 to 90 minutes.So once the session expires, the user would normally have to sign-in again. How to overcome this?With the OIDC workflow, it is possible to retrieve a new access token without prompting the user to provide credentials again. This is done by finding a valid access token from cache or by finding a valid refresh token from cache and then automatically use it to redeem a new access token. The diagram below shows the normal OIDC workflow: Below you can find a sample code snippet which uses the acquire_token_silent method available through the class: msal.PublicClientApplication:def authenticate_azure(app): accounts = app.get_accounts() if accounts: print("Taking the token silently") creds = app.acquire_token_silent(SCOPES, account=accounts[0]) else: print("Taking token interactively") creds = app.acquire_token_interactive(scopes=SCOPES, port=PORT) return credsY

I have a transformation that is to write to a new dataset as seen here:SELECTconcat('CWS:',loc) as externalId,IF(parent_loc='' OR parent_loc IS NULL, '', concat('CWS:',parent_loc)) AS parentExternalId,CAST(lastUpdatedTime AS STRING) AS name,to_metadata(*) AS metadata,description AS description,7089382776719091 AS dataSetId FROM `CWS-Assets-DB`.`CWS-Assets-Tbl` The Preview seems to work fine.When RUN: I get following error:Request with id 96f4a082-9dde-9b41-8065-0b3cf0923197 to https://az-eastus-1.cognitedata.com/api/v1/projects/ra-istc-sandbox/assets/byids failed with status 403: Resource not found. This may also be due to insufficient access rights..The permissions on the dataset for my user group is raw:readraw:writeraw:listdatamodels:readdatamodels:writedatamodelinstances:readdatamodelinstances:writedatasets:readdatasets:writedatasets:ownertimeseries:readtimeseries:writefiles:readfiles:writeevents:readevents:writesequences:readsequences:write3d:read3d:create3d:update3d:deletetransfo

If  publishing a datamodel with TimeSeries attributes defined(like below)in the UI, it will fail with an error message:type Property {  modelName: String  value: TimeSeries} 

I am able to login with clientID and client Secret to our test project using cdf cli. Also the “cdf dm create” is working. However, when publishing a graphql using “cdf dm publish” there was always a validation error even using the demo GraphQL data model. Below is the error message:$ cdf dm publish --file=./test.graphql --external-id="testgraphql1" --space="testgraphql1" --version 1 --verbose2023-02-28T13:24:31.576Z cdf:middleware:auth Fetching auth token from CDF...2023-02-28T13:24:31.577Z cdf Getting token via Client Secret2023-02-28T13:24:31.578Z cdf:auth:client-secret Starting to aquire client credential token2023-02-28T13:24:32.032Z cdf:middleware:auth Finished fetching auth token from CDF2023-02-28T13:24:32.038Z cdf:data-models:publish dataModelVersionsHandler initialized2023-02-28T13:24:32.038Z cdf:data-models:publish Reading graphql file ./test.graphql2023-02-28T13:24:32.039Z cdf:data-models:publish Schema contents 'type Actor {\r\n' + '  name: String!\r\n' + '  age: Int\r\n'

A common setup in CDF (Cognite Data Fusion) is to first build an asset hierarchy, and then to attach events, timeseries etc. to those assets. One way to do it is to use transformations. In this short article, we’ll take events as an example. It works the same way for timeseries and sequences. Let’s assume we already have an asset hierarchy and we want to attach events to it. We’ll suppose that we have a RAW table, from which we want to create events, that looks like this:   In transformations, to attach an asset to a created event, you need to specify the corresponding asset ID (assetId in the target schema). Asset IDs being automatically generated, we usually prefer using external IDs because we know what they are made of: an ID from an ERP, a value following a naming convention, etc.  Since the asset ID is the expected value for the assetId field in the target schema, we need to retrieve for each asset its ID, based on its external ID. As mentioned in the documentation, we can read f

The coordinates of a document in CDF are normalized. In a normalized document, it has (0,0) coordinates in the upper left corner and (1,1) coordinates in the right bottom corner.Convert xml coordinates to normalized coordinatesTo convert xml coordinates to normalized coordinates you need to divide X coordinates with the width of the page and the Y coordinates with the height of the page. If the xml Y coordinate is 0 at the bottom of the page and the <height> is at the top of the page, the Y coordinate should be flipped as below.Normalized Y = 1- Normalized YOnce the coordinates are flipped;NewMinY = 1 - OldMaxYNewMaxY = 1 - OldMinY Example scenarioIf a user would like to add manual annotations to a P&ID file in CDF the user should convert the xml coordinates of the file to normalized coordinates.