Skip to main content

Developer and User Community

Join the discussions about delivering apps and data-driven decisions powered by Cognite Data Fusion. Click the + CREATE TOPIC button in the menu bar to start the conversation.

  • 835 Topics
  • 2,647 Replies
835 Topics
Sverre Dorheim
Committed
Sverre DorheimCommitted
posted in Developer and User Community
Manage data governance at scale in CDF

WhyWhen starting out in Cognite Data Fusion (CDF) project, it's natural to start by creating data governance elements like groups, datasets, and Raw databases from the CDF user interface. But as the solution begins to scale, you'll quickly realize that it is demanding to set up a detailed configuration handling multiple solutions, sources, roles, and other dimensions. For scaling, precise control is needed for access management and data governance and to enforce the guidelines and rules across the solution. The problem is not trivial, and a good way to solve it is by replacing the manual approach with a configuration-driven system, where the configuration language supports higher-level concepts for data lineage and access control. With configuration files as the foundation, you can set up an automated DevOps process and review and approve any changes to the structure before they are deployed. This approach also dramatically simplifies sharing the same configuration across multiple envi

1732
K
Practitioner
2 years ago
R
Active
Rajeev z ranjanActive
asked in Developer and User Community
Retrieving file sizes for CDF files

Hi Team,I am trying to get details of broken files uploaded in CDF . Is there any way to query actual file size from CDF using python sdk without downloading the file? Please add a feature to query file size using python sdk to get the corrupted/broken files. Thanks in advance.

1716
H
Committed
8 months ago
N
Committed
nabanitakarCommitted
asked in Developer and User Community
Can I use Asset Hierarchy in Grafana

My Question is : Is it possible to use  Asset Hierarchy in Grafana?I found many things related to timeseries  but not related  Asset Hierarchy to Grafana. Thanks. 

1718
K
Practitioner
1 year ago
R
Seasoned
RobersoSNSeasoned
posted in Developer and User Community
Aggregates on synthetic time-series

Hey!In our tooling for Power Analysts we compute synthetic-timeseries for them to evaluate scenarios of flow exceeding a threshold  value.     In the current implementation of  SyntheticTimeseries only Average and Interpolation aggregates are allowed. This leads to scenarios where zoomed out (and down-sampled) views of data computed through the SyntheticTimeseries API displays non-informative values. Consider the example below, where the top image is the un-aggregated addition of two timeseries, and the bottom is with the use of SyntheticTimeseries.  

1696
T
Seasoned Practitioner
1 year ago
M
Active
Morten Fredriksen
asked in Developer and User Community
Authenticator alternatives

I wonder if it is possible to use alternative authenticator apps to connect to Cognite HUB? I am asking because I’m already using another app for other sites.  

1683
S
Practitioner
3 years ago
K
Practitioner
Kristian Gjestad VangsnesPractitioner
asked in Developer and User Community
File extractor - SharePoint Online documentation

Hi,I was wondering where I can find information about how to set up the permissions for a file extractor reading only files from a few sites in SharePoint, i.e., not giving the extractor access to all sites. I have solved it using the GraphAPI, but this is quite cumbersome, so I was wondering if we have documentation on how this can be set up. I’ll post my solution below for clarity on how I solved the issue:_______________________________________________________________________________You will need to create two applications. One that is used in the file extractor and one to give access to the file extractor.Register the File Extractor App: Go to the Azure Active Directory portal. Click on "Azure Active Directory" then click on "App registrations". Click on "New registration" to register a new app. Grant API Permissions: Once your app is registered, click on the app's name to go to its dashboard. Click on "API permissions" in the left panel. Click "Add a permission". Choose "Mic

1686
Dilini Fernando
Seasoned Practitioner
8 months ago
K
Committed
kevinpengCommitted
asked in Developer and User Community
cognitedata / cognite-sdk-js

https://github.com/cognitedata/cognite-sdk-js/tree/master/samples/react/msal-browser-react, I am trying to run npm install and $ REACT_APP_CDF_PROJECT=... REACT_APP_AZURE_TENANT_ID=... REACT_APP_AZURE_APP_ID=... npm startgot the following error:node:internal/modules/cjs/loader:1024  throw err;  ^Error: Cannot find module '/Users/kevin.peng/code/cognite/cognite-sdk-js/samples/react/msal-browser-react/start'    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1021:15)    at Function.Module._load (node:internal/modules/cjs/loader:866:27)    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)    at node:internal/main/run_main_module:22:47 {  code: 'MODULE_NOT_FOUND',  requireStack: []}

1689
Dilini Fernando
Seasoned Practitioner
1 year ago
X
Seasoned
Xiaofeng WangSeasoned
asked in Developer and User Community
Could not create transformation for edges/relationships

It seems I can create transformations for types but there is no option to select a relationship when creating a transformation to a data model. Below it shows the “Target type and relationship” dropdown only has types but not relationships. Is there anything I missing?  

1676
X
Seasoned
1 year ago
R
Seasoned
RobersoSNSeasoned
posted in Developer and User Community
Interpolation in synthetic time-series

Hey!In our tooling for the power-analyst we make computations using the SyntheticTimeseries API. Results from these are used in subsequent analysis, where we have identified a problem for us.    When a synthetic timeseries is computed with a specified aggregate and resolution, that specification is returned  irrespective of there being data on the originating timeseries. In our case, we compute aggregates over long stretches of time which results in situations like the image below: In the figure, the opaque line is the comptued by addition of the two other signals. Spanning over roughly a year, the period in the middle has a linear rise in the SyntheticTimeseries while the originals are empty. These values are of course meaningless and should be omitted in the successive steps of the analysis. Have you considered implementing a density filter or the like for these types of situations? Or, do you believe this is best solved client side by identifying the holes prior to a set of Syntheti

1658
M
Practitioner
3 years ago
A
Committed
Anand Sankar AananthakrishnanCommitted
asked in Developer and User Community
OPC-UA Extractor Issues

Attn. @Jatin Sablok We found some issues with the Cognite pre-built OPC-UA extractor’s automatic re-connection to OPC server and to CDF as detailed in the test cases below.OPC-UA server reconnect test: OPC server is online and Cognite OPC-UA extractor is started. Then OPC server is restarted. Cognite extractor loses connection to OPC server, and is unable to re-connect automatically even after OPC server comes back online later. Extractor service has to be manually restarted to re-connect to OPC server.CDF reconnect test: Cognite OPC-UA extractor is started with network connection cable to CDF unplugged. Extractor log shows error messages. And extractor is unable to connect to CDF automatically even after network cable is plugged back in. Extractor service needs to be manually restarted to connect to CDF.Logs from both tests are attached. Could you please review them and let us know if these are known issues?

1656
A
Committed
1 year ago
thomafred
Seasoned
thomafredSeasoned
asked in Developer and User Community
CDF Files block writes `ReadError`

I have implemented block-updates to work with CDF Files. This by itself works quite well, as documented here.However, for very large files there are some problems. After some time, typically around 4 minutes, one of the block updates fail, with a simple `ReadError`. There’s no response from the API.Some logs from our application:DEBUG:odp.odcat.storage.cdffs.async_spec:Uploaded 133120 bytes to https://westeurope-1.cognitedata.com/api/v1/files/storage/cognite/6723545981847516/7406155183547467/wod_gld_2013.nc?sig=REDACTED&rsct=application%2Foctet-stream&sp=cw&sr=bDEBUG:odp.odcat.storage.cdffs.async_spec:Total bytes uploaded: 2660989807DEBUG:odp.odcat.storage.cdffs.async_spec:Data: len=5376000, MD5=6a0c0db1b66dac18ee28e8b082830667DEBUG:odp.odcat.storage.cdffs.async_spec:Buffer cursors: start=0, end=5242880, final=FalseDEBUG:httpx._client:HTTP Request: PUT https://westeurope-1.cognitedata.com/api/v1/files/storage/cognite/6723545981847516/7406155183547467/wod_gld_2013.nc?sig=RED

16516
thomafred
Seasoned
1 year ago
J
Committed
jaydeepCommitted
posted in Developer and User Community
Error running the Cognite replicator with Option 5

I have set up both the environment variables SOURCE_CLIENT_SECRET (Note that the Client secret is generated from the OID widget as per the documentation)and DEST_CLIENT_SECRET and running the replicator with the config file using option 5.I am getting error as per below logAMAC02Z3123LVCJ:oid-replicator j.subhash.parandekar$ poetry run python3 ./oid_replicator/replicate.py 2023-02-17 18:46:48,239 root INFO - Config file - Repeat line 5: 2023-02-17 18:46:48,239 root INFO - Config file - Repeat line 14: 2023-02-17 18:46:48,239 root INFO - Config file - Repeat line 23: Starting replication of resourcesReplicating assets...Traceback (most recent call last):  File "/Users/j.subhash.parandekar/Library/Caches/pypoetry/virtualenvs/oid-replicator-cwaK-6ym-py3.11/lib/python3.11/site-packages/cognite/client/credentials.py", line 364, in _refresh_access_token    token_result = self.__oauth.fetch_token(                   ^^^^^^^^^^^^^^^^^^^^^^^^^  File "/Users/j.subhash.parandekar/Library/Caches/py

16516
G
Seasoned Practitioner
1 year ago
Sofie Haug
Practitioner
Sofie HaugCognite Academy Instructor
published in Developer and User Community
Exciting news: Upcoming Cognite Data Fusion Bootcamps in Houston and Oslo

We are thrilled to announce the dates for our upcoming Cognite Data Fusion (CDF) Delivery Bootcamps in Houston and Oslo. Are you a customer or partner of Cognite working on a future CDF project? If so, this Bootcamp might be perfect for you! The Cognite Data Fusion Bootcamp is a 5-day intensive, in-person training designed to help you develop an end-to-end, production-ready solution with CDF. This program is ideal for technical users who will be ingesting and contextualizing data and creating solutions. To make the most of this Bootcamp, participants must complete the prerequisites and have experience with Python, poetry, SQL, Azure Active Directory, and GitHub/GitOps. Upcoming Bootcamp dates in Houston: September 23rd - September 27th 2024 October 21st - October 25th 2024 November 18th - November 22th 2024 December 9th - December 13th 2024 Upcoming Bootcamp dates in Oslo: October 14th - October 18th 2024 November 11th - November 15th 2024 December 2nd - December 6th 2024     

1640
Sofie Haug
Practitioner
4 months ago
E
Practitioner
eashwar11
asked in Developer and User Community
Timeseries creation - how to make it faster

Hi I have a final computed dataframe that looks like this. sample set alone listed here (15 rows) There are 570 records. I need to create timeseries for sumprod and derivedYields columns as per the value in Yieldcode. I have written the code and it is running and timeseries is getting created. But it is running slow. How to make it get done faster?Any best method? Please advise     sumprod Yieldcode Rank Group derivedYields MBBLD Vol LP-Class 1 532 0.009636 VBALFG1 1 VBAL 0.009636 3.524946 FG1 2 533 0 VBALLP1 2 VBAL 0 0 LP1 3 534 0.210269 VBALFN1 3 VBAL 0.218807 80.0382 FN1 4 535 0.037774 VBALLW1 4 VBAL 0 0 LW1 5 536 0.048263 VBALSK1 5 VBAL 0.094867 34.70158 SK1 6 537 0.017368 VBALHK1 6 VBAL 0 0 HK1 7 538 0.128149 VBALSR1 7 VBAL 0.142624 52.17094 SR1 8 539 0.014722 VBALDS1 8 VBAL 0 0 DS1 9 540 0.034342 VBALAG1 9 VBAL 0.0345

1607
H
Practitioner
1 year ago
A
Committed
Alex NarayananCommitted
asked in Developer and User Community
Read excel files (spreadsheet extractor?)

 Appears to be an obvious question, may have been asked before. Just wondering if there is an extractor available to read the spreadsheet files and load it into raw/staged data. The Excel can contain multiple spread sheet and perhaps with configuration each of that spreadsheet and its columns can be mapped to the staged tables/columns? 

1585
J
Seasoned Practitioner
8 months ago
Pierre Pernot
Seasoned Practitioner
Pierre PernotSeasoned Practitioner
published in Developer and User Community
Cognite Replicator

A common setup that we have at Cognite is having 3 different environments: development, test and production. We have one specific Cognite Data Fusion (CDF) project for each of these environments. This setup allows to develop new features without interfering with production data. Also, it makes the deployment of new features to production much safer, since they are tested before.Sometimes, you still want to have the data you used during your tests, in your production environment. For example, if you extracted some data from sensors and saved them as time series in CDF, you may want to keep this historical data when you move to production. Same goes with files, assets, datasets, functions etc. Another example could be the replication of the data from a full production CDF project, to one of your customer’s project, containing only a subset of the source one.At Cognite, we have a python package (available at https://pypi.org/project/cognite-replicator/) whose purpose is to copy the select

1584
R
Active
4 months ago
D
Committed
Devi Iragavarapu Sri SandhyaCommitted
asked in Developer and User Community
How to Extract data from Honeywell PHD Source System

Hi team,I have a Honeywell PHD source system. I have a got a suggestion to use Cognite OPC Classic Extractor. But I can see Extractor suggested is beta version. Can I know, it is efficient to use the Cognite OPC Classic Extractor, or we should go with custom extractor.If we use ,Cognite OPC Classic Extractor it connects over COM+ or DCOM to OPC servers and streams live and historical data into CDF. Can you please elaborate on COM+COM+ or DCOM connectivity to OPC Servers. How to connect with Honeywell PHD server using either of the ways?    Thanks

15811
D
Committed
1 year ago
Sumit Bondlewad
Committed
Sumit BondlewadCommitted
asked in Developer and User Community
asset transformation

I am doing asset transformation asset Source Target key   categoryId   sourceDb source parentExternalId parentExternalId updatedDate   createdDate   externalId externalId isCriticalLine   description description tag   areaId   isActive   I don't have all the columns for mapping to target schemaThese are the rest target columns what to do if the mapping column is not there? and where should i add the rest additional columns from source? parentId metadata name dataSetId labels

1579
Sumit Bondlewad
Committed
8 months ago
I
MVP
ibrahim.alsyedMVP
asked in Developer and User Community
Question: How to Leverage Drawings that are Contextualized and LIVE and Hotspotted?

scenarios we are working on, To show PI&D diagram with annotations, which help to select the asset and create a threat for that asset. Or draw a rectangle over the PI&D diagram to select multiple assets. For the above two scenarios we are referring APIs in,https://docs.cognite.com/api/v1/#tag/Engineering-diagrams Quickstart — cognite-sdk 5.3.1 documentation (readthedocs-hosted.com) We are successful in getting the PI&D as image by following the below steps,1. List Fileshttps://api.cognitedata.com/api/v1/projects/{project}/files2. Convert a diagram to image formathttps://api.cognitedata.com/api/v1/projects/{project}/context/diagram/convert/3. Get the results for converting an engineering diagram to an imagehttps://api.cognitedata.com/api/v1/projects/{project}/context/diagram/convert/{jobId} Could you please guide with APIs / JavaScript SDK methods we need to work on for the above mentioned two scenarios. 

1568
A
Seasoned Practitioner
1 year ago
X
Seasoned
Xiaofeng WangSeasoned
asked in Developer and User Community
Filtering based on externalID

Hi,To apply a startWith filter on a property, we can do:prop_view = client.data_modeling.views.retrieve(('myspace', 'myview'))filters.Prefix(prop_view[0].as_id().as_property_ref("myPropertyName"),"xxx")This is working fine. However, as the externalId is not a property so the code below doesn’t work:filters.Prefix(prop_view[0].as_id().as_property_ref("externalId"),"xxx")and the error message is: “Invalid prefix filter with property reference [myspace, myview/94b12855a09cf4, externalId]. View myspace:myview/94b12855a09cf4 does not have property externalId.” Could you help find out how to apply a startWith filter on externalId in python sdk? Regards,Xiaofeng

1548
X
Seasoned
1 year ago
jasar
Active
jasarActive
asked in Developer and User Community
Error when run a transformation from the course.

i need support for the course Python SDK Transformations.Im running the comands from notebook that was given in git and a stoped on:result = client.transformations.run(asset_transformation.id, wait=False)The error resulted is :CogniteAPIError: Invalid source/destination credentials: Could not authenticate with the OIDC credentials. Please check your credentials. | code: 403 | X-Request-ID: 78e0e54a-a3ca-9bc4-b2f5-e303ed6a7207iv checked the authentication process and i can perform all task like search list add datasets assets and everything but i cant run the transformation.Any one knows how to solve it ?i have the same problen in the UI  

15410
smrsgv
Committed
1 year ago
thomafred
Seasoned
thomafredSeasoned
posted in Developer and User Community
CDF Modelling Challenge - Open Lineage

Hi everyone,We in C4IR Ocean are starting a series where we are challenging the community to model a certain problem in CDF. The aim with this series is to facilitate discussion and invite community members to share interesting solutions and techniques.The first challenge focuses on Open LineageOpenLineage is an Open standard for metadata and lineage collection designed to instrument jobs as they are running. It defines a generic model of run, job, and dataset entities identified using consistent naming strategies. The core lineage model is extensible by defining specific facets to enrich those entities.In C4IR Ocean, we are dealing with data from multiple providers. Some datasets are open and publicly available, others are closed. It is therefore important to keep track of where data is coming from, and what transformations have been applied to the data after it is read from the provider.We are seeing a good landscape of data lineage solutions - both open and closed source. At the sam

1542
thomafred
Seasoned
2 years ago
E
Active
Erik PerssonActive
asked in Developer and User Community
OData service capabilities

We are interested in retrieving data from cdf through the odata service. We would do this not from Power BI, but from another Power Platform component, namely Dataverse. (the purpose would be to create "virtual tables" there which we could query, and in that way we could control access to the data more granularly then we are able to thorough the Cognite API) The documentation we find only mentions the OData connector in relation to Power BI (e.g. Cognite Power BI connector | Cognite documentation (cogniteapp.com) and Tips and best practices | Cognite Documentation). Is Cognite's OData service specifically tailored for Power BI, or is it a full implementation according to the OData standard? Is anyone able to recommend for or against attempting the proposed approach?In case the service implements the standard, does anyone know which version of the standard is it?

1542
E
Active
1 year ago
D
Committed
Dhande RushikeshCommitted
asked in Developer and User Community
Facing issue in ML model deployment

Hi Team I am trying to deploy ML Model into the AIR but Creating Env Variable Step is showing the error. Also attached image consisting issue details. Regards.

15114
D
Committed
2 years ago
thomafred
Seasoned
thomafredSeasoned
posted in Developer and User Community
CDF Modelling Challenge - SpatioTemporal Asset Catalog

Hi everyone,We in are running a series where we are challenging the community to model a certain problem in CDF. The aim with this series is to facilitate discussion and invite the community members to share interesting solutions and techniques.In this second entry of the series we will focus on the SpatioTemporal Asset Catalog (STAC).The SpatioTemporal Asset Catalog (STAC) specification provides a common language to describe a range of geospatial information, so it can more easily be indexed and discovered. A 'spatiotemporal asset' is any file that represents information about the earth captured in a certain space and time.The goal is for all providers of spatiotemporal assets (Imagery, SAR, Point Clouds, Data Cubes, Full Motion Video, etc) to expose their data as SpatioTemporal Asset Catalogs (STAC), so that new code doesn't need to be written whenever a new data set or API is released.(source)In C4IR Ocean, we are primarily dealing with geospatial data. The data originates from mult

1503
S
Practitioner
2 years ago

Badge winners

  • Cognite Data Fusion Fundamentals
    Jamwal Khushihas earned the badge Cognite Data Fusion Fundamentals
  • Cognite Data Fusion Fundamentals
    pranalipatkarhas earned the badge Cognite Data Fusion Fundamentals
  • Cognite Data Scientist Basics
    rafael.fiorihas earned the badge Cognite Data Scientist Basics
  • Cognite Data Fusion Fundamentals
    hmaruyamahas earned the badge Cognite Data Fusion Fundamentals
  • Cognite Data Fusion Fundamentals
    David Kinloughhas earned the badge Cognite Data Fusion Fundamentals
Show all badges
Terms & ConditionsCookie Settings