Developer and User Community

Join the discussions about delivering apps and data-driven decisions powered by Cognite Data Fusion. Click the + CREATE TOPIC button in the menu bar to start the conversation.

  • 691 Topics
  • 2,167 Replies

691 Topics

Pierre Pernot
Practitioner
Pierre PernotPractitioner
posted in Developer and User Community

How to use contextualization with the SDK

Contextualization is one of CDF’s core strengths. It allows you to link resources from different data sources together, which can be pretty powerful in an industrial context. We can for example think about linking an asset hierarchy based on SAP, to some time series extracted from sensors. You can contextualize CDF resources in the UI. In case you need more filtering capabilities, or you want to automate that, you can also do that programmatically.  In this article, we’ll take the example of contextualizing files and assets with the SDK. The setup we will use will look like that:20 assets  20 files, for which something similar to the asset name appears in a metadata field called “asset” It is often the case that files and assets don’t share something obvious to make a link on, because they might be ingested from different sources. That is where contextualization comes into play, using Machine Learning to find matches between resources based on specified fields. To create the setup, we

3
E
Seasoned Practitioner
4 months ago
V
Committed
Vetle NevlandCommitted
asked in Developer and User Community

Isolated calculations in Charts

Are the calculations provided in Charts (originating from `indsl` python library) available for all CDF projects? I’m wondering if it’s possible to contribute with new calculations to Charts that are written in `indsl` but only deployed to the specific CDF project you’re working on (i.e., not accessible for people outside this project)?I want to write new calculations using indsl that SMEs in my team can use in Charts for specific time series analysis, but I think it would be best to keep them isolated within our CDF project to not “overflow” the global Charts with potentially lots of new calculations that may mostly be relevant for my team/company.Any response is appreciated, thanks :)

5
V
Committed
4 months ago
V
Committed
Vetle NevlandCommitted
asked in Developer and User Community

Importing calculations to Cognite Charts

In Charts you can import your own calculations, but I can’t find any documentation regarding requirements to this end. It seems like the required format is a JSON file, but how should this JSON be set up. If anyone has any examples of imported calculations, sharing them would be very much appreciated.Thank you.

1
K
Seasoned Practitioner
4 months ago
A
Seasoned
Adrian DamsSeasoned
asked in Developer and User Community

Configure an OpcUA Extractor to push a NodeSet to CDF

I am trying to configure an OpcUA extractor to push a NodeSet2.xml file to CDF. I have it basically setup and if I run it, there are no obvious errors, but I don’t see any kinds of assets being imported.  Am I expecting too much? Is the functionality in place to import an OpcUA Nodeset as Assets into my CDF project. If anyone has this working I would really appreciate a copy of the config file you are usingThanksAdrian

7
D
Committed
4 months ago
K
Seasoned
Kristian NymoenSeasoned
posted in Developer and User Community

Monitoring CDF operations

Hi! We’re aiming to monitor all of our CDF operations in the solution we have for monitoring other systems and data flows (Splunk). This would involve monitoring of extractors/extraction pipelines, cognite functions, and general platform availability. I wonder if any other Cognite customers have done this, and if you would be willing to share your experience / architecture for extracting logs from cognite functions etc. into Splunk or a similar service? Cheers,Kristian 

1
Anita Hæhre
Seasoned Practitioner
4 months ago
T
Active
tabdoolActive
asked in Developer and User Community

Synthetic time series query not supporting IF statements in 'expressions' parameter

I am trying to create a synthetic time series query that looks are two other time series and if those values meet a certain condition, is outputs a 1 or 0 as the synthetic time series but I keep getting a syntax error that I can’t solve. Below is a code sample. temp_id_list = [3308928194658464]flow_id_list = [1256034258207391, 3564715377369546]for temperature in temp_id_list:    temp_expression = f"TS{{id:{temperature}}} < 200"    flow_expression = " + ".join([f"(TS{{id:{flowrate}}})" for flowrate in flow_id_list])    final_expression = f"if({temp_expression} and {flow_expression} > 0, 1, 0)"    cv_query = client.time_series.data.synthetic.query(expressions=final_expression, start="1d-ago", end="now")

2
M
Practitioner
5 months ago
D
Committed
Dhande RushikeshCommitted
asked in Developer and User Community

Facing SSL while using CDF SDK.

I am trying to read timeseries data from CDF on my local but facing SSL Error also showing cognite.client.exceptions.CogniteConnectionError.[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)  

5
M
5 months ago
N
Seasoned
Neerajkumar BhatewaraSchlumberger
asked in Developer and User Community

OPC UA Extractor - is there a parameter to handle the source timezone similar to DB Extractor

Hello,In DB Extractor, I see a parameter that could be configured to specify how to treat the timezone parameter.  Is there something similar for OPC-UA Extractor as well ?If not how can the different timezone data could be handled or atleast Local timezone data ?

3
E
Practitioner
5 months ago
A
Active
andcondallActive
asked in Developer and User Community

Download openapi spec in CI

Trying to download the openapi spec over at https://api-docs.cognite.com/20230101/ by using the URL in the “download” link fails with 404 when trying to use the URL with wget/curl and “ADD” in a dockerfile. It seems like the link is generated behind the scenes during pageload by javascript. Is there a way to download the openapi spec in scripts e.g for use in CI pipelines and similar?

2
A
Active
5 months ago
A
Active
andcondallActive
asked in Developer and User Community

Generate client from CDF openapi spec for Go

I am unable to generate client code for the CDF openapi spec at https://api-docs.cognite.com/20230101/ using deepmap/oapi-codegen for Go, seemingly because the spec contains some errors. Do anyone know of a way to generate an api-client in go for the CDF-api spec? Both commands below rely on having openapi spec downloaded to the current directory from https://api-docs.cognite.com/20230101/ Testing the spec for errors with redocly/cli: docker run --rm -it -v $PWD/swagger.json:/swagger.json redocly/cli lint /swagger.jsonFor me this results in “Validation failed with 25 errors and 382 warnings” Generating the client library using deepmap/oapi-codegen: go run github.com/deepmap/oapi-codegen/v2/cmd/oapi-codegen@v2.0.0 --config oapi-codegen-config.yaml swagger.jsonwith the this config file:package: "cdf_api"output: "cdf_api.gen.go"generate: models: true client: truecompatibility: circular-reference-limit: 100For me this results in the message “error generating code: error creating operat

0
A
Active
5 months ago
K
Active
Karguvel kalisekarActive
asked in Developer and User Community

Python SDK - List, Search, Query Instance from FDM

Hi Team,Is there any sample code available for retrieving a list of instance objects from FDM? I've reviewed the documentation's sample code, but I need additional clarification on where to provide Space, Object Type, Version, and Data Model.  Sample code available for Instance List. instance_list = c.data_modeling.instances.list(limit=5) I have also followed the steps outlined in the document URL below.. advancedListInstance queryContent searchInstances Thanks,Karguvel K

3
K
Active
5 months ago
G
Active
grc5084Active
asked in Developer and User Community

Issue Running Transformation to get data into CDF

Hello, I am going through Cognite Data Fusion Fundamentals and am having an issue that I cant figure out. After multiple attempts I am loading 1 row of data to try and figure out my issue.I make it to the “Preview the Transformation” step:  The next step is to “Run the Transformation” step and this is where I am running into an issue. I get the below error message and theres no suggestion on what to do to correct my apparent error. Any idea what to do or how to progress? I just want to finish this course. -George

2
Dilini Fernando
Seasoned Practitioner
5 months ago
R
Active
rmawais791Active
asked in Developer and User Community

CDF Data Points Benchmarks with OBDC and OPC UA to read data from other application

Hi I have an application to read data from and I have two options either to use ODBC or OPC UA directlyWhat are the benchmarks or limitations when it comes to aboveI have 30,000 data points to read with sample time of 1s and 5s and 10s 

2
Dilini Fernando
Seasoned Practitioner
5 months ago
R
Committed
Rajendra PasupuletiCommitted
asked in Developer and User Community

How to upload excel files to CDF files

How to upload excel files to CDF files, I am looking for ways to upload files programatically and schedule this using cron expression.  

7
Dilini Fernando
Seasoned Practitioner
5 months ago
D
Active
Dexter NguyenActive
asked in Developer and User Community

Problem with ingesting datapoint immediately after Timeseries object creation with multiple concurrent jobs.

We’re attempting to execute 10 concurrent jobs, and each one will create approximately 20 Timeseries objects, immediately ingesting datapoints for these newly created Timeseries objects in a loop.However, if we try to run 500 similar jobs with 10 concurrently running at a time, about 10% of the jobs fail due to a ‘Timeseries not found’ error.We theorize that the creation of a Timeseries object can take some time. Therefore, immediately ingesting the datapoint might fail because Cognite doesn’t refresh quickly enough to find the corresponding Timeseries object. This issue only occurs when we attempt to run multiple concurrent jobs.Has anyone else encountered similar issues? 

4
Dilini Fernando
Seasoned Practitioner
5 months ago
P
Committed
Pedersen Jon-RobertCommitted
asked in Developer and User Community

PI Extractor and Extraction Pipelines

Hi, we are using the PI Extractor with Extraction Pipeline and is pushing the config through the pipeline.But we noticed that after updating Config in extraction pipeline we have to restart the PI Extractor  to force it to use the updated config.If we update the config we can se in run history that the extractor is reporting “Seen” on the new config revision but it is not loading and using the new config.Is there a way to force it to use new config?

7
Dilini Fernando
Seasoned Practitioner
5 months ago
G
Active
grc5084Active
asked in Developer and User Community

Uploading CSV to Staging Area

Hello,I have been trying to load a CSV file into a table while going through the Cognite Data Fusion Fundamentals course. I have tried for over an hour to load the data correctly. One of the column headers adds 2 commas after the text and is throwing off the SQL formula. I am using the stock CSV file provided in the course, loading it straight from my desktop and not making any changes to the file itself.   

4
Dilini Fernando
Seasoned Practitioner
5 months ago
G
Committed
gghalebCommitted
asked in Developer and User Community

state for OPC-UA extractor

Hi There,How do you configure the state of OPC-UA extractor in the Raw layer?I know that you can do it through various types of state stores including InfluxDB or local, but I have a dataset and a table in the Staging area which I would like to get populated.Thanks in advance!

2
G
Committed
5 months ago
Chris Selph
Seasoned
Chris SelphSeasoned
asked in Developer and User Community

Cognite DB Extractor 3.0.0 Stops After Execution

Running Cognite DB Extractor 3.0.0 as a Windows Service. Note that previous versions of this extractor have worked fine although some changes have been made to the config.yml since upgrading. Symptoms - According to the log, Upon starting the service, It seems the extractor will run and successfully query and upload (30 rows in this case) and then the service will STOP running immediately as indicated in the Windows Services app.There is an error in the log file about database/table already existing which does not make sense to me b/c usually errors will prevent the service from starting at all.  https://www.yamllint.com/ indicates the config.yml is valid. The only entry in Windows Application log is an “Informational” note that states:The description for Event ID 4098 from source DBExtractor cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer

5
Chris Selph
Seasoned
5 months ago
K
Seasoned
Kristian NymoenSeasoned
asked in Developer and User Community

3D models, scoping to data sets

Hi. What is the recommended method for access control of 3D models? 3D capabilities can be scoped to data sets, but how can I assign a model to a particular data set, so that it is only accessible by a limited user group? Is it possible via the Python SDK or GUI?

2
Andre Alves
Seasoned
5 months ago
K
Seasoned
Kristian NymoenSeasoned
asked in Developer and User Community

Access control. Sharing data.

Hi. When setting up CDF we were recommended to use CDF data sets scoped to data sources to control access to data. However, we have use cases where we would like to share a selection of data from a source with external parties. What would be the recommended setup for this? Would we need build an API layer on top of CDF,  to copy data from a source dataset to multiple other datasets for sharing? Or could the security categories feature be used for this?The person in charge of sharing data in our case will often be a data owner / SME without much coding experience. Is there a way our data owners could move/copy a subset of data from a source dataset to a sharing dataset, or somehow label individual time series or assets in a manner making them accessible to specificaccess groups?

4
N
Practitioner
5 months ago
P
Seasoned
Peter QuinnSeasoned
asked in Developer and User Community

Is there a InfluxDB extractor?

We have customers using InfluxDB as a local on-prem storage mechanism. What’s the best extractor to use to push this data to CDF? I’ve heard OPC-UA is the way to do it, but what I find online are ways to store OPC-UA data into Influx, but not ways to get InfluxDB data into a OPC-UA server so I could use the OPC-UA extractor to get it into CDF.

4
P
Seasoned
5 months ago
E
Active
Espen JacobsenActive
asked in Developer and User Community

Retrieving many raw datapoints from many timeseries

Hi We have a web application that uses the cognite sdk to present timeseries and datapoints.We have two use cases:1. We have a case with 500 time series, and we need to retrieve the latest 1000 data points from the combined 500 time series.2. We have a case with 500 time series, and we need to retrieve all datapoints between two dates. For both cases we also need to know which timeseries each datapoint belongs to and we need raw values and can not use aggregates.What is the most effective approach to achieve this?

8
E
Active
5 months ago
H
Committed
Harshita SaiCommitted
asked in Developer and User Community

I would like to write unittest for Extractor run main function, Please help me to find mistakes in my code.

Hi Team,Please help me to find the mistakes in my code. Let me showcase the dummy codemain code:def main() -> None:    """    Main entrypoint    """    BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))    config_values_vault.set_vara()    with Extractor(        name="SAP_Extractor",        description="An extractor to extract asset hierarchy from SAP based on root node",        config_class=SapConfig,        # version=__version__,        # debugger        version="1.0.0",        run_handle=run,        metrics=metrics,        config_file_path= os.path.join(BASE_DIR, 'config.yaml'),    ) as extractor:        extractor.run() I have build code unittest for above code: def test_main():    with patch('os.path') as path_mock:        with patch.object(path_mock, 'abspath') as mock_abspath:            with patch.object(path_mock, 'dirname') as mock_dirname:                with patch.object(path_mock,'join') as mock_join:                    mock_abspath.return_value = '/p

5
Anita Hæhre
Seasoned Practitioner
5 months ago
R
Seasoned
Rimmi AnandSeasoned
asked in Developer and User Community

Search in JSON object type column data

I am looking for function/query that can search in  JSON type data against array of values. Json can have nested keys. I tried using JSON_SEARCH or JSON_CONTAINS . getting Undefined function:JSON_SEARCH(data->'$[*]', JSON_ARRAY("abc")) Is there a way to search in JSON object keys for list of values ? for e.g :JSON col data {"name":"Back","address":{"location":{"country":[{"city":{"state":[‘jkl’,’fgh’]},"type":"home"},"empId":"e100043867","FullName":"Back"}JSON_ARRAY = [‘name’,’state’,’country’]check if JSON_ARRAY values exists in JSON data ?Any help would be appreciated.

3
Dilini Fernando
Seasoned Practitioner
5 months ago

Badge winners

Show all badges
Terms & ConditionsCookie Settings