Developer and User Community

Join the discussions about delivering apps and data-driven decisions powered by Cognite Data Fusion. Click the + CREATE TOPIC button in the menu bar to start the conversation.

  • 683 Topics
  • 2,140 Replies

683 Topics

V
Practitioner
Vaibhav Narain
asked in Developer and User Community

Unable to create transformation for FDM model

Hi all, I am trying to create a transformation for my FDM below: When I create a new transformation or edit target of an existing transformation for the type: VT_TARGET & VT_TOTAL the screen gets stuck on saving and then throws a 500 error.   The type ‘WELL’ has no issues. the error crops up for the VT_TARGET & VT_TOTALS type. 

3
Dilini Fernando
Seasoned Practitioner
1 year ago
P
Committed
Pedersen Jon-RobertCommitted
asked in Developer and User Community

PI Extractor and Extraction Pipelines

Hi, we are using the PI Extractor with Extraction Pipeline and is pushing the config through the pipeline.But we noticed that after updating Config in extraction pipeline we have to restart the PI Extractor  to force it to use the updated config.If we update the config we can se in run history that the extractor is reporting “Seen” on the new config revision but it is not loading and using the new config.Is there a way to force it to use new config?

7
Dilini Fernando
Seasoned Practitioner
4 months ago
S
Committed
ShwetaCommitted
asked in Developer and User Community

Enable logging to be accessible from Extraction Pipeline UI

Hi Team, I want to change the message for the custom extractor’s run history message by python sdk. Right now, message says “successful shutdown” after successfully running the extractor, but I want to change it with how many datapoints are uploading and in how much time.I tried to do it by following methods:By using abstract class Extractor in our code to overwrite the _report_success method’s message.       2.By using this method  cognite.extractorutils.util.add_extraction_pipeline(extraction_pipeline_ext_id: str, cognite_client: CogniteClient, heartbeat_waiting_time: int = 600, added_message: str = '')       3. By using below class too:cclient.data_classes.extractionpipelines.ExtractionPipeline(id: int | None = None, external_id: str | None = None, last_message: str | None = None)  There is no way of passing the custom message here in my knowledge if it’s there let me know!And is there any way to pass logs in extraction pipeline?

6
S
Committed
2 months ago
K
Seasoned
Kristian NymoenSeasoned
asked in Developer and User Community

Grafana data source. Feature request / alternate solotions?

Hi.It would be useful to access metadata from time series in Grafana. Some of our time series contain simple integers 0, 1, 2..,  and the meaning of each value is a string described in a metadata field set on the time series. Could this functionality be added to the CDF Grafana data source? Or is there a way to access this information in Grafana through another Grafana plugin? I see that there is a CDF PostgreSQL gateway, but its documentation says that it’s only for writing data to CDF, so assume this cannot be used for fetching timeseries metadata through the Grafana PostgreSQL data source?  

7
K
Seasoned
2 months ago
I
Seasoned
ibrahim.alsyedSeasoned
asked in Developer and User Community

CDF: I use to see a link to Jupyter Notebook on CDF FrontPage.

I used to see a link to Jupyter Notebook on CDF Main Page. Now I dont, what happened?

3
K
Seasoned Practitioner
1 year ago
A
Seasoned
Adarsh DhimanSeasoned
asked in Developer and User Community

Client secret management in Python SDK

We are currently exploring workflow development on top of CDF python SDK. The client secret is used in the workflow code in order to connect with CDF project instance. This client secret is bound to expire after predefined interval. It will incur production downtime and manual intervention in order to update the secret.Can we have or do we have some concept similar to refresh tokens in CDF Python SDK.

4
A
Seasoned
1 year ago
A
Seasoned
Andreas KimsåsSeasoned
asked in Developer and User Community

Can data access rights be scoped by other means than datasets?

What are the alternatives for scoping read/write rights to different users?Dataset is the most intuitive way of limiting access rights to subsets of data, but if i would like to share data that is scattered across different datasets, but  easily identifiable by sharing a common label, are there any options? 

4
L
Practitioner
2 years ago
X
Seasoned
Xiaofeng WangSeasoned
asked in Developer and User Community

Which API should be used to query all the linked instances(through edges) based on a root node

type SimulationModel { modelName: String! nodes: [SimNode]}type SimNode { modelName: String input: [Key]}type Key{ modelName: String unit: String}With the above sample schema, I am able to fetch all the SimNodes together with their Keys for a specified SimulationModel instance using the GraphQL:query MyQuery{ listSimulationModel(filter:{externalId: {eq : "00000000-0000-0000-0000-000000000000"}}){ items { externalId modelName nodes(first:100){ items{ externalId modelName input{ items{ externalId modelName unit } } } } } }}How to do do the same with the APIs, could you give an example? Not sure which API should be used(instances/list, instances/byids, instances/search, instances/query). 

2
Dilini Fernando
Seasoned Practitioner
11 months ago
R
Active
Rajas AthaleActive
asked in Developer and User Community

Does CDF support timeseries ingestion with non numeric values

Does CDF support the direct ingestion of timeseries with non-numeric values ( enums).  If the workaround is to convert enums to integers to ingest, this is unnecessary overhead and a limitation. Also have to remember to convert the values back for visualization. 

7
Øystein Aspøy
Committed
3 months ago
Jo Øvstaas
Seasoned
Jo ØvstaasSeasoned
asked in Developer and User Community

Does FDM support RDF, SHACL and SPARQL

Hi from HUB Ocean and the Ocean Data Platform. We are participating in the open ocean data cataloging scene, for instance through the EU ILIAD “Digital twin of the ocean” project.Home | Iliad - Digital Twin of the Ocean (ocean-twin.eu)In order to improve the interoperability between the myriad of ocean data portals, web services and data catalogues out there, it will be very important for the Ocean Data Platform data catalog to support technologies and concepts like RDF (Resource Description Framework), SHACL (Shapes Constraint Language) and SPARQL (RDF query language). Is this already supported by FDM or is it on the roadmap?

2
Jo Øvstaas
Seasoned
1 year ago
Sumit Bondlewad
Committed
Sumit BondlewadCommitted
asked in Developer and User Community

asset transformation

I am doing asset transformation asset Source Target key   categoryId   sourceDb source parentExternalId parentExternalId updatedDate   createdDate   externalId externalId isCriticalLine   description description tag   areaId   isActive   I don't have all the columns for mapping to target schemaThese are the rest target columns what to do if the mapping column is not there? and where should i add the rest additional columns from source? parentId metadata name dataSetId labels

9
Sumit Bondlewad
Committed
3 months ago
R
Committed
robert.rillCommitted
asked in Developer and User Community

Executing transformation from Cognite Function

Is it a good practice to execute transformations from a Cognite Function?Our team implemented a Cognite Function which executes transformations on tables from Cognite RAW. We chose this solution as we wanted to parametrize the SQL query of the transformation, essentially applying a different filter on raw tables every time.Also, it would easier for the end user, e.g. a plant operator, to provide a JSON input to the function then to edit the query every time in multiple locations. E.g., if the JSON is:{ "plant": "Plant.1", "equipment": "equipment.1"}the user can just change the ID for the plant and equipment. Furthermore, these values can be validated by the function before making the replacements in the query and executing the updated transformations.Let me know your thoughts!

2
R
Committed
1 year ago
S
Practitioner
Shehan KarunaratnePrincipal Support Engineer
posted in Developer and User Community

How do I use OIDC credentials when I create transformations using python SDK

You need to provide “source_oidc_credentials” and “destination_oidc_credentials” run your transformation via API/SDKBelow is the code sample how to use OIDC credentials in your transformationstransformations = [ Transformation( external_id="<external_id>", name="<name>", query="<Query>", destination=TransformationDestination.raw("<Database>", "<Table>"), conflict_mode="<upsert>", source_oidc_credentials=OidcCredentials( client_id="<CLIENT_ID>", client_secret="<CLIENT_SECRET>", scopes=f"<scopes>", token_uri=<"TOKEN_URL">, cdf_project_name=<"COGNITE_PROJECT"> ), destination_oidc_credentials=OidcCredentials( client_id="<CLIENT_ID>", client_secret="<CLIENT_SECRET>", scopes=f"<scopes>", token_uri=<"TOKEN_URL">, cdf_project_name=&

0
S
Practitioner
2 years ago
A
Seasoned
Anders WillersrudSeasoned
asked in Developer and User Community

[Bug] Transformation access require group "tranformations" and legacy login

We are currently migrating to OIDC where we need to give access through access groups linked to Azure AD.On our Statnett cluster it seems that a user needs to explicitly be member of a group “transformations” in order to to delete (or edit) a transformation.The admin group has the capabilities  (on “test”) {'transformationsAcl': {'actions': ['READ', 'WRITE'], 'scope': {'all': {}}}}]But we need to login to the legacy login without OIDC and have a service account linked explicitly to the group “transformations” in order to delete a transformation. The group “transformations” has no capabilities set.I have tried both Fusion and the API/python-SDK (read is possible):CogniteAPIError: Transformation not found. This may also be due to insufficient access rights. | code: 403 | X-Request-ID: b7c0beb6-d3e0-9ec4-ba50-895533ac1996

7
C
Seasoned Practitioner
2 years ago
Juan Alvarado
Seasoned
Juan AlvaradoSeasoned
asked in Developer and User Community

Data Modeling API, how to create links between view instances.

Hi, I am trying to use the Data Modeling API (https://pr-ark-codegen-1646.specs.preview.cogniteapp.com/v1.json.html#tag/Instances-(New)/operation/applyNodeAndEdges) to create instances of Views I have defined, including a reference property from one view to another. The reference property in the view looks like this:                "facility": {                    "type": {                        "space": "jca-domain",                        "externalId": "EquipmentFacility"                    },                    "source": {                        "type": "view",                        "space": "jca-domain",                        "externalId": "Facility",                        "version": "0_1"                    },                    "direction": "outwards"                }I’m not including the whole definition as it is kind of lengthy, but basically there is a Facility view and an Equipment view, and the above property is on the Equipment so I can link it to a Facility. I am able

3
K
Practitioner
1 year ago
E
Practitioner
eashwar11
asked in Developer and User Community

CDF Extractors

I have to build two 2 types of extractors.1. PI extractor to connect to the PI server and fetch the data and ingest into CDF2. SharePoint online - Extract data from files present in SharePoint online and ingest into CDF. I wanted to know the construct in building the extractors. Mainly wanted to account for scenarios where the PI server is not available and PI extractor is unable to fetch the data from the pipeline. How to handle these kinds of situations and incorporate them in the code while building the extractor. Also, how to handle monitoring while performing extractor. Are there some sample code repos that can be referenced for getting complete idea for building extractors.    

11
Dilini Fernando
Seasoned Practitioner
10 months ago
H
Practitioner
Håkon V. TreiderPractitioner
published in Developer and User Community

Cognites guide to code quality in Python - Part 2

Introduction In this post we continue to share some of our internal material aimed towards solution builders, such as data scientists, who want to develop their ability to develop high quality solutions by creating more reliable, maintainable and readable code. This is the second part of 2.MotivationHigh code quality is easy to recognize but can be very hard to describe concretely. The assumed benefits are easier maintainability, modifiability, and more. While code style, like formatting, can be a matter of different taste, most parties agree that other code practices that fall under the umbrella term “anti-patterns” should be avoided. To stop endless formatting discussions and the like, having (and adhering to the industry) standard makes reading and understanding code across repositories easier.What this guide is notThis guide will not tackle the topic of  “how to set up a Python project” the right way . Please let us know in the comments if you would like us to share more of our ex

990
H
Practitioner
1 year ago
Y
Active
Yiming DengActive
asked in Developer and User Community

Error - Finding Timestamp is too high

Hi everyone, I tried to upload my own dataset(time-series data) into CDF, but I always get error: Request with id 87b235ea-78dd-963a-b343-ff58a0ee084d to https://westeurope-1.cognitedata.com/api/v1/projects/learn/timeseries/data failed with status 400: Timestamp is too high, must be at most 2556143999999 which is 31-12-2050 23:59:59 GMT.I followed with this link:https://learn.cognite.com/path/cognite-data-fusion-fundamentals/working-with-cdf-integrate. I created an assest, 3 time-series. What I want to do is to upload my data to the 3 time-series, and show the variation of my data in CDF. I change the time_stamp of my data into the format of 2022-02-10T17:00:00.000Z, I am not sure if this is necessary but when I preview my datapoints in RAW explorer and CDF automatically change the this format into for example, 164998440000, so I am wondering if this is why I have this error? I also want to ask what is the meaning of span in IFSDB.sensors? It means this time-series is 100 days? or 100

1
K
Practitioner
1 year ago
I
Seasoned
ibrahim.alsyedSeasoned
posted in Developer and User Community

Cognite Reveal: Cannot handle Large Point Clouds

Cognite RevealWe are being told that our models need to be downsampled which is resulting in better performance in CDF but poor quality on the viewer. We have consistently noticed that we are unable to have one large point cloud together in the viewer and also the viewing quality is very poor compared to what laser scanning vendor solutions show. I need some attention on this from Product?Why should we downsample and decrease the density of the Point Cloud? does not appear to be the right thing to doWhy should we chop up the 3D model in pieces?

5
Lars Moastuen
Seasoned Practitioner
1 year ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
posted in Developer and User Community

New feature: Calculations are now running on up to 100k data points

As we announced in our last release post, we recently updated our calculations backend to run on individual data points if the total count of data points is less than the predefined maximum limit.(You can watch the video walkthrough for more information.)When this improved functionality was released a few weeks ago, this limit was set at the intentionally low value of 10,000 individual data points. We did this to test our infrastructure, gather feedback, and ensure our backend will not crash with these more expensive requests — Thanks to those of you who have provided us with input!As of today, we have released an update which increases this limit from 10,000 data points → 100,000 data points.This 10x improvement in performance will help to provide accurate, trustworthy calculation results for larger ranges of time and data. With this new 100k limit, we’ve reached the maximum number of data points we can retrieve from the Cognite Data Fusion Time series API with a single request.While

3
Philippe Bettler
Practitioner
1 year ago
I
Seasoned
ibrahim.alsyedSeasoned
asked in Developer and User Community

COGNITE REVEAL SDK: POINT CLOUD AND 3D

I realizd that when leveraging Cognite REVEAL in application development, can we not have Point Cloud/360 Images annd 3D in one REVEAL? Why do i have to select which one? is that not embedded within the same where i can turn on or off the Point Cloud or 360 Images?

2
C
Practitioner
1 year ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
posted in Developer and User Community

Latest release of Cognite Data Fusion: Charts Updates

Today is the December release of Cognite Data Fusion. As you might have seen in last week’s Product Tour or read in today’s release post, there are several new and exciting Charts-specific updates in the latest release, namely...Charts is now an integral part of Cognite Data Fusion (fusion.cognite.com).A long awaited improvement, charting and no-code analytics finally lives alongside all other industrial DataOps functionalities available in Cognite Data Fusion. No URL switching and additional sign-in required.For those of you who have been using Charts before, all private and public charts that you and your colleagues have created will still be available via Cognite Data Fusion (fusion.cognite.com).Note that charts.cogniteapp.com will still work and be available. This teams, for those of you who are linking to Charts in your third party applications, Microsoft Teams, etc., this will still function exactly as expected. We will be working on an automatic redirect for coming releases, how

3
Eric Stein-Beldring
Seasoned Practitioner
1 year ago
David Liu
Seasoned Practitioner
David LiuSeasoned Practitioner
posted in Developer and User Community

Flexible Data Modeling: Bug/Feature Request

Found a bug? Have a question about how something works? Want something new with Flexible Data Modelling? We want to hear about it!You can choose to either create a dedicated post (topic) in the Flexible Data Modelling group by clicking the Create topic button OR simply post a reply below in this thread.Be sure to tag with “bugs” or “feature request” Features requests You can choose to either create a dedicated post (topic) in the Charts group by clicking the Create topic button OR simply post a reply below in this thread. Remember to say whether your feature request is Nice to have, Important, or Critical to you and why. Screenshots, sketches, or explanatory videos are also encouraged. We will follow up and share progress on features periodically as well! Bugs Remember to include a screenshot or video to help the product team best understand what exactly you’re talking about or referring to.

2
David Liu
Seasoned Practitioner
1 year ago
H
Committed
Harshita SaiCommitted
asked in Developer and User Community

How to write the unit test for the list row?

Hi Team, I need help to write the unit test using pytest,below is the code:    row_list = cognite.raw.rows.list(file.destination.database, file.Extractor_Last_Run_Table, limit=-1,                               columns=["key", "LastUpdatedDatetime"]).to_pandas()    if not row_list.empty:        val = list(row_list[row_list.index.isin([file.Extractor_key])]['LastUpdatedDatetime'])Could you please help to write the unit test for above code?Thanks

4
H
Committed
5 months ago
Ben Brandt
Seasoned
Ben BrandtSeasoned
asked in Developer and User Community

Similar to 365d-ago for historical, how do we specify relative time for future?

For the Retrieve data points API and a few other API’s that allow date range input, we are able to specify a string where, “The format is N[timeunit]-ago where timeunit is w,d,h,m,s.”  This is able to specify a time in the past.  I am now trying to specify end as “-365d-ago” to set a date 365 days in the future, but this is giving an error.Is there another syntax available for specifying future dates?N[timeunit]-afterNow?365d-afterNow?

4
Ben Brandt
Seasoned
2 years ago

Badge winners

  • Cognite Data Engineer Basics - Integrate
    Sumedh Vaidyahas earned the badge Cognite Data Engineer Basics - Integrate
  • Cognite Data Engineer Basics - Integrate
    Parth Sinhahas earned the badge Cognite Data Engineer Basics - Integrate
  • Working with Cognite InField
    marksasshas earned the badge Working with Cognite InField
  • Cognite Domain Expert Basics
    MihaiCurachas earned the badge Cognite Domain Expert Basics
  • Working with Cognite InField
    nurfarhahas earned the badge Working with Cognite InField
Show all badges
Terms & ConditionsCookie Settings