Skip to main content

Developer and User Community

Join the discussions about delivering apps and data-driven decisions powered by Cognite Data Fusion. Click the + CREATE TOPIC button in the menu bar to start the conversation.

  • 803 Topics
  • 2,539 Replies
803 Topics
Emel Varol
Practitioner
Emel VarolPractitioner
posted in Developer and User Community
Check out our course Integration Fundamentals!

New to data transformations? The course Integration Fundamentals walks you through the data’s journey from source systems through the staging area to being transformed into Cognite Data Fusion (CDF) resource types. Learn what the typical uses of transformations are and watch a demo the built-in transformation tool in CDF.After completing this course, you’ll be able to:Summarize the data’s journey from source systems to the staging area. Explain why it is important to write data to RAW first. Explain why we use transformations. Describe tools you can use to extract and transform data.This course is independent but also part of the Cognite Data Fusion Fundamentals learning path. We recommend that you complete all courses from this path and take the assessment at the end to receive a certificate of completion and the Onboarding badge here on Cognite Hub.Any questions? Let us know!Let’s go! Integration Fundamentals

0
Emel Varol
Practitioner
2 years ago
Sunil Krishnamoorthy
Seasoned Practitioner
Sunil KrishnamoorthyProduct Manager
posted in Developer and User Community
APIs and Python SDK for SQL Transformations

Hey there! I am Sunil, Product Manager for Data Integrations. Since this is our first post in the Hub, let me introduce my team and what we do. My team is responsible for the Ctrl+C Ctrl+V of your data from source systems to Cognite Data Fusion . But on a serious note, we are responsible for ELT - Extract, Load, and Transform of data. We are thrilled that you are interested in learning about the Transformations Python SDK. Yayy!In this post, I list a couple of use-cases/scenarios where the python SDK for transformations makes managing and orchestrating transformation jobs simple and seamless. Our goal is to make the transformations SDK more developer-friendly and would like to hear your feedback. Do check out the API and SDK docs for more details, but now on to a few use cases! Use Case 1: Triggering TransformationsCognite Data Fusion helps liberate industrial data silos and provides a unified and contextualized view of the data. It is now easy to derive insights and build application

0
Sunil Krishnamoorthy
Seasoned Practitioner
2 years ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
posted in Developer and User Community
Charts Product Update: New version of calculations backend ⇒ Faster calculations, bug fixes, ++

If you’ve been feeling like that calculations in your Charts are running faster than before… they are!Yesterday, we released a new version (V2) of our calculations backend to production. This update includes  numerous performance improvements, bug fixes, and quality improvements.In short, it’s an all-around better experience built on higher quality code.Test it for yourself on your CDF project. As always, let us know in this group if you have any questions or feedback!

0
Eric Stein-Beldring
Seasoned Practitioner
2 years ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
posted in Developer and User Community
Thanks for joining yesterday's Charts webinar! Here's the link to the recording.

Thank you again to everyone who joined us live for yesterday’s Product Spotlight webinar! To those of you who couldn’t make it to the live event or if you want to re-watch, you can take a look at the recording, below.If you want to read about the the details of the October CDF release, you can visit the post in Product Updates by clicking here. We already received some great feedback and we’re happy to hear several of you found the product demo with a practical use case example to be helpful. If you haven’t already submitted your feedback about the webinar, please do so here. We’re reading and discussing each response so we can make all of our future product webinars as valuable as possible. We’re already planning a new event about Charts, exclusively available for all of you in our Early Adopter community, so stay tuned for updates and invitations.Have a great rest of your week, everyone!P.S. We also appreciate everyone’s patience with the technical difficulties that were experienced

0
Eric Stein-Beldring
Seasoned Practitioner
2 years ago
Eric Stein-Beldring
Seasoned Practitioner
Eric Stein-Beldring
posted in Developer and User Community
Product Spotlight Webinar on November 3rd, 2021: Special focus on Charts

Hello Charts Group! On November 3rd 15:00-15:45 CET, we will be hosting a live webinar to present highlights from the latest CDF release, with a special focus on CHARTS AND THE CHARTS EARLY ADOPTER GROUP.You will learn about why we’re building Charts, what problems it can solve, and how to use key features and functionalities. We’ll demonstrate how to leverage this new CDF functionality as we build and solve real use cases in Charts step by step. From finding the relevant data, to using P&IDs to gain a systems understanding, to configuring the chart, to creating the no-code calculations to solve the problem at hand. This will be an interactive session for asking questions, discussing ideas, and providing feature requests directly with the team. By the end of this session, you’ll feel confident about using Charts in your day-to-day work and feel empowered to think creatively about the solutions you could build to shape a safer, more efficient, more sustainable industrial future.You

0
Eric Stein-Beldring
Seasoned Practitioner
2 years ago
thomafred
Seasoned
thomafredSeasoned
posted in Developer and User Community
CDF Modelling Challenge - Open Lineage

Hi everyone,We in C4IR Ocean are starting a series where we are challenging the community to model a certain problem in CDF. The aim with this series is to facilitate discussion and invite community members to share interesting solutions and techniques.The first challenge focuses on Open LineageOpenLineage is an Open standard for metadata and lineage collection designed to instrument jobs as they are running. It defines a generic model of run, job, and dataset entities identified using consistent naming strategies. The core lineage model is extensible by defining specific facets to enrich those entities.In C4IR Ocean, we are dealing with data from multiple providers. Some datasets are open and publicly available, others are closed. It is therefore important to keep track of where data is coming from, and what transformations have been applied to the data after it is read from the provider.We are seeing a good landscape of data lineage solutions - both open and closed source. At the sam

2
thomafred
Seasoned
2 years ago
Z
Active
ZachHActive
asked in Developer and User Community
Does CDF support MQTT and Azure IoTHub for Time Series Data?

In the CDF docs I see there is a PI and OPCUA connector. Many industry 4.0 community members see MQTT rather than OPCUA as the future of Industrial IoT cloud based communication, as it is lightweight, report by exception, client driven, etc…If I want to onboard a facility that has an IoT MQTT gateway, how does Cognite recommend we integrate? I have used Azure IoTHub and Azure Functions to connect and transform the data in the past. Do you have a reference architecture for this? Can we connect to the gateway directly with a CDF connector, or do you recommend using Azure IoTHub and Functions (or similar GCP/AWS)?Also Azure has some neat tools like IoT Device Provisioning Services, be nice to know a POV on if/how to utilize those. Thanks! 

2
Z
Active
2 years ago
thomafred
Seasoned
thomafredSeasoned
asked in Developer and User Community
CDF Templates API documentation and itroduction

Hi there,I am looking for resources on CDF templates API.So far I the best introduction I have found is the Python SDK documentation, as well as the information at docs.cognite.com which focuses on using templates in fusion.Can you provide more information about how to use the templates API?

3
J
Practitioner
2 years ago
N
Active
Niros ManoharanActive
asked in Developer and User Community
Data Streaming with CDF

I am trying to stream data from CDF to Azure Event Hub with the Python SDK and cannot find anything related to streaming datasets. Only option so far (as I know) is dps = c.datapoints.retrieve_latest(id=184691546499795)which would need a trigger of some sort to keep running. Data from CDF are so called timeseries from different types of sensors. Is there any documentation on Streaming Data for CDF that I could look at or is Streaming really supported? 

1
K
Practitioner
2 years ago
R
Seasoned
RobersoSNSeasoned
posted in Developer and User Community
The cognite python SDK should batch requests that violate API constraints

Hi, When developing with the Cognite python SDK, a common restriction is  the API imposed restrictions on queries. Quering time-series, for example, is restricted by 100 asset-ids.  I believe the Python SDK should recognize these constraint violations and batch / concurrently dispatch requests in chunks that do not violate constraints. Optionally with a performance-warning to the developer.Is this sane, or do you think that the developer/customer should maintain a wrapper on the SDK for batching each endpoint (as we’ve currently done at Statnett)?

3
R
Seasoned
2 years ago
A
Practitioner
Arild EidePractitioner
asked in Developer and User Community
Feature request CDF login form: Strip protocol from override tenant field

 If a user includes https:// or http:// in the Override Azure Tenant field the url to Azure AD will not include a correct tenant and login is prevented. This has caused login issues for at least a few users.  The request is simply that the form is modified to strip the protocol scheme, ie removing http:// or https:// automatically, if present.  

1
Krystele Uy
Practitioner
2 years ago
R
Seasoned
RobersoSNSeasoned
asked in Developer and User Community
Getting proprietary code to Cognite Functions

Hey, In our current workflow we’re expanding the use of Functions as a tool. We’re somewhat hampered by how “clumsy” it is to bundle and version control proprietary dependencies. Is there something in the pipe to address this issue? Kind regards,Robert

3
A
Seasoned Practitioner
2 years ago
R
Seasoned
RobersoSNSeasoned
posted in Developer and User Community
Interpolation in synthetic time-series

Hey!In our tooling for the power-analyst we make computations using the SyntheticTimeseries API. Results from these are used in subsequent analysis, where we have identified a problem for us.    When a synthetic timeseries is computed with a specified aggregate and resolution, that specification is returned  irrespective of there being data on the originating timeseries. In our case, we compute aggregates over long stretches of time which results in situations like the image below: In the figure, the opaque line is the comptued by addition of the two other signals. Spanning over roughly a year, the period in the middle has a linear rise in the SyntheticTimeseries while the originals are empty. These values are of course meaningless and should be omitted in the successive steps of the analysis. Have you considered implementing a density filter or the like for these types of situations? Or, do you believe this is best solved client side by identifying the holes prior to a set of Syntheti

8
M
Practitioner
2 years ago
R
Seasoned
RobersoSNSeasoned
asked in Developer and User Community
Datapoint replacement inconsistencies

Hey! I’ve a problem with filling a gap from a source to a timeseries in CDF.  Problem descriptionWe’re filling a hole in a time-series from time A to  B. There are some datapoints on the edges of the interval in CDF.Data is extracted from the source, and in python prepped for the datapoints API as a list-of-tuples payload. For an arbitrary period I extract 2976 datapoints which I upload to CDF. Subsequently, I query the time-series for the same period of time and recieve 2928 datapoints. There are no NAN values in the input for either the date-time or value. The data is also hourly, and so I’m wary of it just being an edge effect of poor timestamp specifications for the retrieval. What other PEBCAK things have I missed? Simplified example included below:payload>> [{'externalId': 'ts_externalid', 'datapoints': [...]}]payload[0]["datapoints"][10]>> (1617271200000, 0.0)client.datapoints.insert_multiple(payload)meter_data = client.datapoints.retrieve( start=dates[0], end=dat

4
A
Seasoned
3 years ago
P
Practitioner
Patrick MishimaProduct Specialist
posted in Developer and User Community
Azure with Cognite Data Fusion - Data Integration Part 2

 It's been a while, but as promised, here is the follow-up post to my previous article. If you haven't read the previous article, you can find it here.In the first part, I shared the basic concepts about data integration. Today I will continue on the same theme, but the focus will be on latency and frequency. They are related concepts, but not necessarily the same. Why latency and frequency? They are crucial to defining your data pipeline, and they influence which Azure resources you use to cover your needs.What is data latency and frequency? Data latency is how fast/slow data can be retrieved or stored. Low latency means that the data is available in real-time or close to real-time and is vital in use cases where you need to respond quickly to information. Examples are alerts for critical events on machinery and equipment and online games based on real-time experiences. This leads us to the data frequency concept. Data frequency is how many times in a specific period the data should b

0
P
Practitioner
3 years ago
thomafred
Seasoned
thomafredSeasoned
asked in Developer and User Community
CDF Files and cloud optimized storage

We are looking into cloud-optimized storage formats such as GeoTIFF, Parquet, etc. One of the things we are trying to determine is whether we could fully utilize such cloud optimized formats with CDF Files.Will the download-link returned by CDF Files allow us to do seek-operations and only download parts of the file?

3
E
Practitioner
3 years ago
A
Active
Abusdal MortenActive
asked in Developer and User Community
Get latest TimeSerie entry for a node/collection of TimeSeries

Hi. I’m currently working on an application where we need to check if a node (PLC/PC) have updated any timeseries within x amounts of hours. The way we do it now is to get all timeseries recorded values in a time range, and check if any timeseries have any values. If there is an value on one timeserie, then we consider the node alive. This can be time consuming, since each node can several hundreds/thousands of timeseries. So my question is if there is a way to get the latest recorded value in a collection of timeseries within a timerange? Or just a recorded value in a timerange for a collection of timeseries. In Cognite, we have an asset hierarchy, which preferably would look like this:RigNode 1 TimeSerie1 TimeSerie2 ... Node 2 TimeSerie1 TimeSerie2 ... Node ... TimeSerie1 TimeSerie2 … But the hierarchy could also be completely flat, where all the timeseries are connected to the Rig. Our externalIds for the timeseries are:rigNumber.NodeNumber.SignalNumber So all timeseries wit

1
T
Seasoned Practitioner
3 years ago
P
Practitioner
Patrick MishimaProduct Specialist
posted in Developer and User Community
Azure with Cognite Data Fusion - Data Integration Part 1

Hello All! As this is my first time writing, I will do a quick introduction. My name is Patrick Mishima, and I am one of the Cognite Product Specialists. My main specialization is data integration and governance. I've worked with data since 2011 with different roles, projects, and job titles all related to data management. I also have experience with various ETL and BI tools, mainly from Microsoft and SAP and a few others.I lead the Product Specialists team, and we'll soon have more articles to share our knowledge and experience with you.Today and in my following articles, I will focus on the different options that Microsoft Azure and Cognite offer on data integration between our platforms and tools. But first, let's talk about a few essential and fundamental points when working with cloud providers. Using a cloud provider, most of the time, we think about saving on infrastructure costs. That is indeed true, but it's not the only truth. When you decide to move to a cloud provider, you

0
P
Practitioner
3 years ago
S
Practitioner
Snjezana SlabekSenior Director
published in Developer and User Community
Are you ready for a citizen data scientist adventure?

Cognite's Industrial Digital Academy (IDA), available on Cognite Academy, offers several courses to help you upskill and close any knowledge gaps to understand the value of CDF better.Today, we are eager to present the Data Science Fundamentals learning path we created with Cognite's data scientists. Some of our customers already had a pre-run, and their feedback is making us proud, so we recommend that you try it out. Take this opportunity and learn about data science from an industry point of view. You will be guided through a set of courses showcasing how data scientists solve industry challenges using industrial data.By the end of this learning path, you will: Understand basic principles of data science from an industry point of view Realize the importance of data science workflow Understand the value of understanding the business problem Be able to compare and analyze various data science use-cases  Understand the evolvement of data science, data doers, and citizen data sc

800
S
Practitioner
3 years ago
joar.saether
Practitioner
joar.saetherDirector Site Reliability Cognite
posted in Developer and User Community
Maintenance Windows for Assets and Files - June 2021

Database migrations - assets and filesCognite plans to migrate data from Cloud SQL databases in our Google Cloud-based CDF clusters in a two-week period starting June 25, 2021. We plan to migrate the databases containing data for the assets and files resource types. The migration for the two resource types will occur on different days.  We will email customers the exact time of the maintenance window for their cluster, CDF project, and resource type.  Why is this database migration needed?Cognite has recently made CDF available on Azure in addition to Google Cloud. The code running on Google Cloud is currently dependent on database options only available in Google’s public cloud.  Migrating these databases enables us to run the same code base in Google Cloud and Azure, allowing us to bring recent improvements in our Azure implementation to customers on CDF projects hosted on Google Cloud.   APIs in read-only  modeWe will set the API for the impacted resource types in read-only mode dur

0
joar.saether
Practitioner
3 years ago
S
Committed
Shanthosh PushparajahCommitted
posted in Developer and User Community
Test out the new Interactive Engineering Diagram Flow on Fusion!

Hello Developer Community,The Contextualization team at Cognite released an experimental interactive engineering diagram flow on fusion.cognite.com  . The old interactive engineering diagram flow still exists in fusion.cognite.com, however, it will be replaced with the new flow come August 1st. Thus, we highly encourage you to try out the new flow titled Explore and create interactive engineering diagrams. This new flow includes a plethora of new functionality and updates such as: Overview of all interactive diagrams in fusion.cognite.com  Easy test setup for relevant file and asset links Handling 1000 engineering diagrams in one go  New P&ID preview component Reducing and cleaning up technical debtAgain, we are still at the experimental stage, so any feedback we receive is beneficial to future iterations and to truly polish the interactive engineering diagrams flow. Please provide any feedback to the Contextualization Cognite hub group found here .   

0
S
Committed
3 years ago
E
erlend_sommerNew
posted in Developer and User Community
Size limitiations in RAW

I use RAW as a cache system for sensors that come from BACnet (BAS systems) Since it extremly slow.This have worked for some time now, But now CDF would give me a internal server error.. When this error accure, there is no help what so ever on what the problem is.After debugging, It seems i have reached a size cap on a row (5 MB)  but the table have a size cap of about 5 GB (Unconfirmed)So after i included a size check of my data, I chunk it up and spread it out in the rowsAs fare i can see, this limitiations is not mentioned in the docs so here you go, Liberate your data!

1
D
Practitioner
3 years ago
R
Seasoned
RobersoSNSeasoned
asked in Developer and User Community
Adding info to the explore page in fusion

When exploring the datamodel in CDF through fusion, it would be nice to add labels to the table overview, as shown below.  

3
Eric Stein-Beldring
Seasoned Practitioner
3 years ago
S
Active
Subas SujanActive
asked in Developer and User Community
Download documents from AssetMeta (Gearbox.js) component.

Hi, does anyone know how I can download files from AssetMeta and / or AssetDocumentsPanel component? Are there other options for retrieving documents related to an asset?

1
S
Practitioner
3 years ago
R
Seasoned
RobersoSNSeasoned
asked in Developer and User Community
Race conditions on event updates

Hey, To what extent does CDF handle possible race-condition triggering situations like the update of an object from two different systems?The example we are currently considering is whether a disjoint set of metadata on objects (events) can be updated without regard to timing. /Robert

3
A
Practitioner
3 years ago

Badge winners

Show all badges
Terms & ConditionsCookie Settings