Skip to main content

Community Discussions

Join the conversations to shape a safer, more efficient, and sustainable industrial future!

1247 Topics
N
Active
Niros ManoharanActive
asked in Product User Community
Data Streaming with CDF

I am trying to stream data from CDF to Azure Event Hub with the Python SDK and cannot find anything related to streaming datasets. Only option so far (as I know) is dps = c.datapoints.retrieve_latest(id=184691546499795) which would need a trigger of some sort to keep running. Data from CDF are so called timeseries from different types of sensors. Is there any documentation on Streaming Data for CDF that I could look at or is Streaming really supported?

1221
K
Practitioner
R
Seasoned
RobersoSNSeasoned
posted in Product User Community
The cognite python SDK should batch requests that violate API constraints

Hi, When developing with the Cognite python SDK, a common restriction is the API imposed restrictions on queries. Quering time-series, for example, is restricted by 100 asset-ids. I believe the Python SDK should recognize these constraint violations and batch / concurrently dispatch requests in chunks that do not violate constraints. Optionally with a performance-warning to the developer. Is this sane, or do you think that the developer/customer should maintain a wrapper on the SDK for batching each endpoint (as we’ve currently done at Statnett)?

843
R
Seasoned
Alexandra Cretu
Practitioner
Alexandra CretuProduct Marketing Analyst
published in Digitalization Community
The digital twin: the evolution of a key concept of industry 4.0

The digital twin is the foundation for industrial digitalization efforts, delivering real-time insights, accurate forecasting, and intelligent decision-making. In the almost two decades since the term was invented, industry - and the world - have changed dramatically. So what’s next for digital twin technology? Johan Krebber, IT Strategist at Cognite, summarized his perspective on the evolution of the digital twin concept during a panel at Ignite Talks, 2021’s big industrial digitalization conference. Read below an extended expert interview between Johan and Petteri Vainikka, our Vice President of Product Marketing, on the future of digital twins. Hello, Johan! Thank you, for taking part in our panel at Ignite Talks and especially for taking the time to do a deep-dive interview to expand on your contributions to the panel! Let’s start with a lightning round question. All I need is a simple yes or no. You’ll get to elaborate in a second. Should we sunset talking about digital twins and

720
Alexandra Cretu
Practitioner
Alexandra Cretu
Practitioner
Alexandra CretuProduct Marketing Analyst
published in Digitalization Community
We've released the first-of-its-kind Industrial DataOps Book!

Did you know that only one in four industrial organizations extract value from their data? The lack of tools and processes to connect, contextualize, and govern the data often stand in the way of industrial digitalization. Industrial DataOps is a powerful new way of deploying data and technology to transform an industrial organization. It makes sense of, manages, and extracts value from complex industrial data. And it is is already becoming a driving force in industrial transformations, helping accelerate digital maturity, enabling data teams to deliver more digital products, and realizing more operational value at scale. In a 2020 survey of global companies, McKinsey found organizations that embedded DataOps could see the volume of new features increase by 50 percent because data automation enables quicker development iterations. At Cognite, we’ve released the first-of-its-kind Industrial DataOps book - a guide packed with insights, industry expertise and practical advice on how your

440
Alexandra Cretu
Practitioner
Arild Eide
Seasoned Practitioner
Arild EideSeasoned Practitioner
asked in Product User Community
Feature request CDF login form: Strip protocol from override tenant field

If a user includes https:// or http:// in the Override Azure Tenant field the url to Azure AD will not include a correct tenant and login is prevented. This has caused login issues for at least a few users. The request is simply that the form is modified to strip the protocol scheme, ie removing http:// or https:// automatically, if present.

731
Krystele Uy
Practitioner
R
Seasoned
RobersoSNSeasoned
asked in Product User Community
Getting proprietary code to Cognite Functions

Hey, In our current workflow we’re expanding the use of Functions as a tool. We’re somewhat hampered by how “clumsy” it is to bundle and version control proprietary dependencies. Is there something in the pipe to address this issue? Kind regards, Robert

1163
Andreea Oprea
Seasoned Practitioner
Anita Hæhre
Seasoned Practitioner
Anita HæhreHead of Academy and Community
published in Digitalization Community
Join us for Ignite Talks September 21 - 23. Sign up to watch the recordings live, or on demand!ignite talks

We’re excited to host our fourth global conference, Ignite Talks on September 21-23. We’d like to invite you to this virtual hybrid event, which will bring together global leaders and innovators from technology, industry, and government who are dedicated to meet the carbon net-zero 2050 deadline and create a more innovative, data-driven, sustainable future. You can view every session live or on your own time. The three-day industrial digitalization conference, co-located in Asia, Europe, the Middle East, and the United States, will focus on the innovative technologies that power industries like oil and gas, power and utilities, and manufacturing and enable renewable energy development. You can expect conversations on Industrial DataOps and forward thinking technologies like robotics, artificial intelligence, and data analytics, as well as deep discussion on how to drive profitable sustainability. “As the energy industry reinvents itself and deploys new technologies, we know that data w

380
Anita Hæhre
Seasoned Practitioner
R
Seasoned
RobersoSNSeasoned
posted in Product User Community
Interpolation in synthetic time-series

Hey! In our tooling for the power-analyst we make computations using the SyntheticTimeseries API. Results from these are used in subsequent analysis, where we have identified a problem for us. When a synthetic timeseries is computed with a specified aggregate and resolution, that specification is returned irrespective of there being data on the originating timeseries. In our case, we compute aggregates over long stretches of time which results in situations like the image below: In the figure, the opaque line is the comptued by addition of the two other signals. Spanning over roughly a year, the period in the middle has a linear rise in the SyntheticTimeseries while the originals are empty. These values are of course meaningless and should be omitted in the successive steps of the analysis. Have you considered implementing a density filter or the like for these types of situations? Or, do you believe this is best solved client side by identifying the holes prior to a set of SyntheticTi

1778
M
Practitioner
ArveV
Practitioner
ArveVPractitioner
posted in Digitalization Community
"So... you want to be a software company?"

McKinsey Quarterly did post an interesting set of articles recently, regarding the transition and change management when looking into digital services. I believe some of the topic and views could be interesting to reflect upon. Go here to find out : https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/five-fifty-so-you-want-to-be-a-software-company?cid=other-eml-alt-mip-mck&hdpid=034603b3-3fba-4313-a963-553dca543b0f&hctky=12386931&hlkid=8635b3c5c91d439a8a47f0bf353a4968

320
ArveV
Practitioner
Paulo Botura
Practitioner
Paulo BoturaPractitioner
posted in Digitalization Community
PGS strikes again!

How a DataOps platform can boost improvement in all parts of the business! And this time is focusing on what is most important - HSE! https://www.pgs.com/investor-relations/ir-news-stock-announcements/pgs-digitalization-initiative-improves-crew-safety/ https://www.cognite.com/customers_stories/dataops-in-action-improving-vessel-crew-safety One more big shout out to @Cerys James and @Sverre Olsen ! Congratulations for another amazing use case!

361
alex.farmer
Practitioner
R
Seasoned
RobersoSNSeasoned
asked in Product User Community
Datapoint replacement inconsistencies

Hey! I’ve a problem with filling a gap from a source to a timeseries in CDF. Problem description We’re filling a hole in a time-series from time A to B. There are some datapoints on the edges of the interval in CDF. Data is extracted from the source, and in python prepped for the datapoints API as a list-of-tuples payload. For an arbitrary period I extract 2976 datapoints which I upload to CDF. Subsequently, I query the time-series for the same period of time and recieve 2928 datapoints. There are no NAN values in the input for either the date-time or value. The data is also hourly, and so I’m wary of it just being an edge effect of poor timestamp specifications for the retrieval. What other PEBCAK things have I missed? Simplified example included below: payload>> [{'externalId': 'ts_externalid', 'datapoints': [...]}]payload[0]["datapoints"][10]>> (1617271200000, 0.0) client.datapoints.insert_multiple(payload)meter_data = client.datapoints.retrieve( start=dates[0], end=dates[-1], exter

864
A
Seasoned
P
Practitioner
Patrick MishimaProduct Specialist
posted in Product User Community
Azure with Cognite Data Fusion - Data Integration Part 2

It's been a while, but as promised, here is the follow-up post to my previous article. If you haven't read the previous article, you can find it here. In the first part, I shared the basic concepts about data integration. Today I will continue on the same theme, but the focus will be on latency and frequency. They are related concepts, but not necessarily the same. Why latency and frequency? They are crucial to defining your data pipeline, and they influence which Azure resources you use to cover your needs. What is data latency and frequency? Data latency is how fast/slow data can be retrieved or stored. Low latency means that the data is available in real-time or close to real-time and is vital in use cases where you need to respond quickly to information. Examples are alerts for critical events on machinery and equipment and online games based on real-time experiences. This leads us to the data frequency concept. Data frequency is how many times in a specific period the data should

2210
P
Practitioner
K
Practitioner
Kine ÅrdalPractitioner
posted in Digitalization Community
Neptune Energy: How the pilot enabled collaborative environment in digital well planning

Hi, my name is Kine Årdal, and I have the privilege of being the Customer success manager for Neptune Energy. I’d like to share experiences from our latest project, and hoping to get feedback and hear about similar experiences from you in the community. The project kicked off in April, as a12-weeks proof of concept, where we enabled a collaborative environment with seamless data flow of key well data for well planning. Current workflow: In the current well planning workflow, it is challenging to share data efficiently and reliably across disciplines and collaboration partners. There is no central storage of drilling and well data ready for a digital way of working, and data is often stored in silos. In addition, the collaboration with vendors often relies on manual exports/imports, sharing data on e-mails/ftp, which leads to e.g. unnecessary time spent waiting for data, duplication of data and a challenge to keep track of versions. Result: In this project, the initial Neptune well data

1061
A
Alexandra Cretu
Practitioner
Alexandra CretuProduct Marketing Analyst
published in Digitalization Community
Think Beyond Data Lakes: Data Products and Data Value Measurement

There are two discomforting truths within digital transformation across our key industries; energy, utilities, and manufacturing. Digitalization PoCs are commonplace. Real ROI isn’t. Billions are invested in cloud data warehouses and data lakes. Most data ends there, unused by anyone for anything. At the heart of this data-driven value dilemma lies a confluence of challenges, ranging from the technical (How can we best organize our diverse and fluid data universe?) to the operational (How can we create new information products and services?), to the financial (How can we treat data as an asset?), to the human (How can we improve data literacy and ensure digital solution adoption in the field?). Read also: DataOps: A transformative new approach to data ROI To avoid boiling the ocean, we will focus on what is perhaps the most fundamental question all fellow Chief Data Officers and other digitalisation executives need to consider as their Northstar — and in doing so, we will find ourselve

310
Alexandra Cretu
Practitioner
thomafred
Seasoned
thomafredSeasoned
asked in Product User Community
CDF Files and cloud optimized storage

We are looking into cloud-optimized storage formats such as GeoTIFF, Parquet, etc. One of the things we are trying to determine is whether we could fully utilize such cloud optimized formats with CDF Files. Will the download-link returned by CDF Files allow us to do seek-operations and only download parts of the file?

1203
E
Practitioner
A
Active
Abusdal MortenActive
asked in Product User Community
Get latest TimeSerie entry for a node/collection of TimeSeries

Hi. I’m currently working on an application where we need to check if a node (PLC/PC) have updated any timeseries within x amounts of hours. The way we do it now is to get all timeseries recorded values in a time range, and check if any timeseries have any values. If there is an value on one timeserie, then we consider the node alive. This can be time consuming, since each node can several hundreds/thousands of timeseries. So my question is if there is a way to get the latest recorded value in a collection of timeseries within a timerange? Or just a recorded value in a timerange for a collection of timeseries. In Cognite, we have an asset hierarchy, which preferably would look like this: Rig Node 1 TimeSerie1 TimeSerie2 ... Node 2 TimeSerie1 TimeSerie2 ... Node ... TimeSerie1 TimeSerie2 … But the hierarchy could also be completely flat, where all the timeseries are connected to the Rig. Our externalIds for the timeseries are: rigNumber.NodeNumber.SignalNumber So all timeseries with the

1261
T
Seasoned Practitioner
Anita Hæhre
Seasoned Practitioner
Anita HæhreHead of Academy and Community
published in Digitalization Community
‘DataOps is what we do, but it’s not what we call it’

Most people are talking about DataOps as if it’s an idea that emerged in the last 5 to 10 years. But according to Rolf Thu, it’s something Aarbakke, a world-leading mechanical solutions supplier to the oil and gas industry, has been thinking about since the early 2000s. Thu says what’s today known as DataOps has been long in the making — people just didn’t know what to call it. For Aarbakke, it’s been a steady evolution, introducing digital solutions step by step over the years until the company emerged as the “smart factory” it sees itself as today. “I joined Aarbakke in 1989, and the theme throughout my career has been learning,” Thu said. “And what we’re doing now with data is also about learning. We are learning from best practices, we are increasing the competence of our employees, and we are constantly seeking improvements for our factory through better and smarter uses of data.” Can you explain more about how you use the data to achieve more at Aarbakke? RT: With the powerful da

570
Anita Hæhre
Seasoned Practitioner
ArveV
Practitioner
ArveVPractitioner
posted in Digitalization Community
Check this out! Energima have built Properate on CDF, awesome stuff!

Hi all, take a look at https://properate.com/, the landing page for Properate. This is the solution Energima have built on CDF. Remember to watch the landing page embedded video :-) Energima is taking full advantage of several CDF services and Properate includes a stack of cool features!

241
ArveV
Practitioner
Paulo Botura
Practitioner
Paulo BoturaPractitioner
posted in Digitalization Community
New PGS use case - Vessel Speed Optimization

The happiest moment in a CSM day/week/month/ YEAR is when we see our customers sharing success cases! Thank you PGS for allowing us to be part of your digital journey! Contratualizations for another fantastic achievement @Cerys James @Sverre Olsen !!

1298
Cerys James
Active
Snjezana Slabek
Practitioner
Snjezana SlabekSenior Director
published in Product User Community
Are you ready for a citizen data scientist adventure?

Cognite's Industrial Digital Academy (IDA), available on Cognite Academy , offers several courses to help you upskill and close any knowledge gaps to understand the value of CDF better. Today, we are eager to present the Data Science Fundamentals learning path we created with Cognite's data scientists. Some of our customers already had a pre-run, and their feedback is making us proud, so we recommend that you try it out. Take this opportunity and learn about data science from an industry point of view. You will be guided through a set of courses showcasing how data scientists solve industry challenges using industrial data. By the end of this learning path, you will: Understand basic principles of data science from an industry point of view Realize the importance of data science workflow Understand the value of understanding the business problem Be able to compare and analyze various data science use-cases Understand the evolvement of data science, data doers, and citizen data scientis

820
Snjezana Slabek
Practitioner
joar.saether
Practitioner
joar.saetherDirector Site Reliability Cognite
posted in Product User Community
Maintenance Windows for Assets and Files - June 2021

Database migrations - assets and files Cognite plans to migrate data from Cloud SQL databases in our Google Cloud-based CDF clusters in a two-week period starting June 25, 2021. We plan to migrate the databases containing data for the assets and files resource types. The migration for the two resource types will occur on different days. We will email customers the exact time of the maintenance window for their cluster, CDF project, and resource type. Why is this database migration needed? Cognite has recently made CDF available on Azure in addition to Google Cloud. The code running on Google Cloud is currently dependent on database options only available in Google’s public cloud. Migrating these databases enables us to run the same code base in Google Cloud and Azure, allowing us to bring recent improvements in our Azure implementation to customers on CDF projects hosted on Google Cloud. APIs in read-only mode We will set the API for the impacted resource types in read-only mode during

760
joar.saether
Practitioner
Jørgen Lund
Seasoned Practitioner
Jørgen LundProduct Manager
posted in Digitalization Community
What does it take to succeed with data science and analytics in power trading?

For many operators, power trading remains more of an art than a science that still relies heavily on heuristics and trader intuition. But as markets evolve and deviate from historical patterns, the tools and methods to make smarter trades must adapt in order to maintain or grow profitability and become a sustainable source of competitive advantage. In this webinar, we talk about our experience of integrating data science into forecasting and trading analysis, and how we used Cognite Data Fusion to enable analysts and traders to work iteratively in a data science process across a range of analytical topics within the trading domain. Check it out and leave your comments below! What do you see as the biggest opportunities and challenges within power trading analytics?

300
Jørgen Lund
Seasoned Practitioner
E
erlend_sommerNew
posted in Product User Community
Size limitiations in RAW

I use RAW as a cache system for sensors that come from BACnet (BAS systems) Since it extremly slow. This have worked for some time now, But now CDF would give me a internal server error.. When this error accure, there is no help what so ever on what the problem is. After debugging, It seems i have reached a size cap on a row (5 MB) but the table have a size cap of about 5 GB (Unconfirmed)So after i included a size check of my data, I chunk it up and spread it out in the rows As fare i can see, this limitiations is not mentioned in the docs so here you go, Liberate your data!

961
D
Practitioner
Alexandra Cretu
Practitioner
Alexandra CretuProduct Marketing Analyst
posted in Digitalization Community
How to scale use cases with data contextualization

If there is one thing we at Cognite get a lot of questions on, it’s contextualization. Not so much what is contextualization (luckily we are getting past that phase now), but specifically on two subsequent topics: How does your contextualization engine actually work? How does contextualization make use case scaling order of magnitude (or two!) more efficient? In this article, we will address both the above questions. We will also offer an ‘executive summary’ on data contextualization and its role in modern data management towards the end for completeness. Dive in!

290
Alexandra Cretu
Practitioner
Alexandra Cretu
Practitioner
Alexandra CretuProduct Marketing Analyst
published in Digitalization Community
How to scale use cases with data contextualization

Anatomy of a contextualization engine for AI use case scaling in industry If there is one thing we at Cognite get a lot of questions on, it’s contextualization. Not so much what is contextualization (luckily we are getting past that phase now), but specifically on two subsequent topics: How does your contextualization engine actually work? How does contextualization make use case scaling order of magnitude (or two!) more efficient? In this article, we will address both the above questions. We will also offer an ‘executive summary’ on data contextualization and its role in modern data management towards the end for completeness. Let’s dive in! Read also: The data liberation paradox: drowning in data, starving for context How does Cognite Data Fusion contextualization engine work? First, it is paramount to set some foundations: There is no such thing as the ideal universal data model. Having some pre-defined reference data model (can be based on industry-standard where applicable, or onl

1040
Alexandra Cretu
Practitioner

Popular tags

Training Request Button
Terms & ConditionsCookie Settings

Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings