Welcome to Our Community!
Join the conversation to shape a safer, more efficient, and sustainable industrial future.
I'm working on identifying the falling and rising edges of the VAL_23-KA-9101-M01:HSI.StatusMotorOn signal. For that and shifting the TS and rest the values, if I get -1 is a falling edge and for 1 it is a rising edge. For the first row it works fine: But I’m also getting a lot of values different than 1 or -1. Especially for more recent years. I guess that the problem is at the aggregating step, since I downloaded the data at a 1m frequency I get not int values for some records. Now, I was trying to download this TS at a 1s frequency, but I get a different number of records depending on the year. E.G. for 2014 I get 34 records for a 10 days time windows with a granularity 1m. If I do the same but for 2020 I get 1411, much more than for 2014, but still a tenth of the expected count. If I change the granularity to 1s, I still get the same number of records, but with more precision in the time stamp: Considering this, is there a recommended granularity to download the data and min
This discussion is linked to the course Cognite Data Fusion for Domain Experts.A data-driven approach to maintenance is more efficient than relying on a fixed schedule or fixing something once it’s already broken. There are plenty of opportunities around us to take advantage of smart maintenance. Look around in your home, your office, or think of your work tasks. Where could data-driven maintenance make a difference?Share your thoughts! For example, explain what data you have available or would need in order to do smart maintenance, and how this would be an improvement of current methods.
We are fetching quite a lot of data from CDF, both raw and aggregated, and we have experienced error messages from Postman and Java requests when the aggregated number of datapoints are exceeding the limit (10.000), but we have yet to experience this for the Cognite Python SDK for raw data (limit 100.000). Is there an error message implemented for this intention? From the code in the SDK in GitHub it appears that if the data is exceeding the limit then it would return an empty list, which is a very dangerous return setting. Best regardsOliver
Do you know what courses are available for you to explore the data from the Open Industrial Data project?Cognite Academy creates e-learning courses with an emphasis on providing hands-on experiences so you can learn to use Cognite Data Fusion through interacting with real industrial data. In most of our courses, the training data we use comes from the Open Industrial Data (OID) project, a live stream of industrial data from the Valhall oil platform. While the data is from the oil and gas industry, it is relevant for all asset-heavy industries as it provides insights into dynamic industrial processes. In this article, I will share a series of CDF & Power BI courses where you can explore the Open Industrial Data and solve a simple use case in Power BI: Introduction to CDF & Power BI In this course, you will connect Power BI to CDF, and retrieve the Open Industrial Data to learn how filtering and aggregation work with the connector in Power BI. CDF & Power BI: Solving
Are you a Power BI user who wants to gain hands-on experience with Cognite Data Fusion? Are you curious to learn how to handle a large set of CDF data in Power BI?The Cognite Power BI connector lets you use a CDF project as a data source in Power BI Desktop to query, transform and visualize data, share insights across your organization, or embed dashboards in your app or website. When working with the Cognite Power BI connector, it is important to know how to refresh the data and apply best practices to deliver accurate results. The newest course from Cognite Academy introduces you to implementing incremental refresh - one of the best practices to get the most out of the Cognite Power BI connector. Our instructors first explain when to use incremental refresh, then show you how to set it up in four steps. In this course, the training data comes from the Open Industrial Data (OID) project, a live stream of industrial data from the Valhall oil platform.After completing this course, you’l
A common setup that we have at Cognite is having 3 different environments: development, test and production. We have one specific Cognite Data Fusion (CDF) project for each of these environments. This setup allows to develop new features without interfering with production data. Also, it makes the deployment of new features to production much safer, since they are tested before.Sometimes, you still want to have the data you used during your tests, in your production environment. For example, if you extracted some data from sensors and saved them as time series in CDF, you may want to keep this historical data when you move to production. Same goes with files, assets, datasets, functions etc. Another example could be the replication of the data from a full production CDF project, to one of your customer’s project, containing only a subset of the source one.At Cognite, we have a python package (available at https://pypi.org/project/cognite-replicator/) whose purpose is to copy the select
Can the signals of warnings, alarms, and failures be used to tag anomalies and/or failures in the compressor?
There are some signals that in the names or the description have the words: alarm, warning, or fault. E.G. There is this alarm signal, (in red ) that I’m comparing with one of the vibration signals. And the alarm seems to be on when the vibrations are close to 0 and the Compressor is off. I’m plotting the label as a combination of the name and the description, but the externalId of the alarm is: pi:160268In most cases that alarm is on when the compressor is off, except in this case, Where I do not see any problem with the vibration signal: There are these other two alarms, from the plots: one is on when the compressor is off and the other one when it is starting up. But I do not see any strange behavior in the signal when they occur. Besides the alarms, I’ve also looked into some of the fault signals but those seem to be on for very long times: Is there another way to use the alarm and fault signals to label anomalies or failures? I don’t know if using the vibration signals is the bes
HiI have retrieved timeseries from an asset in PowerBi. That asset is linked through a relationship column to a group of assets. However the asset and the group it belongs to are connected trough 4 steps:groupwanted <- belongsTo - subgroup<- belongsTo - type of group <- connectsTo - group of a few timeseries assets<- belongsTo - timeseries asset Im wondering how to link the relationships in power bi, and how to orient to the right place.
This guide describes how to run the Cognite dB Extractor in a separate Docker container to fetch data from Microsoft SQL server to Cognite Data Fusion. Prerequisites Running instance of MS SQL server and valid credentials to SELECT rows from the table in question Docker host that can access the MS SQL server Running CDF project with an Azure AD service principal with capabilities to write either to CDF. 1 - Prepare Docker ImageCognite provides the database extractor as a Docker image published to Docker Hub, requiring just the addition of an ODBC driver. Since we are connecting to MS SQL we will install drivers provided by Microsoft for Debian 11, using this Dockerfile: Note! Go to https://hub.docker.com/r/cognite/db-extractor-base/tags to see the latest version of the Docker imageFROM cognite/db-extractor-base:2.5.0-beta5RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - && \ curl https://packages.microsoft.com/config/debian/11/prod.list > /
On serveral occations we have encountered limitations in metadata key length, most frequently when flattening JSON-formatted strings from our event stream data source systems. When the source system presents nested structure of as many as 4 levels we frequently encounter metadata keys that require more than 128 bytes. Up to now we have “solved” the issue by abbreviating the metadatakeys at the price of higher maintenance cost of the code and more importantly, that end-users get the perception that we have transformed the data or even dont understand what it represents. We now consider moving towards a solution where we simply put the entire JSON-formatted string into one single metadata value field, and leave to front-end teams and end users to flatten the structure. We have a similar issue with max number of metadata keys for timesseries (16). Question 1) Could you please suggest other, better options for handling these metadata limitations? Question 2) Will templates come to the resc
Hi everyone, I tried to upload my own dataset(time-series data) into CDF, but I always get error: Request with id 87b235ea-78dd-963a-b343-ff58a0ee084d to https://westeurope-1.cognitedata.com/api/v1/projects/learn/timeseries/data failed with status 400: Timestamp is too high, must be at most 2556143999999 which is 31-12-2050 23:59:59 GMT.I followed with this link:https://learn.cognite.com/path/cognite-data-fusion-fundamentals/working-with-cdf-integrate. I created an assest, 3 time-series. What I want to do is to upload my data to the 3 time-series, and show the variation of my data in CDF. I change the time_stamp of my data into the format of 2022-02-10T17:00:00.000Z, I am not sure if this is necessary but when I preview my datapoints in RAW explorer and CDF automatically change the this format into for example, 164998440000, so I am wondering if this is why I have this error? I also want to ask what is the meaning of span in IFSDB.sensors? It means this time-series is 100 days? or 100
I am listing time series for a given asset, and get a lot of results. I need to filter based on the time series external ID, but the only option is by using the “external_id_prefix” argument to the list function. So I build up the prefix left to right. Somewhere in the external id is a parameter that I do not care about, and after comes a new parameter that I want a particular value of. Concrete example of external IDs:IAA_Miros_Weather_Data_WIA_008IAA_Miros_Weather_Data_WIB_008IAA_Miros_Weather_Data_WIC_008IAA_Miros_Weather_Data_WID_008IAA_Miros_Weather_Data_WIE_008I am interested in only getting time series with external IDs that contain “_WI” and that end with “008”. Is there a way to list time series with a wildcard? Something like this:client.time_series.list( asset_ids=[my_asset_id], limit=None, external_id_prefix="IAA_Miros_Weather_Data_WI*_008", partitions=4)I could of course obtain the relevant time series by filtering after the fact. Something like thisresult = cl
When I run the query below, I am getting an error. If I increase the end time by 1 second I can get a result, but it gives me 2 days and I only want 1. All my data is consistently daily at midnight, UTC. Any ideas how to get past this?SELECT dp.id, dp.externalId AS `key`, dp.timestamp, dp.valueFROM `_cdf`.`datapoints` dpWHERE dp.externalId IN ( 'f18ae31cbfe544cb7bca08da91e5245a-SuFMiCo0f7c8eca4ce444208853aedd00ded4fb', '730619216b764ba77bc708da91e5245a-SuFMiCo6bb90df0efbc461d863faedd00df009b' ) AND dp.timestamp >= TO_TIMESTAMP('2022-09-01T00:00:00Z') AND dp.timestamp < TO_TIMESTAMP('2022-09-02T00:00:00Z') Gives me a result, but not what I want: cc @Torgrim Aas @Sunil Krishnamoorthy
A use case we encounter at Cognite is writing data back to SAP. This can happen in several contexts where the main goal is to create or update data in SAP (ex: work orders, notifications etc.) based on the analysis of data stored in Cognite Data Fusion. The ability to write back to SAP allows to take decisions without going back and forth between applications, which can save a lot of time. It is also less error prone than manually filling fields in SAP based on what you read in Cognite Data Fusion. This use case, which is all about automating processes, definitely fits Industry 4.0. In addition to that, SAP is one of the most used ERP systems in the industry. As a side note: we are talking today about SAP, but the same would be possible with other ERPs (as long as they have an API we can send requests to). For example, in a maintenance context: when analyzing data of your industrial machinery, you might notice that one of your machines needs maintenance. Instead of going to SAP, looki
Hi all!I am trying to obtain weather data from Miros for a given asset ID. I am experienced in Python, but new to CDF, and don’t have access to a lot of practical examples of common tasks and patterns. So I thought I’d ask here to speed up my work :) What would be best practice for the given scenario (given in detail to try to avoid the XY problem)?I am going to work on some root cause analyses, and the starting point is likely to be an asset that failed multiple times for the same reason in the past. I want to fetch different types of data for this asset, such as maintenance history, relevant metadata, and weather data at its location during the relevant time window. This will eventually make its way into a simple graphical tool that I can use at the start of new root cause analyses.I have been playing around with retrieving an asset object based on its id or external_id. This seems to be fine. Then I notice that weather time series from Miros are connected to assets. But how would I
One thing that would make the finding of the correct data is to label the CDF resources better with labels. https://docs.cognite.com/dev/concepts/resource_types/labels In the contextualization pipeline could things be automatically labeled such as:Document type: P&ID, PDF(Process Flow Diagram), loop diagrams, datasheets, single line. that theses documents are either labeled from the metadata. document ID schema (regex will do it) or from classification algorithm the file, and then gives it a label to the file. that we can see from the label on the resource what it is.The same is for Asset: Is it a Junction box, Transmitter, Skid, Pump, Motor, valve etc etc. Time series: is it a physical measurement, is an inference / calculation / “soft-tag”, is a parameter (like the gain or integration time on PID controller) this could be determent by simple analyzing of the data-point behavior of the time series. cause now when we search for a time series often a time series of a parameter that
Morten Andreas Strøm / Ben Skal September 12, 2022 What makes Cognite unique? Why is partnering with Cognite the best investment of your time and resources?This is a 3 part series where @Morten and I answer these questions through an indepth look at how our product, Cognite Data Fusion, can help you use industrial data to ignite your digital roadmaps. The topics we discussing are: What is Cognite Data Fusion and why did we build it? (First post) Data modeling grounded in business impact (Previous post) The opportunity cost of custom building your industrial data platform (This post) In the first Why Cognite post, we discussed the data problem Cognite Data Fusion is built to address. The short answer, industrial companies need simple access to complex industrial data. The reason, most operations teams have many business opportunities, but are struggling to effectively use data to improve production. In the
Hi!Almost three years ago we made a feature request regarding search. Specifically we want to:Have support for paging. Do grouping on metadata-fields. Define which fields yields relevance to the search-results. Inclusive search-terms using AND instead of OR.I do understand that satisfying the search-requirements of all your users is a daunting task, but perhaps it would be easier to let us do the job instead? I believe it would be very beneficial if we were able to access ElasticSearch directly in some manner.And please keep in mind that the data itself defines relevancy. No two customers are alike and therefore a good generic solution likely doesn’t exist. @Knut Vidvei @Andreea Pastinaru
I just finished the cognite academy examples on contextualization. I did notice on the PID contextualization example that there were quite a few errors in what seems potentially character recognition pipeline to identify tags in the PDF.The tutorial stated that “these were all good” and “we can accept all”. I suspect such a process would create missing or strange links in the contextualized dataset.Are these known issues?I lack a bit the understanding of the context for the importance of these mispredictions,but I thought to report them anyway just in case.Happy to support you on improving these if they are something that needs improvement.