Impact 2024: The Industrial Data and AI Conference for and by Users | Nominate Speakers Now for a Ch...
very cool. this also give the ability to write values “as found” and “as left” if we do a change. such that we get a more representative data in the interventions that we do in the field.
Into the cliboard it would also be great to get the names and the ID’s / External-IDs of the timeseries used aswell :)
Hi @Knut Vidvei A: you are correctB: you are also correctThanks for understanding the features 😊 Stig
Hi @dvoelxen In the Open Industrial Data publicdata CDF project there is not a SeaWaterLift pump. the majority of the data is for a 1. stage gas treatment system on an upstream oil and gas platform, that compresses, and cools gas in order to condense heavyer hydrocarbons, to separate the heavyer and the lighter hydrocarbons. additionally you find some information on a reverse osmosis system which is a water purifying system. Do you have aditional information on this data quality hands-on? is it in using the indsl functions created https://indsl.docs.cognite.com/data_quality.html
Yeah, I just compare it to what i have available to me on the SCADA system or PI vision or some other place, and do the correction from that. Here is a code for getting the data from this current start of a production day, witch goes from 06:00 to 06:00 each day. dtn = datetime.now()start = datetime(dtn.year,dtn.month,dtn.day,6)end = 'now'# This retreives data in utc atleast from the CDF project I'm working againstdf = client.datapoints.retrieve(id=list_of_int_ids,tart=start, end=end).to_pandas()#shifte datatime away from utc to local time zone witch is one our forwardns = df.index.view(np.int64)ns += 1*60*60*10**9 #adjusting it 1 hours froward such that timzones match.df.set_index(ns.view('datetime64[ns]'))But If there is away of getting the data straight from CDF in my local timezone by default that would be great to know. Hope some here can help for that, does anyone know if it can be automatically retrieve by using timezone environmental variables or something on the API / SDK leve
#the dtype in pandas is in datetime64[ns] so when we view it in the np.int64 dtype we get the number og ns since unix epoc (1.1.1970)#shifte datatime away from utc to local time zonens = df.index.view(np.int64) #converts it to unixepoc in nano secondsns += 1*60*60*10**9 #adjusting it 1 hours froward in nanosecondsdf.set_index(ns.view('datetime64[ns]'))One way of doing it is this :)
Yeah, I also sett that. I usedatetime.utcnow()and convert it manualy to my timezone by shifting the DF with one hour. then the code becomes independant on where ever the code is excecute from. But @Håkon V. Treider has made two good guide for pyhton here: Where he on the the part 2 goes into how to do datetime stuff :)
maybe even data form Osdu can be filled into publicdata aswell? :)
I’ve never heard about OSDU, but in the Forum members i found cognites logo. hope someone in Cognite knows what is going for OID’s prespective :) But this is interesting. is there any good information on how you want to explain this? But CDF is quite the platform and data source agnostic tool so I could see alot of the data thats shared in OSDU could be replicated in a open CDF project to even make it easyer to work with the data :) but its always fantastic to have more open industrial data. Have you worked with any data sets? are there alot of raw data there?
Just use & as a separator - Noted Cognite functions can be use to alot of things, only creativity can hold you back :)
Nice, Knut :) very nice to note down. but often I need to run charts with their calculations that I have created and not on spesific externalID’s cause i have timeseries avaiable in the SCADA systems, but I need a calculation from charts. But hope you guys can note it down as a wanted feature for charts’s future. But accessing externalID in charts with a spesific timesegment is noted, this is a nice link to have from web-apps to just send it to charts. Can we add additional externalID’s there is just one TS in the example. but can more be added by separating them with a separation character? very easy to build <a href=chartsURL> in web apps :)
should be:https://{company}.fusion.cognite.com/{project}/ in the base url. anyhow.. nothing todo with the case.
You need to have a datetime index on your data frame.#check it has the dtype=datetiem64[ns]df.indexIf the datetime is not the index then you can set the ‘datetime_indexed_series’ by setting the series with dtype=datetime64[ns] as the index:df = df.set_index('your_datetime_series_name')then check that your index has now datetime64[ns] dtype. If none of your series have the datetime64[ns] dtype, but you have a string or an int representing the timestamp of the dataframe rows. then convert the string or the int into a datetime64[ns], check out this link https://pandas.pydata.org/docs/reference/api/pandas.to_datetime.html If it is a unix epoc time in ms (milliseconds since 1/1-1970) then you can convert that to a datetime series:#if the series is in integers on unix epoc format, you can use some numpy inherited functionality and convert the dtype there#if it is integer values unix epoc in msdf['timestamp'] = df['unix_epoc_ms_series'].view('datetime64[ms]')df.set_index('timestamp')#or just
Try and merge the units i the y-axis, and zoom closer to the datapoint, because the graph is showing on aggregates when you are trying to view a wide time-range. but Zoom closer to get the actual values datapoints, that can cause this viewing phenomena.
Hi Revathi, Should contact @Shehan Karunaratne, he can help you to setup a replicator of publicdata-data setup for your companies CDF Project. You can also extract everything yourself and create an Cognite Extractor yourself extracting all the available data from it and push it to yours aswell. a nice excecise do.
I made a video that might help you to setup a python developement enviroment for developing extractors. using WSL, Pyenv and Poetry.
Anytime @Alex 😊 Hope I figure out something more on the effect timeseries.
Hi @Alex Since the gas train is mainly for the gas and NGL, and by looking at the ration from here https://www.norskpetroleum.no/en/facts/field/valhall/ you can try to infer the oil production from the valhall field. but then I would use TS.name: VAL_23-FT-92512:X.Value (ext_ID: pi:160182, ID: 622801209626283) although the unit is not set on the timeseries, I’m quite sure it is Sm³/h. this is then the gas from the first separator stage only, since the gas from the ejectors (2. stage separator and from closed-drain- / “flare knockout”-drum) is not metered by this Flow meter, and currently are not available in publicdata. so a complete metered volume balance of the system is not existing now form the timeseries available.When it comes to the differences in the power timeseries. I’m not quite sure what the differances are. https://publicdata.fusion.cognite.com/publicdata/charts/09a3f0a8-8ff8-48f4-bc64-f69eba663a48?cluster=api.cognitedata.com But I will do some digging, and see what ill fi
I totally agree, especially with the second option, with a domain boosted computational science approch, that is near and dear to my heart. aswell as providing newer insight and with this we can augment the data closer for a more actionable insight and create a more factfull culture around industrial data, with not just realtime perspectives but how things behave over time. Thats also the powers of using CDF with industrial data, it democratizes access to industrial data, through open source SDKs and easy to use open API’s, giving a new creative domain to programatically wrangle industrial data 😍
Hi @Luis Ramon Ramirez Rodriguez, I agree with alot of what you are saying.One of the biggest issues we have in using supervised machine learning approches to industrial data is having data from failuremodes, where both the failure mode and time segment is speficied. and this is not that easy. the work orders are extracted from an ERP system, where people have both notified and planned a WorkOrder, and do not nessaserily represent the reality as in the SCADA systems or even the physical systems, when it comes to classifying the time segments. Additionaly the lack of failure data is also something we don’t yet have significant ammounts of. especially the data from Valhall, cause the Valhall assett have very high production reliability, and failures on bigger gas processing machinery is a rare occurance. but having some data to augment / feature engineer the data with can potentially be helpfull, or having dimentionaly independant datasets / models can also benefit cases such as this. Bu
@Eric Stein-Beldring do you know when/if the GitHub repo goes public for InDSL? 😊
I love the digging and exploring in the OID data you are doing @Luis Ramon Ramirez Rodriguez. I’m going offshore to Valhall in a week, and I will note down the timestamps and have a look at what has been happening from the operators end. If there are some events that we don’t yet have in the publicdata CDF project. but right now I think the triggering of the common alarm of the switchgear system can be a lot of things. but I will speak to the guys what is triggering the common alarm on the switch gear for the 1.stg compressor. And I don’t think it is frequency components show in the vibration timeseries, yes it is aggregated but it is aggregated in max length peak to peak vibration of the driveshaft. would be nice with frequency components also from the vibration system in the publicdata project. I can see what we can do something about getting the frequency specturm aswell “live” into the CDF project. if its easily available from Bently Nevadas System 1 that is the 3. party SCADA syst
Not that interested in standard colors, but if its a thing sure, but that a default color schema / pallet that is normally used within a specific company, cause the logo and the marketing within each company have their own color schema that we should be able to import to the specific tenant, such that the customer feel that it the charts and graphics are a natural part of their own digital infrastructure.
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.