Impact 2024: The Industrial Data and AI Conference for and by Users | Nominate Speakers Now for a Ch...
Thanks @Eniko Farkas for the response. I am doing this independently to further my knowledge in CDF.I managed to upload a PID file in ‘Explore Data’ but I am unsure now how to create the tags which are needed to be approved in the ‘Interactive diagrams’I created labels by editing the file and linked those labels / tags to the Assets. But still the file has no tags. Please help.
Thanks @Eniko Farkas . I guess this course is having limited availability and not there for enrolment.
Thanks @Thomas Sjølshagen . When I run SQL transformations with Destination as ‘Assets-Hierarchy’ or even ‘Assets’, I dont see any placeholder to map for ‘geoLocation object.
Thanks @Kristian Gjestad Vangsnes . I can see the environment is activated (as shown below in BLUE). But when i try to install ipykernel directly, it is giving the following messages (highlighted in yellow). I am trying to clone the extractor-utils repo and work to create a custom extractor in my local.D:\pycdf\python-extractor-utils-master\python-extractor-utils-master>poetry env listcognite-extractor-utils-yeiQVxSK-py3.9 (Activated)D:\pycdf\python-extractor-utils-master\python-extractor-utils-master>poetry run python -m ipykernel install --user --name eashwar_nC:\Users\eashwar_n\AppData\Local\pypoetry\Cache\virtualenvs\cognite-extractor-utils-yeiQVxSK-py3.9\Scripts\python.exe: No module named ipykernel
Great! It worked now. I added the ipykernel and then ran the above command. After launching the jupyter notebook, I could see the new env with <myname> showing up in the notebook.Thanks a lot!
Illustration for implementing the Macros computation within CDF
Thanks @roman.chesnokov for the inputs. I also see that the excel has lot of formulas in several columns in the workbook (with around 30+ sheets). I am wondering how to extract all of these individual computation in each of those columns and then use it as cognite-functions. Maybe, I am thinking if I should try setting up RAW tables and then use extended derived columns to make these computations as a part of transformations.
Thanks for the inputs.
Thanks @mathialo for your inputs. Please could you share monitoring guidelines and means of handling the DataStream if there is any hiccups in the PI server connectivity etc. How does the extractor resume from where it stopped / halted. How to handle those scenarios in the extractor scripts.
Thanks for the inputs @mathialo . So, I don't need to invest into doing any custom coding while using prebuilt extractors. I just need to configure with the right parameters and the data extraction can be accomplished.
Thanks @mathialo . I have two environments, DEV and PROD. So, should I manually do the same steps for extraction for PROD as well separately? Can I setup Extraction pipelines using Git actions so that we can automate these steps and not do these steps manually in the respective environments?
Thanks @roman.chesnokov . Could you please share any reference material that has all the technical details provided. (using GH actions etc). I can note that the main documentation site shares the details in some abstract fashion. I am looking for something that has the necessary details like in the boot-camp documentation.
Thanks @Everton Colling for your response. I will explore and see for early adopter program. Meanwhile, I wanted to get my data model construct for this case clean so that I can approach to develop the solution. Are there some sample data models available so that I can have a look at it for quick reference?I can understand your point about not sticking to raw tables but to use clean CDF resource types. For instance, where do I store the Crude Assays input data in CDF, Diet Data (Crude composition data (for various crudes and its index values), mass-balance data etc. I am not sure of the final destination schema to store these data values in CDF targets. Should I be using CDF resource type - sequences here?Please advise.
Thanks @Gaetan Helness for the inputs. Please could you share details on this step. “client IT team to add the app registration to all the different groups in the Azure AD” Is this the same as adding service principal to the AD group.
Thanks @roman.chesnokov for the inputs. But is it okay to store the timeseries tags as ‘assets’?Will that be good? I thought only asset information is only to be maintained as ‘Assets’.Here, I have around 343k timeseries objects in total, but I wanted to maintain a list of 18 parent tags and each of those having 4 tags attached to the parent.
Thanks @roman.chesnokov @Håkon V. Treider It worked. I ran the upgrade command pip install --upgrade cognite-sdkin a project specific folder and when launched the notebook from that folder, it worked.Should I add this SDK in my PATH variable so that it runs for any notebook in any folder?
Hi @Glen Sykes I am doing a use case called “Yield tracking”. Basically, I get lot of inputs and calculate the mass-balancing, the assays normalization, Diet feed, and then doing a dot product of these last two data elements and then use the output to compute swing% (SW%) values based on the type of the crude output (like liquids, gases products) and use those SW% slabs to compute the derived yields for some 500-line items in an assay matrix. These processes have to be done on a daily basis based on the data feed and then stored appropriately with outputs in CDF so that cognite charts can be plotted using them. I have around 137 charts and most of them have three items in it (like Linear, Non-linear and Actual). I have done all these steps as a POC to master this flow in a local jupyter notebook using flat files (csv) and data frames in Python. I also plotted them using matplotlib.Now, I have to translate all of them in cognite using CDF functions, sequences, timeseries and then use to
Hi @HaydenH Thanks for the response. I will try this option. Currently, I have decided to use the raw tables in CDF to store all of them instead of sequences and then use raw tables to fetch the dataset and to my processing.
Hi @roman.chesnokov ,Thanks for the inputs. Is it possible to perform the insertion inside a cognite function but not return any json as output and deploy the function?I think Cognite fucntion expects to have a json output.
Hi @Carin Meems It did help!, Thanks for the inputs.
Thanks for the inputs @Jason Dressel . It helped! Please could you guide on how to add the new creation of asset and hierarchy with a dataset that I already created. When I run the code, it creates the assets and hierarchy but doesn't update them all in my dataset. Dataset is still empty. # Code to create the LP -Classes and store the meta-dataimport pandas as pdfrom cognite.client import CogniteClientfrom cognite.client.data_classes import Assetcsv_file = "lpm_asset_hierarchy.csv"df = pd.read_csv(csv_file)client = CogniteClient()def create_asset(asset_name, description, dataset_id,lims_prefix=None, weight=None,pid = None): metadata = { "LIMS_Prefix": lims_prefix if pd.notna(lims_prefix) else 'NA', "Weight": weight if pd.notna(weight) else 0 } asset = client.assets.create( Asset(name=asset_name, description=description, metadata=metadata,parent_id=pid) ) return assetdataset_id = 6572524964198398root_asset = create_asset("LPM_YT_MODEL", "Root Asset",
Thanks @Håkon V. Treider . Is there a way to delete the dataset I created in SDK?In the script above, I manually created a dataset from UI and then used the ID to add in the asset hierarchy script. I want to delete the dataset and create a new dataset in the script itself (with name and description) and then attach all the assets to the newly created dataset. I am unable to delete the dataset that i created manually. Please help.
Thanks @Chyrus Ramesh for the response. CDF automatically takes the field as numeric only (#) in the raw table. At least that is what i observed after I uploaded from CSV. But when i run this command and fetch it as a data frame and try to see the total sum of this column, it says it has str values. Ideally, if it is stored as # columns then it should be a numeric column in dataframe too, why is the retrieve-dataframe converts the long numeric value into an object in a dataframe. Attaching the csv for your reference. You can try in any CDF project and check it out and do retrieve as a dataframe in a notebook and see the result. from cognite.client import CogniteClientc = CogniteClient()df = c.raw.rows.retrieve_dataframe("database-name", "table-name", limit=None)when you try to check the following, it should not give any error if it was properly retrieved as a numeric column. df[‘0NA’].sum()
Thanks @HaydenH I tried with asset_subtree_ids. It gave the expected result. root = client.assets.list(name='liquid_asset')child_assets=client.assets.list(parent_ids=[root[0].id])list_assets = []for list_item in child_assets: (list_assets.append(list_item.id))time_series_obj = (client.time_series.list(asset_subtree_ids=list_assets,limit=None))df_timeobj = (time_series_obj.to_pandas())df_timeobj[df_timeobj['name'].str.contains('Product_Vol',na=False)].loc[:,'id','external_id','name']]It took around 30 seconds to pull the final table that I wanted. Hope that is transient and won't be a problem when I include this code block in cognite function.
Hi @HaydenH I am implementing this in a cognite function. So, I would want the function to take the file, download and then use that file to read it in pd.read_csv(). All these must happen in-tandem. I don't want to download to my local disk and then read the file from my local. Since it must all happen online within cognite itself, what is the best way to do that?Ideally, it is apt to have a function like client.files.retrieve_dataframe(filename = “myfile.csv”) etc. But I don't have any means like this now. Please advise.
Already have an account? Login
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.