Product User Community

Is there any way to define schedule for custom db extractor (based on cognite-extractor-utils)  alike default extractor. 

I have to do complex calculations and store the resulting data in the form of data frames (tabular form of data structure). The only way I see is to use the ‘sequences’ in CDF resource types. But I think CDF sequences doesn't allow to do data wrangling as we can do in pandas data frames. So, I wish to know if there is any best way to accomplish the storage of tabular data structures like data frames / arrays like what we can usually do in core Python. Basically, I wish to store data in structures like we typically have in core Python. Lists, Dataframes , arrays etc.  Any structure available in CDF?

Hi, what is required as access rights to be able to uplad a function ?  

Hi Team,In GraphQL there is no (known) option to filter null values of FDM View’s direct properties which refer to another view(s).For example: Consider the snapshot of the views.Here if we want to filter all MyTypeWrapper instances which has ‘myType’ property as null.How could it be achieved using GraphQL?-Mohit

I am unable to complete and proceed further, as  the below mentioned course as it is showing as  “Registered” even after completion Please give the solution even after clearing my system cache and tried in other browser as well https://learn.cognite.com/path/data-engineer-basics-transform-and-contextualize/match-entities-concept-and-ui  

Hi there!I have a usecase where a file is uploaded by a user to an API. The API then uploads the file to CDF Files. We want to avoid having to have the full file in memory at the same time, and therefore must stream the file contents from the request handler directly into CDF Files.There are two ways of achieving this:Stream the request body from the request handler directly into CDF Files’ upload URL Chunk the request body and upload each chunk as separate requests.The first option may be achievable, but I don’t believe the second option is possible.Do you have any insight whether it is possible to chunk a file upload like this in CDF Files?

Hi, We like to use Cognite AIR in one of our project .We got to know that it is getting decommissioned by end of 2023.Kindly confirm on that whether we should explore AIR now or we should not as it will not be available after this year. Thanks,  

I am trying to run a code to fetch timeseries based on some tags available in a project. While I execute the same code using jupyter notebooks in CDF online-notebook feature, the code runs fine. When I am trying to run the same code script in local after setting up connectivity using interactive-login and then when I run the timeseries retrieve code, I am getting an error. Please help.Code:from datetime import datetime, timezoneutc = timezone.utcpi= client.time_series.data.retrieve_dataframe(external_id=['pi:2FC1898.DACA.PV','pi:2TC1066.DACA.PV','pi:LAB_133-X013_APIGRAVOB','pi:2FC1898.PIDA.OP'], start=datetime(2023, 1, 1, tzinfo=utc), end=datetime(2023, 5, 1, tzinfo=utc), aggregates=["average"], granularity="1d")  Error- Traceback:---------------------------------------------------------------------------AttributeError Traceback (most recent call last)Cell In [9], line 5 1 from d

Hi, when I create a function that imports pandas I get  ModuleNotFoundError: No module named 'pandas'I expected that the function was hosted in a pre-configured runtime environment with many commonly used libraries already installed, including pandas.Any idea what I am doing wrong?

Hello! I have 3 quick questions that come to mind:From your perspective, what is the 1 sentence value statement of CDF?How can we justify all the manual work required to prepare the data to ingest into the platform?How do our customers save money by using CDF?

Hi ,  I’m trying to deploy an Azure function on Azure function app. But when I Included Cognite related libraries which I need to read and write data to Cognite data model. It was not working  even though I mentioned to include cognite in requirments.txt still not working. Has anybody else faced same issue ? Have you use azure function to connect to cognite (not the cognite functions)/

Hello I noticed that data type for table import thru CSV file is by default as string, how can I change it to other data type like boolean, array and etc?  Regards

We are using the online version of the Jupyter notebook from CDF portal for a client project - DEV and able to get the clientconfig/ client object and create and retrieve assets, run transformations, create datasets etc. Client IT team has created an app and registered in Azure and also shared the tenant ID, Client ID / name and secrets as well. When we use these parameters shared for this app and run the same code locally in a notebook, it is not able to perform certain tasks (such as data set creation etc.). Basically, the online version has all the IAM groups as {data engineer, data scientist Data Analyst, OIDC-Admin.}But when we set the configuration parameters client-ID, Tenant and secrets etc., we don't get the groups entirely as above but only comes as “Data Integration”.  This “Data-integration” has limited scope and doesn't allow to create datasets etc.  So how do we understand this part of roles and access management in CDF construct and applications registered in Azure AD? 

I have a time series data identified with TAGS and that can contain around 1500 to 5000+ records generated per day. I would have to perform a time weighted average and calculate the time-weighted value for the time-series data for the given times. How do I proceed to recreate the computation in Cognite since I got the PI data already sitting within CDF. Basically, got to recreate this function of OSI PI inside CDFPIAdvCalcDat(tagname, stime, etime, interval, mode, calcbasis, minpctgood, cfactor, outcode, PIServer) 

Hi team, In Hess team came up with one request from Documentum Extractor, please refer below context for the request. Currently we have Cognite connected to Documentum via the raw folder, which pulls in the raw file format from EDMS. We actually need to connect to Documentum's rendition folder where the PDF versions of all the files are saved. Cognite can only effectively contextualize PDF's, so we need to connect to the render folder directly. We contacted the documentum team, however they've said that only people on the Documentum team can connect to that folder. We just need the Cognite extractor to have access to that Rendition repository. Please let me know if you need more information. project - hess-dev, hess-us

I have a lot of timeseries objects in CDF datasets. I have a particular set of timeseries tags out of the innumerable list of timeseries objects (16 of them) and for each tag, there are a bunch of sub-tags (4 of them). So, in total, I will need to maintain a hierarchy of 16 tags and each having 4 tags and the overall total of 64 tags. I need to go and retrieve the datapoints for each of those 64 tags. So how do I store my desired list of timeseries tags along with their child tags within CDF. Where do I store them and maintain them. This list may be edited and needs flexibility to be edited based on business users need. IT is completely the enterprise choice.  Please share complete steps simulate them in CDF. This is actually to be done for yield-tracking analytics and all these tags corresponding to the yield groups/products in a refinery.  

i need support for the course Python SDK Transformations.Im running the comands from notebook that was given in git and a stoped on:result = client.transformations.run(asset_transformation.id, wait=False)The error resulted is :CogniteAPIError: Invalid source/destination credentials: Could not authenticate with the OIDC credentials. Please check your credentials. | code: 403 | X-Request-ID: 78e0e54a-a3ca-9bc4-b2f5-e303ed6a7207iv checked the authentication process and i can perform all task like search list add datasets assets and everything but i cant run the transformation.Any one knows how to solve it ?i have the same problen in the UI  

I have many thousand timeseries where I need to find the date of the first datapoints in each series. For each timeseries, I have no idea if the first datapoint is from this year, or from 20 years ago. Fetching data for several decades is not effective.  Any ideas how to get the first datapoint? Getting the last datapoint would also be great.  Thanks!

Executing transform from Raw/Staging to Dataset Events. Codeselect cast(`uniqueid` as BIGINT) as id, cast(`Start Time` as TIMESTAMP) as startTime, cast(`End Time` as TIMESTAMP) as endTime, 1626362640169782 as dataSetId, concat("Chem-Batch", "-", Unit, "-", `Batch ID`) as description, 'Process' as type, 'Batch' as subtype from `Chem-Batch`.`Batch-Unit`where `Start Time` > '2023-06-13'Preview shows expected 3 row results: Run yields this error:Request with id 90f5d941-7f0e-9553-9ca5-317d784434d1 to https://az-eastus-1.cognitedata.com/api/v1/projects/ra-istc-sandbox/events/update failed with status 400: Event id not found. Missing ids: [48, 49].It seems all columns data in Raw rows has appropriate data.Any ideas are welcome.Chris  

Hello Community,we have a option scheduling in cognite functions to run those functions periodically. But is there any way to trigger Cognite function when a particular event occur.thank you

A few quick questions to help with a potential client engagement focusing on generating insights from unstructured docs in file systems:Can we go beyond the 1MB limit to allow OCR, index and search of any free text within the documents. Is this configurable?   LLM/NLP Search: What level of fuzzy or NLP searches are supported on the indexed content of a document. For example could one search for a phrase that isn’t verbatim in the file, but close enough in meaning or spread out within a couple of places within the document.   Automation: I understand we can auto-extract file/folder names and even text from the initial portion of a document to create metadata. Can this metadata be used in an automated entity matching service to automatically relate (contextualize) a new document to the parent field, well, facility etc. Can the same automation work by purely relying on the text within the OCRed document to avoid the human having to categorize into folders and ensuring naming standards f

Hi all - I’m new to CDF.I’d like to call CDF from JavaScript running in a browser using a bearer token as authentication.This request works fine when made from a tool such as Postman, but when the same call is made from JavaScript running in a browser using the same URL and bearer token, the browser’s CORS policy blocks the request because CDF has not responded to the preflight OPTIONS request with the HTTP header Access-Control-Allow-Credentials: trueApart from this, CDF is correctly sending out all the other HTTP headers needed to satisfy the browser’s CORS policy such asAccess-Control-Allow-HeadersAccess-Control-Allow-MethodsAccess-Control-Allow-OriginAccess-Control-Expose-HeadersAccess-Control-Max-AgeWhere do I configure CDF to respond with this additional HTTP header?Thanks  

I have to build two 2 types of extractors.1. PI extractor to connect to the PI server and fetch the data and ingest into CDF2. SharePoint online - Extract data from files present in SharePoint online and ingest into CDF. I wanted to know the construct in building the extractors. Mainly wanted to account for scenarios where the PI server is not available and PI extractor is unable to fetch the data from the pipeline. How to handle these kinds of situations and incorporate them in the code while building the extractor. Also, how to handle monitoring while performing extractor. Are there some sample code repos that can be referenced for getting complete idea for building extractors.    

Is it possible to gain access to the Github repo for the InDSL? I will use Charts in my work, and would like to be able to contribute to the development.

We have a unique use case where there are data points collected from PI server for refinery plant and alongside, we have other refinery data like Crude assays, Diet, Mass Balance. All these data points (in tabular format) are collected and then some formula is applied across to get properties (like swing cut %, CBLISS, Vol%, etc.) and some derived tables are created. Then some input feed is made from the actual data sent to a Petro-Sim tool (math simulation tool/modeling) and then output from that tool is collected and stored. All these data wrangling is done and finally charts are finally made for some 50+ crude product variants to measure yield tracking (Actual data, non-linear model data, linear model data). As per design, we are planning to use the CDF raw tables and then do all the computations and create derived raw tables to fulfil the purpose. Then use Petro-Sim connector and then store the data output from the model tool as well in raw tables. So, we don't tend to see the typi