Community Discussions

Hi there,Are there any plans to make the PI Extractor open source, following the same approach as opcua-extractor-net?

Hi! We have made the following updates available for Workflows.Optional TasksDefining non-critical tasks in a workflow as optional, meaning that in the event of a task failure, the workflow itself should continue, has been a much requested feature. You can now set the policy for how to handle failures and timeouts for each task in a workflow. By default, the workflow will fail if a task fails (after retries) or if the task times out. Alternatively, you can set the parameter to skip the task in the event of a failure or timeout (after retries). This means that the workflow execution will continue even when the task does not complete successfully. The feature is documented in the API specififcation here (navigate items > workflowDefinition > tasks > onFailure). Concurrency Policies for TransformationsIf a job is already running for a given Transformation in CDF, a new job cannot be started concurrently (fails with error message “A job already runs for this transform”). To better

Hi Community Members!We’ve recently published a cloud-based Virtual Flow Metering (VFM) system tailored for a late-life well with high water cuts. This solution employs a hybrid physics-data approach, delivering precise and near-real-time estimates of water production per well in gas fields.Link to the full article: https://www.sciencedirect.com/science/article/pii/S277250812300042X Enhancing Production OptimizationIn the oil and gas production domain, understanding the fluid composition and production rates of multiphase operations is crucial for effective planning and optimization. With wells yielding a diverse mix of oil, gas, and water, often with varying compositions, ensuring accurate measurements is essential for maximizing production efficiency. Ratios change in time, and this gets more and more important as the water cut grows in the late part of a water flooded well life. This article explores how Virtual Flow Meters (VFMs) offer a cost-effective alternative to traditional me

Hello everyone,I'm relatively new to using Grafana, and after reading this post (Cognite Hub) on Cognite Hub, I've come to understand that Alerts in Grafana may not be compatible with the CDF connector. Please feel free to correct me if I'm mistaken.Considering this, I'm wondering if anyone has a good setup for monitoring the data coming from the CDF connector. I have around 900 time series, and listing all of them is not very efficient. Additionally, I'm having trouble figuring out how to write an SQL query to count the time series that have a timestamp older than, let's say, 3 hours. this could be a way, to count data not coming in. Would anyone be willing to share their setup or offer any tips?Thank you in advanced :) 

In September, we embarked on an incredible journey with our partners at CNTXT Academy, bringing Cognite Data Fusion (CDF) training to the heart of Saudi Arabia. We were thrilled to host not one, but two engaging Cognite Data Fusion Fundamentals sessions right at the epicenter of innovation - Saudi Aramco's headquarters in Dhahran. Our participants came from diverse backgrounds, each with their own unique experiences with Cognite Data Fusion. So, we tailored the training to ensure it resonated with everyone, addressing their everyday work challenges that Cognite Data Fusion can brilliantly solve.The first part of the training focused on contextualization, mastering Cognite Charts, and seamlessly connecting CDF with Grafana to tackle a real-world use case. The participants got hands-on experience, putting their newfound knowledge to the test right in CDF. In the second part, our exceptional instructors introduced participants to the Cognite Python SDK, taking their skills to the next lev

Hi Team, I need help to write the unit test using pytest,below is the code:    row_list = cognite.raw.rows.list(file.destination.database, file.Extractor_Last_Run_Table, limit=-1,                               columns=["key", "LastUpdatedDatetime"]).to_pandas()    if not row_list.empty:        val = list(row_list[row_list.index.isin([file.Extractor_key])]['LastUpdatedDatetime'])Could you please help to write the unit test for above code?Thanks

SOLVED:The Cognite client was set up wrong, hence the Cognite API error when uploading data to CDF. See docs posted by Peter for setting up config.yaml for the extractor. Specifically these were the changes:``` COGNITE_BASE_URL=https://westeurope-1.cognitedata.com (not Fusion URL)COGNITE_PROJECT= {some project name}COGNITE_TOKEN_URL= https://login.windows.net/{some_tenant}/oauth2/v2.0/token(Different from OpenID connect token URL)```_______________________________________________Hello curious and helpful members of Cognite Hub,I’m building my first REST extractor to GET a response (sensor data) from a camera_sensor API, where I then aim to upload the data to corresponding timeseries. I have multiple cameras with each its URL path, so it seems @extractor.get_multiple(paths) gives me the desired behaviour of essentially looping @extractor.get through the URLs. However, this error arises when my extractor tries to upload datapoints to the timeseries in CDF.^CTraceback (most recent call la

Hi, I want to transform JSON object to Data model column. JSON is populated from extractor to raw table, so its type is by default Object and its structure will be nested.Is there any built-in/ Custom function using which I can parse this json  and convert the specific value into column. I have tried to_josn () which will convert the json structure to string and then with the help of get_json_object() I can get expected value by passing json path to this function. But problem here is it will always return value as String.  

Hi, I have done the Grafana setup from azure managed grafana from azure portal. But don’t have permission to install cognite data fusion in grafana from azure. Please let me know how to install plugin and what permission needed.   

Hi,I am running the DB Extractor, where my Source is an odbc connection(SQL DB) and destination is Timeseries. But I am getting the error: “ [ERROR   ] QueryExecutor_0 - Unexpected error in UB-Training-Timeseries-objects: 'NoneType' object has no attribute 'timestamp'”Attaching the screenshot of my data in SQL, error Log and config file.I am using DB extraction v3.0.0.Please reach out if you need more info.

Hi! We have made the following updates available for Workflows.Support for Workflows in the Cognite Python SDKThe Cognite Python SDK now has native support for Workflows. Documentation can be found here. We have also updated the example notebook, which you can find attached as a file below (Note: Download the file and remove the .txt postfix to open it as a Jupyter Notebook.)New Access Management CapabilitiesYou can now configure access to Workflows directly in Access Management in CDF. Navigate to “Manage” > “Manage access” and for the relevant Groups add capabilities to read/write Workflows as needed (see screenshot below). This replaces the experimental ACLs which were required initially to interact with the service. The capabilities can also be added to a group using the SDK directly, e.g.:from cognite.client.data_classes import Groupcapabilities = [{"workflowOrchestrationAcl": {"actions": ["READ", "WRITE"], "scope": {"all": { }}}}]group = Group(name="Workflow Orchestration", ca

Learn FDM GraphQL Queries . Aggregate query not supporting sort functionality. How to achieve sort functionality in aggregate  

How to fetch documents from multiple folder locations using single config.yml file by Cognite Documentum Extractor? how to write multiple DQL queries in same config.yml? Which configuration parameters to use?I am working on Cognite Documentum Extractor and using DFC Java SDK mode for connectivity.

Hi! I am trying to update “Planned End Date” for a specified work order in staging.I have prepared this table to join to my work order raw table:This is my sql and the error message I got:Is there a way I could update “Planned End Date”? Thanks in advance. :)

@Jørgen LundWe have a use case for Workflow Orchestration feature in a project – an example being we would like to run some workflows (Cognite functions) after successful runs of certain workflows as they have a dependency between them.There are some conversion and allocation factors that vary periodically and will be updated to SharePoint via end users. Based on the dates for these numbers, we will utilize them for further data munging and calculations in other Cognite functions, given that the previous function run was successful.Can we use workflows for this? If yes, can we enable this in our project?

Hi Team, IS there any unit testing examples for cogex based extractors to test integration with rest api

Hi! I have a question about the data modelling in Cognite Data Fusion.We record data from thousands of sensors from a factory. Today we record them as time series in CDF and organized in the asset hierarchy by manually constructing the relationships.I heared the data modelling toolkit is meant to replace asset hierarchies approach in future. Could you please elaborate a bit on how Cognite would suggest to approach our use case, if we started from scratch today.Thanks!

I have a set of timeseries objects t1,t2,t3,t4,t5…..etc. Now I would like to create a derived timeseries objects based on the existing timeseries objects. For instance  T1 = (t1/1000) *141.5T2 = (t2/100 - t3)T3 = (t3/100 - 14)T4 = t4 * t2/100T5 = t5/100How do I perform these steps using cognite SDK. Is there a way to perform these operations?Please advise. Retrieving dataframe from data-points and then create new timeseries objects (T1,T2….) will be challenging and hence wanted if there is an optimal way to handle this in a better way. Is this possible using synthetic time-series?IS there some examples and code snippets that can be shared so that I can grasp them better?  

I have had this problem before but it fixed it self by just waiting. But is the connection to CDF down when you get this error in PBI? Details: "Cognite: Request failed: The remote server returned an error: (503) Server Unavailable. (Service Unavailable)"

Hi, thanks for enabling the early adopter version.I’ve created a workflow and was planning to disable all my transformations to see if we can rely only on the workflow and get better results than transformation (which sometimes fails causing data latencies), but I’m missing one part: How do we trigger the workflow to run based ion new data arrival or schedule?

Found a bug? Have a question about how something works? We want to hear about it!You can choose to either create a dedicated post (topic) in the Charts group by clicking the Create topic button OR simply post a reply below in this thread. Remember to include a screenshot or video to help the product team best understand what exactly you’re talking about or referring to. 

How do I perform upsert using this dataframe. I also need to associate the timeseries against a specified asset. Please advise. How to use this dataframe and perform associating to an asset id using this line:    client.time_series.data.insert_dataframe(df) I have a dataframe that looks like this as below: DATE LP_Crude API_Meter LP_FRN_KERO_SW_CUTPT LP_KERO_DSL_SW_CUTPT LP_DSL_AGO_SW_CUTPT LP_MVG_HVGO_SW_CUTPT 1/1/2023 28.75472705 271.8662 440.9032 680.3416 928.875 1/2/2023 28.21111702 269.3863 466.2317 686.5167 924.1292 1/3/2023 27.78340123 268.8638 484.6189 684.5542 919.9917 1/4/2023 27.5781529 269.3117 506.7792 686.6708 921.9375 1/5/2023 28.03229217 268.1567 500.2208 688.2208 917.575

Will custom data formats for hosted extractors support events as target type in the future, and if so, do you have a rough estimate on when?