Skip to main content

Product Ideas Pipeline

Filter by Idea Status

Filter by Topic

1354 Ideas

Ibrahim Al-Syed
Practitioner ⭐️
Ibrahim Al-SyedPractitioner ⭐️

Improve CDF Raw StoreParked

Current Observations:When joining multiple Tables (in Cognite Raw) the Transformation (query) is taking a  very long time Cognite is advising that Cognite Raw is a “Key-Value” store that is not optimized for running queries with multiple joins And that, currently, it is not possible to index or partition data in Cognite Raw to support efficient querying of data Given these limitations: Cognite Raw is not an ideal solution to store unfiltered data from source systems, in its original state Cognite Raw therefore should not be considered as equivalent to a Data or Delta Lake “Raw Zone” or “Bronze Layer” Other issues:It is not easy to debug or identify error conditions for long running transformation queries e.g., a long running query (with multiple joins, with say million records in few tables) can fail after ~10 hours if there are invalid node references all this after reading millions of records – only to fail with an error message that a node reference is invalid or that there is a duplicate key Joining on a subset of data (using inner queries) – to limit the number of records processed – still results in full table scan i.e., long running transformation Workaround - Recommendations:Execute long running joins (transformation queries) on source side (say, SAP) – Not always possible Consider piecemeal data ingestion and storage, e.g., by Site Code .. to store data in multiple tables (one for each site), instead of storing data for all sites in one table Using an intermediate data storage and processing layer (between Data Source and Cognite Raw) e.g., Snowflake, Databricks or any other platform It is really not ideal for us to consider other data lakes and solutions to mitigate these constraints.

Akash Sood
MVP ⭐️⭐️⭐️⭐️⭐️
Akash SoodMVP ⭐️⭐️⭐️⭐️⭐️

Unable to see work orders created from Maintain in InFieldParked

Hi Team,We had recently deployed both InField and Maintain for an internal slb team that is showcasing these apps to other customers.The issue that is being faced is that the work orders/activities being generated and planned in Maintain must be executed via tasks under Checklists in InField - this is the complete lifecycle of a work order/activity that is being expected.As mentioned by @Omar Khalil and @STEWART WALLACE in Cognite Hub, we are able to turn a work order in InField space to a Checklist task. But the customer is interested in using Maintain and InField together - an app which creates and plans tasks (i.e. Maintain) to plan and execute them (in InField).We found below issues while troubleshooting:Currently, InField and Maintain use different instance space - For InField, this space is created via the Cognite toolkit whereas for Maintain we use APM_SourceData. Also, since we have two separate configurations for these apps in the APM_Config data model, this is not being achieved. Question is: How do we ensure cohesion between these two apps to achieve the above? Currently it seems the apps are in silos. Steps followed: a. Open Maintain with InField’s configuration and create a new activity. b. It is expected that this work order is now visible in InField We were unable to create an activity from UI in Maintain’s current version - seems a bug in the current build We found that start date is not a mandatory field in Maintain Activity creation - Start date should be mandatory as InField’s work order filters work orders in the Overview section. While troubleshooting we added this one manually in the instance created in Activity view We were finally able to see the work order created in Maintain (with InField’s config selected) in InField when we updated source property of the Activity instance from APP to SAP - Seems InField only shows work orders that are SAP or not APP. Another bug while performing this flow in Maintain was that the test plan was selecting the entire Activity even though it was not selected. Ideally, only the activity selected should be associated to a plan. If nothing is selected, it should not allow you to proceed.@Aditya Kotiyal @kelvin @Kristoffer Knudsen @STEWART WALLACE 

APSHANKAR Sagar
Seasoned ⭐️⭐️
APSHANKAR SagarSeasoned ⭐️⭐️

Synthetic Time Series : Set Points and Process ValuesGathering Interest

As in any industry, we have a lot of time series which correspond to Process Values and Set Points. I set out to do a seemingly simple task. Calculate a new time series that was 1 when the PV was +-10% of the SP and 0 otherwise. The first issue that I faced is that we can’t do logic calculation with Snthetic TS expressions but I won’t go into detail about that since I already made a post about it before. The second issue was unexpected. The Set Point temperature of an oven or other industrial material is seldom modified. For good quality, consistent set points are important. Thus, there are far fewer points in a Set-Point time series than in a Process-Value time series. I was trying to calculate this time series for 11h00 : 11h05.  I had a set point set to 180 DEG_C at 09h00. My process values were in the hundreds, about one every 5 seconds (or whenver the temperature changed beyond a 0.1 DEG_C threshold). With data like this, if I try calculating the expression “PV, SP*0.9, SP*1.1” between 10h00 and 11h00, it will result in NAs for the SP columns! This is because the last known value of the SP is at 09h00 and there are no known SP points between 11h00 and 11h:05. My workaround was to retrieve_latest for the PV and use the time stamp to write to a NEW time series with the default time_series.data.retrieve endpoint, all values within 11h00 and 11h:05 and include_outside_points in that write, and then also forward fill it so that I had SP points in between 11h00 and 11h05. In a prior version of my program, I deleted the new time series at the end of the program but CDF started to give me Timeout errors at the delete (Not 429, just time out). I assume I hit some kind of API limit. We deal with hundreds of such pairs of Set Points and Process Values. I feel that there should be a metadata field in a time series that allows it to be filled-down. That is to say, an indicator that if the last point is at 9h00 and I ask for data at 11h00, CDF can safely assume that no new points indicate no change in value, up to the datetime.now().