Skip to main content

Product Ideas Pipeline

Filter by Idea Status

Filter by Topic

1354 Ideas

Niranjan Madhukar KarvekarSeasoned ⭐️⭐️⭐️

How to use is_new() function with lastUpdatedTime from FDM type in transformationPlanned for development

Hi, When the data is in CDF RAW then we can use is_new function with lastUpdatedTime  in transformation and this works fine but how can we use that function for data coming from FDM model. Let’s say, from CDF RAW we are populating the data in FDM domain model and we are considering only incremental data with is_new function in transformations.This is the domain model (space: dm):type Entity @view(space: "dm", version: "1"){name:String} This is transformation query to populate above Entity type:select  distinct concat('wellbore:',WellUWI)  as externalId,  WellName as name, FROM  `db1`.`wellbores`where is_new(        "db1:wellbores",        lastUpdatedTime      )This is good and we can see only incremental data from CDF RAW in the domain model.  Now this is the solution model (sol_model):type WELLBORE{name:String }This is transformation query to populate above WELLBORE type ( I can directly map the name of WELLBORE of solution model to ENTITY of domain model but for this example I am ignoring it to understand is_new() function):select  entity_table.externalId as externalId,  entity_table.name as name FROM  cdf_nodes('dm','Entity','1') as entity_tablewhere is_new(        "dm:entity",        entity_table.lastUpdatedTime      )Here, I can’t use lastUpdatedTime because even though it is internal attribute of Entity type, it is not possible to get it in the result of cdf_nodes . The cdf_nodes doesn’t give us internal attributes. How can we make sure, the solution model transformations only consider the incremental data coming from domain model?  Am I missing anything here?   

Oussama ALLALI
Seasoned ⭐️⭐️
Oussama ALLALISeasoned ⭐️⭐️

Request for Asynchronous Functionality in Time Series Subscriptions APIImplemented

Dear Team,I am reaching out to provide feedback and suggest enhancements for the new Cognite subscription API, particularly concerning its use in continuous data streaming applications.Challenge with Time Series Subscriptions API:We are currently utilizing your Time series subscription API in a project that requires continuous data streaming. However, the API’s synchronous nature has presented us with significant challenges. The main issue is the inherent latency in data processing, which is exacerbated by the need to implement a sleep mechanism with a fixed delay. Determining the optimal sleep duration is challenging; too short a delay leads to frequent polling, while a longer delay results in data lag.Benefits of Asynchronous Streaming:Implementing asynchronous functionality in the API would greatly alleviate these issues. Asynchronous methods would enable more efficient and responsive data handling, reducing latency and providing a smoother data stream. This is particularly crucial in applications where timely data processing and analysis are vital.Suggestions for Improvement:We suggest the addition of asynchronous streaming capabilities to the API, such as non-blocking data fetches or real-time push mechanisms. This improvement would align with industry standards and significantly enhance the API's utility for various applications, especially those requiring real-time data processing.Best regards,Oussama

Ibrahim Al-Syed
Practitioner ⭐️
Ibrahim Al-SyedPractitioner ⭐️

PowerBI Live connectionParked

it is our intent that the PowerBi can be used as a real time visualization tool for many use cases including: Analytics on Compressor Efficiency Real time monitoring off Natural Gas Consumption in our methanol unit Work Order Wrench Time Optimization Shift Schedule and OT optimization The problem with PowerBi connector is that it currently treats CDF as a data source and then imports data on a scheduled basis. There is no federated connection available where data is not imported or duplicated. Microsoft has confirmed this issue lies with Cognite and not on their side.  The current Powerbi connector forces an import and does data duplication in the Powerbi space. This method has two names, some call it Import Data, and some call it Scheduled Refresh. Both names explain the behavior of this method. With this method data from the source will be loaded into Power BI. Loading in Power BI means consuming memory and disk space. As long as you are developing Power BI on your machine with Power BI Desktop, then it would be the memory and disk space of your machine. When you publish the report on the website, then it will be the memory and disk space of the Power BI cloud machines on AZURE.If you have 1 Million rows in a source table, and you load it into Power BI with no filtering, you end up having the same amount of data rows in Power BI. If you have a database with 1000 tables, however, you only load 10 of those tables in Power BI, then you get memory consumption for only those 10 tables. Bottom line is that you spent memory and disc space as much as you load data into Power BI. This also has limitation on how much can be imported and what frequency. This not only limits us in scale but does not provide data in real time.PowerBi live connection is very similar to DirectQuery in the way that it works with the data source. It will not store data in Power BI, and it will query the data source every time. Power BI will be a visualization layer, then query the data from the data source every time. Power BI will only store metadata of tables (table names, column names, relationships…) but not the data. Power BI file size will be much smaller, and most probably you never hit the limitation of the size because there is no data stored in the model.Because these data sources are modeling engines themselves, Power BI only connects to these and fetches all model metadata (measure names, attribute names, relationships…). With this method, you need to handle all your modeling requirements in the data source which in our case is CDF. and Power BI just surfaces that data through Visualization.I hope this make sense.