Skip to main content
Answer

Cognite Extractors for streaming real time data into cdf

  • May 25, 2023
  • 4 replies
  • 116 views
V
Active
Forum|alt.badge.img+3

Hi, I wanted to understand if Cognite Data Extractors can stream real-time data into CDF (with or without delay) for timeseries analysis , or do we need to schedule and extract a specific number of rows every time we run the extractor?

Best answer by roman.chesnokov

Hey @vaibhavsancheti25 , If you are referring to a PI extractor, it is specifically designed for continuous running and streaming of data.

Ref from doc:

The Cognite PI extractor connects to the OSISoft PI Data Archive and detects and streams time series into Cognite Data Fusion (CDF) in near real-time. In parallel, the extractor ingests historical data (backfill) to make all time series available in CDF. The PI points in the PI Data Archive correspond to the time series in CDF.

Stig Harald Gustavsen

4 replies

roman.chesnokov
Seasoned Practitioner
  • roman.chesnokov
  • Seasoned Practitioner
  • Answer
  • May 25, 2023

Hey @vaibhavsancheti25 , If you are referring to a PI extractor, it is specifically designed for continuous running and streaming of data.

Ref from doc:

The Cognite PI extractor connects to the OSISoft PI Data Archive and detects and streams time series into Cognite Data Fusion (CDF) in near real-time. In parallel, the extractor ingests historical data (backfill) to make all time series available in CDF. The PI points in the PI Data Archive correspond to the time series in CDF.

dinesh.makked
Seasoned

Roman has provided the solution for continuing stream of data from PI (no delay option).  

 

My solution is for the “delay” scenario where there is no continuous streaming of data into CDF. In our use-case, the timeseries data was stored in the Oracle database. We used the DB Extractor to set up the Data Pipeline. The extractor was set up in such a way that it extracts data incrementally. We are running the extractor every 10 minutes which provides the near real-time streaming of data. 

You can find the documentation for the extractor at About the Cognite DB extractor | Cognite Documentation. 

In the configuration section (Configuration settings | Cognite Documentation), you can find instructions about how to set up the incremental extract. 

The last note, we are running the extractor using the Windows Scheduler.

Please let me know if any additional questions.

Thanks,

Dinesh

 

 

 

 

 

 

 

 

 

roman.chesnokov
A
Active
  • adeel.ali
  • Active
  • July 7, 2023

@roman.chesnokov - will other CDF extractors, like the DB extractor, support streaming or event-driven integration? We have investigated something similar by with database change data capture [CDC] but currently do not see the streaming or event driven support with the DB extractor.

roman.chesnokov
Seasoned Practitioner

Hi @adeel.ali Unfortunately, the DB extractor right now only allows to use schedule. 

A
Terms & ConditionsCookie SettingsAccessibility statement