Data Size Limitation

Hi @Ankit Kothawade 
 

It's crucial to understand that not all data from sources needs to be copied to the staging area. Techniques like data streaming and selective data extraction can minimize unnecessary data movement, optimizing performance. Additionally, incremental ingestion, focusing on the data required for specific use cases, is highly advantageous. Cognite offers robust support for incremental ingestion capabilities.

Determining the most suitable ingestion approach often involves gathering use case requirements from data consumers and working backward. Key questions to consider include:

• What are the business needs driving the data ingestion?
• What are the expectations for the data quality, granularity, and timeliness?
• When does the data need to meet these expectations to support business objectives effectively?

The last time I read about Cognite's performance, it was reported that the largest Cognite Data Fusion® Time Series cluster stored around 15 trillion data points. It consistently handled ingesting 40 million data points per second and reading 200 million data points per second. There's an expectation for even higher scalability. However, it's important to consider your actual needs, as managing huge data volumes can become quite expensive.

Hope it can help in your decision.