Hi Cognite Community,We are looking at ingesting approximately 1 million engineering and operational documents into CDF, including equipment manuals, datasheets, inspection reports, SIL/HAZOP studies, operating procedures, technical studies, maintenance reports, and other asset-related documents.Our key requirement is for an Atlas AI agent to answer questions across this entire document repository without the user having to identify or pinpoint a specific document first.For example, a user should be able to ask:“What are the recurring compressor seal issues reported across our engineering and operational documents?”The user may not know which documents contain the relevant information. We therefore expect Atlas AI to discover the appropriate documents/passages, retrieve the relevant content, and generate a grounded response with references to the source documents.According to Cognite's documentation, CDF Semantic Search does create vector embeddings for PDF documents as part of the built-in RAG pipeline. Documents are parsed/OCRed, divided into passages and indexed in a vector store for semantic retrieval. Semantic search - Cognite DocsWe would therefore like to understand how this capability works with Atlas AI at enterprise scale:Can Atlas AI perform semantic retrieval across a repository of approximately 1 million documents without specifying document IDs or filenames in advance? How does Atlas AI determine which documents/passages are relevant to a user's question before sending context to the LLM? Does Atlas AI directly use CDF's document Semantic Search / built-in RAG pipeline for this retrieval? Are embeddings automatically created and maintained for all supported documents ingested into CDF, or does this require additional configuration? Are there practical scalability, indexing, latency, document-count, passage-count, or cost limitations that we should consider at this volume? Can semantic retrieval be combined with CDF metadata and knowledge graph relationships, for example Asset → Equipment → Document, so that semantic similarity and industrial context are used together? How are document updates, revisions, superseded documents, and re-indexing handled? What document formats are supported by the RAG/semantic-search pipeline at this scale, particularly beyond PDF? What architecture and ingestion/indexing approach would Cognite recommend for a document estate of this size?This is important for us because the target experience is enterprise-wide conversational discovery across engineering and operational knowledge, rather than asking users to locate a document first and then ask Atlas AI questions about it.Would appreciate guidance from Cognite on the recommended architecture and any reference implementations or scale benchmarks for this type of use case.

Committed ⭐️⭐️⭐️

Question

Atlas AI: Semantic Search Across Large-Scale Engineering & Operational Document Collections

2 replies
22 views

Hi Cognite Community,

We are looking at ingesting approximately 1 million engineering and operational documents into CDF, including equipment manuals, datasheets, inspection reports, SIL/HAZOP studies, operating procedures, technical studies, maintenance reports, and other asset-related documents.

Our key requirement is for an Atlas AI agent to answer questions across this entire document repository without the user having to identify or pinpoint a specific document first.

For example, a user should be able to ask:

“What are the recurring compressor seal issues reported across our engineering and operational documents?”

The user may not know which documents contain the relevant information. We therefore expect Atlas AI to discover the appropriate documents/passages, retrieve the relevant content, and generate a grounded response with references to the source documents.

According to Cognite's documentation, CDF Semantic Search does create vector embeddings for PDF documents as part of the built-in RAG pipeline. Documents are parsed/OCRed, divided into passages and indexed in a vector store for semantic retrieval. Semantic search - Cognite Docs

We would therefore like to understand how this capability works with Atlas AI at enterprise scale:

Can Atlas AI perform semantic retrieval across a repository of approximately 1 million documents without specifying document IDs or filenames in advance?
How does Atlas AI determine which documents/passages are relevant to a user's question before sending context to the LLM?
Does Atlas AI directly use CDF's document Semantic Search / built-in RAG pipeline for this retrieval?
Are embeddings automatically created and maintained for all supported documents ingested into CDF, or does this require additional configuration?
Are there practical scalability, indexing, latency, document-count, passage-count, or cost limitations that we should consider at this volume?
Can semantic retrieval be combined with CDF metadata and knowledge graph relationships, for example Asset → Equipment → Document, so that semantic similarity and industrial context are used together?
How are document updates, revisions, superseded documents, and re-indexing handled?
What document formats are supported by the RAG/semantic-search pipeline at this scale, particularly beyond PDF?
What architecture and ingestion/indexing approach would Cognite recommend for a document estate of this size?

This is important for us because the target experience is enterprise-wide conversational discovery across engineering and operational knowledge, rather than asking users to locate a document first and then ask Atlas AI questions about it.

Would appreciate guidance from Cognite on the recommended architecture and any reference implementations or scale benchmarks for this type of use case.