Extractor Monitoring Dashboard

While the extractor monitoring in CDF offers a paginated view of extractor status, it lacks a comprehensive overview of all extractors against a specific site. With over 100 pipelines, pinpointing which pipeline is not functioning poses a significant challenge.

Hi @rsiddha @Harsha , thansk for sharing your insight. A follow-up question: in addition to extractors, would it be relevant to include the other processing/transformation steps that happen within CDF (e.g. Transformations, Functios, various contextualization jobs) in the same dashboard? 

Thank you for your thoughtful insight. Regarding your follow-up question, incorporating other processing and transformation steps within CDF, such as Transformations, Functions, and various contextualization jobs, in the same dashboard could indeed offer a more comprehensive and holistic view of the entire data processing workflow.

This would provide users with a unified platform to monitor and analyze the status of extractors, along with associated processing and transformation activities. Such an inclusive dashboard could enhance the overall visibility and management of data workflows within CDF, offering a more integrated and efficient monitoring experience.   Certainly, I understand the need for a quick and intuitive overview.

The primary purpose of the dashboard, at a first glance, should be to provide a clear indication of whether everything is functioning as expected. Users should be able to easily identify the status of extractors, and if any issues arise, they can then delve into more detailed information. The initial dashboard view should offer a high-level summary, employing visual cues such as traffic signal indicators (red/green/blue), to promptly communicate the overall health of the system. This approach ensures that users can swiftly assess the situation before exploring specific details.

I totally support this and we would also like to get access and provide to our users, all the status information from the extractor side that might be available that today is transferred to Prometheus.

One could probably also show the information from status.cognite.com with dependencies to the solutions applicable to us in the same module/visualization.

this is quite important

Thank you all for the great input. We are planning improvements to the monitoring capabilities across extractors and other data onboarding processes in CDF, so I’m changing the status of this Product Idea. If you have more details or related input you’d like to share, please don’t hesitate to add more to the comments. We’ll make sure to bring all of it into the process of shaping a better monitoring experience in CDF.

do you have any update on expected timing?

@Hunter Beck @Jørgen Lund  Can we get some clarity on the timeline, for when we can expect this incorporated? Thanks