Seasoned ⭐️⭐️

Gathering Interest

Feature request — native CDF Time Series push for metrics in PI Extractor and DB Extractor

Related products:Extractors

Forum|Forum|5 days ago
August 6, 2026
5 replies
64 views

Context

We currently run several custom Python extractors built on cognite-extractor-utils. For those, cognite-extractor-utils lets us configure a metrics.cognite block that periodically pushes the extractor's Prometheus
metrics (Counter/Gauge) directly to CDF Time Series — this gives us consistent,
queryable observability (row counts, run duration, throughput, failure counts, etc.)
across all our extractors, without any extra tooling.

Problem

The PI Extractor and the DB Extractor (official Cognite binaries) already expose
Prometheus metrics internally (e.g. pi_extractor_timeseries_states,
pi_extractor_stream_iterations, pi_extractor_pi_data_pipe_events,
pi_extractor_streamer_data_points, pi_extractor_event_queue_size,
pi_extractor_pi_data_loss, pi_extractor_pi_connection_time,
pi_extractor_pi_connections, extractor_utils_cdf_datapoints, …), but today they can
only be exposed via a local Prometheus scrape endpoint (metrics.server) or a Push
Gateway (metrics.push-gateways). There is no equivalent of metrics.cognite to push
these metrics natively to CDF Time Series, unlike our custom extractors.

Impact

Without native CDF Time Series push, we can't monitor these extractors the same way as our custom ones (no unified dashboarding/alerting in CDF/Grafana). We want to keep CDF as the single source of truth for observability, not add a separate tool just to collect metrics.

Request

Could you add support for a metrics.cognite configuration block (or equivalent) in the
PI Extractor and DB Extractor, consistent with the one already available via
cognite-extractor-utils, so their internal Prometheus metrics can be pushed
periodically to CDF Time Series (external-id prefix, target data set, push interval)?

Mithila Jayalath

Expert ⭐️⭐️⭐️⭐️

@Oussama ALLALI shall I convert this into a product idea?

Oussama ALLALIAuthor

Seasoned ⭐️⭐️

yes please !

Mithila Jayalath

Expert ⭐️⭐️⭐️⭐️

@Oussama ALLALI done. Please make sure to upvote the idea.

Peter Arwanitis

Practitioner ⭐️⭐️⭐️

Thank you for raising this gap how Cognite today is supporting metrics, especially pushing them to Cognite TS.

The issue of a centralized and shared metrics-tracking and observability are recognized capabilities, across extractors, workflows, flows and agents. Cognite already added them under the name “Integration Services” to the external roadmap.

Let me check when this new service is available, the planned rollout of capabilities and the Cognite roadmap when existing components (like PI Extractor) are planned to support it.

May I ask in the meantime how critical the issue you raised is for your operations? And if you see options to manage until the described change is delivered?

Oussama ALLALIAuthor

Seasoned ⭐️⭐️

Thank you @Peter Arwanitis

Criticality: It matters to us. It's part of our monitoring roadmap this quarter, so we'll likely need to move ahead with a solution on our side in the meantime. That said, we're also working towards a broader goal of centralizing observability across platforms (Cognite, Databricks, etc.) in a single company-wide monitoring application, so native support would still be a nice fit for us long-term.

Workaround: We're considering putting in place a small scraping module in our extractor launcher that would poll the local Prometheus endpoint of these binaries and forward the metrics to our monitoring pipeline. Nothing built yet, but it looks like a viable option to manage until a native solution is available.

Could you share a bit more on the Integration Services plans in the meantime. Whether it would land as a native capability in the extractors themselves or as a separate service to integrate with, and even a rough indicative timeline? That would help us decide how much to invest in the workaround versus waiting.

Sign up

Welcome to Cognite Hub