Skip to main content

Product Ideas Pipeline

Filter by Idea Status

Filter by Topic

1353 Ideas

Oussama ALLALI
Seasoned ⭐️⭐️
Oussama ALLALISeasoned ⭐️⭐️

Feature request — native CDF Time Series push for metrics in PI Extractor and DB ExtractorGathering Interest

ContextWe currently run several custom Python extractors built on cognite-extractor-utils. For those, cognite-extractor-utils lets us configure a metrics.cognite block that periodically pushes the extractor's Prometheusmetrics (Counter/Gauge) directly to CDF Time Series — this gives us consistent,queryable observability (row counts, run duration, throughput, failure counts, etc.)across all our extractors, without any extra tooling.ProblemThe PI Extractor and the DB Extractor (official Cognite binaries) already exposePrometheus metrics internally (e.g. pi_extractor_timeseries_states,pi_extractor_stream_iterations, pi_extractor_pi_data_pipe_events,pi_extractor_streamer_data_points, pi_extractor_event_queue_size,pi_extractor_pi_data_loss, pi_extractor_pi_connection_time,pi_extractor_pi_connections, extractor_utils_cdf_datapoints, …), but today they canonly be exposed via a local Prometheus scrape endpoint (metrics.server) or a PushGateway (metrics.push-gateways). There is no equivalent of metrics.cognite to pushthese metrics natively to CDF Time Series, unlike our custom extractors.ImpactWithout native CDF Time Series push, we can't monitor these extractors the same way as our custom ones (no unified dashboarding/alerting in CDF/Grafana). We want to keep CDF as the single source of truth for observability, not add a separate tool just to collect metrics.RequestCould you add support for a metrics.cognite configuration block (or equivalent) in thePI Extractor and DB Extractor, consistent with the one already available viacognite-extractor-utils, so their internal Prometheus metrics can be pushedperiodically to CDF Time Series (external-id prefix, target data set, push interval)?