Skip to main content

Product Ideas Pipeline

Filter by Idea Status

Filter by Topic

1298 Ideas

Oussama ALLALI
Seasoned ⭐️⭐️
Oussama ALLALISeasoned ⭐️⭐️

Feature request — native CDF Time Series push for metrics in PI Extractor and DB ExtractorGathering Interest

ContextWe currently run several custom Python extractors built on cognite-extractor-utils. For those, cognite-extractor-utils lets us configure a metrics.cognite block that periodically pushes the extractor's Prometheusmetrics (Counter/Gauge) directly to CDF Time Series — this gives us consistent,queryable observability (row counts, run duration, throughput, failure counts, etc.)across all our extractors, without any extra tooling.ProblemThe PI Extractor and the DB Extractor (official Cognite binaries) already exposePrometheus metrics internally (e.g. pi_extractor_timeseries_states,pi_extractor_stream_iterations, pi_extractor_pi_data_pipe_events,pi_extractor_streamer_data_points, pi_extractor_event_queue_size,pi_extractor_pi_data_loss, pi_extractor_pi_connection_time,pi_extractor_pi_connections, extractor_utils_cdf_datapoints, …), but today they canonly be exposed via a local Prometheus scrape endpoint (metrics.server) or a PushGateway (metrics.push-gateways). There is no equivalent of metrics.cognite to pushthese metrics natively to CDF Time Series, unlike our custom extractors.ImpactWithout native CDF Time Series push, we can't monitor these extractors the same way as our custom ones (no unified dashboarding/alerting in CDF/Grafana). We want to keep CDF as the single source of truth for observability, not add a separate tool just to collect metrics.RequestCould you add support for a metrics.cognite configuration block (or equivalent) in thePI Extractor and DB Extractor, consistent with the one already available viacognite-extractor-utils, so their internal Prometheus metrics can be pushedperiodically to CDF Time Series (external-id prefix, target data set, push interval)?

Gunnar AndreasSeasoned ⭐️⭐️

Full API access to inspect and update instances in ChartsGathering Interest

Today there is limited to no API access to Charts, creating several issues.One obvious issue is that we cannot automate the setup of Charts. Eg, when an anomaly detection method detects an issue, we would like to be able to auto-create the Charts with the necessary timeseries, timerange, calculations and overlay anomaly insight (CogniteActivity extension). This is not possible today (only pre-build url with instances and range).But another issue is around the maintainability of Charts. Due to the limit on the number or Views/revisions, we need to deprecate and delete old revisions on a regular basis (in particular in periods with heavy data model development). As far as I have seen, Atlas, Canvas and Chart lock the View version to the latest version (or maybe based on the location filter) during config. For Atlas agents and Canvas’ we can find the View versions via the API, allowing us to identify users depending on models that are deprecated, and also auto-duplicate with the updated versions/views. However, with Charts this is not possible. If Charts is supposed to only be a do-my-analysis-and-then-throw-it-away tool, that is ok, but if it is intended to be used for more long term analysis (it does offer functionality for this like calculations and threshold monitoring) we need a way to monitor status and update the views/instances in the Charts on an enterprise level (via API) 

tainabouzanPractitioner ⭐️⭐️⭐️

Ability to configure the PI Extractor to include ptsecurity and datasecurity attributes in the extracted metadataGathering Interest

 As a CDF Platform Administrator,I want the ability to configure the PI Extractor to include ptsecurity and datasecurity attributes in the extracted metadata,So that I can use these source-level access control lists (ACLs) to automate and enforce data access security for groups of users in Cognite Data Fusion.  Configuration OptionsGIVEN a standard PI Extractor installation,WHEN looking at the extractor configuration file (e.g., config.yml),THEN there must be a new, optional configuration parameter (e.g., include-internal-security-metadata: true/false or an override setting).GIVEN the configuration parameter is omitted or set to false,WHEN the extractor runs,THEN it must continue to exclude ptsecurity and datasecurity metadata as it currently does (default secure behavior). Extraction BehaviorGIVEN the configuration parameter is set to true,WHEN the extractor performs discovery and extracts Time Series metadata from PI,THEN the ptsecurity and datasecurity attributes must be successfully captured and populated as key-value pairs in the CDF Time Series metadata block.GIVEN the configuration parameter is set to true,WHEN a source PI point has empty or null values for these security fields,THEN the extractor should gracefully skip writing those specific keys to CDF metadata without throwing errors or halting the extraction process.