Skip to main content

Product Ideas Pipeline

Filter by Idea Status

Filter by Topic

1312 Ideas

Gunnar AndreasSeasoned ⭐️⭐️

Several actions in the "Explore and select your data" view in Charts kicks you out and into Search, when using Data ModellingImplemented

When adding data in Charts via the “+Add data”→ “Add time series”→ “Explore and select your data” the behavior using classic vs data modelling is very different.  By using a CDM extended data model we see that a series of actions related to Assets push you out of the “Explore and select your data” and into full Search. We do not see the same behavior when using classic data model.Here is a non-exhaustive list of actions that kick you out of ChartsFrom an Asset, clicking on parent, an asset from the path, or children  from the “assets” reference  From a File, clicking on any Asset reference From an annotated file, clicking on the “Open” button on the popup when you click an Asset annotation From an Activity or TimeSeries, clicking on Asset references Clicking on any Asset in the “bread crumb” trail in the  “Explore and select your data”  Clicking on the back button in the “Explore and select your data”, if the last item in the “bread crumb” is an Asset What does not kick you out:Maneuvering the asset hierarchy using the “Tree view” Clicking on any non-Asset links (what I’ve seen so far) Clicking “Open” on the file link annotation from an annotated file Clicking on Files in the “bread crumb” trail in the  “Explore and select your data”  Clicking on the back button in the “Explore and select your data”, if the last item in the “bread crumb” is not an Asset Except using the “Tree view”, it seems like any Asset related action kicks you out. We do not see the same behavior using classic data model. If this is related to how we have extended our Asset, it would be good to have this documented so we can adjust our UI data model  

Andrew Wagner
Seasoned ⭐️⭐️
Andrew WagnerSeasoned ⭐️⭐️

Feature Request: API for Kafka Connector LogsGathering Interest

Strategic Objective: Establishing CDF as the Definitive Source of TruthFor Koch Ag & Energy Solutions (KAES), the primary goal is for all stakeholders to trust CDF as the most accurate and reliable resource for operational data. To achieve this, we must move away from "silent failures." We require proactive monitoring to ensure that if data is missing from CDF, the system—not the user—is the first to identify and report the gap.New Requirement: Proactive Integrity & Trust MonitoringIn addition to standard execution logs, the Kafka Connector API should provide hooks for proactive health checks:Data Freshness Latency: Real-time reporting of the "age" of the last record written to CDF versus the timestamp of the event in Kafka. Source-to-Sink Parity: Automated counters to verify that $N$ records consumed from Kafka equals $N$ records successfully ingested into CDF. Proactive "No Data" Alerts: The ability to trigger a log event or status change if a high-priority Kafka topic produces zero records over a defined threshold (e.g., 5 minutes).Use Cases: The KAES RCA & Trust AgentThe internal KAES Monitoring Agents will use these API enhancements to fulfill two roles:The Fixer (RCA): When a pipeline breaks, an agent can utilize the logs to identify the issue and attempt to resolve or give detailed RCA, saving data engineers hours of manual tracing. The Guarantor (Trust): If there is a discrepancy between source systems and Cognite models, an agent can assist in reconciling the data. Additionally, if a data stream slows down or encounters frequent issues, the agent can proactively flag these problems and recommend a plan of action for the data engineer to address.Business Value for KAESUser Adoption: Increases trust of the Connected Cognite Data Foundation for our Operations partners. Operational Excellence: Transitions the engineering team from reactive troubleshooting to managing by exception - our team spends ~50% of time on low value data pipeline issues. Scalability: Provides a standardized way to monitor thousands of concurrent data streams across the KAES enterprise. Increases the speed to deploy internal production ready solutions as data is higher quality Technical Specifics for ImplementationHealth Status API: A GET endpoint returning the current "Liveliness" and "Readiness" of specific Kafka consumer groups. Structured Error Categorization: Distinct error codes for Transient (network), Permanent (schema/logic), and Source (empty topic) issues to allow the KAES agent to categorize the RCA automatically. OpenTelemetry Integration: Support for exporting these metrics to external observability stacks.