Skip to main content
Seasoned ⭐️⭐️
Nov 17, 2026 7:34 AM UTC
Question

cognite-data-quality: uniqueness validation fails when the view filter contains a nested filter

We use cognite-data-quality==0.4.9 to validate an external DataProduct. The uniqueness rule (dqs:uniquenessConstraint) works on views with simple filters, but fails on a view whose filter contains a nested filter.

Setup

The Wellbore`view has a filter that follows a direct relation to WellState and keeps only non-archived records:

- nested:
    scope: [<schema_space>, Wellbore, WellState]
    filter:
      equals:
        property: [<schema_space>, WellState, Archived]
        value: false

 

The uniqueness rule checks a different property (TUWBI), not WellState. The nested filter comes only from the view definition.

Error

The uniqueness workflow fails, and every retry fails with the same error:


[aggregate] Aggregate query failed: Property 'WellState' does not exist in container '<schema_space>:Wellbore'
 

What we found

  • In 0.4.9, uniqueness validation uses `client.data_modeling.instances.aggregate(...)` (in `_function_code/handlers/aggregate_uniqueness_validation.py`).
  • When you aggregate on a view, the view filter is applied automatically.
  • The Cognite documentation says nested filters are not supported in the search and aggregation endpoints, except for Core Data Model assets.
  • On this view, even a plain aggregate count fails. List and query requests work.
  • Another view without a nested filter passes uniqueness validation with the same setup.

How to reproduce

  1. Create a view whose filter contains a `nested` filter on a direct relation.
  2. Add a `dqs:uniquenessConstraint` rule for a scalar property of that view.
  3. Deploy with `cognite-data-quality==0.4.9`.
  4. Run the uniqueness workflow with the trigger input.

Expected behavior

Uniqueness validation works on views with nested filters, or the framework clearly reports that the view is not supported.

Temporary workaround

We currently patch the Function code at deployment time:

  • If the view or rule filter contains nested, we use the Query API with HasData(view) instead of aggregate.
  • We read all pages and count duplicate values across the full instance scope.
  • Views without nested filters still use the original aggregate logic.

With this workaround, the uniqueness check works. In our test, it read all instances (about 6,000 over 8 pages), found no duplicates and wrote a pass record.

Questions

  1. Is this a known issue? Is it fixed in a newer version, or is a fix planned?
  2. What is the recommended approach for uniqueness rules on views with nested filters?
  3. Is a full Query API scan an acceptable fallback, or do you recommend a different approach for large views?

1 reply

Practitioner ⭐️⭐️⭐️
Nov 17, 2026 10:39 AM UTC

Hi.

Your diagnosis is right. This is a known gap, and it is not fixed in a newer release, as data modelling query capacity is prioritized for Flows apps and Atlas AI Agents.

Uniqueness validation is aggregate-first. The handler calls client.data_modeling.instances.aggregate(...) grouped by the uniqueness property (TUWBI). Aggregating on a view applies that view’s filter automatically, so the nested WellState filter is part of the aggregate request. The uniqueness property is never reached.

Data Modeling does not support nested filters on the search and aggregation endpoints, except for direct relations to Core Data Model assets. WellState is a custom direct relation, so it falls outside that exception. The aggregation service then returns:

Property 'WellState' does not exist in container '<schema_space>:Wellbore'

That matches the documented limit: Nested filters are only supported for core data model assets. The same rule succeeds on a view whose filter is a plain property predicate, because that filter is legal for aggregate. Retrying the workflow repeats the same rejected request.

0.4.9 and the current 0.5 line (stable 0.5.0, and prereleases through 0.5.1b4) still run uniqueness only through instances.aggregate. There is no Query API fallback, and the failure stays AGGREGATE_API_FAILED with the raw aggregate message.

What to do

Preferred for anything beyond a small view. Keep the scheduled uniqueness workflow on aggregate. Store the predicate on the Wellbore container itself — an indexed boolean such as archived, maintained when WellState changes — and filter the view with equals on that property. Aggregate can apply that filter, and duplicate detection stays a server-side group-by.
Valid while the nested filter has to stay on the view. Use the Query API (instances.query). hasData on a view that defines an explicit filter uses that filter (hasData), and Query can evaluate WellState.Archived = false. Your deployment patch is the right shape: if the view filter contains nested, page with Query and count duplicate values in the function. Leave views without a nested filter on the aggregate path. Include the same instance-space scope the Data Product uses, page until the cursor is exhausted, and skip null keys. Aggregate uniqueness ignores grouped nulls.
Is a full Query scan acceptable?

At the size you tested, yes. About 6,000 instances over 8 pages is a reasonable fallback, and a pass record from a complete scan is a valid uniqueness result.

For a view that will grow well past that, treat the scan as a mitigation. Query has no server-side group-by, so every scheduled run reads the whole population, and each nested filter adds joins. Watch the function timeout and memory, select only the uniqueness property, and run that schedule less often than views that can use aggregate. The built-in overflow behavior (more than 100 distinct duplicate values, or more than 100 instances for one value) is part of the aggregate executor. A full in-function count does not emit global_overflow or value_overflow unless you add that logic yourself.

If this view will keep growing, option 1 is the approach to keep. Option 2 matches the test you already ran.