cognite-data-quality: uniqueness validation fails when the view filter contains a nested filter
We use cognite-data-quality==0.4.9 to validate an external DataProduct. The uniqueness rule (dqs:uniquenessConstraint) works on views with simple filters, but fails on a view whose filter contains a nested filter.
Setup
The Wellbore`view has a filter that follows a direct relation to WellState and keeps only non-archived records:
- nested:
scope: [<schema_space>, Wellbore, WellState]
filter:
equals:
property: [<schema_space>, WellState, Archived]
value: false
The uniqueness rule checks a different property (TUWBI), not WellState. The nested filter comes only from the view definition.
Error
The uniqueness workflow fails, and every retry fails with the same error:
[aggregate] Aggregate query failed: Property 'WellState' does not exist in container '<schema_space>:Wellbore'
What we found
- In 0.4.9, uniqueness validation uses `
client.data_modeling.instances.aggregate(...)` (in `_function_code/handlers/aggregate_uniqueness_validation.py`). - When you aggregate on a view, the view filter is applied automatically.
- The Cognite documentation says nested filters are not supported in the search and aggregation endpoints, except for Core Data Model assets.
- On this view, even a plain aggregate count fails. List and query requests work.
- Another view without a nested filter passes uniqueness validation with the same setup.
How to reproduce
- Create a view whose filter contains a `nested` filter on a direct relation.
- Add a `dqs:uniquenessConstraint` rule for a scalar property of that view.
- Deploy with `cognite-data-quality==0.4.9`.
- Run the uniqueness workflow with the trigger input.
Expected behavior
Uniqueness validation works on views with nested filters, or the framework clearly reports that the view is not supported.
Temporary workaround
We currently patch the Function code at deployment time:
- If the view or rule filter contains nested, we use the Query API with
HasData(view)instead of aggregate. - We read all pages and count duplicate values across the full instance scope.
- Views without nested filters still use the original aggregate logic.
With this workaround, the uniqueness check works. In our test, it read all instances (about 6,000 over 8 pages), found no duplicates and wrote a pass record.
Questions
- Is this a known issue? Is it fixed in a newer version, or is a fix planned?
- What is the recommended approach for uniqueness rules on views with nested filters?
- Is a full Query API scan an acceptable fallback, or do you recommend a different approach for large views?
Check the
documentation
Ask the
Community
Take a look
at
Academy
Cognite
Status
Page
Contact
Cognite Support
