cognite-data-quality: incremental sync trigger is not created when a view has many instance spaces
We use cognite-data-quality==0.4.9 to validate an external DataProduct with config_source: dataproduct. Each view reads from 24 instance spaces and has use_sync_cursor_mode: true.
Historic validation runs, but incremental validation does not work. We found two separate length-limit issues.
Trigger metadata is longer than 255 characters
The historic orchestrator stores json.dumps(instance_spaces) in the workflow trigger's metadata.instance_spaces.
With 24 spaces, this value is 360 characters. CDF allows at most 255 characters per trigger metadata value, so creating the trigger fails:
[':metadata']: Values are limited to 255 characters
The orchestrator catches this error and reports Sync trigger: not created, so it is easy to miss.
Lock and cursor IDs are longer than 256 characters
The lock and cursor external IDs include the full sorted list of instance spaces. With 24 spaces, they are about 330–335 characters, which is above the 256-character external ID limit:
Sync lock acquire skipped (load error: ... externalId size must be between 1 and 256 ...)
When the incremental workflow runs, this lock-read error is reported as:
{"status": "skipped", "reason": "concurrent_execution"}
The workflow shows Completed, so it looks as if another run is holding the lock. In reality, the ID is invalid and no data is validated.
How to reproduce
1. Use `cognite-data-quality==0.4.9` with an external DataProduct.
2. Set `use_sync_cursor_mode: true` for a view.
3. Configure enough instance spaces to exceed the limits. We used 24; the exact number depends on the length of the space and view names.
4. Run `deploy_validation_pipeline` for that view.
Expected behavior
- The incremental trigger is created regardless of the number of instance spaces.
- Lock and cursor IDs stay within the external ID limit.
- A lock-read error is reported as an error, not as
concurrent_execution.
Temporary workaround
We currently patch the Function code at deployment time:
- If a lock or cursor ID is longer than 256 characters, we use a SHA-256-based ID instead.
- If the metadata value is longer than 255 characters, we store `"<count> spaces; sha256=<hash>"`. The full list of spaces is still kept in the trigger input.
- If the lock cannot be read, we raise an error instead of returning
concurrent_execution.
With this workaround, historic and incremental validation work for all 24 spaces.
Questions
- Is this a known issue? Is it fixed in a newer version, or is a fix planned?
- What is the recommended approach for views with many instance spaces?
Check the
documentation
Ask the
Community
Take a look
at
Academy
Cognite
Status
Page
Contact
Cognite Support
