Skip to main content

I have a use case where I need to filter events from CDF based on a “string.contains” pattern. Is this possible with the current API? 

Consider the following metadata field for an event in CDF. 

"EventData": "t{"version":"2.0.0"},{"type":"opcae"},{"id":"1d21002a-77a2-4531-a9bf-27f28876f2e1"},{"source":"47-PA-9810B-M01"},{"time":"5/29/2024 8:56:41 AM"},{"eventType":"Simple"},{"eventCategory":3439269159},{"eventCategoryName":"Process Simple Event"},{"severity":404},{"message":"Start"},{"eventCounter":187718259},{"Class":127},{"ObjectDescription":"PH PotWtr Pump B"},{"PriorityLevel":5}]"

The value for `EventData` is a JSON string that has not been unpacked as proper metadata fields. Several of these fields are relevant for us to filter events down to the relevant subset.

It is unclear from the documentation whether the API supports the “string.contains” pattern. The only class that maybe sounds like it is relevant is the Search class. This is the documentation:

Search

The Search filter performs a full-text search on a specified property.

final classcognite.client.data_classes.filters.Search(property: Unionmstr, Tuplestr, ...], Listpstr], EnumProperty], value: Union/str, float, bool, Sequence, Mapping,str, Any], Label, PropertyReferenceValue, ParameterValue])

 

No detailed explanation. No examples.

Can anyone elaborate on how this use case can or cannot be solved with the API? Currently we fetch maybe 100k events and filter down to maybe 500 that are relevant. In other words, we really want to be able to filter on the EventData string-

Thanks for any advice!

Digging around the docs lead me to this page: https://developer.cognite.com/dev/concepts/resource_filtering_dsl/

 

So `Search` does not allow for string.contains matching., and my use case cannot be solved. Correct?

Hi Anders,

Unfortunately yes, that is a limitation of our Events API as it stands today.

I have previously spoken with some of your colleagues about the challenge associated with this particular format of data where the key value pairs are nested within a single metadata key.  I believe there are moves underway to fix this problem in your OPC UA gateway so that all future data will be stored as properly structured key value pairs and not nested like this.  

In the future, OPC UA event data will be stored in a new storage API we are building called Industrial Log Analytics.  When this is available, each metadata value can be stored in a ‘first class’ property with much richer sorting, filtering and aggregation capabilities that the current Events API.  Please let me know if you’re interested in hearing more about any of this and I’ll be happy to have a call with you.

In the meantime, I’d suggest investigating the use of transformations on these particular events to promote these nested key value pairs ‘up a level’ so they are stored as proper metadata keys in the JSON object.  If you can do that, you will have search and filtering capabilities on each metadata key.

Kind Regards, Glen
 

Reply


Terms & ConditionsCookie Settings