Hi Zahra,
Could you elaborate on the use case a bit please? What is the data, what is the source system and how will it be consumed / what kind of applications will consume it from CDF?
When you say you want to stream this array to cloud, I assume here you mean to be stored in CDF? To which storage API are you trying to send this data?
Kind Regards, Glen
Hi Glen,
The application is to send 500Hz sensor data using OPCUA and store them as time series in CDF. The OPCUA server is Labview which doesn’t support pubsub method for high frequency data, so we use arrays to go around this problem. We split the original 500Hz data into an array of 500 elements that are updated every second (1Hz). We would like to use Cognite OPCUA extractor to send such array and store it in CDF in some datatype. Eventually we would like to merge elements of the array into a time series.
Does Cognite have array datatypes and can Cognite OPCUA extractor send arrays ?
Thank you
Zahra
The OPC UA extractor supports arrays, but it would typically write each array value into a discrete time series instance. We have no native way of storing arrays to enable you to do what you’re describing.
At first glance, whilst it’s an inventive solution, what you’re doing seems to be outside of what OPC UA arrays was intended for.
Have you tried collecting the data points using the normal client-server based approach already? I assume the Labview server is unable to cope with the load?
This seems like a strange workaround for high frequency data. In OPC-UA there is nothing preventing you from creating 500 notifications on a monitored item every second. If the client specified a sampling interval of 0 that would even be within reasonable behavior in the standard.
We generally do not support features that are not somehow rooted in the OPC-UA standard in the extractor, so this is not something we are likely to support.
All that said, the extractor _is_ actually able to ingest arrays. You specify the `extraction.data-types.max-array-size` property, and make sure that the `ArrayDimensions` attribute is set on the variable, and the extractor will create 500 separate time series.
Ok, thank you for your response, so then we know this approach is not supported in Cognite.
Regarding normal server-client OPCUA for sending 500Hz data, we have tried that and communicated with you and concluded that it is not working. We have used UaExpert and verified that the Labview server sends data at 500Hz but the Cognite extractor is not able to catch up and we end up at 20Hz data on CDF. It is wellknown that normal OPCUA server-client is not proper for high frequency data and PubSub method is recommended in OPCUA communities. Unfortunately Labview doesn’t support PubSub method, and I understood from your response that only choice for us is to change our OPCUA server…
P.S. Using arrays to send high frequency data is not invented in our team but an approach that is used in Labview community...
Have you attempted to implement Einar’s first suggestion? i.e. configuration of the sampling interval? Rather than collecting a sample every 2ms, collect a sample every 1s.
That may also be an issue due to queue length on the server. There’s a parameter `subscriptions.queue-length` that you should set high enough that it doesn’t overflow.
Yes, I have tried configuring sampling interval and `subscriptions.queue-length` but the issue persists. As mentioned the OPCUA community suggests PubSub method for high frequency data. So I think wee need to change our OPCUA server.
Sorry to hear that it comes to that, but that does sound like the best approach, to find an OPC UA server that supports pub-sub and which is optimised for these types of data.