Skip to main content
Seasoned ⭐️⭐️⭐️
October 1, 2021
Solved

How to overcome metadata key length and max keys limitation

  • October 1, 2021
  • 11 replies
  • 331 views

On serveral occations we have encountered limitations in metadata key length, most frequently when flattening JSON-formatted strings from our event stream data source systems. When the source system presents nested structure of as many as 4 levels we frequently encounter metadata keys that require more than 128 bytes. 

Up to now we have “solved” the issue by abbreviating the metadatakeys at the price of higher maintenance cost of the code and more importantly, that end-users get the perception that we have transformed the data or even dont understand what it represents. We now consider moving towards a solution where we simply put the entire JSON-formatted string into one single metadata value field, and leave to front-end teams and end users to flatten the structure. 

We have a similar issue with max number of metadata keys for timesseries (16). 

Question 1) Could you please suggest other, better options for handling these metadata limitations? 
Question 2) Will templates come to the rescue?  
Question 3) Any plans for increasing the limits on metadata key length or number of metadata keys on Timeseries? 

Best answer by RobersoSN

Thanks, Thomas. That is a useful clarification regarding the metadata keys/values. And, 256 bytes for keys might be a bandaid for now at least.

For now we seem to be very eagerly awaiting the deployment of schema services to the SN cluster.

 

Thanks for the heads up regarding filtering, and access management in raw.

11 replies

Lead Product Manager
September 30, 2022

Hi @Andreas Kimsås,

With the work going into Flexible Data Modeling, we've not been able to prioritize extending the metadata key length. 

So it's not in the plans for the next couple of months, sorry.