Skip to main content
Solved

How transform compressed data coming from kafka topics


Hi All,

I am working with hosted extractors for kafka and it works pretty well for me with transformations when we have plain json data in kafka topics.

Now I am trying to check if we can work with zlib gzip compressed data coming in topic, I have json string and messagepayload attribute of json string will be holding compressed data instead of whole message as compressed one, is it possible to write transformation for such data.

 

e.g., in kafka topics

{
    "Header": {
        "MessageId": 133367162,
        "MessageType": "DATA_REPORT",
        "Timestamp": 1741122422,
        "PayloadCompression": "Z_LIB_COMPRESSION"        
    },
    "MessagePayload": "eJyqVnJJLctMTi1WsoquVvJLzE1VslIyVNJRckksSQxJTIeIhySmQ6WCA3wVfFMTi0uLUlNgqioLQDIu/qFOPq7xYY4+oa5KOkphiTmlMLNCMnNTi0sScwuUrAzNTQwNjYxMjAwtzA11lAJLE3MySyqVrAxqY2tjawEBAAD//x5aKt0="
}

Best answer by Thomas Sjølshagen

This looks like a Base64 encoded “MessagePayload”, which I guess is compressed before it’s being Base64 encoded?

This is not something we support in the extractor.

 

We do support a fully compressed message (the decompression is configurable in the setup for the extractor). But we do not have any methods for parsing the message and then unencoding and uncompressing portions of the message “on the fly”, sorry.

What is the use case that results in this formatting?

View original
Did this topic help you find an answer to your question?

Forum|alt.badge.img

This looks like a Base64 encoded “MessagePayload”, which I guess is compressed before it’s being Base64 encoded?

This is not something we support in the extractor.

 

We do support a fully compressed message (the decompression is configurable in the setup for the extractor). But we do not have any methods for parsing the message and then unencoding and uncompressing portions of the message “on the fly”, sorry.

What is the use case that results in this formatting?


  • Active
  • March 18, 2025

Ok this answers my query right now, if I need specific changes I would try to raise it as requirement


Reply


Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings