Skip to main content
Active ⭐️⭐️⭐️
mqtttechuser27
Active ⭐️⭐️⭐️
Nov 9, 2026 10:52 AM UTC
Question

Hosted MQTT Extractor – MQTT Message Throughput Limits and Quotas

Hello Cognite Community,

We are evaluating the throughput limits of the Cognite Hosted MQTT Extractor.

Our use case involves subscribing to MQTT topics and storing the received messages in CDF RAW. Each MQTT message has a payload smaller than 1 KB.

We would like to clarify the following:

  1. Are there any hard limits on MQTT message throughput (messages/second)?

    • Per MQTT client connection

    • Per subscription (topic filter)

    • Per Hosted Extractor job

  2. If there are no fixed limits, what is the recommended maximum sustained throughput (messages/second)?

  3. What happens if the incoming message rate exceeds the Hosted MQTT Extractor's processing capacity?

    • Are messages buffered?

    • Are failed messages retried?

    • Is there a risk of message loss?

  4. Do the CDF RAW API write quotas also apply to Hosted MQTT Extractors, or are there separate internal limits?

We would appreciate any official documentation, benchmarks, or practical experience regarding these limitations.

Thank you!

    1 reply

    augtheo
    Practitioner ⭐️⭐️⭐️
    augtheo
    Practitioner ⭐️⭐️⭐️
    Nov 12, 2026 10:49 AM UTC

    Hi,

    There are no hard message-rate limits — not per connection, per subscription, or per job. Throughput is bounded by how fast the pipeline drains, rather than by a configured cap.

    Overload

    Each job subscribes at QoS 1 and processes messages sequentially, buffering rows in memory (approximately 10,000 per destination) and flushing to CDF roughly once a second in batches of up to 5,000 rows.

    When the buffer fills, the extractor stops reading from the socket, applying TCP backpressure to your broker. With QoS 1 and a persistent session, messages queue on the broker rather than being dropped. This means your broker's queue depth and expiry settings become the practical limits during a burst.

    Retries and loss

    Transient CDF errors, including rate limiting, are retried with exponential backoff (up to ten attempts). Transform failures are permanent and are not retried. If the dead letter queue is enabled for your project, those payloads are written to a RAW table.

    Loss is possible in a narrow window: buffered data that has not yet been written can be lost due to a permanent write failure or an ungraceful shutdown. Graceful restarts flush data first.

     

    Deferred Acknowledgement

     If you need stronger delivery guarantees, please contact support, we can enable deferred ack for your project. The extractor then withholds the MQTT acknowledgement until the CDF write is confirmed, and withholds it on a failed write so your broker redelivers on reconnect. 

    RAW quotas

    Writes go through the standard RAW API as your project, so the same project quotas apply. There is no separate internal allowance.

    Monitoring

    Per-job metrics are available at:

    /api/v1/projects/{project}/hostedextractors/jobs/{externalId}/metrics

    Comparing sourceMessages against cdfUploadedValues, along with cdfSkippedValues and cdfFailedValues, is the most direct way to determine whether a job is keeping up.

    Throughput expectations

    We don't publish throughput benchmarks, since achievable rates depend on payload shape, transform complexity, and RAW write patterns.

    For sub-1 KB payloads written to RAW, a single job sustaining a few thousand messages per second is a reasonable expectation. However, validate this with your own data.

    For higher throughput, split the workload across jobs with disjoint topic filters.

    Terms & ConditionsAccessibility statement