Hi,
There are no hard message-rate limits — not per connection, per subscription, or per job. Throughput is bounded by how fast the pipeline drains, rather than by a configured cap.
Overload
Each job subscribes at QoS 1 and processes messages sequentially, buffering rows in memory (approximately 10,000 per destination) and flushing to CDF roughly once a second in batches of up to 5,000 rows.
When the buffer fills, the extractor stops reading from the socket, applying TCP backpressure to your broker. With QoS 1 and a persistent session, messages queue on the broker rather than being dropped. This means your broker's queue depth and expiry settings become the practical limits during a burst.
Retries and loss
Transient CDF errors, including rate limiting, are retried with exponential backoff (up to ten attempts). Transform failures are permanent and are not retried. If the dead letter queue is enabled for your project, those payloads are written to a RAW table.
Loss is possible in a narrow window: buffered data that has not yet been written can be lost due to a permanent write failure or an ungraceful shutdown. Graceful restarts flush data first.
Deferred Acknowledgement
If you need stronger delivery guarantees, please contact support, we can enable deferred ack for your project. The extractor then withholds the MQTT acknowledgement until the CDF write is confirmed, and withholds it on a failed write so your broker redelivers on reconnect.
RAW quotas
Writes go through the standard RAW API as your project, so the same project quotas apply. There is no separate internal allowance.
Monitoring
Per-job metrics are available at:
/api/v1/projects/{project}/hostedextractors/jobs/{externalId}/metrics
Comparing sourceMessages against cdfUploadedValues, along with cdfSkippedValues and cdfFailedValues, is the most direct way to determine whether a job is keeping up.
Throughput expectations
We don't publish throughput benchmarks, since achievable rates depend on payload shape, transform complexity, and RAW write patterns.
For sub-1 KB payloads written to RAW, a single job sustaining a few thousand messages per second is a reasonable expectation. However, validate this with your own data.
For higher throughput, split the workload across jobs with disjoint topic filters.