Advanced
Chunk Size Tuning
The default chunk_seconds=14400 (4 hours) works well for 1-minute granularity.
For other granularities, target 100–5000 points per request to balance HTTP
overhead against API response size limits.
| Granularity | Points/chunk at default | Recommended chunk |
|---|---|---|
| 10 s | 1440 | 3600 (1 h) |
| 60 s | 240 | 14400 (4 h) - default |
| 300 s | 48 | 86400 (24 h) |
| 3600 s | 4 | 604800 (7 days) |
If you see HTTP 400 or 503 errors, your chunk is too large. Halve
chunk_seconds until the errors stop. If batches arrive very quickly and
contain few points, your chunk is too small - increase it to reduce per-request
overhead (TLS handshake, DNS lookup, JSON parsing).
# High-resolution 10-second data: smaller chunks cfg.granularity = 10 cfg.chunk_seconds = 3600 # 360 points per chunk # Low-resolution hourly data: large chunks are fine cfg.granularity = 3600 cfg.chunk_seconds = 604800 # 7 days = 168 points
Retry Strategy
dd_stream retries transparently. You control the budget with max_retries:
| HTTP Status | Behaviour |
|---|---|
| 429 Too Many Requests | Sleep until x-ratelimit-reset, then retry |
| 500–599 Server Error | Exponential backoff (1 s, 2 s, 4 s…), up to max_retries |
| 400–499 Client Error | Propagate immediately - no retry |
| Transport / DNS error | Retry up to max_retries with 1 s delay |
cfg.max_retries = 5 # default: 3 cfg.request_timeout_ms = 60000 # increase for slow networks (default: 30 000) cfg.connect_timeout_ms = 10000 # TCP connect (default: 5 000)
Rate Limiting
Datadog's API returns rate-limit headers on every response. dd_stream reads
x-ratelimit-remaining and x-ratelimit-reset and automatically sleeps before
the next request if remaining == 0. You can observe the remaining budget via
BatchStats:
for batch in stream:
stats = stream.last_batch_stats()
print(f"Rate limit remaining: {stats.rate_limit_remaining}")
# If this reaches 0, the next batch will pause automaticallyDebug Logging
Set log_level to enable structured log output to stderr. This is useful for
diagnosing unexpected pauses, retry storms, or chunking behaviour.
cfg.log_level = 4 # DEBUG: prints every HTTP request, retry, and prefetch event # Log levels: # 0 = NONE (default - silent) # 1 = ERROR (only failures) # 2 = WARN # 3 = INFO (chunk boundaries, reset, exhaust) # 4 = DEBUG (every request, retry, prefetch)
Example DEBUG output:
[ddstream][2026-04-02T09:00:00Z][INFO ] [BatchManager] reset: range=[1743502800, 1743589200] chunk=14400s [ddstream][2026-04-02T09:00:00Z][INFO ] [HttpClient] POST https://api.datadoghq.com/... → 200 (0 retries) [ddstream][2026-04-02T09:00:00Z][DEBUG] [BatchManager] batch #1: 240 times chunk=[...] is_last=0 [ddstream][2026-04-02T09:00:00Z][DEBUG] [BatchManager] kick_prefetch next_start=1743517200
CLI Tool
Build the ddstream_cli binary for quick command-line metric streaming without
writing code. Useful for validating API keys, testing queries, and inspecting
batch statistics.
cmake -B build -DDDSTREAM_BUILD_CLI=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)
# Stream last 1 hour of CPU at 1-min resolution
./build/ddstream_cli \
--api-key "$DD_API_KEY" \
--app-key "$DD_APP_KEY" \
--query "avg:system.cpu.user{*}" \
--start $(date -d '1 hour ago' +%s) \
--end $(date +%s) \
--gran 60
# Output (one line per batch):
# batch=1 points=240 chunk=[1743588000,1743589200] progress=100% retries=0