REST Adapters
Overview
Two generic adapters are provided for common REST pagination patterns. They work with any API - you provide a parser function that knows your response schema, and the adapter handles pagination, retry, and exhaustion detection.
| Adapter | Pagination model | Exhaustion condition |
|---|---|---|
PageAdapter<T> | ?page=N&per_page=M | Response returns fewer than page_size records |
TokenAdapter<T> | ?cursor=TOKEN&limit=N | Token extractor returns nullopt |
Both adapters are templated on the record type T - you define what a "record" is.
The engine doesn't know or care about your data model.
PageAdapter
For APIs that paginate by page number. The adapter appends ?page=N&per_page=M
to the base URL and increments the page on each batch. When a response returns fewer
records than page_size, the stream is exhausted.
struct Item { int id; std::string name; };
// You provide the parser - the engine doesn't know your response schema
auto parser = [](const std::string& body) -> std::vector<Item> {
auto doc = nlohmann::json::parse(body);
std::vector<Item> out;
for (auto& el : doc["items"])
out.push_back({ el["id"], el["name"] });
return out;
};
PageAdapter<Item>::Config cfg;
cfg.base_url = "https://api.example.com/items";
cfg.page_size = 100;
cfg.auth_header = "Authorization";
cfg.auth_value = "Bearer your-token";
Cursor initial;
initial.page = 1;
auto engine = std::make_unique<ExecutionEngine<Item>>(
std::make_unique<PageAdapter<Item>>(cfg, parser),
std::make_unique<CurlTransport>(),
std::make_unique<FixedPolicy>(),
initial
);
while (auto batch = engine->next()) {
if (!batch->ok()) break;
for (auto& item : batch->records)
printf("#%d: %s\n", item.id, item.name.c_str());
}Configuration
| Field | Default | Description |
|---|---|---|
base_url | - | Base URL without query params |
page_size | 100 | Records per page |
page_param | "page" | Query parameter name for page number |
per_page_param | "per_page" | Query parameter name for page size |
auth_header | empty | Authentication header name |
auth_value | empty | Authentication header value |
extra_headers | empty | Additional headers to include |
TokenAdapter
For APIs that return an opaque cursor/token in the response body. You provide two
functions: a parser (body → records) and a token extractor (body → next token).
When the extractor returns nullopt, the stream is exhausted.
auto parser = [](const std::string& body) -> std::vector<Item> {
auto doc = nlohmann::json::parse(body);
std::vector<Item> out;
for (auto& el : doc["data"]) out.push_back({ el["id"], el["name"] });
return out;
};
auto token_extractor = [](const std::string& body) -> std::optional<std::string> {
auto doc = nlohmann::json::parse(body);
if (doc.contains("next_cursor") && !doc["next_cursor"].is_null())
return doc["next_cursor"].get<std::string>();
return std::nullopt; // stream exhausted
};
TokenAdapter<Item>::Config cfg;
cfg.base_url = "https://api.example.com/items";
cfg.page_size = 50;
cfg.cursor_param = "cursor"; // ?cursor=<token>
cfg.page_size_param = "limit"; // ?limit=50
Cursor initial; // no token on first request
auto engine = std::make_unique<ExecutionEngine<Item>>(
std::make_unique<TokenAdapter<Item>>(cfg, parser, token_extractor),
std::make_unique<CurlTransport>(),
std::make_unique<FixedPolicy>(),
initial
);The first request is sent without a cursor parameter. Subsequent requests include the token extracted from the previous response.
Writing Parsers
Parser functions receive the raw HTTP response body as a std::string and must
return a std::vector<T>. They should throw an exception on parse failure - the
adapter catches it and sets last_error().
// Simple JSON array parser
auto parser = [](const std::string& body) -> std::vector<MyRecord> {
auto doc = nlohmann::json::parse(body); // throws on invalid JSON
std::vector<MyRecord> out;
for (auto& el : doc["results"]) {
out.push_back({
el.at("id").get<int>(),
el.at("name").get<std::string>(),
el.value("score", 0.0)
});
}
return out;
};Tips:
- Use
el.at("field")instead ofel["field"]to get clear errors on missing fields - Use
el.value("field", default)for optional fields - Return an empty vector for empty pages - the adapter uses
page_sizecomparison for exhaustion
Error Handling
Both REST adapters retry on 429 and 5xx responses. They parse the standard
Retry-After header (lowercase) for server-mandated wait times. Client errors
(4xx except 429) and parse failures are propagated immediately.
The AdaptivePolicy does not apply to page/token-based adapters since there
is no time window to shrink. Use FixedPolicy for REST adapters.