API Tools — streaming infrastructure for reliable API and AI data

Prerequisites

Linux (Ubuntu / Debian)

bash

sudo apt-get update
sudo apt-get install -y build-essential cmake libcurl4-openssl-dev nlohmann-json3-dev

Linux (Fedora / RHEL)

bash

sudo dnf install -y gcc-c++ cmake libcurl-devel nlohmann-json-devel

macOS

bash

brew install cmake curl nlohmann-json

Required versions

Dependency	Minimum version
CMake	3.16
C++ compiler	C++17 (GCC 9+, Clang 10+, MSVC 19.14+)
libcurl	7.68+
nlohmann/json	3.11+

Build from source

All commands run from the codebase/ directory.

bash

cmake -S source -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build

Build outputs land in build/:

Target	Type	Description
`libapiexec.so`	Shared library	For language bindings (Go, Rust, Python, Java, JS)
`libapiexec_capi.a`	Static library	C API for static linking
`libapiexec_adapters.a`	Static library	All vendor adapters
`libapiexec_transport.a`	Static library	CurlTransport
`libapiexec_policy.a`	Static library	DefaultPolicy + CostAwarePolicy

Install

Step 1: Build

bash

cmake -S source -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build

Step 2: Install

Choose your install location:

System-wide (/usr/local — requires sudo, no environment setup needed):

bash

sudo cmake --install build

User-local (~/.local — no sudo, requires environment setup):

bash

cmake --install build --prefix $HOME/.local

Custom prefix:

bash

cmake --install build --prefix /opt/apiexec

Step 3: Set up environment (skip for system-wide installs)

If you installed to ~/.local or a custom prefix, add these to your ~/.bashrc or ~/.zshrc:

bash

# For ~/.local installs:
export LD_LIBRARY_PATH="$HOME/.local/lib:$LD_LIBRARY_PATH"
export PKG_CONFIG_PATH="$HOME/.local/lib/pkgconfig:$PKG_CONFIG_PATH"
export CMAKE_PREFIX_PATH="$HOME/.local:$CMAKE_PREFIX_PATH"

# For custom prefix installs (replace /opt/apiexec with your prefix):
export LD_LIBRARY_PATH="/opt/apiexec/lib:$LD_LIBRARY_PATH"
export PKG_CONFIG_PATH="/opt/apiexec/lib/pkgconfig:$PKG_CONFIG_PATH"
export CMAKE_PREFIX_PATH="/opt/apiexec:$CMAKE_PREFIX_PATH"

Then reload: source ~/.bashrc

Why are these needed?

/usr/local is in the compiler and linker's default search path, so system-wide installs work out of the box. Other prefixes are not — these exports tell the linker (LD_LIBRARY_PATH), pkg-config (PKG_CONFIG_PATH), and CMake (CMAKE_PREFIX_PATH) where to find the library.

Step 4: Verify

bash

# Check pkg-config finds the library
pkg-config --libs --cflags apiexec
# Expected: -I<prefix>/include/apiexec -L<prefix>/lib -lapiexec -lcurl

# Compile and run a test program
cat > /tmp/test_apiexec.c << 'EOF'
#include "c_api.h"
#include <stdio.h>
int main() {
    printf("ABI version: %d\n", APIEXEC_ABI_VERSION);
    StreamHandle* h = stream_create("generic_rest",
        "{\"base_url\":\"http://localhost\"}", NULL);
    printf("stream_create: %s\n", h ? "OK" : "OK (no server)");
    if (h) stream_destroy(h);
    return 0;
}
EOF

gcc /tmp/test_apiexec.c $(pkg-config --cflags --libs apiexec) -o /tmp/test_apiexec
/tmp/test_apiexec
# Expected:
#   ABI version: 1
#   stream_create: OK

If pkg-config reports Package apiexec was not found, your PKG_CONFIG_PATH is not set — go back to Step 3.

Uninstall

bash

# System-wide
sudo xargs rm < build/install_manifest.txt

# User-local or custom prefix
xargs rm < build/install_manifest.txt

Link the library

Option 1: pkg-config (recommended)

Handles -I and -L flags automatically regardless of install prefix.

bash

# C
gcc my_app.c $(pkg-config --cflags --libs apiexec) -o my_app

# C++
g++ -std=c++17 my_app.cpp $(pkg-config --cflags --libs apiexec) -o my_app

# In a Makefile
CFLAGS  += $(shell pkg-config --cflags apiexec)
LDFLAGS += $(shell pkg-config --libs apiexec)

This works identically for /usr/local, ~/.local, and custom prefix installs — as long as PKG_CONFIG_PATH is set (see Installation Step 3).

Option 2: CMake find_package

cmake

find_package(apiexec REQUIRED)
target_link_libraries(my_app PRIVATE apiexec::apiexec)

bash

# System-wide install — found automatically
cmake -S . -B build

# ~/.local install — tell CMake where to look
cmake -S . -B build -DCMAKE_PREFIX_PATH=$HOME/.local

Option 3: CMake subdirectory (no install needed)

cmake

add_subdirectory(path/to/apiexec/source)
target_link_libraries(my_app PRIVATE apiexec_all)

Option 4: Direct compiler flags

Use this when pkg-config or CMake are not available. Replace <prefix> with your install prefix (/usr/local, $HOME/.local, etc.).

Shared library (preferred — includes all adapters, no --whole-archive needed):

bash

g++ -std=c++17 my_app.cpp \
    -I<prefix>/include/apiexec \
    -L<prefix>/lib \
    -lapiexec -o my_app

Static libraries:

bash

g++ -std=c++17 my_app.cpp \
    -I<prefix>/include/apiexec \
    -L<prefix>/lib \
    -lapiexec_capi \
    -Wl,--whole-archive -lapiexec_adapters -Wl,--no-whole-archive \
    -lapiexec_transport -lapiexec_policy \
    -lcurl -lstdc++ -lpthread

Common mistakes with direct flags

Always include both -I<prefix>/include/apiexec and -L<prefix>/lib. Without -I, the compiler can't find c_api.h; without -L, the linker can't find libapiexec. Use pkg-config (Option 1) to avoid this entirely.

For static builds, --whole-archive on apiexec_adapters is required — it preserves adapter self-registration initialisers that the linker would otherwise strip, causing stream_create to return null for every adapter name.

Write your first stream (C)

The minimal lifecycle is four calls: stream_create → stream_has_next → stream_next_batch_v1 → stream_destroy.

#include "c_api.h"
#include <stdio.h>
#include <stdlib.h>

int main(void) {
    const char* config =
        "{\"base_url\": \"https://api.example.com/v1/data\"}";
    const char* policy =
        "{\"max_retries\": 3, \"prefetch_depth\": 1}";

    StreamHandle* stream = stream_create("generic_rest", config, policy);
    if (!stream) {
        fprintf(stderr, "stream_create failed - check config JSON\n");
        return EXIT_FAILURE;
    }

    char buf[1048576];  /* 1 MiB - tune to your expected batch size */
    int32_t count = 0;

    while (stream_has_next(stream) == 1) {
        int32_t rc = stream_next_batch_v1(stream, buf, sizeof buf, &count);
        if (rc == STREAM_EXHAUSTED) break;
        if (rc != STREAM_OK) {
            fprintf(stderr, "stream error: %d\n", rc);
            break;
        }
        printf("received %d records: %s\n", count, buf);
    }

    stream_destroy(stream);
    return EXIT_SUCCESS;
}

Write your first stream (language bindings)

Every binding wraps the same C ABI. Set up the binding for your language:

Go - cd bindings/go && CGO_ENABLED=1 go test ./apiexec/...
Rust - cd bindings/rust && LD_LIBRARY_PATH=../../build cargo test
Python - cd bindings/python && LD_LIBRARY_PATH=../../build python3 -m pytest -v
Java - requires JNA on the classpath; see bindings/java/README.md
JavaScript - requires ffi-napi; see bindings/js/README.md

main.py

from apiexec import Stream

with Stream("generic_rest",
            '{"base_url": "https://api.example.com/v1/data"}') as s:
    for batch_json, count in s:
        print(f"Got {count} records")

Handle errors

stream_next_batch_v1 returns an int32_t error code. Check it on every call.

int32_t rc = stream_next_batch_v1(stream, buf, sizeof buf, &count);

switch (rc) {
    case STREAM_OK:
        /* process buf */
        break;
    case STREAM_EXHAUSTED:
        /* no more data - normal termination */
        break;
    case STREAM_ERROR_RATE_LIMIT:
        fprintf(stderr, "rate limited after all retries\n");
        break;
    case STREAM_ERROR_CLIENT:
        /* bad config or buf_len too small */
        fprintf(stderr, "client error (buf too small?)\n");
        break;
    default:
        fprintf(stderr, "unexpected error: %d\n", rc);
        break;
}

See the Error Code Table for the full list.

Cancel from another thread

stream_cancel() is the only thread-safe call on a handle. Use it from a signal handler or a watchdog thread; subsequent stream_next_* calls return STREAM_ERROR_CANCELLED.

#include <pthread.h>

static StreamHandle* g_stream = NULL;

void* watchdog(void* arg) {
    (void)arg;
    sleep(30);
    stream_cancel(g_stream);
    return NULL;
}

int main(void) {
    const char* config =
        "{\"base_url\": \"https://api.example.com/v1/data\"}";
    g_stream = stream_create("generic_rest", config, NULL);

    pthread_t tid;
    pthread_create(&tid, NULL, watchdog, NULL);

    char buf[1048576];
    int32_t count = 0;
    while (stream_has_next(g_stream) == 1) {
        int32_t rc = stream_next_batch_v1(g_stream, buf, sizeof buf, &count);
        if (rc == STREAM_ERROR_CANCELLED) break;
        if (rc != STREAM_OK) break;
        /* process buf */
    }

    stream_destroy(g_stream);
    pthread_join(tid, NULL);
    return EXIT_SUCCESS;
}

Next steps

Concepts - how the engine works and why it is designed that way
Architecture - module layers, ABI boundary, adapter interface
Adapters - config keys for generic_rest, datadog_metrics, openai, anthropic
Stream API - full function table and null-safety contract
Metrics - runtime observability and Prometheus export
Examples - real-world patterns in all languages

Prerequisites

Linux (Ubuntu / Debian)

bash

sudo apt-get update
sudo apt-get install -y build-essential cmake libcurl4-openssl-dev nlohmann-json3-dev

Linux (Fedora / RHEL)

bash

sudo dnf install -y gcc-c++ cmake libcurl-devel nlohmann-json-devel

macOS

bash

brew install cmake curl nlohmann-json

Required versions

Dependency	Minimum version
CMake	3.16
C++ compiler	C++17 (GCC 9+, Clang 10+, MSVC 19.14+)
libcurl	7.68+
nlohmann/json	3.11+

Build from source

All commands run from the codebase/ directory.

bash

cmake -S source -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build

Build outputs land in build/:

Target	Type	Description
`libapiexec.so`	Shared library	For language bindings (Go, Rust, Python, Java, JS)
`libapiexec_capi.a`	Static library	C API for static linking
`libapiexec_adapters.a`	Static library	All vendor adapters
`libapiexec_transport.a`	Static library	CurlTransport
`libapiexec_policy.a`	Static library	DefaultPolicy + CostAwarePolicy

Install

Step 1: Build

bash

cmake -S source -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build

Step 2: Install

Choose your install location:

System-wide (/usr/local — requires sudo, no environment setup needed):

bash

sudo cmake --install build

User-local (~/.local — no sudo, requires environment setup):

bash

cmake --install build --prefix $HOME/.local

Custom prefix:

bash

cmake --install build --prefix /opt/apiexec

Step 3: Set up environment (skip for system-wide installs)

If you installed to ~/.local or a custom prefix, add these to your ~/.bashrc or ~/.zshrc:

bash

# For ~/.local installs:
export LD_LIBRARY_PATH="$HOME/.local/lib:$LD_LIBRARY_PATH"
export PKG_CONFIG_PATH="$HOME/.local/lib/pkgconfig:$PKG_CONFIG_PATH"
export CMAKE_PREFIX_PATH="$HOME/.local:$CMAKE_PREFIX_PATH"

# For custom prefix installs (replace /opt/apiexec with your prefix):
export LD_LIBRARY_PATH="/opt/apiexec/lib:$LD_LIBRARY_PATH"
export PKG_CONFIG_PATH="/opt/apiexec/lib/pkgconfig:$PKG_CONFIG_PATH"
export CMAKE_PREFIX_PATH="/opt/apiexec:$CMAKE_PREFIX_PATH"

Then reload: source ~/.bashrc

Why are these needed?

Step 4: Verify

bash

# Check pkg-config finds the library
pkg-config --libs --cflags apiexec
# Expected: -I<prefix>/include/apiexec -L<prefix>/lib -lapiexec -lcurl

# Compile and run a test program
cat > /tmp/test_apiexec.c << 'EOF'
#include "c_api.h"
#include <stdio.h>
int main() {
    printf("ABI version: %d\n", APIEXEC_ABI_VERSION);
    StreamHandle* h = stream_create("generic_rest",
        "{\"base_url\":\"http://localhost\"}", NULL);
    printf("stream_create: %s\n", h ? "OK" : "OK (no server)");
    if (h) stream_destroy(h);
    return 0;
}
EOF

gcc /tmp/test_apiexec.c $(pkg-config --cflags --libs apiexec) -o /tmp/test_apiexec
/tmp/test_apiexec
# Expected:
#   ABI version: 1
#   stream_create: OK

If pkg-config reports Package apiexec was not found, your PKG_CONFIG_PATH is not set — go back to Step 3.

Uninstall

bash

# System-wide
sudo xargs rm < build/install_manifest.txt

# User-local or custom prefix
xargs rm < build/install_manifest.txt

Link the library

Option 1: pkg-config (recommended)

Handles -I and -L flags automatically regardless of install prefix.

bash

# C
gcc my_app.c $(pkg-config --cflags --libs apiexec) -o my_app

# C++
g++ -std=c++17 my_app.cpp $(pkg-config --cflags --libs apiexec) -o my_app

# In a Makefile
CFLAGS  += $(shell pkg-config --cflags apiexec)
LDFLAGS += $(shell pkg-config --libs apiexec)

This works identically for /usr/local, ~/.local, and custom prefix installs — as long as PKG_CONFIG_PATH is set (see Installation Step 3).

Option 2: CMake find_package

cmake

find_package(apiexec REQUIRED)
target_link_libraries(my_app PRIVATE apiexec::apiexec)

bash

# System-wide install — found automatically
cmake -S . -B build

# ~/.local install — tell CMake where to look
cmake -S . -B build -DCMAKE_PREFIX_PATH=$HOME/.local

Option 3: CMake subdirectory (no install needed)

cmake

add_subdirectory(path/to/apiexec/source)
target_link_libraries(my_app PRIVATE apiexec_all)

Option 4: Direct compiler flags

Use this when pkg-config or CMake are not available. Replace <prefix> with your install prefix (/usr/local, $HOME/.local, etc.).

Shared library (preferred — includes all adapters, no --whole-archive needed):

bash

g++ -std=c++17 my_app.cpp \
    -I<prefix>/include/apiexec \
    -L<prefix>/lib \
    -lapiexec -o my_app

Static libraries:

bash

g++ -std=c++17 my_app.cpp \
    -I<prefix>/include/apiexec \
    -L<prefix>/lib \
    -lapiexec_capi \
    -Wl,--whole-archive -lapiexec_adapters -Wl,--no-whole-archive \
    -lapiexec_transport -lapiexec_policy \
    -lcurl -lstdc++ -lpthread

Common mistakes with direct flags

Write your first stream (C)

The minimal lifecycle is four calls: stream_create → stream_has_next → stream_next_batch_v1 → stream_destroy.

#include "c_api.h"
#include <stdio.h>
#include <stdlib.h>

int main(void) {
    const char* config =
        "{\"base_url\": \"https://api.example.com/v1/data\"}";
    const char* policy =
        "{\"max_retries\": 3, \"prefetch_depth\": 1}";

    StreamHandle* stream = stream_create("generic_rest", config, policy);
    if (!stream) {
        fprintf(stderr, "stream_create failed - check config JSON\n");
        return EXIT_FAILURE;
    }

    char buf[1048576];  /* 1 MiB - tune to your expected batch size */
    int32_t count = 0;

    while (stream_has_next(stream) == 1) {
        int32_t rc = stream_next_batch_v1(stream, buf, sizeof buf, &count);
        if (rc == STREAM_EXHAUSTED) break;
        if (rc != STREAM_OK) {
            fprintf(stderr, "stream error: %d\n", rc);
            break;
        }
        printf("received %d records: %s\n", count, buf);
    }

    stream_destroy(stream);
    return EXIT_SUCCESS;
}

Write your first stream (language bindings)

Every binding wraps the same C ABI. Set up the binding for your language:

Go - cd bindings/go && CGO_ENABLED=1 go test ./apiexec/...
Rust - cd bindings/rust && LD_LIBRARY_PATH=../../build cargo test
Python - cd bindings/python && LD_LIBRARY_PATH=../../build python3 -m pytest -v
Java - requires JNA on the classpath; see bindings/java/README.md
JavaScript - requires ffi-napi; see bindings/js/README.md

main.py

from apiexec import Stream

with Stream("generic_rest",
            '{"base_url": "https://api.example.com/v1/data"}') as s:
    for batch_json, count in s:
        print(f"Got {count} records")

Handle errors

stream_next_batch_v1 returns an int32_t error code. Check it on every call.

int32_t rc = stream_next_batch_v1(stream, buf, sizeof buf, &count);

switch (rc) {
    case STREAM_OK:
        /* process buf */
        break;
    case STREAM_EXHAUSTED:
        /* no more data - normal termination */
        break;
    case STREAM_ERROR_RATE_LIMIT:
        fprintf(stderr, "rate limited after all retries\n");
        break;
    case STREAM_ERROR_CLIENT:
        /* bad config or buf_len too small */
        fprintf(stderr, "client error (buf too small?)\n");
        break;
    default:
        fprintf(stderr, "unexpected error: %d\n", rc);
        break;
}

See the Error Code Table for the full list.

Cancel from another thread

stream_cancel() is the only thread-safe call on a handle. Use it from a signal handler or a watchdog thread; subsequent stream_next_* calls return STREAM_ERROR_CANCELLED.

#include <pthread.h>

static StreamHandle* g_stream = NULL;

void* watchdog(void* arg) {
    (void)arg;
    sleep(30);
    stream_cancel(g_stream);
    return NULL;
}

int main(void) {
    const char* config =
        "{\"base_url\": \"https://api.example.com/v1/data\"}";
    g_stream = stream_create("generic_rest", config, NULL);

    pthread_t tid;
    pthread_create(&tid, NULL, watchdog, NULL);

    char buf[1048576];
    int32_t count = 0;
    while (stream_has_next(g_stream) == 1) {
        int32_t rc = stream_next_batch_v1(g_stream, buf, sizeof buf, &count);
        if (rc == STREAM_ERROR_CANCELLED) break;
        if (rc != STREAM_OK) break;
        /* process buf */
    }

    stream_destroy(g_stream);
    pthread_join(tid, NULL);
    return EXIT_SUCCESS;
}

Next steps

Concepts - how the engine works and why it is designed that way
Architecture - module layers, ABI boundary, adapter interface
Adapters - config keys for generic_rest, datadog_metrics, openai, anthropic
Stream API - full function table and null-safety contract
Metrics - runtime observability and Prometheus export
Examples - real-world patterns in all languages

Quick Start

Prerequisites

Linux (Ubuntu / Debian)

Linux (Fedora / RHEL)

macOS

Required versions

Build from source

Install

Step 1: Build

Step 2: Install

Step 3: Set up environment (skip for system-wide installs)

Step 4: Verify

Uninstall

Link the library

Option 1: pkg-config (recommended)

Option 2: CMake find_package

Option 3: CMake subdirectory (no install needed)

Option 4: Direct compiler flags

Write your first stream (C)

Write your first stream (language bindings)

Handle errors

Cancel from another thread

Next steps

Quick Start

Prerequisites

Linux (Ubuntu / Debian)

Linux (Fedora / RHEL)

macOS

Required versions

Build from source

Install

Step 1: Build

Step 2: Install

Step 3: Set up environment (skip for system-wide installs)

Step 4: Verify

Uninstall

Link the library

Option 1: pkg-config (recommended)

Option 2: CMake find_package

Option 3: CMake subdirectory (no install needed)

Option 4: Direct compiler flags

Write your first stream (C)

Write your first stream (language bindings)

Handle errors

Cancel from another thread

Next steps