Low-Level Design and Data Schema Modeling of API Clients like Requestly

An API client like Requestly goes beyond a simple curl wrapper, offering a rich set of features for developers to interact with APIs efficiently. This document delves into the low-level design and data schema modeling required to build such a sophisticated tool from scratch, exploring key features like variable interpolation, request chaining, collection runners, and test assertion frameworks.

Key Features of a Full-Fledged API Client

Building a comprehensive API client necessitates several core features:

1. Scripting Runtime: For pre-request and post-response scripts.
2. Scoped Variables: Managing variables with different scopes and precedence.
3. Test Assertion Framework: To validate API responses.
4. Collection Runner: Executing multiple requests in a defined sequence.
5. Migration Layer: For importing/exporting data from/to other tools (e.g., cURL, Postman).

Variable Interpolation

Variable interpolation is a fundamental feature allowing dynamic values in requests, such as base URLs that change across environments (e.g., dev.example.com, staging.example.com). Instead of duplicating collections, variables are defined and referenced using a syntax like {{variable_name}}.

Implementation Considerations

Implementing variable interpolation involves three critical aspects:

1. Scoping: Defining where a variable is accessible.
   • Global: Accessible across all collections (e.g., request_timeout, default_user_id).
   • Environment: Specific to an environment (e.g., base_url for dev, staging, production).
   • Collection: Specific to a particular collection (e.g., default_payment_amount for a payments collection).
   • Runtime: Session-scoped overrides, not synced.
2. Precedence: Determining which variable value takes priority if defined in multiple scopes.
   • Runtime (highest) > Collection > Environment > Global (lowest).
3. Resolution Timing: When variables are resolved (e.g., dynamic variables resolved at invocation).

Data Model for Variables

To store variable configurations, a database schema might include tables like:

-- Variables Table
CREATE TABLE variables (
    id TEXT PRIMARY KEY,
    workspace_id TEXT, -- For multi-tenancy/workspace support
    environment_id TEXT, -- NULL for Global, specific ID for Environment scope
    collection_id TEXT, -- NULL for Global/Environment, specific ID for Collection scope
    scope TEXT NOT NULL, -- 'global', 'environment', 'collection'
    key TEXT NOT NULL,
    initial_value TEXT, -- Default value, synced across workspace
    is_secret BOOLEAN DEFAULT FALSE
);

-- Environments Table
CREATE TABLE environments (
    id TEXT PRIMARY KEY,
    workspace_id TEXT,
    name TEXT NOT NULL
);

-- Collections Table
CREATE TABLE collections (
    id TEXT PRIMARY KEY,
    workspace_id TEXT,
    parent_id TEXT, -- For nested collections (tree view)
    name TEXT NOT NULL
);

-- Runtime Variables (local to machine, not synced)
-- This would typically be managed in-memory or in local storage, not a synced database table.
-- For a local SQLite, it might be a table:
CREATE TABLE runtime_variables (
    id TEXT PRIMARY KEY,
    session_id TEXT, -- Or user_id, machine_id
    key TEXT NOT NULL,
    current_value TEXT
);

Initial Value vs. Current Value

• Initial Value: The default value, synced across the workspace (e.g., cloud).
• Current Value: A local override, not synced, specific to the user’s machine/session. This distinction prevents accidental syncing of sensitive or temporary local changes.

Dynamic Variables

These are special variables (e.g., $random_uuid, $timestamp) that are resolved freshly at the time of invocation, providing unique or time-sensitive values.

Request Chaining

Request chaining enables using parts of one request’s response in subsequent requests. A classic example is extracting a JWT token from an authentication response and using it in subsequent API calls.

Mechanism

API clients typically expose a global object (e.g., RQ in Requestly) within the scripting environment. This RQ object contains:

• RQ.request: The current request object.
• RQ.response: The current response object.
• RQ.environment: A key-value store where scripts can temporarily store data (e.g., RQ.environment.set('jwt_token', response.json().token)). This data is then accessible to subsequent requests within the same execution context.
• RQ.globals: Global variables.

Scripting Hooks

Scripts are executed at specific points during a request’s lifecycle:

• Pre-request scripts: Run before the request is sent, useful for setting headers, body, or variables.
• Post-response scripts: Run after the response is received, useful for parsing responses, setting environment variables, or running assertions. These scripts are typically written in JavaScript and have access to the RQ object.

Collection Runner

A collection runner allows executing all requests within a collection in a sequential order. This is crucial for workflows that involve multiple dependent API calls.

Execution Flow

When a collection is run:

1. Each request in the collection is executed sequentially.
2. For each request, its pre-request and post-response scripts are executed.
3. Variable resolution occurs dynamically for each request.
4. Test assertions (discussed next) are run against the response.
5. Data stored in RQ.environment by one request’s post-response script is available to subsequent requests.

Data Model for Collection Runs

To track and store the results of collection runs, the following schema could be used:

-- Collections Table (revisited)
CREATE TABLE collections (
    id TEXT PRIMARY KEY,
    workspace_id TEXT,
    parent_id TEXT,
    name TEXT NOT NULL
);

-- Collection Runs Table
CREATE TABLE collection_runs (
    id TEXT PRIMARY KEY,
    collection_id TEXT NOT NULL,
    environment_id TEXT, -- Environment used for this run
    iteration_count INTEGER,
    status TEXT, -- 'passed', 'failed', 'in_progress'
    start_time DATETIME,
    end_time DATETIME
);

-- Requests Table (defines individual requests within a collection)
CREATE TABLE requests (
    id TEXT PRIMARY KEY,
    collection_id TEXT NOT NULL,
    name TEXT NOT NULL,
    method TEXT NOT NULL, -- e.g., 'GET', 'POST'
    url_template TEXT NOT NULL, -- URL with variable placeholders
    headers_json TEXT, -- JSON string of headers
    query_params_json TEXT, -- JSON string of query parameters
    body_type TEXT, -- e.g., 'raw', 'form-data', 'x-www-form-urlencoded'
    body_content TEXT, -- The actual body content (JSON, XML, etc.)
    auth_type TEXT, -- e.g., 'bearer', 'basic', 'oauth'
    auth_config_json TEXT -- JSON string of authentication configuration
);

-- Run Results Table (stores results for each request within a collection run)
CREATE TABLE run_results (
    id TEXT PRIMARY KEY,
    collection_run_id TEXT NOT NULL,
    request_id TEXT NOT NULL,
    iteration_index INTEGER,
    status TEXT, -- 'passed', 'failed', 'skipped' for individual request
    response_time_ms INTEGER,
    response_body TEXT,
    response_headers TEXT,
    test_results_json TEXT -- JSON string of test assertion results
);

-- Scripts Table (stores pre-request and post-response scripts)
CREATE TABLE scripts (
    id TEXT PRIMARY KEY,
    request_id TEXT NOT NULL,
    phase TEXT NOT NULL, -- 'pre-request', 'post-response'
    source_code TEXT NOT NULL, -- The actual JavaScript code
    timeout_ms INTEGER DEFAULT 5000 -- Script execution timeout
);

Test Assertion Framework

A test assertion framework allows users to define tests that validate API responses. This is crucial for ensuring API contracts are maintained and catching breaking changes early.

Concept

Tests are grouped into named test cases. For example, a test case “User should have required fields” might include assertions like:

• expect(response.body.email).to.match(/regex/);
• expect(response.body.roles).to.include('admin');
• expect(response.body).to.have.property('id'); Another test case “Status should be 201” would assert expect(response.status).to.equal(201);.

Implementation

API clients often wrap existing popular JavaScript testing frameworks (e.g., Chai.js, Jest) to provide a familiar syntax. The client exposes methods like expect and test within the scripting environment. These test scripts are executed as part of the post-response hook. If any assertion fails, the corresponding request (and potentially the entire collection run) is marked as failed. This integrates API testing directly into the development workflow.

Data Model for Test Cases

-- Test Cases Table
CREATE TABLE test_cases (
    id TEXT PRIMARY KEY,
    request_id TEXT NOT NULL,
    name TEXT NOT NULL, -- Name of the test case
    source_code TEXT NOT NULL -- The JavaScript code for assertions
);

Uncovered Aspects and Future Scope

While this design covers core functionalities, several advanced aspects were not detailed:

1. Authentication: Supporting various authentication types (API Key, Bearer Token, Basic Auth, OAuth 2.0) and protocols (HTTP, gRPC) requires significant implementation effort to handle different credential flows and header manipulations.
2. Scripting Runtime Environment: The execution environment for pre-request and post-response scripts is a complex system in itself. It needs to be:
   • Isolated: To prevent scripts from corrupting the main application or other scripts. This often involves sandboxed JavaScript environments (e.g., using Node.js vm module or Web Workers).
   • Preloaded: With common libraries (e.g., Moment.js, Lodash) and the RQ object readily available.
   • Secure: Especially when running user-provided code. Implementing such a robust and flexible runtime is a significant engineering challenge.

Conclusion

Designing an API client like Requestly is far more intricate than merely wrapping curl. It involves sophisticated data modeling for variables, requests, collections, and test results, alongside a robust execution engine for scripting and assertions. The depth of features like variable scoping, request chaining, and integrated testing transforms a simple HTTP client into a powerful development and testing tool, highlighting the complexity and interesting problems involved in building such systems.