High HTTP throughput on your agent

Choose the High throughput installation profile when you need to fetch many addresses. It adds a network service with a shared capacity of up to 6000 concurrent HTTP requests. Scripts, schedules, results, exports and updates remain part of your existing agent.

When to use it

For example, you may collect catalog prices or check a large list of pages. While a website is preparing one response, the agent can wait for other responses too. Parsing, saving results and the source's own response time also affect run duration.

The number 6000 describes HTTP requests in progress. It is not a count of scripts, browser sessions or requests per second. Tasks that require a browser have separate settings and resource requirements.

Enable the profile

1. Open Agents and prepare an installation.
2. At the capabilities step, select High throughput as the installation size.
3. Check the server. This profile needs more memory than a regular agent.
4. Complete installation. The agent card will show the network service status.
5. Assign the agent to your site and configure its request limits.

Regular profiles do not enable the service automatically. An existing installation requires a planned transition by an operator: preserve keys, passwords and volumes, update the installation bundle and add the network service during maintenance. Do not replace the whole existing .env with a newly generated bundle: this could change the encryption key used by stored data. Existing agents keep working in their original mode until a transition is prepared.

Three different limits

The agent's capacity is an additional upper bound. Site settings, robots.txt, proxy routing and other policies still apply. A high capacity is not a reason to set every site to its maximum. Start with a moderate load and inspect responses, errors and run duration.

Reduce high limits when switching the site to the cloud or a regular agent. The server validates these settings when you save them.

Use it in your script

All current SDK languages provide pool(). It fetches independent addresses in parallel and returns responses in input order. Regular profiles accept up to 8 addresses with at most two requests in progress. High throughput accepts up to 6000 addresses, subject to the site's concurrency and shared network capacity. Waiting for responses does not prevent the next permitted requests from starting. Rate limits still apply: a site limited to 100 requests per second will not receive all 6000 requests immediately. With sufficiently slow responses, the number of requests in progress may gradually reach the chosen limit when the wait budget and overall execution deadline allow it.

PHP, inside the script function:

$urls = $ctx->param('product_urls', []);
$responses = $ctx->http()->withoutCache()->pool($urls);
foreach ($responses as $response) {
    $ctx->log('Page received', ['status' => $response->status]);
}

JavaScript, inside async function run(ctx):

const urls = ctx.param('product_urls', []);
const responses = await ctx.http().withoutCache().pool(urls);
for (const response of responses) {
  ctx.log('Page received', { status: response.status });
}

TypeScript uses the same method:

const urls = ctx.param('product_urls', []) as string[];
const responses = await ctx.http().withoutCache().pool(urls);
for (const response of responses) {
  ctx.log('Page received', { status: response.status });
}

Python, inside def run(ctx):

urls = ctx.param('product_urls', [])
responses = ctx.http().without_cache().pool(urls)
for response in responses:
    ctx.log('Page received', {'status': response.status})

withoutCache() or without_cache() requests fresh pages. Omit it when a saved response is suitable. The optional pool() parameter controlling automatic CAPTCHA handling keeps its existing default. This profile adds no new challenge types.

Choose a suitable batch size

One SDK result is limited to 8,000,000 bytes including metadata. Six thousand small responses may fit; six thousand large HTML pages will not. For example, 6000 pages of 100 KB each already occupy about 600 MB before metadata.

Use smaller batches for large pages, for example 16 or 32 addresses. Process one batch before requesting the next, instead of accumulating all responses in script memory. The current pool() returns a completed list rather than an unlimited stream. A network exception or oversized result fails the call. An HTTP status such as 404 or 500 is not a network exception by itself: inspect status and ok() on each response. Pages are never silently truncated.

A pool session allows up to 20,000 network hops, including redirects and retries, and lasts no longer than 10 minutes. This is a separate limit: one URL may need several network hops. Reduce the batch size if that limit is reached.

Each address consumes one SDK call-budget unit. Trial-run budgets and script time limits still apply. Future script languages can use the same host-call transport; the profile itself does not add support for running Go scripts.

Monitor the service

The agent card shows HTTP requests in progress, shared capacity, the peak since the service started, and completed requests. History covers the selected period. When communication is unavailable, the card retains the last received values; check the measurement timestamp.

Cancelling a run stops its remaining requests. An unavailable or busy network service produces a run error. Check connectivity, service health and batch size before retrying. A failed pool does not return a partially successful list; keep this in mind for requests that change data at the source. The service uses the agent's signed update mechanism, with manual updates remaining the default. Component compatibility is checked before an update is completed. If that check fails, the agent restores the previous version.

Method references: PHP, JavaScript, TypeScript, Python.