JavaScript SDK: Logs, progress, and safe retries
For a long collection, users need to see what is happening. ctx.log() adds an explanation to the run log, ctx.progress() shows the current stage, and ctx.metric() records a number. These methods are local and need no await.
Useful logs
Record the start of a category, page counts, skipped items, and reasons for stopping. Do not log passwords, tokens, private mail text, or complete private documents. Use ctx.fail(reason) with a concrete observation, such as “page returned HTTP 403”, instead of a vague message.
Progress and cancellation
Pass done and total when the total is known. Otherwise pass null for the total and update the stage label. Do not display 100% until all records have been saved.
async function run(ctx) {
const urls = ctx.param('urls', []);
if (!Array.isArray(urls)) ctx.abort('Enter a URL list');
let processed = 0;
ctx.log('Starting collection', { pages: urls.length });
for (const url of urls) {
if (await ctx.cancelled()) ctx.abort('Stopped by the user');
const response = await ctx.http().get(String(url));
if (!response.ok()) ctx.fail(`HTTP ${response.status}: ${url}`);
processed++;
ctx.metric('pages_processed', 1);
ctx.progress(processed, urls.length, 'Loading pages');
}
ctx.log('Collection finished', { processed });
}
State between runs
await ctx.state().get(key, fallback) reads a saved cursor; set() and incr() update it. State is not an output database. Use ctx.emit(record, key) with a stable key for products. Repeat a trial with the same inputs and check for duplicates or missing records.
async function run(ctx) {
const state = ctx.state();
const nextPage = Number(await state.get('next_page', 1));
const url = ctx.url().withParam(ctx.param('catalog_url'), 'page', nextPage);
const response = await ctx.http().get(url);
if (!response.ok()) ctx.fail(`HTTP ${response.status}`);
const cards = response.dom().qa('.product-card');
for (const card of cards) {
const sku = card.attr('data-sku');
if (sku) ctx.emit({ sku, name: card.text() }, sku);
}
if (cards.length) await state.set('next_page', nextPage + 1);
}
Update the cursor after successfully reading the page. ctx.flush() sends buffered records for processing, but does not itself confirm durable storage. Check the final run status and use stable record keys so retries are safe.