Logs, progress, and safe retries in Python

log(message, context, level) writes a structured run event, progress(done, total, label, percent) shows the current stage, and metric(name, value) records a numeric measurement. They work in trials and live runs. Use these methods instead of treating print as the run log.

def run(ctx):
    urls = ctx.param('urls', [])
    if not isinstance(urls, list):
        ctx.abort('Enter a list of URLs')
    ctx.log('Starting collection', {'pages': len(urls)})
    processed = 0
    for url in urls:
        if ctx.cancelled():
            ctx.abort('Collection stopped by the user')
        response = ctx.http().get(str(url))
        if not response.ok():
            ctx.fail('HTTP ' + str(response.status))
        processed += 1
        ctx.metric('pages_processed', 1)
        ctx.progress(processed, len(urls), 'Loading pages')
    ctx.log('Collection completed', {'processed': processed})

Pass None when the total is unknown. Do not invent exact percentages or show completion before processing all records. Logs should explain actions and skips without passwords, tokens, private email, or document contents.

State between runs

def run(ctx):
    state = ctx.state()
    number = int(state.get('next_page', 1))
    url = ctx.url().with_param(ctx.param('catalog_url'), 'page', number)
    response = ctx.http().get(url)
    if not response.ok():
        ctx.fail('HTTP ' + str(response.status))
    cards = response.dom().qa('.product-card')
    for card in cards:
        sku = card.attr('data-sku')
        if sku:
            ctx.emit({'sku': sku, 'name': card.text()}, sku)
    ctx.flush()
    if cards:
        state.set('next_page', number + 1)

State stores a small cursor, not the catalog. flush() submits buffered records to the pipeline; it is not an acknowledgment of the whole run's final successful commit. A retry can reprocess a page, so stable keys remain essential. For a full refresh, define when the cursor resets to page one rather than incrementing forever.

State has shared key-count and value-size limits. Trial and live execution can differ in persistence context. Check the intended behavior with a controlled run. See KeyValue.