PHP SDK: Collecting multiple pages and categories

If a catalog has 20 pages, reading only the first misses most products. First identify how to move forward: a Next link, page number, API cursor, or Load more button.

Example: follow a Next link

Set catalog_url in the run parameters. Replace .product-card and a.next with your selectors. The site schema needs sku and name fields.

<?php
function run(\Sdk\Context $ctx): void
{
    $url = (string) $ctx->param('catalog_url', '');
    if ($url === '') {
        $ctx->abort('Set catalog_url');
    }
    $visited = [];
    $maxPages = (int) $ctx->param('max_pages', 100);
    $page = 0;
    while ($url !== '') {
        if ($ctx->cancelled()) {
            $ctx->abort('Collection cancelled');
        }
        if (isset($visited[$url])) {
            $ctx->abort('Pagination returned a page already processed');
        }
        if ($page >= $maxPages) {
            $ctx->abort('Reached max_pages. The catalog is incomplete');
        }
        $visited[$url] = true;
        $response = $ctx->http()->get($url);
        if (!$response->ok()) {
            $ctx->abort('Could not read page: HTTP ' . $response->status);
        }
        $dom = $response->dom();
        $cards = $dom->qa('.product-card');
        if (count($cards) === 0) {
            $ctx->abort('Empty catalog page. Inspect the snapshot');
        }
        foreach ($cards as $card) {
            $sku = $card->attr('data-sku');
            if ($sku === null) {
                $ctx->log('Card without SKU', ['page' => $page + 1], 'warning');
                continue;
            }
            $ctx->emit(['sku' => $sku, 'name' => $card->q('.product-title')?->text()], $sku);
        }
        $page++;
        $ctx->progress($page, null, 'Walking catalog pages');
        $ctx->log('Page complete', ['page' => $page, 'cards' => count($cards)]);
        $url = $dom->q('a.next')?->abs() ?? '';
    }
    $ctx->progress($page, $page, 'Catalog traversal complete');
}

$visited stops a loop when the site returns the same link again. max_pages prevents an endless traversal, but reaching the limit means the collection is incomplete. Do not report that run as fully complete.

If the page range is known

The SDK supports $ctx->http()->pages($template, $from, $to). The template uses {page}, for example https://shop.example.invalid/catalog?page={page}. Determine the last page from the site's data. Do not hard-code a guessed number: the catalog may grow.

For categories, first gather every category link, remove duplicates, then visit the pages of each category. A product may belong to more than one category. Use a stable product key across the whole catalog and decide separately how to store its category list.

Common mistakes

• A Next button is present but disabled. Refine the selector or check its state.
• Tracking query parameters change on every page. Normalize links so $visited can recognize repeats.
• The server returns HTTP 200 with an access-check page. Check for product cards, not just the status code.
• Prices or stock are present only on product detail pages. Add a detail-page step after reading the listing.

Before publishing, compare the page count and several products from the beginning, middle, and end of the catalog.