Paginating categories and pages in JavaScript

Pagination must cover the intended pages without missing products or looping on the last page. First identify whether the page number comes from a URL, a Next button, or a JSON response. Then set an upper bound and choose a stable record key.

Page number in a URL

For a page URL parameter, use ctx.url().withParam(). Check the HTTP status, whether cards appeared, and cancellation after each request. An empty page and a maximum page count are complementary stop conditions.

async function run(ctx) {
  const base = ctx.param('catalog_url', '');
  if (!base) ctx.abort('Enter a catalog URL');
  const maxPages = Math.min(100, Math.max(1, Number(ctx.param('max_pages', 10))));
  const seen = new Set();

  for (let number = 1; number <= maxPages; number++) {
    if (await ctx.cancelled()) ctx.abort('Collection stopped');
    const url = ctx.url().withParam(base, 'page', number);
    const response = await ctx.http().get(url);
    if (!response.ok()) ctx.fail(`Page ${number}: HTTP ${response.status}`);
    const cards = response.dom().qa('.product-card');
    if (!cards.length) break;

    let added = 0;
    for (const card of cards) {
      const sku = card.attr('data-sku');
      if (!sku || seen.has(sku)) continue;
      seen.add(sku);
      added++;
      ctx.emit({ sku, name: card.q('.title')?.text() ?? '' }, sku);
    }
    ctx.progress(number, maxPages, 'Catalog pages');
    if (added === 0) break;
  }
}

The .product-card, data-sku, and .title selectors are examples. Choose selectors for your website and verify them in a trial. Set removes duplicates only within the current run; a stable ctx.emit() key connects records across runs.

Multiple categories

The category list can come from scenario parameters. Validate its type so a string or empty value does not silently become one URL. Bound the number of categories and pages per category. Use a shared seen if an SKU should have one record across categories; include the category in the key if records should remain separate.

async function run(ctx) {
  const categories = ctx.param('category_urls', []);
  if (!Array.isArray(categories)) ctx.abort('Enter a category list');
  const seen = new Set();
  for (const categoryUrl of categories.slice(0, 20)) {
    if (await ctx.cancelled()) ctx.abort('Collection stopped');
    const response = await ctx.http().get(String(categoryUrl));
    if (!response.ok()) ctx.fail(`Category: HTTP ${response.status}`);
    for (const card of response.dom().qa('.product-card')) {
      const sku = card.attr('data-sku');
      if (!sku || seen.has(sku)) continue;
      seen.add(sku);
      ctx.emit({ sku, category: String(categoryUrl), name: card.text() }, sku);
    }
  }
}

For a Next button, infinite scroll, or XHR, use browser actions and wait for a new card or network response. A click alone does not prove that data changed.

Lifecycle · HttpClient · JSON data