Documentation

Infinite scrolling: detect the end of a feed (Python)

Collect products from a feed that loads content as the page scrolls. Track the visible card count and check the completeness of the final list.

Choose Python, paste the complete source from this article and set the run parameters below. The source for this collection is ScrapingCourse. This walkthrough applies to the specified demonstration page.

Example settings

{
  "path": "/infinite-scrolling",
  "mode": "scroll",
  "expected": 147
}

path selects the page, mode selects an algorithm branch, and expected sets the control count for this example. It is not a plan limit. When the source dataset changes, check the expected count against it.

How the algorithm works

Open /infinite-scrolling in a browser and wait for the first nonempty product name.
Call scroll(3, 400) inside the loop: three scroll steps with a 400-millisecond pause between steps. The second argument is not a pixel distance.
Read the current HTML and collect products after scrolling. seen retains every unique product encountered during this run.
Compare the current DOM card count with the previous value. Increment stable only when the count is unchanged; reset it when the count changes.
Finish after three consecutive unchanged observations. A separate limit allows at most 40 loop iterations.
Read the final page again, compare the result with expected and close the browser regardless of the outcome.

Products are processed by collect(html). Read the name first, skip empty names, and use the card link as the key, falling back to the name when necessary. seen prevents the same key being emitted twice. This matters when a later HTML read includes products already collected.

Details that matter

The page card count and the number of newly emitted rows are different. stable tracks cards, while expected is checked against unique keys in seen.

Python details

The Python SDK is sequential here: no async or await is needed. Method names use snake_case: without_cache, wait_for, wait_for_response. The nested collect uses a shared key set and returns the current HTML card count. finally calls page.close().

Expected output

The configured demonstration dataset expects 147 unique records. The final check compares the actual unique key count with expected.

Field	Contents
`name`	Product name with leading and trailing whitespace removed.
`price`	Price converted from text to a number.
`image`	Image URL read from the card src attribute.

Complete scenario source

The source is shared by this collection. This example runs its scroll branch using the parameters above. Copy the complete block into the Python editor; no fixed sign-in credentials or service keys need to be added.

import re


def run(ctx):
    origin = "https://www.scrapingcourse.com"
    path = str(ctx.param("path", "/pagination"))
    mode = str(ctx.param("mode", "http"))
    auto = ctx.param("auto_captcha", True) is not False
    seen = set()

    def collect(html):
        dom = ctx.dom(html)
        for card in dom.qa(".product-item, li.product"):
            title = card.q(".product-name, .woocommerce-loop-product__title")
            name = title.text().strip() if title else ""
            link = card.q("a")
            key = (link.attr("href") if link else "") or name
            if not name or key in seen:
                continue
            price = card.q(".product-price, .price")
            image = card.q("img")
            seen.add(key)
            ctx.emit({"name": name, "price": ctx.text().money(price.text() if price else ""),
                      "image": image.attr("src") if image else ""}, key)
        return len(dom.qa(".product-item, li.product"))

    if mode == "http":
        url = origin + path
        visited = set()
        while url and len(visited) < 50:
            if url in visited:
                raise ValueError("Pagination returned an already processed page")
            visited.add(url)
            response = ctx.http().without_cache().get(url, auto)
            if not response.ok():
                raise ValueError("The source could not be opened: " + str(response.status))
            html = response.body
            collect(html)
            next_url = ctx.dom(html).attr('a[rel="next"], a.next.page-numbers', "href")
            url = ctx.url().absolutize(url, next_url) if next_url else None
        if url:
            raise ValueError("The pagination safety limit was reached")
    else:
        page = ctx.browser().open(origin + path, ["*/ajax/products*"], {}, auto)
        try:
            if mode == "login":
                dom = ctx.dom(page.content())
                hint = next((p.text() for p in dom.qa("p") if "Demo Credentials:" in p.text()), "")
                password = re.search(r"Password:\s*([^\s|]+)", hint, re.I)
                email_match = re.search(r"Email:\s*([^\s|]+@[^\s|]+)", hint, re.I)
                email = email_match.group(1) if email_match else ""
                encoded = dom.attr("[data-cfemail]", "data-cfemail")
                if not email and encoded and len(encoded) > 2:
                    key = int(encoded[:2], 16)
                    email = "".join(chr(int(encoded[i:i + 2], 16) ^ key) for i in range(2, len(encoded), 2))
                if not email or not password:
                    raise ValueError("The page does not contain demo credentials")
                page.type("#email", email)
                page.type("#password", password.group(1))
                page.click('button[type="submit"]')
            if mode == "challenge":
                text = re.sub(r"\s+", " ", ctx.dom(page.content()).text("body") or "")
                if not re.search(r"You (?:successfully )?bypassed|successfully bypassed|Congratulations", text, re.I):
                    raise ValueError("The expected page content was not received")
                ctx.emit({"name": path, "passed": True}, path)
            else:
                page.wait_for(".product-name:not(:empty)", 15000)
                if mode == "button":
                    complete = False
                    for step in range(30):
                        html = page.content()
                        before = ctx.dom(html)
                        collect(html)
                        button = before.q("#load-more-btn")
                        if not button or re.search(r"display:\s*none", button.attr("style") or ""):
                            complete = True
                            break
                        offset = button.attr("data-offset")
                        count = len(before.qa(".product-item"))
                        page.click("#load-more-btn")
                        response = page.wait_for_response("*offset=" + offset, 15000)
                        if not response or not 200 <= response["status"] < 300 or response["truncated"] or response["error"]:
                            raise ValueError("The next batch could not be loaded")
                        if not (response.get("body") or "").strip():
                            complete = True
                            break
                        page.wait_for(".product-item:nth-child(" + str(count + 1) + ")")
                    if not complete:
                        raise ValueError("The batch loading safety limit was reached")
                if mode == "scroll":
                    stable, previous = 0, 0
                    for step in range(40):
                        page.scroll(3, 400)
                        count = collect(page.content())
                        stable = stable + 1 if count == previous else 0
                        previous = count
                        if stable >= 3:
                            break
                    if stable < 3:
                        raise ValueError("The feed is still loading: increase the scroll limit")
                collect(page.content())
        finally:
            page.close()
    expected = int(ctx.param("expected", 1))
    if mode != "challenge" and len(seen) != expected:
        raise ValueError("Incomplete result: " + str(len(seen)) + " instead of " + str(expected))