Python SDK: Working with CAPTCHA: scenarios and their structure

Explore 12 ScrapingCourse scenarios in Python: complete catalogs, pagination, batch loading, scrolling, sign-in, JavaScript rendering, tables and control pages. Each walkthrough provides settings, complete source and step-by-step explanations.

Getting started

1. Open the site with base URL https://www.scrapingcourse.com, or add it to your workspace.
2. Choose a scenario below and select Python.
3. Copy the complete source from its article. Set path, mode and expected exactly as shown.
4. For catalogs, prepare name (string), price (number) and image (string) fields. Control pages use name (string) and passed (boolean).
5. Check a trial result, then save a version and configure a schedule for your task.

Where CAPTCHA fits

The scenario defines data actions: open a page, fill a form, wait for products and emit records. Site settings and auto_captcha govern additional page checks. Their internal mechanisms are separate from the field-extraction algorithm.

In the complete source, auto_captcha defaults to true. Set it to false to disable automatic handling for requests and the browser session in this run. The SDK argument is named autoCaptcha in PHP, JavaScript and TypeScript and auto_captcha in Python; these sources pass the value of auto positionally.

The examples do not invoke a separate challenge-solving method. Scenario success depends on received data or the control phrase on the page.

CAPTCHA settings and costs · SDK and API parameters

Shared source parameters

One source supports all variants below. Parameters select the scenario, so shared logic does not need rewriting when switching from pagination to sign-in.

Field extraction

A new seen collection tracks product keys already processed during the run. The collector selects .product-item, li.product, reads the name and skips elements without a name. A card link is the key, with the name as a fallback. Products with equal names and different links therefore remain separate records.

name comes from .product-name, .woocommerce-loop-product__title; price comes from .product-price, .price and is converted with money. image contains the URL in img[src]. Actual file storage is described in Storage.

The second emit argument supplies the record key. Reading the whole page again does not emit another result for an encountered key because seen filters it out. This collection lasts only for the current run; the example does not persist it in KeyValue between runs.

The HTTP branch has a separate visited collection for page URLs. Different pages can contain the same product, so these collections serve different purposes. The browser branch closes its page in finally, including on errors.

Python details

The Python SDK is sequential here: no async or await is needed. Method names use snake_case: without_cache, wait_for, wait_for_response. The nested collect uses a shared key set and returns the current HTML card count. finally calls page.close().

For sign-in, read the email from the text hint first. The code also supports a page that represents it with a data-cfemail attribute. The password comes from the same public demonstration hint.

All 12 scenarios

The counts refer to the configured demonstration dataset. Each article explains completeness checks, stopping conditions and field meanings. Copy the complete source from a walkthrough and keep the parameters for the selected example.

Cloud and your own agent

The same source can run in the cloud or on your own agent. Select the execution location in the site settings. For browser examples, the agent needs browser support and the selected language runtime. Field extraction and run parameters stay the same. See agents for setup.