Documentation

Sign in and collect the catalog (PHP)

Submit the demonstration site sign-in form and read its catalog in the same browser session. The credentials come from a public hint in the page.

Choose PHP, paste the complete source from this article and set the run parameters below. The source for this collection is ScrapingCourse. This walkthrough applies to the specified demonstration page.

Example settings

{
  "path": "/login",
  "mode": "login",
  "expected": 12
}

path selects the page, mode selects an algorithm branch, and expected sets the control count for this example. It is not a plan limit. When the source dataset changes, check the expected count against it.

How the algorithm works

Open /login with the login mode, which selects the browser branch of the shared source.
Read the paragraphs and find the Demo Credentials: hint. Extract its demonstration email and password.
Require both values before submitting, so a changed hint does not cause an empty sign-in attempt.
Fill #email and #password, then click the ordinary submit button. The credentials are not fixed in the source and are not emitted as records.
Wait for a nonempty .product-name in that same session to identify the catalog after sign-in.
Collect unique product cards, compare the count with expected and close the page in finally.

Products are processed by courseCollect($ctx, $html, $seen). Read the name first, skip empty names, and use the card link as the key, falling back to the name when necessary. seen prevents the same key being emitted twice. This matters when a later HTML read includes products already collected.

Details that matter

Keep the session open between form submission and catalog extraction: it preserves the sign-in state for subsequent actions.

PHP details

PHP calls network and browser methods sequentially with ->. courseCollect receives $seen and returns its updated array, so assign each result back. Close the page in finally; $ctx->fail() stops the run with a readable message.

Expected output

The configured demonstration dataset expects 12 unique records. The final check compares the actual unique key count with expected.

Field	Contents
`name`	Product name with leading and trailing whitespace removed.
`price`	Price converted from text to a number.
`image`	Image URL read from the card src attribute.

Complete scenario source

The source is shared by this collection. This example runs its login branch using the parameters above. Copy the complete block into the PHP editor; no fixed sign-in credentials or service keys need to be added.

<?php

function run(\Sdk\Context $ctx): void
{
    $origin = 'https://www.scrapingcourse.com';
    $path = (string) $ctx->param('path', '/pagination');
    $mode = (string) $ctx->param('mode', 'http');
    $auto = $ctx->param('auto_captcha', true) !== false;
    $seen = [];
    if ($mode === 'http') {
        $url = $origin.$path;
        $visited = [];
        while ($url !== null && count($visited) < 50) {
            if (isset($visited[$url])) {
                $ctx->fail('Pagination returned an already processed page');
            }
            $visited[$url] = true;
            $response = $ctx->http()->withoutCache()->get($url, $auto);
            if (! $response->ok()) {
                $ctx->fail('The source could not be opened: '.$response->status);
            }
            $html = $response->body;
            $seen = courseCollect($ctx, $html, $seen);
            $next = $ctx->dom($html)->attr('a[rel="next"], a.next.page-numbers', 'href');
            $url = $next ? $ctx->url()->absolutize($url, $next) : null;
        }
        if ($url !== null) {
            $ctx->fail('The pagination safety limit was reached');
        }
    } else {
        $page = $ctx->browser()->open($origin.$path, ['*/ajax/products*'], [], $auto);
        try {
            if ($mode === 'login') {
                $dom = $ctx->dom($page->content());
                $hint = '';
                foreach ($dom->qa('p') as $p) {
                    if (str_contains($p->text(), 'Demo Credentials:')) {
                        $hint = $p->text();
                    }
                }
                preg_match('/Password:\s*([^\s|]+)/i', $hint, $password);
                preg_match('/Email:\s*([^\s|]+@[^\s|]+)/i', $hint, $emailMatch);
                $email = $emailMatch[1] ?? '';
                if ($email === '' || ! isset($password[1])) {
                    $ctx->fail('The page does not contain demo credentials');
                }
                $page->type('#email', $email)->type('#password', $password[1])->click('button[type="submit"]');
            }
            if ($mode === 'challenge') {
                $text = preg_replace('/\s+/', ' ', $ctx->dom($page->content())->text('body') ?? '');
                if (! preg_match('/You (?:successfully )?bypassed|successfully bypassed|Congratulations/i', $text)) {
                    $ctx->fail('The expected page content was not received');
                }
                $ctx->emit(['name' => $path, 'passed' => true], $path);
            } else {
                $page->waitFor('.product-name:not(:empty)', 15000);
                if ($mode === 'button') {
                    $complete = false;
                    for ($i = 0; $i < 30; $i++) {
                        $html = $page->content();
                        $before = $ctx->dom($html);
                        $seen = courseCollect($ctx, $html, $seen);
                        $button = $before->q('#load-more-btn');
                        if ($button === null || preg_match('/display:\s*none/', $button->attr('style') ?? '')) {
                            $complete = true;
                            break;
                        }
                        $offset = $button->attr('data-offset');
                        $count = count($before->qa('.product-item'));
                        $page->click('#load-more-btn');
                        $response = $page->waitForResponse('*offset='.$offset, 15000);
                        if ($response === null || ($response['status'] < 200 || $response['status'] >= 300) || $response['truncated'] || $response['error']) {
                            $ctx->fail('The next batch could not be loaded');
                        }
                        if (trim($response['body'] ?? '') === '') {
                            $complete = true;
                            break;
                        }
                        $page->waitFor('.product-item:nth-child('.($count + 1).')');
                    }
                    if (! $complete) {
                        $ctx->fail('The batch loading safety limit was reached');
                    }
                }
                if ($mode === 'scroll') {
                    $stable = 0;
                    $previous = 0;
                    for ($i = 0; $i < 40 && $stable < 3; $i++) {
                        $page->scroll(3, 400);
                        $html = $page->content();
                        $seen = courseCollect($ctx, $html, $seen);
                        $count = count($ctx->dom($html)->qa('.product-item'));
                        $stable = $count === $previous ? $stable + 1 : 0;
                        $previous = $count;
                    }
                    if ($stable < 3) {
                        $ctx->fail('The feed is still loading: increase the scroll limit');
                    }
                }
                $seen = courseCollect($ctx, $page->content(), $seen);
            }
        } finally {
            $page->close();
        }
    }
    $expected = (int) $ctx->param('expected', 1);
    if ($mode !== 'challenge' && count($seen) !== $expected) {
        $ctx->fail('Incomplete result: '.count($seen).' instead of '.$expected);
    }
}

function courseCollect(\Sdk\Context $ctx, string $html, array $seen): array
{
    $dom = $ctx->dom($html);
    foreach ($dom->qa('.product-item, li.product') as $card) {
        $name = trim($card->q('.product-name, .woocommerce-loop-product__title')?->text() ?? '');
        $key = $card->q('a')?->attr('href') ?? $name;
        if ($name === '' || isset($seen[$key])) {
            continue;
        }
        $seen[$key] = true;
        $ctx->emit(['name' => $name, 'price' => $ctx->text()->money($card->q('.product-price, .price')?->text() ?? ''),
            'image' => $card->q('img')?->attr('src') ?? ''], $key);
    }

    return $seen;
}