Web scraping

Rayobyte runs this browser in production at billion-page-per-month scale against sites that block plain headless Chromium on the first request.

What a stealth browser buys you

Detection checks dozens of signals at once, many of them below JavaScript, and looks for a set that describes one real machine. Stock headless Chromium fails on navigator.webdriver alone, and then again on the GPU renderer, the font list, the screen metrics and how it renders.

rayobrowse sets more than 50 signals from one real device profile inside the browser engine, and runs headful by default so the page is rendered the way desktop Chrome renders it. Your scraper connects over CDP and does not change. See Fingerprint spoofing and Headless and headful.

The shape of a run

import httpx
from playwright.sync_api import sync_playwright

PROXY = "http://USERNAME:[email protected]:8000"
KEY = "rb_live_YOUR_KEY"

resp = httpx.get(
    "https://browser.rayobyte.com/connect",
    params={"os": "windows", "proxy": PROXY},
    headers={"x-api-key": KEY},
    timeout=120,
)
if resp.status_code != 200:
    raise SystemExit(f"{resp.status_code}: {resp.text}")
cdp_url = resp.text.strip()
session_id = resp.headers["x-session-id"]

try:
    with sync_playwright() as p:
        browser = p.chromium.connect_over_cdp(cdp_url)
        context = browser.contexts[0]
        page = context.pages[0] if context.pages else context.new_page()
        for n in range(1, 4):
            page.goto(f"https://books.toscrape.com/catalogue/page-{n}.html")
            print(n, page.locator("article.product_pod").count())
finally:
    httpx.post(
        "https://browser.rayobyte.com/api/browser/close",
        json={"browserId": session_id},
        headers={"x-api-key": KEY},
        timeout=30,
    )

Without the finally block, a scraper that raises keeps its browser running until maxLifetime expires, 7,200 seconds later by default, and all of it counts against your browser hours.

One browser per target session, not per page

Creating a browser per URL wastes the startup on every request and runs into the 5 concurrent limit immediately. Reuse one browser across many page.goto() calls and close it at the end of the batch.

Pin the exit IP

Append -session- and 8 characters to the proxy password so every request in the browser leaves from one IP. A cart, a login or a search session that moves IP mid-flow is thrown out by the site whatever the fingerprint says. See Sticky sessions.

Scaling past 5 browsers

5 concurrent browsers and 48 lifetime hours are what every account includes. A production crawl passes both. Email [email protected] and both are raised on your account.

For crawling, Scrapy with scrapy-playwright uses the browser as a download handler.