Scrapy

scrapy-playwright renders pages in a browser. Set PLAYWRIGHT_CDP_URL and it attaches to rayobrowse instead of starting its own. The setup below creates the browser when a crawl starts and closes it when the crawl ends.

1. Install and create a project

   pip install scrapy scrapy-playwright httpx
   scrapy startproject quotes_project
   cd quotes_project

   Scrapy 2.13 or newer. Skip playwright install: the browser is not on your machine.

2. Add quotes_project/rayobyte.py

   One function that creates the browser, and one extension that closes it:

   import os
   
   import httpx
   from scrapy import signals
   
   API = "https://browser.rayobyte.com"
   
   
   def create_browser(settings):
       resp = httpx.get(
           f"{API}/connect",
           params={"os": "windows", "proxy": os.environ["RAYOBYTE_PROXY"]},
           headers={"x-api-key": os.environ["RAYOBYTE_KEY"]},
           timeout=120,
       )
       if resp.status_code != 200:
           raise RuntimeError(f"{resp.status_code}: {resp.text}")
       settings.set("PLAYWRIGHT_CDP_URL", resp.text.strip(), priority="spider")
       settings.set("RAYOBYTE_SESSION_ID", resp.headers["x-session-id"], priority="spider")
   
   
   class CloseRayobyteBrowser:
       def __init__(self, session_id):
           self.session_id = session_id
   
       @classmethod
       def from_crawler(cls, crawler):
           ext = cls(crawler.settings.get("RAYOBYTE_SESSION_ID"))
           crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed)
           return ext
   
       def spider_closed(self, spider):
           if not self.session_id:
               return
           httpx.post(
               f"{API}/api/browser/close",
               json={"browserId": self.session_id},
               headers={"x-api-key": os.environ["RAYOBYTE_KEY"]},
               timeout=30,
           )

3. Add to quotes_project/settings.py

   DOWNLOAD_HANDLERS = {
       "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
       "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
   }
   
   TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
   
   PLAYWRIGHT_PROCESS_REQUEST_HEADERS = None
   PLAYWRIGHT_CONTEXTS = {"default": {"no_viewport": True}}
   
   EXTENSIONS = {"quotes_project.rayobyte.CloseRayobyteBrowser": 500}

   Caution

   Two of these lines protect the fingerprint. PLAYWRIGHT_PROCESS_REQUEST_HEADERS = None stops Scrapy writing its own User-Agent over the one the fingerprint set, which would leave the header disagreeing with the client hints and the platform. no_viewport stops Playwright emulating a 1280×720 viewport in the context scrapy-playwright creates, which would leave the screen and window sizes disagreeing with each other.

4. Write the spider in quotes_project/spiders/quotes.py

   import scrapy
   
   from quotes_project.rayobyte import create_browser
   
   
   class QuotesSpider(scrapy.Spider):
       name = "quotes"
       start_urls = ["https://quotes.toscrape.com/js/"]
   
       @classmethod
       def update_settings(cls, settings):
           super().update_settings(settings)
           create_browser(settings)
   
       async def start(self):
           for url in self.start_urls:
               yield scrapy.Request(
                   url,
                   meta={"playwright": True, "playwright_include_page": True},
               )
   
       async def parse(self, response):
           page = response.meta["playwright_page"]
           await page.close()
   
           for quote in response.css("div.quote"):
               yield {
                   "text": quote.css("span.text::text").get(),
                   "author": quote.css("small.author::text").get(),
               }

5. Run it

   export RAYOBYTE_KEY=rb_live_YOUR_KEY
   export RAYOBYTE_PROXY='http://USERNAME:[email protected]:8000'
   scrapy crawl quotes -o quotes.json

   The log ends with item_scraped_count: 10, and quotes.json holds the 10 quotes from the first page.

Where the browser is created, and why

update_settings runs only when Scrapy builds a crawler for this spider, so scrapy list and other commands that only load the project do not start a browser. Creating the browser in settings.py instead would start one on every scrapy command, and only crawl would ever close it.

The extension closes the session on spider_closed. A crawl that exits without it leaves the browser running for the rest of its 7,200 second default lifetime, all of it against your 48 browser hours.

On Scrapy older than 2.13, rename start to start_requests and make it a plain def. Newer Scrapy ignores start_requests when start is not defined, so the requests go out without the playwright flag, skip the browser, and parse fails with KeyError: 'playwright_page'.

One browser, one crawl

PLAYWRIGHT_CDP_URL is a single browser and every concurrent request in the crawl shares it. Five concurrent browsers is your account limit, so five crawls is the ceiling before the sixth gets 429 Concurrent browser limit reached.