Modules

A module decides how we get the page. The same URL through two modules can cost different amounts and return a different shape.

Sending one

curl -X POST "http://api.scraping.rayobyte.com/?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","module":"HtmlRequestScraper"}'

Omit module and you get a plain HTML fetch.

The three kinds

Plain HTTP fetch. We request the page and return what came back. Fastest and cheapest. Works on any site that renders server-side. HtmlRequestScraper is the general-purpose one.

Browser rendering. We load the page in a real browser and let its JavaScript run before returning the HTML. Slower and more expensive. Necessary when the content you want is not in the initial response.

Site-specific extractors. Built for one target, returning structured JSON instead of HTML. Use one when it exists for your target, because it survives layout changes that would break your own selectors.

Which to use

Start with the plain fetch. If the data you want is missing from the response but visible in a browser, the page renders client-side and you need browser rendering.

The cheap test: curl the URL directly and search the HTML for a string you expect. If it is not there, no plain fetch will find it.

Not sure which module fits your target? Ask support and we will tell you which one to send.