Making requests

Endpoint: http://api.scraping.rayobyte.com/

GET

Everything in the query string. Good for a quick fetch.

curl -G "http://api.scraping.rayobyte.com/" \
  --data-urlencode "token=YOUR_TOKEN" \
  --data-urlencode "url=https://example.com"

Use -G with --data-urlencode rather than building the string by hand. The url parameter is a URL inside a URL, and an unencoded & in your target ends the parameter early.

POST

A JSON body instead. This is the form to use once you are sending options, because nothing needs encoding twice.

curl -X POST "http://api.scraping.rayobyte.com/?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","module":"HtmlRequestScraper"}'

Python

import requests

r = requests.post(
    "http://api.scraping.rayobyte.com/",
    params={"token": "YOUR_TOKEN"},
    json={"url": "https://example.com", "module": "HtmlRequestScraper"},
    timeout=120,
)
print(r.text)

Node

const res = await fetch(
  `http://api.scraping.rayobyte.com/?token=${process.env.RAYOBYTE_TOKEN}`,
  {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      url: 'https://example.com',
      module: 'HtmlRequestScraper',
    }),
  }
);
console.log(await res.text());

Parameters

Field	Required	Description
token	Yes	Your API token. Always in the query string.
url	Yes	The page to fetch.
module	No	How the page is fetched and parsed. Defaults to a plain HTML fetch.

The full list is on Parameters.

Choosing a module

module decides whether the page is fetched with a plain HTTP request, rendered in a real browser, or handled by an extractor written for that specific site.

HtmlRequestScraper is the general-purpose HTML fetch and the right default. See Modules.

Timeouts

We may render in a browser and retry several times before answering, so a single request can take tens of seconds. Set your client timeout to at least 120 seconds. Anything shorter will cut off requests that were going to succeed and you will still be billed for the work.

Concurrency

Send requests in parallel rather than raising your timeout and waiting. Each request is independent and there is no session to keep alive between them.