Getting Started

1. Welcome to Effortless Web Scraping

Welcome to the Rayobyte Web Scraping API! This powerful tool lets you scrape websites at scale without worrying about blocks, CAPTCHAs, or IP bans. It’s designed to be easy to use and flexible for both beginners and advanced developers.

➡️ Learn More: https://rayobyte.com/products/web-scraper-api/

2. What Makes This API a Game-Changer?

Say goodbye to web scraping headaches:

Reliable page loads even from challenging sites.
Built-in handling for CAPTCHAs, blocks, and rate limits.
Flexible data parsing options like XPath or CSS selectors.
Support for geolocation targeting and JavaScript rendering.

Tip: For websites with specific structures or complex data needs, check out our Custom Modules for platforms like Google and Amazon. Contact support if you're interested.

3. Let’s Get You Started!

Step 1: Grab Your API Key

First, get your API key from your account dashboard. This key authenticates your requests and ensures they are processed correctly.

➡️ Sign up or login today at: https://app.rayobyte.com/

Tip: You can find your scraping information and key on the left hand menu under "Scraping API". You can also trial any of Rayobyte's other data products as well, such as data center proxies, ISP proxies, residential proxies, and mobile proxies.

Step 2: Send Your First Request

Use the base URL format to request a page. Below are code samples in different languages:

curl 'https://api.scraping.rayobyte.com/?token=<API-KEY>&url=https://tools-api.rayobyte.com/free-ip-lookup/'

const axios = require('axios');

// Define the API URL and parameters
const url = 'https://api.scraping.rayobyte.com/';
const params = {
    token: '<API-KEY>',
    url: 'https://tools-api.rayobyte.com/free-ip-lookup/'
};

// Make the GET request
axios.get(url, { params })
    .then(response => {
        console.log("Response:", response.data);
    })
    .catch(error => {
        if (error.response) {
            console.error(`Error: ${error.response.status} - ${error.response.data}`);
        } else {
            console.error(`Error: ${error.message}`);
        }
    });

https://api.scraping.rayobyte.com/?token=<API-KEY>&url=https://tools-api.rayobyte.com/free-ip-lookup/

import requests

# Define the API URL and parameters
url = "https://api.scraping.rayobyte.com/"
params = {
    "token": "<API-KEY>",
    "url": "https://tools-api.rayobyte.com/free-ip-lookup/"
}

# Make the GET request
response = requests.get(url, params=params)

# Check and print the response
if response.status_code == 200:
    print("Response:", response.json())
else:
    print(f"Error: {response.status_code} - {response.text}")

Expected Result (JSON)

When you make a request, our API returns a structured JSON response containing all the key details you need. Here's a breakdown of the fields you'll see in the response:

{
  "status": "SUCCESS",
  "date": "Thu, 19 Dec 2024 07:56:04 GMT",
  "httpCode": 200,
  "headers": {
    "<Headers returned>"
  },
  "result": "<HTML content of the scraped page>"
}

Output from above API call

{
  "status": "SUCCESS",
  "date": "Thu, 19 Dec 2024 07:56:04 GMT",
  "httpCode": 200,
  "headers": {
    "server": "nginx/1.20.1",
    "content-length": "621",
    "access-control-max-age": "3600"
  },
  "result": {
    "ipVersion": 4,
    "ipAddress": "1.2.3.4",
    "latitude": -27.467541,
    "longitude": 153.028091,
    "countryName": "Australia",
    "countryCode": "AU",
    "timeZone": "+10:00",
    "zipCode": "4000",
    "cityName": "Brisbane",
    "regionName": "Queensland",
    "isProxy": false,
    "continent": "Oceania",
    "continentCode": "OC",
    "currency": {
      "code": "AUD",
      "name": "Australian Dollar"
    },
    "language": "English",
    "timeZones": [
      "Antarctica/Macquarie",
      "Australia/Adelaide",
      "Australia/Brisbane",
      "Australia/Broken_Hill",
      "Australia/Darwin",
      "Australia/Eucla",
      "Australia/Hobart",
      "Australia/Lindeman",
      "Australia/Lord_Howe",
      "Australia/Melbourne",
      "Australia/Perth",
      "Australia/Sydney"
    ],
    "tlds": [
      ".au"
    ]
  }

Explanation of the Fields:

status: Indicates whether the request was successful (SUCCESS) or encountered an issue (FAIL).
date: A timestamp of when the API processed your request.
httpCode: The HTTP status code of the target website, such as 200 for success or 404 for not found.
headers: A collection of HTTP headers returned by the target website, which may include details like server type, content length, or caching information.
result: The HTML content of the page you requested, which you can process further to extract the data you need.

➡️ Learn more about response codes and troubleshooting.

4. Features You’ll Want to Explore

Make scraping smarter by adding these features to your requests:

Geolocation Targeting: Scrape localized content based on country or region.
JavaScript Rendering: Use headless browser rendering to scrape websites that require JS.
Custom Headers and Cookies: Mimic real-world browsers or pass unique identifiers.

➡️ See all Features here.

5. Common Roadblocks? We’ve Got You.

Here’s how to solve some common issues:

Invalid API Key: Double-check your key or regenerate it in your dashboard.
Timeouts or Slow Responses: Adjust timeout settings (learn more here).
Response Errors: Refer to the Response Codes page for troubleshooting specific error codes.

Tip: If you’re stuck, our support team is just a click away!

6. Ready to Level Up?

Take your scraping to the next level:

Google Module: Get structured SERP data for Google.
Amazon Module: Perfect for extracting product information.

➡️ Explore the Modules section for more advanced scraping modules. Contact support if you're interested.

Got a question?

Our awesome, human, support team is just a click away!

NextFeatures

Last updated 6 months ago