Custom Headers and Cookies

Custom headers allow you to control the exact information your scraping requests send to a target website. This can be helpful for specialized scenarios where the default setup doesn’t quite cut it. Let’s explore how to use custom headers, when they’re useful, and why they might not always be your best option.

---

When (and Why) Use Custom Headers?

By default, Rayobyte’s API manages headers dynamically. Our intelligent system adjusts headers like user-agent and cookies for optimal success rates. This works great for most scraping projects.

But what if you need something extra? For instance:

• You want to impersonate a specific browser or device using a custom user agent.

• You need to provide a session ID or preferences using cookies.

In such cases, custom headers can be a game-changer. But remember:

• Control comes with responsibility. You override our dynamic header system, which may reduce success rates if your headers aren’t spot-on.

• Stick to custom headers only when the website specifically requires them.

---

How to Send Custom Headers

Using custom headers in Rayobyte’s API is easy! Simply add the &customHeaders=true parameter to your API query and include the headers in your request.

Query Example

Here’s what a request with custom headers looks like:

Example Requests:

cURL

curl 'https://api.scraping.rayobyte.com/?token=<API-KEY>&customHeaders=true&url=https://httpbin.org/headers'

JavaScript

const axios = require('axios');

// Define the API URL and parameters
const url = 'https://api.scraping.rayobyte.com/';
const params = {
    token: '<API-KEY>',
    customHeaders: 'true',
    url: 'https://httpbin.org/headers'
};

// Make the GET request
axios.get(url, { params })
    .then(response => {
        console.log("Response:", response.data);
    })
    .catch(error => {
        if (error.response) {
            console.error(`Error: ${error.response.status} - ${error.response.data}`);
        } else {
            console.error(`Error: ${error.message}`);
        }
    });

GET/HTTP

https://api.scraping.rayobyte.com/?token=<API-KEY>&customHeaders=true&url=https://httpbin.org/headers

Python

import requests

# Define the API URL and parameters
url = "https://api.scraping.rayobyte.com/"
params = {
    "token": "<API-KEY>",
    "customHeaders": "true",
    "url": "https://httpbin.org/headers"
}

# Make the GET request
response = requests.get(url, params=params)

# Check and print the response
if response.status_code == 200:
    print("Response:", response.json())
else:
    print(f"Error: {response.status_code} - {response.text}")

How to Add Headers

Send your desired headers in your HTTP Headers, and we’ll forward them to the target site on your behalf.

---

Useful Headers for Web Scraping

Again, please note that using this functionality may break your scraping efforts. Use with caution ⚠️!

1. Custom User-Agent

Mimic a specific browser, operating system, or device to avoid detection. Example: User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36

2. Cookies

Pass session information, user preferences, or authentication data. Example: Cookie: sessionid=abc123; theme=dark

3. Accept-Language

Specify the preferred language for the content you want. Example: Accept-Language: en-US,en;q=0.9

4. Referer

Simulate a previous page the request came from, useful for some site workflows. Example: Referer: https://example.com/previous-page

---

Important Tips

• Be Specific: Ensure your custom headers match the target site’s requirements. Sending incomplete or incorrect headers may lead to failed requests.

• Limit Usage: Stick to custom headers only when absolutely necessary. Rayobyte’s built-in system is designed to handle headers optimally for most cases.

• Test Thoroughly: Websites may change how they handle headers, so test regularly to ensure compatibility