Crawleo Documentation

Crawler API

curl --request GET \
  --url https://api.crawleo.dev/crawl \
  --header 'x-api-key: <x-api-key>'

const options = {method: 'GET', headers: {'x-api-key': '<x-api-key>'}};

fetch('https://api.crawleo.dev/crawl', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

{
  "results": [
    {
      "url": "<string>",
      "status_code": 123,
      "raw_html": "<string>",
      "enhanced_html": "<string>",
      "markdown": "<string>",
      "page_text": "<string>",
      "screenshot": "<string>",
      "error": "<string>"
    }
  ],
  "credits": 123,
  "successful_pages": 123
}

GET

https://api.crawleo.dev

crawl

Crawler API

curl --request GET \
  --url https://api.crawleo.dev/crawl \
  --header 'x-api-key: <x-api-key>'

const options = {method: 'GET', headers: {'x-api-key': '<x-api-key>'}};

fetch('https://api.crawleo.dev/crawl', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

{
  "results": [
    {
      "url": "<string>",
      "status_code": 123,
      "raw_html": "<string>",
      "enhanced_html": "<string>",
      "markdown": "<string>",
      "page_text": "<string>",
      "screenshot": "<string>",
      "error": "<string>"
    }
  ],
  "credits": 123,
  "successful_pages": 123
}

Overview

The Crawler API performs direct crawling of specified URLs with JavaScript rendering support. Ideal for extracting content from single pages or multiple URLs in a single request.

Endpoint

GET https://api.crawleo.dev/crawl

Parameters

Required Headers

string

required

Your Crawleo API key for authentication. (Alternatively, use the Authorization: Bearer YOUR_API_KEY header.)Example: x-api-key: YOUR_API_KEY or Authorization: Bearer YOUR_API_KEY

Required Parameters

string

required

URL(s) to crawl. Can be a single URL or comma-separated list.Example: https://example.com or https://example.com,https://example.org

Rendering Options

boolean

default:"false"

Enable browser rendering for JavaScript-heavy sites.

true - Browser rendering (10 credits/URL)
false - HTTP request (1 credit/URL)

string

ISO 3166-1 alpha-2 country code for geolocation (e.g., us, gb, de).Example: geolocation=us

Output Format Parameters

boolean

default:"false"

Return the raw HTML as-is from the page source.

boolean

default:"true"

Return cleaned/sanitized HTML with ads, scripts, and tracking removed.

boolean

default:"false"

Return plain-text extraction of the page content.

boolean

default:"true"

Return markdown conversion (recommended for RAG/LLM applications).

Screenshot Options

boolean

default:"false"

Capture a screenshot of the page. Only available when render_js=true.

boolean

default:"false"

Capture full-page screenshot vs viewport only.

Screenshots are only available when render_js=true. Setting screenshot=true without JavaScript rendering will be ignored.

Example Requests

Basic Crawl with Markdown Output

curl -X GET "https://api.crawleo.dev/crawl?urls=https://example.com&markdown=true" \
  -H "x-api-key: YOUR_API_KEY"

import requests

response = requests.get(
    "https://api.crawleo.dev/crawl",
    params={
        "urls": "https://example.com",
        "markdown": True
    },
    headers={"x-api-key": "YOUR_API_KEY"}
)

data = response.json()

const response = await fetch(
  "https://api.crawleo.dev/crawl?urls=https://example.com&markdown=true",
  {
    headers: { "x-api-key": "YOUR_API_KEY" }
  }
);

const data = await response.json();

Crawl Multiple URLs

curl -X GET "https://api.crawleo.dev/crawl?urls=https://example.com,https://example.org&markdown=true" \
  -H "x-api-key: YOUR_API_KEY"

import requests

urls = [
    "https://example.com",
    "https://example.org"
]

response = requests.get(
    "https://api.crawleo.dev/crawl",
    params={
        "urls": ",".join(urls),
        "markdown": True
    },
    headers={"x-api-key": "YOUR_API_KEY"}
)

data = response.json()

const urls = [
  "https://example.com",
  "https://example.org"
].join(",");

const response = await fetch(
  `https://api.crawleo.dev/crawl?urls=${encodeURIComponent(urls)}&markdown=true`,
  {
    headers: { "x-api-key": "YOUR_API_KEY" }
  }
);

const data = await response.json();

Crawl with JavaScript Rendering

curl -X GET "https://api.crawleo.dev/crawl?urls=https://example.com&render_js=true&markdown=true" \
  -H "x-api-key: YOUR_API_KEY"

import requests

response = requests.get(
    "https://api.crawleo.dev/crawl",
    params={
        "urls": "https://example.com",
        "render_js": True,
        "markdown": True
    },
    headers={"x-api-key": "YOUR_API_KEY"}
)

data = response.json()

Crawl with Screenshot and Geolocation

curl -X GET "https://api.crawleo.dev/crawl?urls=https://example.com&render_js=true&screenshot=true&screenshot_full_page=true&geolocation=gb" \
  -H "x-api-key: YOUR_API_KEY"

import requests

response = requests.get(
    "https://api.crawleo.dev/crawl",
    params={
        "urls": "https://example.com",
        "render_js": True,
        "screenshot": True,
        "screenshot_full_page": True,
        "geolocation": "gb"
    },
    headers={"x-api-key": "YOUR_API_KEY"}
)

data = response.json()

Get Multiple Output Formats

curl -X GET "https://api.crawleo.dev/crawl?urls=https://example.com&raw_html=true&enhanced_html=true&markdown=true&page_text=true" \
  -H "x-api-key: YOUR_API_KEY"

import requests

response = requests.get(
    "https://api.crawleo.dev/crawl",
    params={
        "urls": "https://example.com",
        "raw_html": True,
        "enhanced_html": True,
        "markdown": True,
        "page_text": True
    },
    headers={"x-api-key": "YOUR_API_KEY"}
)

data = response.json()

Response

A successful response returns crawled content for each URL:

{
  "results": [
    {
      "url": "https://example.com",
      "status_code": 200,
      "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
      "enhanced_html": "<html>...</html>"
    }
  ],
  "credits": 1,
  "successful_pages": 1
}

array

Array of crawl result objects.

Show Result object properties

string

The crawled URL.

integer

HTTP status code of the crawled page.

string

Full HTML source (if raw_html=true).

string

Cleaned/sanitized HTML (if enhanced_html=true).

string

Markdown content (if markdown=true).

string

Plain-text extraction (if page_text=true).

string

Base64-encoded screenshot (if screenshot=true).

string

Error message if the crawl failed for this URL.

integer

Number of credits consumed by this request. Varies based on rendering:

HTTP request (render_js=false): 1 credit per URL
Browser rendering (render_js=true): 10 credits per URL
Failed requests: 0 credits

integer

Number of URLs that were successfully crawled.

Use Cases

RAG Data Ingestion

Crawl documentation pages or knowledge bases and convert to Markdown for vector database ingestion.

# Example: Crawl docs for RAG
response = requests.get(
    "https://api.crawleo.dev/crawl",
    params={
        "urls": "https://docs.example.com/guide,https://docs.example.com/api",
        "markdown": True
    },
    headers={"x-api-key": "YOUR_API_KEY"}
)

for result in response.json()["results"]:
    # Add to vector database
    vector_db.add(result["markdown"], metadata={"url": result["url"]})

Content Extraction

Extract clean content from web pages for analysis or processing.

Web Scraping

Scrape multiple pages in a single API call with JavaScript rendering support.

AI Agent Tools

Provide AI agents with the ability to read and understand web pages.

Tips

For LLM applications, always use markdown=true to get clean, structured content that minimizes token usage. The markdown format is enabled by default.

Ensure you have permission to crawl the target URLs. Respect robots.txt and website terms of service.

Last modified on April 21, 2026

Getting Started

API Overview

Search APIs

Crawling APIs

Core Concepts

MCP Integration

Integrations

Crawler API

Overview

Endpoint

Parameters

Required Headers

Required Parameters

Rendering Options

Output Format Parameters

Screenshot Options

Example Requests

Basic Crawl with Markdown Output

Crawl Multiple URLs

Crawl with JavaScript Rendering

Crawl with Screenshot and Geolocation

Get Multiple Output Formats

Response

Use Cases

Tips

​Overview

​Endpoint

​Parameters

​Required Headers

​Required Parameters

​Rendering Options

​Output Format Parameters

​Screenshot Options

​Example Requests

​Basic Crawl with Markdown Output

​Crawl Multiple URLs

​Crawl with JavaScript Rendering

​Crawl with Screenshot and Geolocation

​Get Multiple Output Formats

​Response

​Use Cases

​Tips

Overview

Endpoint

Parameters

Required Headers

Required Parameters

Rendering Options

Output Format Parameters

Screenshot Options

Example Requests

Basic Crawl with Markdown Output

Crawl Multiple URLs

Crawl with JavaScript Rendering

Crawl with Screenshot and Geolocation

Get Multiple Output Formats

Response

Use Cases

Tips