Skip to main content
GET
https://api.crawleo.dev
/
crawl
Crawler API
curl --request GET \
  --url https://api.crawleo.dev/crawl \
  --header 'x-api-key: <x-api-key>'
{
  "results": [
    {
      "url": "<string>",
      "status_code": 123,
      "raw_html": "<string>",
      "enhanced_html": "<string>",
      "markdown": "<string>",
      "page_text": "<string>",
      "screenshot": "<string>",
      "error": "<string>"
    }
  ],
  "credits": 123,
  "successful_pages": 123
}

​
Overview

The Crawler API performs direct crawling of specified URLs with JavaScript rendering support. Ideal for extracting content from single pages or multiple URLs in a single request.

​
Endpoint

GET https://api.crawleo.dev/crawl

​
Parameters

​
Required Headers

​
x-api-key
string
required
Your Crawleo API key for authentication. (Alternatively, use the Authorization: Bearer YOUR_API_KEY header.)Example: x-api-key: YOUR_API_KEY or Authorization: Bearer YOUR_API_KEY

​
Required Parameters

​
urls
string
required
URL(s) to crawl. Can be a single URL or comma-separated list.Example: https://example.com or https://example.com,https://example.org

​
Rendering Options

​
render_js
boolean
default:"false"
Enable browser rendering for JavaScript-heavy sites.
  • true - Browser rendering (10 credits/URL)
  • false - HTTP request (1 credit/URL)
​
geolocation
string
ISO 3166-1 alpha-2 country code for geolocation (e.g., us, gb, de).Example: geolocation=us

​
Output Format Parameters

​
raw_html
boolean
default:"false"
Return the raw HTML as-is from the page source.
​
enhanced_html
boolean
default:"true"
Return cleaned/sanitized HTML with ads, scripts, and tracking removed.
​
page_text
boolean
default:"false"
Return plain-text extraction of the page content.
​
markdown
boolean
default:"true"
Return markdown conversion (recommended for RAG/LLM applications).

​
Screenshot Options

​
screenshot
boolean
default:"false"
Capture a screenshot of the page. Only available when render_js=true.
​
screenshot_full_page
boolean
default:"false"
Capture full-page screenshot vs viewport only.
Screenshots are only available when render_js=true. Setting screenshot=true without JavaScript rendering will be ignored.

​
Example Requests

​
Basic Crawl with Markdown Output

curl -X GET "https://api.crawleo.dev/crawl?urls=https://example.com&markdown=true" \
  -H "x-api-key: YOUR_API_KEY"

​
Crawl Multiple URLs

curl -X GET "https://api.crawleo.dev/crawl?urls=https://example.com,https://example.org&markdown=true" \
  -H "x-api-key: YOUR_API_KEY"

​
Crawl with JavaScript Rendering

curl -X GET "https://api.crawleo.dev/crawl?urls=https://example.com&render_js=true&markdown=true" \
  -H "x-api-key: YOUR_API_KEY"

​
Crawl with Screenshot and Geolocation

curl -X GET "https://api.crawleo.dev/crawl?urls=https://example.com&render_js=true&screenshot=true&screenshot_full_page=true&geolocation=gb" \
  -H "x-api-key: YOUR_API_KEY"

​
Get Multiple Output Formats

curl -X GET "https://api.crawleo.dev/crawl?urls=https://example.com&raw_html=true&enhanced_html=true&markdown=true&page_text=true" \
  -H "x-api-key: YOUR_API_KEY"

​
Response

A successful response returns crawled content for each URL: 
{
  "results": [
    {
      "url": "https://example.com",
      "status_code": 200,
      "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
      "enhanced_html": "<html>...</html>"
    }
  ],
  "credits": 1,
  "successful_pages": 1
}
​
results
array
Array of crawl result objects.
​
credits
integer
Number of credits consumed by this request. Varies based on rendering:
  • HTTP request (render_js=false): 1 credit per URL
  • Browser rendering (render_js=true): 10 credits per URL
  • Failed requests: 0 credits
​
successful_pages
integer
Number of URLs that were successfully crawled.

​
Use Cases

Crawl documentation pages or knowledge bases and convert to Markdown for vector database ingestion.
# Example: Crawl docs for RAG
response = requests.get(
    "https://api.crawleo.dev/crawl",
    params={
        "urls": "https://docs.example.com/guide,https://docs.example.com/api",
        "markdown": True
    },
    headers={"x-api-key": "YOUR_API_KEY"}
)

for result in response.json()["results"]:
    # Add to vector database
    vector_db.add(result["markdown"], metadata={"url": result["url"]})
Extract clean content from web pages for analysis or processing.
Scrape multiple pages in a single API call with JavaScript rendering support.
Provide AI agents with the ability to read and understand web pages.

​
Tips

For LLM applications, always use markdown=true to get clean, structured content that minimizes token usage. The markdown format is enabled by default.
Ensure you have permission to crawl the target URLs. Respect robots.txt and website terms of service.
Last modified on April 21, 2026