> ## Documentation Index
> Fetch the complete documentation index at: https://docs.crawleo.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawler API

> Direct URL crawling API with JavaScript rendering, automatic content extraction, multiple output formats (HTML, Markdown, plain text), and intelligent data cleanup. Ideal for web scraping and content extraction.

## Overview

The Crawler API performs direct crawling of specified URLs with JavaScript rendering support. Ideal for extracting content from single pages or multiple URLs in a single request.

## Endpoint

```
GET https://api.crawleo.dev/crawl
```

## Parameters

### Required Headers

<ParamField header="x-api-key" type="string" required>
  Your Crawleo API key for authentication. (Alternatively, use the `Authorization: Bearer YOUR_API_KEY` header.)

  Example: `x-api-key: YOUR_API_KEY` or `Authorization: Bearer YOUR_API_KEY`
</ParamField>

### Required Parameters

<ParamField query="urls" type="string" required>
  URL(s) to crawl. Can be a single URL or comma-separated list.

  Example: `https://example.com` or `https://example.com,https://example.org`
</ParamField>

### Rendering Options

<ParamField query="render_js" type="boolean" default="false">
  Enable browser rendering for JavaScript-heavy sites.

  * `true` - Browser rendering (10 credits/URL)
  * `false` - HTTP request (1 credit/URL)
</ParamField>

<ParamField query="geolocation" type="string">
  ISO 3166-1 alpha-2 country code for geolocation (e.g., `us`, `gb`, `de`).

  Example: `geolocation=us`
</ParamField>

### Output Format Parameters

<ParamField query="raw_html" type="boolean" default="false">
  Return the raw HTML as-is from the page source.
</ParamField>

<ParamField query="enhanced_html" type="boolean" default="true">
  Return cleaned/sanitized HTML with ads, scripts, and tracking removed.
</ParamField>

<ParamField query="page_text" type="boolean" default="false">
  Return plain-text extraction of the page content.
</ParamField>

<ParamField query="markdown" type="boolean" default="true">
  Return markdown conversion (recommended for RAG/LLM applications).
</ParamField>

### Screenshot Options

<ParamField query="screenshot" type="boolean" default="false">
  Capture a screenshot of the page. Only available when `render_js=true`.
</ParamField>

<ParamField query="screenshot_full_page" type="boolean" default="false">
  Capture full-page screenshot vs viewport only.
</ParamField>

<Warning>
  Screenshots are only available when `render_js=true`. Setting `screenshot=true` without JavaScript rendering will be ignored.
</Warning>

## Example Requests

### Basic Crawl with Markdown Output

<CodeGroup>
  ```bash cURL theme={null}
  curl -X GET "https://api.crawleo.dev/crawl?urls=https://example.com&markdown=true" \
    -H "x-api-key: YOUR_API_KEY"
  ```

  ```python Python theme={null}
  import requests

  response = requests.get(
      "https://api.crawleo.dev/crawl",
      params={
          "urls": "https://example.com",
          "markdown": True
      },
      headers={"x-api-key": "YOUR_API_KEY"}
  )

  data = response.json()
  ```

  ```javascript JavaScript theme={null}
  const response = await fetch(
    "https://api.crawleo.dev/crawl?urls=https://example.com&markdown=true",
    {
      headers: { "x-api-key": "YOUR_API_KEY" }
    }
  );

  const data = await response.json();
  ```
</CodeGroup>

### Crawl Multiple URLs

<CodeGroup>
  ```bash cURL theme={null}
  curl -X GET "https://api.crawleo.dev/crawl?urls=https://example.com,https://example.org&markdown=true" \
    -H "x-api-key: YOUR_API_KEY"
  ```

  ```python Python theme={null}
  import requests

  urls = [
      "https://example.com",
      "https://example.org"
  ]

  response = requests.get(
      "https://api.crawleo.dev/crawl",
      params={
          "urls": ",".join(urls),
          "markdown": True
      },
      headers={"x-api-key": "YOUR_API_KEY"}
  )

  data = response.json()
  ```

  ```javascript JavaScript theme={null}
  const urls = [
    "https://example.com",
    "https://example.org"
  ].join(",");

  const response = await fetch(
    `https://api.crawleo.dev/crawl?urls=${encodeURIComponent(urls)}&markdown=true`,
    {
      headers: { "x-api-key": "YOUR_API_KEY" }
    }
  );

  const data = await response.json();
  ```
</CodeGroup>

### Crawl with JavaScript Rendering

<CodeGroup>
  ```bash cURL theme={null}
  curl -X GET "https://api.crawleo.dev/crawl?urls=https://example.com&render_js=true&markdown=true" \
    -H "x-api-key: YOUR_API_KEY"
  ```

  ```python Python theme={null}
  import requests

  response = requests.get(
      "https://api.crawleo.dev/crawl",
      params={
          "urls": "https://example.com",
          "render_js": True,
          "markdown": True
      },
      headers={"x-api-key": "YOUR_API_KEY"}
  )

  data = response.json()
  ```
</CodeGroup>

### Crawl with Screenshot and Geolocation

<CodeGroup>
  ```bash cURL theme={null}
  curl -X GET "https://api.crawleo.dev/crawl?urls=https://example.com&render_js=true&screenshot=true&screenshot_full_page=true&geolocation=gb" \
    -H "x-api-key: YOUR_API_KEY"
  ```

  ```python Python theme={null}
  import requests

  response = requests.get(
      "https://api.crawleo.dev/crawl",
      params={
          "urls": "https://example.com",
          "render_js": True,
          "screenshot": True,
          "screenshot_full_page": True,
          "geolocation": "gb"
      },
      headers={"x-api-key": "YOUR_API_KEY"}
  )

  data = response.json()
  ```
</CodeGroup>

### Get Multiple Output Formats

<CodeGroup>
  ```bash cURL theme={null}
  curl -X GET "https://api.crawleo.dev/crawl?urls=https://example.com&raw_html=true&enhanced_html=true&markdown=true&page_text=true" \
    -H "x-api-key: YOUR_API_KEY"
  ```

  ```python Python theme={null}
  import requests

  response = requests.get(
      "https://api.crawleo.dev/crawl",
      params={
          "urls": "https://example.com",
          "raw_html": True,
          "enhanced_html": True,
          "markdown": True,
          "page_text": True
      },
      headers={"x-api-key": "YOUR_API_KEY"}
  )

  data = response.json()
  ```
</CodeGroup>

## Response

A successful response returns crawled content for each URL:

```json theme={null}
{
  "results": [
    {
      "url": "https://example.com",
      "status_code": 200,
      "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
      "enhanced_html": "<html>...</html>"
    }
  ],
  "credits": 1,
  "successful_pages": 1
}
```

<ResponseField name="results" type="array">
  Array of crawl result objects.

  <Expandable title="Result object properties">
    <ResponseField name="url" type="string">
      The crawled URL.
    </ResponseField>

    <ResponseField name="status_code" type="integer">
      HTTP status code of the crawled page.
    </ResponseField>

    <ResponseField name="raw_html" type="string">
      Full HTML source (if `raw_html=true`).
    </ResponseField>

    <ResponseField name="enhanced_html" type="string">
      Cleaned/sanitized HTML (if `enhanced_html=true`).
    </ResponseField>

    <ResponseField name="markdown" type="string">
      Markdown content (if `markdown=true`).
    </ResponseField>

    <ResponseField name="page_text" type="string">
      Plain-text extraction (if `page_text=true`).
    </ResponseField>

    <ResponseField name="screenshot" type="string">
      Base64-encoded screenshot (if `screenshot=true`).
    </ResponseField>

    <ResponseField name="error" type="string">
      Error message if the crawl failed for this URL.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="credits" type="integer">
  Number of credits consumed by this request. Varies based on rendering:

  * HTTP request (render\_js=false): 1 credit per URL
  * Browser rendering (render\_js=true): 10 credits per URL
  * Failed requests: 0 credits
</ResponseField>

<ResponseField name="successful_pages" type="integer">
  Number of URLs that were successfully crawled.
</ResponseField>

## Use Cases

<AccordionGroup>
  <Accordion title="RAG Data Ingestion" icon="database">
    Crawl documentation pages or knowledge bases and convert to Markdown for vector database ingestion.

    ```python theme={null}
    # Example: Crawl docs for RAG
    response = requests.get(
        "https://api.crawleo.dev/crawl",
        params={
            "urls": "https://docs.example.com/guide,https://docs.example.com/api",
            "markdown": True
        },
        headers={"x-api-key": "YOUR_API_KEY"}
    )

    for result in response.json()["results"]:
        # Add to vector database
        vector_db.add(result["markdown"], metadata={"url": result["url"]})
    ```
  </Accordion>

  <Accordion title="Content Extraction" icon="file-lines">
    Extract clean content from web pages for analysis or processing.
  </Accordion>

  <Accordion title="Web Scraping" icon="spider">
    Scrape multiple pages in a single API call with JavaScript rendering support.
  </Accordion>

  <Accordion title="AI Agent Tools" icon="robot">
    Provide AI agents with the ability to read and understand web pages.
  </Accordion>
</AccordionGroup>

## Tips

<Tip>
  **For LLM applications**, always use `markdown=true` to get clean, structured content that minimizes token usage. The markdown format is enabled by default.
</Tip>

<Warning>
  Ensure you have permission to crawl the target URLs. Respect robots.txt and website terms of service.
</Warning>