> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mindcase.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Crawl API

> Crawl a website from one start URL and get every page it links to as clean Markdown, with title, description and link depth. 50 pages by default, up to 1,000. Only pages that came back are charged.

`POST /v1/data/web/crawl/run`

**Pricing:** $0.003 per page ($3.00 per 1,000).

## Request

Send the inputs under `params`. Only the keys below are accepted; any other key returns `422 invalid_params`.

**url** `string` *required*

The page to start from, like [https://docs.example.com](https://docs.example.com). Links are followed from here.

Format e.g. `https://www.example.com`.

**maxPages** `integer` (default: 50)

How many pages to crawl. Defaults to 50; at most 1,000.

Min `1`; max `1000`.

**includePaths** `string[]`

Only crawl paths that match, like /blog/\* or /docs/\*.

**excludePaths** `string[]`

Skip paths that match, like /tag/\* or /author/\*.

## Example

```bash cURL theme={null}
curl -X POST "https://api.mindcase.co/v1/data/web/crawl/run?wait=true" \
  -H "Authorization: Bearer $MINDCASE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "params": {
    "url": "https://docs.firecrawl.dev",
    "maxPages": 10
  }
}'
```

```python Python theme={null}
import os, requests

r = requests.post(
    "https://api.mindcase.co/v1/data/web/crawl/run",
    params={"wait": "true"},
    headers={"Authorization": f"Bearer {os.environ['MINDCASE_API_KEY']}"},
    json={"params": {
    "url": "https://docs.firecrawl.dev",
    "maxPages": 10
}},
)
job = r.json()
rows = job.get("data", [])  # if job["status"] == "running", poll /v1/jobs/{job_id}/results
```

```javascript Node.js theme={null}
const r = await fetch("https://api.mindcase.co/v1/data/web/crawl/run?wait=true", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.MINDCASE_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ params: {
  "url": "https://docs.firecrawl.dev",
  "maxPages": 10
} }),
});
const job = await r.json();
const rows = job.data ?? []; // if job.status === "running", poll /v1/jobs/{job_id}/results
```

## Response fields

Each row in `data` is one page, keyed by these fields.

| Field | Name | Type |
| - | - | - |
| `url` | URL | text |
| `title` | Title | text |
| `description` | Description | text |
| `markdown` | Markdown | text |
| `wordCount` | Word Count | number |
| `depth` | Depth | integer |
| `statusCode` | Status Code | integer |
| `fetchedAt` | Fetched At | text |
| `engine` | Engine | text |
| `servedFromCache` | Served From Cache | boolean |

## Example response

Abridged: long text and lists are shortened.

```json theme={null}
{
  "job_id": "7f3a1c2e-0b9d-4c55-9a51-3f2e8d1b6a40",
  "agent": "web/crawl",
  "status": "completed",
  "row_count": 1,
  "truncated": false,
  "data": [
    {
      "depth": 2,
      "engine": "crawler",
      "wordCount": 50895,
      "markdown": "> ## Documentation Index\n>\n> Fetch the complete documentation index at: [/llms.txt](https://docs.firecrawl.dev/llms.txt)\n>\n> Use this file to discover all av...",
      "description": "Turn any URL into clean Markdown or structured data",
      "url": "https://docs.firecrawl.dev/features/scrape",
      "statusCode": 200,
      "title": "Scrape | Firecrawl",
      "fetchedAt": "2026-11-09T10:28:08Z",
      "servedFromCache": true
    }
  ]
}
```

## Related APIs

For one page use [`web/scrape`](https://docs.mindcase.co/apis/web/scrape); for just the URLs, up to 5,000, much cheaper use [`web/map`](https://docs.mindcase.co/apis/web/map); for named fields as JSON use [`web/extract`](https://docs.mindcase.co/apis/web/extract).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.