> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mindcase.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Scraping API

> Scrape any web page or PDF live into clean Markdown with every link kept, plus title, description, author, published date, word count and the cleaned HTML. Pages that cannot be scraped come back with a typed error and are not charged. Scrape a list of URLs, or search by keyword and scrape the top results.

`POST /v1/data/web/scrape/run`

**Pricing:** $0.0025 per page ($2.50 per 1,000).

## Ways to call it

* **By URL**: set `urls`
* **Search**: set `query` (optional: `maxResults`)

## Request

Send the inputs under `params`. Only the keys below are accepted; any other key returns `422 invalid_params`.

**urls** `string[]`

Web pages or PDF links to scrape, up to 1,000 per request. Failed pages are returned with an error and not charged.

**query** `string`

Keyword to search the web for. The top results are fetched and converted to Markdown.

**maxResults** `integer` (default: 10)

How many search results to scrape. Defaults to 10; at most 20.

Max `20`.

## Example

```bash cURL theme={null}
curl -X POST "https://api.mindcase.co/v1/data/web/scrape/run?wait=true" \
  -H "Authorization: Bearer $MINDCASE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "params": {
    "urls": [
      "https://www.python.org/this-page-does-not-exist-h2h",
      "https://news.ycombinator.com/",
      "https://quotes.toscrape.com/js/"
    ]
  }
}'
```

```python Python theme={null}
import os, requests

r = requests.post(
    "https://api.mindcase.co/v1/data/web/scrape/run",
    params={"wait": "true"},
    headers={"Authorization": f"Bearer {os.environ['MINDCASE_API_KEY']}"},
    json={"params": {
    "urls": [
        "https://www.python.org/this-page-does-not-exist-h2h",
        "https://news.ycombinator.com/",
        "https://quotes.toscrape.com/js/"
    ]
}},
)
job = r.json()
rows = job.get("data", [])  # if job["status"] == "running", poll /v1/jobs/{job_id}/results
```

```javascript Node.js theme={null}
const r = await fetch("https://api.mindcase.co/v1/data/web/scrape/run?wait=true", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.MINDCASE_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ params: {
  "urls": [
    "https://www.python.org/this-page-does-not-exist-h2h",
    "https://news.ycombinator.com/",
    "https://quotes.toscrape.com/js/"
  ]
} }),
});
const job = await r.json();
const rows = job.data ?? []; // if job.status === "running", poll /v1/jobs/{job_id}/results
```

## Response fields

Each row in `data` is one page, keyed by these fields.

| Field | Name | Type |
| - | - | - |
| `url` | URL | text |
| `query` | Query | text |
| `rank` | Rank | number |
| `finalUrl` | Final URL | text |
| `statusCode` | Status Code | integer |
| `fetchStatus` | Fetch Status | text |
| `title` | Title | text |
| `domain` | Domain | text |
| `description` | Description | text |
| `snippet` | Snippet | text |
| `author` | Author | text |
| `publishedDate` | Published Date | text |
| `language` | Language | text |
| `wordCount` | Word Count | number |
| `markdown` | Markdown | text |
| `html` | Cleaned HTML | text |
| `htmlLength` | Html Length | integer |
| `meta` | Meta | object |
| `fetchedAt` | Fetched At | text |
| `engine` | Engine | text |
| `errorType` | Error Type | text |
| `errorMessage` | Error Message | text |

## Example response

Abridged: long text and lists are shortened.

```json theme={null}
{
  "job_id": "7f3a1c2e-0b9d-4c55-9a51-3f2e8d1b6a40",
  "agent": "web/scrape",
  "status": "completed",
  "row_count": 1,
  "truncated": false,
  "data": [
    {
      "errorType": "not_found",
      "author": null,
      "errorMessage": "The page does not exist (the site answered 404 or 410). Check the URL. Not charged.",
      "engine": null,
      "wordCount": null,
      "meta": null,
      "finalUrl": null,
      "htmlLength": null,
      "markdown": null,
      "rank": null,
      "language": null,
      "domain": "python.org",
      "description": null,
      "url": "https://www.python.org/this-page-does-not-exist-h2h",
      "snippet": null,
      "html": null,
      "query": null,
      "publishedDate": null,
      "statusCode": 404,
      "title": null,
      "fetchedAt": "2026-11-09T10:29:20Z",
      "fetchStatus": "failed"
    }
  ]
}
```

## Related APIs

For a whole site rather than one page use [`web/crawl`](https://docs.mindcase.co/apis/web/crawl); for a site's URL list without page content use [`web/map`](https://docs.mindcase.co/apis/web/map); for named fields as JSON use [`web/extract`](https://docs.mindcase.co/apis/web/extract).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.