Skip to main content
POST /v1/data/web/crawl/run Pricing: $0.003 per page ($3.00 per 1,000).

Request

Send the inputs under params. Only the keys below are accepted; any other key returns 422 invalid_params.
string
required
The page to start from, like https://docs.example.com. Links are followed from here.Format e.g. https://www.example.com.
integer
default:"50"
How many pages to crawl. Defaults to 50; at most 1,000.Min 1; max 1000.
string[]
Only crawl paths that match, like /blog/* or /docs/*.
string[]
Skip paths that match, like /tag/* or /author/*.

Example

Response fields

Each row in data is one page, keyed by these fields.

Example response

Abridged: long text and lists are shortened.
For one page use web/scrape; for just the URLs, up to 5,000, much cheaper use web/map; for named fields as JSON use web/extract.