A whole website from one request
Send one URL. The crawler follows the site's links and sitemap, renders every page in a real browser, and gives you each one as a screenshot, Markdown, JSON and HTML. Then download the whole site as one file.
Crawls need Starter or a higher plan. One credit per captured page; failed and skipped pages are free.
- discovered
- 1
- completed
- 0
- failed
- 0
- skipped
- 0
- charged
- 0
- queued
- 1
Depth 0 (start URL) · 1 page
Depth 1 · 0 pages
Depth 2 · 0 pages
Simulation over a real crawl of docs.allscreenshots.com (71 pages captured, depth 3, limit 100, finished in 11 min). Pages, links, screenshots and outcomes are from that crawl; the order and limits follow the crawler's rules. Pages the real crawl didn't reach show their path instead of a screenshot.
A replay of a real crawl of our docs: 73 pages found, 71 captured, and 2 broken links caught, in 11 minutes. Change the settings and press Play.
What you get for every page
Pick any mix of four outputs. A page costs one credit whichever you pick.
screenshotScreenshot
The whole page, top to bottom, as WebP, PNG or JPEG. The median page of our crawl was 243 KB.
markdownMarkdown
The page body as Markdown, ready for search or a language model. 7 KB for the median docs page.
jsonJSON page document
Title, description, canonical URL, HTTP status, depth, the page that linked to it, headings, links marked internal or external, visible text and OpenGraph tags.
htmlHTML
The HTML as the browser loaded it, after JavaScript ran. Stored safely: it can’t run scripts when you open it from the API.

It stays where you point it
A crawl stays on the site and in the folder you start in. Patterns narrow it further.
Default
"url": "https://example.com/docs/intro"
- example.com/docs/introfollowed
- example.com/docs/api/authfollowed
- www.example.com/docs/faqfollowed
- example.com/blog/launchnot followed
- help.example.com/docs/startnot followed
entireSite
"entireSite": true
- example.com/docs/introfollowed
- example.com/docs/api/authfollowed
- www.example.com/docs/faqfollowed
- example.com/blog/launchfollowed
- help.example.com/docs/startnot followed
entireSite + includeSubdomains
"entireSite": true, "includeSubdomains": true
- example.com/docs/introfollowed
- example.com/docs/api/authfollowed
- www.example.com/docs/faqfollowed
- example.com/blog/launchfollowed
- help.example.com/docs/startfollowed
https://example.com/docs/intro. By default the crawl stays under /docs/, the folder of the start URL; www. and the bare domain count as the same site. entireSite lifts the folder limit and includeSubdomains adds other subdomains.- Stays on the start site, and in the start folder unless you set entireSite.
- Obeys robots.txt as AllScreenshotsBot, and honours Crawl-delay up to 30 seconds.
- Waits at least 500 ms between requests to a site.
- Reads the sitemap, so pages nothing links to are found too.
- Skips links to PDFs, images, videos and other files that aren’t pages.
- Follows links in the page content before menu and footer links.
Try include and exclude patterns
Patterns match the whole URL. * matches anything, slashes included, and an exclude always wins. These are the URLs of the real crawl.
16 of 73 URLs from the real crawl would be followed.
- /not followed:matches no include
- /api-reference/async-jobsnot followed:matches no include
- /api-reference/batchesnot followed:matches no include
- /api-reference/bulknot followed:matches no include
- /api-reference/composenot followed:matches no include
- /api-reference/crawlsnot followed:matches no include
- /api-reference/outputsnot followed:matches no include
- /api-reference/schedulesnot followed:matches no include
- /api-reference/screenshotsnot followed:matches no include
- /api-reference/webhooksnot followed:matches no include
- /dashboard/domainsnot followed:matches no include
- /features/ai-cleanupnot followed:matches no include
- /features/ai-extractionnot followed:matches no include
- /features/blockingnot followed:matches no include
- /features/full-page-capturenot followed:matches no include
- /features/interactive-actionsnot followed:matches no include
- /features/sessionsnot followed:matches no include
- /features/social-imagesnot followed:matches no include
- /features/stealth-modenot followed:matches no include
- /features/stylingnot followed:matches no include
- /features/viewport-and-devicesnot followed:matches no include
- /getting-started/authenticationnot followed:matches no include
- /getting-started/quickstartnot followed:matches no include
- /getting-started/rate-limitsnot followed:matches no include
- /guides/canvas-exportsfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/canvas-image-processingfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/canvas-webhooksfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/canvas-workflowsfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/ci-cd-integrationfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/cicd-integrationfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/crawl-a-websitefollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/full-page-screenshotsfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/multi-output-extractionfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/openclaw-integrationfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/pdf-generationfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/remove-popups-and-overlaysfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/scheduled-screenshotsfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/social-media-previewsfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/visual-regression-testingfollowed:matches https://docs.allscreenshots.com/guides/*
- /guides/website-monitoringfollowed:matches https://docs.allscreenshots.com/guides/*
- /integrations/n8nnot followed:matches no include
- /llm-txtnot followed:matches no include
- /sdksnot followed:matches no include
- /sdks/csharpnot followed:matches no include
- /sdks/gonot followed:matches no include
- /sdks/javanot followed:matches no include
- /sdks/javascriptnot followed:matches no include
- /sdks/kotlinnot followed:matches no include
- /sdks/phpnot followed:matches no include
- /sdks/pythonnot followed:matches no include
- /sdks/rubynot followed:matches no include
- /sdks/rustnot followed:matches no include
- /sdks/swiftnot followed:matches no include
- /sdks/typescriptnot followed:matches no include
- /supportnot followed:matches no include
- /tools/agent-pluginsnot followed:matches no include
- /tools/canvanot followed:matches no include
- /tools/clinot followed:matches no include
- /tools/figmanot followed:matches no include
- /tools/mcpnot followed:matches no include
- /tools/mkdocsnot followed:matches no include
- /use-casesnot followed:matches no include
- /use-cases/ci-cd-workflowsnot followed:matches no include
- /use-cases/compliance-auditingnot followed:matches no include
- /use-cases/content-archivingnot followed:matches no include
- /use-cases/customer-onboardingnot followed:matches no include
- /use-cases/ecommerce-monitoringnot followed:matches no include
- /use-cases/legal-evidencenot followed:matches no include
- /use-cases/report-generationnot followed:matches no include
- /use-cases/social-media-og-imagesnot followed:matches no include
- /use-cases/structured-extractionnot followed:matches no include
- /use-cases/visual-regression-testingnot followed:matches no include
- /use-cases/website-monitoringnot followed:matches no include
Patterns match the whole URL, including https:// and the host, so a pattern that starts with a path matches nothing. The start URL is always captured; patterns decide which discovered links are followed.
The whole site in one download
When a crawl ends, one call returns every page. Sizes are from the same 71-page crawl.
ZIP
18.2 MBformat=zipA folder per page with its files, plus manifest.json with every page’s URL, title, status, depth and parent.
One Markdown file
535 KBformat=markdownAll 71 pages in crawl order, each headed by its source URL and title. Paste it into a model or a search index.
JSON Lines
1.7 MBformat=jsonlOne line per page, failed pages included, with the Markdown, the JSON document and a screenshot link.
Crawler, bulk, or your own script
Three ways to capture many pages of one site.
| Comparison | Your own sitemap script | Bulk screenshots | Website crawler |
|---|---|---|---|
| Finds the pages | Only what the sitemap lists | You send every URL | Follows links and reads the sitemap |
| Renders JavaScript | If you run a browser | Yes | Yes |
| Per page | What you build | Screenshot, or any outputs | Screenshot, HTML, Markdown, JSON |
| Page tree | No | No | Depth and parent for every page |
| robots.txt and crawl delay | You implement them | Not applied: you pick the URLs | Obeyed |
| Whole-site download | You build it | Per URL | ZIP, Markdown or JSON Lines |
| Pages per request | Any | 100 | 100 |
| Plans | — | Every plan | Starter and higher |
Use bulk when you already have the list of URLs, or need PDFs, logged-in pages or device presets. Use the crawler when the site should tell you which pages exist.
Three calls
Start the crawl, wait for it, download the result.
1. Start a crawl
curl -X POST https://api.allscreenshots.com/v1/crawls \
-H "X-API-Key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/docs/",
"depth": 2,
"limit": 25,
"outputs": ["screenshot", "markdown", "json"]
}'2. Check progress, or set a webhookUrl
curl https://api.allscreenshots.com/v1/crawls/CRAWL_ID \
-H "X-API-Key: YOUR_API_KEY"3. Download every page
curl "https://api.allscreenshots.com/v1/crawls/CRAWL_ID/export?format=markdown" \
-H "X-API-Key: YOUR_API_KEY" --output site.mdQuestions
What does a crawl cost?
One screenshot credit per captured page, whichever outputs you pick. Pages that fail, are skipped, or show a bot check or an error page are not charged. The crawler is included on Starter and every higher plan.
How many pages can one crawl capture?
Up to 100 pages and 5 links deep. Two crawls run at once per organization; more wait in a queue. Split larger sites into sections, such as one crawl for /docs/ and one for /blog/.
Which pages does it crawl?
It starts at your URL, reads the site’s sitemap and follows links on each rendered page. It stays on the same site and, unless you set entireSite, under the folder you start in. Include and exclude patterns narrow it further.
Does it respect robots.txt?
Yes. It identifies as AllScreenshotsBot, skips disallowed URLs, honours Crawl-delay up to 30 seconds, and waits at least 500 ms between requests to a site.
How do I know when a crawl has finished?
Poll GET /v1/crawls/{id}, which shows progress counters, or add a webhookUrl and get one signed call when the crawl ends.
How long are the results kept?
Screenshots and other outputs are kept for 7 days. The crawl’s page list stays until you delete it.
Can it log in, click or use a mobile device preset?
No. Crawls don’t support sessions, actions, device presets, PDF output or stealth mode. Set a viewport for a mobile-width crawl, or send the URLs you need to the screenshot API, which supports all of those.
Is there an SDK or MCP tool for crawls?
Not yet. Crawls are plain JSON over HTTP, so any HTTP client works, and the dashboard has a Crawler view with a site atlas.
Crawl your own docs first
Our first crawl of our own docs found two broken links. Point the crawler at a site you know and see what it finds.