Generally availableOn Starter and higher plans

A whole website from one request

Send one URL. The crawler follows the site's links and sitemap, renders every page in a real browser, and gives you each one as a screenshot, Markdown, JSON and HTML. Then download the whole site as one file.

Crawls need Starter or a higher plan. One credit per captured page; failed and skipped pages are free.

sitemap

Links on pages plus the sitemap

discovered
1
completed
0
failed
0
skipped
0
charged
0
queued
1

Depth 0 (start URL) · 1 page

Depth 1 · 0 pages

    Depth 2 · 0 pages

      CapturedQueuedSkipped (robots.txt, duplicate)FailedSkipped: LIMIT_REACHED

      Simulation over a real crawl of docs.allscreenshots.com (71 pages captured, depth 3, limit 100, finished in 11 min). Pages, links, screenshots and outcomes are from that crawl; the order and limits follow the crawler's rules. Pages the real crawl didn't reach show their path instead of a screenshot.

      A replay of a real crawl of our docs: 73 pages found, 71 captured, and 2 broken links caught, in 11 minutes. Change the settings and press Play.

      What you get for every page

      Pick any mix of four outputs. A page costs one credit whichever you pick.

      screenshot

      Screenshot

      The whole page, top to bottom, as WebP, PNG or JPEG. The median page of our crawl was 243 KB.

      markdown

      Markdown

      The page body as Markdown, ready for search or a language model. 7 KB for the median docs page.

      json

      JSON page document

      Title, description, canonical URL, HTTP status, depth, the page that linked to it, headings, links marked internal or external, visible text and OpenGraph tags.

      html

      HTML

      The HTML as the browser loaded it, after JavaScript ran. Stored safely: it can’t run scripts when you open it from the API.

      A grid of 24 page screenshots from a crawl of the Allscreenshots documentation.
      The first 24 of 71 pages from the real crawl. Median render time: 9.3 seconds a page.

      It stays where you point it

      A crawl stays on the site and in the folder you start in. Patterns narrow it further.

      Default

      "url": "https://example.com/docs/intro"

      • example.com/docs/introfollowed
      • example.com/docs/api/authfollowed
      • www.example.com/docs/faqfollowed
      • example.com/blog/launchnot followed
      • help.example.com/docs/startnot followed

      entireSite

      "entireSite": true

      • example.com/docs/introfollowed
      • example.com/docs/api/authfollowed
      • www.example.com/docs/faqfollowed
      • example.com/blog/launchfollowed
      • help.example.com/docs/startnot followed

      entireSite + includeSubdomains

      "entireSite": true, "includeSubdomains": true

      • example.com/docs/introfollowed
      • example.com/docs/api/authfollowed
      • www.example.com/docs/faqfollowed
      • example.com/blog/launchfollowed
      • help.example.com/docs/startfollowed
      Starting at https://example.com/docs/intro. By default the crawl stays under /docs/, the folder of the start URL; www. and the bare domain count as the same site. entireSite lifts the folder limit and includeSubdomains adds other subdomains.
      • Stays on the start site, and in the start folder unless you set entireSite.
      • Obeys robots.txt as AllScreenshotsBot, and honours Crawl-delay up to 30 seconds.
      • Waits at least 500 ms between requests to a site.
      • Reads the sitemap, so pages nothing links to are found too.
      • Skips links to PDFs, images, videos and other files that aren’t pages.
      • Follows links in the page content before menu and footer links.

      Try include and exclude patterns

      Patterns match the whole URL. * matches anything, slashes included, and an exclude always wins. These are the URLs of the real crawl.

      Try

      16 of 73 URLs from the real crawl would be followed.

      • /not followed:
      • /api-reference/async-jobsnot followed:
      • /api-reference/batchesnot followed:
      • /api-reference/bulknot followed:
      • /api-reference/composenot followed:
      • /api-reference/crawlsnot followed:
      • /api-reference/outputsnot followed:
      • /api-reference/schedulesnot followed:
      • /api-reference/screenshotsnot followed:
      • /api-reference/webhooksnot followed:
      • /dashboard/domainsnot followed:
      • /features/ai-cleanupnot followed:
      • /features/ai-extractionnot followed:
      • /features/blockingnot followed:
      • /features/full-page-capturenot followed:
      • /features/interactive-actionsnot followed:
      • /features/sessionsnot followed:
      • /features/social-imagesnot followed:
      • /features/stealth-modenot followed:
      • /features/stylingnot followed:
      • /features/viewport-and-devicesnot followed:
      • /getting-started/authenticationnot followed:
      • /getting-started/quickstartnot followed:
      • /getting-started/rate-limitsnot followed:
      • /guides/canvas-exportsfollowed:
      • /guides/canvas-image-processingfollowed:
      • /guides/canvas-webhooksfollowed:
      • /guides/canvas-workflowsfollowed:
      • /guides/ci-cd-integrationfollowed:
      • /guides/cicd-integrationfollowed:
      • /guides/crawl-a-websitefollowed:
      • /guides/full-page-screenshotsfollowed:
      • /guides/multi-output-extractionfollowed:
      • /guides/openclaw-integrationfollowed:
      • /guides/pdf-generationfollowed:
      • /guides/remove-popups-and-overlaysfollowed:
      • /guides/scheduled-screenshotsfollowed:
      • /guides/social-media-previewsfollowed:
      • /guides/visual-regression-testingfollowed:
      • /guides/website-monitoringfollowed:
      • /integrations/n8nnot followed:
      • /llm-txtnot followed:
      • /sdksnot followed:
      • /sdks/csharpnot followed:
      • /sdks/gonot followed:
      • /sdks/javanot followed:
      • /sdks/javascriptnot followed:
      • /sdks/kotlinnot followed:
      • /sdks/phpnot followed:
      • /sdks/pythonnot followed:
      • /sdks/rubynot followed:
      • /sdks/rustnot followed:
      • /sdks/swiftnot followed:
      • /sdks/typescriptnot followed:
      • /supportnot followed:
      • /tools/agent-pluginsnot followed:
      • /tools/canvanot followed:
      • /tools/clinot followed:
      • /tools/figmanot followed:
      • /tools/mcpnot followed:
      • /tools/mkdocsnot followed:
      • /use-casesnot followed:
      • /use-cases/ci-cd-workflowsnot followed:
      • /use-cases/compliance-auditingnot followed:
      • /use-cases/content-archivingnot followed:
      • /use-cases/customer-onboardingnot followed:
      • /use-cases/ecommerce-monitoringnot followed:
      • /use-cases/legal-evidencenot followed:
      • /use-cases/report-generationnot followed:
      • /use-cases/social-media-og-imagesnot followed:
      • /use-cases/structured-extractionnot followed:
      • /use-cases/visual-regression-testingnot followed:
      • /use-cases/website-monitoringnot followed:

      Patterns match the whole URL, including https:// and the host, so a pattern that starts with a path matches nothing. The start URL is always captured; patterns decide which discovered links are followed.

      The whole site in one download

      When a crawl ends, one call returns every page. Sizes are from the same 71-page crawl.

      ZIP

      18.2 MB
      format=zip

      A folder per page with its files, plus manifest.json with every page’s URL, title, status, depth and parent.

      One Markdown file

      535 KB
      format=markdown

      All 71 pages in crawl order, each headed by its source URL and title. Paste it into a model or a search index.

      JSON Lines

      1.7 MB
      format=jsonl

      One line per page, failed pages included, with the Markdown, the JSON document and a screenshot link.

      Crawler, bulk, or your own script

      Three ways to capture many pages of one site.

      ComparisonYour own sitemap scriptBulk screenshotsWebsite crawler
      Finds the pagesOnly what the sitemap listsYou send every URLFollows links and reads the sitemap
      Renders JavaScriptIf you run a browserYesYes
      Per pageWhat you buildScreenshot, or any outputsScreenshot, HTML, Markdown, JSON
      Page treeNoNoDepth and parent for every page
      robots.txt and crawl delayYou implement themNot applied: you pick the URLsObeyed
      Whole-site downloadYou build itPer URLZIP, Markdown or JSON Lines
      Pages per requestAny100100
      Plans—Every planStarter and higher

      Use bulk when you already have the list of URLs, or need PDFs, logged-in pages or device presets. Use the crawler when the site should tell you which pages exist.

      Three calls

      Start the crawl, wait for it, download the result.

      1. Start a crawl

      curl -X POST https://api.allscreenshots.com/v1/crawls \ -H "X-API-Key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com/docs/", "depth": 2, "limit": 25, "outputs": ["screenshot", "markdown", "json"] }'

      2. Check progress, or set a webhookUrl

      curl https://api.allscreenshots.com/v1/crawls/CRAWL_ID \ -H "X-API-Key: YOUR_API_KEY"

      3. Download every page

      curl "https://api.allscreenshots.com/v1/crawls/CRAWL_ID/export?format=markdown" \ -H "X-API-Key: YOUR_API_KEY" --output site.md

      Questions

      What does a crawl cost?

      One screenshot credit per captured page, whichever outputs you pick. Pages that fail, are skipped, or show a bot check or an error page are not charged. The crawler is included on Starter and every higher plan.

      How many pages can one crawl capture?

      Up to 100 pages and 5 links deep. Two crawls run at once per organization; more wait in a queue. Split larger sites into sections, such as one crawl for /docs/ and one for /blog/.

      Which pages does it crawl?

      It starts at your URL, reads the site’s sitemap and follows links on each rendered page. It stays on the same site and, unless you set entireSite, under the folder you start in. Include and exclude patterns narrow it further.

      Does it respect robots.txt?

      Yes. It identifies as AllScreenshotsBot, skips disallowed URLs, honours Crawl-delay up to 30 seconds, and waits at least 500 ms between requests to a site.

      How do I know when a crawl has finished?

      Poll GET /v1/crawls/{id}, which shows progress counters, or add a webhookUrl and get one signed call when the crawl ends.

      How long are the results kept?

      Screenshots and other outputs are kept for 7 days. The crawl’s page list stays until you delete it.

      Can it log in, click or use a mobile device preset?

      No. Crawls don’t support sessions, actions, device presets, PDF output or stealth mode. Set a viewport for a mobile-width crawl, or send the URLs you need to the screenshot API, which supports all of those.

      Is there an SDK or MCP tool for crawls?

      Not yet. Crawls are plain JSON over HTTP, so any HTTP client works, and the dashboard has a Crawler view with a site atlas.

      Crawl your own docs first

      Our first crawl of our own docs found two broken links. Point the crawler at a site you know and see what it finds.