Overview
Crawler.sh is a desktop app and command-line tool that lets you crawl entire websites, analyze SEO health, and export structured data — all from your own machine, with no account required to get started.
It targets developers, SEO professionals, and content teams who need a fast, local-first solution for site audits, sitemap generation, and bulk content extraction. Unlike cloud-based AI web scraping services, Crawler.sh runs entirely on your hardware, keeping your data private and eliminating per-crawl fees.
The tool ships as two interfaces: a CLI with four subcommands for scripting and automation, and a visual desktop app with a live feed dashboard for interactive analysis. The CLI outputs NDJSON stream, JSON array, W3C-compliant Sitemap XML, CSV, and TXT; the desktop app exports JSON archive and Sitemap XML, with Content Archive (Markdown) available on the Pro plan.
Key Features
Full-site crawler — Crawls any website following only same-host links (cross-domain URLs are discovered but not followed). Configurable concurrency, depth limits, and polite request delays let you crawl up to 1,000 pages per crawl (600 on the free tier) without triggering rate limits.
16 automated SEO checks — Detects missing titles, duplicate meta descriptions, noindex directives, thin content, long URLs, non-self canonicals, and more across every crawled page. Export issues as CSV or human-readable TXT for your team.
Clean Markdown extraction — Automatically strips navigation, ads, and boilerplate from each page and converts the main article content to clean Markdown with word count, author byline, and excerpt included.
Multiple export formats — CLI streams results as NDJSON during the crawl for real-time piping, then exports to JSON array, Sitemap XML, CSV, and TXT. The desktop app exports JSON archive and Sitemap XML; Content Archive (Markdown) is a Pro-only export.
CLI with four subcommands —
crawlkicks off a crawl,infoinspects saved.crawlfiles,exportconverts to JSON or Sitemap XML, andseoruns the full AI SEO analysis with CSV/TXT output. Ideal for automation scripts and CI pipelines.Visual desktop dashboard — Eight interactive cards (Live Feed, SEO Issues, Page Status, Settings, Downloads, Content Viewer, Newsletter, Premium) give you a real-time view of the crawl without leaving the app.
Pricing & Plans
Crawler.sh is a freemium tool. Pricing is publicly listed on the download page.
| Plan | Price | Page limit per crawl | Highlights |
|---|---|---|---|
| Free | $0 | 600 pages | Full CLI, desktop app, all export formats, 16 SEO checks |
| Pro | $99/year | 1,000 pages | Everything in Free + Content Archive export |
The free tier includes the complete crawling engine, content extraction, SEO analysis, and all standard export formats — with a 600-page limit per crawl. Upgrading to Pro ($99/year) raises the per-crawl limit to 1,000 pages and unlocks Content Archive export. Both tiers are available without creating an account for the local app. An upgrade prompt and purchase button are accessible directly from the Premium card inside the desktop dashboard.
Platform availability:
- macOS: Universal binary (Apple Silicon + Intel) — available now
- Linux: x86_64 & ARM64 (.deb package) — available now; AppImage coming soon
- Windows: Coming soon (CLI and desktop app — in progress on the roadmap)
CLI installation (macOS/Linux):
curl -fsSL https://install.crawler.sh | sh
Best For
- SEO professionals who need a fast, offline site auditor without paying per-crawl fees
- Developers building content pipelines who want clean Markdown output from any URL
- DevOps and automation engineers who prefer a CLI-first tool that integrates with AI agent workflows, shell scripts, and CI
- Content teams performing site migrations or audits on large websites (thousands of pages)
- Webmasters who need accurate, up-to-date Sitemap XML generated from a live crawl
FAQ
Is Crawler.sh completely free?
The core product — CLI and desktop app — is free to download with no account required. The free tier includes all features (crawling, 16 SEO checks, content extraction, all export formats) with a limit of 600 pages per crawl. Upgrading to Pro ($99/year) raises the limit to 1,000 pages per crawl and adds Content Archive export.
Does Crawler.sh require an account to use?
No. The local CLI and desktop app can be downloaded and used immediately without creating an account or providing an email address. A cloud version is mentioned in the roadmap and may require an account when it launches.
What platforms does Crawler.sh support?
macOS (Apple Silicon and Intel) and Linux (x86_64 and ARM64 via .deb) are currently supported. The CLI install script (curl -fsSL https://install.crawler.sh | sh) works on macOS and Linux. Windows support — both the CLI and desktop app — is listed as in progress on the roadmap.
Can Crawler.sh handle JavaScript-rendered pages?
Official documentation does not mention headless browser or JavaScript rendering capabilities. For sites that rely heavily on client-side rendering (SPAs built with React, Vue, etc.), crawled HTML may be incomplete — you can verify by checking whether the raw HTML contains actual content or just empty containers. For JS-heavy sites, consider pre-rendering pages with a tool like Playwright and feeding the output into Crawler.sh for analysis.
What output formats does Crawler.sh support?
The CLI exports in five formats: NDJSON (streamed in real time during the crawl), JSON array (full results), W3C-compliant Sitemap XML, CSV, and plain TXT. The desktop app exports JSON archive and Sitemap XML; Content Archive (clean Markdown of all crawled pages) is available on the Pro plan.
How does the SEO analysis work?
The SEO engine runs 16 automated checks on every crawled page, including detection of missing title tags, duplicate meta descriptions, noindex directives, thin content, excessively long URLs, and non-self canonicals. Results can be exported as a CSV (for spreadsheet analysis) or a human-readable TXT file.
Is there a page limit on how much I can crawl?
Yes. The free tier allows up to 600 pages per crawl; the Pro plan ($99/year) raises this to 1,000 pages per crawl. Within those limits, you can also configure a lower Max Pages value in the settings to control crawl scope. Performance also depends on your local machine and the target server's response speed.
How does Crawler.sh compare to cloud-based SEO crawlers?
Unlike cloud tools such as Screaming Frog (desktop, paid) or browser-based audit platforms, Crawler.sh is fully local, free at its core, and outputs developer-friendly formats like NDJSON and Markdown. It trades cloud collaboration features for privacy, speed, and zero per-crawl cost — making it well-suited for individual developers and technical SEO professionals who prefer AI keyword for SEO research and audit workflows that stay on their own hardware.
