# How to Vibe Code Your Own Sitebulb (and Stop Paying for It)

> Website Crawler for Better SEO Audits

- Site: https://sitebulb.com
- Category: SEO & Marketing SaaS
- Verdict: **Serious undertaking** (38/100 vibecodeable)
- Estimated effort: 3-6 months of part-time development

## Verdict

Build a scaled-down web scraping dashboard instead, because replicating Sitebulb's multi-threaded crawler, Chromium rendering cluster, and 300+ SEO hint rules requires months of low-level systems engineering.

Sitebulb is an engineering masterpiece disguised as an SEO tool. The core challenge is not the UI dashboard; it's writing a fault-tolerant asynchronous web crawler in a language like Go or Rust that handles circular redirects, malformed HTML, and rate limits, coupled with a headless Chromium rendering pool that does not exhaust system memory. Storing half a million crawled pages with all outgoing links, response headers, and asset trees requires robust local relational storage and heavy indexing. While an AI agent can scaffold the frontend and basic scraper scripts, debugging edge cases like client-side DOM hydration failures, connection timeouts, and memory management across massive link graphs will consume weeks of frustrating debugging.

### What you can't replicate

- The exact depth and edge-case tuning of 300+ specialized technical SEO rules
- Seamless local-to-cloud transition handling millions of link graph nodes smoothly
- Years of battle-tested robustness against hostile or malformed target websites

## What it does

A technical SEO auditing platform that crawls websites to discover, analyze, and prioritize search engine optimization and security issues across 300+ automated hints.

### Core features

- Multi-threaded asynchronous web crawler respecting robots.txt
- Headless Evergreen Chromium rendering engine for JavaScript-heavy sites
- Local embedded relational database storing billions of link graph nodes
- Over 300 automated technical SEO and security Hint rules
- Audit comparison engine tracking metric deltas over time
- Customizable PDF report generator and CSV/Sheets exports

## The business

### Pricing

- Lite (Desktop): $18/mo — For freelancers and small audits up to 10,000 URLs.
- Pro (Desktop): $42/mo — For SEO professionals and agencies.
- Cloud Mini: $125/mo — Cloud-hosted crawls for teams.

Founded 2017.
Team size: Small boutique team.

## The hard parts

- Building a performant crawler engine that processes millions of pages without memory leaks or getting banned
- Scaling headless browser rendering to evaluate client-side frameworks without crashing local RAM
- Designing a local database schema and indexing strategy capable of sub-second relational graph queries across 500,000+ nodes
- Implementing 300+ deterministic SEO and security rule evaluations accurately against raw DOM trees

## How to vibe code Sitebulb

### Prerequisites

- Node.js (free): Required for running the full-stack web application interface and backend crawler runner.
- GitHub (free): Version control and repository management for your codebase.

### Recommended AI tools

- Claude Code: Best-in-class terminal coding agent for scaffolding complex backend crawling loops, database schemas, and multi-file codebases.
- Cursor: Ideal for inspecting complex data table components, charts, and report layouts via inline diff reviews.

### Stack

- Frontend: Next.js
- Backend: Next.js API Routes / Node.js worker service
- Database: Turso
- Auth: better-auth
- Payments: none
- Other: Playwright, Tailwind CSS, shadcn/ui

### Hosting

- Vercel (Hosting the Next.js frontend and dashboard interface): $0/mo
- Fly.io (Running long-lived background crawler worker containers with persistent disk volumes): $5-10/mo

### Build guide

1. **Project Scaffolding & Database Schema** — Initialize the Next.js application with Tailwind CSS, shadcn/ui, better-auth, and Turso SQLite database connections for storing sites, audits, pages, links, and hints.

```
Scramble a new Next.js project configured with TypeScript, Tailwind CSS, and shadcn/ui. Set up better-auth for local user authentication. Configure a Turso SQLite database client using libSQL. Create a comprehensive relational database schema with tables for 'projects', 'audits', 'pages' (storing URL, status code, title, meta description, depth, response time), 'links' (source page ID, target URL, anchor text, follow/nofollow), and 'hints' (page ID, hint code, severity, message). Ensure proper indexing on foreign keys for fast relational lookups.
```

2. **Asynchronous Web Crawler Engine** — Build a robust, concurrent web scraper worker in Node.js utilizing Playwright and Axios to fetch pages, parse HTML using Cheerio, extract links, and respect depth limits.

```
Build a robust asynchronous web crawler service in Node.js designed to run as a background worker. The crawler must take a seed URL, respect crawl depth limits (e.g., max depth 10), parse 'robots.txt' before crawling, and maintain a visited URL set to prevent loops. For each fetched page, extract the HTTP status code, response time, title tag, meta description, canonical tag, headings (H1-H6), and all internal/external anchor links. Store the raw page metrics and link relationships in the Turso database in batches to handle thousands of pages efficiently without memory exhaustion.
```

3. **JavaScript Rendering & DOM Extraction** — Integrate headless browser execution via Playwright to evaluate client-side rendered Single Page Applications and compare raw response vs rendered DOM.

```
Enhance the crawler worker to support optional JavaScript rendering using Playwright. When JavaScript crawling is enabled for a project, launch a headless browser context, navigate to the target URL, wait for network idle, and extract the fully rendered DOM tree. Compare the raw HTML response body against the rendered DOM to identify content discrepancies (e.g. missing links or text only present after hydration). Store these comparison metrics in the page record.
```

4. **Automated Hints Rule Engine** — Implement a deterministic rules engine executing over 300 technical SEO and security checks against stored crawl data, categorizing findings into Critical, High, Medium, and Low severity.

```
Write a modular rule engine service that executes automated checks ('Hints') against the crawled pages and link graph stored in the database. Implement at least 30 core SEO and security rules: missing H1 tags, duplicate title tags, broken internal links (404 responses), chains of redirects (>2 hops), non-secure HTTP links on HTTPS pages (mixed content), missing meta descriptions, and orphaned pages with zero internal links. Each triggered rule must generate a 'hint' record linked to affected pages with assigned severity levels (Critical, High, Medium, Low) and explanatory educational text.
```

5. **Audit Dashboard & Visualizations** — Build interactive dashboard screens displaying audit overview metrics, severity distribution charts, crawl maps, and filterable URL data tables.

```
Build a comprehensive audit dashboard UI in Next.js using Tailwind CSS and shadcn/ui. Create an overview page displaying key crawl metrics (total pages crawled, average response time, overall health score) and charts breaking down issues by severity using Recharts. Add a filterable, sortable URL Explorer data table allowing users to search pages by status code, content type, title, and assigned hints, with pagination and CSV export functionality.
```

6. **Audit Comparison & PDF Exporting** — Implement audit comparison logic to highlight metrics that changed between consecutive runs and generate clean client-facing reports.

```
Implement an audit comparison feature allowing users to select two completed audits for the same project and view side-by-side metric deltas (e.g., fixed issues, new issues introduced, pages added or removed). Add a report generation view optimized for printing and PDF export, presenting executive summaries, prioritized hints lists, and charts suitable for client presentations.
```

### Cost vs paying

**Starting costs (one-time):**

- Domain name: $12/yr
- Total: ~$12 one-time

**Ongoing costs (monthly):**

- Fly.io worker VM hosting: $8/mo
- Turso database storage: $0/mo
- Total: ~$8/mo

- Paying for the SaaS instead: $42/mo (Pro)
- Build time: 40-60 hours
- AI tool credits: ~$20 (Claude Pro)
- Break-even: Not financially rational (purely for educational mastery of web scrapers and graph databases)

## Sources

- [Sitebulb Official Website](https://sitebulb.com)
- [Sitebulb Pricing & Plans Page](https://sitebulb.com/pricing/)
- [Sitebulb Cloud Server Pricing Page](https://sitebulb.com/cloud/pricing/)