How to vibe codeOncrawl

Prerequisites

• Node.jsfree

  Runtime for building the Next.js app and scraper workers

• GitHubfree

  Code repository and CI/CD triggers

• Supabase Accountfree

  Provides Postgres and vector storage for crawl results and logs

AI coding tools

• Claude Code

  Best-in-class agentic coding tool for scaffolding complex multi-file crawler services and parsers

• Cursor

  Ideal for iterative UI work on the dashboards and segmentation matrices

Build guide

1. 01Project Scaffolding & Database Schema

   Initialize the Next.js 16 app with Tailwind CSS, shadcn/ui, and configure Supabase Postgres tables for projects, URLs, crawl sessions, and raw log records.

   Initialize a Next.js 16 project with TypeScript, Tailwind CSS, and shadcn/ui components. Configure better-auth for single-user authentication. Set up Supabase client connections and create SQL migrations for relational tables: `projects` (id, name, target_url), `crawls` (id, project_id, status, started_at, finished_at), `urls` (id, crawl_id, url, status_code, depth, title, load_time, content_type), and `server_logs` (id, project_id, ip, timestamp, method, path, user_agent, status_code). Ensure full type safety across all database queries.

   ⧉copy promptopen in Claude CodeCopen in Codexopen in Cursor

2. 02Headless Crawler Worker Engine

   Build a recursive web crawler using Playwright that respects robots.txt, renders JavaScript, extracts internal links, and streams results back to the database.

   Build a robust Node.js crawler worker script using Playwright and Cheerio. The worker should take a `crawl_id` and `target_url`, fetch `robots.txt`, parse sitemaps if available, and recursively visit internal links up to a configurable depth limit. For each page visited, capture the HTTP status code, page load time, title, meta description, and extract all outgoing anchor links. Handle timeouts, redirects, and JS rendering errors gracefully. Stream the parsed URL data in batches back to Supabase to prevent out-of-memory errors on large crawls.

   ⧉copy promptopen in Claude CodeCopen in Codexopen in Cursor

3. 03Server Log Ingestion & Bot Hit Analyzer

   Implement a log parser endpoint that ingests standard Apache/Nginx log formats or custom JSON logs, parsing bot signatures to distinguish Googlebot, Bingbot, and AI crawlers.

   Create a backend log ingestion service and API route in Next.js that accepts raw server log uploads (CSV or text files). Write a robust regex parser that extracts IP addresses, timestamps, HTTP methods, request paths, status codes, and User-Agents from each log line. Automatically classify User-Agents into categories (Googlebot, Bingbot, AI Bots like ChatGPT-User/ClaudeBot, and Standard Users). Store parsed logs in the `server_logs` table, indexed by path and timestamp for fast cross-referencing against crawl results.

   ⧉copy promptopen in Claude CodeCopen in Codexopen in Cursor

4. 04Link Graph & Architecture Analysis Module

   Develop analytical queries and visual components to calculate internal link distribution, spot orphan pages, and detect structural bottlenecks.

   Build an analytics calculation engine and UI dashboard for site architecture. Write SQL/TypeScript logic to analyze the `urls` table and compute: internal link count per page, distribution of crawl depth, identification of orphan pages (pages with 0 incoming internal links), and broken internal links (404/5xx status codes). Present these findings in a clean dashboard using shadcn/ui data tables and Recharts charts, allowing users to filter by depth and status code.

   ⧉copy promptopen in Claude CodeCopen in Codexopen in Cursor

5. 05AI Search Lens & Content Evaluation

   Integrate the Vercel AI SDK and OpenAI API to analyze scraped page content quality and track AI bot activity patterns from ingested logs.

   Implement an 'AI Search Lens' and 'Content Lens' feature using the Vercel AI SDK and OpenAI API. Create a background job that samples page content from completed crawls, sends it to the LLM to grade SEO readiness, readability, and content depth (returning a structured score and actionable recommendations). Also, build a log aggregation view specifically isolating AI bot hits (ChatGPT-User, ClaudeBot, PerplexityBot) against site pages to show which sections of the site receive the most AI crawler attention.

   ⧉copy promptopen in Claude CodeCopen in Codexopen in Cursor

6. 06Export API & Final Polish

   Add a full REST API for data export and polish the overall workflow with status indicators, notifications, and error boundary handling.

   Build a secure REST API protected by bearer tokens that lets users trigger new crawls, check crawl status, and export crawl data and log analysis summaries as JSON or CSV. Add comprehensive error handling, toast notifications for worker actions, and clean up the UI layout to match a professional enterprise SEO dashboard aesthetic.

   ⧉copy promptopen in Claude CodeCopen in Codexopen in Cursor

Cost vs paying for Oncrawl