How to vibe codeBrowse AI

Prerequisites

• Node.jsfree

  Required runtime for Next.js backend and Playwright automation scripts.

• GitHubfree

  Source code repository and CI/CD triggers for deployment.

AI coding tools

• Claude Code

  Unmatched capability for scaffolding full-stack Next.js applications, complex database schemas, and writing robust Playwright extraction scripts.

• Cursor

  Exceptional for interactive frontend UI work, reviewing diffs, and refining the visual robot configuration builder dashboard.

Build guide

1. 01Project Scaffolding and Database Schema Setup

   Initialize the Next.js project with Tailwind CSS, configure Shadcn UI, set up Supabase database connection, and implement better-auth for single-user authentication.

   Create a new Next.js 16 project with Tailwind CSS, TypeScript, and App Router. Set up better-auth for local user authentication using Supabase PostgreSQL as the database provider. Define the database schema in Prisma or Drizzle for 'robots' (storing name, target_url, extraction_selectors JSON, schedule_cron), 'robot_runs' (storing status, started_at, finished_at, error_message), and 'extracted_data_items' (storing run_id, payload JSON). Create clean layout wrappers with Shadcn UI components for a modern dashboard layout featuring a sidebar navigation and dark mode support.

   ⧉copy promptopen in Claude CodeCopen in Codexopen in Cursor

2. 02Headless Browser Automation Engine (Playwright)

   Build a modular scraping service using Playwright that launches a headless browser, navigates to target URLs, executes custom actions, and extracts requested DOM elements.

   Create a backend scraping utility using Playwright in TypeScript. Write an execution function that accepts a target URL and a JSON array of extraction rules (containing selector types, CSS/XPath locators, attribute targets, and data keys). The function must spin up a headless browser instance with realistic user-agent headers, wait for network idle, handle basic pagination or scrolling if specified, and extract structured key-value datasets into clean JSON objects. Handle timeouts, missing elements gracefully, and return detailed error logs if the target page fails to render.

   ⧉copy promptopen in Claude CodeCopen in Codexopen in Cursor

3. 03Background Job Queue & Cron Scheduling (BullMQ + Fly.io)

   Configure an asynchronous worker queue using BullMQ and Redis to execute scheduled scraping jobs reliably without timing out HTTP requests.

   Implement a background job processing pipeline using BullMQ and Upstash Redis. Create an API route in Next.js that triggers a robot run by pushing a job into the scraping queue. Set up a standalone worker script designed to run in a Fly.io Docker container that pulls jobs from the queue, executes the Playwright scraping utility from Step 2, updates the 'robot_runs' table status in Supabase (pending -> running -> success/failed), and saves extracted records. Add support for cron-based recurring schedules using BullMQ repeatable jobs.

   ⧉copy promptopen in Claude CodeCopen in Codexopen in Cursor

4. 04Robot Configuration Dashboard & Selector Builder UI

   Build an intuitive frontend dashboard allowing users to create robots, input target URLs, define extraction fields, and view past execution logs.

   Build a comprehensive dashboard interface in Next.js App Router for managing scraping robots. Include a 'Create Robot' wizard where users enter a name, target URL, and configure a dynamic list of data extraction fields (Field Name, CSS Selector, Extraction Type like Text/Attribute/HTML). Add a detailed robot view showing execution history logs with success/failure badges, duration timers, and a preview table displaying extracted JSON datasets. Include controls to manually trigger a run and toggle automated cron schedules (hourly, daily).

   ⧉copy promptopen in Claude CodeCopen in Codexopen in Cursor

5. 05AI Change Detection & Self-Healing Fallback

   Integrate an AI model layer (using Anthropic API or OpenAI API) to automatically inspect DOM changes and suggest updated selectors when extraction rules fail.

   Integrate an AI self-healing module into the scraping pipeline using the Anthropic API. When a designated CSS selector returns null or fails during a Playwright extraction run, capture the outer HTML snippet of the target container area and send it to the LLM along with the original field label. Instruct the model to analyze the new HTML structure and return an updated, working CSS or XPath selector. Automatically update the robot's configuration schema in Supabase with the healed selector, log a warning event, and successfully complete the extraction run without failing.

   ⧉copy promptopen in Claude CodeCopen in Codexopen in Cursor

6. 06Data Export, Webhooks, and Notifications

   Implement export capabilities allowing users to download extracted data as CSV/JSON, dispatch webhooks to external endpoints, and send email alerts on failure.

   Implement data export and notification features for the scraping platform. Add endpoints and UI buttons to download extracted run data as a formatted CSV or JSON file. Create a webhook dispatching system that fires an HTTP POST request with the structured JSON dataset to a user-configured webhook URL immediately upon successful run completion. Integrate Resend to send an automated alert email to the user if a scheduled robot run fails consecutively three times.

   ⧉copy promptopen in Claude CodeCopen in Codexopen in Cursor

Cost vs paying for Browse AI