Multi-Agent Systems in Node.js: Web Scraping to MySQL Tutorial

Multi-Agent E-commerce System in Node.js: Web Scraping with MySQL Integration

Build Multi-Agent E-commerce System with OpenKBS and Claude Code

Learn how to build a multi-agent system that scrapes web data and stores it in MySQL. Use Claude Code to generate the code with natural language instructions, and OpenKBS to deploy and run the agents.

Prerequisites

1. Install Required Tools

npm i -g @anthropic-ai/claude-code
npm i -g openkbs

2. Setup Local MySQL

mysql -u root -p

CREATE DATABASE demo;
CREATE USER 'demo'@'localhost' IDENTIFIED BY 'demo';
GRANT ALL PRIVILEGES ON demo.* TO 'demo'@'localhost';
FLUSH PRIVILEGES;
EXIT;

Step 1: Product Search Agent

Create your first OpenKBS agent:

openkbs create brand-discovery
cd brand-discovery
claude

Give Claude this prompt:

First, create a brand-discovery agent that starts locally, takes a category as input (e.g., "electronics"), researches the top 3 most popular brands in that category, and when receive the response stores them in my local MySQL (localhost, DB_NAME: demo) with the brand name, description, logo/image, and category association.

Then, create a product-research agent that, for each brand, researches and finds five products, and stores them with the name, price, description, image, and association to the brand.

The flow: Input "electronics" → agent finds brands (e.g., TechCorp, ElectroMax, FutureTech) → stores brands and runs → product agents run simultaneously for all brands → rapidly populates products for the entire category.

After pasting the prompt, Claude will ask for approval at each step - simply click "Yes" to proceed through each phase. When Claude announces that the agent system is complete and README.md is ready, you can start using your agents by following the deployment steps.

What Claude Creates

• Main brand-discovery agent - with MySQL integration
• Related product-research agent - in related-agents/ folder
• Complete database schemas - with brands and products tables
• Orchestration scripts - that coordinate both agents
• README.md file - with YOUR EXACT setup instructions

Login with CLI:

openkbs login

Follow README.md to setup and start the agents:

open README.md

The README will guide you through:

• Configuring MySQL credentials - Setting up database connection
• Deploying agents - Using openkbs push to deploy to cloud
• Configuring API Keys - Initializing access between local scripts and cloud agents
• Starting the orchestrator - Running your multi-agent system

When configuring API keys, you'll be prompted with URLs. For each agent:

API Key Creation - Click 'Copy API Key' button to copy your key

Monitor Agent Execution in Real-Time

After setting up API keys, click on the Chat tab to monitor your agent's execution flow in real-time. Here you can see every tool call, decision, and data extraction as it happens:

Agent Execution Logs - Real-time monitoring of agent operations in the Chat tab

What to Expect: Multi-Agent System in Action

When you run your orchestrator (as specified in YOUR README.md), you'll see output similar to:

Starting brand discovery for category: electronics
✓ Connected to MySQL database
✓ Running brand discovery agent...
✓ Found 3 brands: TechCorp, ElectroMax, FutureTech
✓ Running product research agents (parallel)
✓ Stored 5 products for TechCorp
✓ Stored 5 products for ElectroMax
✓ Stored 5 products for FutureTech

Complete! Stored 3 brands and 15 products

Note: The exact command and parameters will be in your README.md file.

Understanding the Hierarchical Agent Architecture

    ┌─────────────────────────────────────┐
    │   Orchestration Script (Local)      │
    │   - Manages database connections    │
    │   - Coordinates agent execution     │
    └──────────────┬──────────────────────┘
                   │
                   ▼
    ┌─────────────────────────────────────┐
    │  Brand Discovery Agent (Cloud)      │
    │  - Searches for top 3 brands        │
    │  - Extracts brand information       │
    │  - Returns structured JSON          │
    └──────────────┬──────────────────────┘
                   │
                   ▼ (Parallel Execution)
          ┌────────┴────────┬────────────┐
          ▼                 ▼            ▼
    ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
    │Product Agent │ │Product Agent │ │Product Agent │
    │  (TechCorp)  │ │ (ElectroMax) │ │ (FutureTech) │
    └──────────────┘ └──────────────┘ └──────────────┘

Key Architecture Components:

• Orchestration Script - Local coordinator managing database connections and agent execution
• Brand Discovery Agent - Cloud-based agent that discovers and extracts brand information
• Product Agents - Multiple parallel agents (one per brand) for efficient data collection
• Parallel Execution - All product agents run simultaneously for maximum throughput
• Cloud Jobs - Each agent creates a separate job in the OpenKBS cloud infrastructure

Try It Yourself!

Give Claude this prompt in the same chat session:

Create a simple ecommerce website using this database

Watch Claude build a complete web interface where users can browse the brands and products your agents discovered. From raw data to user experience in minutes.

Result: Full E-commerce Website

npm start

Your data is now live in a web interface:

Product page with AI-extracted descriptions and pricing

The website automatically displays everything your agents discovered - real brands, products, prices, and descriptions!