GitHub - vaadin/vaadin-mcp: A complete solution for ingesting, indexing, and retrieving Vaadin documentation through semantic search. (original) (raw)

Vaadin Documentation RAG Service

A sophisticated, hierarchically-aware Retrieval-Augmented Generation (RAG) system for Vaadin documentation that understands document structure, provides framework-specific filtering, and enables intelligent parent-child navigation through documentation sections.

🎯 Project Overview

This project provides an advanced RAG system with enhanced hybrid search that:

🏗️ Architecture

vaadin-documentation-services/
├── packages/
│   ├── core-types/              # Shared TypeScript interfaces
│   ├── 1-asciidoc-converter/    # AsciiDoc → Markdown + metadata extraction
│   ├── 2-embedding-generator/   # Markdown → Vector database with hierarchical chunking
│   ├── rest-server/             # Enhanced REST API with hybrid search + reranking
│   └── mcp-server/              # MCP server with hierarchical navigation
├── package.json                 # Bun workspace configuration
└── PROJECT_PLAN.md             # Complete project documentation

Data Flow

flowchart TD subgraph "Step 1: Documentation Processing" VaadinDocs["📚 Vaadin Docs
(AsciiDoc)"] Converter["🔄 AsciiDoc Converter
• Framework detection
• URL generation
• Markdown output"] Processor["⚡ Embedding Generator
• Hierarchical chunking
• Parent-child relationships
• OpenAI embeddings"] end

subgraph "Step 2: Enhanced Retrieval"
    Pinecone["🗄️ Pinecone Vector DB<br/>• Rich metadata<br/>• Hierarchical relationships<br/>• Framework tags"]
    RestAPI["🌐 REST API<br/>• Enhanced hybrid search<br/>• Native Pinecone reranking<br/>• Framework filtering"]
end

subgraph "Step 3: Agent Integration"
    MCP["🤖 MCP Server<br/>• search_vaadin_docs<br/>• get_full_document<br/>• Full document retrieval"]
    IDEs["💻 IDE Assistants<br/>• Context-aware search<br/>• Hierarchical exploration<br/>• Framework-specific help"]
end

VaadinDocs --> Converter
Converter --> Processor
Processor --> Pinecone
Pinecone <--> RestAPI
RestAPI <--> MCP
MCP <--> IDEs

classDef processing fill:#e1f5fe,stroke:#01579b,stroke-width:2px
classDef storage fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
classDef api fill:#e8f5e8,stroke:#2e7d32,stroke-width:2px
classDef agent fill:#fff3e0,stroke:#e65100,stroke-width:2px

class VaadinDocs,Converter,Processor processing
class Pinecone,RestAPI storage
class MCP api
class IDEs agent

Loading

✨ Key Features

🌳 Hierarchical Navigation

🎛️ Developer Experience

🚀 Quick Start

Prerequisites

Installation

Clone and install dependencies

git clone https://github.com/vaadin/vaadin-documentation-services cd vaadin-documentation-services bun install

Environment Setup

Create .env file with your API keys

echo "OPENAI_API_KEY=your_openai_api_key" > .env echo "PINECONE_API_KEY=your_pinecone_api_key" >> .env echo "PINECONE_INDEX=your_pinecone_index" >> .env

Running the System

1. Process Documentation (One-time setup)

Convert AsciiDoc to Markdown with metadata

cd packages/1-asciidoc-converter bun run convert

Generate embeddings and populate vector database

cd ../2-embedding-generator bun run generate

2. Start REST API Server

cd packages/rest-server bun run start

Server runs at http://localhost:3001

3. Use MCP Server with IDE Assistant

The MCP server is deployed and available remotely via HTTP transport at:https://mcp.vaadin.com/

Configure your IDE assistant to use the Streamable HTTP transport:

import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js"; const transport = new StreamableHTTPClientTransport( new URL("https://mcp.vaadin.com/") );

📦 Package Details

Core Types (packages/core-types/)

Shared TypeScript interfaces used across all packages:

AsciiDoc Converter (packages/1-asciidoc-converter/)

Converts Vaadin AsciiDoc documentation to Markdown with metadata:

cd packages/1-asciidoc-converter bun run convert # Convert all documentation bun run test # Run framework detection tests

Embedding Generator (packages/2-embedding-generator/)

Creates vector embeddings with hierarchical relationships:

cd packages/2-embedding-generator bun run generate # Generate embeddings from Markdown bun run test # Run chunking and relationship tests

REST Server (packages/rest-server/)

Enhanced API server with hybrid search capabilities:

cd packages/rest-server bun run start # Start production server bun run test # Run comprehensive test suite bun run test:verbose # Detailed test output

API Endpoints:

MCP Server (packages/mcp-server/)

Model Context Protocol server for IDE assistant integration:

cd packages/mcp-server bun run build # Build for distribution bun run test # Run document-based tests

Available Tools:

🧪 Testing

Each package includes comprehensive test suites:

Test individual packages

cd packages/1-asciidoc-converter && bun run test cd packages/2-embedding-generator && bun run test
cd packages/rest-server && bun run test cd packages/mcp-server && bun run test

Run REST server against live endpoint

cd packages/rest-server && bun run test:server

📈 Performance & Metrics

Search Quality

System Performance

Production Readiness

🌐 Deployment

REST Server

The REST server is available at:

MCP Server

The MCP server is available at:

Documentation Processing

Automated via GitHub Actions:

🔧 Development

Workspace Structure

This project uses Bun workspaces for package management:

bun install # Install all dependencies bun run build # Build all packages bun run test # Test all packages

Adding New Features

  1. Core Types: Add interfaces to packages/core-types/
  2. Processing: Extend converters in packages/1-asciidoc-converter/ or packages/2-embedding-generator/
  3. API: Enhance search in packages/rest-server/
  4. Integration: Update MCP tools in packages/mcp-server/

Architecture Principles

📚 Documentation

🏆 Project Success

This project successfully delivered:

✅ Sophisticated RAG System: Replaced naive implementation with hierarchically-aware search
✅ Enhanced User Experience: Agents can now navigate from specific details to broader context
✅ Production Quality: Clean architecture, comprehensive testing, and error handling
✅ Framework Intelligence: Accurate Flow/Hilla content separation with common content inclusion
✅ Developer Integration: Seamless IDE assistant integration via MCP protocol

The system now provides intelligent, context-aware documentation search that understands the hierarchical structure of Vaadin documentation and enables sophisticated agent interactions.

📄 License

MIT - See license file for details.

Built with ❤️ for the Vaadin developer community