Automated API Schema Crawler & Extractor

This workflow automates the process of discovering, scraping, and extracting REST API documentation from online sources. It searches for API schemas related to a given service or website, retrieves relevant documentation, processes and chunks large content, and uses Google Gemini models for classification, information extraction, and embeddings. The extracted API operations (endpoints, methods, resources, and descriptions) are stored in a structured format within Google Sheets for further use.

Key Features

• 🔍 Automated Web Search: Finds API documentation URLs related to a given service.
• 📄 Web Scraping: Extracts and cleans webpage content from identified sources.
• 🧠 AI-Powered Processing: Uses Google Gemini models for classification, embeddings, and structured information extraction.
• 📊 Data Organization: Splits and processes large documents into manageable chunks for accurate embeddings.
• 🗂 API Operations Extraction: Identifies and structures REST API endpoints with method, resource, operation, and description.
• 📑 Google Sheets Integration: Logs extracted API schema data (service, resource, operation, method, description, documentation link).
• 🔄 Event-Driven Workflow: Supports research, extraction, and generation stages triggered dynamically.
• ⚡ Error Handling & Deduplication: Filters duplicate results and manages failed extractions gracefully.

Use Cases

• Automating research of public APIs for integration projects.
• Building a centralized API catalog from multiple services.
• Preparing structured API schemas for AI-driven analysis or documentation.
• Supporting data engineers, integrators, and developers in discovering hidden APIs.