WaterCrawl

WaterCrawl

🕷️ WaterCrawl is a powerful web application that uses Python, Django, Scrapy, and Celery to crawl web pages and extract relevant data.

✨ Features

• 🕸️ Advanced Web Crawling & Scraping - Crawl websites with highly customizable options for depth, speed, and targeting specific content
• 🔍 Powerful Search Engine - Find relevant content across the web with multiple search depths (basic, advanced, ultimate)
• 🌐 Multi-language Support - Search and crawl content in different languages with country-specific targeting
• ⚡ Asynchronous Processing - Monitor real-time progress of crawls and searches via Server-Sent Events (SSE)
• 🔄 REST API with OpenAPI - Comprehensive API with detailed documentation and client libraries
• 🔌 Rich Ecosystem - Integrations with Dify, N8N, and other AI/automation platforms
• 🏠 Self-hosted & Open Source - Full control over your data with easy deployment options
• 📊 Advanced Results Handling - Download and process search results with customizable parameters

Check our API Overview to learn more about these features.

🛠️ Client SDKs

• ✅ Python Client - Full-featured SDK with support for all API endpoints
• ✅ Node.js Client - Complete JavaScript/TypeScript integration
• ✅ Go Client - Full-featured SDK with support for all API endpoints
• ✅ PHP Client - Full-featured SDK with support for all API endpoints
• 🔜 Rust Client - Coming soon

🔌 Integrations

• ✅ Dify Plugin (source code)
• ✅ N8N workflow node (source code)
• ✅ Dify Knowledge Base
• 🔄 Langflow (Pull Request - Not Merged yet)
• 🔜 Flowise (Coming soon)