WaterCrawl
🕷️ WaterCrawl is a powerful web application that uses Python, Django, Scrapy, and Celery to crawl web pages and extract relevant data.
✨ Features
- 🕸️ Advanced Web Crawling & Scraping - Crawl websites with highly customizable options for depth, speed, and targeting specific content
- 🔍 Powerful Search Engine - Find relevant content across the web with multiple search depths (basic, advanced, ultimate)
- 🌐 Multi-language Support - Search and crawl content in different languages with country-specific targeting
- ⚡ Asynchronous Processing - Monitor real-time progress of crawls and searches via Server-Sent Events (SSE)
- 🔄 REST API with OpenAPI - Comprehensive API with detailed documentation and client libraries
- 🔌 Rich Ecosystem - Integrations with Dify, N8N, and other AI/automation platforms
- 🏠 Self-hosted & Open Source - Full control over your data with easy deployment options
- 📊 Advanced Results Handling - Download and process search results with customizable parameters
Check our API Overview to learn more about these features.
🛠️ Client SDKs
- ✅ Python Client - Full-featured SDK with support for all API endpoints
- ✅ Node.js Client - Complete JavaScript/TypeScript integration
- ✅ Go Client - Full-featured SDK with support for all API endpoints
- ✅ PHP Client - Full-featured SDK with support for all API endpoints
- 🔜 Rust Client - Coming soon
🔌 Integrations
- ✅ Dify Plugin (source code)
- ✅ N8N workflow node (source code)
- ✅ Dify Knowledge Base
- 🔄 Langflow (Pull Request - Not Merged yet)
- 🔜 Flowise (Coming soon)
