Metadata-Version: 2.4
Name: search-engine-tool-mcp
Version: 0.5.0
Summary: A Python MCP server for zero-config web search and content extraction
Author: Search Engine Tool Contributors
License-Expression: MIT
Project-URL: Homepage, https://github.com/YiHarvest/search-engine-tool-mcp
Project-URL: Documentation, https://github.com/YiHarvest/search-engine-tool-mcp#readme
Project-URL: Repository, https://github.com/YiHarvest/search-engine-tool-mcp
Project-URL: Issues, https://github.com/YiHarvest/search-engine-tool-mcp/issues
Keywords: mcp,search,web,extraction,ddgs,searxng,tavily
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: ddgs<10,>=9.14
Requires-Dist: mcp<2,>=1.0.0
Requires-Dist: httpx[socks]>=0.27.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: trafilatura>=1.12
Requires-Dist: beautifulsoup4>=4.12
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: build>=0.10.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Dynamic: license-file

# Search Engine Tool MCP

A Python MCP (Model Context Protocol) Server for web search and extraction with multiple providers.

## Version

Current version: **0.5.0**

## Features

- 🔍 **Web Search**: Zero-configuration search with DDGS, plus SearXNG, TalorData, You.com, and Tavily providers
- 📄 **Content Extraction**: Extract text from public HTTP(S) URLs with local extraction or Tavily
- 🔄 **Fixed Auto Search**: Uses SearXNG when configured, then DDGS
- 🔑 **No Search Key Required**: DDGS search and local extraction work without API keys
- 🔒 **Safe Local Extraction**: DNS-aware SSRF checks, per-hop redirect validation, and bounded responses
- 🚀 **MCP Compatible**: Full support for Model Context Protocol

## Installation

### From PyPI (Recommended)

```bash
# Using pip
pip install search-engine-tool-mcp

# Or using uv (recommended)
uv pip install search-engine-tool-mcp

# Or using uvx (no installation required)
uvx search-engine-tool-mcp@latest
```

### From Source

```bash
git clone https://github.com/YiHarvest/search-engine-tool.git
cd search-engine-tool

# Using uv (recommended)
uv sync

# Or using pip
pip install -e .
```

## Configuration

### Environment Variables

- **SEARXNG_BASE_URL** (optional): Your SearXNG instance URL. Example: `http://127.0.0.1:8080`
- **DDGS_REGION**, **DDGS_BACKEND**, **DDGS_SAFESEARCH**, **DDGS_TIMEOUT**, and **DDGS_PROXY** (optional): Tune DDGS. DDGS requires no API key.
- **SEARCH_TOTAL_TIMEOUT** (optional, default `20`): Maximum seconds for the complete `web_search` operation, including automatic fallback.
- **EXTRACT_MAX_BYTES** (optional, default `5000000`): Maximum downloaded response size for local extraction.
- **EXTRACT_MAX_CHARS** (optional, default `100000`): Maximum extracted character count.
- **EXTRACT_MAX_REDIRECTS** (optional, default `5`): Maximum manually followed redirects; every hop is validated.
- **TALORDATA_API_KEY** (optional): Your TalorData SERP API key for explicit `provider="talordata"` searches. TalorData supports `web_search`, not `web_extract`.
- **YDC_API_KEY** (optional): Your You.com API key for explicit `provider="you"` searches.
- **TAVILY_API_KEY** (optional): Used for explicit Tavily search/extraction and automatic extraction fallback.

The server loads only `.env` from its current working directory. It does not search parent or home directories.

### Provider Capabilities

| Provider | Configuration | Search | Extract | Participates in `web_search auto` |
|---|---|---:|---:|---:|
| DDGS | None | ✅ | ❌ | ✅ |
| SearXNG | `SEARXNG_BASE_URL` | ✅ | ❌ | First, when configured |
| TalorData | `TALORDATA_API_KEY` | ✅ | ❌ | ❌ |
| You.com | `YDC_API_KEY` | ✅ | ❌ | ❌ |
| Tavily | `TAVILY_API_KEY` | ✅ | ✅ | ❌ |
| Local | None | ❌ | ✅ | Not applicable |

### Fixed Auto Rules

For **web_search**:

- If `SEARXNG_BASE_URL` exists: SearXNG, then DDGS when SearXNG fails or returns no results.
- Without `SEARXNG_BASE_URL`: DDGS.
- TalorData, You.com, and Tavily never participate in `auto`. They require API keys and explicit provider selection, preventing accidental API usage.

For **web_extract**:

- Default provider is `local` (free, no API key required)
- If `provider="auto"` and local extraction fails:
  - Falls back to Tavily if `TAVILY_API_KEY` is available
  - Otherwise returns error
- `provider="tavily"` requires `TAVILY_API_KEY`

### Quick Configuration

1. Copy the example configuration file:

```bash
cp .env.example .env
```

2. Optional: edit `.env` only when you want SearXNG, custom DDGS settings, or an explicitly selected paid provider:

```bash
# SEARXNG_BASE_URL="http://127.0.0.1:8080"
DDGS_REGION="us-en"
DDGS_BACKEND="auto"
DDGS_SAFESEARCH="moderate"
DDGS_TIMEOUT="10"
# DDGS_PROXY="socks5h://127.0.0.1:9150"
SEARCH_TOTAL_TIMEOUT="20"
EXTRACT_MAX_BYTES="5000000"
EXTRACT_MAX_CHARS="100000"
EXTRACT_MAX_REDIRECTS="5"

# Paid providers: uncomment only when explicitly needed
# TALORDATA_API_KEY="your-talordata-api-key-here"
# YDC_API_KEY="your-ydc-api-key-here"
# TAVILY_API_KEY="your-tavily-api-key-here"
```

### Run SearXNG Locally

`SEARXNG_BASE_URL` is not an API key. It is the URL of a SearXNG instance that you run. This MCP server uses SearXNG's JSON search endpoint, so `json` must be enabled in `settings.yml`.

1. Create the configuration directory and `searxng/settings.yml`:

```bash
mkdir -p searxng
```

```yaml
# searxng/settings.yml
use_default_settings: true

search:
  formats:
    - html
    - json

server:
  # Replace this even for local use: openssl rand -hex 32
  secret_key: "replace-with-a-random-secret"
  limiter: false
  image_proxy: true
```

2. Start a SearXNG container bound only to localhost:

```bash
docker run -d \
  --name searxng \
  --restart unless-stopped \
  -p 127.0.0.1:8080:8080 \
  -v "$(pwd)/searxng:/etc/searxng:rw" \
  docker.io/searxng/searxng:latest
```

3. Verify the JSON endpoint:

```bash
curl "http://127.0.0.1:8080/search?q=hello&format=json"
```

After it returns JSON containing `results`, configure:

```bash
SEARXNG_BASE_URL="http://127.0.0.1:8080"
```

Use `docker stop searxng` and `docker start searxng` to stop and restart it. For production or public deployment, add a reverse proxy, HTTPS, access controls, and a random `secret_key`; see the [official SearXNG container documentation](https://docs.searxng.org/admin/installation-docker.html).

### MCP Client Configuration

No environment variables are required for DDGS. Optional SearXNG settings and paid-provider credentials must be passed to the MCP server process. Never commit real credentials to Git.

#### JSON clients (Trae and similar)

Zero-configuration DDGS search:

```json
{
  "mcpServers": {
    "search-engine-tool": {
      "command": "uvx",
      "args": [
        "search-engine-tool-mcp@latest"
      ]
    }
  }
}
```

To use SearXNG or a paid provider, add only the variables you actually use and replace every placeholder:

```json
{
  "mcpServers": {
    "search-engine-tool": {
      "command": "uvx",
      "args": [
        "search-engine-tool-mcp@latest"
      ],
      "env": {
        "SEARXNG_BASE_URL": "http://127.0.0.1:8080",
        "TALORDATA_API_KEY": "<your-talordata-api-key>",
        "YDC_API_KEY": "<your-you-api-key>",
        "TAVILY_API_KEY": "<your-tavily-api-key>"
      }
    }
  }
}
```

Delete unused entries. In particular, do not leave a placeholder `TAVILY_API_KEY`: `web_extract(provider="auto")` treats any non-empty value as configured and may attempt Tavily fallback. Other variables from the environment-variable list can be passed through the same `env` object.

Restart or refresh the MCP server after saving. If Trae and SearXNG run on different hosts or container networks, `127.0.0.1` refers to the environment running Trae/the MCP process; replace it with a SearXNG URL reachable from that environment.

#### Codex (TOML)

Codex CLI and the Codex IDE extension share `config.toml`. Add this to the user configuration at `~/.codex/config.toml`, or to `.codex/config.toml` in a trusted project:

```toml
[mcp_servers.search-engine-tool]
command = "uvx"
args = [
  "search-engine-tool-mcp@latest",
]
startup_timeout_sec = 30
tool_timeout_sec = 60

# Add only the entries you use.
[mcp_servers.search-engine-tool.env]
SEARXNG_BASE_URL = "http://127.0.0.1:8080"
TALORDATA_API_KEY = "<your-talordata-api-key>"
YDC_API_KEY = "<your-you-api-key>"
TAVILY_API_KEY = "<your-tavily-api-key>"
```

The environment table is optional. Remove unused entries and never commit a configuration file containing real credentials.

Restart Codex, then run `codex mcp list` or use `/mcp` in a Codex session to verify the connection. See the [official Codex MCP documentation](https://developers.openai.com/codex/mcp/) for configuration options.

## Usage

### Available Tools

The MCP server exposes two tools:

#### 1. web_search

Search the web for information.

**Parameters:**

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `query` | string | ✅ | - | Search query string |
| `provider` | string | ❌ | `"auto"` | Provider: `"auto"`, `"searxng"`, `"ddgs"`, `"talordata"`, `"you"`, or `"tavily"` |
| `max_results` | integer | ❌ | `5` | Maximum number of results (1-20) |
| `search_depth` | string | ❌ | `"basic"` | Search depth: `"basic"` or `"advanced"` (Tavily only) |
| `include_answer` | boolean | ❌ | `false` | Include AI-generated answer (Tavily and TalorData only) |

**Response:**

```json
{
  "query": "search query",
  "provider": "ddgs",
  "count": 5,
  "results": [
    {
      "href": "https://example.com",
      "title": "Result Title",
      "abstract": "Result snippet..."
    }
  ]
}
```

#### 2. web_extract

Extract content from a public HTTP(S) URL.

**Parameters:**

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `url` | string | ✅ | - | Public HTTP(S) URL to extract |
| `provider` | string | ❌ | `"auto"` | Provider to use: `"auto"`, `"local"`, or `"tavily"` |

**Provider Options:**

- **`local`** (default free provider): Free local extraction, no API key required. Uses trafilatura + BeautifulSoup for content extraction.
  - ⚠️ **Limitations:**
    - Does not execute JavaScript
    - Cannot bypass login walls, paywalls, or CAPTCHA
    - Dynamic/spa-style pages may have incomplete extraction
    - Timeout: 20 seconds (default)
    - Only public HTTP(S) targets are allowed; all resolved IPs and every redirect hop are validated
    - Downloads, extracted characters, redirects, and response Content-Type are bounded
- **`tavily`**: Tavily API extraction, requires `TAVILY_API_KEY`.
- **`auto`** (default): Tries local first, falls back to Tavily if `TAVILY_API_KEY` is available and local fails.

**Response:**

```json
{
  "url": "https://example.com",
  "content": "Extracted content...",
  "provider": "local"
}
```

## Development

### Setup Development Environment

```bash
# Clone the repository
git clone https://github.com/YiHarvest/search-engine-tool.git
cd search-engine-tool

uv sync
```

### Run Tests

```bash
uv run pytest -v
```

### Build Package

```bash
uv build
```

### Check Package

```bash
uv run twine check dist/*
```

## Project Structure

```
search-engine-tool/
├── src/search_engine_tool_mcp/
│   ├── __init__.py
│   ├── server.py          # MCP server implementation
│   ├── search.py          # Web search functionality
│   ├── extract.py         # Content extraction
│   ├── errors.py          # Stable, client-safe error types
│   ├── schemas.py         # Pydantic data models
│   ├── security/
│   │   └── url_validator.py  # DNS-aware SSRF validation
│   └── providers/
│       ├── __init__.py
│       ├── ddgs.py        # DDGS provider (free search)
│       ├── searxng.py     # SearXNG provider (self-hosted search)
│       ├── talordata.py   # TalorData provider (search)
│       ├── you.py         # You.com provider (search)
│       ├── tavily.py      # Tavily provider (search + extract)
│       └── local_extract.py  # Local extraction provider (free)
├── tests/
│   ├── test_search.py
│   ├── test_ddgs.py
│   ├── test_url_validator.py
│   ├── test_local_extract.py
│   ├── test_providers.py
│   └── test_extract.py
├── pyproject.toml
└── README.md
```
## License

MIT License

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## Support

For issues and feature requests, please use the [GitHub Issues](https://github.com/YiHarvest/search-engine-tool-mcp/issues) page.
