Metadata-Version: 2.4
Name: scihub-cli
Version: 0.5.0
Summary: Multi-source academic paper downloader (Sci-Hub, Unpaywall, CORE)
Author-email: Sci-Hub CLI <example@example.com>
License-Expression: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.25.1
Requires-Dist: beautifulsoup4>=4.9.3
Requires-Dist: curl-cffi>=0.13.0
Requires-Dist: cloudscraper>=1.2.71
Requires-Dist: pymupdf4llm>=0.2.0
Dynamic: license-file

# Sci-Hub CLI

A command-line tool for batch downloading academic papers with multi-source support (OpenAlex, Europe PMC, Sci-Hub, Unpaywall, arXiv, CORE).

*Read this in other languages: [English](README.md), [简体中文](README.zh-CN.md)*

## Features

- **Multi-Source Support**: Intelligently routes downloads across multiple sources
  - **OpenAlex**: OA metadata + full-text link discovery (no email required)
  - **Europe PMC**: OA full-text links for biomedical literature (no email required)
  - **arXiv**: Prioritized for preprints (free, no API key needed)
  - **Unpaywall**: For open access papers (requires email)
  - **Sci-Hub**: Fallback for older papers (coverage-driven, slower)
  - **CORE**: Additional OA fallback (disabled by default; opt in with `--enable-core`)
- **Smart Year-Based Routing**:
  - Papers before 2021: OA sources first, Sci-Hub fallback
  - Papers 2021+: OA sources only (skip Sci-Hub)
- **Parallel Source Querying**: Fast sources queried in parallel with slow-source fallback
- **Parallel Mirror Testing**: Quickly finds working Sci-Hub mirrors (typically <2s)
- **Smart Metadata Caching**: Avoids redundant API calls across sources
- **Smart Fallback**: Automatically tries alternative sources if primary fails
- **Flexible Input**: Download papers using DOIs, arXiv IDs, or URLs
- Batch processing from a text file
- Automatic mirror selection and testing
- Customizable output directory
- Robust error handling and retries
- PDF validation (rejects HTML files)
- Progress reporting
- **Metadata-based Filenames**: Automatically names files as `[YYYY] - [Title].pdf` for easy organization

## Recent Updates

### v0.4.1

- Improved `--to-md` stability by serializing Markdown conversion internally, reducing random converter crashes under parallel downloads
- Added release guidance for stale `uv tool` environments that may still run old converter code

### v0.5.0

- Enabled `--academic-only` filtering by default to exclude obvious non-academic links before download
- Added high-signal URL normalization/deduplication for noisy inputs (tracking params/fragments/`www` variants)
- Added OpenAlex source integration and improved OA-first routing behavior
- Added Europe PMC OA source (biomedical OA coverage, no email required)
- Improved fail-fast behavior for challenge/paywall pages to reduce wasted retries
- Added robust PMC fallback to EuropePMC render endpoints when primary PMC PDF URLs return HTML
- Updated default parallelism to `16` for better speed/success balance on large batches

## Installation

[uv](https://docs.astral.sh/uv/) is an extremely fast Python package and project manager, written in Rust.

### Install uv

```bash
# macOS and Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
```

### Install scihub-cli (recommended)

```bash
# From PyPI
pip install scihub-cli

# Or install globally with uv
uv tool install scihub-cli

# Try without installing (temporary run)
uvx scihub-cli papers.txt
```

### Development / source install

```bash
# Install from the current directory while developing
uv tool install .

# Or install from GitHub for an unreleased snapshot
uv tool install git+https://github.com/Oxidane-bot/scihub-cli.git
```

**Note**: `pip install` and `uv tool install scihub-cli` use the published release. `uv tool install .` is best for local development.

### Global vs Temporary Usage

- **Global Installation**: Use `pip install scihub-cli` or `uv tool install scihub-cli`
- **Temporary Usage**: Use `uvx scihub-cli` to run the tool without installing it
- **Source Code**: Clone the repo and run directly with Python for development

### Manual Installation (Alternative)

If you prefer to run directly from source:

1. Clone this repository:
   ```
   git clone https://github.com/Oxidane-bot/scihub-cli.git
   cd scihub-cli
   ```

2. Sync dependencies with the lockfile:
   ```bash
   uv sync --frozen
   ```

3. Run directly with Python:
   ```bash
   uv run python -m scihub_cli input_file.txt
   ```

### Troubleshooting Installation

If you encounter issues with the installation, try the following:

1. Ensure you have Python 3.10+ installed:
   ```bash
   python --version
   ```

2. Verify uv is installed correctly:
   ```bash
   uv --version
   ```

3. Check if the command is in your PATH:
   ```bash
   # On Windows
   where scihub-cli
   
   # On macOS/Linux
   which scihub-cli
   ```

4. If you get "command not found" errors after installation:
   ```bash
   # Update shell environment
   uv tool update-shell
   
   # Manual PATH refresh
   # On Windows
   $env:Path = [System.Environment]::GetEnvironmentVariable("Path","User")
   
   # On macOS/Linux
   source ~/.bashrc  # or .zshrc, .bash_profile, etc.
   ```

5. If having issues, try:
   ```bash
   # Upgrade a PyPI install
   pip install --upgrade scihub-cli

   # List installed tools
   uv tool list
   
   # Upgrade a tool
   uv tool upgrade scihub-cli
   
   # Reinstall
   uv tool uninstall scihub-cli
   uv tool install scihub-cli
   ```

6. If `--to-md` still behaves like an old version after upgrade, force refresh the local tool environment:
   ```bash
   # Published release
   uv tool install --force --reinstall --refresh scihub-cli

   # Local development checkout
   uv tool install --force --reinstall --refresh .
   ```

## Usage

### Basic Usage

```bash
# If installed with uv
scihub-cli input_file.txt

# If running temporarily with uv
uvx scihub-cli input_file.txt

# If running directly from source repository
uv run python -m scihub_cli input_file.txt
```

Where `input_file.txt` is a text file containing DOIs or paper URLs, one per line.

### Input File Format

Create a text file with one identifier per line. Supports:
- **DOIs**: `10.1038/nature12373`
- **arXiv IDs**: `2301.12345` or `arxiv:2401.00001`
- **URLs**:
  - DOI URLs: `https://doi.org/10.1126/science.abc1234`
  - Direct PDF URLs: `https://files.eric.ed.gov/fulltext/EJ1358705.pdf`
  - PMC article URLs: `https://pmc.ncbi.nlm.nih.gov/articles/PMC6505544/`
  - OA landing pages (auto PDF extraction): `https://example.org/article/123`

Example `papers.txt`:
```
# Comments start with a hash symbol
10.1038/s41586-020-2649-2
2301.12345
arxiv:2401.00001
https://files.eric.ed.gov/fulltext/EJ1358705.pdf
https://pmc.ncbi.nlm.nih.gov/articles/PMC6505544/
10.1016/s1003-6326(21)65629-7
```

### Optional Email (Unpaywall)

Set an email if you want Unpaywall open-access lookup. If no email is provided, Unpaywall is skipped automatically.

```bash
# Enable Unpaywall by setting email
scihub-cli papers.txt --email your-email@university.edu
```

The email is saved to `~/.scihub-cli/config.json` and sent only to Unpaywall for rate limiting (not tracking).

### Command-Line Options

```
usage: scihub-cli [-h] [-o OUTPUT] [-m MIRROR] [-t TIMEOUT] [-r RETRIES] [-p PARALLEL]
                  [--enable-core] [--fast-fail] [--academic-only]
                  [--no-academic-only]
                  [--email EMAIL] [-v] [--version] input_file

Download academic papers from Sci-Hub and Unpaywall in batch mode.

positional arguments:
  input_file            Text file containing DOIs or URLs (one per line)

options:
  -h, --help            show this help message and exit
  -o OUTPUT, --output OUTPUT
                        Output directory for downloaded PDFs (default: ./downloads)
  -m MIRROR, --mirror MIRROR
                        Specific Sci-Hub mirror to use
  -t TIMEOUT, --timeout TIMEOUT
                        Request timeout in seconds (default: 15)
  -r RETRIES, --retries RETRIES
                        Number of retries for failed downloads (default: 3)
  -p PARALLEL, --parallel PARALLEL
                        Number of parallel downloads (threads) (default: 16)
  --to-md              Convert downloaded PDFs to Markdown
  --md-output MD_OUTPUT
                        Output directory for generated Markdown (default: <pdf_output>/md)
  --md-backend MD_BACKEND
                        Markdown conversion backend (default: pymupdf4llm)
  --md-overwrite        Overwrite existing Markdown files
  --md-warn-only        Do not fail the run if Markdown conversion fails
  --trace-html          Capture and persist HTML snapshots for failed downloads
  --trace-html-dir TRACE_HTML_DIR
                        Directory for HTML snapshots (default: <output>/trace-html)
  --trace-html-max-chars TRACE_HTML_MAX_CHARS
                        Maximum characters per HTML snapshot file (default: 2000000)
  --enable-core         Enable CORE source lookups (disabled by default to avoid rate-limit slowdown)
  --fast-fail           Skip bypass and HTML recovery on permanent failures (faster, may reduce success rate)
  --academic-only       Filter out obvious non-academic URLs before downloading (default)
  --no-academic-only    Disable academic-only filtering and process all input URLs
  --email EMAIL         Email for Unpaywall API (saves to config file)
  -v, --verbose         Enable verbose logging
  --version             show program's version number and exit
```

### Configuration

scihub-cli stores configuration in `~/.scihub-cli/config.json`:

```json
{
  "email": "your-email@university.edu"
}
```

You can edit this file directly or use `--email` to update it.

### Examples

```bash
# Basic usage
scihub-cli papers.txt

# Download PDFs and convert them to Markdown
# Default markdown output: <pdf_output_dir>/md
scihub-cli --to-md papers.txt

# Custom markdown output directory
scihub-cli --to-md --md-output research/markdown papers.txt

# Enable failure diagnostics (source attempts + HTML snapshots in download-report.json)
scihub-cli --to-md --md-warn-only --trace-html papers.txt

# Fast-fail is enabled by default (and default parallelism is 16)
scihub-cli -r 1 -t 8 papers.txt

# Academic-only filtering is enabled by default
scihub-cli papers.txt

# Disable academic-only filtering (process all URLs)
scihub-cli --no-academic-only papers.txt

# Disable fast-fail when you want slower but more aggressive recovery
scihub-cli --no-fast-fail papers.txt

# Opt in to CORE (off by default)
scihub-cli --enable-core papers.txt

# Specify output directory
scihub-cli -o research/papers papers.txt

# Use specific mirror
scihub-cli -m https://sci-hub.se papers.txt

# Increase verbosity
scihub-cli -v papers.txt
```

## How It Works

The tool uses intelligent multi-source routing:

1. **Year Detection**: Queries Crossref API to determine publication year
2. **Smart Routing**:
   - Papers before 2021 → Try OA sources first, then Sci-Hub fallback
   - Papers 2021+ → Try OA sources only (skip Sci-Hub)
   - Unknown year → OA sources first with Sci-Hub fallback
3. **Download Process**:
   - Get PDF URL from selected source
   - Download with progress tracking
   - Validate PDF (reject HTML files)
   - Generate filename from metadata: `[YYYY] - [Title].pdf`

### Why Multi-Source?

- **Sci-Hub**: Strong fallback for pre-2021 coverage, but stopped updating in 2020
- **Unpaywall**: Best for 2021+ open access papers (25-35% coverage for recent papers)
- **Combined**: OA-first speed with Sci-Hub fallback for older literature

### Domain-Specific User-Agents

The tool automatically adapts HTTP headers for different publishers:
- **MDPI**: Uses `curl/8.0.0` (required by their CDN)
- **Others**: Uses browser User-Agent for compatibility

## Coverage and Success Rates

| Year Range | Primary Source | Success Rate |
|-----------|---------------|--------------|
| Before 2021 | OA first (Sci-Hub fallback) | 85-90% |
| 2021+ | OA (Unpaywall/arXiv/CORE) | 25-35% |
| Overall | Multi-source | 75-80% |

## Limitations

- Not all papers are available through Sci-Hub or Unpaywall
- Unpaywall only covers open access papers
- Some publishers may block automated downloads
- Sci-Hub mirrors may change or become unavailable

## Use with AI Agents (MCP)

[paper-download-mcp](https://github.com/Oxidane-bot/paper-download-mcp) is a separate MCP server built on the same core and published on PyPI. It exposes paper downloading as a tool your agent can call. Install `uv` first (`uvx --version` to verify), then:

### Claude Code

```bash
claude mcp add --transport stdio --scope project --env PAPER_DOWNLOAD_EMAIL=your-email@university.edu paper-download -- uvx paper-download-mcp
```

### Claude Desktop

Edit MCP config (`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS, `%APPDATA%\Claude\claude_desktop_config.json` on Windows):

```json
{
  "mcpServers": {
    "paper-download": {
      "command": "uvx",
      "args": ["paper-download-mcp"],
      "env": {
        "PAPER_DOWNLOAD_EMAIL": "your-email@university.edu"
      }
    }
  }
}
```

### Codex

```bash
codex mcp add paper-download --env PAPER_DOWNLOAD_EMAIL=your-email@university.edu -- uvx paper-download-mcp
```

## Legal Disclaimer

This tool is provided for educational and research purposes only. Users are responsible for ensuring they comply with applicable laws and regulations when using this tool.

## Longrun Iteration Workflow

For the agent-driven longrun process that runs the downloader, inspects failure diagnostics, updates code or configuration, and keeps the best final version, see:

- `docs/longrun_benchmark.md`

## Testing

The project includes comprehensive tests for multi-source functionality:

### Running Tests

```bash
# Run all tests
cd tests
uv run python test_functionality.py
uv run python -m unittest test_metadata_utils.py -v

# Or run all unit tests
uv run python -m unittest discover -v
```

### Test Coverage

The test suite covers:

- ✅ **Multi-Source Download**: Tests OA-first routing with Sci-Hub fallback for pre-2021 papers
- ✅ **PDF Validation**: Verifies downloaded files have valid PDF headers
- ✅ **Mirror Connectivity**: Tests all Sci-Hub mirrors for accessibility
- ✅ **Metadata Extraction**: Tests Unpaywall API metadata retrieval
- ✅ **Filename Generation**: Tests filename sanitization and edge cases

### Recent Test Results

```
Multi-source download: 2/2 PASS
- 2013 paper (944 KB) via OA-first (Sci-Hub fallback if OA fails) ✓
- 2021 paper (1.6 MB) via OA (Sci-Hub skipped) ✓
PDF validation: All valid ✓
Metadata extraction: PASS ✓
```

## License

This project is licensed under the MIT License - see the LICENSE file for details. 
