Metadata-Version: 2.4
Name: grabber-cli
Version: 2.0.0
Summary: Template-based web crawler with async support — grab any site with one command
Author: unohee
Maintainer: unohee
License: MIT
Project-URL: Homepage, https://github.com/unohee/grabber
Project-URL: Bug Reports, https://github.com/unohee/grabber/issues
Project-URL: Source, https://github.com/unohee/grabber
Keywords: web-scraping,crawler,template,news-crawler,stock-data,beautifulsoup,selenium,automated-scraping,data-extraction,async-crawler,high-performance,bot-detection-bypass,aiohttp,concurrent-crawling
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Topic :: Office/Business :: Financial :: Investment
Classifier: Topic :: Utilities
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28.0
Requires-Dist: beautifulsoup4>=4.11.0
Requires-Dist: lxml>=4.9.0
Requires-Dist: numpy>=1.22.0
Requires-Dist: scikit-learn>=1.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: aiohttp>=3.8.0
Requires-Dist: aiodns>=3.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: tqdm>=4.64.0
Requires-Dist: click>=8.0.0
Provides-Extra: js
Requires-Dist: playwright>=1.40.0; extra == "js"
Provides-Extra: selenium
Requires-Dist: selenium>=4.18.1; extra == "selenium"
Requires-Dist: webdriver-manager>=4.0.1; extra == "selenium"
Requires-Dist: undetected-chromedriver>=3.5.0; extra == "selenium"
Provides-Extra: nlp
Requires-Dist: konlpy>=0.6.0; extra == "nlp"
Provides-Extra: all
Requires-Dist: grabber-cli[js,nlp,selenium]; extra == "all"
Requires-Dist: pandas>=1.5.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: pytest-mock>=3.10.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: black>=22.0.0; extra == "dev"
Requires-Dist: flake8>=5.0.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: sphinx>=5.0.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=1.0.0; extra == "docs"
Provides-Extra: performance
Requires-Dist: psutil>=5.9.0; extra == "performance"
Requires-Dist: memory-profiler>=0.60.0; extra == "performance"
Requires-Dist: line-profiler>=4.0.0; extra == "performance"
Dynamic: license-file

# Grabber

[![Python](https://img.shields.io/badge/python-3.8%2B-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

A flexible and extensible template-based web crawler with automatic site detection. Easily crawl news articles, stock data, and more from Korean websites.

## ✨ Features

- 🎯 **Automatic Template Detection**: Automatically selects the optimal template based on URL
- 🚀 **Simple Interface**: One-line crawling with the `Grabber` class
- ⚡ **High-Performance Async**: Optimized async crawling up to 10,000+ RPS locally
- 📰 **30+ Supported Sites**: Major Korean news sites and financial platforms
- 🔧 **Extensible**: Easy to add custom templates for new sites
- 💾 **Multiple Export Formats**: Save as JSON, CSV, or TXT
- 🔄 **Batch Processing**: Crawl multiple URLs efficiently with connection pooling
- 🛡️ **Smart Bot Detection Bypass**: Advanced header management and session persistence
- 🎛️ **Adaptive Performance**: Dynamic concurrency and delay adjustment based on site response
- 📊 **Site Profiling**: Pre-configured optimal settings for each news site
- 🖥️ **Mode-based CLI**: One `grabber` command for HTML, news, audio and site-exploration modes
- 🤖 **Agent-friendly**: JSON/NDJSON output, meaningful exit codes, and a self-describing `capabilities` command

## 📦 Installation

```bash
git clone https://github.com/unohee/grabber.git
cd grabber
pip install -e .

# With JS rendering support (SPA/SSR news sites)
pip install -e ".[js]" && playwright install chromium
```

## 🖥️ Command Line

Installing the package puts a `grabber` command on your PATH. Modes are
subcommand groups:

```bash
grabber doctor                 # check binaries, optional extras and credentials
grabber grab <URL>             # HTML / article scraping
grabber news search "반도체"    # keyword search via the Naver News API
grabber audio get <URL>        # YouTube / SoundCloud audio download
grabber explore run suno       # Suno / Udio media discovery
grabber probe <URL>            # which template and backend would handle this URL?
grabber capabilities           # dump the whole command surface as JSON
```

### HTML mode

```bash
# Single URL, human output
grabber grab "https://www.newsis.com/view/NISX20240101_0002"

# Batch from a file, 8 workers, machine-readable stream
grabber grab -f urls.txt -w 8 --ndjson > out.jsonl

# From stdin, trimmed for an LLM's context budget
cat urls.txt | grabber grab -f - --json --max-chars 2000 --fields url,title,content

# Export to CSV / Markdown (format inferred from the extension)
grabber grab "$URL" -o article.md
```

### News mode

Needs `NAVER_CLIENT_ID` / `NAVER_CLIENT_SECRET` in `.env` (or `--env-file`).

```bash
grabber news search "삼성전자" --days 3 --max 50 --json
grabber news collect "환율" --max 10 --max-chars 1500 -o out.md   # search + scrape
```

### Audio mode

Requires [`yt-dlp`](https://github.com/yt-dlp/yt-dlp) on PATH. Downloads go to
`$GRABBER_DATA_DIR/audio/<source>/` (default `~/.grabber`), or `-d <dir>`.

```bash
grabber audio info "https://youtu.be/VIDEOID" --json      # metadata only
grabber audio search "lofi piano" -n 10 --urls-only       # SoundCloud search
grabber audio get "https://youtu.be/VIDEOID" -d ./downloads

# Long corpus runs: queue once, drain repeatedly (resume-safe)
grabber audio queue -f songs.csv --source youtube
grabber audio batch --source youtube -w 4 --limit 500
```

`audio queue` accepts a plain URL list or a CSV carrying a `youtube_id`,
`video_id`, `track_id` or `url` column.

Connections are direct by default; pass `--proxy URL` or `--proxy-pool` to route
through a proxy. Recent yt-dlp versions need a JavaScript runtime (e.g. `deno`)
for YouTube downloads — `grabber doctor` flags this. SoundCloud is unaffected.

### Explore mode

Requires the `js` extra (`pip install -e ".[js]" && playwright install chromium`).

```bash
grabber explore run suno --pages 20 --batches 3
grabber explore stats suno --json
grabber explore export suno --type audio -o suno_audio.txt
grabber explore download suno --type audio --limit 100
```

### For agents

```bash
grabber capabilities --json     # every command, option, type, default and choice
```

stdout carries only the payload; progress and logs go to stderr. Exit codes:
`0` success · `1` partial · `2` usage error · `3` all failed · `4` missing
dependency or credentials · `5` no results. Full contract — record schema, error
codes, recipes — in [docs/AGENT_CLI.md](docs/AGENT_CLI.md).

## 🚀 Quick Start

### Simple Usage

```python
from grabber import Grabber

# One-line crawling
data = Grabber.quick_grab("https://www.ajunews.com/view/20240101000000000")

if data:
    print(f"Title: {data.data.get('title')}")
    print(f"Content: {data.data.get('content')}")
```

### Basic Usage

```python
from grabber import Grabber

# Create a Grabber instance
grabber = Grabber()

# Crawl a news article (automatic template detection)
data = grabber.grab("https://n.news.naver.com/article/001/0014000000")

if data:
    print(f"Source: {data.source}")
    print(f"Title: {data.data.get('title')}")
    print(f"Content: {data.data.get('content')}")
```

### Batch Crawling

```python
# Crawl multiple URLs
urls = [
    "https://www.ajunews.com/view/20240101000000000",
    "https://www.businesspost.co.kr/BP?command=article_view&num=123456",
    "https://www.thebell.co.kr/free/content/ArticleView.asp?key=202401010000000000"
]

results = grabber.grab_batch(urls)

for result in results:
    if result:
        print(f"{result.source}: {result.data.get('title')}")
```

### Stock Data Crawling

```python
# Crawl stock information
data = grabber.grab_stock("005930", source="naver")  # Samsung Electronics

if data:
    print(f"Company: {data.data.get('company_name')}")
    print(f"Price: {data.data.get('current_price')}")
```

### 🆕 Asynchronous Crawling (High Performance)

```python
import asyncio
from grabber.core import AsyncTemplateCrawler
from templates.async_naver_news_template import AsyncNaverNewsTemplate

async def async_crawl():
    # Create async template with optimization
    template = AsyncNaverNewsTemplate()
    
    # Create async crawler with performance tuning
    async with AsyncTemplateCrawler(
        template=template, 
        max_concurrent=32,  # Optimized for Ryzen 5800X
        delay=0.3,          # Optimal delay for news sites
        timeout=30
    ) as crawler:
        # Single crawl
        result = await crawler.crawl("인공지능")
        
        # Multiple crawls concurrently  
        keywords = ["AI", "머신러닝", "딥러닝"]
        results = await crawler.crawl_multiple(keywords)
        
        # Performance: 10x faster than synchronous
        print(f"Crawled {len(results)} keywords")
        
# Run async crawling
asyncio.run(async_crawl())
```

## 📋 Supported Sites

### News Sites
- 아주뉴스 (ajunews.com)
- 비즈니스포스트 (businesspost.co.kr)
- 이데일리 (edaily.co.kr)
- 이투데이 (etoday.co.kr)
- 한국경제 (hankyung.com)
- 매일경제 (mk.co.kr)
- 네이버 뉴스 (news.naver.com)
- 뉴시스 (newsis.com)
- 뉴스핌 (newspim.com)
- 더벨 (thebell.co.kr)
- And 20+ more...

### Financial Sites
- 네이버 금융 (finance.naver.com)
- FnGuide (comp.fnguide.com)

## 🔧 Advanced Usage

### Custom Template

```python
from grabber import Grabber, SiteTemplate

class MyCustomTemplate(SiteTemplate):
    def get_site_name(self):
        return "MyCustomSite"
    
    def build_url(self, target):
        return f"https://mycustomsite.com/{target}"
    
    def extract_data(self, soup):
        return {
            "title": soup.find("h1").text,
            "content": soup.find("article").text
        }

# Use custom template
grabber = Grabber(template=MyCustomTemplate)
data = grabber.grab("article/123")
```

### Save Results

```python
# Save crawled data
data = grabber.grab(url)

if data:
    # Save as JSON
    grabber.save_to_file(data, "output.json", format="json")
    
    # Save as CSV
    grabber.save_to_file(data, "output.csv", format="csv")
    
    # Save as TXT
    grabber.save_to_file(data, "output.txt", format="txt")
```

### List Available Templates

```python
# Get all available templates
templates = grabber.list_templates()
for name, site_name in templates.items():
    print(f"{name}: {site_name}")

# Get supported domains
domains = grabber.get_supported_domains()
print(f"Supported domains: {domains}")
```

## 🏗️ Architecture

```
grabber/
├── grabber/     # Core package
│   ├── core/            # Core components
│   ├── grabber.py       # Main interface
│   └── tools/           # Utilities
├── templates/           # Site-specific templates
│   ├── *_template.py    # Individual site templates
│   ├── extractors/      # Data extractors
│   └── validators/      # Data validators
└── examples/            # Usage examples
```

## 🧪 Testing

```bash
# Install development dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run with coverage
pytest --cov=grabber
```

## 🤝 Contributing

Contributions are welcome! To add support for a new site:

1. Create a new template in `templates/` directory
2. Inherit from `SiteTemplate` class
3. Implement required methods:
   - `get_site_name()`
   - `build_url(target)`
   - `extract_data(soup)`
   - `clean_data(raw_data)`
   - `validate_data(data)`

Example:
```python
from grabber.core.site_template import SiteTemplate

class NewSiteTemplate(SiteTemplate):
    def get_site_name(self):
        return "NewSite"
    
    def build_url(self, target):
        return f"https://newsite.com/{target}"
    
    def extract_data(self, soup):
        # Implementation here
        pass
```

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- Built with BeautifulSoup4 and Selenium
- Uses undetected-chromedriver for anti-detection
- Inspired by the need for flexible web scraping solutions

## 📧 Contact

- Issues: [GitHub Issues](https://github.com/unohee/grabber/issues)

## 🚀 Performance Optimization

### System Requirements
- **CPU**: Multi-core processor recommended (tested on AMD Ryzen 7 5800X)
- **RAM**: 4GB+ recommended for large-scale crawling
- **Network**: Stable internet connection with low latency

### Performance Benchmarks
| Site | RPS | Concurrency | Success Rate |
|------|-----|-------------|--------------|
| Naver News | 11.9 | 32 | 100% |
| HTTPBin (local) | 10,544 | 128 | 100% |
| General News | 7-15 | 16-32 | 95%+ |

### Optimization Tips
1. **Use async crawling** for I/O-bound operations
2. **Adjust concurrency** based on target site's rate limits
3. **Enable connection pooling** for batch operations
4. **Use site profiling** for automatic optimization

## 📈 Roadmap

- [ ] Add more news sites
- [ ] Support for international sites
- [x] Async crawling support (v1.1.0)
- [x] Performance optimization for news sites (v1.2.0)
- [x] Smart bot detection bypass (v1.2.0)
- [ ] REST API interface
- [ ] Docker support
- [ ] Cloud deployment guides
- [ ] Distributed crawling support

---

Made with ❤️ by unohee
