Metadata-Version: 2.2
Name: kmvahini
Version: 1.2.0
Summary: A Python package to scrape agricultural data from Karnataka's market website.
Home-page: https://github.com/patilmanojkumar/kmvahini
Author: Manojkumar Patil
Author-email: patil.manojkumar@hotmail.com
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: selenium
Requires-Dist: pandas
Requires-Dist: tqdm
Requires-Dist: lxml
Provides-Extra: cache
Requires-Dist: pyarrow>=10.0; extra == "cache"
Requires-Dist: filelock>=3.0; extra == "cache"
Provides-Extra: cloud
Requires-Dist: pyarrow>=10.0; extra == "cloud"
Requires-Dist: filelock>=3.0; extra == "cloud"
Requires-Dist: huggingface_hub>=0.16; extra == "cloud"
Provides-Extra: full
Requires-Dist: pyarrow>=10.0; extra == "full"
Requires-Dist: filelock>=3.0; extra == "full"
Requires-Dist: huggingface_hub>=0.16; extra == "full"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# kmvahini

[![PyPI version](https://img.shields.io/pypi/v/kmvahini.svg)](https://pypi.org/project/kmvahini/)
[![Python Versions](https://img.shields.io/pypi/pyversions/kmvahini.svg)](https://pypi.org/project/kmvahini/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

> **High-Performance Agricultural Market Data Toolkit for Karnataka (KSAMB / Krishi Marata Vahini)**

`kmvahini` provides an asynchronous and parallel interface to programmatically extract, clean, cache, and analyze daily agricultural commodity price and arrival data from the [Karnataka State Agricultural Marketing Board (KSAMB)](https://krama.karnataka.gov.in/).

---

## Key Features

- **⚡ High-Throughput Parallel Scraping**: Decomposes multi-year and multi-commodity queries into concurrent subtasks utilizing up to **80% of available CPU cores**, slashing scrape times by up to **20x** (e.g. 25 years of data extracted in ~40 seconds).
- **🛡️ Robust Error Recovery & Edge Handling**: Gracefully handles *"No Records Found"* periods, dropdown variations, and transient network timeouts with automatic retries and headless browser recycling.
- **🧹 Clean, Analysis-Ready Data**: Returns standardized `pandas.DataFrame` objects with formatted DatetimeIndex, forward-filled markets, and sanitized categorical values.
- **💾 Dual-Tier Caching (Local & Hugging Face Cloud)**: Built-in Parquet caching with automatic month-level deduplication and cloud synchronization.
- **📊 260+ Commodities & 160+ Markets**: Historical records dating from **2002 to present**.

---

## Installation

Install via pip:

```bash
pip install kmvahini
```

To install with cloud caching support:

```bash
pip install "kmvahini[cloud]"
```

For development / editable installation:

```bash
git clone https://github.com/patilmanojkumar/kmvahini.git
cd kmvahini
pip install -e .
```

---

## Quick Start

### 1. High-Speed Parallel Scraping with `scrape_website`

The core `scrape_website` function automatically parallelizes across available CPU cores:

```python
import calendar
from kmvahini.scraper import scrape_website

# Define parameters
years = [str(y) for y in range(2010, 2026)]
months = [m.upper() for m in calendar.month_name if m]  # All 12 months
commodities = ["SAFFLOWER", "SOYABEEN", "MAIZE"]
markets = ["AllMarkets"]

# Scrape using automatic ~80% CPU core parallelization
df = scrape_website(
    months=months,
    years=years,
    commodities=commodities,
    markets=markets,
    workers=None,        # None = auto 80% cores; or pass an explicit integer e.g. 32
    show_progress=True   # Unified tqdm progress bar
)

print(df.info())
print(df.head())
```

### 2. Incremental Caching API with `get_data`

For pipelines requiring incremental synchronization and local Parquet caching:

```python
import kmvahini as kmv

# Fetches from local cache if present, otherwise scrapes missing months
df = kmv.get_data(
    commodity="SAFFLOWER",
    market="AllMarkets",
    start_year=2020,
    end_year=2025
)
```

---

## Performance & Architecture

Under standard sequential scraping, querying 25 years of daily market records (300 monthly form submits) takes ~18 minutes per commodity due to single-browser request roundtrips and DOM rendering waits.

`kmvahini` introduces **`(Commodity × Year)` task decomposition** powered by Python's `ProcessPoolExecutor`:

```
User Query (e.g., SAFFLOWER, 2002–2026)
    │
    ├── Task 1: (SAFFLOWER, 2002) ──► Worker Process 1 (Headless Chrome)
    ├── Task 2: (SAFFLOWER, 2003) ──► Worker Process 2 (Headless Chrome)
    ├── ...
    └── Task 25: (SAFFLOWER, 2026) ─► Worker Process 25 (Headless Chrome)
    │
    ▼
Consolidated, Sorted & Cleaned DatetimeIndexed DataFrame (~40 seconds total)
```

### Benchmark on Multi-Core Workstations

| Query | Sequential Time | `kmvahini` Parallel (80% Cores) | Speedup |
|:---|:---:|:---:|:---:|
| 1 Commodity (25 Years) | ~18 min | **~42 seconds** | **~25x** |
| 5 Commodities (25 Years) | ~90 min | **~3.5 minutes** | **~26x** |

---

## API Reference

### `kmvahini.scraper.scrape_website`

```python
scrape_website(
    months: List[str],
    years: List[str],
    commodities: List[str],
    markets: List[str],
    workers: Optional[int] = None,
    show_progress: bool = True
) -> pd.DataFrame
```

- **`months`**: List of month names (case-insensitive: `"JANUARY"`, `"January"`, etc.).
- **`years`**: List of four-digit year strings (e.g. `["2023", "2024"]`).
- **`commodities`**: List of commodity names as displayed in KSAMB dropdowns.
- **`markets`**: List of market names (use `["AllMarkets"]` for all markets).
- **`workers`**: Number of parallel worker processes. Default `None` utilizes `int(cpu_count * 0.8)`. Set `workers=1` for sequential execution.
- **`show_progress`**: Set to `False` to silence progress indicators.

---

## Legal & Compliance Disclaimer

*Krishi Marata Vahini* is the registered trademark and service of the **Karnataka State Agricultural Marketing Board (KSAMB)** and the **Department of Agricultural Marketing, Government of Karnataka**.

`kmvahini` is an independent open-source research toolkit and is **not** affiliated, sponsored, or endorsed by KSAMB or the Government of Karnataka. Data retrieved via this package is subject to the source terms published at [KSAMB Terms of Use](https://krama.karnataka.gov.in/department.aspx?page=terms).

---

## License & Contributing

Distributed under the **MIT License**. See `LICENSE` for details.

- **Author**: Manojkumar Patil ([patil.manojkumar@hotmail.com](mailto:patil.manojkumar@hotmail.com))
- **Source Code**: [GitHub](https://github.com/patilmanojkumar/kmvahini)
- **Issue Tracker**: [GitHub Issues](https://github.com/patilmanojkumar/kmvahini/issues)
