Metadata-Version: 2.4
Name: maxpreps-scraper
Version: 0.1.5
Summary: A Python scraper for MaxPreps high school sports data
Author-email: Raghav Dhir <dhir.raghav@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/raghavdhir03/maxpreps_scraper
Project-URL: Repository, https://github.com/raghavdhir03/maxpreps_scraper
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests
Requires-Dist: beautifulsoup4
Requires-Dist: tqdm
Requires-Dist: pandas
Requires-Dist: lxml
Requires-Dist: html5lib
Dynamic: license-file

# MaxPreps Web Scraper

This Python module scrapes high school sports data from [MaxPreps.com](https://www.maxpreps.com), enabling efficient extraction of team rankings and game results across states and sports.

## Features

- Retrieve team rankings with strength of schedule, rating, and team links.
- Scrape team schedules, game outcomes, rankings
- Includes address/location data for each team.
- Multi-threaded scraping for speed and efficiency.
- Cleans and structures game metadata (venue, game type, outcome, etc.).

## File Structure

- `maxpreps_scraper/scraper.py`: Contains the `MaxPrepsScraper` class.

## Installation

```bash
python -m pip install maxpreps-scraper
```
## Functions

##### `MaxPrepsScraper(requests_per_second=None, max_retries=0, backoff_factor=1.0, school_retry_attempts=0, school_retry_delay_seconds=5.0)`

Constructor. All parameters are optional and default to today's behavior (no
throttling, no retries).
- `requests_per_second` (`float`, optional): When set, throttles every HTTP
  request made through the scraper's shared session — including all threads
  in `get_contests()`'s and `get_districts()`'s thread pools — to at most
  this many requests per second, collectively (not per-thread).
- `max_retries` (`int`, optional): When > 0, retries individual requests that
  fail with status 429, 500, 502, 503, 504, or 403, using exponential
  backoff, before giving up.
- `backoff_factor` (`float`, optional): Backoff multiplier used between
  retries when `max_retries > 0`.
- `school_retry_attempts` (`int`, optional): Number of extra retry passes
  `get_contests()` performs over just the schools whose schedule page failed
  to scrape during the main pass.
- `school_retry_delay_seconds` (`float`, optional): Delay before each
  `get_contests()` school retry pass.

##### `get_rankings(state: str, sport: str, year: str, boys: bool = True)`

Retrieves state rankings for a given sport and academic year.
**Parameters:**
- `state` (`str`): Two-letter state abbreviation (e.g., `'tx'` for Texas)
- `sport` (`str`): Sport name (e.g., `'basketball'`)
- `year` (`str`): Academic year of the season (e.g., `'21-22'`)
- `boys` (`bool`, optional): Set to `True` for boys' sports and `False` for girls'. Defaults to `True`.

**Returns:**  
`pandas.DataFrame` containing:
- School Name
- State Rank
- Strength of Schedule (SOS)
- Team Rating
- Team URL

---

##### `get_contests(state: str, sport: str, year: str, boys: bool = True, cities: list = None)`

Scrapes all contests (games) for selected schools by state, sport, and year.
**Parameters:**
- `state` (`str`): Two-letter state abbreviation (e.g., `'tx'`)
- `sport` (`str`): Sport name (e.g., `'basketball'`)
- `year` (`str`): Academic year of the season (e.g., `'21-22'`)
- `boys` (`bool`, optional): Set to `True` for boys' sports and `False` for girls'. Defaults to `True`.
- `cities` (`list`, optional): A list of cities to filter for, e.g., `['austin', 'el paso']` (if omitted, full state will be scraped)

**Returns:**  
`pandas.DataFrame` containing game-level contest data, including:
- Date, opponent, venue, scores
- Team location details (address, city, state, zipcode)
- URLs to the MaxPreps pages of both teams

The returned DataFrame also carries scrape-completeness metadata in `.attrs`
(inert pandas metadata; ignored by callers who don't look for it):
- `schools_discovered`: number of schools found for the given filters
- `schools_scraped`: number of schools successfully scraped, after any
  `school_retry_attempts` retry passes
- `failed_schools`: list of `{"school": ..., "url": ...}` entries for schools
  that could not be scraped after retries were exhausted

#### Supported Sports and Limitations
These functions only support the following sports:
['basketball', 'football', 'baseball', 'soccer', 'volleyball', 'lacrosse', 'softball']

⚠️ Note: Soccer data is only available for the following states:
['tx', 'la', 'ms', 'hi', 'ca', 'fl', 'az']
## Usage and Example Output
```
from maxpreps_scraper.scraper import MaxPrepsScraper
scraper = MaxPrepsScraper()

# Get Team Rankings
rankings_df = scraper.get_rankings(state = 'de', sport = 'football', year = '23-24')

# Get Contest Data
contests_df = scraper.get_contests(state='tx', sport='basketball', year='21-22', boys=False, cities = ['san antonio'])

# Output dataframes
rankings_df.head(10)
contests_df.head(10)
```
![Get Rankings](images/rankings_demo.png)
![Get Contests](images/contests_demo.png)

## Testing

Run the deterministic tests with:

```bash
python -m pytest -q
```

The integration tests fetch a live MaxPreps schedule and may be skipped when
the site blocks automated requests:

```bash
python -m pytest -q tests/integration_tests.py
```

## Release Checklist

```bash
rm -rf build dist
python -m build
python -m twine check dist/*
python -m twine upload dist/*
```

## Version 0.1.5

- Added concurrent district scraping via `get_districts()`.
- Added optional session-wide rate limiting (`requests_per_second`) and
  retry-with-backoff (`max_retries`, `backoff_factor`) on the scraper's
  constructor, applied transparently to every request across all methods and
  thread pools.
- `get_contests()` now retries schools that failed to scrape
  (`school_retry_attempts`, `school_retry_delay_seconds`) and reports
  scrape completeness via `schools_discovered`/`schools_scraped`/
  `failed_schools` in the returned DataFrame's `.attrs`.

## Version 0.1.4

- Improved Team 1 and Team 2 name cleanup.
- Improved date and time normalization.
- Improved school URL and city filtering.
- Added deterministic unit tests and live integration tests.



