Metadata-Version: 2.1
Name: scape-academy
Version: 0.0.1
Summary: A framework and an utility helps you to develop web scraping applications.
Home-page: https://github.com/sojin-project/scrape-academy
Author: Atsuo Ishimoto
License: MIT
Project-URL: Documentation, https://github.com/sojin-project/scrape-academy
Project-URL: Source, https://github.com/sojin-project/scrape-academy
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: License :: OSI Approved :: MIT License
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: aiohttp
Requires-Dist: aiolimiter
Requires-Dist: click
Provides-Extra: dev
Requires-Dist: wheel ; extra == 'dev'
Requires-Dist: twine ; extra == 'dev'
Requires-Dist: pytest ; extra == 'dev'
Requires-Dist: mypy ; extra == 'dev'
Requires-Dist: black ; extra == 'dev'
Requires-Dist: flake8 ; extra == 'dev'
Requires-Dist: autoflake ; extra == 'dev'
Requires-Dist: pre-commit ; extra == 'dev'

# Scrape Academy

Scrape Academy provides a framework and an utility helps you to develop web scraping applications.

## Simple web page  scraping

Scrape Academy helps you to download web pages to scrape.

```python
# Download a page from https://www.python.jp

from bs4 import BeautifulSoup
from scrapeacademy import context, run

async def run_simple():
    page = await context.get("https://www.python.jp")
    soup = BeautifulSoup(page, features="html.parser")
    print(soup.title.text)

run(run_simple())
```

`scrapeacademy.run()` starts [asyncio](https://docs.python.org/3/library/asyncio.html) event loop and run a scraping function.

In the async function, you can use `context.get()` method to download the page. The `context.get()` throttle the requests to the server. By default, `context.get()` waits 0.1 seconds between requests.

## Cache downloaded files

While developping the scraper, you usually need to investigate the HTML over and over. To help investigations, you can save the downloaded files to the cache directory.

The `context.get()` method saves downloaded file to the cache directory if `name` parameter is supplied.

```python
# Save https://www.python.jp

from scrapeacademy import context, run

async def save_index():
    page = await context.get("https://www.python.jp", name="python_jp_index")

run(run_simple())
```

Later, you can load the saved HTML from the cache to scrape using another script.

```python
# Parse saved HTML file.

from scrapeacademy import context

html = context.load("python_jp_index")
soup = BeautifulSoup(page, features="html.parser")
print(soup.title.text)
```

## Command line utility

Scrape Academy provides the `scrapeacademy` command to make development easier.

You can inspect the cached files with web browser.

```sh
$ scrapeacademy open python_jp_index
```

Or, you can view the file with vi editor as follow.

```sh
$ vi `scrapeacademy path python_jp_index`
```




