Metadata-Version: 2.1
Name: mathgenproject-jgolden17
Version: 0.1.0
Summary: Mathematics Genealogy Project Spider
Home-page: https://github.com/jgolden17/mathgenproject
Author: Jonathan Golden
License: UNKNOWN
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown

# mathgenproject
> Mathematics Genealogy Project Spider

A webspider for the [Mathematics Genealogy Project](https://genealogy.math.ndsu.nodak.edu/index.php).

## Installation

Use the package manager pip to install the spider.

```bash
pip install mathgenproject
```

## Usage

Define a pipeline through which to process each mathematician returned from the spider.

```python
class MyPipeline(object):
    def open_spider(self, spider):
        ...

    def process_item(self, item, spider):
        print(item['name'])
        return item

    def close_spider(self, spider):
        ...

```

Run the spider using scrapy's `CrawlerProcess`, passing in the mathematician's MGP ID.

```python
from scrapy.crawler import CrawlerProcess
from mathgenproject.spiders import MathGenProjectSpider

process = CrawlerProcess(settings={
    'FEED_FORMAT': 'json',
    'FEED_URI': 'items.json',
    'ITEM_PIPELINES': {
        'MyPipeline': 300,
    },
})

process.crawl(MathGenProjectSpider, mgp_id='216087')
process.start()
```

## Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

## License

[MIT](https://choosealicense.com/licenses/mit/)


