Metadata-Version: 2.1
Name: Protego
Version: 0.1.dev0
Summary: Pure-Python robots.txt parser with support for modern conventions
Home-page: UNKNOWN
Author: Anubhav Patel
Author-email: anubhavp28@gmail.com
License: BSD
Description: # Protego
        
        ## Overview
        Protego is a pure-Python `robots.txt` parser with support for modern conventions.
        
        ## Requirements
        * Python 2.7 or Python 3.4+
        * Works on Linux, Windows, Mac OSX, BSD
        
        ## Install
        
        To install Protego, simply use pip:
        
        ```
        pip install protego
        ```
        
        ## Usage
        
        ```python
        >> from protego import Protego
        >> import requests
        >> r = requests.get('https://google.com/robots.txt')
        >> rp = Protego.parse(r.text)
        >> # That's it! We can now perform queries.
        >> rp.can_fetch('https://google.com/search', 'mybot')
        False
        >> rp.can_fetch('https://google.com/search/about', 'mybot')
        True
        >> list(rp.sitemaps)
        ['https://www.google.com/sitemap.xml']
        ```
Keywords: robots.txt,parser,robots,rep
Platform: UNKNOWN
Classifier: Framework :: Scrapy
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*
Description-Content-Type: text/markdown
