Metadata-Version: 2.0
Name: cdx-toolkit
Version: 0.9.2
Summary: A toolkit for working with CDX indices
Home-page: https://github.com/cocrawler/cdx_toolkit
Author: Greg Lindahl and others
Author-email: lindahl@pbm.com
License: Apache 2.0
Description-Content-Type: UNKNOWN
Platform: UNKNOWN
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3 :: Only

cdx\_toolkit
============

cdx\_toolkit is a set of tools for working with CDX indices of web
crawls and archives, including those at CommonCrawl and the Internet
Archive's Wayback Machine.

CommonCrawl uses Ilya Kramer's pywb to serve the CDX API, which is
somewhat different from the Internet Archive's CDX API. cdx\_toolkit
hides these differences as best it can. cdx\_toolkit also knits together
the monthly Common Crawl CDX indices into a single, virtual index.

https://github.com/webrecorder/pywb/wiki/CDX-Server-API
https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server

Example
-------

::

    import cdx_toolkit

    cdx = cdx_toolkit.CDXFetcher(source='cc', cc_duration='90d')
    url = 'commoncrawl.org/*'

    print(url, 'size estimate', cdx.get_size_estimate(url))

    for obj in cdx.items(url, limit=10):
        print(obj)

at the moment will print:

::

    size estimate 6000
    http://commoncrawl.org/ 200
    http://commoncrawl.org/ 200
    http://commoncrawl.org/ 200
    http://www.commoncrawl.org/ 301
    https://www.commoncrawl.org/ 301
    http://www.commoncrawl.org/ 301
    http://commoncrawl.org/ 200
    http://commoncrawl.org/2011/12/mapreduce-for-the-masses/ 200
    http://commoncrawl.org/2012/03/data-2-0-summit/ 200
    http://commoncrawl.org/2012/03/twelve-steps-to-running-your-ruby-code-across-five-billion-web-pages/ 200

Command-line tools
------------------

The above example can also be done as

::

    $ cdx_size.py 'commoncrawl.org/*' --cc
    $ cdx_iter.py 'commoncrawl.org/*' --cc --limit 10 --cc-duration='90d'

or

::

    $ cdx_size.py 'commoncrawl.org/*' --ia
    $ cdx_iter.py 'commoncrawl.org/*' --ia --limit 10

cdx\_iter can generate jsonl or csv outputs; see

::

    $ cdx_iter.py --help

for details.

Status
------

cdx\_toolkit has reached the "I hacked this together out of some other
code for a hackathon this weekend" stage of development.

License
-------

Apache 2.0


