Metadata-Version: 1.1
Name: amo2kinto
Version: 1.7.0
Summary: Generate a blocklists.xml file from the Kinto collections.
Home-page: https://github.com/mozilla-services/amo2kinto
Author: Mozilla Services
Author-email: services-dev@mozilla.com
License: Apache License (2.0)
Description: amo2kinto
        #########
        
        .. image:: https://img.shields.io/travis/mozilla-services/amo2kinto/master.svg
                :target: https://travis-ci.org/mozilla-services/amo2kinto
        
        .. image:: https://img.shields.io/pypi/v/amo2kinto.svg
                :target: https://pypi.python.org/pypi/amo2kinto
        
        .. image:: https://coveralls.io/repos/mozilla-services/amo2kinto/badge.svg?branch=master
                :target: https://coveralls.io/r/mozilla-services/amo2kinto
        
        - **json2kinto**: Import ``addons-server`` blocklist into kinto
        - **kinto2xml**: Export the blocklist in the legacy XML format
        - **xml-verifier**: Show the differences between the ``addons-server`` XML file and the ``kinto2xml`` generated one.
        
        
        Installation
        ============
        
        To install the release::
        
            pip install amo2kinto
        
        
        To install the development environment::
        
            make install
        
        
        System dependencies
        -------------------
        
        In order to build ``amo2kinto`` dependencies you may need the following libraries:
        
        - Building **lxml** will require: ``libxml2-dev libxslt-dev python-dev``
        - Building **cryptography** will require: ``build-essential libffi-dev libssl-dev python-dev``
        
        Otherwise you will run into errors like:
        
        - ``libxml/xmlversion.h: File not found``
        - ``ffi.h: File not found``
        - ``ssl.h: File not found``
        
        
        Importing blocklists from the addons server into Kinto
        ======================================================
        
        The `addons-server <https://github.com/mozilla/addons-server/>`_ is
        able to export its blocklists database in JSON.
        
        The script will create the bucket/collection with a given schema.
        
        The last version of the schema file can be find here:
        https://github.com/mozilla-services/amo-blocklist-ui/blob/master/amo-blocklist.json
        
        You can use the ``make update-schemas`` command to fetch its last version.
        
        You can use the ``json2kinto`` script to load this database into a
        Kinto server::
        
            json2kinto --server http://localhost:8888/v1 \
                       --addons-server https://addons.mozilla.org/  \
                       --schema-file schemas.json
        
        
        The script is able to synchronize (add new blocked items and remove old ones).
        
        However if a blocked item already exists it won't be altered.
        
        **json2kinto** gives you the ability to configure what you want to
        load, with which user credentials and from which bucket and collection.
        
        .. code-block::
        
            usage: json2kinto [-h] [--addons-server ADDONS_SERVER] [-s SERVER] [-a AUTH]
                              [-v] [-q] [-D] [-S SCHEMA_FILE] [--no-schema]
                              [--editor-auth EDITOR_AUTH]
                              [--reviewer-auth REVIEWER_AUTH]
                              [--certificates-bucket CERTIFICATES_BUCKET]
                              [--certificates-collection CERTIFICATES_COLLECTION]
                              [--gfx-bucket GFX_BUCKET] [--gfx-collection GFX_COLLECTION]
                              [--addons-bucket ADDONS_BUCKET]
                              [--addons-collection ADDONS_COLLECTION]
                              [--plugins-bucket PLUGINS_BUCKET]
                              [--plugins-collection PLUGINS_COLLECTION]
                              [-C] [-G] [-A] [-P]
        
            Import the blocklists from the addons server into Kinto.
        
            optional arguments:
              -h, --help            show this help message and exit
              --addons-server ADDONS_SERVER
                                    The addons server to import from
              -s SERVER, --server SERVER
                                    The location of the remote server (with prefix)
              -a AUTH, --auth AUTH  BasicAuth token:my-secret
              -v, --verbose         Show all messages.
              -q, --quiet           Show only critical errors.
              -D, --debug           Show all messages, including debug messages.
              -S SCHEMA_FILE, --schema-file SCHEMA_FILE
                                    JSON Schemas file
              --no-schema           Should we handle schemas
              --editor-auth EDITOR_AUTH
                                    Credentials to be used for requesting a review
              --reviewer-auth REVIEWER_AUTH
                                    Credentials to be used for validating the review
              --certificates-bucket CERTIFICATES_BUCKET
                                    Bucket name for certificates
              --certificates-collection CERTIFICATES_COLLECTION
                                    Collection name for certificates
              --gfx-bucket GFX_BUCKET
                                    Bucket name for gfx
              --gfx-collection GFX_COLLECTION
                                    Collection name for gfx
              --addons-bucket ADDONS_BUCKET
                                    Bucket name for addons
              --addons-collection ADDONS_COLLECTION
                                    Collection name for addon
              --plugins-bucket PLUGINS_BUCKET
                                    Bucket name for plugins
              --plugins-collection PLUGINS_COLLECTION
                                    Collection name for plugin
              -C, --certificates    Only import certificates
              -G, --gfx             Only import GFX drivers
              -A, --addons          Only import addons
              -P, --plugins         Only import plugins
        
        
        Generate a blocklist.xml file from Kinto collections
        ====================================================
        
        If you want to export blocklists stored in Kinto in the addons server XML export
        format, you can use the ``kinto2xml`` script::
        
            kinto2xml -s http://localhost:8888/v1
        
        
        **kinto2xml** gives you the ability to configure what you want to
        export and in which bucket and collection are the data stored.
        
        .. code-block::
        
            usage: kinto2xml [-h] [-s SERVER] [-a AUTH] [-v] [-q] [-D]
                             [--certificates-bucket CERTIFICATES_BUCKET]
                             [--certificates-collection CERTIFICATES_COLLECTION]
                             [--gfx-bucket GFX_BUCKET] [--gfx-collection GFX_COLLECTION]
                             [--addons-bucket ADDONS_BUCKET]
                             [--addons-collection ADDONS_COLLECTION]
                             [--plugins-bucket PLUGINS_BUCKET]
                             [--plugins-collection PLUGINS_COLLECTION]
                             [--api-version API_VERSION] [--app APP]
                             [--app-version APP_VERSION] [-o OUT]
            
            Build a blocklists.xml file from Kinto blocklists.
            
            optional arguments:
              -h, --help            show this help message and exit
              -s SERVER, --server SERVER
                                    The location of the remote server (with prefix)
              -a AUTH, --auth AUTH  BasicAuth token:my-secret
              -v, --verbose         Show all messages.
              -q, --quiet           Show only critical errors.
              -D, --debug           Show all messages, including debug messages.
              --certificates-bucket CERTIFICATES_BUCKET
                                    Bucket name for certificates
              --certificates-collection CERTIFICATES_COLLECTION
                                    Collection name for certificates
              --gfx-bucket GFX_BUCKET
                                    Bucket name for gfx
              --gfx-collection GFX_COLLECTION
                                    Collection name for gfx
              --addons-bucket ADDONS_BUCKET
                                    Bucket name for addons
              --addons-collection ADDONS_COLLECTION
                                    Collection name for addon
              --plugins-bucket PLUGINS_BUCKET
                                    Bucket name for plugins
              --plugins-collection PLUGINS_COLLECTION
                                    Collection name for plugin
              --api-version API_VERSION
                                    Targeted blocklists.xml APP id
              --app APP             Targeted blocklists.xml APP id
              --app-version APP_VERSION
                                    The targetted app version
              -o OUT, --out OUT     Output XML file.
        
        
        Show differences between two XML files
        ======================================
        
        Once you've imported the data and exported them, you may want to
        validate that both Kinto and addons server export the exact same data in XML.
        
        You can use the ``xml-verifier`` command to validate that both files
        are rendered the same way.
        
        The **xml-verifier** command accept both files path and files URLs.
        
        
        .. code-block::
        
            APPID="{ec8030f7-c20a-464f-9b0e-13a3a9e97384}"
            kinto2xml --app $APPID --app-version 46.0 -o generated-blocklists.xml
            xml-verifier https://blocklist.addons.mozilla.org/blocklist/3/$APPID/46.0/ generated-blocklists.xml
        
        
        Generate blocked addons and plugins description pages
        =====================================================
        
        You might want to export the human readable list and description of
        add-ons and plugins that were blocked.
        
        You can do that using the ``blockpages-generator`` cli tool:
        
        .. code-block::
        
            usage: blockpages-generator [-h] [-s SERVER] [-a AUTH] [-b BUCKET] [-v] [-q]
                                        [-D] [--addons-collection ADDONS_COLLECTION]
                                        [--plugins-collection PLUGINS_COLLECTION]
                                        [-d TARGET_DIR]
            
            Generate blocked item description files.
            
            optional arguments:
              -h, --help            show this help message and exit
              -s SERVER, --server SERVER
                                    The location of the remote server (with prefix)
              -a AUTH, --auth AUTH  BasicAuth token:my-secret
              -b BUCKET, --bucket BUCKET
                                    Bucket name.
              -v, --verbose         Show all messages.
              -q, --quiet           Show only critical errors.
              -D, --debug           Show all messages, including debug messages.
              --addons-collection ADDONS_COLLECTION
                                    Collection name for addon
              --plugins-collection PLUGINS_COLLECTION
                                    Collection name for plugin
              -d TARGET_DIR, --target-dir TARGET_DIR
                                    Destination directory to write files in.
        
        It will generate an ``index.html`` file with the list of records
        present in the ``addons`` and ``plugins`` collections in the
        ``target-dir`` directory.
        
        It will also generate a file per ``add-on`` and ``plugin`` using the
        ``blockID`` or the ``id``. e.g ``i487.html`` or
        ``08db5018-2c80-4c4d-aa98-dafe6aacc28c.html``
        
        
        CHANGELOG
        #########
        
        This document describes changes between each past release.
        
        1.7.0 (2016-11-21)
        ==================
        
        **New features**
        
        - Retry downloading the XML in case it fails the first time. (#50)
        
        
        **Bug fix**
        
        - Remove redundant footer when generating pages. (#51)
        - Allow the importer to work even without the permission to create buckets. (#53)
        
        **Internal changes**
        
        - Improve the way the Makefile works (#52)
        
        
        1.6.0 (2016-10-04)
        ==================
        
        **New features**
        
        - Add a configurable signoff workflow to-review → to-sign (#48)
        
        
        1.5.1 (2016-09-08)
        ==================
        
        **Bug fix**
        
        - session_kwargs is not accessible anymore with last kinto-http client release. (#45)
        
        
        1.5.0 (2016-08-25)
        ==================
        
        - Add the ``blockpages-generator`` CLI tool (#43)
        
        
        1.4.1 (2016-08-02)
        ==================
        
        - Fix blockID ordering to make it the same as the addons-server.
        
        
        1.4.0 (2016-07-18)
        ==================
        
        **New features**
        
        - Add version ranges for GFX items (#39) â `Bug 1283601`_
        
        .. _`Bug 1283601`: https://bugzilla.mozilla.org/show_bug.cgi?id=1283601
        
        
        1.3.1 (2016-07-06)
        ==================
        
        **Bug fix**
        
        - Fix patch_records properties.
        
        
        1.3.0 (2016-07-06)
        ==================
        
        - Update records that were changed (#37)
        
        
        1.2.1 (2016-05-26)
        ==================
        
        **Bug fix**
        
        - kinto_client.delete_records API changed. (#35)
        
        
        1.2.0 (2016-05-26)
        ==================
        
        **New features**
        
        - Let people specify ``--app-version`` and ``--api-version`` while using kinto2xml (#33)
        - Add functional tests for both version of the XML file format. (#33)
        - Configure the kinto-admin as well as the JSONSchema (#32)
        
        **Bug fix**
        
        - Group addons by the biggest guid (#33)
        
        
        1.1.0 (2016-05-18)
        ==================
        
        **Bug fix**
        
        - In case there is a diff using xml-verifier, the command now returns an error code (#28)
        
        **Internal changes**
        
        - ``json2kinto`` does not set destination collections as publicly readable (#27)
        
        
        1.0.0 (2016-05-12)
        ==================
        
        **Breaking changes**
        
        - kinto2xml was renamed amo2kinto (#21)
        - New JSON Schema file format with a "collection" prefix (#22)
        
        **New features**
        
        - Use the schema to validate AMO records in the importer script. (#5)
        - Warn if the server does not have the schema capability (#24)
        
        **Internal changes**
        
        - Document amo2kinto commands: json2kinto, kinto2xml and verifier. (#23)
        
        
        0.1.0 (2016-04-27)
        ==================
        
        **Initial version**
        
        - Create collection with the definition of the JSON schema.
        - Fetch AMO blocklists information from the /blocked/blocklists.json AMO endpoint.
        - Handle import configuration on the CLI.
          - Bucket / Collection names
          - Remote AMO endpoint configuration
          - Schema file path configuration
          - Schema or not schema
          - Verbosity level
          - Server selection
          - Auth credentials
          - Importation type selection
        - Support for kinto-signer triggering
        - Full SSL support for Python 2.7
        - Full Python 2.7 and Python 3.4/3.5 support
        - Handle the enabled flag to activate records
        - Makefile rule to update the schema definition
        - Export kinto blocklists in XML blocklist file version 2
        - Export kinto blocklists in XML blocklist file version 3
        - XML verifier that create a diff of two XML files
        
Keywords: web services
Platform: UNKNOWN
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: License :: OSI Approved :: Apache Software License
