Metadata-Version: 2.0
Name: amo2kinto
Version: 1.3.1
Summary: Generate a blocklists.xml file from the Kinto collections.
Home-page: https://github.com/mozilla-services/amo2kinto
Author: Mozilla Services
Author-email: services-dev@mozilla.com
License: Apache License (2.0)
Keywords: web services
Platform: UNKNOWN
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: License :: OSI Approved :: Apache Software License
Requires-Dist: jsonschema
Requires-Dist: kinto-client
Requires-Dist: lxml
Requires-Dist: requests
Requires-Dist: six
Requires-Dist: xmltodict

amo2kinto
#########

.. image:: https://img.shields.io/travis/mozilla-services/amo2kinto/master.svg
        :target: https://travis-ci.org/mozilla-services/amo2kinto

.. image:: https://img.shields.io/pypi/v/amo2kinto.svg
        :target: https://pypi.python.org/pypi/amo2kinto

.. image:: https://coveralls.io/repos/mozilla-services/amo2kinto/badge.svg?branch=master
        :target: https://coveralls.io/r/mozilla-services/amo2kinto

- **json2kinto**: Import ``addons-server`` blocklist into kinto
- **kinto2xml**: Export the blocklist in the legacy XML format
- **xml-verifier**: Show the differences between the ``addons-server`` XML file and the ``kinto2xml`` generated one.


Installation
============

To install the release::

    pip install amo2kinto


To install the development environment::

    make install


System dependencies
-------------------

In order to build ``amo2kinto`` dependencies you may need the following libraries:

- Building **lxml** will require: ``libxml2-dev libxslt-dev python-dev``
- Building **cryptography** will require: ``build-essential libffi-dev libssl-dev python-dev``

Otherwise you will run into errors like:

- ``libxml/xmlversion.h: File not found``
- ``ffi.h: File not found``
- ``ssl.h: File not found``


Importing blocklists from the addons server into Kinto
======================================================

The `addons-server <https://github.com/mozilla/addons-server/>`_ is
able to export its blocklists database in JSON.

The script will create the bucket/collection with a given schema.

The last version of the schema file can be find here:
https://github.com/mozilla-services/amo-blocklist-ui/blob/master/amo-blocklist.json

You can use the ``make update-schemas`` command to fetch its last version.

You can use the ``json2kinto`` script to load this database into a
Kinto server::

    json2kinto --server http://localhost:8888/v1 \
               --addons-server https://addons.mozilla.org/  \
               --schema-file schemas.json


The script is able to synchronize (add new blocked items and remove old ones).

However if a blocked item already exists it won't be altered.

**json2kinto** gives you the ability to configure what you want to
load, with which user credentials and from which bucket and collection.

.. code-block::

    usage: json2kinto [-h] [--addons-server ADDONS_SERVER] [-s SERVER] [-a AUTH]
                      [-v] [-q] [-D] [-S SCHEMA_FILE] [--no-schema]
                      [--certificates-bucket CERTIFICATES_BUCKET]
                      [--certificates-collection CERTIFICATES_COLLECTION]
                      [--gfx-bucket GFX_BUCKET] [--gfx-collection GFX_COLLECTION]
                      [--addons-bucket ADDONS_BUCKET]
                      [--addons-collection ADDONS_COLLECTION]
                      [--plugins-bucket PLUGINS_BUCKET]
                      [--plugins-collection PLUGINS_COLLECTION]
                      [-C] [-G] [-A] [-P]

    Import the blocklists from the addons server into Kinto.

    optional arguments:
      -h, --help            show this help message and exit
      --addons-server ADDONS_SERVER
                            The addons server to import from
      -s SERVER, --server SERVER
                            The location of the remote server (with prefix)
      -a AUTH, --auth AUTH  BasicAuth token:my-secret
      -v, --verbose         Show all messages.
      -q, --quiet           Show only critical errors.
      -D, --debug           Show all messages, including debug messages.
      -S SCHEMA_FILE, --schema-file SCHEMA_FILE
                            JSON Schemas file
      --no-schema           Should we handle schemas
      --certificates-bucket CERTIFICATES_BUCKET
                            Bucket name for certificates
      --certificates-collection CERTIFICATES_COLLECTION
                            Collection name for certificates
      --gfx-bucket GFX_BUCKET
                            Bucket name for gfx
      --gfx-collection GFX_COLLECTION
                            Collection name for gfx
      --addons-bucket ADDONS_BUCKET
                            Bucket name for addons
      --addons-collection ADDONS_COLLECTION
                            Collection name for addon
      --plugins-bucket PLUGINS_BUCKET
                            Bucket name for plugins
      --plugins-collection PLUGINS_COLLECTION
                            Collection name for plugin
      -C, --certificates    Only import certificates
      -G, --gfx             Only import GFX drivers
      -A, --addons          Only import addons
      -P, --plugins         Only import plugins


Generate a blocklist.xml file from Kinto collections
====================================================

If you want to export blocklists stored in Kinto in the addons server XML export
format, you can use the ``kinto2xml`` script::

    kinto2xml -s http://localhost:8888/v1


**kinto2xml** gives you the ability to configure what you want to
export and in which bucket and collection are the data stored.

.. code-block::

    usage: kinto2xml [-h] [-s SERVER] [-a AUTH] [-v] [-q] [-D]
                     [--certificates-bucket CERTIFICATES_BUCKET]
                     [--certificates-collection CERTIFICATES_COLLECTION]
                     [--gfx-bucket GFX_BUCKET] [--gfx-collection GFX_COLLECTION]
                     [--addons-bucket ADDONS_BUCKET]
                     [--addons-collection ADDONS_COLLECTION]
                     [--plugins-bucket PLUGINS_BUCKET]
                     [--plugins-collection PLUGINS_COLLECTION]
                     [--api-version API_VERSION] [--app APP]
                     [--app-version APP_VERSION] [-o OUT]

    Build a blocklists.xml file from Kinto blocklists.

    optional arguments:
      -h, --help            show this help message and exit
      -s SERVER, --server SERVER
                            The location of the remote server (with prefix)
      -a AUTH, --auth AUTH  BasicAuth token:my-secret
      -v, --verbose         Show all messages.
      -q, --quiet           Show only critical errors.
      -D, --debug           Show all messages, including debug messages.
      --certificates-bucket CERTIFICATES_BUCKET
                            Bucket name for certificates
      --certificates-collection CERTIFICATES_COLLECTION
                            Collection name for certificates
      --gfx-bucket GFX_BUCKET
                            Bucket name for gfx
      --gfx-collection GFX_COLLECTION
                            Collection name for gfx
      --addons-bucket ADDONS_BUCKET
                            Bucket name for addons
      --addons-collection ADDONS_COLLECTION
                            Collection name for addon
      --plugins-bucket PLUGINS_BUCKET
                            Bucket name for plugins
      --plugins-collection PLUGINS_COLLECTION
                            Collection name for plugin
      --api-version API_VERSION
                            Targeted blocklists.xml APP id
      --app APP             Targeted blocklists.xml APP id
      --app-version APP_VERSION
                            The targetted app version
      -o OUT, --out OUT     Output XML file.


Show differences between two XML files
======================================

Once you've imported the data and exported them, you may want to
validate that both Kinto and addons server export the exact same data in XML.

You can use the ``xml-verifier`` command to validate that both files
are rendered the same way.

The **xml-verifier** command accept both files path and files URLs.


.. code-block::

    APPID="{ec8030f7-c20a-464f-9b0e-13a3a9e97384}"
    kinto2xml --app $APPID --app-version 46.0 -o generated-blocklists.xml
    xml-verifier https://blocklist.addons.mozilla.org/blocklist/3/$APPID/46.0/ generated-blocklists.xml


CHANGELOG
#########

This document describes changes between each past release.

1.3.1 (2016-07-06)
==================

**Bug fix**

- Fix patch_records properties.


1.3.0 (2016-07-06)
==================

- Update records that were changed (#37)


1.2.1 (2016-05-26)
==================

**Bug fix**

- kinto_client.delete_records API changed. (#35)


1.2.0 (2016-05-26)
==================

**New features**

- Let people specify ``--app-version`` and ``--api-version`` while using kinto2xml (#33)
- Add functional tests for both version of the XML file format. (#33)
- Configure the kinto-admin as well as the JSONSchema (#32)

**Bug fix**

- Group addons by the biggest guid (#33)


1.1.0 (2016-05-18)
==================

**Bug fix**

- In case there is a diff using xml-verifier, the command now returns an error code (#28)

**Internal changes**

- ``json2kinto`` does not set destination collections as publicly readable (#27)


1.0.0 (2016-05-12)
==================

**Breaking changes**

- kinto2xml was renamed amo2kinto (#21)
- New JSON Schema file format with a "collection" prefix (#22)

**New features**

- Use the schema to validate AMO records in the importer script. (#5)
- Warn if the server does not have the schema capability (#24)

**Internal changes**

- Document amo2kinto commands: json2kinto, kinto2xml and verifier. (#23)


0.1.0 (2016-04-27)
==================

**Initial version**

- Create collection with the definition of the JSON schema.
- Fetch AMO blocklists information from the /blocked/blocklists.json AMO endpoint.
- Handle import configuration on the CLI.
  - Bucket / Collection names
  - Remote AMO endpoint configuration
  - Schema file path configuration
  - Schema or not schema
  - Verbosity level
  - Server selection
  - Auth credentials
  - Importation type selection
- Support for kinto-signer triggering
- Full SSL support for Python 2.7
- Full Python 2.7 and Python 3.4/3.5 support
- Handle the enabled flag to activate records
- Makefile rule to update the schema definition
- Export kinto blocklists in XML blocklist file version 2
- Export kinto blocklists in XML blocklist file version 3
- XML verifier that create a diff of two XML files


