Metadata-Version: 2.1
Name: amsterdam-schema-tools
Version: 0.16.3
Summary: Tools to work with Amsterdam schema.
Home-page: https://github.com/amsterdam/schema-tools
Author: Jan Murre
Author-email: jan.murre@catalyz.nl
License: Mozilla Public 2.0
Platform: UNKNOWN
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Requires-Dist: cachetools
Requires-Dist: click
Requires-Dist: geoalchemy2
Requires-Dist: jinja2
Requires-Dist: jsonpath-rw
Requires-Dist: jsonschema
Requires-Dist: mappyfile
Requires-Dist: ndjson (>=0.3.0)
Requires-Dist: pg-grant
Requires-Dist: psycopg2
Requires-Dist: python-dateutil
Requires-Dist: python-string-utils
Requires-Dist: requests
Requires-Dist: shapely
Provides-Extra: django
Requires-Dist: django-environ ; extra == 'django'
Requires-Dist: django-gisserver (>=0.5) ; extra == 'django'
Requires-Dist: django-postgres-unlimited-varchar (>=1.1.0) ; extra == 'django'
Requires-Dist: django (>=3.0.4) ; extra == 'django'
Provides-Extra: tests
Requires-Dist: flake8 ; extra == 'tests'
Requires-Dist: pytest ; extra == 'tests'
Requires-Dist: pytest-cov ; extra == 'tests'
Requires-Dist: pytest-django ; extra == 'tests'
Requires-Dist: pytest-sqlalchemy ; extra == 'tests'
Requires-Dist: requests-mock ; extra == 'tests'

# amsterdam-schema-tools

Set of libraries and tools to work with Amsterdam schema.

Install the package with: `pip install amsterdam-schema-tools`

Currently, the following cli commands are available:

- schema import events
- schema import ndjson
- schema show schema <dataset-id>
- schema show tablenames
- schema introspect db <dataset-id> <list-of-tablenames>
- schema introspect geojson <dataset-id> \*.geojson
- schema validate
- schema permissions apply

The tools expect either a `DATABASE_URL` environment variable or a command-line option `--db-url` with a DSN.

The output is a json-schema output according to the Amsterdam schemas
definition for the tables that are being processed.

## Generate amsterdam schema from existing database tables

The --prefix argument controls whether table prefixes are removed in the
schema, because that is required for Django models.

As example we can generate a BAG schema. Point `DATABASE_URL` to `bag_v11` database and then run :

    schema show tablenames | sort | awk '/^bag_/{print}' | xargs schema introspect db bag --prefix bag_ | jq

The **jq** formats it nicely and it can be redirected to the correct directory
in the schemas repository directly.

## Express amsterdam schema information in relational tables

Amsterdam schema is expressed as jsonschema. However, to make it easier for people with a
more relational mind- or toolset it is possible to express amsterdam schema as a set of
relational tables. These tables are *meta_dataset*, *meta_table* and *meta_field*.

It is possible to convert a jsonschema into the relational table structure and vice-versa.

This command converts a dataset from an existing dataset in jsonschema format:

    schema import schema <id of dataset>

To convert from relational tables back to jsonschema:

    schema show schema <id of dataset>


## Generating amsterdam schema from existing GeoJSON files

The following command can be used to inspect and import the GeoJSON files:

    schema introspect geojson <dataset-id> *.geojson > schema.json
    edit schema.json  # fine-tune the table names
    schema import geojson schema.json <table1> file1.geojson
    schema import geojson schema.json <table2> file2.geojson

## Importing GOB events

The schematools library has a module that read GOB events into database tables that are
defines by an Amsterdam schema. This module can be used to read GOB events from a Kafka stream.
It is also possible to read GOB events from a batch file with line-separeted events using:

    schema import events <path-to-dataset> <path-to-file-with-events>




