Metadata-Version: 2.1
Name: datasette-block-robots
Version: 1.0
Summary: Datasette plugin that blocks all robots using robots.txt
Home-page: https://github.com/simonw/datasette-block-robots
Author: Simon Willison
License: Apache License, Version 2.0
Project-URL: Issues, https://github.com/simonw/datasette-block-robots/issues
Project-URL: CI, https://github.com/simonw/datasette-block-robots/actions
Project-URL: Changelog, https://github.com/simonw/datasette-block-robots/releases
Platform: UNKNOWN
Description-Content-Type: text/markdown
Requires-Dist: datasette (>=0.50)
Provides-Extra: test
Requires-Dist: pytest ; extra == 'test'
Requires-Dist: pytest-asyncio ; extra == 'test'
Requires-Dist: httpx ; extra == 'test'

# datasette-block-robots

[![PyPI](https://img.shields.io/pypi/v/datasette-block-robots.svg)](https://pypi.org/project/datasette-block-robots/)
[![Changelog](https://img.shields.io/github/v/release/simonw/datasette-block-robots?label=changelog)](https://github.com/simonw/datasette-block-robots/releases)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-block-robots/blob/master/LICENSE)

Datasette plugin that blocks robots and crawlers using robots.txt

## Installation

Install this plugin in the same environment as Datasette.

    $ pip install datasette-block-robots

## Usage

Having installed the plugin, `/robots.txt` on your Datasette instance will return the following:

    User-agent: *
    Disallow: /

This will request all robots and crawlers not to visit any of the pages on your site.

Here's a demo of the plugin in action: https://sqlite-generate-demo.datasette.io/robots.txt

## Configuration

By default the plugin will block all access to the site, using `Disallow: /`.

If you want the index page to be indexed by search engines without crawling the database, table or row pages themselves, you can use the following:

```json
{
    "plugins": {
        "datasette-block-robots": {
            "allow_only_index": true
        }
    }
}
```
This will return a `/robots.txt` like so:

    User-agent: *
    Disallow: /db1
    Disallow: /db2

With a `Disallow` line for every attached database.

To block access to specific areas of the site using custom paths, add this to your `metadata.json` configuration file:

```json
{
    "plugins": {
        "datasette-block-robots": {
            "disallow": ["/mydatabase/mytable"]
        }
    }
}
```
This will result in a `/robots.txt` that looks like this:

    User-agent: *
    Disallow: /mydatabase/mytable

Alternatively you can set the full contents of the `robots.txt` file using the `literal` configuration option. Here's how to do that if you are using YAML rather than JSON and have a `metadata.yml` file:

```yaml
plugins:
    datasette-block-robots:
        literal: |-
            User-agent: *
            Disallow: /
            User-agent: Bingbot
            User-agent: Googlebot
            Disallow:
```
This example would block all crawlers with the exception of Googlebot and Bingbot, which are allowed to crawl the entire site.

## Development

To set up this plugin locally, first checkout the code. Then create a new virtual environment:

    cd datasette-block-robots
    python3 -mvenv venv
    source venv/bin/activate

Or if you are using `pipenv`:

    pipenv shell

Now install the dependencies and tests:

    pip install -e '.[test]'

To run the tests:

    pytest


