Metadata-Version: 2.4
Name: orcli
Version: 0.1.3
Summary: A Python client for interacting with OpenRefine
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE.md
Requires-Dist: requests>=2.28.0
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Dynamic: license-file

﻿
# orcli -- open-refine client

A Python client library for interacting with [OpenRefine](https://openrefine.org/) via its REST API.

For simple project creation, data transformation, metadata management and export operations.

Find the [orcli package](https://pypi.org/project/orcli/) on PyPI.

## Features

  - Create and delete projects from local files.
  - Retrieve and manage project metadata.
  - Apply OpenRefine operations individually or in batches.
  - Load operations from JSON files.
  - Export project data using OpenRefine's supported export formats (TSV, CSV, JSON).
  - Retrieve column information and project models and convert row data to Python lists.
  - Error handling with detailed response logging.

## Installation

### Requirements

Requires Python 3.10+ and a running OpenRefine server instance.

### Quick Start

1. Download:

Clone the repository (see below).

```bash
git clone https://github.com/rkraasch/orcli.git
cd orcli
```

Or download from [orcli from PyPI](https://pypi.org/project/orcli/) (see below).

```bash
python -m venv ./venv
./venv/bin/python -m pip install --upgrade pip
./venv/bin/python -m pip install orcli
./venv/bin/python -c "from orcli import Refine; print(Refine)"
```

2. Optionally run tests:

[Download](https://openrefine.org/download) and run OpenRefine, then run the tests as shown below.

```bash
python -m pytest tests/ -v
```
This creates temporary test projects in OpenRefine which should be cleaned up automatically.
If a project prefixed `pytest_orcli_` remains visible in the Open project tab (under [#open-project](http://127.0.0.1:3333/#open-project)), something went wrong.

3. Basic usage:

```python
from orcli import Refine

# Initialize the client
refine = Refine(base_url="http://127.0.0.1:3333")

# Create a project
project_id = refine.create_project("input_file.csv", "My Project")

# Get column names
columns = refine.get_column_names(project_id)
print(f"Columns: {columns}")

# Apply an operation
operation = {
    "op": "core/column-removal",
    "columnName": "unwanted_column",
    "description": "Remove column"
}
refine.apply_operation(operation, project_id)

# Export data
refine.export_data("output_file.tsv", fmt="tsv", project_id=project_id)

# Clean up
refine.delete_project(project_id)
```

## Examples

The example projects can be found in [./examples/](./examples/).

### Example 1: Access Project Metadata

Demonstrates how to access and modify OpenRefine project metadata.

See files at [./examples/01_access-project-metadata/](./examples/01_access-project-metadata/).

```python
# Set metadata
refine.set_project_metadata("name", "Project Name", project_id)

# Get all projects
projects = refine.get_all_projects_metadata()
for pid, metadata in projects.items():
    print(f"{metadata['name']} (ID: {pid})")

# Find by name
project_id = refine.get_project_id_by_name("Project Name")
```

### Example 2: Batch Operations

Apply multiple operations directly or load them from a JSON file.

See files at [./examples/02_batch-operations](./examples/02_batch-operations).

```python
operations = [
    {"op": "core/column-removal", "columnName": "col1"},
    {"op": "core/column-removal", "columnName": "col2"}
]

refine.apply_operations(operations, project_id, wait=True)

refine.apply_operations_from_file(
    "operations.json",
    project_id,
    wait=True,
)
```

### Example 3: Data Pipeline

Create a project, transform its data, export the result and clean up.

See files at [./examples/03_data-pipeline](./examples/03_data-pipeline).

```python
# Transform data
operations = [
    {
        "op": "core/column-removal",
        "columnName": "temp_field",
    },
    {
        "op": "core/text-transform",
        "engineConfig": {
            "facets": [],
            "mode": "row-based",
        },
        "columnName": "email",
        "expression": "value.toLowercase()",
        "onError": "keep-original",
        "repeat": False,
        "repeatCount": 10,
    },
]

refine.apply_operations(operations, project_id, wait=True)
```

### Example 4: Batch Processing

Process every CSV file in a directory using the same operations file.

See files at [./examples/04_batch-processing](./examples/04_batch-processing).

```python
for filename in os.listdir("input_dir/"):
    if filename.endswith(".csv"):
        project_id = refine.create_project(
            f"input_dir/{filename}",
            filename,
        )

        refine.apply_operations_from_file(
            "operations.json",
            project_id,
            wait=True,
        )

        refine.export_data(
            f"output_dir/{filename}",
            fmt="csv",
            project_id=project_id,
        )

        refine.delete_project(project_id)
```

### Example 5: Logging

Configure the Python logger to control the output produced by `orcli` at different log levels.

See files at [./examples/05_logging/](./examples/05_logging/).

```python
import logging

from orcli import Refine

logger = logging.getLogger("orcli.client")

handler = logging.StreamHandler()
handler.setFormatter(
    logging.Formatter("%(levelname)s: %(message)s")
)

logger.addHandler(handler)
logger.propagate = False

# DEBUG: show all diagnostic messages.
logger.setLevel(logging.DEBUG)

refine = Refine()

# Use Refine as usual.
project_id = refine.create_project(
    project_file="input.csv",
    project_name="Logging Example",
)

# INFO: suppress DEBUG messages.
logger.setLevel(logging.INFO)

# WARNING: suppress DEBUG and INFO messages.
logger.setLevel(logging.WARNING)

# ERROR: only show errors and critical messages.
logger.setLevel(logging.ERROR)
```

## API Reference

### Initialization

```python
Refine(base_url=None)
```

Parameters:
- base_url: OpenRefine server URL (default: http://127.0.0.1:3333)

### Methods

| Method | Description |
|--------|-------------|
| create_project(file, name) | Create project |
| delete_project(project_id) | Delete project |
| get_all_projects_metadata() | Get all projects |
| get_project_id_by_name(name) | Find project by name |
| set_project_metadata(field, value, id) | Update metadata |
| apply_operation(op, id, wait) | Apply operation |
| apply_operations(ops, id, wait) | Apply multiple |
| apply_operations_from_file(file, id, wait) | Load from file |
| get_models(id) | Get models |
| get_column_names(id) | Get columns |
| export_data(file, fmt, id) | Export data |
| rows_as_list(data) | Convert rows |
| wait_until_idle(id, delay) | Wait for completion |

## Configuration

### Logging

`orcli` uses Python's standard `logging` module. The library does not configure
logging output itself, allowing applications to decide which log messages to display.

See [Example 5: Logging](#example-5-logging) for a code example.

### Custom server

```python
refine = Refine(base_url="http://example.com:3333")
```

## Troubleshooting

  - **ConnectionError:** Ensure OpenRefine is running, verify the server URL and check network connectivity.
  - **CSRF token errors:** Check server status and logs.
  - **FileNotFoundError:** Use absolute paths, verify the file exists and check permissions.

## Support and Contribute

Open an issue on the project's [issues tab](https://github.com/rkraasch/orcli/issues) on Github.

Or contribute via Github:

  - fork the repository,
  - create a feature branch,
  - commit changes,
  - push to branch,
  - open Pull Request.

## References

References:

  - [OpenRefine Documentation](https://docs.openrefine.org/)
  - [OpenRefine REST API](https://docs.openrefine.org/manual/running)

Similar Projects:

  - [paulmakepeace/refine-client-py](https://github.com/paulmakepeace/refine-client-py): OpenRefine Python 2 Client (last update 11 years ago).
  - [opencultureconsulting/openrefine-client](https://github.com/opencultureconsulting/openrefine-client): OpenRefine Python Client (archived 2024).

## License

This project is released under [CC0 1.0 Universal License](https://creativecommons.org/publicdomain/zero/1.0/).

For a plain text version see this project's [LICENSE file](./LICENSE.md) or visit [creativecommons.org](https://creativecommons.org/2011/04/15/plaintext-versions-of-creative-commons-licenses-and-cc0/).

