Metadata-Version: 2.4
Name: wbdutil
Version: 0.2.0
Summary: OSDU Wellbore DDMS Data Loader - command line interface
Author-email: "BP, SLB" <greg.harris1@bp.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader
Project-URL: Repository, https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader
Project-URL: Issues, https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues
Keywords: osdu,wellbore,welllog,las,ddms,energy
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering
Classifier: Operating System :: OS Independent
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: anyio
Requires-Dist: argcomplete
Requires-Dist: certifi
Requires-Dist: charset-normalizer
Requires-Dist: colorama
Requires-Dist: h11
Requires-Dist: httpcore
Requires-Dist: httpx
Requires-Dist: idna
Requires-Dist: jmespath
Requires-Dist: knack
Requires-Dist: lasio
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: pyarrow
Requires-Dist: pygments
Requires-Dist: python-dateutil
Requires-Dist: pytz
Requires-Dist: pyyaml
Requires-Dist: rfc3986[idna2008]
Requires-Dist: six
Requires-Dist: sniffio
Requires-Dist: tabulate
Dynamic: license-file

# Data Loader Utility for Wellbore DDMS (`wbdutil`)

`wbdutil` is a command line utility for loading well log data into, and retrieving it
from, an [OSDU&trade;](https://osduforum.org/) platform via the
[Wellbore Domain Data Management Services (Wellbore DDMS) API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/master/spec/generated/openapi.json).

It acts as a bridge between the industry-standard [LAS](https://www.cwls.org/products/#products-las)
(Log ASCII Standard) file format and OSDU wellbore / well log records:

-   **Ingest** LAS files (single files or whole folders) as OSDU wellbore and well log records, including bulk curve data.
-   **Download** OSDU well log and curve data back out to LAS files.
-   **List / search** wellbore and well log records held in an OSDU instance.
-   **Parse** LAS files offline (a dry run of ingest) to inspect the JSON that would be uploaded.
-   **Update** the curve families of an existing well log.
-   Fully **customisable mappings** between LAS data and OSDU kinds (see [Custom mappings](#custom-mappings)).

## Table of contents

-   [Requirements](#requirements)
-   [Installation](#installation)
-   [Quick start](#quick-start)
-   [Authentication and configuration](#authentication-and-configuration)
    -   [Environment variables](#environment-variables)
    -   [Config file](#config-file)
-   [Command reference](#command-reference)
-   [Custom mappings](#custom-mappings)
-   [Development](#development)

## Requirements

-   **Python 3.10 or newer** (3.10 – 3.13 are tested). Python 3.10 is the minimum so that
    `wbdutil` can also run inside Microsoft Fabric notebooks.
-   Network access to the target OSDU / Wellbore DDMS instance.
-   A valid OSDU bearer token and a [configuration file](#config-file) describing the instance
    and your legal / ACL settings.

## Installation

Install the latest release from PyPI:

```
pip install wbdutil
```

Alternatively, install from the OSDU community package registry (useful for pre-release builds):

```
pip install wbdutil --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple
```

To install into an isolated environment, use a virtual environment or a tool such as
[`uv`](https://docs.astral.sh/uv/) or [`pipx`](https://pipx.pypa.io/):

```
uv tool install wbdutil          # or: pipx install wbdutil
```

Verify the installation:

```
wbdutil --version
```

## Quick start

```bash
# 1. Point wbdutil at your configuration file and OSDU token.
export CONFIGPATH=./config.json
export OSDUTOKEN=<your-bearer-token>

# 2. Do an offline dry run to see what would be ingested (no upload, no token needed).
wbdutil parse convert --path ./well.las --wellbore_id "opendes:master-data--Wellbore:abc123"

# 3. Ingest the wellbore + well log into OSDU.
wbdutil ingest wellbore --path ./well.las

# 4. Read data back out of OSDU as a LAS file.
wbdutil download welllog --welllog_id "opendes:work-product-component--WellLog:xyz789" --out ./out.las
```

Every group and command supports `-h` / `--help`, e.g. `wbdutil ingest -h` or
`wbdutil download welllog -h`.

## Authentication and configuration

`wbdutil` requires a [configuration file](#config-file) that describes the target OSDU
instance. Its path must be supplied either through the `CONFIGPATH` environment variable or
the `--config_path` / `-c` option on each command.

All commands that connect to an OSDU instance (everything except `parse`) also require a
bearer token, supplied either through the `OSDUTOKEN` environment variable or the `--token` /
`-t` option.

### Environment variables

| Variable name | Overriding command option | Comment                                                    |
| ------------- | ------------------------- | ---------------------------------------------------------- |
| OSDUTOKEN     | `--token` `-t`            | The JWT required to authenticate against an OSDU instance. |
| CONFIGPATH    | `--config_path` `-c`      | The path to the configuration file.                        |

### Config file

The `wbdutil` requires a configuration file that has the following JSON structure:

```
{
    "base_url": "https://osdu-ship.msft-osdu-test.org",
    "data_partition_id": "opendes",
    "legal":
    {
        "legaltags": ["opendes-public-usa-dataset-7643990"],
        "otherRelevantDataCountries": ["US"],
        "status": "compliant"
    },
    "data": {
        "default": {
            "viewers": [ "data.default.viewers@opendes.contoso.com" ],
            "owners": [ "data.default.owners@opendes.contoso.com" ]
        }
    },
    "wellbore_mapping": {},
    "welllog_mapping": {}
}
```

The `base_url` and `data_partition_id` must be correct for the OSDU instance that you want to connect to.
`wellbore_mapping` and `welllog_mapping` are optional features described in the [custom mappings](#custom-mappings) section.

## Command reference

The CLI has the general syntax:

```
wbdutil <group> <command> [options]
```

Every command accepts the common options `-c` / `--config_path` (path to the
[config file](#config-file)) and, for commands that talk to OSDU, `-t` / `--token` (bearer
token). Both fall back to the `CONFIGPATH` / `OSDUTOKEN` environment variables. Add `-h` to any
group or command for full help.

### `parse` — offline inspection (no OSDU connection)

| Command         | Description                                                                 | Key options                                        |
| --------------- | --------------------------------------------------------------------------- | -------------------------------------------------- |
| `parse print`   | Print the header of a LAS file, or of every LAS file in a folder.           | `-p`/`--path`                                      |
| `parse convert` | Convert a LAS file/folder to wellbore + well log JSON files (dry-run ingest).| `-p`/`--path`, `--wellbore_id`, `-c`/`--config_path` |

```bash
wbdutil parse print --path ./well.las
wbdutil parse convert --path ./las_folder --wellbore_id "opendes:master-data--Wellbore:abc123" -c ./config.json
```

### `ingest` — upload to OSDU

| Command           | Description                                                              | Key options                                              |
| ----------------- | ------------------------------------------------------------------------ | -------------------------------------------------------- |
| `ingest wellbore` | Ingest a LAS file or a folder of LAS files as wellbore + well log records.| `-p`/`--path`, `--norecognize`, `-t`/`--token`           |
| `ingest data`     | Write bulk curve data from a LAS file to an existing well log.            | `--welllog_id`, `-p`/`--path`, `-t`/`--token`            |

`--norecognize` skips automatic curve-family recognition.

```bash
wbdutil ingest wellbore --path ./well.las
wbdutil ingest data --welllog_id "opendes:work-product-component--WellLog:xyz789" --path ./well.las
```

### `list` — read records from OSDU

| Command        | Description                                        | Key options                                   |
| -------------- | -------------------------------------------------- | --------------------------------------------- |
| `list wellbore`| Retrieve and print a wellbore record.              | `--wellbore_id`, `-t`/`--token`               |
| `list welllog` | Retrieve and print a well log record.              | `--welllog_id`, `--curveids`, `-t`/`--token`  |
| `list curves`  | Retrieve and print a well log's bulk curve data.   | `--welllog_id`, `--curves`, `-t`/`--token`    |

`--curveids` prints only the curve ids of a well log. `--curves` takes a space-separated list of
curve names (all curves if omitted).

```bash
wbdutil list wellbore --wellbore_id "opendes:master-data--Wellbore:abc123"
wbdutil list welllog --welllog_id "opendes:work-product-component--WellLog:xyz789" --curveids
wbdutil list curves --welllog_id "opendes:work-product-component--WellLog:xyz789" --curves GR RHOB
```

### `search` — find wellbores by name

| Command          | Description                                       | Key options                    |
| ---------------- | ------------------------------------------------- | ------------------------------ |
| `search wellbore`| Print the ids of wellbores matching a facility name.| `--name`, `-t`/`--token`     |

```bash
wbdutil search wellbore --name "MY-WELL-01"
```

### `update` — modify existing records

| Command         | Description                                      | Key options                                  |
| --------------- | ------------------------------------------------ | -------------------------------------------- |
| `update welllog`| Update an existing well log's curve families.    | `--welllog_id`, `--recognize`, `-t`/`--token`|

`--recognize` recognizes and updates the curve families of the well log.

```bash
wbdutil update welllog --welllog_id "opendes:work-product-component--WellLog:xyz789" --recognize
```

### `download` — export OSDU data to LAS

| Command           | Description                                                   | Key options                                             |
| ----------------- | ------------------------------------------------------------- | ------------------------------------------------------- |
| `download welllog`| Download a well log and its curve data to a LAS file.         | `--welllog_id`, `--out`, `--curves`, `-t`/`--token`     |

`--curves` limits the download to a space-separated list of curves (all curves if omitted).

```bash
wbdutil download welllog --welllog_id "opendes:work-product-component--WellLog:xyz789" --out ./out.las
wbdutil download welllog --welllog_id "opendes:work-product-component--WellLog:xyz789" --out ./out.las --curves GR RHOB
```

## Custom mappings

**Custom mappings are an advanced feature of `wbdutil` that require knowledge of both `lasio.LASFile` and OSDU data object schemas and should be used with care.**
The configuration file can be used to define optional custom mappings between `lasio.LASFile` data objects
and OSDU wellbore and welllog objects of a specified kind.
It is recommended that a new mapping is thoroughly tested using the `parse` command group, before upload to OSDU.

There are 3 mapping definitions `wellbore_mapping`, `welllog_mapping` and `las_file_mapping`.
The first 2 (`wellbore_mapping` and `welllog_mapping`) define mappings from LAS format data to OSDU wellbore and welllog objects.
`las_file_mapping` defines the mapping from OSDU well log, well bore and curve data to LAS format data (a `lasio.LASFile` object).

All the mapping definitions must contain a `mapping` attribute, in addition the LAS to OSDU mapping definitions (`wellbore_mapping` and `welllog_mapping`) must contain a `kind` attribute.
If `wellbore_mapping`, `welllog_mapping` and `las_file_mapping` are not defined in the configuration file `wbdutil` will use the default mappings.
The `mapping` attribute describes how data in the incoming object should be transformed into the outgoing data type.
The `kind` attribute defines the target OSDU data type (kind), for example `osdu:wks:work-product-component--WellLog:1.1.0`.
Here is an example mapping for a welllog that could be added to a configuration file.

```
{
    "welllog_mapping": {
        "kind": "osdu:wks:work-product-component--WellLog:1.1.0",
        "mapping":
        {
            "acl.viewers": "CONFIGURATION.data.default.viewers",
            "acl.owners": "CONFIGURATION.data.default.owners",
            "legal.legaltags": "CONFIGURATION.legal.legaltags",
            "legal.otherRelevantDataCountries": "CONFIGURATION.legal.otherRelevantDataCountries",
            "legal.status": "CONFIGURATION.legal.status",
            "data.ReferenceCurveID": "curves[0].mnemonic",
            "data.WellboreID": {
                "type": "function",
                "function": "get_wellbore_id",
                "args": []
            },
            "data.Curves": {
                "type": "array",
                "source": "curves",
                "mapping": {
                    "CurveID": "mnemonic",
                    "Mnemonic": "mnemonic",
                    "CurveUnit": {
                        "type": "function",
                        "function": "las2osdu_curve_uom_converter",
                        "args": [
                            "unit",
                            "CONFIGURATION.data_partition_id"
                        ]
                    }
                }
            }
        }
    }
}
```

The simple data mappings take the form of a key and string value pair.
The key (string to the left of the semi-colon) is the target field within the OSDU data kind and
the value string defines the source of the data. For example:
`"data.ReferenceCurveID": "curves[0].mnemonic"` will set the `data.ReferenceCurveID` field of the output OSDU object
to the value of the `mnemonic` field of the first element in the `curves` array of the input `lasio.LASFile` object.
The `CONFIGURATION` keyword indicates that data should be taken from the configuration file, for example:
`"acl.viewers": "CONFIGURATION.data.default.viewers"` will set the `acl.viewers` field of the output OSDU object
to the value of the `data.default.viewers` field of the configuration. The simple mapping form supports the direct copying of
all objects including arrays from the incoming LAS data to the output OSDU data kind.

There are often more complex transformations that need to be performed on the incoming data,
`wbdutil` supports two types of complex mapping `array` and `function`.
The `function` complex mapping type makes a call to a hard coded function to perform a transformation on the incoming data.
For example:

```
"CurveUnit": {
    "type": "function",
    "function": "las2osdu_curve_uom_converter",
    "args": [
        "unit",
        "CONFIGURATION.data_partition_id"
    ]
}
```

This will set the value of the `CurveUnit` field to the output of the function `las2osdu_curve_uom_converter` using the input arguments in the args array. The `args` section defines the argument for the function, each `arg` is not the direct input argument to the function. An `arg` is a reference to piece of data in the incoming data or configuration file.
In this case `unit` references data in the input data and `data_partition_id` data in the configuration file.

The second complex mapping is `array` this should be used if the elements of an incoming array need to be changed in some way.
This could be a field name change, a change in the object structure or to call a function on specific data within each element.
Here is an example:

```
{
    "data.Curves": {
        "type": "array",
        "source": "curves",
        "mapping": {
            "CurveID": "mnemonic",
            "Mnemonic": "mnemonic",
            "CurveUnit": {
                "type": "function",
                "function": "las2osdu_curve_uom_converter",
                "args": [
                    "unit",
                    "CONFIGURATION.data_partition_id"
                ]
            }
        }
    }
}
```

This mapping will iterate over the `curves` array of the input `lasio.LASFile` object and apply an
inner mapping to each element in the array.
In this case the inner mapping is defined so that the `mnemonic` field of the `curve` element is
mapped to both the `CurveID` and `Mnemonic` output fields, and the `CurveUnit` output field is set
to the return value of the function `las2osdu_curve_uom_converter`
that takes the `unit` field of array element and the `data_partition_id` (from configuration) as arguments.
The resulting output array is mapped to the `data.Curves` field of the output OSDU kind.

Here is an example `las_file_mapping` section:

```
"las_file_mapping": {
    "mapping": {
        "Well.WELL": "WELLBORE.data.FacilityName",
        "Well.UWI": {
            "type": "function",
            "function": "extract_uwi_from_name_aliases",
            "args": ["WELLBORE.data.NameAliases"]
        },
        "Curves": {
            "type": "function",
            "function": "build_curves_section",
            "args": ["WELLLOG.data.Curves", "CURVES"]
        }
    }
}
```

With OSDU to LAS mappings, data is drawn from 3 types of OSDU data object: wellbore, welllog and curves.
Each of these incoming OSDU data can be referenced by the keywords `WELLBORE`, `WELLLOG` and `CURVES`.
Where `WELLBORE` and `WELLLOG` are wellbore and welllog OSDU kinds and `CURVES` is a Pandas DataFrame that
contains the incoming curves data from OSDU.

Example configuration files that are set up for the preship OSDU instance are given in the
`src/` directory (`example_opendes_configuration_AWS.json`, `example_opendes_configuration_Azure.json`
and `example_opendes_configuration_GC.json`),
it also contains example custom mappings for the `osdu:wks:master-data--Wellbore:1.0.0` wellbore kind and the `osdu:wks:work-product-component--WellLog:1.1.0` welllog kind.

This table summarises the available keywords.

| Keyword         | Valid Mapping type | Incomming data source     |
| --------------- | ------------------ | ------------------------- |
| `CONFIGURATION` | All                | The configuration file    |
| `WELLBORE`      | `las_file_mapping` | The OSDU wellbore object  |
| `WELLLOG`       | `las_file_mapping` | The OSDU welllog object   |
| `CURVES`        | `las_file_mapping` | The OSDU Curves DataFrame |

There are a limited number of mapping functions available these are listed below:

| Function name                                              | Mapping type       | Purpose                                                                                                  |
| ---------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------- |
| `build_wellbore_name_aliases(uwi, data_partition_id)`      | `wellbore_mapping` | Constructs a name alias object from the LAS UWI and the data partition id.                               |
| `get_wellbore_id()`                                        | `welllog_mapping`  | Returns the wellbore id from the wellbore that corresponds to the welllog                                |
| `las2osdu_curve_uom_converter(unit, data_partition_id)`    | `welllog_mapping`  | This function converts a LAS format unit of measure to an OSDU format UoM.                               |
| `extract_uwi_from_name_aliases(NameAliases: list)`         | `las_file_mapping` | Return the first name alias or None if none exist                                                        |
| `build_curves_section(wl_curves: list, curves: DataFrame)` | `las_file_mapping` | Iterates over curves, converting units of measure from OSDU to LAS form. Returns the updated curve data. |

These are hard coded functions, so a change request will need to be raised if additional functions are required. We have avoided user defined functions, because such functions represent a small security risk.

## Development
Refer to [System Maintenance Guide](doc/SMG.md) for instructions on setting up a development environment.
