Metadata-Version: 2.4
Name: cccd
Version: 1.0.0
Summary: Create self documenting tabular data with toml-like metadata header
Author: Wojciech Sadowski
Project-URL: Homepage, https://gitlab.ruhr-uni-bochum.de/sadowwgh/c3d
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=3.0.5
Dynamic: license-file

# c3d (cccd) - Convert CSV files to Commented Data

Add human-readable metadata while saving your tabular data. Metadata
is stored in a _toml-like_ format, which allows for easy parsing both by
humans and programs. A parser for metadata is provided.

<!--toc:start-->
- [c3d (cccd) - Convert CSV files to Commented Data](#c3d-cccd-convert-csv-files-to-commented-data)
  - [Use case example](#use-case-example)
    - [Create a file](#create-a-file)
    - [Reading the metadata from a file](#reading-the-metadata-from-a-file)
    - [Reading only the data](#reading-only-the-data)
  - [c3d .dat File Format Specification](#c3d-dat-file-format-specification)
    - [Title-Level Fields](#title-level-fields)
    - [Data Description Section](#data-description-section)
    - [Column Sub-Sections](#column-sub-sections)
      - [Column Naming Rules](#column-naming-rules)
      - [Per-Column Fields](#per-column-fields)
    - [Data Section](#data-section)
    - [Complete Example](#complete-example)
    - [Supported Custom Field Types](#supported-custom-field-types)
<!--toc:end-->

## Use case example

### Create a file

```python
from cccd import models as m
from cccd import writer as w
import pandas as pd

# We will process a folowing table:
#
# names of students | happines of phds
# -------------------------------------
# Stefan            | 1
# Alexandra         | 2
# Happy Guy         | 10

data = pd.DataFrame(
    {
        "names of students": ["Stefan", "Alexandra", "Happy guy"],
        "happines of phds": [1, 2, 10],
    }
)

# We want to document what is happines, link to the original stady and indicate
# other important informationtion about the metadata.

# We start by renaming and describing the columns
data = data.rename(
    columns={"names of students": "names", "happines of phds": "happines"}
)

description_names = m.ColumnMetadata(
    name="names",
    unit=m.NO_UNIT,  # this is a string (NO_UNIT is equivalent of writing "-" )
    description="the names of the students",
    custom={
        "anonymisation": "all names has been changed"
    },  # we can add any other custom information in the form of a dictionary
)

description_happines = m.ColumnMetadata("happines", m.NO_UNIT, "a number out of 10")

# We have to provide the description of the dataset as a whole, by combining
# the descriptions of individual columns into a list and specifying which
# delimiter we would like to use to separate the columns.
data_description = m.DataDescription(
    columns=[description_happines, description_names],
    delimiter=",",  # defaults to "\t" (that is, tab separated values tsv)
    # n_columns=2,
    # n_rows=3,
    custom={"year": 2026, "university": "RUB", "data_collection": "campus survey"},
)

# Finally we describe the whole table by providing the name of this table, the
# permalink (DOI) the original study and the link to the related dataset
# (hopefully hosted in a university database).
m = m.Metadata(
    title="Data on happines",
    publication_doi="doi1",
    dataset_doi="doi2",
    figure="4c",  # if the data is plotted in a figure in the related publication, 
                  # we specify it here
    data_description=data_description,
)

# Finally we write the file
file_name = "test.dat"
w.write_dat_file(file_name, m, data)
```

The above script will save this simple dataframe as

```
# title = "Data on happines"
# publication_doi = "doi1"
# dataset_doi = "doi2"
# figure = "4c"
#
# header_size = 24
#
# [data_description]
# delimiter = ","
# year = 2026
# university = "RUB"
# data_collection = "campus survey"
#
# [data_description.happines]
# unit = "-"
# description = "a number out of 10"
#
# [data_description.names]
# unit = "-"
# description = "the names of the students"
# anonymisation = "all names has been changed"
#
#
# [data]
names,happines
Stefan,1
Alexandra,2
Happy guy,10
```

### Reading the metadata from a file

You can extract a `Metadata` object from a file with

```python
from cccd.parser import extract_metadata

# Get metadata for a file (information in the header)
meta= extract_metadata("test.dat")
print(meta)
# prints:
#
# Title: ........... Data on happines
# Publication DOI: . doi1
# Dataset DOI: ..... doi2
# Figure: .......... 4c
# 
# Data Description:
#   Delimiter: .... ','
# 
# Columns:
#   Column          | Unit            | Description
#   happines        | -               | a number out of 10
#   names           | -               | the names of the students
```

All of the above information are accessible as fields of the `Metadata` object.

### Reading only the data

Simply:

```python
from cccd.parser import extract_dataframe
dataframe, meta = extract_dataframe("test.dat")
```

or, alternatively,

```python
import pandas as pd
df = pd.read_csv("test.dat", comment="#", delimiter=",")
```

## c3d .dat File Format Specification

The `.dat` format is a self-describing tabular data file consisting of two parts:

1. **Header** — a TOML-encoded metadata section describing the dataset and its columns
2. **Data** — delimiter-separated tabular values

The metadata fields are commented by `#` for easy reading by standard software.
In case when the amount of lines to skip has to be provided manually, the
number of lines in the header is stored in the `header_size` field.

```
# <title-level fields>
# 
# header_size = <int>
# 
# [data_description]
# <global data fields>
# 
# [data_description.<column_name_1>]
# unit = "<string>"
# description = "<string>"   (optional)
# 
# [data_description.<column_name_2>]
# unit = "<string>"
# description = "<string>"   (optional)
# ...
# 
# [data]
column_name_1   column_name_2   ...
row1_col1       row1_col2       ...
row2_col1       row2_col2       ...
```

### Title-Level Fields

| Field              | Type     | Required | Description                                      |
|--------------------|----------|----------|--------------------------------------------------|
| `title`            | string   | **Yes**  | Human-readable title of the dataset              |
| `publication_doi`  | string   | **Yes**  | DOI of the publication the data belongs to       |
| `dataset_doi`      | string   | **Yes**  | DOI of the dataset itself                        |
| `figure`           | string   | No       | Optional reference to a figure in the paper      |
| `header_size`      | integer  | No       | Number of header lines before `[data]`            |

- `title`, `publication_doi`, and `dataset_doi` are mandatory. If any is
  missing, parsing will fail.
- `figure` is optional and typically contains a figure label (e.g., `"3c"`).
- `header_size` is auto-generated by the writer and tells readers exactly how
  many lines to skip to reach the data section.

### Data Description Section

The `[data_description]` section contains global metadata about the tabular data:

| Field         | Type     | Required | Description                                |
|---------------|----------|----------|--------------------------------------------|
| `delimiter`   | string   | Yes       | Character separating columns in data rows  |
| `n_columns`   | integer  | No       | Total number of columns in the data        |
| `n_rows`      | integer  | No       | Total number of data rows                  |

Custom fields may be added here. Any key-value pair that is not recognized as a
built-in field and does not reference a column subsection will be stored as
custom metadata.

### Column Sub-Sections

Each column is described in its own subsection named
`[data_description.<column_name>]`, where `<column_name>` matches the
identifier of the column.

#### Column Naming Rules

- Column names must be alphanumeric (`isalnum()`).
- If a name contains spaces or special characters, it must be quoted in TOML:
  `[data_description."my column"]`.

#### Per-Column Fields

| Field         | Type     | Required | Description                        |
|---------------|----------|----------|------------------------------------|
| `unit`        | string   | **Yes**  | Measurement unit (e.g., `"mm"`, `"K"`, `"-"`) |
| `description` | string   | No       | Human-readable column description  |

- `unit` is mandatory. Parsing will raise a `ValueError` if a column is missing
  its unit.
- Use `"-"` (constant `NO_UNIT`) for unitless columns.
- Additional custom fields may be attached to individual columns and will be
  preserved during round-trip serialization.

### Data Section

The `[data]` section marker signals the start of tabular data. All lines after
this marker are parsed as delimiter-separated values using whichever delimiter
was specified in `data_description.delimiter` (default: tab).

- The data has no header row — column names come from the metadata
  subsections.
- Values containing the delimiter character should be quoted (standard CSV
  quoting rules apply).
- Whitespace after delimiters is skipped (`skipinitialspace=True`).

### Complete Example

```
# title = "Jet-to-surface distance measurements"
# publication_doi = "10.1234/abcd"
# dataset_doi = "10.5678/xyz"
# figure = "2a"
# 
# header_size = 18
# 
# [data_description]
# delimiter = "\t"
# n_columns = 3
# n_rows = 100
# 
# [data_description.distance]
# unit = "mm"
# description = "Distance from jet to liquid surface"
# 
# [data_description.concentration]
# unit = "%"
# description = "Mass concentration"
# 
# [data_description.temperature]
# unit = "K"
# description = "Ambient temperature"
# 
# [data]
distance  concentration temperature
1.0    50.2    293.15
2.0    48.7    293.40
...
```

### Supported Custom Field Types

Custom fields at any level support these TOML value types:

| Type     | Example        |
|----------|----------------|
| string   | `"hello"`      |
| integer  | `42`           |
| float    | `3.14`         |
| boolean  | `true` / `false` |
| array    | `[1, 2, 3]`    |

Nested tables (dicts) are **not** supported as custom field values.
