Metadata-Version: 2.4
Name: idsdata
Version: 0.1.0
Summary: Versioned, checksummed loaders and a linter for intrusion-detection datasets
Keywords: intrusion detection,datasets,CIC-IDS2017,reproducibility
Author: Samuel Justus Schwertfeger
Author-email: Samuel Justus Schwertfeger <113480134+SamuelSchwertfeger@users.noreply.github.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Requires-Dist: pandas>=2.3.3
Requires-Dist: pyarrow>=25.0.1
Requires-Python: >=3.10
Project-URL: Repository, https://github.com/SamuelSchwertfeger/idsdata
Project-URL: Issues, https://github.com/SamuelSchwertfeger/idsdata/issues
Description-Content-Type: text/markdown

# idsdata

Load intrusion-detection datasets by name and version, check them against recorded SHA-256 hashes, and get the right citation. `ids-lint` flags known quality problems in a dataset CSV file.

Papers that say "we used CIC-IDS2017" can mean several different files. `idsdata` makes the version part of the name, and refuses to load a file whose hash does not match.

## Install

```console
pip install idsdata
```

It needs Python 3.10 or newer and installs pandas and pyarrow.

## Quickstart

```console
idsdata list
idsdata info cic-ids2017 engelen2021
idsdata download cic-ids2017 engelen2021
idsdata verify cic-ids2017 engelen2021
```

```python
import idsdata

flows = idsdata.load('cic-ids2017', 'engelen2021')
print(flows.shape)  # (2100814, 88)
print(flows['label'].value_counts())
print(idsdata.cite('cic-ids2017', 'engelen2021'))
```

`load` keeps every flow and every value as it is in the source files. It gives the columns clean names (`Flow Bytes/s` becomes `flow_bytes_s`), keeps the source label as `label_raw`, and adds four columns:

| column | meaning |
|---|---|
| `label` | the label in lower case with underscores |
| `is_attack` | `True` for attack flows |
| `is_attempted` | `True` for flows the corrected release marks as "Attempted": part of an attack class, but with no malicious payload. These have `is_attack == False`. |
| `source_file` | the file the row came from |

Keeping "Attempted" flows out of `is_attack` is a choice this package makes, following the corrected release. It means `is_attack` and the class name can disagree: a `dos_hulk_attempted` row has `is_attack == False`. Decide what you want before training a binary classifier:

```python
flows = flows[~flows['is_attempted']]  # drop them
attack = flows['is_attack'] | flows['is_attempted']  # or count them as attacks
```

The first load reads the CSV files and writes a parquet cache next to them. Later loads read the cache.

## Train/test splits

```python
train, test = idsdata.split('cic-ids2017', 'engelen2021', 'time')
```

A split has a name, and the name always means the same rows. Both splits put 20% of the flows in the test set and keep every class on both sides.

| split | how the test set is chosen | use it for |
|---|---|---|
| `time` (default) | For each class in each source file, the latest 20% of the flows by timestamp | New results. A model is tested on flows that come after the ones it was trained on. |
| `random` | 20% of each class, drawn with a fixed seed | Comparison with earlier work that shuffled the rows. Near-duplicate flows land on both sides, so scores are optimistic. |

Things to know:

- The cut is made per class, not once per day. Each attack runs in a short window, so a single cut-off time per day would leave most attack classes out of the test set.
- Timestamps have one-second resolution. Flows that share the second at the cut keep the order they have in the source file.
- A class with two or more flows always gets at least one test flow; a class with a single flow stays in train. Some classes are very small, so their test sets are tiny.
- A split is of the whole release. Filter afterwards, for example `train[~train['is_attempted']]`.

## Datasets

| name | version | what it is | how you get it |
|---|---|---|---|
| `cic-ids2017` | `engelen2021` | Corrected flows and labels by Engelen, Rimmer and Joosen | `idsdata download`, or by hand from the authors' page |
| `cic-ids2017` | `original-flows` | The original release from the Canadian Institute for Cybersecurity, with flow ids, addresses and timestamps (`GeneratedLabelledFlows.zip`) | By hand, through the request form on the publisher's page |
| `cic-ids2017` | `original-ml` | The same flows without ids, addresses and timestamps (`MachineLearningCSV.zip`) | By hand, through the same form |

For the two original versions, fill in the form, take the zip from the `CSVs` folder and put it in the directory `idsdata where cic-ids2017 original-flows` (or `original-ml`) prints. `verify` checks the zip and `load` unpacks it.

Both original versions hold the same 2,830,743 flows in 15 classes. They are kept as published, and four things about them are worth knowing:

- The column ` Fwd Header Length` appears twice in every file. Both are kept; the second is named `fwd_header_length_1`. In these files the two hold the same values.
- One file of `original-flows` holds 288,602 lines without any value. `load` leaves them out. This is the only case where lines of a source file are not returned, and the number is recorded and checked.
- The dash in the three web attack labels is not plain text. `original-flows` is read as Windows-1252, which makes it an en dash; in `original-ml` it is the Unicode replacement character. `label_raw` shows what the file holds, and `label` is the same in both (`web_attack_xss`).
- The timestamps of `original-flows` come in more than one format and without AM or PM, so only the `random` split is offered for the original versions.

## Data policy

- The package contains no dataset files and this repository will never contain any.
- Nothing is mirrored. Files come from the official host, or you download them yourself.
- `idsdata download` works only where the host offers a direct link. It prints the terms and asks before it fetches anything (`--yes` skips the question).
- A release behind a request form is never fetched by the package. You fill in the form and place the files; the package checks them.
- Every recorded hash was computed from a real download, and the registry records the date of that download. If the host replaces a file, verification fails and says so. It also fails when a table is missing and the archive it comes from is not there.

Files are kept in `~/.cache/idsdata/<name>/<version>/`. Set `IDSDATA_DIR` or pass `--data-dir` to use another place. `idsdata where NAME VERSION` prints the directory.

## How to cite

To cite `idsdata` itself, use `CITATION.cff` (the "Cite this repository" button on GitHub).

Cite the papers behind the data you use. This prints the BibTeX entries:

```console
idsdata cite cic-ids2017 engelen2021
```

For `engelen2021` that is the paper that corrected the dataset and the paper that introduced it:

- G. Engelen, V. Rimmer and W. Joosen, "Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study", 2021 IEEE Security and Privacy Workshops (SPW), 2021. https://doi.org/10.1109/SPW53761.2021.00009
- I. Sharafaldin, A. Habibi Lashkari and A. Ghorbani, "Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization", ICISSP 2018. https://doi.org/10.5220/0006639801080116

## ids-lint

```console
ids-lint Friday-WorkingHours.csv
ids-lint --release cic-ids2017 engelen2021 *.csv
ids-lint --rules
```

The exit status is 1 when a rule of severity `error` fires. With `--strict`, warnings count as well. `--release` also checks that every label string in the file belongs to that release, and reads the file in the text encoding recorded for it.

<!-- rules:start -->
| id | rule | severity | what it flags | source |
|---|---|---|---|---|
| IDS001 | duplicated-column-names | error | Two columns share a name, so readers rename or overwrite one of them without saying so. | [Engelen et al., documentation](https://intrusion-detection.distrinet-research.be/WTMC2021/extended_doc.html) |
| IDS002 | non-finite-values | error | Missing, NaN or infinite feature values stop most learners or get dropped along the way. | [Rosay et al. 2022](https://doi.org/10.5220/0010774000003120) |
| IDS003 | label-strings | error | Missing or unknown label strings lose rows or split one class into several. | [Engelen et al. 2021](https://doi.org/10.1109/SPW53761.2021.00009), [Flood et al. 2024](https://doi.org/10.1109/EuroSP60621.2024.00042) |
| IDS004 | duplicate-rows | warn | Rows that repeat exactly can end up on both sides of a train/test split. | [Flood et al. 2024](https://doi.org/10.1109/EuroSP60621.2024.00042) |
| IDS005 | identifier-columns | warn | Flow ids, addresses, ports and timestamps describe the capture, and a model can learn them as a shortcut. | [Engelen et al. 2021](https://doi.org/10.1109/SPW53761.2021.00009), [D'hooge et al. 2022](https://doi.org/10.1007/978-3-031-09484-2_2) |
| IDS006 | constant-columns | warn | A column that holds a single value carries no information. | [Rosay et al. 2022](https://doi.org/10.5220/0010774000003120) |
<!-- rules:end -->

`ids-lint --rules` prints the full reference for each source. The source of IDS001 is the authors' documentation page, which is not peer reviewed. Flood et al. describe near-duplicate flows; IDS004 reports only exact repeats. The sources of IDS003 describe flows that carry the wrong label; IDS003 can only see label strings that are missing or that a release does not define.

The linter reports and changes nothing. A warning is a prompt to decide, and what to do with the flagged rows or columns is up to you.

## How it was tested

- **Unit tests.** The test suite runs on every pull request, on Python 3.10 and 3.12 (GitHub Actions, Linux), together with `ruff` and `basedpyright`. They use small made-up files only, never real data: checksums, the download rules, loading, the cache, label mapping, both splits and every lint rule. The `random` split is pinned to exact rows, so a change in pandas that moves rows between train and test is noticed. One test checks that the rule table above matches the code.
- **Real data, by hand.** The numbers in the registry (hashes, sizes, row counts, column names, label strings) were read from real downloads made on 2026-10-09, not typed in. The MD5 files the publisher ships with the original release match the two zips. These checks were then run on Windows 11 with Python 3.12, and are not part of CI because the data cannot be redistributed:

| check | `engelen2021` | `original-flows` | `original-ml` |
|---|---|---|---|
| `verify` passes on the downloaded archive | yes | yes | yes |
| `load` straight from the archive | 2,100,814 rows, 88 columns, 25 classes | 2,830,743 rows, 89 columns, 15 classes | 2,830,743 rows, 83 columns, 15 classes |
| rows per file equal the recorded counts | yes | yes | yes |
| `split(..., 'time')` train / test | 1,680,663 / 420,151 | not offered | not offered |
| `split(..., 'random')` train / test | 1,680,661 / 420,153 | 2,264,600 / 566,143 | 2,264,600 / 566,143 |
| every class on both sides of each split | yes | yes | yes |
| `ids-lint --release` on every file: unknown label strings | none | none | none |
| `ids-lint` rules that fire | IDS002, IDS004, IDS005, IDS006 | all six | all but IDS003 |

The class counts of the two original versions are equal label by label.

What was not tested: other operating systems and Python versions with real data, and `idsdata download` beyond the one release that has a direct link.

## Not in this version

- A time split for the original versions.
- Other datasets.

## Acknowledgements

The CI setup is adapted from [DFAIR-LAB-Augusta/XSecIoT](https://github.com/DFAIR-LAB-Augusta/XSecIoT) (Seth Barrett).
