Metadata-Version: 2.4
Name: mdfreader
Version: 4.3
Summary: A Measured Data Format file parser
Home-page: https://github.com/ratal/mdfreader
Author: Aymeric Rateau
Author-email: aymeric.rateau@gmail.com
License: GPL3
Keywords: Parser MDF file
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.14
Requires-Dist: sympy
Requires-Dist: lxml
Provides-Extra: export
Requires-Dist: hdf5storage; extra == "export"
Requires-Dist: h5py; extra == "export"
Requires-Dist: scipy; extra == "export"
Requires-Dist: openpyxl>2.0; extra == "export"
Requires-Dist: pandas; extra == "export"
Requires-Dist: fastparquet; extra == "export"
Provides-Extra: plot
Requires-Dist: matplotlib; extra == "plot"
Requires-Dist: mpldatacursor; extra == "plot"
Provides-Extra: converter
Requires-Dist: PyQt5; extra == "converter"
Provides-Extra: experimental
Requires-Dist: bitarray; extra == "experimental"
Provides-Extra: compression
Requires-Dist: blosc; extra == "compression"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

**MDFREADER**

**************



Abstract:

=========

This module imports MDF files (Measured Data Format V3.x and V4.x), typically

from INCA (ETAS), CANape or CANoe. It is widely used in the automotive industry

to record data from ECUs. The main module `mdfreader.py` inherits from two

module pairs (one per MDF version): the first reads the file's block structure

(`mdfinfoX`), and the second reads the raw data (`mdfXreader`). It can

optionally run multithreaded and was designed for efficient batch processing of

large endurance-evaluation files for data mining.



Performance:

============

When Cython is available (strongly recommended), mdfreader uses several

low-level optimisations:



* **Fast CN/CC/SI/TX metadata reader** (`read_cn_chain_fast` in `dataRead.pyx`):

  walks the entire MDF4 channel linked list in a single Cython function using

  POSIX `pread()` (no Python file-object dispatch, no GIL during I/O) and

  C packed-struct `memcpy` parsing. A fast `<TX>…</TX>` bytes scan replaces

  `lxml.objectify` for the common MD-block pattern (~95% of files). Result:

  **3–4× speedup** on large files compared to the pure-Python path.



* **SymBufReader**: a Cython bidirectional-buffered wrapper around the raw file

  object. MDF4 metadata blocks are linked by backward-pointing pointers;

  `SymBufReader` keeps a 64 KB buffer centred on the current position so that

  most seeks are served from cache without a kernel `read()`.



* **Vectorised data reading**: sorted channel groups are read in a single

  `readinto()` call into a flat `uint8` buffer that is then reinterpreted as a

  structured record array — zero copies, no per-chunk Python loop.



Typical timings on a 184 MB / 36 000-channel MDF4 file:



| Scenario          | Time   |

|-------------------|--------|

| Pure Python path  | ~1.9 s |

| v4.2 with Cython  | ~1.9 s |

| v4.3 (this version) | **~0.6 s** |



The structure of the mdf object inheriting from python dict

===========================================================

For each channel `mdf[channelName]` the following keys exist:



| Key | Description |

|-----|-------------|

| `data` | numpy array of channel values |

| `unit` | unit string |

| `master` | name of the master (time/angle/…) channel |

| `masterType` | master channel type: 0=None, 1=Time, 2=Angle, 3=Distance, 4=Index |

| `description` | channel description string |

| `conversion` | present when `convert_after_read=False`; dict describing raw→physical mapping |



`mdf.masterChannelList` is a dict mapping each master channel name to the list

of channels sampled at the same raster.



Mdfreader module methods:

=========================

* resample channels to one sampling frequency

* merge files

* plot one channel, several channels on one graph (list) or several channels on subplots (list of lists)



It is also possible to export mdf data into:

* CSV file (Excel dialect by default)

* NetCDF file for compatibility with Uniplot (needs `netcdf4`, `Scientific.IO`)

* HDF5 (needs `h5py`)

* Excel 95–2003 (needs `xlwt` — very slow for large files)

* Excel 2007/2010 (needs `openpyxl` — can also be slow with large files)

* Matlab `.mat` (needs `hdf5storage`)

* MDF file — allows creating, converting or modifying data, units and descriptions

* Pandas DataFrame(s) (command line only, not in mdfconverter) — one DataFrame per raster



Compatibility:

==============

Python 3.9+ — tested on Linux and Windows (x86-64)



Requirements:

=============

Core: `numpy`, `lxml`, `sympy`



`lxml` is used for MDF4 metadata XML blocks. When Cython is compiled, the fast

path handles the common `<TX>…</TX>` pattern directly from bytes and only falls

back to `lxml` for complex XML (CDATA, namespaces).



Reading channels defined by a formula requires `sympy`.



Cython is strongly advised. It compiles `dataRead.pyx`, which provides:

* fast metadata parsing via `pread()` + C packed structs

* the `SymBufReader` bidirectional file buffer

* bit-exact reading for non-byte-aligned or record-padded channels

* VLSD/VLSC string data reading helpers



If Cython compilation fails, `bitarray` is used as a fallback (slower, pure Python).



Export requirements (optional): `scipy`, `h5py`, `hdf5storage`, `openpyxl`, `pandas`, `fastparquet`



Data compression in memory (optional): `blosc`



Graphical converter: `PyQt5`



Installation:

=============

From PyPI:

```shell

pip install mdfreader

```

From source:

```shell

pip install cython numpy        # build prerequisites

python setup.py build_ext --inplace

python setup.py develop

```



Graphical interface: mdfconverter

==================================

A PyQt5 GUI to convert batches of files. Launch with:

```shell

mdfconverter

```

Right-click a channel in the list to plot it. Channels can be dragged between

columns. A `.lab` channel-list file can be imported. Multiple files can be

merged into one and resampled.



Memory-saving options:

======================

For large files or limited memory:



* **Channel list only** — pass `channel_list=['ch1', 'ch2']`; call

  `mdfreader.MdfInfo(file)` to get the full channel list without loading data.

* **Raw data mode** — pass `convert_after_read=False`; data stays as stored in

  the MDF file and is converted on-the-fly by `get_channel_data`, `plot`,

  `export_to_*`, etc.

* **Blosc compression** — pass `compression=True` (default level 9) to compress

  data in memory after reading.

* **No-data skeleton** — pass `no_data_loading=True` to build the channel

  metadata dict without reading any samples; data is fetched on demand via

  `get_channel_data`.



For data visualisation, a dataPlugin for Veusz (≥ 1.16) is also available;

follow the instructions in Veusz's documentation and the plugin file's header.



Command example in ipython:

===========================

```python

    import mdfreader

    # loads whole mdf file content in yop mdf object.

    yop=mdfreader.Mdf('NameOfFile')

    # you can print file content in ipython with a simple:

    yop

    # alternatively, for max speed and smaller memory footprint, read only few channels

    yop=mdfreader.Mdf('NameOfFile', channel_list=['channel1', 'channel2'], convert_after_read=False)

    # also possible to keep data compressed for small memory footprint, using Blosc module

    yop=mdfreader.Mdf('NameOfFile', compression=True)

    # for interactive file exploration, possible to read the file but not its data to save memory

    yop=mdfreader.Mdf('NameOfFile', no_data_loading=True) # channel data will be loaded from file if needed

    # parsing xml metadata from mdf4.x for many channels can take more than just reading data.

    # You can reduce to minimum metadata reading with below argument (no source information, attachment, etc.) 

    yop=mdfreader.Mdf('NameOfFile', metadata=0)  # 0: full, 2: minimal

    # only for mdf4.x, you can search for the mdf key of a channel name that can have been recorded by different sources

    yop.get_channel_name4('channelName', 'source path or name')  # returns list of mdf keys

    # to yield one channel and keep its content in mdf object

    yop.get_channel('channelName')

    # to yield one channel numpy array

    yop.get_channel_data('channelName')

    # to get file mdf version

    yop.MDFVersionNumber

    # to get file structure or attachments, you can create a mdfinfo instance

    info=mdfreader.MdfInfo()

    info.list_channels('NameOfFile') # returns only the list of channels

    info.read_info('NameOfFile') # complete file structure object

    yop.info # same class is stored in mdfreader class

    # to list channels names after reading

    yop.keys()

    # to list channels names grouped by raster, below dict mdf attribute contains

    # pairs (key=masterChannelName : value=listOfChannelNamesForThisMaster)

    yop.masterChannelList

    # quick plot or subplot (with lists) of channel(s)

    yop.plot(['channel1',['channel2','channel3']])

    # file manipulations

    yop.resample(0.1)

    # or

    yop.resample(master_channel='master3')

    # keep only data between begin and end

    yop.cut(begin=10, end=15)

    # export to other file formats :

    yop.export_to_csv(sampling=0.01)

    yop.export_to_NetCDF()

    yop.export_to_hdf5()

    yop.export_to_matlab()

    yop.export_to_xlsx()

    yop.export_to_parquet()

    # return pandas dataframe from master channel name

    yop.return_pandas_dataframe('master_channel_name')

    # converts data groups into pandas dataframes and keeps it in mdf object

    yop.convert_to_pandas()

    # drops all the channels except the one in argument

    yop.keep_channels({'channel1','channel2','channel3'})

    # merge 2 files

    yop2=mdfreader.Mdf('NameOfFile_2')

    yop.merge_mdf(yop2)

    # can write mdf file after modifications or creation from scratch

    # write4 and write3 also allow to convert file versions

    yop.write('NewNameOfFile')  # write in same version as original file after modifications

    yop.write4('NameOfFile', compression=True)  # write mdf version 4.1 file, data compressed

    yop.write3()  # write mdf version 3 file

    yop.attachments  # to get attachments, embedded or paths to files 

```

<a href="https://scan.coverity.com/projects/ratal-mdfreader">

  <img alt="Coverity Scan Build Status"

       src="https://scan.coverity.com/projects/21368/badge.svg"/>

</a>
