Metadata-Version: 2.1 Name: SeqTools Version: 1.3.0 Summary: A library for transparent transformation of indexable containers (lists, etc.) Author-email: Nicolas Granger License: Mozilla Public License 2.0 (MPL 2.0) Project-URL: PyPi, https://pypi.org/project/SeqTools Project-URL: Documentation, http://seqtools-doc.readthedocs.io Project-URL: Repository, https://github.com/nlgranger/SeqTools Keywords: mapping,lazy,delayed,pipeline,processing Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Operating System :: OS Independent Classifier: License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0) Classifier: Development Status :: 3 - Alpha Classifier: Topic :: Software Development :: Libraries :: Python Modules Classifier: Intended Audience :: Developers Classifier: Intended Audience :: Science/Research Requires-Python: >=3.8 Description-Content-Type: text/x-rst License-File: LICENSE.txt Requires-Dist: tblib Provides-Extra: documentation Requires-Dist: sphinx ; extra == 'documentation' Requires-Dist: furo ; extra == 'documentation' Requires-Dist: ipython ; extra == 'documentation' Requires-Dist: ipykernel ; extra == 'documentation' Requires-Dist: nbconvert ; extra == 'documentation' Requires-Dist: nbsphinx ; extra == 'documentation' Requires-Dist: Pillow ; extra == 'documentation' Requires-Dist: numpy ; extra == 'documentation' Provides-Extra: tests Requires-Dist: pytest ; extra == 'tests' Requires-Dist: pytest-timeout ; extra == 'tests' Requires-Dist: numpy ; extra == 'tests' Requires-Dist: pytest-coverage ; extra == 'tests' .. image:: https://badge.fury.io/py/SeqTools.svg :target: https://pypi.org/project/SeqTools :alt: PyPi package .. image:: https://readthedocs.org/projects/seqtools-doc/badge :target: http://seqtools-doc.readthedocs.io :alt: Documentation SeqTools ======== SeqTools extends the functionalities of itertools to indexable (list-like) objects. Some of the provided functionalities include: element-wise function mapping, reordering, reindexing, concatenation, joining, slicing, minibatching, `etc `_. SeqTools functions implement **on-demand evaluation** under the hood: operations and transformations are only applied to individual items when they are actually accessed. A simple but powerful prefetch function is also provided to eagerly evaluate elements in background threads or processes. SeqTools originally targets data science, more precisely the data preprocessing stages. Being aware of the experimental nature of this usage, on-demand execution is made as transparent as possible by providing **fault-tolerant functions and insightful error message**. Example ------- >>> def count_lines(filename): ... with open(filename) as f: ... return len(f.readlines()) >>> >>> def count_words(filename): ... with open(filename) as f: ... return len(f.read().split()) >>> >>> filenames = ["a.txt", "b.txt", "c.txt", "d.txt"] >>> lc = seqtools.smap(count_lines, filenames) >>> wc = seqtools.smap(count_words, filenames) >>> counts = seqtools.collate([lc, wc]) >>> # no computations so far! >>> lc[2] # only evaluates on index 2 3 >>> counts[1] # same for index 1 (1, 2) Batteries included! ------------------- The library comes with a set of functions to manipulate sequences: .. |concatenate| image:: docs/_static/concatenate.svg .. _concatenate: https://seqtools-doc.readthedocs.io/en/stable/reference.html#seqtools.concatenate .. |batch| image:: docs/_static/batch.svg .. _batch: https://seqtools-doc.readthedocs.io/en/stable/reference.html#seqtools.batch .. |gather| image:: docs/_static/gather.svg .. _gather: https://seqtools-doc.readthedocs.io/en/stable/reference.html#seqtools.gather .. |prefetch| image:: docs/_static/prefetch.svg .. _prefetch: https://seqtools-doc.readthedocs.io/en/stable/reference.html#seqtools.prefetch .. |interleave| image:: docs/_static/interleave.svg .. _interleave: https://seqtools-doc.readthedocs.io/en/stable/reference.html#seqtools.interleave .. |uniter| image:: docs/_static/uniter.svg .. _uniter: https://seqtools-doc.readthedocs.io/en/stable/reference.html#seqtools.uniter +-------------------+---------------+ | `concatenate`_ | |concatenate| | +-------------------+---------------+ | `batch`_ | |batch| | +-------------------+---------------+ | `gather`_ | |gather| | +-------------------+---------------+ | `prefetch`_ | |prefetch| | +-------------------+---------------+ | `interleave`_ | |interleave| | +-------------------+---------------+ | `uniter`_ | |uniter| | +-------------------+---------------+ and others (suggestions are also welcome). Installation ------------ .. code-block:: bash pip install seqtools Documentation ------------- The documentation is hosted at `https://seqtools-doc.readthedocs.io `_. Contributing and Support ------------------------ Use the `issue tracker `_ to request features, propose improvements or report issues. For questions regarding usage, please send an `email `_. Related libraries ----------------- `Joblib `_, proposes low-level functions with many optimization settings to optimize pipelined transformations. This library notably provides advanced caching mechanisms which are not the primary concern of SeqTool. SeqTool uses a simpler container-oriented interface with multiple utility functions in order to assist fast prototyping. On-demand evaluation is its default behaviour and applies at all layers of a transformation pipeline. Eager evaluation of elements in SeqTools does not break the list-like interface and can be used in the middle of a transformation pipeline. SeqTools is conceived to connect nicely to the data loading pipeline of Machine Learning libraries such as PyTorch's `torch.utils.data `_ and `torchvision.transforms `_ or Tensorflow's `tf.data `_. The interface of these libraries focuses on `iterators `_ to access transformed elements, contrary to SeqTools which also provides arbitrary reads via indexing.