<div align="center">
  <a href="http://www.openucx.org/"><img src="./docs/doxygen/UCX_Logo_930x933.png" width="200"></a>
  <br>
  <a href="https://twitter.com/intent/follow?screen_name=openucx"> <img src="https://img.shields.io/twitter/follow/openucx?style=social&logo=twitter" alt="follow on Twitter"></a>
  <a href="https://openucx.github.io/ucx/api/latest/html/"><img src="docs/doxygen/api.svg"></a>
  <a href='https://openucx.readthedocs.io/en/master/?badge=master'><img src='https://readthedocs.org/projects/openucx/badge/?version=master' alt='Documentation Status' />
  <a href="https://github.com/openucx/ucx/releases/latest"><img src="docs/doxygen/release.svg"></a>
</div>

<!-- TOC generated by https://github.com/ekalinin/github-markdown-toc -->

<hr>

   * [Unified Communication X](#unified-communication-x)
      * [Using UCX](#using-ucx)
         * [Building and Running Internal Unit Tests](#building-and-running-internal-unit-tests)
         * [UCX Performance Test](#ucx-performance-test)
      * [Our Community](#our-community)
      * [Licenses](#licenses)
      * [Contributor Agreement and Guidelines](#contributor-agreement-and-guidelines)
      * [UCX Publications](#ucx-publications)
      * [UCX Architecture](#ucx-architecture)
      * [Supported Transports](#supported-transports)
      * [Supported CPU Architectures](#supported-cpu-architectures)

<hr>

# Unified Communication X

Unified Communication X (UCX) provides an optimized communication
layer for Message Passing ([MPI](https://www.mpi-forum.org/)),
[PGAS](http://www.pgas.org/)/[OpenSHMEM](http://www.openshmem.org/)
libraries and RPC/data-centric applications.

UCX utilizes high-speed networks for inter-node communication, and
shared memory mechanisms for efficient intra-node communication.

## Using UCX

### Known critical issues
* UCX version 1.8.0 has a bug that may cause data corruption when TCP transport
  is used in conjunction with shared memory transport. It is advised to upgrade
  to UCX version 1.9.0 and above. UCX versions released before 1.8.0 don't have
  this bug.   
### Release Builds

Building UCX is typically a combination of running "configure" and "make".
Execute the following commands to install the UCX system from within the
directory at the top of the tree:

```sh
$ ./autogen.sh
$ ./contrib/configure-release --prefix=/where/to/install
$ make -j8
$ make install
```

NOTE: Compiling support for various networks or other specific hardware may
require additional command line flags when running configure.

### Developer Builds

```bash
$ ./autogen.sh
$ ./contrib/configure-devel --prefix=$PWD/install-debug
```

*** NOTE: Developer builds of UCX typically include a large performance 
penalty at run-time because of extra debugging code.

### Running internal unit tests

```sh
$ make -C test/gtest test
```

### Build RPM package
```bash
$ contrib/buildrpm.sh -s -b
```

### Build DEB package
```bash
$ dpkg-buildpackage -us -uc
```

### Build Doxygen documentation
```bash
$ make docs
```

### OpenMPI and OpenSHMEM installation with UCX
[Wiki page](http://github.com/openucx/ucx/wiki/OpenMPI-and-OpenSHMEM-installation-with-UCX)

### MPICH installation with UCX
[Wiki page](http://github.com/openucx/ucx/wiki/MPICH-installation-with-UCX)

### UCX Performance Test

Start server:

```sh
$ ./src/tools/perf/ucx_perftest -c 0
```

Connect client:

```sh
$ ./src/tools/perf/ucx_perftest <server-hostname> -t tag_lat -c 1
```
Note: the `-c` flag sets CPU affinity. If running both commands on same host, make sure you set the affinity to different CPU cores.

## Our Community

* [Project Website](http://www.openucx.org/)
* [ReadTheDocs](https://openucx.readthedocs.io/en/master/)
* [Github](http://www.github.com/openucx/ucx/)
* [Software Releases](http://www.github.com/openucx/ucx/releases)
* [Mailing List](https://elist.ornl.gov/mailman/listinfo/ucx-group)
* [Twitter](https://twitter.com/openucx)

## Licenses

UCX is licensed as:

* [BSD3](LICENSE)

## Contributor Agreement and Guidelines

In order to contribute to UCX, please sign up with an appropriate
[Contributor Agreement](http://www.openucx.org/license/).

Follow these
[instructions](https://github.com/openucx/ucx/wiki/Guidance-for-contributors)
when submitting contributions and changes.

## UCX Publications

To reference UCX in a publication, please use the following entry:

```bibtex
@inproceedings{shamis2015ucx,
  title={UCX: an open source framework for HPC network APIs and beyond},
  author={Shamis, Pavel and Venkata, Manjunath Gorentla and Lopez, M Graham and Baker, Matthew B and Hernandez, Oscar and Itigin, Yossi and Dubman, Mike and Shainer, Gilad and Graham, Richard L and Liss, Liran and others},
  booktitle={2015 IEEE 23rd Annual Symposium on High-Performance Interconnects},
  pages={40--43},
  year={2015},
  organization={IEEE}
}
```

To reference the UCX website:

```bibtex
@misc{openucx-website,
    title = {{The Unified Communication X Library}},
    key = {{{The Unified Communication X Library}},
    howpublished = {{\url{http://www.openucx.org}}}
}
```

## UCX Architecture

![](docs/doxygen/Architecture.png)

| Component | Role        | Description |
| :---:     | :---:       | ---         |
| UCP | Protocol          | Implements high-level abstractions such as tag-matching, streams, connection negotiation and establishment, multi-rail, and handling different memory types |
| UCT | Transport         | Implements low-level communication primitives such as active messages, remote memory access, and atomic operations |
| UCS | Services          | A collection of data structures, algorithms, and system utilities for common use |
| UCM | Memory            | Intercepts memory allocation and release events, used by the  memory registration cache |

## Supported Transports

* [Infiniband](https://www.infinibandta.org/)
* [Omni-Path](https://www.intel.com/content/www/us/en/high-performance-computing-fabrics/omni-path-driving-exascale-computing.html)
* [RoCE](http://www.roceinitiative.org/)
* [Cray Gemini and Aries](https://www.cray.com/)
* [CUDA](https://developer.nvidia.com/cuda-zone)
* [ROCm](https://rocm.github.io/)
* Shared Memory
    * posix, sysv, [cma](https://dl.acm.org/citation.cfm?id=2616532), [knem](http://knem.gforge.inria.fr/), and [xpmem](https://github.com/hjelmn/xpmem)
* TCP/IP

## Supported CPU Architectures

* [x86_64](https://en.wikipedia.org/wiki/X86-64)
* [Power8/9](https://www.ibm.com/support/knowledgecenter/en/POWER9/p9hdx/POWER9welcome.htm)
* [Arm v8](https://www.arm.com/products/silicon-ip-cpu)
