Metadata-Version: 2.4
Name: cloudrepo-migrate
Version: 0.7.0
Summary: Move a Nexus 2, Nexus 3 or Artifactory server's Maven and npm repositories, and Nexus 3's PyPI and Docker repositories, into CloudRepo, verifying every file.
Project-URL: Homepage, https://www.cloudrepo.io
Requires-Python: >=3.8
Description-Content-Type: text/markdown

# cloudrepo-migrate

Moves a Nexus 2, Nexus 3 or Artifactory server's repositories into CloudRepo and verifies every file: `cloudrepo-migrate nexus2`, `cloudrepo-migrate nexus3` or `cloudrepo-migrate artifactory`, with the same flags, report and exit codes.

- **Maven**: every file of each hosted (Nexus) or local (Artifactory) Maven repository, POMs, `maven-metadata.xml` and checksum files included.
- **npm**: every version of each package in each hosted or local npm repository, scoped packages included, oldest first, then each dist-tag the source has (`latest`, `next` and every other) that CloudRepo lacks. A dist-tag CloudRepo already holds is never moved or removed: each one that names another version than the source's, or that the source lacks, is listed as `kept` in the report, which is not a failure. A deprecated version moves with its deprecation message; a version the source has unpublished is not in its packument, so it does not move. This is the default from version 0.4.0 for Nexus, and from 0.5.0, the first version with `artifactory`, for Artifactory. Before 0.4.0, `nexus2` and `nexus3` moved only the version tagged `latest`, which `--latest-only` still does. For Artifactory, npm is tested against a stand-in for Artifactory's npm API, not yet against a real Artifactory server.
- **PyPI** (Nexus 3; Artifactory from 0.7.0, tested against a stand-in for Artifactory's PyPI API, not yet against a real Artifactory server): every wheel and sdist of each hosted (Nexus 3) or local (Artifactory) PyPI repository, uploaded through CloudRepo's PyPI upload form under the project name the file's own metadata gives (other files beside them, such as signatures, are counted in the report, not moved). `--login` does not cover PyPI yet: move it with `CLOUDREPO_TOKEN`.
- **Docker** (Nexus 3 from 0.6.0; Artifactory from 0.7.0, tested against a stand-in for Artifactory's Docker API, not yet against a real Artifactory server): every tag of each hosted (Nexus 3) or local (Artifactory) Docker repository, with its manifest and every layer it names, a multi-platform tag with each platform's image (those first). Each layer must hash to the digest its manifest names and is uploaded by that digest; a layer CloudRepo already holds is not uploaded again, and one several images share is uploaded once. Each manifest is uploaded as the source serves it and read back by digest. `CLOUDREPO_TOKEN` must be a CloudRepo repository token with write access to the target Docker repository (no username to set). `--login` does not cover Docker yet.
- **Skipped with a reason**: proxy and remote repositories (create a CloudRepo proxy instead), group repositories (a Maven or Docker group: create a CloudRepo group of the moved ones; an npm or PyPI group: point the client at a hosted repository for your own packages plus a proxy repository for the public ones, since CloudRepo groups are Maven and Docker only; a group of any other format: CloudRepo has no group for that format, so point the client at the moved hosted repositories directly), virtual and federated repositories, other formats, and files the server generates itself (Nexus's `.index`, `.meta` and root `archetype-catalog.xml`; Artifactory's `.index` and `.npm`). An Artifactory repository's root `archetype-catalog.xml` is not moved either (this version moves no archetype catalog), and the report names it on that repository's line.

Before upload, each file's bytes are compared with the checksum the source recorded for it (Nexus 2's sha1; every checksum a Nexus 3 asset lists, a wheel's or sdist's sha256 among them; the sha1 and sha256 Artifactory's AQL search lists, which Artifactory OSS answers as Pro does; an npm package's `dist.shasum` and `dist.integrity`), and a file that differs is not uploaded. After upload, every Maven, npm and PyPI file is read back from CloudRepo and its sha1 and sha256 compared with the source's bytes; a wheel or sdist must also be listed in its project's index, where pip looks for it. For Docker, every blob and manifest is checked by sha256 digest before upload and each manifest is read back by digest, but layers are not read back. A checksum file the source carries (`X.sha1`, `X.sha256`, `X.sha512`, `X.md5`) must name the digest of `X`. Artifactory serves `X.sha1` and `X.md5` without storing them, so for Artifactory the run writes both from `X`'s bytes as downloaded and checked, never from Artifactory's numbers alone, and Maven finds them in CloudRepo as it did in Artifactory. Any mismatch fails the run and names the file. The source is only read, never written.

## Install

```sh
pipx install cloudrepo-migrate
```

That installs it from pypi.org and puts `cloudrepo-migrate` on the PATH (`pip install cloudrepo-migrate` in a virtual environment works too).

It is one file, Python 3.8 or later, with no dependencies, so copying `cloudrepo_migrate.py` to a machine that can reach the Nexus or Artifactory server and running `python3 cloudrepo_migrate.py --help` works too.

## Run

Create each target repository in CloudRepo first (same name as the source repository, or map it with `--repo SOURCE=TARGET`) at https://admin.cloudrepo.io/repositories/create, and a repository token that can write them.

Set each credential in your own shell (bash or zsh). Each `IFS= read -rs` waits for you to type or paste the value and press Enter: nothing is shown, the value never lands on a command line or in the shell's history, and `IFS=` keeps a space or tab at either end of it.

```sh
export NEXUS_USERNAME=deployer                    # leave NEXUS_USERNAME and NEXUS_PASSWORD unset for anonymous read
IFS= read -rs NEXUS_PASSWORD && export NEXUS_PASSWORD
IFS= read -rs CLOUDREPO_TOKEN && export CLOUDREPO_TOKEN
cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org acme --dry-run
cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org acme --report-json report.json
# Nexus 3: the same, with its base URL (no /nexus on a default install)
cloudrepo-migrate nexus3 --nexus-url https://nexus.example.com --org acme --dry-run
cloudrepo-migrate nexus3 --nexus-url https://nexus.example.com --org acme --report-json report.json
```

Artifactory (self-hosted or JFrog Cloud), with an access token, or a legacy API key in `ARTIFACTORY_API_KEY`:

```sh
IFS= read -rs ARTIFACTORY_ACCESS_TOKEN && export ARTIFACTORY_ACCESS_TOKEN   # read access to the local repositories you move
IFS= read -rs CLOUDREPO_TOKEN && export CLOUDREPO_TOKEN
cloudrepo-migrate artifactory --artifactory-url https://artifactory.example.com/artifactory --org acme --dry-run
cloudrepo-migrate artifactory --artifactory-url https://artifactory.example.com/artifactory --org acme --report-json report.json
```

| Flag                                 | Meaning                                                                                                                 |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- |
| `--nexus-url URL`                    | The Nexus base URL: ending in `/nexus` on a default Nexus 2 install, the bare host on Nexus 3.                          |
| `--artifactory-url URL`              | For `artifactory`: the Artifactory base URL, ending in `/artifactory`.                                                  |
| `--org ORG` or `--cloudrepo-url URL` | Your CloudRepo organization, or its full base URL (https).                                                              |
| `--repo SOURCE[=TARGET]`             | Move only these repositories (repeatable). Default: every repository of a format the list above moves from that server. |
| `--dry-run`                          | List what would move. Uploads nothing. Checks the token and targets when `CLOUDREPO_TOKEN` is set.                      |
| `--latest-only`                      | For npm, move only the version each package's `latest` dist-tag names (default: every version, with its dist-tags).     |
| `--report-json PATH`                 | Also write the report as JSON.                                                                                          |
| `--json`                             | Print the report as one JSON document on stdout, and the text on stderr (see JSON result, below).                       |
| `--jobs N`                           | Parallel Maven and PyPI uploads (default 4). npm packages and Docker tags move one at a time.                           |
| `--work-dir DIR`                     | Where each file is spooled while it moves (default `TMPDIR`).                                                           |
| `--nexus-ca-file PEM`                | CA bundle for a Nexus with a private certificate (`--artifactory-ca-file` for Artifactory).                             |

With the environment set, nothing prompts. On a terminal with a credential missing it asks for it.

**Resume**: run the same command again. A file CloudRepo already holds with the same bytes is counted as already present and not uploaded again; a `409` on a release CloudRepo already holds counts as moved once its bytes match.

**Exit codes**: `0` everything moved and verified (with `--dry-run`: planned, and nothing written); `1` at least one file failed (every failure is listed with its path; re-run to resume); `2` a usage or preflight problem, such as a missing target repository or a refused token (nothing moved); `4` the `--login` approval was denied, expired or refused (nothing moved); `130` stopped with Ctrl-C (re-run to resume).

## JSON result

`--json` prints one JSON document on stdout, the same one `--report-json` writes, and the text report on stderr. A dry run of two Nexus 3 repositories prints:

```json
{
  "exit_code": 0,
  "source": "https://nexus.example.com",
  "target": "https://acme.mycloudrepo.io",
  "dry_run": true,
  "repositories": [
    {
      "source": "central",
      "target": "",
      "format": "maven2",
      "action": "skipped",
      "reason": "proxy of https://repo1.maven.org: create a CloudRepo proxy repository instead",
      "files": 0,
      "bytes": 0,
      "verified": 0,
      "already_present": 0,
      "failed": 0
    },
    {
      "source": "releases",
      "target": "releases",
      "format": "maven",
      "action": "would move",
      "reason": "",
      "files": 9,
      "bytes": 21880,
      "verified": 0,
      "already_present": 0,
      "failed": 0
    }
  ],
  "failures": [],
  "kept": [],
  "minutes": 0.0,
  "totals": {
    "files": 9,
    "bytes": 21880,
    "verified": 0,
    "already_present": 0,
    "repositories": 1,
    "failed": 0
  }
}
```

- `exit_code`: the run's exit code, as above.
- `dry_run`: `true` with `--dry-run`, which writes nothing to CloudRepo.
- `source` and `target`: the source's base URL and CloudRepo's.
- `repositories`: one entry per repository. Its `action` is `would move` (a dry run), `moved`, `skipped`, or, when a target is missing or refused (exit code `2`, nothing moved), `blocked` or `ready`. Its `reason` says why it was skipped or blocked, or notes what a moved one left behind. Its `format` is CloudRepo's name for a format the tool moves, else the source's. `files`, `bytes`, `verified`, `already_present` (verified files CloudRepo already held) and `failed` count its files.
- `failures`: each file that failed, with its `repository`, `path` and `reason`.
- `kept`: each npm dist-tag CloudRepo already held and kept as it was, with its `repository`, `path` (the package), `tag`, `version` (CloudRepo's) and `source` (the source's version, or `null`).
- `totals`: `repositories`, `files`, `bytes`, `verified`, `already_present` and `failed`, over the repositories that move.
- `minutes`: how long the run took.

A run that stops without a report (exit code `2` for a usage problem, `4` or `130`) prints only `exit_code` and `error`, its message. A command line that cannot be parsed prints its usage on stderr, nothing on stdout, and exits `2`.

## Credentials

- Read only from the environment or a terminal prompt: never from the command line, never printed, never written to the report.
- Artifactory: `ARTIFACTORY_ACCESS_TOKEN` is sent as `Authorization: Bearer`; without it, `ARTIFACTORY_API_KEY` is sent as `X-JFrog-Art-Api` (JFrog has deprecated API keys, so prefer a token). Neither is ever prompted for.
- The source's login is sent only to `--nexus-url` or `--artifactory-url`, the CloudRepo token only to CloudRepo.
- Redirects are never followed, and no URL is taken from a server's answer (a Nexus 3 asset's `downloadUrl` is never requested; a packument's `dist.tarball` contributes only its file name), so neither credential can be steered to another host.
- CloudRepo must be https (plain http only to localhost). A plain-http Nexus or Artifactory gets a warning.

## For agents

A coding agent can drive the whole move. The same steps ship as an Agent Skill, https://www.cloudrepo.io/docs/agents/cloudrepo-migrate/SKILL.md, and each guide has them for its server: [Nexus 2](https://www.cloudrepo.io/docs/migrate/from-nexus-2.html#for-agents), [Nexus 3](https://www.cloudrepo.io/docs/migrate/from-nexus-3.html#for-agents), [Artifactory](https://www.cloudrepo.io/docs/migrate/from-artifactory.html#for-agents).

Tested against a real server of each kind:

- Nexus 2: hosted Maven and npm repositories.
- Nexus 3: hosted Maven, npm, PyPI and Docker repositories.
- Artifactory: local Maven repositories.

Ask the human to set up:

1. A CloudRepo organization ([sign up](https://www.cloudrepo.io/docs/get-started/sign-up.html) if they have none).
2. A CloudRepo repository for each one that moves, of the same format and with the source repository's name (or the names to map with `--repo SOURCE=TARGET`), at https://admin.cloudrepo.io/repositories/create.
3. A CloudRepo repository token with Read + write on those repositories.
4. Read access to the Nexus or Artifactory repositories: a Nexus login that can read them (none when Nexus allows anonymous reads), or an Artifactory access token (a legacy API key works too).

Ask them for the server's base URL and the CloudRepo organization.

Every credential is an environment variable the human sets in their own shell: never on a command line, never in a file, and never pasted into the chat. The tool reads only these:

- `NEXUS_USERNAME` and `NEXUS_PASSWORD`, for `nexus2` and `nexus3` (leave both unset to read anonymously).
- `ARTIFACTORY_ACCESS_TOKEN`, for `artifactory`, or `ARTIFACTORY_API_KEY` for a legacy API key.
- `CLOUDREPO_TOKEN`, the repository token.

Your commands see only the environment you were started with. Check each variable without printing it, such as `test -n "$CLOUDREPO_TOKEN"`. If one is missing, ask the human to set it in their shell and start you again from that shell, or to run the commands themselves. If a credential is pasted into the chat anyway, do not use it: ask the human to replace it.

Steps:

1. Check that `python3 --version` is 3.8 or later. If `cloudrepo-migrate --version` does not answer, run `pipx install cloudrepo-migrate`; if it answers a version older than 0.6.0, run `pipx upgrade cloudrepo-migrate`.
2. Run the dry run, with the human's address and organization. Expect exit code `0` and a last line that starts `Would move`. On exit code `2`, read the message: if a target repository is missing, ask the human to create it, then repeat the dry run. If the dry run lists a repository outside the tested set, ask the human before moving it, and name the ones to move with `--repo`.

   ```sh
   cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org your-org --dry-run
   cloudrepo-migrate nexus3 --nexus-url https://nexus.example.com --org your-org --dry-run
   cloudrepo-migrate artifactory --artifactory-url https://artifactory.example.com/artifactory --org your-org --dry-run
   ```

3. Run the same command with `--report-json report.json` in place of `--dry-run`.
4. Verify all of these: the exit code is `0`; the last line starts `Moved` and ends `V of F verified, 0 failed.` with `V` equal to `F`; in `report.json`, `failures` is empty, `totals.failed` is `0` and `totals.verified` equals `totals.files`; and no entry in `repositories` has the action `blocked`.
5. Tell the human the last line, every repository with the action `skipped` and its reason, and every entry in `kept`.

On exit code `1`, read each failure, remove its cause, and run the same command again: it resumes. If the same file fails twice, stop and report its path and reason. Change nothing on the Nexus or Artifactory server, and do not repoint builds or CI to CloudRepo until the human says so.

## Browser login (not live yet)

CloudRepo's server side for `--login` is not live yet; until it is, use `CLOUDREPO_TOKEN` as above. With it, `nexus2 --org ORG --login` (or `nexus3`, or `artifactory`) needs no token: the run builds its plan, prints a code and the page where your organization's owner approves exactly those repositories, and waits. The token it receives lives in memory for that run only, never on disk. `--open-browser` also opens the page on this machine. A set `CLOUDREPO_TOKEN` still wins. Exit code `4`: the approval was denied, expired or refused (nothing moved).
