Metadata-Version: 2.5
Name: geolens-cli
Version: 1.28.0
Summary: Apache-2.0 command-line interface for the GeoLens API. Login, scan, publish, and export STAC against any GeoLens instance.
Project-URL: Homepage, https://github.com/geolens-io/geolens
Project-URL: Repository, https://github.com/geolens-io/geolens
Project-URL: Documentation, https://docs.getgeolens.com/
Author-email: GeoLens contributors <noreply@getgeolens.com>
License: Apache-2.0
License-File: LICENSE
Keywords: cli,geolens,geospatial,openapi,stac
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: GIS
Requires-Python: >=3.11
Requires-Dist: cryptography>=50.0.0; sys_platform == 'linux'
Requires-Dist: geolens<2.0.0,>=1.27.0
Requires-Dist: idna>=3.18
Requires-Dist: jsonschema<5.0.0,>=4.19.0
Requires-Dist: keyring<26.0.0,>=25.0.0
Requires-Dist: platformdirs<5.0.0,>=4.0.0
Requires-Dist: pyyaml<7.0.0,>=6.0.0
Requires-Dist: rich<16.0.0,>=14.0.0
Requires-Dist: structlog<26.0.0,>=25.0.0
Requires-Dist: tomli-w<2.0.0,>=1.0.0
Requires-Dist: typer<0.26.0,>=0.25.0
Provides-Extra: dev
Requires-Dist: pytest<10.0.0,>=9.0.0; extra == 'dev'
Description-Content-Type: text/markdown

# geolens (CLI)

Apache-2.0 command-line interface for the [GeoLens](https://github.com/geolens-io/geolens) API.

Log in, scan local directories of spatial data, publish files, apply manifest-driven catalogs, replace and refresh datasets, inspect source status, run PostGIS analysis operations, export STAC metadata and plan an ArcGIS migration against any GeoLens instance.

## Install

```bash
pip install geolens-cli      # installs the `geolens` command; Python 3.11 or newer
```

## Quick start

```bash
geolens login https://geolens.example.com/api
geolens publish ./data/cities.geojson
geolens status DATASET_ID        # the UUID at the end of the URL publish printed
```

The CLI wraps the [`geolens`](https://pypi.org/project/geolens/) Python SDK package.

## Migrate from ArcGIS

Three commands move an ArcGIS Online organization or a self-hosted ArcGIS portal into GeoLens. They are written for an ArcGIS organization administrator, because a portal search only returns items shared with the caller and a non-admin never sees other people's private items. On the GeoLens side the session must be an admin (`geolens login` as one), since the run reads the user list and hands ownership to the people the plan names.

```bash
geolens arcgis inventory --portal-url https://yourorg.maps.arcgis.com --scope org -o ./run
geolens arcgis plan ./run/arcgis-inventory.json --owner-map owners.json
geolens arcgis migrate ./run/arcgis-migration-plan.json
```

1. `inventory` reads the portal and writes `arcgis-inventory.json` and a readable report.
2. `plan` turns it into `arcgis-migration-plan.json`, which you review and edit. Each item has an action (`import_layers`, `import_file`, `import_map`, `export_only` or `skip`), a target owner, visibility, collections and title. `owners.json` maps ArcGIS usernames to GeoLens usernames or emails; an empty value means no owner yet.
3. `migrate` runs the plan one item at a time, in dependency order. Add `--dry-run` first to check the plan and the owners without importing anything. With `--username`, the dry run also makes the one ArcGIS sign-in the run starts with, so a refused password exits 3 before the real run; a token isn't checked.

### What each action does

- `import_layers`: GeoLens imports the feature layers and tables of a hosted or federated service itself; the CLI never contacts the service.
- `import_file`: a CSV, GeoJSON, Shapefile or File Geodatabase item is downloaded from the portal and uploaded. It is bounded by the instance's upload size limit; an item over it is skipped with its size.
- `export_only`: items GeoLens can't host (dashboards, forms, StoryMaps, Hub pages and so on) are saved under `export/<type>/<item id>/` next to the state file, with `item.json`, the item's data, its resources and, for a story, a readable `story.md`. `export/INDEX.md` lists them, and the source files of hosted layers, with what to use instead. Uploaded files keep their type: a CSV is saved as `data.csv` and a PDF as `data.pdf`, from the portal's file name or the item type, and `data.bin` only when neither gives one. A `story.md` links each map to the GeoLens map made from it at the instance's `PUBLIC_APP_URL`, or at the instance URL without `/api` when `PUBLIC_APP_URL` isn't set, so set it on an install that serves the web app from another origin. A map that hasn't been migrated is linked to its ArcGIS item page instead, which stops working when the portal is retired, so replace those links before cutover. `--export-cap-mb` (default 200) limits each item. An export can finish with files missing: one over the cap, JSON that can't be redacted, a request the portal refused, or an unsafe resource name. The item notes, the report and `INDEX.md` (status `partial` or `complete, with errors`) say which, so check them before retiring the ArcGIS source. A network or server error while exporting leaves the item failed, and a rerun retries it.
- `import_map`: a Web Map becomes a GeoLens map through the server's web map translator. Its operational layers are matched to the datasets the run imported for their service items, so the map waits for those items (`depends_on`); a layer from another organization or a service that was not imported is left out. Basemaps follow the plan's `basemap_map`. A matched layer that the web map leaves to its source is drawn with the renderer saved on its layer item, read from the portal, or else its service layer's own renderer, read only from a host the run trusts with its token (`--trust-service-host`). `webmap-loss/<item id>.json` next to the state file lists what was not carried over, and the report lists the path under its loss reports and in the follow-ups. The state keeps each ArcGIS layer id's map layer id. The map gets the web map item's thumbnail, read from the portal with the run's credential, and a rerun gives it to a migrated map that has no thumbnail; a thumbnail already on the map is never replaced. A PNG or JPEG over the size GeoLens takes for a thumbnail, any other image type, or a failed read or upload is reported on the item and leaves the map without one. A dry run notes each new map that would get one.

Items are imported as you, then handed to the owner the plan names. Each target user needs a role with `upload` (datasets) or `edit_metadata` (maps). `--unmapped-owner` names an owner for items whose plan owner can't take them yet: the mapping is empty, the user hasn't signed in to GeoLens yet (an OIDC user before first sign-in), is inactive, or has a role without that permission, such as the `viewer` role new OIDC users get. Those items go to the `--unmapped-owner` for now and are recorded as waiting; a later run hands each one to its owner once the owner can take it, or to a new `--unmapped-owner` if you pass another. For a role, the run asks you to promote the user to editor instead of letting the transfer fail, and when GeoLens itself refuses a transfer because of the user's role, the item fails with the same request. Without `--unmapped-owner` such items stop the run before anything is imported, except a role without the permission, which is a warning and the server decides. When the refusal is an empty `owner_map` entry and GeoLens already has an account with the same username and the email the inventory read for that owner, the message names it; `migrate` doesn't map it for you, because an owner gets the item itself, which is more than group membership. A migrated item whose plan owner you later edit to an empty value, a name that matches no user or one that matches two is listed in the output and the report, and stays with its current owner. Folders become collections (`defaults.collection_from` in the plan: `folder` or `none`). A folder whose name another collection holds is filed under `<portal name> - <folder>`. When that name is held too, the run stops before importing anything, as it does when a group's name and `ArcGIS - <name>` are both taken; rename the folder in the items' `target.collections`, or set `collection_from` to `none`.

### Sharing and groups

ArcGIS sharing maps to GeoLens visibility: `private` stays private, `org` becomes internal, `public` stays public, and an item shared only with groups becomes `restricted`, shared with a GeoLens group for each of its ArcGIS groups. Under the `restrict_public_visibility` setting, a public item whose owner isn't a GeoLens admin becomes internal instead; when a run has public items and can't read that setting, it stops before importing anything. Each ArcGIS group's entry in the plan's `group_map` decides its GeoLens group through `geolens_group`. Left `null`, `migrate` creates a group named after the ArcGIS group (or `ArcGIS - <name>` when another group already has that name; it never joins a group it didn't create) and adds the members the inventory read. A member resolves like an owner: through `owner_map` when it lists them, otherwise by the same username or email in GeoLens, but only when the email the inventory read for that ArcGIS user, if any, is the account's. `--dry-run` and the report list which GeoLens user each ArcGIS member became, so check that a same-named account belongs to the same person. A member who isn't a GeoLens user yet, or whom GeoLens refused to add, stays pending and is added by a later run (the output counts members without a GeoLens account once and lists their names, and names `geolens admin users import --from-inventory` as the way to create them; the report keeps one line per member and group); a refused add also makes the run exit 1. Set `geolens_group` to an existing group's exact name to share with that group instead (a name that matches several groups apart from case is refused); its members are left as they are, since adding to it would also give them the rest of what it can read, and the ArcGIS members it lacks are listed. An empty string maps the ArcGIS group to no GeoLens group, and an item none of whose groups maps to one becomes private. `plan` also writes each entry's `collection`, the ArcGIS group's title, which names a group `migrate` creates only when the inventory has no title for it, and `visibility`, the group's ArcGIS access, which `migrate` doesn't read. A rerun follows the plan: an item gets the groups its plan entry now names, and loses the groups `migrate` itself added that the entry no longer names (none while its groups weren't read). Groups shared with an item in GeoLens stay, even when the plan later names them too, and a group someone removes from an item in GeoLens isn't added again or removed later. GeoLens makes a private dataset or map shared with any group restricted, so one the plan keeps private while groups shared in GeoLens remain on it is reported instead of changed. A rerun that can't bring an item up to date leaves its groups as they are and lists the changes still waiting. Sharing replaces an item's whole group list, so a group change someone makes to the same item while `migrate` shares it can be overwritten; run it while nobody is editing sharing. A map is shared with its groups without widening its layers, which keep the sharing of their own items. A member `migrate` added to a group it created is removed again on a rerun once the ArcGIS member it was added for has left the group. A member still listed but no longer resolving (renamed, deactivated, or mapped to someone else in `owner_map`) keeps the account it was added as, nothing is removed while a group's members weren't read, and anyone added or removed in GeoLens stays as they are. The output, `--dry-run` and the report list each removal. A state file from before version 4 doesn't say which members `migrate` added, so the next run lists them once for you to check by hand and removes none of them. It doesn't say which groups `migrate` shared an item with either, so a group the plan no longer names that is still on an item is listed once for you to check, and left. A rerun prints each item it shared with a group or took a group off, and the report lists them under "Item sharing changed by this run".

Only an organization administrator sees every group an item is shared with, and a group can hide its members from everyone but its managers and administrators. `plan` lists items shared only with groups that a non-admin's inventory may have missed, and the report lists, as `under_shared` follow-ups, items whose groups weren't all read or carried over. A rerun changes a migrated map's groups the same way while the map is as `migrate` left it and both it and its plan entry are private or restricted; a map changed in GeoLens since, or one whose plan now gives it another visibility, is reported rather than changed.

### Users

Owners and group members have to be GeoLens users before `migrate` can hand them items or add them to groups. `geolens admin users import users.csv` creates them from a CSV with a header row naming `username`, `email` and `role` (`admin`, `editor` or `viewer`; empty means `viewer`), and skips anyone GeoLens already has with that username and email. A row whose username belongs to an account with another email, or whose email belongs to another account, is a conflict: it isn't created, and the run exits 1. `--from-inventory arcgis-inventory.json` takes the users from an inventory instead: every item owner as an editor, with the email the inventory read, and every member of a group an item is shared with as a viewer. Add `--seed-csv users.csv` to write those rows to a new file you can review and edit, creating nothing. It refuses to overwrite a file, and a cell starting with `=`, `+`, `-` or `@` gets a leading `'` so a spreadsheet doesn't run it as a formula. The import drops that `'` again, so a name or email that really starts with `'` followed by one of those characters can't be given in a CSV. `--dry-run` says what would be created. The command needs a GeoLens admin session and exits 3 without one. It exits 1 when any user couldn't be created.

Each new user gets a generated 72-character password, the longest GeoLens accepts, so it meets any password policy. `--print-passwords` shows each one once on standard output; nothing else writes them anywhere, so without it nobody knows them and you set one for each user through the admin password reset (Admin, Users) and pass it on.

An instance that signs in through an identity provider (OpenID Connect, SAML or a sign-in extension) creates its users at their first sign-in, so the import creates nobody there: those owners and members stay pending, and a `migrate` rerun after they sign in hands them their items and adds them to their groups. `--link-by-email` creates them anyway, with a random password nobody is shown, so their first sign-in links to the account by email. GeoLens links only when the provider sends `email_verified`; without it that sign-in is refused. Users without an email stay pending. GeoLens counts an email an admin enters as verified, so whoever signs in through the provider with that address gets the account: check every email in a seeded CSV before importing it with `--link-by-email`. Ctrl-C stops before the next user and still prints the passwords made so far.

### Credentials

Public items need none. For anything else pass `--token`, `ARCGIS_TOKEN` or `--token-stdin`, or `--username` with the password in `ARCGIS_PASSWORD`, `--password-stdin` or a prompt. A token made from a password is renewed before it expires. A token you pasted is not: when it runs out the run stops, writes the state file and tells you to resume with a fresh one. Accounts that sign in with SAML or OpenID Connect have no password to give here, so use a token. Tokens and passwords are never written to the state file, the report or the export folder.

A private service hosted on a federated ArcGIS Server whose host differs from the portal's (or from `arcgis.com`) only receives the token when you name its host with `--trust-service-host <host>` (repeatable); without it the item fails before any import and the error names the host to pass.

GeoLens needs `REDIS_URL` set to take a token for a private service import; without it the instance answers 503 and the run stops with that message.

### Updating the plan

Run `plan` again on a newer inventory to bring an existing plan up to date. The plan records the defaults it gave each item, so the merge can tell your edits apart: a value you edited stays, and one you left alone takes the new default. When both changed, your value stays and `plan` prints what the inventory now gives. An edit stays an edit until you set it back to its default, even when the portal later changes to match it. New portal items are added at the end with defaults; the rest keep your order. An item gone from the portal stays in the plan with `"removed": true`; `migrate` doesn't run it again, and what it already migrated stays in GeoLens. Only a complete `--scope org` inventory read by an organization administrator marks items removed, since anyone else misses items not shared with them. Any other inventory (partial, truncated, user scope, anonymous, a non-admin account, or one written before the CLI recorded the account's role) marks the items it doesn't list `"unlisted": true`, and `migrate` leaves those pending, so the run exits 1 until a merge lists them again. If you never run a complete org inventory as an admin, an item deleted from the portal stays unlisted; set its action to `skip`, or delete it from the plan, to finish. A non-admin also may not see every group an item is shared with, so its inventory only adds groups to an item's `groups`; a group it doesn't list is kept until an admin's inventory leaves it out. `plan` and `migrate` refuse an inventory of another ArcGIS organization than the plan's, which matters on `www.arcgis.com`, where every organization shares one URL; a plan from an earlier version records no organization, so only its portal URL is checked. Your `owner_map`, `group_map`, `basemap_map` and plan `defaults` entries win over the inventory and `--owner-map`. To leave an item out, set its action to `skip`: an item deleted from the plan comes back with defaults on the next merge. `--force` replaces the plan with fresh defaults instead.

Plans written by earlier versions still migrate. They record no defaults and no inventory scope, so their items keep every value as an edit on every merge, `depends_on` included. Delete an item from the plan, or use `--force`, to give it defaults that follow the portal.

### Resuming and reports

Progress is saved in `arcgis-migration-state.json` (mode 0600) after every item. Run the same command again to resume: failed items are retried (`--no-retry` leaves them alone), and layers already imported are never imported twice. `--only <item id>` runs one item plus the unfinished items it depends on. Only one run can use a state file at a time. A run that finds no state file at its path checks whether GeoLens already holds datasets imported from the plan's items, and when it does it stops before importing anything and names a few of them, since it would import them all again. Point `--state` at the earlier run's state file, or pass `--start-fresh` to migrate them again anyway.

A rerun first hands waiting items to their owners, then imports what isn't finished, then brings every item migrated before up to date. Each dataset of a service item is refreshed from its source with the run's ArcGIS credential, so private items need `--token` or `--username` on every rerun. A data file item is downloaded again and replaces its datasets. An `export_only` item is exported again when the portal changed it since its bundle was written, which the new bundle replaces in one step; if that export fails, the earlier bundle stays and the item is reported as not up to date. A service item is probed even when it had nothing to import, so a layer added since is reported. A migrated web map is not translated again: one whose portal item changed since its import is reported as not up to date, so you can update the GeoLens map by hand. The item's metadata is written again on each dataset that a refresh or replacement published, and a field the portal has emptied since is emptied too, unless it was edited in GeoLens. A visibility the portal's sharing changed since the import is applied to each dataset, unless it was changed in GeoLens or the state file can't tell, which is reported instead. Every service item is refreshed, because an item's `modified` date doesn't move when its features are edited. A refresh that stops for review (live edits, a destructive schema change, an empty result) publishes nothing: the run reports it with the `geolens refresh <dataset> --accept-blocked-run <run>` command that accepts it, and never accepts it for you. A rerun recognises the run you accept it with, and a refresh GeoLens accepted but whose answer was lost, as this command's own. A migrated web map whose owner was changed in GeoLens is reported, and one waiting for its planned owner is not handed over after such a change. `--only` limits the rerun to the items it names.

A rerun never overwrites a change made in GeoLens. After each write the state file records the dataset's last update, feature count and data revision, which counts every feature and column edit, your own included. Each dataset is compared right before its own refresh or replacement; one whose values differ (edited, transferred or refreshed in GeoLens) is reported and left alone, and so is one deleted there. The request carries the recorded revision, so GeoLens stops the publication for review if the data is edited after the check, and a file replacement is checked again after its upload. A GeoLens too old to report data revisions gets no refresh or replacement at all; each dataset is reported until you upgrade it. A refresh that doesn't publish leaves the dataset's record as it was, and one edited while its refresh ran is reported and not recorded. Set the item's action to `skip` to stop bringing it up to date. Change a finished `export_only` or `skip` item's action to `import_file`, `import_layers` or `import_map` and the next run imports it (the export bundle stays on disk but is no longer tracked); other action changes on a finished item aren't acted on. A state file from an earlier version has no recorded values, so a dataset is taken as unchanged only when your GeoLens account made its last change, before the item finished. The state file is now version 4, which an earlier CLI refuses to read rather than rewrite.

A run that gets as far as the item loop writes `arcgis-migration-report.md` and `.json` next to the state file, also when it is interrupted or stops on a failed request. `--dry-run` writes none, and neither does a run refused before it starts (an unusable plan, state file or owner mapping, or a failed GeoLens check). The report covers every item of the plan: the manual follow-ups (failed items, skipped files and layers, group-only sharing, partial exports, items that can't run yet or wait on others) and the unsupported items with where each was saved.

Exit codes:

- 0: no item of the plan is failed, no item the run selected was left waiting or unable to run, and no group membership change failed; or a `--cutover` run that didn't stop refused nothing, because every check that failed was accepted with its `--accept-*` flag, and no group membership change failed. Skipped items (a file over the upload limit or with too many layers) and pending items the run did not select, such as items that can't run yet, don't change it, so read the report's counts and follow-ups before treating the migration as complete.
- 1: an item of the plan is failed (including one this run left out with `--only` or `--no-retry`, and a portal download or GeoLens server error once items are running), an item the run selected was left pending because its action can't run yet or it waits on another item, a migrated item couldn't be brought up to date or handed to its owner, a group membership change failed, `--cutover` refused (any failing check left unaccepted), or the run stopped (for example on an expired pasted token).
- 2: the plan, state file, owner mapping or credentials couldn't be used, the GeoLens session isn't an admin's, a group or collection name the run needs is taken, there is no state file while GeoLens holds datasets migrated from the plan's items, or the portal URL redirected before any item ran. `--dry-run` exits 2 on a taken name or those datasets too. A redirect refused once items are running fails that item (exit 1).
- 3: authentication failed before any item ran (the GeoLens checks, or an ArcGIS password sign-in), or the server rejected a `GEOLENS_TOKEN` at any point, which stops the run with this code. A stored login that is rejected once items are running fails that item (exit 1).
- 4: the portal couldn't be reached while signing in, or a GeoLens request couldn't connect or timed out at any point, which stops the run with this code. The exception is a timeout on the status poll of an import job: it is retried until the poll's deadline, and a deadline that runs out fails that item (exit 1). A portal download that fails once items are running also fails only that item (exit 1).
- 5: a server error from the GeoLens checks, before any item ran.
- 130: interrupted; run the same command to resume.

For a large migration, run the GeoLens worker with `WORKER_CONCURRENCY=2` or more, since each layer import and refresh is a background job.

### Cutover

A migration ends with one last run after the portal stops changing, marked with `--cutover`:

1. Tell the portal's users that it is frozen, and stop edits to its items and services.
2. Run `inventory --scope org` again as an organization administrator, then `plan` on it, so the plan holds everything the portal has now.
3. Run `migrate --dry-run --cutover` to see what the cutover would refuse, and fix what you can: retry failed items, map owners, promote owners to editor, accept or fix refreshes that stopped for review.
4. Run `migrate --cutover` with the same credentials as before. It runs like any rerun, then checks what wouldn't carry over and marks the run as the final one only when nothing is left, or when you accept each loss:
   - `--accept-unfinished-items`: a plan item isn't done (failed, skipped, can't run yet or waits on another item), is unlisted (even a finished one, since a rerun doesn't bring it up to date) or isn't up to date (its refresh failed or stopped for review, or GeoLens changed it). Items set to `skip` and items gone from the portal don't count.
   - `--accept-incomplete-inventory`: the plan wasn't last merged from a complete `--scope org` inventory read by an organization administrator, or the portal has items of the organization changed since that inventory started reading it, by the portal's own clock, so the plan may miss or misstate them. An inventory written by an earlier version, or by a portal that sent no time, records no start and fails this check; run it again. The run asks the portal with its ArcGIS credential, which must be an organization administrator's; without one, or when the portal can't answer, this check fails too.
   - `--accept-attachment-loss`: a migrated layer has attachments, or its service doesn't say. GeoLens doesn't import attachments yet.
   - `--accept-unresolved-users`: an item still waits for its owner, its plan owner doesn't name one GeoLens user, or its plan now names another owner than the one it was given; or a member of an ArcGIS group isn't in its GeoLens group yet.
   - `--accept-under-sharing`: an item shared only with ArcGIS groups may reach fewer people in GeoLens: the inventory wasn't read by an organization administrator, one of its groups maps to no GeoLens group, or a group's members couldn't be read.
5. Read `arcgis-migration-report.md`, which records the cutover and the losses you accepted, and keep it with the state file. Then retire the portal. A refresh after that fails with the service's error, and the datasets stay as they are.

A refused cutover exits 1 and leaves the run unmarked; nothing it imported or refreshed is undone. A marked cutover exits 0 even when the items you accepted are still failed or not up to date, and the report lists them. `--cutover` runs the whole plan, so it can't be combined with `--only`.

## Documentation

The full command reference, manifest schema, exit codes and CI setup are in the [CLI guide](https://docs.getgeolens.com/guides/cli/).
