HYPER-SHELL(1)

NAME
hyper-shell - Process shell commands over a distributed, asynchronous queue

SYNOPSIS

hs [-h] [-v] [--citation] ...
    Top-level command. Show help, version, or citation info.

hs cluster [-h] FILE [--ssh HOST... | --mpi | --launcher ARGS...] ...
    Run managed cluster.

hs server [-h] FILE ...
    Run stand-alone server.

hs client [-h] ...
    Run stand-alone client.

hs submit [-h] [ARGS... | -f FILE] ...
    Submit tasks from file.

hs initdb [-h] [--truncate | --vacuum | --rotate | --backup PATH] [--yes]
    Initialize database.

hs info [-h] ID [--stdout | --stderr | -x FIELD] [-f FORMAT]
    Get metadata and/or task outputs.

hs wait [-h] ID [-n SEC] [--info [-f FORMAT] | --status | --return]
    Wait for task to complete.

hs run [-h] [-n SEC] [-t TAG...] -- ARGS...
    Submit individual task and wait for completion.

hs list [-h] [FIELD [FIELD ...]] [-w COND [COND ...]] [-t TAG [TAG...]] ...
    Search tasks in the database.

hs update [-h] ARG [ARG...] [--cancel | --revert | --delete] ...
    Update tasks in the database.

hs config [-h] {get | set | which | edit } ...
    Configuration management.

DESCRIPTION

HyperShell is an elegant, cross-platform, high-throughput computing utility for processing shell commands over a distributed, asynchronous queue. It is a highly scalable workflow automation tool for many-task scenarios.

Typically, ad hoc usage or batch jobs will use the cluster workflow. This automatically stands up the server and one or more client instances on remote servers and processes the commands from some input FILE until completion.

Operate with an in-memory queue, or configure a database to manage task scheduling and persistence. Stand up the server on its own and scale clients as desired, and submit tasks independently.

CLUSTER USAGE

hs cluster [-h] [FILE | --restart | --forever]
    [-N NUM] [-t CMD] [-b SIZE] [-w SEC] [-p PORT] [-r NUM [--eager]] [-f PATH] [--capture | [-o PATH] [-e PATH]] [--ssh [HOST... | --ssh-group NAME] [--env] | --mpi | --launcher=ARGS...] [--no-db | --initdb] [--no-confirm] [-d SEC] [-T SEC] [-W SEC] [-S SEC] [--autoscaling [MODE] [-P SEC] [-F VALUE] [-I NUM] [-X NUM] [-Y NUM]]

    Start cluster locally or with remote clients over ssh or a custom launcher. This mode should be the most common entry-point for general usage. It fully encompasses all of the different agents in the system in a concise workflow.

    The input source for tasks is file-like, either a local path, or from stdin if no argument is given. The command-line tasks are pulled in and either directly published to a distributed queue (see --no-db) or committed to a database first before being scheduled later.

    For large, long running workflows, it might be a good idea to configure a database and run an initial submit job to populate the database, and then run the cluster with --restart and no input FILE. If the cluster is interrupted for whatever reason it can gracefully restart where it left off.

    Use --autoscaling with either fixed or dynamic to run a persistent, elastically scalable cluster using an external --launcher to bring up clients as needed.

    Use hsx in place of hs cluster.

  Arguments

FILE
    Path to input task file (default: <stdin>).

  Modes

--ssh HOST...
    Launch directly with SSH host(s). This can be a single host, a comma-separated list of hosts, or an expandable pattern, e.g., "cluster-a[00-04].xyz".

    See also --ssh-group and --ssh-args.

--mpi
    Same as --launcher=mpirun.

--launcher ARGS...
    Use specific launch interface. This can be any program that handles process management on a distributed system. For example, on a SLURM cluster one might want to use srun. In this case you would specify --launcher=srun; however, the ARGS are not merely the executable but the full listing, e.g., --launcher='srun --mpi=pmi2'.

  Options

-N, --num-tasks NUM
    Number of task executors per client (default: 1).

    For example, -N4 would create four workers, but -N4 --ssh 'cluster-a[00-01].xyz' creates two clients and a total of eight workers.

-t, --template CMD
    Command-line template pattern (default: "{}").

    This is expanded by the client just before execution. With the default "{}" the input command-line will be run verbatim. Specifying a template pattern allows for simple input arguments (e.g., file paths) to be transformed into some common form; such as -t './some_command.py {} >outputs/{/-}.out'.

    See section on templates.

-p, --port NUM
    Port number (default: 50001).

    This is an arbitrary choice and simply must be an available port. The default option chosen here is typically available on most platforms and is not expected by any known major software.

-b, --bundlesize SIZE
    Size of task bundle (default: 1).

    The default value allows for greater concurrency and responsiveness on small scales. This is used by the submit thread to accumulate bundles for either database commits and/or publishing to the queue. If a database is in use, the scheduler thread selects tasks from the database in batches of this size.

    Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

    See also --num-tasks and --bundlewait.

-w, --bundlewait SEC
    Seconds to wait before flushing tasks (default: 5).

    This is used by both the submit thread and forwarded to each client. The client collector thread that accumulates finished task bundles to return to the server will push out a bundle after this period of time regardless of whether it has reached the preferred bundle size.

    See also --bundlesize.

-r, --max-retries NUM
    Auto-retry failed tasks (default: 0).

    If a database is in use, then there is an opportunity to automatically retry failed tasks. A task is considered to have failed if it has a non-zero exit status. Setting this value greater than zero defines the number of attempts for the task. The original is not over-written, a new task is submitted and later scheduled.

    See also --eager.

--eager
    Schedule failed tasks before new tasks. If --max-retries is greater than one, this option defines the appetite for re-submitting failed tasks. By default, failed tasks will only be scheduled when there are no more remaining novel tasks.

--no-db
    Disable database (submit directly to clients).

    By default, a scheduler thread selects tasks from a database that were previously submitted. With --no-db enabled, there is no scheduler and instead the submit thread publishes bundles directly to the queue.

--initdb
    Auto-initialize database.

    If a database is configured for use with the workflow (e.g., Postgres), auto-initialize tables if they don't already exist. This is a short-hand for pre-creating tables with the hs initdb command. This happens by default with SQLite databases.

    Mutually exclusive to --no-db. See hs initdb command.

--no-confirm
    Disable client confirmation of task bundle received.

    To achieve even higher throughput at large scales, optionally disable confirmation payloads from clients. Consider using this option when also using --no-db.

--forever
    Schedule forever.

    Typically, the cluster will process some finite set of submitted tasks. When there are no more tasks left to schedule, the cluster will begin its shutdown procedure. With --forever enabled, the scheduler will continue to wait for new tasks indefinitely.

    Conflicts with --no-db and mutually exclusive to --restart.

--restart
    Start scheduling from last completed task.

    Instead of pulling a new list of tasks from some input FILE, with --restart enabled the cluster will restart scheduling tasks where it left off. Any task in the database that was previously scheduled but not completed will be reverted.

    For very large workflows, an effective strategy is to first use the submit workflow to populate the database, and then to use --restart so that if the cluster is interrupted, it can easily continue where it left off, halting if nothing to be done.

    Conflicts with --no-db and mutually exclusive to --forever.

--ssh-args ARGS...
    Command-line arguments for SSH. For example, --ssh-args '-i ~/.ssh/my_key'.

--ssh-group NAME
    SSH nodelist group in config.

    In your configuration under [ssh.nodelist] can be one or more named lists. These lists should contain host names to associate with the group name.

    See configuration section.

-E, --env
    Send environment variables. Only for --ssh mode, all HYPERSHELL_ prefixed environment variables can be exported to the remote clients.

--remote-exe PATH
    Path to remote executable on the client side.

    By default the executable path used to launch clients is the same as that for the cluster. This option is necessary for clients with a different install path on client hosts.

-d, --delay-start SEC
    Delay time in seconds for launching clients (default: 0).

    At larger scales it can be advantageous to uniformly delay the client launch sequence. Hundreds or thousands of clients connecting to the server all at once is a challenge. Even if the server could handle the load, your task throughput would be unbalanced, coming in waves.

    Use --delay-start with a negative number to impose a uniform random delay up to the magnitude specified (e.g., --delay-start=-600 would delay the client up to ten minutes). This also has the effect of staggering the workload. If your tasks take on the order of 30 minutes and you have 1000 nodes, choose --delay-start=-1800.

-c, --capture
    Capture individual task <stdout> and <stderr>.

    By default, the stdout and stderr streams of all tasks are fused with that of the client thread, and in turn the cluster. If tasks are producing output that needs to be isolated, the tasks need to manage their own output, you can specify a redirect as part of a --template, or use --capture to capture these as .out and .err files.

    These are stored local to the client. Task outputs can be automatically retrieved via SFTP, see task usage.

-o, --output PATH
    File path for task outputs (default: <stdout>).

    If local only (not --ssh, --mpi or --launcher), then the client can redirect all stdout from tasks to some file PATH together.

-e, --errors PATH
    File path for task errors (default: <stderr>).

    If local only (not --ssh, --mpi or --launcher), then the client can redirect all stderr from tasks to some file PATH together.

-f, --failures PATH
    File path to write failed task args (default: <none>).

    The server acts like a sieve, reading task args from stdin and redirecting those original args to stdout if the task had a non-zero exit status. The cluster will run the server for you and if --failures is enabled these task args will be sent to a local file PATH.

-T, --timeout SEC
    Timeout in seconds for clients. Automatically shutdown if no tasks received (default: never).

    This option is only valid for an --autoscaling cluster. This feature allows for gracefully scaling down a cluster when task throughput subsides.

-W, --task-timeout SEC
    Task-level walltime limit (default: none).

    Executors will send a progression of SIGINT, SIGTERM, and SIGKILL. If the process still persists the executor itself will shutdown.

-S, --signalwait SEC
    Task-level signal escalation wait period in seconds (default: 10).

    When tasks fail to halt following an initial SIGINT, the program waits this interval in seconds before escalating to the next level of interrupt.

    See also --task-timeout.

-A, --autoscaling [MODE]
    Enable autoscaling (default: disabled). Used with --launcher.

    Specifying this option on its own triggers the use of the autoscaler, with the default policy or the configured policy. The policy can be specified directly here as either fixed or dynamic (e.g., --autoscaling=dynamic). The default is fixed.

    The specified --launcher is used to bring up each individual instance of the client as a discrete scaling unit. This is different than using --launcher on its own where it specifies a single invocation that should launch all clients (e.g., like an mpirun). Without this option, clients will simply be run locally.

    A fixed policy will seek to maintain a definite size and allows for recovery in the event that clients halt for some reason (e.g., due to expected faults or timeouts).

    A dynamic policy maintains a --min-size (default: 0) and grows up to some --max-size depending on the observed task pressure given the specified scaling --factor.

    See also --factor, --period, --init-size, --min-size, and --max-size.

-F, --factor VALUE
    Scaling factor (default: 1).

    A configurable, dimensionless quantity used by the --autoscaling=dynamic policy. This value expresses some multiple of the average task duration in seconds.

    The autoscaler periodically checks toc / (factor x avg_duration), where toc is the estimated time of completion for all remaining tasks given current throughput of active clients. This ratio is referred to as task pressure, and if it exceeds 1, the pressure is considered high and we will add another client if we are not already at the given --max-size of the cluster.

    For example, if the average task length is 30 minutes, and we set --factor=2, then if the estimated time of completion of remaining tasks given currently connected executors exceeds 1 hour, we will scale up by one unit.

    See also --period. Only valid with --autoscaling.

-P, --period SEC
    Scaling period in seconds (default: 60).

    The autoscaler waits for this period of time in between checks and scaling events. A shorter period makes the scaling behavior more responsive but can effect database performance if checks happen too rapidly.

    Only valid with --autoscaling.

-I, --init-size SIZE
    Initial size of cluster (default: 1).

    When the cluster starts, this number of clients will be launched. For a fixed policy cluster, this should be given with a --min-size, and likely the same value.

    Only valid with --autoscaling.

-X, --min-size SIZE
    Minimum size of cluster (default: 0).

    Regardless of autoscaling policy, if the number of launched clients drops below this value we will scale up by one. Allowing --min-size=0 is an important feature for efficient use of computing resources in the absence of tasks.

    Only valid with --autoscaling.

-Y, --max-size SIZE
    Maximum size of cluster (default: 2).

    For a dynamic autoscaling policy, this sets an upper limit on the number of launched clients. When this number is reached, scaling stops regardless of task pressure.

    Only valid with --autoscaling.

SERVER USAGE

hs server [-h] [FILE | --forever | --restart]
    [-b NUM] [-w SEC] [-r NUM [--eager]] [-H ADDR] [-p PORT] [-k KEY] [--no-db | --initdb] [--print | -f PATH] [--no-confirm]

    Launch server, schedule directly or asynchronously from database.

    The server includes a scheduler component that pulls tasks from the database and offers them up on a distributed queue to clients. It also has a receiver that collects the results of finished tasks. Optionally, the server can submit tasks (FILE). When submitting tasks, the -w/--bundlewait and -b/--bundlesize options are the same as for the submit workflow.

    With --max-retries greater than zero, the scheduler will check for a non-zero exit status for tasks and re-submit them if their previous number of attempts is less.

    Tasks are bundled and clients pull them in these bundles. However, by default the bundle size is one, meaning that at small scales there is greater concurrency.

  Arguments

FILE
    Path to input task file (default: <stdin>).

  Options

-H, --bind ADDR
    Bind address (default: localhost).

    When running locally, the default is recommended. To allow remote clients to connect over the network, bind the server to 0.0.0.0.

-p, --port NUM
    Port number (default: 50001).

    This is an arbitrary choice and simply must be an available port. The default option chosen here is typically available on most platforms and is not expected by any known major software.

-k, --auth KEY
    Cryptographic authentication key to connect with server (default: <not secure>).

    The default KEY used by the server and client is not secure and only a place holder. It is expected that the user choose a secure KEY. The cluster automatically generates a secure one-time KEY.

-b, --bundlesize SIZE
    Size of task bundle (default: 1).

    The default value allows for greater concurrency and responsiveness on small scales. This is used by the submit thread to accumulate bundles for either database commits and/or publishing to the queue. If a database is in use, the scheduler thread selects tasks from the database in batches of this size.

    Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

    See also --num-tasks and --bundlewait.

-w, --bundlewait SEC
    Seconds to wait before flushing tasks (default: 5).

    This is used by both the submit thread and forwarded to each client. The client collector thread that accumulates finished task bundles to return to the server will push out a bundle after this period of time regardless of whether it has reached the preferred bundle size.

    See also --bundlesize.

-r, --max-retries NUM
    Auto-retry failed tasks (default: 0).

    If a database is in use, then there is an opportunity to automatically retry failed tasks. A task is considered to have failed if it has a non-zero exit status. Setting this value greater than zero defines the number of attempts for the task. The original is not over-written, a new task is submitted and later scheduled.

    See also --eager.

--eager
    Schedule failed tasks before new tasks. If --max-retries is greater than one, this option defines the appetite for re-submitting failed tasks. By default, failed tasks will only be scheduled when there are no more remaining novel tasks.

--no-db
    Disable database (submit directly to clients).

    By default, a scheduler thread selects tasks from a database that were previously submitted. With --no-db enabled, there is no scheduler and instead the submit thread publishes bundles directly to the queue.

--initdb
    Auto-initialize database.

    If a database is configured for use with the workflow (e.g., Postgres), auto-initialize tables if they don't already exist. This is a short-hand for pre-creating tables with the hs initdb command. This happens by default with SQLite databases.

    Mutually exclusive to --no-db. See hs initdb command.

--no-confirm
    Disable client confirmation of task bundle received.

    To achieve even higher throughput at large scales, optionally disable confirmation payloads from clients. Consider using this option when also using --no-db.

--forever
    Schedule forever.

    Typically, the cluster will process some finite set of submitted tasks. When there are no more tasks left to schedule, the cluster will begin its shutdown procedure. With --forever enabled, the scheduler will continue to wait for new tasks indefinitely.

    Conflicts with --no-db and mutually exclusive to --restart.

--restart
    Start scheduling from last completed task.

    Instead of pulling a new list of tasks from some input FILE, with --restart enabled the cluster will restart scheduling tasks where it left off. Any task in the database that was previously scheduled but not completed will be reverted.

    For very large workflows, an effective strategy is to first use the submit workflow to populate the database, and then to use --restart so that if the cluster is interrupted, it can easily continue where it left off, halting if nothing to be done.

    Conflicts with --no-db and mutually exclusive to --forever.

--print
    Print failed task args to <stdout>.

    Mutually exclusive to -f/--failures.

-f, --failures PATH
    File path to write failed task args (default: <none>).

    The server acts like a sieve, reading task args from some input source. Tasks with a non-zero exit status can have their original command-line args printed to an output stream. With -f/--failures, specify a local file PATH.

    Mutually exclusive to --print.

CLIENT USAGE

hs client [-h]
    [-N NUM] [-t TEMPLATE] [-b SIZE] [-w SEC] [-H ADDR] [-p PORT] [-k KEY] [--capture | [-o PATH] [-e PATH]] [--no-confirm] [-d SEC] [-T SEC] [-W SEC] [-S SEC]

    Launch client directly, run tasks in parallel.

    The client connects to the server and pulls bundles of tasks off the shared queue. These tasks are run locally by some number of a parallel task executors.

    The environment for tasks are the same as for the client. Standard output and error for tasks are forwarded to that of the client, unless --capture is used, in which these are directed to individual files for each task.

  Options

-N, --num-tasks NUM
    Number of task executors (default: 1).

-t, --template CMD
    Command-line template pattern (default: "{}").

    This is expanded by the client just before execution. With the default "{}" the input command-line will be run verbatim. Specifying a template pattern allows for simple input arguments (e.g., file paths) to be transformed into some common form; such as -t './some_command.py {} >outputs/{/-}.out'.

    See section on templates.

-b, --bundlesize SIZE
    Size of task bundle (default: 1).

    Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

    See also --num-tasks and --bundlewait.

-w, --bundlewait SEC
    Seconds to wait before flushing tasks (default: 5).

    The client collector thread that accumulates finished task bundles to return to the server will push out a bundle after this period of time regardless of whether it has reached the preferred bundle size.

    See also --bundlesize.

-H, --host ADDR
    Hostname or IP address to connect with server (default: localhost).

-p, --port NUM
    Port number to connect with server (default: 50001).

-k, --auth KEY
    Cryptographic authentication key to connect with server (default: <not secure>).

    The default KEY used by the server and client is not secure and only a place holder. It is expected that the user choose a secure KEY. The cluster automatically generates a secure one-time KEY.

-d, --delay-start SEC
    Delay time in seconds before connecting to server (default: 0).

    At larger scales it can be advantageous to uniformly delay the client launch sequence. Hundreds or thousands of clients connecting to the server all at once is a challenge. Even if the server could handle the load, your task throughput would be unbalanced, coming in waves.

    Use --delay-start with a negative number to impose a uniform random delay up to the magnitude specified (e.g., --delay-start=-600 would delay the client up to ten minutes). This also has the effect of staggering the workload. If your tasks take on the order of 30 minutes and you have 1000 nodes, choose --delay-start=-1800.

--no-confirm
    Disable client confirmation of task bundle received.

    To achieve even higher throughput at large scales, optionally disable confirmation payloads from clients. Consider using this option when also using --no-db.

-o, --output PATH
    File path for task outputs (default: <stdout>).

-e, --errors PATH
    File path for task errors (default: <stderr>).

-c, --capture
    Capture individual task <stdout> and <stderr>.

    By default, the stdout and stderr streams of all tasks are fused with that of the client thread, and in turn the cluster. If tasks are producing output that needs to be isolated, the tasks need to manage their own output, you can specify a redirect as part of a --template, or use --capture to capture these as .out and .err files.

    These are stored local to the client under <prefix>/lib/task/<uuid>.[out,err]. Task outputs can be automatically retrieved via SFTP, see task usage.

    Mutually exclusive with both --output and --errors.

--timeout SEC
    Timeout in seconds for client. Automatically shutdown if no tasks received (default: never).

    This feature allows for gracefully scaling down a cluster when task throughput subsides.

--task-timeout SEC
    Task-level walltime limit (default: none).

    Executors will send a progression of SIGINT, SIGTERM, and SIGKILL. If the process still persists the executor itself will shutdown.

-S, --signalwait SEC
    Task-level signal escalation wait period in seconds (default: 10).

    When tasks fail to halt following an initial SIGINT, the program waits this interval in seconds before escalating to the next level of interrupt.

    See also --task-timeout.

SUBMIT USAGE

hs submit [-h] [ARGS... | -f FILE]
    [-b NUM] [-w SEC] [--template CMD] [-t TAG...] [--initdb]

    Submit one or more tasks.

    Submit one task as positional arguments. If a single positional argument refers to a non-executable file path, tasks will be read from the file, one per line (use -f/--task-file to be explicit).

    Tasks are accumulated and published in bundles to the database. The -b/--bundlesize and -w/--bundlewait options control the size of these bundles and how long to wait before flushing tasks regardless of how many have accumulated.

    Pre-format tasks at submit-time with template expansion using --template. Any tags specified with -t/--tag are applied to all tasks submitted.

    Use the special comment syntax, # HYPERSHELL: ..., to include tags inline. If the comment is alone on the line it will be applied to all tasks that follow.

  Arguments

ARGS...
    Command-line task arguments (for single task submission).

  Options

-f, --task-file FILE
    Input file containing one task per line.

--template CMD
    Command-line template pattern (default: "{}").

    This is expanded at submit-time before sending to the database. With the default "{}" the input command-line will be run verbatim. Specifying a template pattern allows for simple input arguments (e.g., file paths) to be transformed into some common form; such as --template './some_command.py {} >outputs/{/-}.out'.

    See section on templates.

-b, --bundlesize SIZE
    Size of task bundle (default: 1).

    The default value allows for greater concurrency and responsiveness on small scales. Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

    See also --bundlewait.

-w, --bundlewait SEC
    Seconds to wait before flushing tasks (default: 5).

    If this period of time expires since the previous bundle was pushed to the database, The current bundle will be pushed regardless of how many tasks have been accumulated.

    See also --bundlesize.

--initdb
    Auto-initialize database.

    If a database is configured for use with the workflow (e.g., Postgres), auto-initialize tables if they don't already exist. This is a short-hand for pre-creating tables with the hs initdb command. This happens by default with SQLite databases.

    See hs initdb command.

-t, --tag TAG...
    Assign one or more tags.

    Tags allow for user-defined tracking of information related to individual tasks or large groups of tasks. They are defined with both a key and value (e.g., --tag file:a). The default value for tags is blank. When searching with tags, not specifying a value will return any task with that key defined regardless of value (including blank).

INITDB USAGE

hs initdb [-h] [--truncate | --vacuum | --rotate | --backup PATH] [--yes]

Initialize database.

For SQLite this happens automatically. See also --initdb for the hs cluster command.

The available special actions are mutually exclusive. The --rotate operation migrates completed tasks to the next database partition, and applies a special purpose part:N tag to the new partition and remaining tasks.

  Actions

--vacuum
    Vacuum an existing database.

--backup PATH
    Vacuum into backup file (SQLite only).

--rotate
    Rotate completed tasks to new database (SQLite only).

-t, --truncate
    Truncate database (task metadata will be lost).

  Options

-y, --yes
    Auto-confirm action (default will prompt).

INFO USAGE

hs info [-h]
    ID [--stdout | --stderr | -x FIELD] [-f FORMAT]

    Get metadata and/or task outputs.

    Query for the full metadata on task by ID. Extract a specific field using -x/--extract.

    Alternatively, dump the captured standard output or errors from the finished task (if captured). These files are written on the client host and synced with SFTP if necessary.

  Arguments

ID  Unique task UUID.

  Options

-f, --format FORMAT
    Format task info ([normal], json, yaml).

--json
    Format metadata output as JSON.

--yaml
    Format metadata output as YAML.

-x, --extract FIELD
    Print this field only (e.g., -x submit_time).

--stdout
    Print <stdout> of task if captured, fetch from client if necessary.

--stderr
    Print <stderr> of task if captured, fetch from client if necessary.

WAIT USAGE

hs wait [-h]
    ID [-n SEC] [--info [-f FORMAT] | --status | --return]

    Wait for task to complete.

    Poll the database periodically for the completion status of the task. Block until completed.

    Optionally, print task metadata after completion.

  Arguments

ID  Unique task UUID.

  Options

-n, --interval SEC
    Time in seconds to wait between polling (default: 5).

-i, --info
    Print task info after completion.

    See task info command.

-f, --format FORMAT
    Format task info ([normal], json, yaml).

--json
    Format metadata output as JSON.

--yaml
    Format metadata output as YAML.

-s, --status
    Print task exit status only.

-r, --return
    Use exit status from task.

RUN USAGE

hs run [-h] [-n SEC] [-t TAG [TAG...]] -- ARGS...

Submit individual task and wait for completion.

A database must be configured. The task will not run until scheduled by the server process. The <stdout> and <stderr> are written locally as if run locally, requires the client to have --capture enabled.

See info and wait commands.

  Arguments

ARGS...
    Command-line arguments.

    The full command-line for some shell task. To use options, preface with leading --.

  Options

-n, --interval SEC
    Time in seconds to wait between polling (default: 5).

-t, --tag TAG...
    Assign one or more tags.

    Tags allow for user-defined tracking of information related to individual tasks or large groups of tasks. They are defined with both a key and value (e.g., --tag file:a). The default value for tags is blank. When searching with tags, not specifying a value will return any task with that key defined regardless of value (including blank).

LIST USAGE

hs list [-h]
    [FIELD [FIELD ...]] [-w COND [COND ...]] [-t TAG [TAG...]] [--order-by FIELD [--desc]] [--count | --limit NUM] [-f FORMAT | --json | --csv] [-d CHAR] [--failed | --succeeded | --completed | --remaining]

    hs list --fields

    hs list --tag-keys

    hs list --tag-values KEY

    Search tasks in the database.

    A database must be configured. Specifying FIELD names defines what is included in the output (by default all fields are included).

    This command maps directly to underlying SQL queries.

  Arguments

FIELD
    Select specific named fields to include in output. Default is to include all fields.

  Options

-w, --where COND...
    List of conditional statements to filter results (e.g., -w 'duration >= 600').

    Operators include ==, !=, >=, <=, >, <, ~. The ~ operator applies a regular expression.

-t, --with-tag TAG...
    Filter on one or more tags. (e.g., -t special or -t file:a).

    Leaving the value unspecified will return any task for which the key exists. Specifying a full key:value pair will match on both.

-s, --order-by FIELD [--desc]
    Order results by field. Optionally, in descending order.

-F, --failed
    Alias for -w 'exit_status != 0'.

-S, --succeeded
    Alias for -w 'exit_status == 0'.

-C, --completed
    Alias for -w 'exit_status != null'.

    For backwards compatibility, --finished is also valid.

-R, --remaining
    Alias for -w 'exit_status == null'.

-f, --format FORMAT
    Specify output format (either normal, plain, table, csv, json).

    Default is normal for whole-task output. If any FIELD names are given, output is formatted in simple plain text; use csv for compliant output. The pretty-printed table formatting is good for presentation on wide screens.

    See --csv, --json, and --delimiter.

--csv
    Format output as CSV. (Shorthand for --format=csv).

--json
    Format output as JSON. (Shorthand for --format=json).

-d, --delimiter CHAR
    Field seperator for plain/csv formats.

-l, --limit NUM
    Limit the number of results.

-c, --count
    Show count of results.

-i, --ignore-partitions
    Suppress auto-union feature (SQLite only).

    When using the sqlite provider, all databases matching the numbering pattern applied by hs initdb --rotate will automatically be attached as partitions and used within a temporary view, making full task history searchable.

UPDATE USAGE

hs update [-h] ID FIELD VALUE

hs update
    [ARG [ARG ...]] [--cancel | --revert | --delete | --remove-tag KEY [KEY...]] [-w COND [COND ...]] [-t TAG [TAG...]] [--order-by FIELD [--desc] --limit NUM] [--failed | --succeeded | --completed | --remaining] [--no-confirm]

    Update tasks in the database.

    Include any number of FIELD=VALUE or tag KEY:VALUE positional arguments.

    The -w/--where and -t/--with-tag operate just as in the search command.

    Using --cancel sets schedule_time to now and exit_status to -1.

    Using --revert reverts everything as if the task was new again.

    Using --delete drops the row from the database entirely.

    The legacy interface for updating a single task with the ID, FIELD, and VALUE as positional arguments remains valid.

  Arguments

ARGS...
    Assignment pairs for update.

  Options

--cancel
    Cancel specified tasks.

    Cancelling a task means it will no longer be scheduled. This is done by setting the schedule_time to now and the exit_status to -1. A task cannot be cancelled after it is sent to remote clients.

--revert
    Revert specified tasks.

    A reverted task retains its ID and submit info along with any tags. It will be as if it were new and never scheduled.

--delete
    Delete specified tasks.

    Deleting a task fully drops the record from the database. All task information will be lost and not recoverable.

--remove-tag TAG...
    Strip the specified tag from task records.

-w, --where COND...
    List of conditional statements to filter results (e.g., -w 'duration >= 600').

    Operators include ==, !=, >=, <=, >, <, ~. The ~ operator applies a regular expression.

-t, --with-tag TAG...
    Filter on one or more tags. (e.g., -t special or -t file:a).

    Leaving the value unspecified will return any task for which the key exists. Specifying a full key:value pair will match on both.

-s, --order-by FIELD [--desc]
    Order results by field. Optionally, in descending order.

    When used in an update command, a --limit is required. For example, to delete the most recently submitted task, --order-by submit_time --desc --limit 1.

-l, --limit NUM
    Limit the number of results.

-F, --failed
    Alias for -w 'exit_status != 0'.

-S, --succeeded
    Alias for -w 'exit_status == 0'.

-C, --completed
    Alias for -w 'exit_status != null'.

-R, --remaining
    Alias for -w 'exit_status == null'.

-f, --no-confirm
    Do not ask for confirmation.

    The program first checks the number of affected tasks. The user must confirm the update interactively unless provided with --no-confirm.

  Legacy Mode

In previous releases of the software the update command had the following signature. Executing the command with these three positional arguments is still valid.

ID  Unique UUID.

FIELD
    Task field name (e.g., "args").

VALUE
    New value.

    Use key:value notation for tags. Updating tag will add or update any pre-existing tag with that key.

CONFIG GET USAGE

hs config get [-h]
    [-x] [-r] SECTION[...].VAR [--system | --user | --local | --default]

    Get configuration option.

    If source is not specified; e.g., --local, the output is the merged configuration from all sources. Use the config which command to see where a specific option originates from.

    Output is pretty-printed using the configured console theme (default: monokai). Colorization is disabled for non-TTY invocations.

  Arguments

SECTION[...].VAR
    Path to variable (default is entire configuration).

  Options

--system
    Load from system configuration.

--user
    Load from user configuration.

--local
    Load from local configuration.

--default
    Load from default configuration.

-x, --expand
    Expand variable.

    If the special _env or _eval variant of the option is present in the configuration, it will expand the environment variable or shell command, respectively.

-r, --raw
    Disable formatting on single value output.

CONFIG SET USAGE

hs config set [-h]
    SECTION[...].VAR VALUE [--system | --user | --local]

    Set configuration option.

  Arguments

SECTION[...].VAR
    Path to variable.

VALUE
    Value to be set.

  Options

--system
    Apply to system configuration.

--user
    Apply to user configuration. (default)

--local
    Apply to local configuration.

CONFIG EDIT USAGE

hs config edit [-h]
    [--system | --user | --local]

    Edit configuration with default editor.

    The EDITOR/VISUAL environment variable must be set.

  Options

--system
    Edit system configuration.

--user
    Edit user configuration (default).

--local
    Edit local configuration.

CONFIG WHICH USAGE

hs config which [-h]
    SECTION[...].VAR [--site]

    Show origin of configuration option.

  Arguments

SECTION[...].VAR
    Path to variable.

  Options

--site
    Output originating site name only.

TEMPLATES

Incoming command-line arguments are expanded using a template pattern. Workloads often have a common form and only a small part of the shell command need be different.

Braces, {}, are used in all cases. Empty braces will substitute the full incoming argument line. Use one of the below patterns as a shorthand notation for many common scenarios.

The -t/--template argument is used by the client command to expand templates just prior to execution. The cluster command simply forwards this argument to all clients.

In some situations it may be useful to expand a template with the submit command. These are expanded prior to scheduling as the actual args for the task.

  Filepath Operations

Shell commands often operate on filepaths. In such cases, it may be useful to manipulate these paths. Instead of using a shell interpolation (see below), use one of the available shorthand notations listed here.

{.} Expand to immediate parent directory of given file. E.g., /some/path/to/file.h5 translates to /some/path/to.

{..}
    Expand to second parent directory of given file. E.g., /some/path/to/file.h5 translates to /some/path.

{/} The basename of the given file. E.g., /some/path/to/file.h5 translates to file.h5.

{/-}
    The basename of the given file without its file type extension. E.g., /some/path/to/file.h5 translates to file.

{-} The full path of the given file without the extension. This is useful for targeting adjacent files with a different extension. E.g., /some/path/to/file.h5 translates to /some/path/to/file.

{+} The file type extension for the given file. E.g., /some/path/to/file.h5 translates to .h5.

{++}
    The file type extension for the given file without the leading dot. E.g., /some/path/to/file.h5 translates to h5.

  Argument Slicing

Command-line inputs are understood as individual arguments delimited by whitespace. Slice into the argument vector using the {[]} notation. Arguments follow zero-based indexing. Negative index values are counting backwards from the end.

Select with a singular value.

{[0]}
    The first argument.

{[1]}
    The second argument.

{[-1]}
    The last argument.

    Select a range with a start and stop value, non-inclusive of the stop value. Including a leading or trailing colon implies the default value (inclusive).

{[1:3]}
    The second and third argument.

{[:4]}
    The first four arguments.

{[-2:]}
    The last two arguments.

    Include a third value as a step (or sometimes referred to as stride). Leaving out the start and step value implies starting from the first element (inclusive).

{[::2]}
    Every second argument starting from the first.

{[1::2]}
    Every odd argument.

  Shell Expansion

General purpose shell commands can be expanded with the {% %} notation. The incoming command-line args can be substituted with an @.

{% basename @ %}
    Equivalent to {/}.

{% mktemp -d %}
    Create temporary directory and insert its path.

  Lambda Expressions

Arbitrary Python expressions can be expanded with the {= =} notation. The input argument can be used within the expression with the variable x.

Exposed standard library modules include os, os.path as path, math and datetime as dt.

Incoming arguments are intelligently coerced into the expected type. E.g., 2 will be an integer, 4.67 a float, null and none will be a Python None, and true/false will be the appropriate boolean value.

{= x * math.pi =}
    Multiply the incoming argument (expected to be a float) by Pi.

{= dt.datetime.fromtimestamp(x) =}
    Convert incoming POSIX timestamp to ISO format.

CONFIGURATION

Most of the choices the program makes about timing, task bundling, coordination, logging, and such are configurable by the user. This configuration is loaded when the program starts and is constructed from several sources including an ordered merger of files, environment variables, and command-line options.

In order of precedence (lowest to highest), three files are loaded:
Level | Linux / POSIX | macOS / Darwin
System | /etc/hypershell.toml | /Library/Preferences/HyperShell/config.toml
User | ~/.hypershell/config.toml | ~/Library/Preferences/HyperShell/config.toml
Local | ./.hypershell/config.toml | ./.hypershell/config.toml

The TOML format is modern and minimal.

Every configurable option can be set in one of these files. Further, every option can also be set by an environment variable, where the name aligns to the path to that option, delimited by underscores.

For example, set the logging level at the user level with a command:

$ hs config set logging.level info --user

The file should now look something like this:

# File automatically created on 2022-07-02 11:57:29.332993
# Settings here are merged automatically with defaults and environment variables
[logging]
level = "info"

Alternatively, you can set an environment variable and the runtime configuration would be equivalent:

$ export HYPERSHELL_LOGGING_LEVEL=INFO

Finally, any option defined within a configuration file that ends with _env or _eval is automatically expanded by the given environment variable or shell expression, respectively. This is useful as both a dynamic feature but also as a means to obfuscate sensitive information, such as database connection details.

# File automatically created on 2022-07-02 11:57:29.332993
# Settings here are merged automatically with defaults and environment variables
[logging]
level = "info"
[database]
provider = "postgres"
database = "hypershell"
host = "my.instance.university.edu"
user = "me"
password_eval = "pass hypershell/database/password"  # Decrypt using GNU Pass

  Parameter Reference

[logging]
    Logging configuration. See also logging section.

.level
    One of TRACE, DEBUG, INFO, WARNING, ERROR, or CRITICAL (default: WARNING)

    INFO level messages are reserved for clients when tasks begin running. There are numerous WARNING events (e.g., non-zero exit status of a task). DEBUG level messages signal component thread start/stop and individual task level behavior. TRACE contains detailed information on all other behavior, particular iterative messages while components are waiting for something.

    ERROR messages track when things fail but the application can continue; e.g., when command template expansion fails on an individual task.

    CRITICAL messages are emitted when the application will halt or crash. Some of these are expected (such as incorrect command-line arguments) but in the event of an uncaught exception within the application a full traceback is written to a file and logged.

.datefmt
    Date/time format, standard codes apply (default: '%Y-%m-%d %H:%M:%S'`)

.format
    Log message format.

    Default set by the "default" logging.style. See the available attributes defined by the underlying Python logging interface.

    Additional attributes provided beyond the standard include app_id, hostname, hostname_short, relative_name, time formats in elapsed, elapsed_ms, elapsed_delta, and elapsed_hms, as well as all ANSI colors and formats as ansi_x where x is one of reset, bold, faint, italic, underline, black, red, green, yellow, blue, magenta, cyan, white, and ansi_level contains the standard color for the current message severity level.

.style
    Presets for logging.format which can be difficult to define correctly. Options are default, detailed, detailed-compact, and system.

[database]
    Database configuration and connection details. See also database section.

.provider
    Database provider (default: 'sqlite'). Supported alternatives include 'postgres' (or compatible). Support for other providers may be considered in the future.

.file
    Only applicable for SQLite provider. SQLite does not understand any other connection detail.

.database
    Name for database. Not applicable for SQLite.

.schema
    Not applicable for all RDMS providers. For Postgres the default schema is public. Specifying the schema may be useful for having multiple instances within the same database.

.host
    Hostname or address of database server (default: localhost).

.port
    Port number to connect with database server. The default value depends on the provider, e.g., 5432 for Postgres.

.user
    Username for databaser server account. If provided a password must also be provided. Default is the local account.

.password
    Password for database server account. If provided a user must also be provided. Default is the local account.

    See also note on _env and _eval.

.echo
    Special parameter enables verbose logging of all database transactions.

[connection_args]
    Specify additional connection details for the underlying SQL dialect provider, e.g., sqlite3 or psycopg2.

[pragmas]
    Specify one or more pragmas to apply to database connections (SQLite only). For example pragmas = { journal_mode = "wal" } to enable WAL-mode.

*   Any additional arguments are forwarded to the provider, e.g., encoding = 'utf-8'.

[server]
    Section for server workflow parameters.

.bind
    Bind address (default: localhost).

    When running locally, the default is recommended. To allow remote clients to connect over the network, bind the server to 0.0.0.0.

.port
    Port number (default: 50001).

    This is an arbitrary choice and simply must be an available port. The default option chosen here is typically available on most platforms and is not expected by any known major software.

.auth
    Cryptographic authentication key to connect with server (default: <not secure>).

    The default KEY used by the server and client is not secure and only a place holder. It is expected that the user choose a secure KEY. The cluster automatically generates a secure one-time KEY.

.queuesize
    Maximum number of task bundles on the shared queue (default: 1).

    This blocks the next bundle from being published by the scheduler until a client has taken the current prepared bundle. On smaller scales this is probably best and is only of modest performance impact, limiting the scheduler from getting so far ahead of the currently running tasks.

    On large scale workflows with many clients (e.g., 100) it may be advantageous to allow the scheduler to work ahead in selecting new tasks.

.bundlesize
    Size of task bundle (default: 1).

    The default value allows for greater concurrency and responsiveness on small scales. This is used by the submit thread to accumulate bundles for either database commits and/or publishing to the queue. If a database is in use, the scheduler thread selects tasks from the database in batches of this size.

    Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

    See also -b/--bundlesize command-line option.

.attempts
    Attempts for auto-retry on failed tasks (default: 1).

    If a database is in use, then there is an opportunity to automatically retry failed tasks. A task is considered to have failed if it has a non-zero exit status. The original is not over-written, a new task is submitted and later scheduled.

    Counterpart to the -r/--max-retries command-line option. Setting --max-retries 1 is equivalent to setting .attempts to 2.

    See also .eager.

.eager
    Schedule failed tasks before new tasks (default: false).

    If .attempts is greater than one, this option defines the appetite for re-submitting failed tasks. By default, failed tasks will only be scheduled when there are no more remaining novel tasks.

.wait
    Polling interval in seconds for database queries during scheduling (default: 5). This waiting only occurs when no tasks are returned by the query.

.evict
    Eviction period in seconds for clients (default: 600).

    If a client fails to register a heartbeat after this period of time it is considered defunct and is evicted. When there are no more tasks to schedule the server sends a disconnect request to all registered clients, and waits until a confirmation is returned for each. If a client is defunct, this will hang the shutdown process.

[client]
    Section for client workflow parameters.

.bundlesize
    Size of task bundle (default: 1).

    The default value allows for greater concurrency and responsiveness on small scales.

    Using larger bundles is a good idea for larger distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client. It is also a good idea to coordinate bundle size between the client and server so that the client returns the same sized bundles that it receives.

    See also -b/--bundlesize command-line option.

.bundlewait
    Seconds to wait before flushing task bundle (default: 5).

    If this period of time expires since the previous bundle was returned to the server, the current group of finished tasks will be pushed regardless of bundlesize.

    For larger distributed workflows it is a good idea to make this waiting period sufficiently long so that most bundles are returned whole.

    See also -w/--bundlewait command-line option.

.heartrate
    Interval in seconds between heartbeats sent to server (default 10).

    Even on the largest scales the default interval should be fine.

.timeout
    Timeout in seconds for client. Automatically shutdown if no tasks received (default: never).

    This feature allows for gracefully scaling down a cluster when task throughput subsides.

[submit]
    Section for submit workflow parameters.

.bundlesize
    Size of task bundle (default: 1).

    The default value allows for greater concurrency and responsiveness on small scales. Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

    See also -b/--bundlesize command-line option.

.bundlewait
    Seconds to wait before flushing tasks (default: 5).

    If this period of time expires since the previous bundle was pushed to the database, the current bundle will be pushed regardless of how many tasks have been accumulated.

    See also -w/--bundlewait command-line option.

[task]
    Section for task runtime settings.

.cwd
    Explicitly set the working directory for all tasks.

.timeout
    Task-level walltime limit (default: none).

    Executors will send a progression of SIGINT, SIGTERM, and SIGKILL. If the process still persists the executor itself will shutdown.

.signalwait
    Wait period in seconds between signal escalation on task cancellation (default: 10).

    See also -S, --signalwait command-line option.

[ssh]
    SSH configuration section.

.args
    SSH connection arguments; e.g., -i ~/.ssh/some.key. It is preferable to configure SSH directly however, in ~/.ssh/config.

[nodelist]
    This can be a single list of hostnames or a section when multiple named lists. Reference named groups from the command-line with --ssh-group.

    Such as,

    .mycluster = ['mycluster-01', 'mycluster-02', 'mycluster-03']

[autoscale]
    Define an autoscaling policy and parameters.

.policy
    Either fixed or dynamic.

    A fixed policy will seek to maintain a definite size and allows for recovery in the event that clients halt for some reason (e.g., due to expected faults or timeouts).

    A dynamic policy maintains a minimum size and grows up to some maximum size depending on the observed task pressure given the specified scaling factor.

    See also .factor, .period, .size.init, .size.min, and .size.max.

.factor
    Scaling factor (default: 1).

    A dimensionless quantity used by the dynamic policy. This value expresses some multiple of the average task duration in seconds.

    The autoscaler periodically checks toc / (factor x avg_duration), where toc is the estimated time of completion for all remaining tasks given current throughput of active clients. This ratio is referred to as task pressure, and if it exceeds 1, the pressure is considered high and we will add another client if we are not already at the maximum size of the cluster.

    For example, if the average task length is 30 minutes, and we set factor = 2, then if the estimated time of completion of remaining tasks given currently connected executors exceeds 1 hour, we will scale up by one unit.

    See also .period.

.period
    Scaling period in seconds (default: 60).

    The autoscaler waits for this period of time in between checks and scaling events. A shorter period makes the scaling behavior more responsive but can effect database performance if checks happen too rapidly.

[size]

.init
    Initial size of cluster (default: 1).

    When the the cluster starts, this number of clients will be launched. For a fixed policy cluster, this should be given with a .min size, and likely the same value.

.min
    Minimum size of cluster (default: 0).

    Regardless of autoscaling policy, if the number of launched clients drops below this value we will scale up by one. Allowing min = 0 is an important feature for efficient use of computing resources in the absence of tasks.

.max
    Maximum size of cluster (default: 2).

    For a dynamic autoscaling policy, this sets an upper limit on the number of launched clients. When this number is reached, scaling stops regardless of task pressure.

[console]
    Rich text display and output parameters.

.theme
    Color scheme to use by default in output (such as with task info and task search).

    This option is passed to the rich library.

[export]
    Any variable defined here is injected as an environment variable for tasks.

    Example,

foo = 1
    The environment variable FOO=1 would be defined for all tasks.

DATABASE

The program can operate entirely within memory and does not require a database. This is desirable for ad-hoc, short-lived, or in extreme high-throughput scenarios. However, some capabilities are only possible when a database is active (such as server restart, task retry, task history, etc).

Connection details must to be specified by your configuration. This can be entirely through environment variables or within files. SQLite is automatically initialized upon opening the connection but Postgres must be explicitly initialized via hs initdb or at launch with the --initdb command-line option.

SQLite only needs the local file path.

[database]
file = "/var/lib/hypershell/main.db"

Or via a single environment variable, HYPERSHELL_DATABASE_FILE=/tmp/pipeline/task.db.

The default provider is SQLite; Postgres must be enabled. A local database with local account authentication simply needs to know the database name.

[database]
provider = "postgres"
database = "hypershell"

The port is assumed to be 5432 but can be specified. If a user is given then so too must a password. The default public schema is assumed but can also be specified. Anything else is forwarded to the database as a connection parameter; e.g., encoding = "utf-8".

Any parameter ending in the special suffixes _env or _eval are interpolated into the configuration. E.g., password_env treats its value as the name of an environment variable and password_eval executes its value as a shell command, both exposing password as the expanded version.

The special .echo parameter can be set to true to enable verbose logging of all database transactions. These will be emitted as INFO level messages.

ENVIRONMENT VARIABLES

As stated for configuration, any environment variable prefixed as HYPERSHELL_ where the name aligns to the path to some option, delimited by underscores, will set that option. Example, HYPERSHELL_CLIENT_TIMEOUT maps to the corresponding configuration option.

The following environment variables must be specified as such and cannot be configurable within files.

HYPERSHELL_SITE
    The default site path for outputs is either the user or system site based on whether the program was run as root/admin. The local path is always the current working directory for the program. If this environment variable is set then it takes the place of the local site and used as the default.

HYPERSHELL_CONFIG_FILE
    Under normal operations the program searches all three system, user, and local sites to load the full configuration in addition to all prefixed environment variables. This can actually cause issues for scenarios with many instances of the program running on the same file system, such as an HPC cluster. In order to protect against unintended crashes from incidental configuration changes, defining this environment variable specifies the one and only path to a configuration file and all others will be ignored. Setting this to empty results in no files being loaded (an environment only runtime).

    The following environment variable may be set to harden the installation against accidentally polluting the package resolution of HyperShell itself from user software.

HYPERSHELL_PYTHONPATH
    As a Python application HyperShell may be exposed to PYTHONPATH. This can happen in instances where the user tasks themselves are Python based. To avoid possible collisions we can guard HyperShell by explicitly setting the path list (sys.path). This environment variable may be set in a similar fashion as the standard PYTHONPATH. Or it may instead contain a file path, in which case the content of the file should have one path per line.

    A few common environment variables are defined for every task.

TASK_ID
    Universal identifier (UUID) for the current task.

TASK_ARGS
    Original input command-line argument line. Equivalent to {}, see templates section.

TASK_SUBMIT_ID
    Universal identifier (UUID) for submitting application instance.

TASK_SUBMIT_HOST
    Hostname of submitting application instance.

TASK_SUBMIT_TIME
    Timestamp task was submitted.

TASK_SERVER_ID
    Universal identifier (UUID) for server application instance.

TASK_SERVER_HOST
    Hostname of server application instance.

TASK_SCHEDULE_TIME
    Timestamp task was scheduled by server.

TASK_CLIENT_ID
    Universal identifier (UUID) for client application instance.

TASK_CLIENT_HOST
    Hostname of client application instance.

TASK_COMMAND
    Final command line for task.

TASK_ATTEMPT
    Integer number of attempts for current task (starts at 1).

TASK_PREVIOUS_ID
    Universal identifier (UUID) for previous attempt (if any).

TASK_CWD
    Current working directory for the current task.

TASK_START_TIME
    Timestamp task began execution.

TASK_WAITED
    Time in seconds between task submit time and schedule time.

TASK_OUTPATH
    Absolute file path where standard output is directed (if defined).

TASK_ERRPATH
    Absolute file path where standard error is directed (if defined).

    Further, any environment variable starting with HYPERSHELL_EXPORT_ will be injected into the task environment sans prefix; e.g., HYPERSHELL_EXPORT_FOO would define FOO in the task environment. You can also define such variables in the export section of your configuration file(s); e.g.,

    # File automatically created on 2022-07-02 11:57:29.332993
    # Settings here are merged automatically with defaults and environment variables
    [logging]
    level = "info"
    # Options defined as a list will be joined with a ":" on BSD/Linux or ";" on Windows
    # Environment variables will be in all-caps (e.g., FOO and PATH).
    [export]
    foo = "value"
    path = ["/some/bin", "/some/other/bin"]

    We also respect setting the following environment variables to force disable/enable the use of colors in all console output.

NO_COLOR
    If this variable is set to anything but a blank string, all colors are disabled. See no-color.org for details.

FORCE_COLOR
    If this variable is set to anything but a blank string, colors will be enabled regardless of whether stdout or stderr are a TTY.

SIGNALS

HyperShell traps the following UNIX signals (does not apply on Microsoft Windows).

SIGUSR1
    Sending the SIGUSR1 signal will trigger the schedulers to halt and begin shutdown procedures. On the client side, this means that all current tasks (and any in the local queue) will be allowed to complete, but the system will drain and shutdown at the completion of these tasks.

SIGUSR2
    Sending the SIGUSR2 signal implies the same, but on the client side will set a flag to send local interrupts to tasks to come down faster. As described in the previous release with regard to the task.timeout feature, we send SIGINT, SIGTERM, and SIGKILL in an escalating fashion to halt running tasks.

EXIT STATUS

0   Success

1   Usage statement printed

2   Bad argument

3   Bad configuration

4   Signal interrupt (keyboard interrupt)

5   Generic runtime error (non-specific)

6   Internal error (unexpected)

SEE ALSO

ssh(1), mpirun(1)

AUTHOR
Geoffrey Lentner <glentner@purdue.edu>.

COPYRIGHT
2019-2025 Geoffrey Lentner