Skip to content

API Reference

Core Modules

GearContext

Context for gear runtime.

  • Provides methods for interacting with gear runtime objects such as:
    • config.json (Runtime configuration)
    • manifest.json (Defined configuration and gear metadata)
    • .metadata.json (Hierarchy metadata to be updated at end of gear run)
  • Configures gear logging

Parameters:

Name Type Description Default
gear_path Optional[str]

Path to gear directory, default behavior will use /flywheel/v0.

'/flywheel/v0'
manifest_path Optional[str]

Path to the gear's manifest.json file, defaults to self.path/'manifest.json'.

'/flywheel/v0/manifest.json'
config_path Optional[str]

Path to the gear's config.json file, defaults to self._path/'config.json'.

'/flywheel/v0/config.json'
log_metadata bool

Whether to log .metadata.json to log upon write, defaults to True.

True
clean_on_error bool

Whether to clean output directory upon exit with error, defaults to False.

False
Source code in fw_gear/context.py
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
class GearContext:
    """Context for gear runtime.

    * Provides methods for interacting with gear runtime objects such as:
        * `config.json` (Runtime configuration)
        * `manifest.json` (Defined configuration and gear metadata)
        * `.metadata.json` (Hierarchy metadata to be updated at end of gear run)
    * Configures gear logging


    Args:
        gear_path: Path to gear directory, default behavior will use `/flywheel/v0`.
        manifest_path: Path to the gear's manifest.json file,
            defaults to ``self.path/'manifest.json'``.
        config_path: Path to the gear's config.json file, defaults to
            ``self._path/'config.json'``.
        log_metadata: Whether to log .metadata.json to log upon write,
            defaults to True.
        clean_on_error: Whether to clean output directory upon exit
            with error, defaults to False.
    """

    def __init__(
        self,
        gear_path: t.Optional[str] = "/flywheel/v0",
        manifest_path: t.Optional[str] = "/flywheel/v0/manifest.json",
        config_path: t.Optional[str] = "/flywheel/v0/config.json",
        log_metadata: bool = True,
        fail_on_validation: bool = True,
        clean_on_error: bool = False,
    ):  # noqa: D107
        self._path = Path(gear_path or Path.cwd()).resolve()
        self.config = Config(None, self._load_json(config_path))
        self._client: t.Optional["flywheel.Client"] = self.get_client()
        self.config._client = self.config._client or self._client
        self._out_dir = None
        self._work_dir = None
        self._log_metadata = log_metadata
        self.fail_on_validation = fail_on_validation
        self._clean_on_error = clean_on_error
        self.manifest = None

        # Having a manifest.json is not required per gear specifications
        try:
            self.manifest = Manifest(manifest_path)
        except FileNotFoundError:
            log.debug(
                f"Manifest file not found at {manifest_path}. Continuing without manifest."
            )

        # Needs to be last
        self.metadata = Metadata(self)

    def init_logging(self, default_config_name=None, update_config=None):
        """Configures logging via `fw_gear.logging.configure_logging`.

        If no `default_config_name` is provided, will get `debug` from the
        configuration options. If `debug` is False or not defined in the gear
        configuration options, `default_config_name` will be set to info.

        If `update_config` is not provided, manifest['custom']['log_config'] will be
        used (if defined in the manifest).

        Args:
            default_config_name (str, optional): A string, 'info' or 'debug', indicating
                the default template to use. (Defaults to 'info').
            update_config (dict, optional): A dictionary containing the keys, sub-keys,
                and values of the templates to update. (Defaults to
                None).
        """
        if not default_config_name:
            if self.config.opts.get("debug"):
                default_config_name = "debug"
            else:
                default_config_name = "info"
        if not update_config:
            # Only try to get log_config from manifest if manifest exists
            if self.manifest is not None:
                if isinstance(self.manifest.get_value("custom"), dict):
                    update_config = self.manifest.get_value("custom").get(
                        "log_config", None
                    )

        configure_logging(
            default_config_name=default_config_name,
            update_config=update_config,
        )
        return default_config_name, update_config

    @property
    def work_dir(self) -> Path:
        """Get the absolute path to a work directory.

        Returns:
            pathlib.Path: The absolute path to work.
        """
        if self._work_dir is None:
            self._work_dir = self._path / "work"
            if not self._work_dir.exists():
                self._work_dir.mkdir(parents=True)
        return self._work_dir

    @property
    def output_dir(self) -> Path:
        """Get the absolute path to the output directory.

        Returns:
            pathlib.Path: The absolute path to outputs.
        """
        if self._out_dir is None:
            self._out_dir = self._path / "output"
            if not self._out_dir.exists():
                self._out_dir.mkdir(parents=True)
        return self._out_dir

    @property
    def client(self) -> t.Optional["flywheel.Client"]:  # noqa: D402
        """Wrapper around self.get_client()."""
        if not self._client:
            self._client = self.get_client()
        return self._client

    def is_fw_context(self) -> bool:
        """Return True if the gear is running in a Flywheel gear context, False otherwise."""

        gear_path = Path("/flywheel/v0")

        return gear_path.exists() and Path(gear_path / "config.json").exists()

    def get_client(self) -> t.Optional["flywheel.Client"]:
        """Get the SDK client, if an api key input exists or CLI client exists.

        Returns:
          flywheel.Client: The Flywheel SDK client.


        Raises:
            GearContextError: If the Flywheel SDK is not installed, no API key is found in `config.json`, or initializing the Flywheel Client fails.
        """
        if not HAVE_FLYWHEEL:
            warnings.warn(
                "Please install the `sdk` extra or install `flywheel-sdk` within"
                " your gear in order to use the SDK client."
            )
            return None
        api_key = None
        client = None

        for inp in self.config.inputs.values():
            if inp["base"] == "api-key" and inp["key"]:
                api_key = inp["key"]
                break
        if api_key:
            try:
                client = flywheel.Client(api_key)

            except Exception as exc:  # pylint: disable=broad-except
                log.exception(
                    "An exception was raised when initializing the Flywheel Client with the provided API key."
                )
                raise GearContextError(
                    "Failed to initialize the Flywheel Client with the provided API key."
                ) from exc
        return client

    def log_config(self):
        """Print the configuration and input files to the logger."""
        # Log destination
        log.info(
            "Destination is %s=%s",
            self.config.destination.get("type"),
            self.config.destination.get("id"),
        )

        # Log file inputs
        for inp_name, inp in self.config.inputs.items():
            if inp["base"] != "file":
                continue

            container_type = inp.get("hierarchy", {}).get("type")
            container_id = inp.get("hierarchy", {}).get("id")
            file_name = inp.get("location", {}).get("name")

            log.info(
                'Input file "%s" is %s from %s=%s',
                inp_name,
                file_name,
                container_type,
                container_id,
            )

        # Log configuration values
        for key, value in self.config.opts.items():
            log.info('Config "%s=%s"', key, value)

    def open_output(self, name: str, mode: str = "w", **kwargs) -> t.IO:
        """Open the named output file.

        Args:
            name: The name of the output.
            mode: The open mode (Default value = 'w').
            **kwargs (dict): Keyword arguments for `open`.

        Returns:
            File: The file object.
        """
        path = self.output_dir / name
        return path.open(mode, **kwargs)

    def get_context_value(self, name: str) -> t.Optional[dict]:
        """Get the context input for name.

        Args:
            name: The name of the input.

        Returns:
            dict: The input context value, or None if not found.
        """
        inp = self.config.get_input(name)
        if not inp:
            return None
        if inp["base"] != "context":
            raise ValueError(f"The specified input {name} is not a context input")
        return inp.get("value")

    def __enter__(self):  # noqa: D105
        return self

    def __exit__(self, exc_type, exc_value, _traceback):
        if exc_type is None or (
            issubclass(exc_type, SystemExit) and exc_value.code == 0
        ):
            self.metadata.update_zip_member_count(
                self.output_dir,
                container_type=self.config.destination.get("type"),
            )
            self.metadata.write(
                self.output_dir, self.fail_on_validation, self._log_metadata
            )
        elif self._clean_on_error:  # noqa: PLR5501
            log.info("Cleaning output folder.")
            self._clean_output()
        else:
            log.debug(f"Skipping cleanup of {self.output_dir}")

        if self._client:
            sdk_usage = getattr(
                self._client.api_client.rest_client, "request_counts", None
            )
            if not sdk_usage:
                log.debug("SDK profiling not available in this version of flywheel-sdk")
            else:
                log.debug(f"SDK usage:\n{pformat(sdk_usage, indent=2)}")

    def _clean_output(self):
        for path in Path(self.output_dir).glob("**/*"):
            if path.is_file():
                path.unlink()
            elif path.is_dir():
                rmtree(path)

    @staticmethod
    def _load_json(filepath):
        """Return dictionary for input json file.

        Args:
          filepath (str): Path to a JSON file.

        Raises:
            RuntimeError: If filepath cannot be parsed as JSON.

        Returns:
            (dict): The dictionary representation of the JSON file at filepath.
        """
        json_dict = dict()
        if os.path.isfile(filepath):
            try:
                with open(filepath, "r") as f:
                    json_dict = json.load(f)
            except json.JSONDecodeError:
                raise RuntimeError(f"Cannot parse {filepath} as JSON.")

        return json_dict

client property

Wrapper around self.get_client().

output_dir property

Get the absolute path to the output directory.

Returns:

Type Description
Path

pathlib.Path: The absolute path to outputs.

work_dir property

Get the absolute path to a work directory.

Returns:

Type Description
Path

pathlib.Path: The absolute path to work.

get_client()

Get the SDK client, if an api key input exists or CLI client exists.

Returns:

Type Description
Optional[Client]

flywheel.Client: The Flywheel SDK client.

Raises:

Type Description
GearContextError

If the Flywheel SDK is not installed, no API key is found in config.json, or initializing the Flywheel Client fails.

Source code in fw_gear/context.py
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
def get_client(self) -> t.Optional["flywheel.Client"]:
    """Get the SDK client, if an api key input exists or CLI client exists.

    Returns:
      flywheel.Client: The Flywheel SDK client.


    Raises:
        GearContextError: If the Flywheel SDK is not installed, no API key is found in `config.json`, or initializing the Flywheel Client fails.
    """
    if not HAVE_FLYWHEEL:
        warnings.warn(
            "Please install the `sdk` extra or install `flywheel-sdk` within"
            " your gear in order to use the SDK client."
        )
        return None
    api_key = None
    client = None

    for inp in self.config.inputs.values():
        if inp["base"] == "api-key" and inp["key"]:
            api_key = inp["key"]
            break
    if api_key:
        try:
            client = flywheel.Client(api_key)

        except Exception as exc:  # pylint: disable=broad-except
            log.exception(
                "An exception was raised when initializing the Flywheel Client with the provided API key."
            )
            raise GearContextError(
                "Failed to initialize the Flywheel Client with the provided API key."
            ) from exc
    return client

get_context_value(name)

Get the context input for name.

Parameters:

Name Type Description Default
name str

The name of the input.

required

Returns:

Name Type Description
dict Optional[dict]

The input context value, or None if not found.

Source code in fw_gear/context.py
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
def get_context_value(self, name: str) -> t.Optional[dict]:
    """Get the context input for name.

    Args:
        name: The name of the input.

    Returns:
        dict: The input context value, or None if not found.
    """
    inp = self.config.get_input(name)
    if not inp:
        return None
    if inp["base"] != "context":
        raise ValueError(f"The specified input {name} is not a context input")
    return inp.get("value")

init_logging(default_config_name=None, update_config=None)

Configures logging via fw_gear.logging.configure_logging.

If no default_config_name is provided, will get debug from the configuration options. If debug is False or not defined in the gear configuration options, default_config_name will be set to info.

If update_config is not provided, manifest['custom']['log_config'] will be used (if defined in the manifest).

Parameters:

Name Type Description Default
default_config_name str

A string, 'info' or 'debug', indicating the default template to use. (Defaults to 'info').

None
update_config dict

A dictionary containing the keys, sub-keys, and values of the templates to update. (Defaults to None).

None
Source code in fw_gear/context.py
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
def init_logging(self, default_config_name=None, update_config=None):
    """Configures logging via `fw_gear.logging.configure_logging`.

    If no `default_config_name` is provided, will get `debug` from the
    configuration options. If `debug` is False or not defined in the gear
    configuration options, `default_config_name` will be set to info.

    If `update_config` is not provided, manifest['custom']['log_config'] will be
    used (if defined in the manifest).

    Args:
        default_config_name (str, optional): A string, 'info' or 'debug', indicating
            the default template to use. (Defaults to 'info').
        update_config (dict, optional): A dictionary containing the keys, sub-keys,
            and values of the templates to update. (Defaults to
            None).
    """
    if not default_config_name:
        if self.config.opts.get("debug"):
            default_config_name = "debug"
        else:
            default_config_name = "info"
    if not update_config:
        # Only try to get log_config from manifest if manifest exists
        if self.manifest is not None:
            if isinstance(self.manifest.get_value("custom"), dict):
                update_config = self.manifest.get_value("custom").get(
                    "log_config", None
                )

    configure_logging(
        default_config_name=default_config_name,
        update_config=update_config,
    )
    return default_config_name, update_config

is_fw_context()

Return True if the gear is running in a Flywheel gear context, False otherwise.

Source code in fw_gear/context.py
165
166
167
168
169
170
def is_fw_context(self) -> bool:
    """Return True if the gear is running in a Flywheel gear context, False otherwise."""

    gear_path = Path("/flywheel/v0")

    return gear_path.exists() and Path(gear_path / "config.json").exists()

log_config()

Print the configuration and input files to the logger.

Source code in fw_gear/context.py
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
def log_config(self):
    """Print the configuration and input files to the logger."""
    # Log destination
    log.info(
        "Destination is %s=%s",
        self.config.destination.get("type"),
        self.config.destination.get("id"),
    )

    # Log file inputs
    for inp_name, inp in self.config.inputs.items():
        if inp["base"] != "file":
            continue

        container_type = inp.get("hierarchy", {}).get("type")
        container_id = inp.get("hierarchy", {}).get("id")
        file_name = inp.get("location", {}).get("name")

        log.info(
            'Input file "%s" is %s from %s=%s',
            inp_name,
            file_name,
            container_type,
            container_id,
        )

    # Log configuration values
    for key, value in self.config.opts.items():
        log.info('Config "%s=%s"', key, value)

open_output(name, mode='w', **kwargs)

Open the named output file.

Parameters:

Name Type Description Default
name str

The name of the output.

required
mode str

The open mode (Default value = 'w').

'w'
**kwargs dict

Keyword arguments for open.

{}

Returns:

Name Type Description
File IO

The file object.

Source code in fw_gear/context.py
238
239
240
241
242
243
244
245
246
247
248
249
250
def open_output(self, name: str, mode: str = "w", **kwargs) -> t.IO:
    """Open the named output file.

    Args:
        name: The name of the output.
        mode: The open mode (Default value = 'w').
        **kwargs (dict): Keyword arguments for `open`.

    Returns:
        File: The file object.
    """
    path = self.output_dir / name
    return path.open(mode, **kwargs)

Config

Basic Config class akin to Manifest class.

This class is mainly used as a helper for the gear config create function of the python-cli, however it may have unforeseen programmatic usages.

There is some confusing nomenclature here. The gear configuration is stored in a file named config.json, within the config.json file, there are at three keys: * config: configuration options for the gear, often booleans, integers, floats, or short strings. * inputs: Gear inputs, usually in the form of files, longer text. * destination: Information on where the gear is being run within the flywheel hierarchy.

This class represents the config.json file, meaning that is has attributes which store the configuration options, inputs, and destination. That means that the class Config has a config attribute (configuration options) and the constructor accepts a config dictionary that contains a config (configuration options) key.

Source code in fw_gear/config.py
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
class Config:
    """Basic Config class akin to Manifest class.

    This class is mainly used as a helper for the
    `gear config create` function of the python-cli,
    however it may have unforeseen programmatic usages.

    There is some confusing nomenclature here.  The gear
    configuration is stored in a file named `config.json`,
    within the config.json file, there are at three keys:
        * config:  configuration options for the gear,
            often booleans, integers, floats, or short
            strings.
        * inputs: Gear inputs, usually in the form of files,
            longer text.
        * destination: Information on where the gear is
            being run within the flywheel hierarchy.

    This class represents the `config.json` file, meaning
    that is has attributes which store the configuration
    options, inputs, and destination. That means that the
    class `Config` has a `config` attribute (configuration
    options) and the constructor accepts a `config`
    dictionary that contains a `config` (configuration
    options) key.
    """

    def __init__(
        self,
        client: t.Optional["flywheel.Client"],
        config: t.Optional[dict],
        path: Path = Path.cwd(),
    ):
        """Load or generate config.

        Args:
            client: Optional flywheel SDK client.
            config: Config dictionary with any of
                the following keys: 'config','inputs','destination'.
                Defaults to None.
            path: Path to existing
                configuration or path to directory in which to create
                config. Defaults to Path.cwd().

        Raises:
            ConfigValidationError:
                1. When a config kwarg has been passed but is
                    not of type dict.
                2. When a config.json exists at the passed path,
                    but it is not valid JSON
                3. When a file is passed in for path, but it does
                    not exist.
        """
        self._opts = dict()
        self._inputs = dict()
        self._destination = dict()
        self._path = Path(path)
        self._job = dict()
        self._client = client
        if config is not None:
            self._load_config_from_dict(config)
        else:
            self._load_config_from_file()

    def _load_config_from_dict(self, config: dict):
        """Load config values from a provided dictionary."""
        if not isinstance(config, dict):
            raise ConfigValidationError("Passed in", ["Cannot read config"])

        self.opts = config.get("config", {})
        self.inputs = config.get("inputs", {})
        self.destination = config.get("destination", {})
        self.job = config.get("job", {})

    def _load_config_from_file(self):
        """Load config values from a JSON file."""
        if self._path.is_dir():
            self._path = self._path / "config.json"
        if not self._path.exists() or not self._path.is_file():
            raise ConfigValidationError(self._path, ["File doesn't exist"])

        try:
            with open(self._path, "r") as fp:
                cfg = json.load(fp)
            self.opts = cfg.get("config", {})
            self.inputs = cfg.get("inputs", {})
            self.destination = cfg.get("destination", {})
            self.job = cfg.get("job", {})
        except json.JSONDecodeError:
            raise ConfigValidationError(self._path, ["Cannot read config file"])

    @property
    def opts(self):
        """Get provided configuration values."""
        return self._opts

    @property
    def job(self):
        """Get provided job info when show-job set to true."""
        return self._job

    @property
    def inputs(self):
        """Get provided inputs."""
        return self._inputs

    @property
    def destination(self):
        """Get gear run destination."""
        return self._destination

    @opts.setter
    def opts(self, config: dict):
        """Set gear config (for use in building gear local runs)."""
        self._opts = config

    @job.setter
    def job(self, job: dict):
        """Set gear config (for use in building gear local runs)."""
        self._job = job

    @inputs.setter
    def inputs(self, inputs: dict):
        """Set gear inputs (for use in building gear local runs)."""
        self._inputs = inputs

    @destination.setter
    def destination(self, dest: dict):
        """Set gear destination (for use in building gear local runs)."""
        self._destination = dest

    def update_opts(self, vals: dict):
        """Perform dictionary update on configuration values."""
        self._opts.update(vals)

    def update_destination(self, dest: dict):
        """Perform dictionary on destination dictionary."""
        self._destination.update(dest)

    def add_input(
        self, name: str, val: str, type_: str = "file", file_: t.Optional[t.Any] = None
    ):
        """Add an input to the config.

        Args:
            name: name of input
            val (str): input value, file path, api-key, context
            type_: file, api-key, or context. Defaults to "file".
            file_: File object to set

        Raises:
            ValueError: When file doesn't exist.
            NotImplementedError: When type is not file or api-key
        """
        if type_ == "file":
            path = Path(val)
            try:
                stat_result = path.resolve().stat()
            except FileNotFoundError:
                raise ValueError(
                    f"Cannot resolve file input at {path}, is the path correct?"
                )

            if file_:
                obj = {
                    "size": file_.size,
                    "type": file_.type,
                    "mimetype": file_.mimetype,
                    "modality": file_.modality,
                    "classification": file_.classification,
                    "tags": file_.tags,
                    "info": file_.info,
                    "zip_member_count": file_.zip_member_count,
                    "version": file_.version,
                    "file_id": file_.file_id,
                    "origin": {"type": "user", "id": ""},
                }
            else:
                obj = {
                    "size": stat_result.st_size,
                    "type": None,
                    "mimetype": "application/octet-stream",
                    "modality": None,
                    "classification": {},
                    "tags": [],
                    "info": {},
                    "zip_member_count": None,
                    "version": 1,
                    "file_id": "",
                    "origin": {"type": "user", "id": ""},
                }
            file = {
                "base": "file",
                "location": {
                    "name": path.name,
                    "path": f"/flywheel/v0/input/{name}/{path.name}",
                },
                "object": obj,
            }
            self.inputs.update({name: file})
        elif type_ == "api-key":
            self.inputs.update({name: {"base": "api-key", "key": val}})
        else:
            raise NotImplementedError(f"Unknown input type {type_}")

    @classmethod
    def default_config_from_manifest(
        cls, manifest: t.Union[Path, Manifest]
    ) -> "Config":
        """Create a default config.json from a manifest file.

        Args:
            manifest: Path to manifest or instantiated Manifest object.

        Raises:
            ValueError: When there is a problem parsing the manifest.

        Returns:
            Config: new config class with a default configuration.
        """
        if not isinstance(manifest, Manifest):
            try:
                manifest = Manifest(manifest)
            except ManifestValidationError:
                raise ValueError("Could not load manifest to generate config")

        config = {}

        for k, v in manifest.config.items():
            if "default" in v:
                config[k] = v["default"]

        return cls(None, config={"config": config})

    def get_input(self, name: str) -> t.Optional[dict]:
        """Get raw input object by name.

        Args:
            name: Name of input.
        """
        return self._inputs.get(name)

    def get_input_path(self, name: str) -> t.Optional[Path]:
        """Get input filepath by name.

        Args:
            name: Input name

        Raises:
            ValueError: If found input is not a file.

        Returns:
            t.Optional[Path]: Input filepath if found
        """
        input_ = self.get_input(name)
        if input_ is None:
            return None
        if input_.get("base") != "file":
            raise ValueError(f"The specified input {name} is not a file")
        return input_.get("location", {}).get("path")

    def get_input_filename(self, name) -> t.Optional[str]:
        """Get the the filename of given input file.
        Sourced from the 'inputs' field in the manifest.json

        Args:
            name (str): The name of the input.

        Raises:
            ValueError: if the input exists, but is not a file.

        Returns:
            str: The filename to the input file if it exists, otherwise None.
        """
        input_ = self.get_input(name)
        if input_ is None:
            return None
        if input_.get("base") != "file":
            raise ValueError(f"The specified input {name} is not a file")
        return input_.get("location", {}).get("name")

    def get_input_file_object(self, name: str) -> t.Optional[dict]:
        """Get the specified input file object from config.json

        Args:
            name (str): The name of the input.

        Returns:
            dict: The input dictionary, or None if not found.
        """
        input_ = self.get_input(name)
        if input_ is None:
            return None
        if input_.get("base") != "file":
            raise ValueError(f"The specified input {name} is not a file")
        return input_.get("object")

    def get_input_file_object_value(self, name: str, key: str) -> t.Any:
        """Get the value of the input file metadata from config.json.

        Args:
            name (str): The name of the input.
            key (str): The name of the file container metadata.

        Raises:
            ValueError: if the input exists, but is not a file.

        Returns:
            Union[str, list, dict]: The value of the specified file metadata, or None if not found.

        """
        inp_obj = self.get_input_file_object(name)
        return inp_obj.get(key, None) if key in inp_obj.keys() else None

    def open_input(self, name: str, mode: str = "r", **kwargs) -> t.IO:
        """Open the named input file, derived from the 'inputs' field in the manifest.

        Args:
            name: The name of the input.
            mode: The open mode (Default value = 'r').
            **kwargs (dict): Keyword arguments for `open`.

        Raises:
            ValueError: If input `name` is not defined in config.json.
            FileNotFoundError: If the path in the config.json for input `name` is
                not a file/

        Returns:
            The file object.
        """
        path = self.get_input_path(name)
        if path is None:
            raise ValueError(f"Input {name} is not defined in the config.json")
        if not Path(path).is_file():
            raise FileNotFoundError(f"Input {name} does not exist at {path}")
        return open(path, mode, **kwargs)

    def get_destination_container(self):
        """Returns the destination container."""
        if not self._client:
            # Error that we need API key.
            unauthenticated("Config.get_destination_container")
        return get_container_from_ref(self._client, self.destination)

    def get_destination_parent(self):
        """Returns the parent container of the destination container."""
        if not self._client:
            # Error that we need API key.
            unauthenticated("Config.get_destination_container")
        dest = self.get_destination_container()
        return get_parent(self._client, dest)

    ############### Utilities
    def to_json(self, path: t.Optional[Path] = None):
        """Dump config to a file in config.json format."""
        to_write = {
            "config": self.opts,
            "inputs": self.inputs,
            "destination": self.destination,
        }
        with open(path if path is not None else str(self._path), "w") as fp:
            json.dump(to_write, fp, indent=4)

    def __str__(self):  # pragma: no cover
        """Return string representation."""
        to_write = {
            "config": self.opts,
            "inputs": self.inputs,
            "destination": self.destination,
        }
        return json.dumps(to_write, indent=4)

destination property writable

Get gear run destination.

inputs property writable

Get provided inputs.

job property writable

Get provided job info when show-job set to true.

opts property writable

Get provided configuration values.

__init__(client, config, path=Path.cwd())

Load or generate config.

Parameters:

Name Type Description Default
client Optional[Client]

Optional flywheel SDK client.

required
config Optional[dict]

Config dictionary with any of the following keys: 'config','inputs','destination'. Defaults to None.

required
path Path

Path to existing configuration or path to directory in which to create config. Defaults to Path.cwd().

cwd()

Raises:

Type Description
ConfigValidationError
  1. When a config kwarg has been passed but is not of type dict.
  2. When a config.json exists at the passed path, but it is not valid JSON
  3. When a file is passed in for path, but it does not exist.
Source code in fw_gear/config.py
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
def __init__(
    self,
    client: t.Optional["flywheel.Client"],
    config: t.Optional[dict],
    path: Path = Path.cwd(),
):
    """Load or generate config.

    Args:
        client: Optional flywheel SDK client.
        config: Config dictionary with any of
            the following keys: 'config','inputs','destination'.
            Defaults to None.
        path: Path to existing
            configuration or path to directory in which to create
            config. Defaults to Path.cwd().

    Raises:
        ConfigValidationError:
            1. When a config kwarg has been passed but is
                not of type dict.
            2. When a config.json exists at the passed path,
                but it is not valid JSON
            3. When a file is passed in for path, but it does
                not exist.
    """
    self._opts = dict()
    self._inputs = dict()
    self._destination = dict()
    self._path = Path(path)
    self._job = dict()
    self._client = client
    if config is not None:
        self._load_config_from_dict(config)
    else:
        self._load_config_from_file()

__str__()

Return string representation.

Source code in fw_gear/config.py
378
379
380
381
382
383
384
385
def __str__(self):  # pragma: no cover
    """Return string representation."""
    to_write = {
        "config": self.opts,
        "inputs": self.inputs,
        "destination": self.destination,
    }
    return json.dumps(to_write, indent=4)

add_input(name, val, type_='file', file_=None)

Add an input to the config.

Parameters:

Name Type Description Default
name str

name of input

required
val str

input value, file path, api-key, context

required
type_ str

file, api-key, or context. Defaults to "file".

'file'
file_ Optional[Any]

File object to set

None

Raises:

Type Description
ValueError

When file doesn't exist.

NotImplementedError

When type is not file or api-key

Source code in fw_gear/config.py
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
def add_input(
    self, name: str, val: str, type_: str = "file", file_: t.Optional[t.Any] = None
):
    """Add an input to the config.

    Args:
        name: name of input
        val (str): input value, file path, api-key, context
        type_: file, api-key, or context. Defaults to "file".
        file_: File object to set

    Raises:
        ValueError: When file doesn't exist.
        NotImplementedError: When type is not file or api-key
    """
    if type_ == "file":
        path = Path(val)
        try:
            stat_result = path.resolve().stat()
        except FileNotFoundError:
            raise ValueError(
                f"Cannot resolve file input at {path}, is the path correct?"
            )

        if file_:
            obj = {
                "size": file_.size,
                "type": file_.type,
                "mimetype": file_.mimetype,
                "modality": file_.modality,
                "classification": file_.classification,
                "tags": file_.tags,
                "info": file_.info,
                "zip_member_count": file_.zip_member_count,
                "version": file_.version,
                "file_id": file_.file_id,
                "origin": {"type": "user", "id": ""},
            }
        else:
            obj = {
                "size": stat_result.st_size,
                "type": None,
                "mimetype": "application/octet-stream",
                "modality": None,
                "classification": {},
                "tags": [],
                "info": {},
                "zip_member_count": None,
                "version": 1,
                "file_id": "",
                "origin": {"type": "user", "id": ""},
            }
        file = {
            "base": "file",
            "location": {
                "name": path.name,
                "path": f"/flywheel/v0/input/{name}/{path.name}",
            },
            "object": obj,
        }
        self.inputs.update({name: file})
    elif type_ == "api-key":
        self.inputs.update({name: {"base": "api-key", "key": val}})
    else:
        raise NotImplementedError(f"Unknown input type {type_}")

default_config_from_manifest(manifest) classmethod

Create a default config.json from a manifest file.

Parameters:

Name Type Description Default
manifest Union[Path, Manifest]

Path to manifest or instantiated Manifest object.

required

Raises:

Type Description
ValueError

When there is a problem parsing the manifest.

Returns:

Name Type Description
Config Config

new config class with a default configuration.

Source code in fw_gear/config.py
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
@classmethod
def default_config_from_manifest(
    cls, manifest: t.Union[Path, Manifest]
) -> "Config":
    """Create a default config.json from a manifest file.

    Args:
        manifest: Path to manifest or instantiated Manifest object.

    Raises:
        ValueError: When there is a problem parsing the manifest.

    Returns:
        Config: new config class with a default configuration.
    """
    if not isinstance(manifest, Manifest):
        try:
            manifest = Manifest(manifest)
        except ManifestValidationError:
            raise ValueError("Could not load manifest to generate config")

    config = {}

    for k, v in manifest.config.items():
        if "default" in v:
            config[k] = v["default"]

    return cls(None, config={"config": config})

get_destination_container()

Returns the destination container.

Source code in fw_gear/config.py
352
353
354
355
356
357
def get_destination_container(self):
    """Returns the destination container."""
    if not self._client:
        # Error that we need API key.
        unauthenticated("Config.get_destination_container")
    return get_container_from_ref(self._client, self.destination)

get_destination_parent()

Returns the parent container of the destination container.

Source code in fw_gear/config.py
359
360
361
362
363
364
365
def get_destination_parent(self):
    """Returns the parent container of the destination container."""
    if not self._client:
        # Error that we need API key.
        unauthenticated("Config.get_destination_container")
    dest = self.get_destination_container()
    return get_parent(self._client, dest)

get_input(name)

Get raw input object by name.

Parameters:

Name Type Description Default
name str

Name of input.

required
Source code in fw_gear/config.py
249
250
251
252
253
254
255
def get_input(self, name: str) -> t.Optional[dict]:
    """Get raw input object by name.

    Args:
        name: Name of input.
    """
    return self._inputs.get(name)

get_input_file_object(name)

Get the specified input file object from config.json

Parameters:

Name Type Description Default
name str

The name of the input.

required

Returns:

Name Type Description
dict Optional[dict]

The input dictionary, or None if not found.

Source code in fw_gear/config.py
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
def get_input_file_object(self, name: str) -> t.Optional[dict]:
    """Get the specified input file object from config.json

    Args:
        name (str): The name of the input.

    Returns:
        dict: The input dictionary, or None if not found.
    """
    input_ = self.get_input(name)
    if input_ is None:
        return None
    if input_.get("base") != "file":
        raise ValueError(f"The specified input {name} is not a file")
    return input_.get("object")

get_input_file_object_value(name, key)

Get the value of the input file metadata from config.json.

Parameters:

Name Type Description Default
name str

The name of the input.

required
key str

The name of the file container metadata.

required

Raises:

Type Description
ValueError

if the input exists, but is not a file.

Returns:

Type Description
Any

Union[str, list, dict]: The value of the specified file metadata, or None if not found.

Source code in fw_gear/config.py
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
def get_input_file_object_value(self, name: str, key: str) -> t.Any:
    """Get the value of the input file metadata from config.json.

    Args:
        name (str): The name of the input.
        key (str): The name of the file container metadata.

    Raises:
        ValueError: if the input exists, but is not a file.

    Returns:
        Union[str, list, dict]: The value of the specified file metadata, or None if not found.

    """
    inp_obj = self.get_input_file_object(name)
    return inp_obj.get(key, None) if key in inp_obj.keys() else None

get_input_filename(name)

Get the the filename of given input file. Sourced from the 'inputs' field in the manifest.json

Parameters:

Name Type Description Default
name str

The name of the input.

required

Raises:

Type Description
ValueError

if the input exists, but is not a file.

Returns:

Name Type Description
str Optional[str]

The filename to the input file if it exists, otherwise None.

Source code in fw_gear/config.py
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
def get_input_filename(self, name) -> t.Optional[str]:
    """Get the the filename of given input file.
    Sourced from the 'inputs' field in the manifest.json

    Args:
        name (str): The name of the input.

    Raises:
        ValueError: if the input exists, but is not a file.

    Returns:
        str: The filename to the input file if it exists, otherwise None.
    """
    input_ = self.get_input(name)
    if input_ is None:
        return None
    if input_.get("base") != "file":
        raise ValueError(f"The specified input {name} is not a file")
    return input_.get("location", {}).get("name")

get_input_path(name)

Get input filepath by name.

Parameters:

Name Type Description Default
name str

Input name

required

Raises:

Type Description
ValueError

If found input is not a file.

Returns:

Type Description
Optional[Path]

t.Optional[Path]: Input filepath if found

Source code in fw_gear/config.py
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
def get_input_path(self, name: str) -> t.Optional[Path]:
    """Get input filepath by name.

    Args:
        name: Input name

    Raises:
        ValueError: If found input is not a file.

    Returns:
        t.Optional[Path]: Input filepath if found
    """
    input_ = self.get_input(name)
    if input_ is None:
        return None
    if input_.get("base") != "file":
        raise ValueError(f"The specified input {name} is not a file")
    return input_.get("location", {}).get("path")

open_input(name, mode='r', **kwargs)

Open the named input file, derived from the 'inputs' field in the manifest.

Parameters:

Name Type Description Default
name str

The name of the input.

required
mode str

The open mode (Default value = 'r').

'r'
**kwargs dict

Keyword arguments for open.

{}

Raises:

Type Description
ValueError

If input name is not defined in config.json.

FileNotFoundError

If the path in the config.json for input name is not a file/

Returns:

Type Description
IO

The file object.

Source code in fw_gear/config.py
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
def open_input(self, name: str, mode: str = "r", **kwargs) -> t.IO:
    """Open the named input file, derived from the 'inputs' field in the manifest.

    Args:
        name: The name of the input.
        mode: The open mode (Default value = 'r').
        **kwargs (dict): Keyword arguments for `open`.

    Raises:
        ValueError: If input `name` is not defined in config.json.
        FileNotFoundError: If the path in the config.json for input `name` is
            not a file/

    Returns:
        The file object.
    """
    path = self.get_input_path(name)
    if path is None:
        raise ValueError(f"Input {name} is not defined in the config.json")
    if not Path(path).is_file():
        raise FileNotFoundError(f"Input {name} does not exist at {path}")
    return open(path, mode, **kwargs)

to_json(path=None)

Dump config to a file in config.json format.

Source code in fw_gear/config.py
368
369
370
371
372
373
374
375
376
def to_json(self, path: t.Optional[Path] = None):
    """Dump config to a file in config.json format."""
    to_write = {
        "config": self.opts,
        "inputs": self.inputs,
        "destination": self.destination,
    }
    with open(path if path is not None else str(self._path), "w") as fp:
        json.dump(to_write, fp, indent=4)

update_destination(dest)

Perform dictionary on destination dictionary.

Source code in fw_gear/config.py
150
151
152
def update_destination(self, dest: dict):
    """Perform dictionary on destination dictionary."""
    self._destination.update(dest)

update_opts(vals)

Perform dictionary update on configuration values.

Source code in fw_gear/config.py
146
147
148
def update_opts(self, vals: dict):
    """Perform dictionary update on configuration values."""
    self._opts.update(vals)

Manifest

A class to host a manifest and a methods around it.

Parameters:

Name Type Description Default
manifest Path - like or dict

Path to a manifest.json file or a manifest dictionary. If none is provided, try to load manifest.json from the current working directory.

None

Attributes:

Name Type Description
manifest Dotty

Dictionary (dotty-enhanced version to be exact) of the manifest.json file.

Raises:

Type Description
FileNotFoundError

If no manifest.json found

Source code in fw_gear/manifest.py
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
class Manifest:
    """A class to host a manifest and a methods around it.

    Args:
        manifest (Path-like or dict): Path to a manifest.json file or a manifest dictionary.
            If none is provided, try to load manifest.json from the current working directory.

    Attributes:
        manifest (dotty_dict.Dotty): Dictionary (dotty-enhanced version to be exact) of the
            manifest.json file.

    Raises:
        FileNotFoundError: If no manifest.json found
    """

    def __init__(self, manifest=None):  # noqa: D107
        self._schema = None
        self._path = None
        self._gear_spec_schema = None
        self.manifest = None
        if isinstance(manifest, dict):
            self.manifest = dotty(manifest)
        elif not manifest:  # noqa: PLR5501
            # look into current directory
            self._path = (Path(".") / "manifest.json").absolute()
            if not self._path.exists():
                raise FileNotFoundError(
                    f"manifest.json not found in cwd ({os.getcwd()}"
                )
            log.info("Using manifest.json in current working directory.")
            self.manifest = dotty(self.get_manifest_from_file(path=self._path))
        else:  # Load from path
            self._path = Path(manifest)
            self.manifest = dotty(self.get_manifest_from_file(path=self._path))

    @property
    def schema(self):
        """Returns the json schema definition of the manifest.json file."""
        if not self._schema:
            self._schema = get_master_schema()
        return self._schema

    @property
    def gear_spec_schema(self):
        """Returns the json schema of the gear spec."""
        if not self._gear_spec_schema:
            self._gear_spec_schema = get_master_gear_schema()
        return self._gear_spec_schema

    @property
    def author(self):
        """Returns manifest author."""
        return self.get_value("author")

    @property
    def config(self):
        """Returns manifest config."""
        return self.get_value("config")

    @property
    def description(self):
        """Returns manifest description."""
        return self.get_value("description")

    @property
    def inputs(self):
        """Returns manifest inputs."""
        return self.get_value("inputs")

    @property
    def label(self):
        """Returns manifest label."""
        return self.get_value("label")

    @property
    def license(self):
        """Returns manifest license."""
        return self.get_value("license")

    @property
    def name(self):
        """Returns manifest name."""
        return self.get_value("name")

    @property
    def source(self):
        """Returns manifest source."""
        return self.get_value("source")

    @property
    def url(self):
        """Returns manifest url."""
        return self.get_value("url")

    @property
    def version(self):
        """Returns manifest version."""
        return self.get_value("version")

    @property
    def environment(self):  # noqa: D102
        return self.get_value("environment")

    def __getitem__(self, dotty_key):
        """Return any value of the manifest by passing the dotty-dict key.

        Args:
            dotty_key (str): A string representing a key or nested key (e.g.
                custom.gear-builder.image)
        """
        return self.get_value(dotty_key)

    @staticmethod
    def get_manifest_from_file(path=None):
        """Returns the dictionary representation of the manifest.json at path.

        Args:
            path (Path-like): Path to manifest. If None, look for manifest.json in current
                directory.

        Returns:
            (dict): The manifest as a dictionary
        """
        if isinstance(path, str):
            path = Path(path)

        if not path.exists():
            raise FileNotFoundError(str(path))

        with open(path, "r") as fp:
            try:
                manifest = json.load(fp, object_pairs_hook=OrderedDict)
            except JSONDecodeError as e:
                raise ManifestValidationError(
                    path,
                    [f"Error decoding at line {e.lineno}, column {e.colno}"],
                )

        return manifest

    def get_value(self, dotty_key):
        """Returns value found at dotty_key (e.g. 'custom.gear-builder.image') if any.

        More on dotty-dict notation at [here](https://github.com/pawelzny/dotty_dict)

        Args:
            dotty_key (str): A string representing a key or nested key (e.g.
                custom.gear-builder.image)
        """
        return self.manifest.get(dotty_key)

    def to_json(self, path, validate=True, ensure_ascii=True):
        """Save as json file to `path`."""
        if not Path(path).name.endswith(".json"):
            raise ValueError("Incorrect path. Must end with .json")

        if validate:
            try:
                self.validate()
            except ManifestValidationError:
                log.warning("The saved manifest.json is invalid")

        with open(path, "w") as fp:
            json.dump(dict(self.manifest), fp, indent=2, ensure_ascii=ensure_ascii)

    def get_docker_image_name_tag(self):
        """Returns docker image tag from either locations.

        Look first at 'custom.gear-builder.image'. If not defined there, look at
        'custom.docker-image' and log a warning.
        """
        image = self.get_value("custom.gear-builder.image")
        if not image:
            deprecated_image = self.get_value("custom.docker-image")
            if deprecated_image:
                log.warning(
                    'Defining image in "custom.docker-image" is deprecated. '
                    'Please use "custom.gear-builder.image" instead'
                )
            image = None
        return image

    def validate(self, validate_classification=False):
        """Validates manifest.

        Raises:
            ManifestValidationError: If errors are found in the manifest.
        """
        errors = self._validate(validate_classification)
        if errors:
            raise ManifestValidationError(self._path, errors)

    def is_valid(self):
        """Returns True if manifest is valid, False otherwise."""
        try:
            self.validate()
        except ManifestValidationError:
            return False
        return True

    def _validate(self, validate_classification=False):
        """Runs validation checks on manifest.

        Returns:
            (list): List of errors found.
        """
        errors = []
        errors += self._validate_schema()
        errors += self._validate_config_default()
        errors += self._is_docker_image_defined()
        errors += self._is_version_matches()
        errors += self._is_docker_images_match()
        if validate_classification:
            errors += self._validate_gear_suite()
            errors += self._validate_gear_classification()
        # Don't need to enforce this
        # errors += self._is_docker_names_match()
        return errors

    def _validate_schema(self):
        """Validates the manifest against its schema definition."""
        validator = jsonschema.Draft4Validator(self.schema)
        errors = []
        for error in sorted(validator.iter_errors(dict(self.manifest)), key=str):
            errors.append(error.message)
        return errors

    def _validate_config_default(self):
        """Validate default value in manifest."""
        errors = []
        if "config" in self.manifest:
            config_schema = self.derive_config_schema(full=False)
            validator = jsonschema.Draft4Validator(config_schema)
            # mock a config dict with default values from manifest
            config = {
                "config": {
                    k: v["default"]
                    for k, v in self.manifest.get("config", {}).items()
                    if "default" in v
                }
            }

            for error in sorted(validator.iter_errors(config), key=str):
                errors.append(error.message)
        return errors

    def _validate_gear_suite(self):
        """Validates gear suite."""
        errors = []
        # get the suite
        manifest_suite = self.manifest.get("custom.flywheel.suite", None)

        # Check if the suite is missing
        if manifest_suite is None:
            errors += ["Suite is missing in manifest."]
        # Check if it is a string
        elif not isinstance(manifest_suite, str):
            errors += ["Suite must be a string"]

        # check if it is a valid suite
        suite_vals = self.gear_spec_schema["suite"]
        if manifest_suite not in suite_vals:
            errors += [
                f"Value {manifest_suite} not found in schema suite. Available values are {suite_vals}"
            ]
        return errors

    def _validate_gear_classification(self):
        """Validates gear classification."""
        errors = []
        manifest_classification = self.manifest.get(
            "custom.flywheel.classification", None
        )

        # Check if the classification is missing
        if manifest_classification is None:
            errors += ["Classification is missing in manifest."]
        # Check if it's a dictionary
        elif not isinstance(manifest_classification, dict):
            errors += ["Classification must be a dictionary"]
        else:
            # Iterate over the classification items and validate
            for classification, vals in manifest_classification.items():
                if classification not in self.gear_spec_schema.keys():
                    errors += [
                        f"Classification {classification} not found in gear spec schema"
                    ]
                    continue  # skip to the next classification if not found

                schema_val_list = self.gear_spec_schema[classification]

                for val in vals:
                    if val not in schema_val_list:
                        errors += [
                            f"Value {val} not found in {classification} classification"
                        ]
        return errors

    def derive_config_schema(self, full=True):
        """Returns json schema for config from manifest.

        Adapted from
        https://gitlab.com/flywheel-io/public/gears/-/blob/master/gears/generator.py#L66.

        Args:
            full (bool): Whether to derive a "full" config schema, to be used
                when validating a real config that has been populated.  Set
                "full" to false if you are just validating the default
                values in a config.
        """
        # Config jsonschema
        schema = {
            "title": "Config schema for manifest",
            "$schema": "http://json-schema.org/draft-04/schema#",
            "type": "object",
            "properties": {
                "config": {"type": "object", "properties": {}, "required": []},
            },
            "required": ["config"],
        }

        # Copy over constraints from manifest
        for key in self.manifest["config"]:
            # Copy constraints, removing 'base' and 'description' keywords which are not constraints
            value = copy.deepcopy(self.manifest["config"][key])
            value.pop("base", None)
            value.pop("description", None)
            optional = value.pop("optional", False)
            schema["properties"]["config"]["properties"][key] = value

            # Require the key be present unless optional flag is set.
            if not optional:
                # Always add the required key if full
                if full:
                    schema["properties"]["config"]["required"].append(key)
                # Else only add the required key if a default value is
                # specified (GEAR-1866)
                elif value.get("default"):
                    schema["properties"]["config"]["required"].append(key)

        # After handling each key, remove required array if none are present.
        # Required by jsonschema (minItems 1).
        if len(schema["properties"]["config"]["required"]) == 0:
            schema["properties"]["config"].pop("required", None)

        # Important: check our work - the schema must be a valid schema.
        jsonschema.Draft4Validator.check_schema(schema)

        return schema

    def _is_docker_image_defined(self):
        """Returns a list of errors if not defined."""
        errors = []
        docker_image = self.get_docker_image_name_tag()
        if not docker_image:
            errors += ['"custom.gear-builder.image" missing from manifest']
        return errors

    def _is_docker_images_match(self):
        """Returns a list of errors if docker images are different."""
        errors = []
        docker_image_1 = self.get_value("custom.gear-builder.image")
        docker_image_2 = self.get_value("custom.docker-image")
        if docker_image_1 and docker_image_2 and docker_image_1 != docker_image_2:
            errors += [
                '"custom.gear-builder.image" is different from "custom.docker-image"'
            ]
        return errors

    def _is_docker_names_match(self):
        """Returns a list of errors if docker name and image name are different."""
        errors = []
        docker_image = self.get_docker_image_name_tag()
        docker_name = self.name
        if docker_image and ":" in docker_image:
            im_path, _ = docker_image.split(":")
            if "/" in im_path:
                _, im_name = im_path.split("/")
                if im_name != docker_name:
                    errors += ['"custom.gear-builder.image" is different from "name"']
        return errors

    def _is_version_matches(self):
        """Returns a list of errors if version does not match."""
        errors = []
        docker_image = self.get_docker_image_name_tag()
        if docker_image and ":" in docker_image:
            _, im_tag = docker_image.split(":")
            if im_tag != self.version:
                errors += ["version and docker image tag do not match"]

        return errors

author property

Returns manifest author.

config property

Returns manifest config.

description property

Returns manifest description.

gear_spec_schema property

Returns the json schema of the gear spec.

inputs property

Returns manifest inputs.

label property

Returns manifest label.

license property

Returns manifest license.

name property

Returns manifest name.

schema property

Returns the json schema definition of the manifest.json file.

source property

Returns manifest source.

url property

Returns manifest url.

version property

Returns manifest version.

__getitem__(dotty_key)

Return any value of the manifest by passing the dotty-dict key.

Parameters:

Name Type Description Default
dotty_key str

A string representing a key or nested key (e.g. custom.gear-builder.image)

required
Source code in fw_gear/manifest.py
122
123
124
125
126
127
128
129
def __getitem__(self, dotty_key):
    """Return any value of the manifest by passing the dotty-dict key.

    Args:
        dotty_key (str): A string representing a key or nested key (e.g.
            custom.gear-builder.image)
    """
    return self.get_value(dotty_key)

derive_config_schema(full=True)

Returns json schema for config from manifest.

Adapted from https://gitlab.com/flywheel-io/public/gears/-/blob/master/gears/generator.py#L66.

Parameters:

Name Type Description Default
full bool

Whether to derive a "full" config schema, to be used when validating a real config that has been populated. Set "full" to false if you are just validating the default values in a config.

True
Source code in fw_gear/manifest.py
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
def derive_config_schema(self, full=True):
    """Returns json schema for config from manifest.

    Adapted from
    https://gitlab.com/flywheel-io/public/gears/-/blob/master/gears/generator.py#L66.

    Args:
        full (bool): Whether to derive a "full" config schema, to be used
            when validating a real config that has been populated.  Set
            "full" to false if you are just validating the default
            values in a config.
    """
    # Config jsonschema
    schema = {
        "title": "Config schema for manifest",
        "$schema": "http://json-schema.org/draft-04/schema#",
        "type": "object",
        "properties": {
            "config": {"type": "object", "properties": {}, "required": []},
        },
        "required": ["config"],
    }

    # Copy over constraints from manifest
    for key in self.manifest["config"]:
        # Copy constraints, removing 'base' and 'description' keywords which are not constraints
        value = copy.deepcopy(self.manifest["config"][key])
        value.pop("base", None)
        value.pop("description", None)
        optional = value.pop("optional", False)
        schema["properties"]["config"]["properties"][key] = value

        # Require the key be present unless optional flag is set.
        if not optional:
            # Always add the required key if full
            if full:
                schema["properties"]["config"]["required"].append(key)
            # Else only add the required key if a default value is
            # specified (GEAR-1866)
            elif value.get("default"):
                schema["properties"]["config"]["required"].append(key)

    # After handling each key, remove required array if none are present.
    # Required by jsonschema (minItems 1).
    if len(schema["properties"]["config"]["required"]) == 0:
        schema["properties"]["config"].pop("required", None)

    # Important: check our work - the schema must be a valid schema.
    jsonschema.Draft4Validator.check_schema(schema)

    return schema

get_docker_image_name_tag()

Returns docker image tag from either locations.

Look first at 'custom.gear-builder.image'. If not defined there, look at 'custom.docker-image' and log a warning.

Source code in fw_gear/manifest.py
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
def get_docker_image_name_tag(self):
    """Returns docker image tag from either locations.

    Look first at 'custom.gear-builder.image'. If not defined there, look at
    'custom.docker-image' and log a warning.
    """
    image = self.get_value("custom.gear-builder.image")
    if not image:
        deprecated_image = self.get_value("custom.docker-image")
        if deprecated_image:
            log.warning(
                'Defining image in "custom.docker-image" is deprecated. '
                'Please use "custom.gear-builder.image" instead'
            )
        image = None
    return image

get_manifest_from_file(path=None) staticmethod

Returns the dictionary representation of the manifest.json at path.

Parameters:

Name Type Description Default
path Path - like

Path to manifest. If None, look for manifest.json in current directory.

None

Returns:

Type Description
dict

The manifest as a dictionary

Source code in fw_gear/manifest.py
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
@staticmethod
def get_manifest_from_file(path=None):
    """Returns the dictionary representation of the manifest.json at path.

    Args:
        path (Path-like): Path to manifest. If None, look for manifest.json in current
            directory.

    Returns:
        (dict): The manifest as a dictionary
    """
    if isinstance(path, str):
        path = Path(path)

    if not path.exists():
        raise FileNotFoundError(str(path))

    with open(path, "r") as fp:
        try:
            manifest = json.load(fp, object_pairs_hook=OrderedDict)
        except JSONDecodeError as e:
            raise ManifestValidationError(
                path,
                [f"Error decoding at line {e.lineno}, column {e.colno}"],
            )

    return manifest

get_value(dotty_key)

Returns value found at dotty_key (e.g. 'custom.gear-builder.image') if any.

More on dotty-dict notation at here

Parameters:

Name Type Description Default
dotty_key str

A string representing a key or nested key (e.g. custom.gear-builder.image)

required
Source code in fw_gear/manifest.py
159
160
161
162
163
164
165
166
167
168
def get_value(self, dotty_key):
    """Returns value found at dotty_key (e.g. 'custom.gear-builder.image') if any.

    More on dotty-dict notation at [here](https://github.com/pawelzny/dotty_dict)

    Args:
        dotty_key (str): A string representing a key or nested key (e.g.
            custom.gear-builder.image)
    """
    return self.manifest.get(dotty_key)

is_valid()

Returns True if manifest is valid, False otherwise.

Source code in fw_gear/manifest.py
211
212
213
214
215
216
217
def is_valid(self):
    """Returns True if manifest is valid, False otherwise."""
    try:
        self.validate()
    except ManifestValidationError:
        return False
    return True

to_json(path, validate=True, ensure_ascii=True)

Save as json file to path.

Source code in fw_gear/manifest.py
170
171
172
173
174
175
176
177
178
179
180
181
182
def to_json(self, path, validate=True, ensure_ascii=True):
    """Save as json file to `path`."""
    if not Path(path).name.endswith(".json"):
        raise ValueError("Incorrect path. Must end with .json")

    if validate:
        try:
            self.validate()
        except ManifestValidationError:
            log.warning("The saved manifest.json is invalid")

    with open(path, "w") as fp:
        json.dump(dict(self.manifest), fp, indent=2, ensure_ascii=ensure_ascii)

validate(validate_classification=False)

Validates manifest.

Raises:

Type Description
ManifestValidationError

If errors are found in the manifest.

Source code in fw_gear/manifest.py
201
202
203
204
205
206
207
208
209
def validate(self, validate_classification=False):
    """Validates manifest.

    Raises:
        ManifestValidationError: If errors are found in the manifest.
    """
    errors = self._validate(validate_classification)
    if errors:
        raise ManifestValidationError(self._path, errors)

Metadata

Class to store and interact with Flywheel metadata via .metadata.json or SDK call.

Source code in fw_gear/metadata.py
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
class Metadata:
    """Class to store and interact with Flywheel metadata via .metadata.json or SDK call."""

    def __init__(
        self,
        context: t.Optional["GearContext"] = None,
        name_override: str = "",
        version_override: str = "",
    ):
        """Initialize metadata class.

        Args:
            context (Optional[GearContext]): GearContext from
                which to populate gear info (inputs, config, destination, ...)
            name_override (str, optional): Optional gear name to use, overrides
                manifest value from GearContext. Defaults to "".
            version_override (str, optional): Optional gear version to use,
                overrides manifest value from GearContext.
                Defaults to "".
        """
        self._metadata: dict = {}
        # Read-only gear context for configuration options
        self._context = context
        self._has_job_info = False
        self.name_override = name_override
        self.version_override = version_override

    ###################
    # HELPER FUNCTION #
    ###################
    def _validate_container_type(self, container_type: str) -> bool:
        """Validate container type is part of the destination hierarchy.

        Args:
            container_type (str): Container type to validate.
        """
        if container_type not in CONT_ORDER:
            raise ValueError(
                f"Container type {container_type} is not part of the hierarchy."
            )

        if self._context.config.destination.get("type") == "analysis":
            parent_container = self._context.config.get_destination_parent()
        else:
            parent_container = self._context.config.get_destination_container()
        # validate if container type is part of the hierarchy
        if parent_container.container_type not in CONT_ORDER:
            raise ValueError(
                f"Parent container type {parent_container.container_type} is not part of the hierarchy."
            )
        parent_container_index = CONT_ORDER.index(parent_container.container_type)

        container_index = CONT_ORDER.index(container_type)
        if container_index > parent_container_index:
            return False  # not allowed to update children
        return True  # ok to update same level or parent

    ##################
    # UPDATE METHODS #
    ##################
    # update methods with .metadata.json
    def update_container(
        self, container_type: str, deep: bool = True, **kwargs
    ) -> None:
        """Update metadata for the given container type in the hierarchy.

        Args:
            container_type (str): The container type (e.g. session or
                acquisition).
            deep (bool): Perform a deep (recursive) update on subdictionaries.
                Default: True
            **kwargs (dict): Update arguments
        """
        if not self._validate_container_type(container_type):
            raise ValueError(
                f"Container type {container_type} is outside the hierarchy that can be updated via .metadata.json."
            )
        dest = self._metadata.setdefault(container_type, {})
        if deep:
            deep_merge(dest, **kwargs)
        else:
            dest.update(**kwargs)

    def update_file_metadata(
        self,
        file_: t.Any,
        container_type: str,
        deep: bool = True,
        **kwargs,
    ) -> None:
        """Update file metadata by name.

        Update a file by name in the hierarchy at the given container type.

        Args:
            file_: File name (str), SDK file (flywheel.FileEntry) or
                dictionary from config.json
            container_type (str): Type of parent container.
            deep (bool): Perform a deep (recursive) update on subdictionaries.
                Default: True
            **kwargs (dict): Update arguments
        """
        if container_type != "analysis" and not self._validate_container_type(
            container_type
        ):
            raise ValueError(
                f"Container type {container_type} is outside the hierarchy that can be updated via .metadata.json."
            )

        # Find file by name in the given container
        file_obj = get_file(file_, self._context, container_type)
        parent = self._metadata.setdefault(file_obj.parent_type, {})
        files = parent.setdefault("files", [])
        file_entry = None
        for fe in files:
            if fe.get("name") == file_obj.name:
                file_entry = fe
                break
        # No metadata entry for given file exists yet.
        if file_entry is None:
            # Initialize with file name and any existing info if present
            file_entry = {"name": file_obj.name, "info": file_obj.info}
            # Add this file entry to the parent container's 'file' key
            files.append(file_entry)
        # Placeholder reference to location in the full dictionary we want
        # to update, i.e. the file entry
        target = file_entry
        # Update metadata with kwargs
        if deep:
            deep_merge(target, **kwargs)
        else:
            target.update(kwargs)

    def update_zip_member_count(
        self,
        path: Path,
        container_type: t.Optional[str] = None,
    ) -> None:
        """Update metadata with zip-member count.

        Add the zip member count for files in a given directory the metadata

        Args:
            path (Path): directory or file.
            container_type (Optional[str]): Container type found files should
                be associated with.
        """
        if container_type is None:
            # This will happen, e.g., if we are running a gear locally.
            log.warning("No container type provided, skipping zip member count update.")
            return
        if not path.exists():
            log.warning("Provided path does not exist: %s", path)
            return
        if path.is_file():
            files = iter([path])
        else:
            files = path.rglob("*")
        for file in files:
            if file.suffix != ".zip":
                continue
            try:
                with zipfile.ZipFile(file) as archive:
                    count = len(archive.infolist())
                    self.update_file_metadata(
                        file.name,
                        container_type=container_type,
                        zip_member_count=count,
                    )
            except zipfile.BadZipFile:
                log.warning("Invalid zip file %s. Skipping update", file)

    # update methods with HTTP requests
    def modify_container_file_info(
        self,
        file_: File,
        **info,
    ) -> None:
        """Modify file info with HTTP requests.

        Wrapper for modify_container_file_info SDK method call.
        Using set method (add on) to update.

        Args:
            file_ (File): File Object
            info (e.Dict[t.Any]): Info dictionary that will be used to update file info.
        """
        if self._context.client is None:
            raise AttributeError("FW Client was not initialized or set.")

        self._context.client.modify_container_file_info(
            file_.parent_id, file_.name, {"set": info}
        )

    def modify_container_info(self, cont_id: str, **info) -> None:
        """Update metadata for the given container type in the hierarchy.

        Wrapper for modify_container_info SDK method call.
        Using set method (add on) to update.

        Args:
            cont_id (str): Container ID that will be used to update container info
            info (e.Dict[t.Any]): Info dictionary that will be used to update the container info.
        """
        if self._context.client is None:
            raise AttributeError("FW Client was not initialized or set.")

        self._context.client.modify_container_info(cont_id, {"set": info})

    ###################
    # WRITING METHODS #
    ###################
    # methods that will be used with .metadata.json
    def clean(self):
        """Clean metadata before writing."""
        self._metadata = convert_nan_in_dict(self._metadata)
        self._metadata = sanitize_periods(self._metadata)

        # Serialize to JSON string
        json_string = json.dumps(self._metadata, cls=MetadataEncoder)

        # Check size
        size = len(json_string.encode("utf-8"))
        if size > 16 * 1024 * 1024:  # 16MB limit
            warnings.warn(
                f"Metadata size exceeds 16MB: {size} bytes. Consider breaking metadata into smaller chunks.",
                UserWarning,
            )

        # Deserialize to prepare for writing (if needed)
        self._metadata = json.loads(json_string)

    def log(self):
        """Log representation of metadata."""
        rep = copy.deepcopy(self._metadata)
        log.info(
            ".metadata.json:\n%s",
            json.dumps(trim(rep), indent=2, cls=MetadataEncoder),
        )

    def write(
        self, directory: Path, fail_on_validation: bool = False, log_meta: bool = True
    ) -> None:
        """Write metadata to .metadata.json in the given directory.

        Args:
            directory (Path): Directory in which to write .metadata.json
            fail_on_validation (bool): Fail if engine metadata schema
                validation has errors. Default: False
            log_meta (bool): Also log cleaned metadata. Default: True
        """
        if not self._metadata:
            return
        self.clean()
        log.debug("Validating generated metadata")
        with open(PARENT_DIR / "engine_metadata_schema.json") as fp:
            engine_metadata_schema = json.load(fp)
        validator = jsonschema.Draft7Validator(engine_metadata_schema)
        errors = []
        for err in validator.iter_errors(self._metadata):
            errors.append(err)
        if errors:
            log.error("Error(s) validating produced metadata.")
            for err in errors:
                log.error(err)
            if fail_on_validation:
                self.log()
                sys.exit(1)
        if log_meta:
            self.log()

        with open(directory / ".metadata.json", "w", encoding="utf-8") as f:
            json.dump(self._metadata, f, indent=2, cls=MetadataEncoder)

    ##############
    # QC METHODS #
    ##############
    # adding qc result to .metadata.json
    def add_qc_result(self, file_: t.Any, name: str, state: str, **data) -> None:
        """Add a pass/fail/na QC results to a file and update .metadata.json.

        Args:
            file_ (t.Any): File object
            name (str): QC result name
            state (str): pass, fail, or na
            data: custom data
        """
        qc_result = create_qc_result_dict(name, state, **data)
        file_obj = get_file(file_, self._context)
        updated_file_info = self.add_gear_info("qc", file_, **qc_result)  # type: ignore
        self.update_file_metadata(
            file_, container_type=file_obj.parent_type, info=updated_file_info
        )

    # add qc methods for analysis container via .metadata.json
    def add_qc_result_to_analysis(self, name: str, state: str, **data) -> None:
        """Add a pass/fail/na QC results to the destination analysis container via update .metadata.json.

        Args:
            name (str): QC result name
            state (str): pass, fail, or na
            data: custom data
        """
        qc_result = create_qc_result_dict(name, state, **data)
        if self._context.config.destination.get("type") != "analysis":
            raise TypeError("Destination container is not an analysis container.")
        analysis_container = self._context.config.get_destination_container()
        updated_file_info = self.add_gear_info("qc", analysis_container, **qc_result)  # type: ignore
        self.update_container("analysis", info=updated_file_info)

    # add qc methods via API call
    def add_qc_result_via_sdk(self, cont_: t.Any, name: str, state: str, **data):
        """Add a pass/fail/na QC results to a container/file and update the container/file info via HTTP methods.

        Args:
            cont_ (t.Any): Flywheel Container object
            name (str): QC result name
            state (str): pass, fail, or na
            data: custom data

        """
        # generate qc result with provided information
        qc_result = create_qc_result_dict(name, state, **data)

        if isinstance(cont_, (dict, str)) or not hasattr(cont_, "files"):
            file_obj = get_file(cont_, self._context)
            updated_file_info = self.add_gear_info("qc", file_obj, **qc_result)
            self.modify_container_file_info(file_obj, **updated_file_info)
        else:
            updated_cont_info = self.add_gear_info("qc", cont_, **qc_result)
            self.modify_container_info(cont_.id, **updated_cont_info)

    # adding file tag to .metadata.json
    def add_file_tags(self, file_: t.Any, tags: t.Union[str, t.Iterable[str]]) -> None:
        """Add tag(s) to a file.

        Args:
            file_ (t.Any): File Object
            tags (str or Iterable[str]): Tags to add.
        """
        file_obj = get_file(file_, self._context)
        to_add = tags
        if not tags:
            to_add = []
        if isinstance(tags, str):
            to_add = [tags]
        exist_tags = file_obj.tags
        new_tags = list(set([*exist_tags, *to_add]))  # Ensure unique tags
        self.update_file_metadata(
            file_, tags=new_tags, container_type=file_obj.parent_type
        )

    ####################
    # JOB/GEAR METHODS #
    ####################
    def pull_job_info(self):
        """Pull job info from GearContext if present.

        Get job input files, configuration, and job id.

        Args:
            name (str): Gear name override.
            version (str): Gear version override.
        """
        name = self.name_override
        version = self.version_override
        # Don't attempt to set gear info if context not passed in.
        self.job = ""
        if not self._context:
            log.info("Context not provided, adding gear info from name and version")
            info = {name: {"job_info": version}}
            self.job_info = info
            self.name = name
            self.version = version
            self._has_job_info = True
            return
        # Add gear information, configuration, inputs, etc.
        gear_inputs = {}
        for i_name, fe in self._context.config.inputs.items():
            if fe["base"] != "file":
                continue
            obj = fe.get("object", {})
            gear_inputs[i_name] = {
                "parent": fe.get("hierarchy"),
                "file_id": obj.get("file_id", ""),
                "version": obj.get("version", ""),
                "file_name": fe.get("location", {}).get("name", ""),
            }
        # First check for job provided in config
        if self._context.config.job:
            self.job = self._context.config.job.get("id", "")
        elif self._context.config.destination.get("type", "") == "analysis":  # noqa: PLR5501
            try:
                dest = self._context.config.get_destination_container()
                self.job = dest.get("job", {}).get("id", "")
            except AttributeError:
                # When client = None
                log.warning("Could not use SDK to look up job id for analysis")
        if not self.job:
            log.warning("Could not determine job id.")

        m = self._context.manifest
        # Handle the case when manifest is None
        if m is None:
            if not name or not version:
                log.debug(
                    "Manifest is None and no name/version override provided. Using defaults."
                )
                name = name or "unknown-gear"
                version = version or "0.0.0"
        else:
            name = name if name else m.name
            version = version if version else m.version
        info = {
            name: {
                "job_info": {
                    "version": version,
                    "job_id": self.job,
                    "inputs": gear_inputs,
                    "config": self._context.config.opts,
                },
            }
        }
        self.job_info = info
        self.name = name
        self.version = version
        self._has_job_info = True

    def add_or_update_gear_info(
        self, top_level_keys: t.List[str], cont_info: t.Any, **kwargs: t.Any
    ) -> dict:
        """Modify container info section with current job run info
        including info about the gear."""

        if not self._has_job_info:
            self.pull_job_info()

        top = cont_info
        for key in top_level_keys:
            # Check if the current value is not a dictionary
            if key in top and not isinstance(top[key], dict):
                raise TypeError(
                    f"Expected a dictionary at key '{key}', but got {type(top[key]).__name__}."
                )

            # Set default to an empty dictionary if the key doesn't exist
            top = top.setdefault(key, {})

        # Check if gear info has already been added
        if self.name in top:
            # Same (present) job id --> don't update job_info
            existing_info = top[self.name].get("job_info", {})
            if not (id := existing_info.get("job_id", "")) or id != self.job:
                top.update(self.job_info)
        else:
            top.update(self.job_info)

        top = top[self.name]
        top.update(**kwargs)

        return cont_info

    def add_gear_info(
        self,
        top_level: str,
        cont_: t.Any,
        **kwargs: t.Any,
    ) -> dict:
        """Add arbitrary gear info to a particular file/container.

        Add predefined info on gear configuration, as well as custom free-form
        information to a given file/container.

        Args:
            top_level (str): Top level info key under "info" in which to store
                the data, i.e. "qc.result" stores info under "info.qc.result".
            cont_ (t.Any]): File/container to store information on
            kwargs: Custom information to add under gear info.
        """
        if not self._has_job_info:
            self.pull_job_info()

        if isinstance(cont_, (dict, str)) or not hasattr(cont_, "files"):
            if isinstance(cont_, File):
                cont_obj = cont_
            else:
                # Get existing info on file
                cont_obj = get_file(cont_, self._context)
        else:
            cont_obj = cont_

        # Insert or get top level key(s)
        keys = top_level.split(".")

        # updated file/container info object
        return self.add_or_update_gear_info(keys, cont_obj.info, **kwargs)

__init__(context=None, name_override='', version_override='')

Initialize metadata class.

Parameters:

Name Type Description Default
context Optional[GearContext]

GearContext from which to populate gear info (inputs, config, destination, ...)

None
name_override str

Optional gear name to use, overrides manifest value from GearContext. Defaults to "".

''
version_override str

Optional gear version to use, overrides manifest value from GearContext. Defaults to "".

''
Source code in fw_gear/metadata.py
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
def __init__(
    self,
    context: t.Optional["GearContext"] = None,
    name_override: str = "",
    version_override: str = "",
):
    """Initialize metadata class.

    Args:
        context (Optional[GearContext]): GearContext from
            which to populate gear info (inputs, config, destination, ...)
        name_override (str, optional): Optional gear name to use, overrides
            manifest value from GearContext. Defaults to "".
        version_override (str, optional): Optional gear version to use,
            overrides manifest value from GearContext.
            Defaults to "".
    """
    self._metadata: dict = {}
    # Read-only gear context for configuration options
    self._context = context
    self._has_job_info = False
    self.name_override = name_override
    self.version_override = version_override

add_file_tags(file_, tags)

Add tag(s) to a file.

Parameters:

Name Type Description Default
file_ Any

File Object

required
tags str or Iterable[str]

Tags to add.

required
Source code in fw_gear/metadata.py
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
def add_file_tags(self, file_: t.Any, tags: t.Union[str, t.Iterable[str]]) -> None:
    """Add tag(s) to a file.

    Args:
        file_ (t.Any): File Object
        tags (str or Iterable[str]): Tags to add.
    """
    file_obj = get_file(file_, self._context)
    to_add = tags
    if not tags:
        to_add = []
    if isinstance(tags, str):
        to_add = [tags]
    exist_tags = file_obj.tags
    new_tags = list(set([*exist_tags, *to_add]))  # Ensure unique tags
    self.update_file_metadata(
        file_, tags=new_tags, container_type=file_obj.parent_type
    )

add_gear_info(top_level, cont_, **kwargs)

Add arbitrary gear info to a particular file/container.

Add predefined info on gear configuration, as well as custom free-form information to a given file/container.

Parameters:

Name Type Description Default
top_level str

Top level info key under "info" in which to store the data, i.e. "qc.result" stores info under "info.qc.result".

required
cont_ t.Any]

File/container to store information on

required
kwargs Any

Custom information to add under gear info.

{}
Source code in fw_gear/metadata.py
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
def add_gear_info(
    self,
    top_level: str,
    cont_: t.Any,
    **kwargs: t.Any,
) -> dict:
    """Add arbitrary gear info to a particular file/container.

    Add predefined info on gear configuration, as well as custom free-form
    information to a given file/container.

    Args:
        top_level (str): Top level info key under "info" in which to store
            the data, i.e. "qc.result" stores info under "info.qc.result".
        cont_ (t.Any]): File/container to store information on
        kwargs: Custom information to add under gear info.
    """
    if not self._has_job_info:
        self.pull_job_info()

    if isinstance(cont_, (dict, str)) or not hasattr(cont_, "files"):
        if isinstance(cont_, File):
            cont_obj = cont_
        else:
            # Get existing info on file
            cont_obj = get_file(cont_, self._context)
    else:
        cont_obj = cont_

    # Insert or get top level key(s)
    keys = top_level.split(".")

    # updated file/container info object
    return self.add_or_update_gear_info(keys, cont_obj.info, **kwargs)

add_or_update_gear_info(top_level_keys, cont_info, **kwargs)

Modify container info section with current job run info including info about the gear.

Source code in fw_gear/metadata.py
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
def add_or_update_gear_info(
    self, top_level_keys: t.List[str], cont_info: t.Any, **kwargs: t.Any
) -> dict:
    """Modify container info section with current job run info
    including info about the gear."""

    if not self._has_job_info:
        self.pull_job_info()

    top = cont_info
    for key in top_level_keys:
        # Check if the current value is not a dictionary
        if key in top and not isinstance(top[key], dict):
            raise TypeError(
                f"Expected a dictionary at key '{key}', but got {type(top[key]).__name__}."
            )

        # Set default to an empty dictionary if the key doesn't exist
        top = top.setdefault(key, {})

    # Check if gear info has already been added
    if self.name in top:
        # Same (present) job id --> don't update job_info
        existing_info = top[self.name].get("job_info", {})
        if not (id := existing_info.get("job_id", "")) or id != self.job:
            top.update(self.job_info)
    else:
        top.update(self.job_info)

    top = top[self.name]
    top.update(**kwargs)

    return cont_info

add_qc_result(file_, name, state, **data)

Add a pass/fail/na QC results to a file and update .metadata.json.

Parameters:

Name Type Description Default
file_ Any

File object

required
name str

QC result name

required
state str

pass, fail, or na

required
data

custom data

{}
Source code in fw_gear/metadata.py
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
def add_qc_result(self, file_: t.Any, name: str, state: str, **data) -> None:
    """Add a pass/fail/na QC results to a file and update .metadata.json.

    Args:
        file_ (t.Any): File object
        name (str): QC result name
        state (str): pass, fail, or na
        data: custom data
    """
    qc_result = create_qc_result_dict(name, state, **data)
    file_obj = get_file(file_, self._context)
    updated_file_info = self.add_gear_info("qc", file_, **qc_result)  # type: ignore
    self.update_file_metadata(
        file_, container_type=file_obj.parent_type, info=updated_file_info
    )

add_qc_result_to_analysis(name, state, **data)

Add a pass/fail/na QC results to the destination analysis container via update .metadata.json.

Parameters:

Name Type Description Default
name str

QC result name

required
state str

pass, fail, or na

required
data

custom data

{}
Source code in fw_gear/metadata.py
363
364
365
366
367
368
369
370
371
372
373
374
375
376
def add_qc_result_to_analysis(self, name: str, state: str, **data) -> None:
    """Add a pass/fail/na QC results to the destination analysis container via update .metadata.json.

    Args:
        name (str): QC result name
        state (str): pass, fail, or na
        data: custom data
    """
    qc_result = create_qc_result_dict(name, state, **data)
    if self._context.config.destination.get("type") != "analysis":
        raise TypeError("Destination container is not an analysis container.")
    analysis_container = self._context.config.get_destination_container()
    updated_file_info = self.add_gear_info("qc", analysis_container, **qc_result)  # type: ignore
    self.update_container("analysis", info=updated_file_info)

add_qc_result_via_sdk(cont_, name, state, **data)

Add a pass/fail/na QC results to a container/file and update the container/file info via HTTP methods.

Parameters:

Name Type Description Default
cont_ Any

Flywheel Container object

required
name str

QC result name

required
state str

pass, fail, or na

required
data

custom data

{}
Source code in fw_gear/metadata.py
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
def add_qc_result_via_sdk(self, cont_: t.Any, name: str, state: str, **data):
    """Add a pass/fail/na QC results to a container/file and update the container/file info via HTTP methods.

    Args:
        cont_ (t.Any): Flywheel Container object
        name (str): QC result name
        state (str): pass, fail, or na
        data: custom data

    """
    # generate qc result with provided information
    qc_result = create_qc_result_dict(name, state, **data)

    if isinstance(cont_, (dict, str)) or not hasattr(cont_, "files"):
        file_obj = get_file(cont_, self._context)
        updated_file_info = self.add_gear_info("qc", file_obj, **qc_result)
        self.modify_container_file_info(file_obj, **updated_file_info)
    else:
        updated_cont_info = self.add_gear_info("qc", cont_, **qc_result)
        self.modify_container_info(cont_.id, **updated_cont_info)

clean()

Clean metadata before writing.

Source code in fw_gear/metadata.py
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
def clean(self):
    """Clean metadata before writing."""
    self._metadata = convert_nan_in_dict(self._metadata)
    self._metadata = sanitize_periods(self._metadata)

    # Serialize to JSON string
    json_string = json.dumps(self._metadata, cls=MetadataEncoder)

    # Check size
    size = len(json_string.encode("utf-8"))
    if size > 16 * 1024 * 1024:  # 16MB limit
        warnings.warn(
            f"Metadata size exceeds 16MB: {size} bytes. Consider breaking metadata into smaller chunks.",
            UserWarning,
        )

    # Deserialize to prepare for writing (if needed)
    self._metadata = json.loads(json_string)

log()

Log representation of metadata.

Source code in fw_gear/metadata.py
300
301
302
303
304
305
306
def log(self):
    """Log representation of metadata."""
    rep = copy.deepcopy(self._metadata)
    log.info(
        ".metadata.json:\n%s",
        json.dumps(trim(rep), indent=2, cls=MetadataEncoder),
    )

modify_container_file_info(file_, **info)

Modify file info with HTTP requests.

Wrapper for modify_container_file_info SDK method call. Using set method (add on) to update.

Parameters:

Name Type Description Default
file_ File

File Object

required
info Dict[Any]

Info dictionary that will be used to update file info.

{}
Source code in fw_gear/metadata.py
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
def modify_container_file_info(
    self,
    file_: File,
    **info,
) -> None:
    """Modify file info with HTTP requests.

    Wrapper for modify_container_file_info SDK method call.
    Using set method (add on) to update.

    Args:
        file_ (File): File Object
        info (e.Dict[t.Any]): Info dictionary that will be used to update file info.
    """
    if self._context.client is None:
        raise AttributeError("FW Client was not initialized or set.")

    self._context.client.modify_container_file_info(
        file_.parent_id, file_.name, {"set": info}
    )

modify_container_info(cont_id, **info)

Update metadata for the given container type in the hierarchy.

Wrapper for modify_container_info SDK method call. Using set method (add on) to update.

Parameters:

Name Type Description Default
cont_id str

Container ID that will be used to update container info

required
info Dict[Any]

Info dictionary that will be used to update the container info.

{}
Source code in fw_gear/metadata.py
262
263
264
265
266
267
268
269
270
271
272
273
274
275
def modify_container_info(self, cont_id: str, **info) -> None:
    """Update metadata for the given container type in the hierarchy.

    Wrapper for modify_container_info SDK method call.
    Using set method (add on) to update.

    Args:
        cont_id (str): Container ID that will be used to update container info
        info (e.Dict[t.Any]): Info dictionary that will be used to update the container info.
    """
    if self._context.client is None:
        raise AttributeError("FW Client was not initialized or set.")

    self._context.client.modify_container_info(cont_id, {"set": info})

pull_job_info()

Pull job info from GearContext if present.

Get job input files, configuration, and job id.

Parameters:

Name Type Description Default
name str

Gear name override.

required
version str

Gear version override.

required
Source code in fw_gear/metadata.py
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
def pull_job_info(self):
    """Pull job info from GearContext if present.

    Get job input files, configuration, and job id.

    Args:
        name (str): Gear name override.
        version (str): Gear version override.
    """
    name = self.name_override
    version = self.version_override
    # Don't attempt to set gear info if context not passed in.
    self.job = ""
    if not self._context:
        log.info("Context not provided, adding gear info from name and version")
        info = {name: {"job_info": version}}
        self.job_info = info
        self.name = name
        self.version = version
        self._has_job_info = True
        return
    # Add gear information, configuration, inputs, etc.
    gear_inputs = {}
    for i_name, fe in self._context.config.inputs.items():
        if fe["base"] != "file":
            continue
        obj = fe.get("object", {})
        gear_inputs[i_name] = {
            "parent": fe.get("hierarchy"),
            "file_id": obj.get("file_id", ""),
            "version": obj.get("version", ""),
            "file_name": fe.get("location", {}).get("name", ""),
        }
    # First check for job provided in config
    if self._context.config.job:
        self.job = self._context.config.job.get("id", "")
    elif self._context.config.destination.get("type", "") == "analysis":  # noqa: PLR5501
        try:
            dest = self._context.config.get_destination_container()
            self.job = dest.get("job", {}).get("id", "")
        except AttributeError:
            # When client = None
            log.warning("Could not use SDK to look up job id for analysis")
    if not self.job:
        log.warning("Could not determine job id.")

    m = self._context.manifest
    # Handle the case when manifest is None
    if m is None:
        if not name or not version:
            log.debug(
                "Manifest is None and no name/version override provided. Using defaults."
            )
            name = name or "unknown-gear"
            version = version or "0.0.0"
    else:
        name = name if name else m.name
        version = version if version else m.version
    info = {
        name: {
            "job_info": {
                "version": version,
                "job_id": self.job,
                "inputs": gear_inputs,
                "config": self._context.config.opts,
            },
        }
    }
    self.job_info = info
    self.name = name
    self.version = version
    self._has_job_info = True

update_container(container_type, deep=True, **kwargs)

Update metadata for the given container type in the hierarchy.

Parameters:

Name Type Description Default
container_type str

The container type (e.g. session or acquisition).

required
deep bool

Perform a deep (recursive) update on subdictionaries. Default: True

True
**kwargs dict

Update arguments

{}
Source code in fw_gear/metadata.py
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
def update_container(
    self, container_type: str, deep: bool = True, **kwargs
) -> None:
    """Update metadata for the given container type in the hierarchy.

    Args:
        container_type (str): The container type (e.g. session or
            acquisition).
        deep (bool): Perform a deep (recursive) update on subdictionaries.
            Default: True
        **kwargs (dict): Update arguments
    """
    if not self._validate_container_type(container_type):
        raise ValueError(
            f"Container type {container_type} is outside the hierarchy that can be updated via .metadata.json."
        )
    dest = self._metadata.setdefault(container_type, {})
    if deep:
        deep_merge(dest, **kwargs)
    else:
        dest.update(**kwargs)

update_file_metadata(file_, container_type, deep=True, **kwargs)

Update file metadata by name.

Update a file by name in the hierarchy at the given container type.

Parameters:

Name Type Description Default
file_ Any

File name (str), SDK file (flywheel.FileEntry) or dictionary from config.json

required
container_type str

Type of parent container.

required
deep bool

Perform a deep (recursive) update on subdictionaries. Default: True

True
**kwargs dict

Update arguments

{}
Source code in fw_gear/metadata.py
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
def update_file_metadata(
    self,
    file_: t.Any,
    container_type: str,
    deep: bool = True,
    **kwargs,
) -> None:
    """Update file metadata by name.

    Update a file by name in the hierarchy at the given container type.

    Args:
        file_: File name (str), SDK file (flywheel.FileEntry) or
            dictionary from config.json
        container_type (str): Type of parent container.
        deep (bool): Perform a deep (recursive) update on subdictionaries.
            Default: True
        **kwargs (dict): Update arguments
    """
    if container_type != "analysis" and not self._validate_container_type(
        container_type
    ):
        raise ValueError(
            f"Container type {container_type} is outside the hierarchy that can be updated via .metadata.json."
        )

    # Find file by name in the given container
    file_obj = get_file(file_, self._context, container_type)
    parent = self._metadata.setdefault(file_obj.parent_type, {})
    files = parent.setdefault("files", [])
    file_entry = None
    for fe in files:
        if fe.get("name") == file_obj.name:
            file_entry = fe
            break
    # No metadata entry for given file exists yet.
    if file_entry is None:
        # Initialize with file name and any existing info if present
        file_entry = {"name": file_obj.name, "info": file_obj.info}
        # Add this file entry to the parent container's 'file' key
        files.append(file_entry)
    # Placeholder reference to location in the full dictionary we want
    # to update, i.e. the file entry
    target = file_entry
    # Update metadata with kwargs
    if deep:
        deep_merge(target, **kwargs)
    else:
        target.update(kwargs)

update_zip_member_count(path, container_type=None)

Update metadata with zip-member count.

Add the zip member count for files in a given directory the metadata

Parameters:

Name Type Description Default
path Path

directory or file.

required
container_type Optional[str]

Container type found files should be associated with.

None
Source code in fw_gear/metadata.py
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
def update_zip_member_count(
    self,
    path: Path,
    container_type: t.Optional[str] = None,
) -> None:
    """Update metadata with zip-member count.

    Add the zip member count for files in a given directory the metadata

    Args:
        path (Path): directory or file.
        container_type (Optional[str]): Container type found files should
            be associated with.
    """
    if container_type is None:
        # This will happen, e.g., if we are running a gear locally.
        log.warning("No container type provided, skipping zip member count update.")
        return
    if not path.exists():
        log.warning("Provided path does not exist: %s", path)
        return
    if path.is_file():
        files = iter([path])
    else:
        files = path.rglob("*")
    for file in files:
        if file.suffix != ".zip":
            continue
        try:
            with zipfile.ZipFile(file) as archive:
                count = len(archive.infolist())
                self.update_file_metadata(
                    file.name,
                    container_type=container_type,
                    zip_member_count=count,
                )
        except zipfile.BadZipFile:
            log.warning("Invalid zip file %s. Skipping update", file)

write(directory, fail_on_validation=False, log_meta=True)

Write metadata to .metadata.json in the given directory.

Parameters:

Name Type Description Default
directory Path

Directory in which to write .metadata.json

required
fail_on_validation bool

Fail if engine metadata schema validation has errors. Default: False

False
log_meta bool

Also log cleaned metadata. Default: True

True
Source code in fw_gear/metadata.py
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
def write(
    self, directory: Path, fail_on_validation: bool = False, log_meta: bool = True
) -> None:
    """Write metadata to .metadata.json in the given directory.

    Args:
        directory (Path): Directory in which to write .metadata.json
        fail_on_validation (bool): Fail if engine metadata schema
            validation has errors. Default: False
        log_meta (bool): Also log cleaned metadata. Default: True
    """
    if not self._metadata:
        return
    self.clean()
    log.debug("Validating generated metadata")
    with open(PARENT_DIR / "engine_metadata_schema.json") as fp:
        engine_metadata_schema = json.load(fp)
    validator = jsonschema.Draft7Validator(engine_metadata_schema)
    errors = []
    for err in validator.iter_errors(self._metadata):
        errors.append(err)
    if errors:
        log.error("Error(s) validating produced metadata.")
        for err in errors:
            log.error(err)
        if fail_on_validation:
            self.log()
            sys.exit(1)
    if log_meta:
        self.log()

    with open(directory / ".metadata.json", "w", encoding="utf-8") as f:
        json.dump(self._metadata, f, indent=2, cls=MetadataEncoder)

File

Module for class represent Flywheel file object.

File dataclass

Class representing a context agnostic Flywheel file.

The SDK and config.json represent files differently. This object holds all the metadata associated with a file in a predictable manner and allows for converting from SDK or config.json file representations.

Attributes:

Name Type Description
name str

File name.

parent_type str

Container type of file's parent.

modality str

File modality.

type str

Flywheel file type.

mimetype str

File mimetype.

classification Dict[str, List[str]]

Classification dictionary.

tags List[str]

List of tags.

info dict

Custom information dict.

_local_path Optional[Path]

Local path to file

parents Dict[str, str]

File parents.

zip_member_count Optional[int]

File zip member count.

version Optional[int]

File version.

file_id Optional[str]

File id.

size Optional[int]

File size in bytes.

fw_path(str) Optional[int]

Path to file in Flywheel.

Source code in fw_gear/file.py
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
@dataclass
class File:
    """Class representing a context agnostic Flywheel file.

    The SDK and config.json represent files differently. This object
    holds all the metadata associated with a file in a predictable manner
    and allows for converting from SDK or config.json file representations.

    Attributes:
        name (str): File name.
        parent_type (str): Container type of file's parent.
        modality (str): File modality.
        type (str): Flywheel file type.
        mimetype (str): File mimetype.
        classification (Dict[str, List[str]]): Classification dictionary.
        tags (List[str]): List of tags.
        info (dict): Custom information dict.
        _local_path (Optional[Path]): Local path to file
        parents (Dict[str, str]): File parents.
        zip_member_count (Optional[int]): File zip member count.
        version (Optional[int]): File version.
        file_id (Optional[str]): File id.
        size (Optional[int]): File size in bytes.
        fw_path(str): Path to file in Flywheel.
    """

    name: str
    parent_type: str
    modality: str = ""
    type: str = ""
    mimetype: str = ""
    classification: t.Dict[str, t.List[str]] = field(default_factory=dict)
    tags: t.List[str] = field(default_factory=list)
    info: dict = field(default_factory=dict)
    _local_path: t.Optional[Path] = None
    parents: t.Dict[str, str] = field(default_factory=dict)
    zip_member_count: t.Optional[int] = None
    version: t.Optional[int] = None
    file_id: t.Optional[str] = None
    size: t.Optional[int] = None
    parent_id: t.Optional[str] = None
    source: str = ""
    _client: t.Any = None
    _fw_path: t.Optional[str] = field(init=False, default=None)

    @classmethod
    def from_config(cls, file_: dict, context: t.Any) -> "File":
        """Returns a File object from a GearContext.

        Args:
            file_ (dict): Config.json dictionary representing the file.
        """
        # file_ passed in from config.json
        obj = file_.get("object", {})
        origin_file_id = obj.get("file_id", None)
        if not origin_file_id:
            raise ValueError("File ID not found in file object")

        parents_ = context.client.get_file(origin_file_id).get("parents", {})

        return cls(
            name=file_.get("location", {}).get("name", ""),
            parent_type=file_.get("hierarchy", {}).get("type", ""),
            parent_id=file_.get("hierarchy", {}).get("id", None),
            modality=obj.get("modality", ""),
            type=obj.get("type", ""),
            mimetype=obj.get("mimetype", ""),
            classification=obj.get("classification", {}),
            tags=obj.get("tags", []),
            info=obj.get("info", {}),
            _local_path=file_.get("location", {}).get("path", None),
            zip_member_count=obj.get("zip_member_count", None),
            version=obj.get("version", None),
            file_id=origin_file_id,
            size=obj.get("size", None),
            parents=parents_,
            source="gear_config",
            _client=context.client,
        )

    @classmethod
    def from_sdk(
        cls, file_: "flywheel.models.file_output.FileOutput", context: t.Any
    ) -> "File":
        """Returns a File from an SDK "file".

        Args:
            file_ (flywheel.models.file_output.FileOutput): SDK "file" object
        """
        parent_type = file_.get("parent_ref", {}).get("type", "")

        parents_ = file_.get("parents", {})

        return cls(
            name=file_.get("name", ""),
            parent_type=parent_type,
            modality=file_.get("modality", ""),
            type=file_.get("type", ""),
            mimetype=file_.get("mimetype", ""),
            classification=file_.get("classification", {}),
            tags=file_.get("tags", []),
            info=file_.get("info", {}),
            parents=parents_,
            zip_member_count=file_.get("zip_member_count", None),
            version=file_.get("version", None),
            file_id=file_.get("file_id", None),
            size=file_.get("size", None),
            parent_id=file_.get("parents", {}).get(parent_type, ""),
            source="sdk",
            _client=context.client,
        )

    @property
    def fw_path(self) -> str:
        """Lazy-load the Flywheel path to the file."""
        if self._fw_path is None:
            # Assuming you can construct the Flywheel path from existing attributes
            self._fw_path = self._construct_fw_path()
        return self._fw_path

    def _construct_fw_path(self) -> str:
        """Construct the Flywheel path to the file."""
        if not self.parents:
            log.warning("Parents information is missing, using default parents.")
            self.parents = {key: "unknown" for key in FLYWHEEL_HIERARCHY}
        if not all(key in self.parents for key in FLYWHEEL_HIERARCHY):
            log.warning(
                "Some hierarchy information is missing, using defaults where necessary."
            )
            for key in FLYWHEEL_HIERARCHY:
                self.parents.setdefault(key, "unknown")

        path_components = []

        for key in FLYWHEEL_HIERARCHY:
            cont_id = self.parents.get(key)

            if cont_id is not None:
                try:
                    cont_label = self._client.get(cont_id).label
                    if key == "analysis":
                        path_components.append("analyses")
                    path_components.append(cont_label)
                except Exception as e:
                    raise ValueError(f"Failed to get container label for {key}: {e}")
        path_components.extend(["files", self.name])
        # return path as list of strings
        return path_components

    @property
    def local_path(self) -> t.Optional[Path]:
        """Lazy-load the local path by downloading the file on first access."""
        if self._local_path is None:
            if self._client:
                file_cont = self._client.get_file(self.file_id)
                self._local_path = self.download_file_to_temp(file_cont)
            else:
                log.warning("Client not set, cannot download file.")
                self._local_path = None
        return self._local_path

    @staticmethod
    def download_file_to_temp(file_: "flywheel.models.file_output.FileOutput") -> Path:
        """Downloads a file to a temporary directory and returns the path.

        Args:
            file_ (flywheel.models.file_output.FileOutput): FW file object.

        Returns:
            Path: Local path to the downloaded file.
        """

        try:
            # Create a temporary directory
            temp_dir = tempfile.mkdtemp()

            # Download the file to the temporary directory
            local_file_path = os.path.join(temp_dir, file_.name)
            file_.download(local_file_path)

            return Path(local_file_path)
        except Exception as e:
            log.error(f"Failed to download file: {e}")
            raise

    def __del__(self):
        """Clean up the temporary file if the source is 'sdk'."""
        if self.source == "sdk" and self._local_path:
            try:
                if self._local_path.exists():
                    shutil.rmtree(self._local_path.parent)
            except Exception as e:
                log.error(f"Failed to delete temporary file: {e}")
fw_path property

Lazy-load the Flywheel path to the file.

local_path property

Lazy-load the local path by downloading the file on first access.

__del__()

Clean up the temporary file if the source is 'sdk'.

Source code in fw_gear/file.py
221
222
223
224
225
226
227
228
def __del__(self):
    """Clean up the temporary file if the source is 'sdk'."""
    if self.source == "sdk" and self._local_path:
        try:
            if self._local_path.exists():
                shutil.rmtree(self._local_path.parent)
        except Exception as e:
            log.error(f"Failed to delete temporary file: {e}")
download_file_to_temp(file_) staticmethod

Downloads a file to a temporary directory and returns the path.

Parameters:

Name Type Description Default
file_ FileOutput

FW file object.

required

Returns:

Name Type Description
Path Path

Local path to the downloaded file.

Source code in fw_gear/file.py
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
@staticmethod
def download_file_to_temp(file_: "flywheel.models.file_output.FileOutput") -> Path:
    """Downloads a file to a temporary directory and returns the path.

    Args:
        file_ (flywheel.models.file_output.FileOutput): FW file object.

    Returns:
        Path: Local path to the downloaded file.
    """

    try:
        # Create a temporary directory
        temp_dir = tempfile.mkdtemp()

        # Download the file to the temporary directory
        local_file_path = os.path.join(temp_dir, file_.name)
        file_.download(local_file_path)

        return Path(local_file_path)
    except Exception as e:
        log.error(f"Failed to download file: {e}")
        raise
from_config(file_, context) classmethod

Returns a File object from a GearContext.

Parameters:

Name Type Description Default
file_ dict

Config.json dictionary representing the file.

required
Source code in fw_gear/file.py
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
@classmethod
def from_config(cls, file_: dict, context: t.Any) -> "File":
    """Returns a File object from a GearContext.

    Args:
        file_ (dict): Config.json dictionary representing the file.
    """
    # file_ passed in from config.json
    obj = file_.get("object", {})
    origin_file_id = obj.get("file_id", None)
    if not origin_file_id:
        raise ValueError("File ID not found in file object")

    parents_ = context.client.get_file(origin_file_id).get("parents", {})

    return cls(
        name=file_.get("location", {}).get("name", ""),
        parent_type=file_.get("hierarchy", {}).get("type", ""),
        parent_id=file_.get("hierarchy", {}).get("id", None),
        modality=obj.get("modality", ""),
        type=obj.get("type", ""),
        mimetype=obj.get("mimetype", ""),
        classification=obj.get("classification", {}),
        tags=obj.get("tags", []),
        info=obj.get("info", {}),
        _local_path=file_.get("location", {}).get("path", None),
        zip_member_count=obj.get("zip_member_count", None),
        version=obj.get("version", None),
        file_id=origin_file_id,
        size=obj.get("size", None),
        parents=parents_,
        source="gear_config",
        _client=context.client,
    )
from_sdk(file_, context) classmethod

Returns a File from an SDK "file".

Parameters:

Name Type Description Default
file_ FileOutput

SDK "file" object

required
Source code in fw_gear/file.py
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
@classmethod
def from_sdk(
    cls, file_: "flywheel.models.file_output.FileOutput", context: t.Any
) -> "File":
    """Returns a File from an SDK "file".

    Args:
        file_ (flywheel.models.file_output.FileOutput): SDK "file" object
    """
    parent_type = file_.get("parent_ref", {}).get("type", "")

    parents_ = file_.get("parents", {})

    return cls(
        name=file_.get("name", ""),
        parent_type=parent_type,
        modality=file_.get("modality", ""),
        type=file_.get("type", ""),
        mimetype=file_.get("mimetype", ""),
        classification=file_.get("classification", {}),
        tags=file_.get("tags", []),
        info=file_.get("info", {}),
        parents=parents_,
        zip_member_count=file_.get("zip_member_count", None),
        version=file_.get("version", None),
        file_id=file_.get("file_id", None),
        size=file_.get("size", None),
        parent_id=file_.get("parents", {}).get(parent_type, ""),
        source="sdk",
        _client=context.client,
    )

Logging

Module for default logging configuration and helpers.

This module utilizes an optionally provided configuration dictionary to update default logging configuration.

Alternatives to updating the default logging templates can be >>> log_level = logging.INFO # logging.DEBUG, etc. >>> log_format = '[%(levelname)8s] %(message)s' >>> log_dtfmt = '' >>> logging.basicConfig(level=log_level, format=log_format, datefmt=log_dtfmt)

Or, >>> logging.config.dictConfig(config)

For a dictionary logging configuration, config, as below.

For more in-depth information on python logging, see https://docs.python.org/3.6/library/logging.html.

ErrFilter

Bases: Filter

Class with single boolean function for filtering error log records.

Source code in fw_gear/logging/__init__.py
51
52
53
54
55
56
57
58
class ErrFilter(logging.Filter):
    """Class with single boolean function for filtering error log records."""

    def __init__(self, param=None):  # noqa: D107
        self.param = param

    def filter(self, record):  # noqa: D102
        return record.levelno not in [logging.INFO, logging.WARNING]

OutFilter

Bases: Filter

Class with single boolean function for filtering non-error log records.

Source code in fw_gear/logging/__init__.py
41
42
43
44
45
46
47
48
class OutFilter(logging.Filter):
    """Class with single boolean function for filtering non-error log records."""

    def __init__(self, param=None):  # noqa: D107
        self.param = param

    def filter(self, record):  # noqa: D102
        return record.levelno in [logging.INFO, logging.WARNING]

configure_logging(default_config_name='info', update_config=None)

Configures logging with default settings or a dictionary of log settings.

Configures the logging for the gear-toolkit module using one of either two default json templates (info, debug -- see below). Whichever one is used can be updated with an optional update_config dictionary.

Logging sent to DEBUG, ERROR, and CRITICAL will be filtered and sent to stderr.

Logging sent to INFO and WARNING will be filtered and sent to stdout.

Parameters:

Name Type Description Default
default_config_name str

A string, 'info' or 'debug', indicating the default template to use. Defaults to 'info'.

'info'
update_config dict

A dictionary containing the keys, subkeys, and values of the templates to update. Defaults to None.

None
Example

.. code-block:: python

config = {
    "formatters": {
        "gear": {
            "format": "[TEST] %(levelname)4s %(message)s"
        }
    }
}
configure_logging(update_config = config)

will update the ['formatters']['gear']['format'] value in the info. Default configuration below.

Note

Default logging configuration dictionaries are stored in defaults/<prefix>.json.

info:

.. code-block:: json

{
    "version": 1,
    "filters": {
        "out_filter": {
            "()": "PlaceHolder",
            "param": "noshow"
        },
        "err_filter": {
            "()": "PlaceHolder",
            "param": "noshow"
        }
    },
    "formatters": {
        "gear": {
            "format":"[%(asctime)s - %(levelname)s - %(name)s] %(message)s",
            "datefmt": "%Y%m%d"
        }
    },
    "handlers": {
        "gear": {
            "level":"INFO",
            "formatter": "gear",
            "class":"logging.StreamHandler"
        }
    },
    "loggers": {
        "": {
            "handlers": ["gear"],
            "level":"INFO"
        }
    },
    "disable_existing_loggers": false
}

debug:

.. code-block:: json

{
    "version": 1,
    "filters": {
        "out_filter": {
            "()": "PlaceHolder",
            "param": "noshow"
        },
        "err_filter": {
            "()": "PlaceHolder",
            "param": "noshow"
        }
    },
    "formatters": {
        "gear": {
            "format":  "[%(asctime)s - %(levelname)s - %(name)s:%(lineno)d] %(message)s",
            "datefmt": "%Y%m%d"
        }
    },
    "handlers": {
        "gear": {
            "level":"DEBUG",
            "formatter": "gear",
            "class":"logging.StreamHandler"
        }
    },
    "loggers": {
        "": {
            "handlers": ["gear"],
            "level":"DEBUG"
        }
    },
    "disable_existing_loggers": false
}
Source code in fw_gear/logging/__init__.py
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
def configure_logging(default_config_name="info", update_config=None):
    """Configures logging with default settings or a dictionary of log settings.

    Configures the logging for the gear-toolkit module using one
    of either two default json templates (`info`, `debug` -- see below).
    Whichever one is used can be updated with an optional `update_config`
    dictionary.

    Logging sent to DEBUG, ERROR, and CRITICAL will be filtered and sent to stderr.

    Logging sent to INFO and WARNING will be filtered and sent to stdout.

    Args:
        default_config_name (str, optional): A string, 'info' or 'debug', indicating
            the default template to use. Defaults to 'info'.
        update_config (dict, optional): A dictionary containing the keys,
            subkeys, and values of the templates to update.
            Defaults to `None`.

    Example:
        .. code-block:: python

            config = {
                "formatters": {
                    "gear": {
                        "format": "[TEST] %(levelname)4s %(message)s"
                    }
                }
            }
            configure_logging(update_config = config)


        will update the ['formatters']['gear']['format'] value in the `info`.
        Default configuration below.

    Note:
        Default logging configuration dictionaries are stored in
        ``defaults/<prefix>.json``.

        `info`:

        .. code-block:: json

            {
                "version": 1,
                "filters": {
                    "out_filter": {
                        "()": "PlaceHolder",
                        "param": "noshow"
                    },
                    "err_filter": {
                        "()": "PlaceHolder",
                        "param": "noshow"
                    }
                },
                "formatters": {
                    "gear": {
                        "format":"[%(asctime)s - %(levelname)s - %(name)s] %(message)s",
                        "datefmt": "%Y%m%d"
                    }
                },
                "handlers": {
                    "gear": {
                        "level":"INFO",
                        "formatter": "gear",
                        "class":"logging.StreamHandler"
                    }
                },
                "loggers": {
                    "": {
                        "handlers": ["gear"],
                        "level":"INFO"
                    }
                },
                "disable_existing_loggers": false
            }

        `debug`:

        .. code-block:: json

            {
                "version": 1,
                "filters": {
                    "out_filter": {
                        "()": "PlaceHolder",
                        "param": "noshow"
                    },
                    "err_filter": {
                        "()": "PlaceHolder",
                        "param": "noshow"
                    }
                },
                "formatters": {
                    "gear": {
                        "format":  "[%(asctime)s - %(levelname)s - %(name)s:%(lineno)d] %(message)s",
                        "datefmt": "%Y%m%d"
                    }
                },
                "handlers": {
                    "gear": {
                        "level":"DEBUG",
                        "formatter": "gear",
                        "class":"logging.StreamHandler"
                    }
                },
                "loggers": {
                    "": {
                        "handlers": ["gear"],
                        "level":"DEBUG"
                    }
                },
                "disable_existing_loggers": false
            }
    """
    available_names = ["info", "debug"]

    if default_config_name not in available_names:
        log.warning(
            "The value of %s is not in %s. Reverting to 'info'.",
            default_config_name,
            available_names,
        )
        default_config_name = "info"

    json_file = PARENT_DIR / f"defaults/{default_config_name}.json"

    with open(json_file, "r") as f:
        config = json.load(f)

    if update_config:
        config = _recursive_update(config, update_config)
    if config.get("filters"):
        if config["filters"].get("out_filter"):
            config["filters"]["out_filter"]["()"] = OutFilter
        if config["filters"].get("err_filter"):
            config["filters"]["err_filter"]["()"] = ErrFilter

    logging.config.dictConfig(config)

    log.info("Log level is %s", logging.getLevelName(logging.root.level))

Invocation Schema

Flywheel Gear Invocation Schema Generator.

derive_invocation_schema(manifest)

Creates an invocation schema from a gear manifest.

This can be used to validate the files and configuration offered to run a gear.

Source code in fw_gear/invocation_schema.py
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
def derive_invocation_schema(manifest):
    """Creates an invocation schema from a gear manifest.

    This can be used to validate the files and configuration offered to run a gear.
    """
    validate_manifest(manifest)

    result = {
        "title": "Invocation manifest for " + manifest["label"],
        "$schema": "http://json-schema.org/draft-04/schema#",
        "type": "object",
        "properties": {
            "config": {
                "type": "object",
                "properties": {},
                "required": [],
                "additionalProperties": False,
            },
            "inputs": {"type": "object", "properties": {}, "required": []},
        },
        "required": ["config", "inputs"],
    }

    # Copy over constraints from manifest
    for kind in ["config", "inputs"]:
        for key in manifest[kind]:
            # Copy constraints, removing 'base' and 'description' keywords which are not constraints
            value = copy.deepcopy(manifest[kind][key])
            value.pop("base", None)
            value.pop("description", None)
            optional = value.pop("optional", False)

            # The config map holds scalars, while the inputs map holds objects.
            if kind == "config":
                result["properties"][kind]["properties"][key] = value
            else:
                keyType = manifest[kind][key]["base"]
                spec = {}

                if keyType == "file":
                    # Object with any particular properties suggested from the manifest
                    # Does not validate the upstream file object schema; that is left to other tools.
                    spec = {
                        "type": "object",
                        "properties": value,  # copy over schema snippet from manifest
                    }

                elif keyType == "api-key":
                    # There is currently only an implicit declaration of how api-key type inputs are provisioned.
                    # For now, declare an object. Should be improved later.
                    spec = {"type": "object"}

                elif keyType == "context":
                    # Object with information about a lookup value
                    spec = {
                        "type": "object",
                        "properties": {
                            "base": {
                                "type": "string",
                            },
                            "found": {
                                "type": "boolean",
                            },
                            "value": {},  # Context inputs can have any type, or none at all
                        },
                        "required": ["base", "found", "value"],
                    }
                else:
                    # Whitelist input types
                    raise Exception("Unknown input type " + str(keyType))

                # Save into result
                result["properties"][kind]["properties"][key] = spec

            # Require the key be present unless optional flag is set.
            if not optional:
                result["properties"][kind]["required"].append(key)

        # After handling each key, remove required array if none are present.
        # Required by jsonschema (minItems 1).
        if len(result["properties"][kind]["required"]) == 0:
            result["properties"][kind].pop("required", None)

    # Important: check our work - the result must be a valid schema.
    Draft4Validator.check_schema(result)
    return result

format_json(d)

Convenience method to pretty-print JSON.

Source code in fw_gear/invocation_schema.py
22
23
24
def format_json(d):
    """Convenience method to pretty-print JSON."""
    return json.dumps(d, sort_keys=True, indent=4, separators=(",", ": "))

get_master_gear_schema()

Return gear spec schema.

Source code in fw_gear/invocation_schema.py
44
45
46
47
48
49
50
51
52
53
def get_master_gear_schema():
    """Return gear spec schema."""

    global master_gear_schema  # noqa: PLW0603

    if master_gear_schema is None:
        master_gear_schema_path = PROJ_DIR / "gear_spec_schema.json"
        master_gear_schema = load_json_from_file(master_gear_schema_path)

    return master_gear_schema

get_master_schema()

Returns the master schema. This will load it from disk and cache in memory.

Source code in fw_gear/invocation_schema.py
33
34
35
36
37
38
39
40
41
def get_master_schema():
    """Returns the master schema. This will load it from disk and cache in memory."""
    global master_schema  # noqa: PLW0603

    if master_schema is None:
        master_schema_path = PROJ_DIR / "manifest.schema.json"
        master_schema = load_json_from_file(master_schema_path)

    return master_schema

isolate_config_invocation(invocation)

Given an invocation schema, isolate just the config portion.

Useful to validate configuration options separately from files.

Source code in fw_gear/invocation_schema.py
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
def isolate_config_invocation(invocation):
    """Given an invocation schema, isolate just the config portion.

    Useful to validate configuration options separately from files.
    """
    inv = copy.deepcopy(invocation)
    fis = inv["properties"]["config"]

    fis["title"] = "Config invocation manifest"
    fis["$schema"] = "http://json-schema.org/draft-04/schema#"
    fis["type"] = "object"

    # Important: check our work - the result must be a valid schema.
    Draft4Validator.check_schema(fis)
    return fis

isolate_file_invocation(invocation, input_name)

Given an invocation schema, isolate just a specific file.

Useful to validate a single input.

Source code in fw_gear/invocation_schema.py
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
def isolate_file_invocation(invocation, input_name):
    """Given an invocation schema, isolate just a specific file.

    Useful to validate a single input.
    """
    inv = copy.deepcopy(invocation)
    fis = inv["properties"]["inputs"]["properties"][input_name]

    fis["title"] = "Input invocation manifest for " + input_name
    fis["$schema"] = "http://json-schema.org/draft-04/schema#"
    fis["type"] = "object"

    # Important: check our work - the result must be a valid schema.
    Draft4Validator.check_schema(fis)
    return fis

load_json_from_file(path)

Loads a JSON file from disk and returns the parsed map.

Source code in fw_gear/invocation_schema.py
27
28
29
30
def load_json_from_file(path):
    """Loads a JSON file from disk and returns the parsed map."""
    with open(path) as fp:
        return json.load(fp)

validate_invocation(manifest, invocation)

Validates a configuration against a gear's manifest.

Expect a jsonschema.exceptions.ValidationError error to be raised on failure.

Source code in fw_gear/invocation_schema.py
187
188
189
190
191
192
193
194
def validate_invocation(manifest, invocation):
    """Validates a configuration against a gear's manifest.

    Expect a jsonschema.exceptions.ValidationError error to be raised on failure.
    """
    inv_schema = derive_invocation_schema(manifest)
    jsonschema.validate(invocation, inv_schema)
    return inv_schema

validate_manifest(manifest)

Validates a gear manifest against the master schema.

Expect a jsonschema.exceptions.ValidationError error to be raised on failure.

Source code in fw_gear/invocation_schema.py
56
57
58
59
60
61
62
def validate_manifest(manifest):
    """Validates a gear manifest against the master schema.

    Expect a jsonschema.exceptions.ValidationError error to be raised on failure.
    """
    schema = get_master_schema()
    jsonschema.validate(manifest, schema)

Utils Module

Core Utilities

Utility functions for fw-gears, including data processing and SDK helpers.

convert_nan_in_dict(data)

Convert NaN values in a dictionary to None.

Source code in fw_gear/utils/utils_helpers.py
41
42
43
44
def convert_nan_in_dict(data: dict) -> dict:  # noqa: D103
    """Convert NaN values in a dictionary to None."""
    # Note: convert_nan_in_dict is borrowed from core-api
    return {key: _convert_nan(value) for key, value in data.items()}

deep_merge(base, **update)

Recursive merging of update dict on base dict.

Instead of updating only top-level keys, deep_merge recurse down to perform a "nested" update.

Source code in fw_gear/utils/utils_helpers.py
47
48
49
50
51
52
53
54
55
56
57
def deep_merge(base, **update):
    """Recursive merging of `update` dict on `base` dict.

    Instead of updating only top-level keys, `deep_merge` recurse down to
    perform a "nested" update.
    """
    for k, v in update.items():
        if k in base and isinstance(base[k], dict) and isinstance(v, dict):
            deep_merge(base[k], **v)
        else:
            base[k] = v

trim(obj)

Trim object for printing.

Source code in fw_gear/utils/utils_helpers.py
60
61
62
def trim(obj: dict):
    """Trim object for printing."""
    return {key: trim_lists(val) for key, val in obj.items()}

trim_lists(obj)

Replace a long list with a representation.

List/Arrays greater than 5 in length will be replaced with the first two items followed by ... then the last two items

Source code in fw_gear/utils/utils_helpers.py
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
def trim_lists(obj: t.Any):
    """Replace a long list with a representation.

    List/Arrays greater than 5 in length will be replaced with the first two
    items followed by `...` then the last two items
    """
    if isinstance(obj, (list, tuple)):
        # Trim list
        if len(obj) > 5:
            return [*obj[:1], f"...{len(obj) - 2} more items...", *obj[-1:]]
        # Recurse into lists
        return [trim_lists(v) for v in obj]
    # Recurse into dictionaries
    if isinstance(obj, dict):
        return {key: trim_lists(val) for key, val in obj.items()}
    return obj

install_requirements(req_file)

Install requirements from a file programmatically

Parameters:

Name Type Description Default
req_file str

Path to requirements file

required

Raises:

Type Description
SystemExit

If there was an error from pip

Source code in fw_gear/utils/utils_helpers.py
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
def install_requirements(req_file):
    """Install requirements from a file programmatically

    Args:
        req_file (str): Path to requirements file

    Raises:
        SystemExit: If there was an error from pip
    """
    try:
        req_path = validate_requirements_file(req_file)
        _run_pip_install(req_path)
    except (ValueError, subprocess.CalledProcessError) as e:
        log.error(f"Requirement installation failed: {e}")
        sys.exit(1)

SDK Helpers

get_container_from_ref(client, ref)

Returns the container from its reference.

Parameters:

Name Type Description Default
client Client

Authenticated Flywheel SDK client

required
ref dict

A dictionary with type and id keys defined.

required

Returns:

Name Type Description
Container

A flywheel container.

Source code in fw_gear/utils/sdk_helpers.py
13
14
15
16
17
18
19
20
21
22
23
24
25
def get_container_from_ref(client: "flywheel.Client", ref: dict):
    """Returns the container from its reference.

    Args:
        client: Authenticated Flywheel SDK client
        ref: A dictionary with type and id keys defined.

    Returns:
        Container: A flywheel container.
    """
    container_type = ref.get("type")
    getter = getattr(client, f"get_{container_type}")
    return getter(ref.get("id"))

get_parent(client, container)

Returns parent container of container input.

Source code in fw_gear/utils/sdk_helpers.py
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
def get_parent(client: "flywheel.Client", container):
    """Returns parent container of container input."""
    if container.container_type == "analysis":
        return get_container_from_ref(client, container.parent)
    elif container.container_type == "file":
        return container.parent
    elif container.container_type == "group":
        raise TypeError("Group container does not have a parent.")
    else:
        for cont_type in BOTTOM_UP_PARENT_HIERARCHY:
            if not container.parents.get(cont_type):
                # not defined, parent must be up the hierarchy
                continue
            return get_container_from_ref(
                client, {"type": cont_type, "id": container.parents.get(cont_type)}
            )

setup_gear_run(client, gear_name, gear_args)

Setup gear run for a specified gear with provided gear arguments.

Parameters:

Name Type Description Default
client Client

Authenticated Flywheel SDK client.

required
gear_name str

Name of the gear to run.

required
gear_args dict

Dictionary of gear inputs and configuration.

required

Raises:

Type Description
ValueError

If the specified gear does not exist.

ValueError

If a required input / configuration for the gear is missing.

Returns:

Name Type Description
Tuple Tuple[GearDoc, dict, dict]

A tuple containing an object of the gear document, input dictionary, and configuration dictionary.

Source code in fw_gear/utils/sdk_helpers.py
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
def setup_gear_run(
    client: "flywheel.Client", gear_name: str, gear_args: dict
) -> t.Tuple["flywheel.GearDoc", dict, dict]:
    """Setup gear run for a specified gear with provided gear arguments.

    Args:
        client (flywheel.Client): Authenticated Flywheel SDK client.
        gear_name (str): Name of the gear to run.
        gear_args (dict): Dictionary of gear inputs and configuration.

    Raises:
        ValueError: If the specified gear does not exist.
        ValueError: If a required input / configuration for the gear is missing.

    Returns:
        Tuple: A tuple containing an object of the gear document, input dictionary, and configuration dictionary.
    """

    geardoc = client.gears.find_first(f"gear.name={gear_name}")
    if geardoc is None:
        raise ValueError(f"Gear {gear_name} does not exist.")

    # Gear Input Setup
    input_args_template = geardoc.gear.get("inputs").copy()
    input_args_template.pop("api-key", None)

    input_dict = {}

    for k, v in gear_args.items():
        if k in input_args_template:
            input_dict[k] = v

    for input_key, val in input_args_template.items():
        if not val["optional"] and input_key not in input_dict:
            raise ValueError(f"Missing required input for {gear_name}: {input_key}.")

    # Gear Configuration Setup
    geardoc_config = geardoc.gear.get("config")

    config_dict = {}
    for config_key, key_info in geardoc_config.items():
        config_default_val = key_info.get("default")
        config_dict[config_key] = (
            gear_args[config_key]
            if config_key in gear_args.keys()
            else config_default_val
        )
        if "optional" in key_info.keys() and not key_info.get("optional"):
            # Check required config is not empty or None
            if not config_dict[config_key]:
                raise ValueError(
                    f"{gear_name}'s {config_key} config should be provided prior to running."
                )

    return geardoc, input_dict, config_dict

Context Utilities

report_open_fds(sockets_only)

Decorator which reports on number of open file descriptors before and after a function.

Parameters:

Name Type Description Default
sockets_only bool

Report on only sockets, or all open FDs.

required
Source code in fw_gear/utils/contextutils.py
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
def report_open_fds(sockets_only):
    """Decorator which reports on number of open file
        descriptors before and after a function.

    Args:
        sockets_only (bool): Report on only sockets, or all open FDs.
    """

    def wrap(func):
        # Report on open sockets
        def wrapper(*args, **kwargs):
            # Get number of open fds before function
            pid = os.getpid()
            p_args = ["lsof", "-a", "-p", str(pid)]
            if sockets_only:
                p_args.insert(1, "-i")
            ps = subprocess.Popen(p_args, stdout=subprocess.PIPE)
            before = (
                subprocess.check_output(("wc", "-l"), stdin=ps.stdout)
                .decode("utf-8")
                .strip("\r\n")
            )
            func(*args, **kwargs)
            ps = subprocess.Popen(p_args, stdout=subprocess.PIPE)
            after = (
                subprocess.check_output(("wc", "-l"), stdin=ps.stdout)
                .decode("utf-8")
                .strip("\r\n")
            )

            log.debug(
                f"Number of fds ({func.__name__}) before: {before}, after: {after}"
            )

        return wrapper

    return wrap

report_usage_stats(interval=1.0, save_output=False, plot=True, save_dir='/flywheel/v0/output', cpu_usage=True, disk_usage=True, mem_usage=True, net_usage=True, disk_monitor_dirs=None, output_format=None)

Decorator to report on resource usage during a function.

Optional configuration to:

  • Change the update interval,
  • Turn off certain usage measurements
  • Adjust outputs save location
  • Adjust output types
  • Adjust which disk locations are monitored for disk usage.
Source code in fw_gear/utils/contextutils.py
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
def report_usage_stats(
    interval: float = 1.0,
    save_output: bool = False,
    plot: bool = True,
    save_dir: str = "/flywheel/v0/output",
    cpu_usage: bool = True,
    disk_usage: bool = True,
    mem_usage: bool = True,
    net_usage: bool = True,
    disk_monitor_dirs: t.Optional[t.List[str]] = None,
    output_format: t.Optional[str] = None,
) -> t.Callable:
    """Decorator to report on resource usage during a function.

    Optional configuration to:

    * Change the update interval,
    * Turn off certain usage measurements
    * Adjust outputs save location
    * Adjust output types
    * Adjust which disk locations are monitored for disk usage.

    """

    def wrap(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            # spawn subprocess that tracks cpu usage

            save_file = Path(save_dir) / f"{str(func.__name__)}"
            nonlocal disk_monitor_dirs
            if not disk_monitor_dirs:
                disk_monitor_dirs = ["/"]
            process, finished, path = _setup_usage_stats_worker(
                interval,
                cpu_usage,
                disk_usage,
                disk_monitor_dirs,
                mem_usage,
                net_usage,
            )
            func(*args, **kwargs)
            _finish_usage_stats_worker(
                process, finished, path, save_file, plot, output_format
            )

        return wrapper

    return wrap

sdk_delete_404_handler(client)

Ignore 404 errors on retries of deletes.

Source code in fw_gear/utils/contextutils.py
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
@contextmanager
def sdk_delete_404_handler(client):
    """Ignore 404 errors on retries of deletes."""
    orig_resp_hook = client.api_client.rest_client.session.hooks["response"]
    try:

        def ignore_404(self, resp, *args, **kwargs):
            if resp.status_code == 404:
                log.debug("Ignoring 404 response on {req.method} {req.url}")
            else:
                orig_resp_hook(self, resp, *args, *kwargs)

        client.api_client.rest_client.session.hooks["response"] = ignore_404
        yield
    finally:
        client.api_client.rest_client.session.hooks["response"] = orig_resp_hook

sdk_post_retry_handler(client)

Patch the SDK session object to retry on specific errors

Safe to use on
  • updating container info
  • updating file classification
  • adding notes
  • adding tags
Not safe to use on
  • Creating containers
  • Uploading files
Source code in fw_gear/utils/contextutils.py
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
@contextmanager
def sdk_post_retry_handler(client):  # pragma: no cover  # noqa: PLC0415
    """Patch the SDK session object to retry on specific errors

    Safe to use on:
        - updating container info
        - updating file classification
        - adding notes
        - adding tags

    Not safe to use on:
        - Creating containers
        - Uploading files
    """
    if not requests or not urllib3:
        raise ImportError("requests and urllib3 are required for SDK retry handler")

    orig_adapter = client.api_client.rest_client.session.adapters["https://"]
    try:
        BACKOFF_FACTOR = float(os.getenv("FLYWHEEL_SDK_BACKOFF_FACTOR", 0.5))  # noqa: PLW1508
        retry = urllib3.util.Retry(
            total=5,
            backoff_factor=BACKOFF_FACTOR,
            allowed_methods=["DELETE", "GET", "HEAD", "POST", "PUT", "OPTIONS"],
            status_forcelist=[429, 500, 502, 503, 504],
        )
        adapter = requests.adapters.HTTPAdapter(max_retries=retry)
        client.api_client.rest_client.session.mount("http://", adapter)
        client.api_client.rest_client.session.mount("https://", adapter)
        yield
    finally:
        client.api_client.rest_client.session.mount("https://", orig_adapter)
        client.api_client.rest_client.session.mount("http://", orig_adapter)

Archive/ZIP Management

Modules for zip manager utilities.

get_config_from_zip(zipfile_path, search_str='_config\\.json')

Extracts and returns configuration data from a JSON file within a ZIP archive.

This function searches for a file within zipfile_path whose name matches the search_str pattern and returns its contents as a Python dictionary.

Parameters:

Name Type Description Default
zipfile_path str

Absolute path to the ZIP file to be searched.

required
search_str str

Regular expression pattern for identifying the target JSON file. Defaults to '_config.json'.

'_config\\.json'

Returns:

Type Description
Optional[Dict]

dict or None: A dictionary containing the extracted JSON data if found,

Optional[Dict]

otherwise None.

Example

config = get_config_from_zip('/flywheel/v0/inputs/ZIP/file.zip')

Source code in fw_gear/utils/archive/zip_manager.py
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
def get_config_from_zip(
    zipfile_path: str, search_str: str = r"_config\.json"
) -> t.Optional[t.Dict]:
    """Extracts and returns configuration data from a JSON file within a ZIP archive.

    This function searches for a file within `zipfile_path` whose name matches
    the `search_str` pattern and returns its contents as a Python dictionary.

    Args:
        zipfile_path (str): Absolute path to the ZIP file to be searched.
        search_str (str, optional): Regular expression pattern for identifying
            the target JSON file. Defaults to '_config.json'.

    Returns:
        dict or None: A dictionary containing the extracted JSON data if found,
        otherwise None.

    Example:
        >>> config = get_config_from_zip('/flywheel/v0/inputs/ZIP/file.zip')
    """

    config = {}
    zf = ZipFile(zipfile_path)
    for fl in zf.filelist:
        if fl.filename[-1] != os.path.sep:  # not (fl.is_dir()):
            # if search_str in filename
            if re.search(search_str, fl.filename):
                json_str = zf.read(fl.filename).decode()
                config = json.loads(json_str)

                # This corrects for leaving the initial "config" key out
                # of previous gear versions without error
                if "config" not in config.keys():
                    config = {"config": config}

    if not config:
        log.warning("Configuration file is empty or not found.")
        return None

    return config

unzip_archive(zipfile_path, output_dir, dry_run=False)

Extracts the contents of a ZIP archive to a specified directory.

This function extracts the contents of zipfile_path into output_dir. Extraction is performed only if dry_run is set to False.

Parameters:

Name Type Description Default
zipfile_path str

Absolute path to the ZIP file.

required
output_dir str

Absolute path to the directory where the extracted files will be placed.

required
dry_run bool

If True, the extraction is skipped, allowing for debugging without modifying files.

False

Examples:

>>> unzip_archive('/flywheel/v0/inputs/ZIP/file.zip', '/flywheel/v0/work/')
>>> unzip_archive('/flywheel/v0/inputs/ZIP/file.zip', '/flywheel/v0/work/', dry_run=True)
Source code in fw_gear/utils/archive/zip_manager.py
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
def unzip_archive(zipfile_path: str, output_dir: str, dry_run: bool = False) -> None:
    """Extracts the contents of a ZIP archive to a specified directory.

    This function extracts the contents of `zipfile_path` into `output_dir`.
    Extraction is performed only if `dry_run` is set to False.

    Args:
        zipfile_path (str): Absolute path to the ZIP file.
        output_dir (str): Absolute path to the directory where the extracted
            files will be placed.
        dry_run (bool, optional): If True, the extraction is skipped, allowing
            for debugging without modifying files.

    Examples:
        >>> unzip_archive('/flywheel/v0/inputs/ZIP/file.zip', '/flywheel/v0/work/')
        >>> unzip_archive('/flywheel/v0/inputs/ZIP/file.zip', '/flywheel/v0/work/', dry_run=True)

    """
    input_zip = ZipFile(zipfile_path, "r")
    log.info(f"Unzipping file, {zipfile_path}")

    if not dry_run:
        input_zip.extractall(output_dir)

zip_info(zipfile_path)

Retrieves a list of relative file paths stored in a ZIP archive.

Parameters:

Name Type Description Default
zipfile_path str

Absolute path to the ZIP archive.

required

Returns:

Type Description
List[str]

List[str]: A sorted list of relative file paths contained in the ZIP archive.

Example

file_info = zip_info('/path/to/zipfile.zip')

Source code in fw_gear/utils/archive/zip_manager.py
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
def zip_info(zipfile_path: str) -> t.List[str]:
    """
    Retrieves a list of relative file paths stored in a ZIP archive.

    Args:
        zipfile_path (str): Absolute path to the ZIP archive.

    Returns:
        List[str]: A sorted list of relative file paths contained in the ZIP archive.

    Example:
        >>> file_info = zip_info('/path/to/zipfile.zip')
    """
    return sorted(
        filter(
            lambda x: len(x) > 0,
            [
                x.filename if x.filename[-1] != os.path.sep else ""
                for x in ZipFile(zipfile_path).filelist
            ],
        )
    )

zip_output(root_dir, source_dir, output_zip_filename, dry_run=False, exclude_files=None)

Compresses a directory into a ZIP archive.

This function creates a ZIP file containing the contents of source_dir relative to root_dir. The resulting ZIP archive is saved at output_zip_filename.

Parameters:

Name Type Description Default
root_dir str

The root directory to zip relative to.

required
source_dir str

Subdirectory within root_dir to be compressed.

required
output_zip_filename str

Full path of the output ZIP archive.

required
dry_run bool

If True, skips actual compression for debugging.

False
exclude_files Optional[List[str]]

List of file paths to exclude from the ZIP archive. Defaults to None.

None

Raises:

Type Description
FileNotFoundError

If root_dir does not exist.

Examples:

>>> zip_output('/flywheel/v0/work', 'gear_output', 'output.zip')
>>> zip_output('/flywheel/v0/work', 'gear_output', 'output.zip', dry_run=True)

.. code-block:: python

zip_output(
    '/flywheel/v0/work', 'gear_output', 'output.zip',
    exclude_files=['sub_dir1/file1.txt', 'sub_dir2/file2.txt']
)
Source code in fw_gear/utils/archive/zip_manager.py
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
def zip_output(
    root_dir: str,
    source_dir: str,
    output_zip_filename: str,
    dry_run: bool = False,
    exclude_files: t.Optional[t.List[str]] = None,
) -> None:
    """
    Compresses a directory into a ZIP archive.

    This function creates a ZIP file containing the contents of `source_dir`
    relative to `root_dir`. The resulting ZIP archive is saved at `output_zip_filename`.

    Args:
        root_dir (str): The root directory to zip relative to.
        source_dir (str): Subdirectory within `root_dir` to be compressed.
        output_zip_filename (str): Full path of the output ZIP archive.
        dry_run (bool, optional): If True, skips actual compression for debugging.
        exclude_files (Optional[List[str]], optional): List of file paths to exclude
            from the ZIP archive. Defaults to None.

    Raises:
        FileNotFoundError: If `root_dir` does not exist.

    Examples:
        >>> zip_output('/flywheel/v0/work', 'gear_output', 'output.zip')
        >>> zip_output('/flywheel/v0/work', 'gear_output', 'output.zip', dry_run=True)

        .. code-block:: python

            zip_output(
                '/flywheel/v0/work', 'gear_output', 'output.zip',
                exclude_files=['sub_dir1/file1.txt', 'sub_dir2/file2.txt']
            )
    """
    exclude_from_output = exclude_files if exclude_files else []

    if not op.exists(root_dir):
        raise FileNotFoundError(f"The directory '{root_dir}' does not exist.")

    log.info(f"Zipping output file: {output_zip_filename}")

    if not dry_run:
        current_directory = os.getcwd()
        os.chdir(root_dir)
        try:
            os.remove(output_zip_filename)
        except FileNotFoundError:
            pass

        with ZipFile(output_zip_filename, "w", ZIP_DEFLATED) as outzip:
            for root, subdirs, files in os.walk(source_dir):
                for fl in files + subdirs:
                    fl_path = op.join(root, fl)
                    if fl_path not in exclude_from_output:
                        outzip.write(fl_path)

        os.chdir(current_directory)

FreeSurfer License

Utilities for installing the Freesurfer license file in the expected location for algorithm execution.

check_project_for_license(context)

Retrieves the FreeSurfer license text from a Flywheel project metadata.

This function checks the project's info field for a license key under either FREESURFER-LICENSE or FREESURFER_LICENSE.

Parameters:

Name Type Description Default
context GearContext

The gear context containing configuration and inputs.

required

Returns:

Type Description
Optional[str]

Optional[str]: The extracted license text if found, otherwise None.

Source code in fw_gear/utils/licenses/freesurfer.py
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
def check_project_for_license(
    context: GearContext,
) -> t.Optional[str]:
    """
    Retrieves the FreeSurfer license text from a Flywheel project metadata.

    This function checks the project's `info` field for a license key under
    either `FREESURFER-LICENSE` or `FREESURFER_LICENSE`.

    Args:
        context (fw_gear.GearContext): The gear context containing configuration and inputs.

    Returns:
        Optional[str]: The extracted license text if found, otherwise None.

    """
    fly = context.client
    destination_id = context.config.destination.get("id")
    project_id = fly.get_analysis(destination_id)["parents"]["project"]
    project = fly.get_project(project_id)
    if any(
        lic in ("FREESURFER-LICENSE", "FREESURFER_LICENSE") for lic in project["info"]
    ):
        try:
            space_separated_text = project["info"]["FREESURFER-LICENSE"]
        except KeyError:
            space_separated_text = project["info"]["FREESURFER_LICENSE"]
        license_info = "\n".join(space_separated_text.split())
        log.info("Using FreeSurfer license in project info.")
        return license_info

find_license_info(context)

Retrieves the FreeSurfer license text from available sources.

It checks for the license in the following order: 1) Input file (freesurfer_license_file). 2) Config parameter (freesurfer_license_key). 3) Flywheel project metadata (FREESURFER_LICENSE).

Parameters:

Name Type Description Default
context GearContext

The gear context containing configuration and inputs.

required

Returns:

Type Description
Optional[str]

Optional[str]: The extracted license text, or None if not found.

Source code in fw_gear/utils/licenses/freesurfer.py
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
def find_license_info(context: GearContext) -> t.Optional[str]:
    """
    Retrieves the FreeSurfer license text from available sources.

    It checks for the license in the following order:
    1) Input file (`freesurfer_license_file`).
    2) Config parameter (`freesurfer_license_key`).
    3) Flywheel project metadata (`FREESURFER_LICENSE`).

    Args:
        context (fw_gear.GearContext): The gear context containing configuration and inputs.

    Returns:
        Optional[str]: The extracted license text, or None if not found.
    """
    license_info = ""
    config_key = isolate_key_name(context)
    if context.config.get_input_path("freesurfer_license_file"):
        license_info = read_input_license(context)
        log.info("Using input file for FreeSurfer license information.")
    elif config_key:
        fs_arg = context.config.opts[config_key]
        license_info = "\n".join(fs_arg.split())
        log.info("Using FreeSurfer license in gear configuration argument.")
    else:
        license_info = check_project_for_license(context)
    return license_info

get_fs_license_path(fs_license_path=None)

Determines the path where the FreeSurfer license should be installed.

If a path is provided, ensures it ends with 'license.txt'. Otherwise, it attempts to infer the path from the $FREESURFER_HOME environment variable or by locating the FreeSurfer installation.

Parameters:

Name Type Description Default
fs_license_path Optional[PathLike]

Custom path for the license file. Defaults to None.

None

Returns:

Name Type Description
Path Path

The resolved path for the FreeSurfer license file.

Raises:

Type Description
RuntimeError

If the FreeSurfer installation path cannot be determined.

Source code in fw_gear/utils/licenses/freesurfer.py
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
def get_fs_license_path(fs_license_path: t.Optional[os.PathLike] = None) -> Path:
    """
    Determines the path where the FreeSurfer license should be installed.

    If a path is provided, ensures it ends with 'license.txt'. Otherwise, it
    attempts to infer the path from the `$FREESURFER_HOME` environment variable
    or by locating the FreeSurfer installation.

    Args:
        fs_license_path (Optional[os.PathLike]): Custom path for the license file. Defaults to None.

    Returns:
        Path: The resolved path for the FreeSurfer license file.

    Raises:
        RuntimeError: If the FreeSurfer installation path cannot be determined.
    """
    if fs_license_path:
        if Path(fs_license_path).suffix != ".txt":
            fs_license_path = Path(fs_license_path) / "license.txt"
    elif os.getenv("FREESURFER_HOME"):
        fs_license_path = Path(os.getenv("FREESURFER_HOME"), "license.txt")
    else:
        try:
            log.debug(
                "FREESURFER_HOME is either not defined in the manifest"
                "or does not exist as defined. (${FREESURFER_HOME})\n"
                "Trying to locate freesurfer."
            )
            which_output = subprocess.check_output(["which", "recon-all"], text=True)
            pattern = ".*freesurfer.*?"
            match = re.search(pattern, which_output)
            if match:
                fs_license_path = Path(match.group(), "license.txt")
            else:
                log.error(f"Could not isolate FreeSurfer path from {which_output}")
                raise RuntimeError("Failed to determine FreeSurfer license path.")
        except subprocess.CalledProcessError as e:
            log.error(e.output)
            raise
    return fs_license_path

install_freesurfer_license(context, fs_license_path=None)

Installs the FreeSurfer license file in the expected location.

The license text is obtained using one of the following methods, in order of priority:

1) Provided as an input file (freesurfer_license_file in the manifest). 2) Retrieved from the "freesurfer_license_key" config parameter. 3) Extracted from a Flywheel project's info metadata (FREESURFER_LICENSE).

Once retrieved, the license file is written to the FreeSurfer top-level directory, enabling FreeSurfer usage within the gear.

See How to include a Freesurfer license file... <https://docs.flywheel.io/user/compute/gears/freesurfer/user_including_a_freesurfer_license_file_to_run_a_freesurfer_or_fmriprep_gear/>_

Parameters:

Name Type Description Default
context GearContext

The gear context with core functionality.

required
fs_license_path str

The path where the license should be installed, typically $FREESURFER_HOME/license.txt. Defaults to None.

None

Raises:

Type Description
FileNotFoundError

If the license cannot be found.

Examples:

>>> from fw_gear.utils.licenses.freesurfer import install_freesurfer_license
>>> install_freesurfer_license(context, '/opt/freesurfer/license.txt')
Source code in fw_gear/utils/licenses/freesurfer.py
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
def install_freesurfer_license(
    context: GearContext,
    fs_license_path: t.Optional[os.PathLike] = None,
) -> None:
    """
    Installs the FreeSurfer license file in the expected location.

    The license text is obtained using one of the following methods, in order of priority:

    1) Provided as an input file (`freesurfer_license_file` in the manifest).
    2) Retrieved from the "freesurfer_license_key" config parameter.
    3) Extracted from a Flywheel project's `info` metadata (`FREESURFER_LICENSE`).

    Once retrieved, the license file is written to the FreeSurfer top-level directory,
    enabling FreeSurfer usage within the gear.

    See `How to include a Freesurfer license file...
    <https://docs.flywheel.io/user/compute/gears/freesurfer/user_including_a_freesurfer_license_file_to_run_a_freesurfer_or_fmriprep_gear/>`_

    Args:
        context (fw_gear.GearContext): The gear context with core
            functionality.
        fs_license_path (str): The path where the license should be installed,
            typically `$FREESURFER_HOME/license.txt`. Defaults to None.

    Raises:
        FileNotFoundError: If the license cannot be found.

    Examples:
        >>> from fw_gear.utils.licenses.freesurfer import install_freesurfer_license
        >>> install_freesurfer_license(context, '/opt/freesurfer/license.txt')
    """
    log.debug("Locating FreeSurfer installation")
    fs_license_path = get_fs_license_path(fs_license_path)

    log.debug("Searching for FreeSurfer license")
    license_info = find_license_info(context)

    if license_info:
        write_license_info(fs_license_path, license_info)
    else:
        raise FileNotFoundError(
            f"Could not find FreeSurfer license ({fs_license_path})."
        )

isolate_key_name(context)

Identifies the relevant FreeSurfer license key from the gear configuration.

This function searches for a predefined set of FreeSurfer license key names within the gear's configuration and returns the first matching key.

Parameters:

Name Type Description Default
context GearContext

The gear context containing configuration.

required

Returns:

Type Description
Optional[str]

Optional[str]: The matched license key name if found, otherwise None.

Source code in fw_gear/utils/licenses/freesurfer.py
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
def isolate_key_name(context: GearContext) -> t.Optional[str]:
    """
    Identifies the relevant FreeSurfer license key from the gear configuration.

    This function searches for a predefined set of FreeSurfer license key names
    within the gear's configuration and returns the first matching key.

    Args:
        context (fw_gear.GearContext): The gear context containing configuration.

    Returns:
        Optional[str]: The matched license key name if found, otherwise None.

    """
    keys = [
        "freesurfer_license_key",
        "FREESURFER_LICENSE",
        "FREESURFER-LICENSE",
        "gear-FREESURFER_LICENSE",
    ]

    for k in keys:
        if context.config.opts.get(k):
            return k
        if context.config.opts.get(k.lower()):
            return k.lower()

    log.info("No matching FreeSurfer license key found in config.")
    return None

read_input_license(context)

Reads and returns the FreeSurfer license text from an input file.

This function retrieves the license file specified as freesurfer_license_file in the gear's inputs and returns its contents as a formatted string.

Parameters:

Name Type Description Default
context GearContext

The gear context containing configuration and inputs.

required

Returns:

Type Description
Optional[str]

Optional[str]: The license text as a formatted string, or None if the file is not found.

Example

license_text = read_input_license(context) if license_text: print("License successfully retrieved.")

Source code in fw_gear/utils/licenses/freesurfer.py
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
def read_input_license(context: GearContext) -> t.Optional[str]:
    """
    Reads and returns the FreeSurfer license text from an input file.

    This function retrieves the license file specified as `freesurfer_license_file`
    in the gear's inputs and returns its contents as a formatted string.

    Args:
        context (fw_gear.GearContext): The gear context containing configuration and inputs.

    Returns:
        Optional[str]: The license text as a formatted string, or None if the file is not found.

    Example:
        >>> license_text = read_input_license(context)
        >>> if license_text:
        >>>     print("License successfully retrieved.")
    """
    input_license = context.config.get_input_path("freesurfer_license_file")
    if input_license:
        with open(input_license) as lic:
            license_info = lic.read()
            license_info = "\n".join(license_info.split())
        return license_info

write_license_info(fs_license_path, license_info)

Writes the FreeSurfer license text to the specified file.

This function ensures that the target directory exists before writing the license text to the specified path.

Parameters:

Name Type Description Default
fs_license_path Path

Path to the FreeSurfer license file.

required
license_info str

The license text to be written.

required

Raises:

Type Description
OSError

If unable to write the license file.

Source code in fw_gear/utils/licenses/freesurfer.py
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
def write_license_info(fs_license_path: Path, license_info: str) -> None:
    """
    Writes the FreeSurfer license text to the specified file.

    This function ensures that the target directory exists before writing the
    license text to the specified path.

    Args:
        fs_license_path (Path): Path to the FreeSurfer license file.
        license_info (str): The license text to be written.

    Raises:
        OSError: If unable to write the license file.
    """
    head = Path(fs_license_path).parents[0]
    if not Path(head).exists():
        Path(head).mkdir(parents=True)
        log.debug(f"Created directory {head}")
    with open(fs_license_path, "w") as flp:
        flp.write(license_info)
        log.debug(f"Wrote license {license_info}")
        log.debug(f" to license file {fs_license_path}")

Command Wrapper Module

Command Execution

Wrapper for executing shell commands using Python's subprocess library.

This module provides a streamlined interface for constructing and executing command-line processes. It simplifies handling subprocess calls by dynamically building command-line arguments from configuration parameters (e.g., config.json), managing environment variables, and providing structured error handling.

Features: - Builds command-line arguments dynamically from key-value parameter mappings. - Executes shell commands with logging, error handling, and real-time output streaming. - Supports dry-run mode for debugging without executing commands. - Captures stdout and stderr, raising an exception for non-zero exit codes.

Examples:

>>> command = ["ls"]
>>> param_list = {"l": True, "a": True, "h": True}
>>> command = build_command_list(command, param_list)
>>> command
["ls", "-l", "-a", "-h"]
>>> exec_command(command)

Another example using parameters with values:

>>> command = ["du"]
>>> params = {"a": True, "human-readable": True, "max-depth": 3}
>>> command = build_command_list(command, params)
>>> command
["du", "-a", "--human-readable", "--max-depth=3"]
>>> params = {"dir1": ".", "dir2": "/tmp"}
>>> command = build_command_list(command, params, include_keys=False)
>>> command
["du", ".", "/tmp"]
>>> exec_command(command)

These examples demonstrate how command-line parameters can be dynamically constructed and executed from gear configuration options.

build_command_list(command, param_list, include_keys=True)

Constructs a command-line argument list for subprocess execution.

Parameters:

Name Type Description Default
command List[str]

Base command (e.g., ["ls"]), including required parameters.

required
param_list Dict[str, any]

Dictionary of key-value pairs representing command-line arguments. - Boolean values (True) result in flags (-k or --key). - String/integer values are formatted as --key=value or -k value. - None or empty strings are ignored.

required
include_keys bool

Whether to include parameter keys in the command list. Defaults to True.

True

Returns:

Type Description
List[str]

List[str]: A formatted command list suitable for subprocess.Popen.

Example

command = ["du"] params = {"a": True, "human-readable": True, "max-depth": 3} command = build_command_list(command, params) command ["du", "-a", "--human-readable", "--max-depth=3"]

Source code in fw_gear/utils/wrapper/command.py
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
def build_command_list(
    command: t.List[str], param_list: t.Dict[str, t.Any], include_keys: bool = True
) -> t.List[str]:
    """Constructs a command-line argument list for subprocess execution.

    Args:
        command (List[str]): Base command (e.g., `["ls"]`), including required parameters.
        param_list (Dict[str, any]): Dictionary of key-value pairs representing command-line arguments.
            - Boolean values (`True`) result in flags (`-k` or `--key`).
            - String/integer values are formatted as `--key=value` or `-k value`.
            - `None` or empty strings are ignored.
        include_keys (bool, optional): Whether to include parameter keys in the command list. Defaults to True.

    Returns:
        List[str]: A formatted command list suitable for `subprocess.Popen`.

    Example:
        >>> command = ["du"]
        >>> params = {"a": True, "human-readable": True, "max-depth": 3}
        >>> command = build_command_list(command, params)
        >>> command
        ["du", "-a", "--human-readable", "--max-depth=3"]
    """
    param_list = _remove_prohibited_values(param_list)
    for key in param_list.keys():
        # Single character command-line parameters are preceded by a single "-"
        if len(key) == 1:
            # If Param is boolean and true, include, else exclude
            if isinstance(param_list[key], bool) and param_list[key]:
                command.append("-" + key)
            else:
                if include_keys:
                    command.append("-" + key)
                if str(param_list[key]):
                    command.append(str(param_list[key]))
        # Multi-Character command-line parameters are preceded by a double "--"
        # If Param is boolean and true include, else exclude
        elif isinstance(param_list[key], bool):
            if param_list[key] and include_keys:
                command.append("--" + key)
        else:
            item = ""
            if include_keys:
                item = "--" + key
                item = item + "="
            item = item + str(param_list[key])
            command.append(item)
    return command

exec_command(command, dry_run=False, environ=None, shell=False, stream=False, stream_mode=None, throttle_sec=1.0, logfile=None)

Executes a shell command using the subprocess module with flexible output handling.

Parameters:

Name Type Description Default
command List[str]

List of command-line arguments, starting with the command itself.

required
dry_run bool

If True, logs the command without executing it. Returns ("", "", 0). Defaults to False.

False
environ Optional[Dict[str, str]]

Additional environment variables to merge with the current environment. Defaults to None.

None
shell bool

Run via system shell (required for redirects/pipes). When True, only literal tokens (">", ">>", "<", "1>", "1>>", "2>", "2>>", "2>&1", "1>&2") are left unquoted; all other arguments are safely quoted. Defaults to False.

False
stream bool

Enable live streaming of output line-by-line (deadlock-safe). When True, stderr is merged into stdout. Defaults to False.

False
stream_mode Optional[str]

Controls what is printed during streaming: - "all": Print every line in real-time (default) - "filter_only": Only print lines matching important patterns (errors, warnings, etc.) - "throttled": Print important lines immediately, rate-limit others Full output is always captured regardless of mode. Defaults to "all".

None
throttle_sec float

Minimum time in seconds between printing non-important lines in "throttled" mode. Defaults to 1.0.

1.0
logfile Optional[Union[str, PathLike]]

Path to append the full unfiltered output stream. Works with all streaming modes. Defaults to None.

None

Returns:

Type Description
Tuple[str, str, int]

Tuple[str, str, int]: A tuple containing: - stdout (str): Standard output from the command. When stream=True, contains merged stdout+stderr. - stderr (str): Standard error output. Empty string ("") when stream=True. - returncode (int): Command exit status (0 for success).

Raises:

Type Description
RuntimeError

If the command returns a non-zero exit code, or if the logfile cannot be opened for writing.

Notes
  • Streaming is automatically disabled when shell redirection tokens are detected
  • Important lines are detected via regex pattern (configurable via EXEC_ALWAYS_PRINT_RE env var)
  • In "throttled" mode, a final suppressed line count is printed at the end
  • The logfile path is validated before command execution (fail-fast behavior)
Example

command = ["du"] params = {"a": True, "human-readable": True, "max-depth":3} command = build_command_list(command, params) params = {"dir1":".","dir2":"/tmp"} command = build_command_list(command, params, include_keys=False) exec_command(command)

1) Basic run (no live streaming)

stdout, stderr, rc = exec_command(command)

2) Live stream only important lines; keep full transcript

stdout, stderr, rc = exec_command( ... command, stream=True, stream_mode="filter_only", logfile="du.stream.log" ... )

3) If you truly need shell redirection (no streaming)

cmd2 = command + [">>", "du.out.log", "2>&1"] stdout, stderr, rc = exec_command(cmd2, shell=True, stream=False)

Source code in fw_gear/utils/wrapper/command.py
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
def exec_command(
    command: t.List[str],
    dry_run: bool = False,
    environ: t.Optional[t.Dict[str, str]] = None,
    shell: bool = False,
    stream: bool = False,
    stream_mode: t.Optional[str] = None,  # "all" | "filter_only" | "throttled" | None
    throttle_sec: float = 1.0,  # used if stream_mode="throttled"
    logfile: t.Optional[t.Union[str, os.PathLike]] = None,  # tee full stream to file
) -> t.Tuple[str, str, int]:  # noqa: PLR0912, PLR0915
    """
    Executes a shell command using the subprocess module with flexible output handling.

    Args:
        command (List[str]): List of command-line arguments, starting with the command itself.
        dry_run (bool, optional): If True, logs the command without executing it. Returns ("", "", 0).
            Defaults to False.
        environ (Optional[Dict[str, str]], optional): Additional environment variables to merge with
            the current environment. Defaults to None.
        shell (bool, optional): Run via system shell (required for redirects/pipes). When True,
            only literal tokens (">", ">>", "<", "1>", "1>>", "2>", "2>>", "2>&1", "1>&2")
            are left unquoted; all other arguments are safely quoted. Defaults to False.
        stream (bool, optional): Enable live streaming of output line-by-line (deadlock-safe).
            When True, stderr is merged into stdout. Defaults to False.
        stream_mode (Optional[str], optional): Controls what is printed during streaming:
            - "all": Print every line in real-time (default)
            - "filter_only": Only print lines matching important patterns (errors, warnings, etc.)
            - "throttled": Print important lines immediately, rate-limit others
            Full output is always captured regardless of mode. Defaults to "all".
        throttle_sec (float, optional): Minimum time in seconds between printing non-important
            lines in "throttled" mode. Defaults to 1.0.
        logfile (Optional[Union[str, os.PathLike]], optional): Path to append the full unfiltered
            output stream. Works with all streaming modes. Defaults to None.

    Returns:
        Tuple[str, str, int]: A tuple containing:
            - stdout (str): Standard output from the command. When stream=True, contains
              merged stdout+stderr.
            - stderr (str): Standard error output. Empty string ("") when stream=True.
            - returncode (int): Command exit status (0 for success).

    Raises:
        RuntimeError: If the command returns a non-zero exit code, or if the logfile
            cannot be opened for writing.

    Notes:
        - Streaming is automatically disabled when shell redirection tokens are detected
        - Important lines are detected via regex pattern (configurable via EXEC_ALWAYS_PRINT_RE env var)
        - In "throttled" mode, a final suppressed line count is printed at the end
        - The logfile path is validated before command execution (fail-fast behavior)

    Example:
            >>> command = ["du"]
            >>> params = {"a": True, "human-readable": True, "max-depth":3}
            >>> command = build_command_list(command, params)
            >>> params = {"dir1":".","dir2":"/tmp"}
            >>> command = build_command_list(command, params, include_keys=False)
            >>> exec_command(command)
            # 1) Basic run (no live streaming)
            >>> stdout, stderr, rc = exec_command(command)

            # 2) Live stream only important lines; keep full transcript
            >>> stdout, stderr, rc = exec_command(
            ...     command, stream=True, stream_mode="filter_only", logfile="du.stream.log"
            ... )

            # 3) If you truly need shell redirection (no streaming)
            >>> cmd2 = command + [">>", "du.out.log", "2>&1"]
            >>> stdout, stderr, rc = exec_command(cmd2, shell=True, stream=False)
    """
    # grab env - preserve PATH etc
    env = os.environ.copy()
    if environ:
        env.update(environ)

    # sanitize shell
    if shell:
        run_command = " ".join(
            (s if s in _SHELL_REDIRECT_TOKENS else shlex.quote(s))
            for s in map(str, command)
        )
    else:
        # No shell: pass a list (no quoting).
        run_command = [str(a) for a in command]

    # Log the command that will be executed (after quoting/sanitization)
    if shell:
        log.info(f"Executing command:\n {run_command}\n\n")
    else:
        log.info(f"Executing command:\n {shlex.join(run_command)}\n\n")

    # Return dry-run immediately after logging
    if dry_run:
        return "", "", 0

    # Validate logfile if specified before running command
    if logfile is not None:
        try:
            with open(logfile, "a"):
                pass  # Just test if we can open for appending
        except (OSError, IOError) as e:
            raise RuntimeError(f"Cannot open logfile '{logfile}': {e}") from e

    # to disable streaming if redirecting
    has_redirect = shell and any(str(a) in _SHELL_REDIRECT_TOKENS for a in command)

    # merge stderr only for streaming (keeps non-streaming behavior closer to before)
    stderr_target = sp.STDOUT if stream else sp.PIPE

    result = sp.Popen(
        run_command,
        stdout=sp.PIPE,
        stderr=stderr_target,  # merge in streaming to avoid deadlock
        stdin=sp.DEVNULL,  # child won't wait for input
        universal_newlines=True,  # text=True
        env=env,
        shell=shell,
        bufsize=1,  # line-buffered for timely lines
    )

    # streaming path (live), unless redirecting (breaks streaming)
    if stream and not has_redirect:
        out_lines: t.List[str] = []

        # normalize mode (default = "all")
        mode = (stream_mode or "all").lower()
        if mode not in {"all", "filter_only", "throttled"}:
            mode = "all"

        last_print = 0.0
        suppressed = 0

        with (
            open(logfile, "a") if logfile is not None else contextlib.nullcontext(None)
        ) as lf:
            for line in iter(result.stdout.readline, ""):
                out_lines.append(line)
                if lf is not None:
                    lf.write(line)
                    lf.flush()

                # what to print
                show_now = True
                if mode == "filter_only":
                    show_now = bool(_ALWAYS_PRINT_RE.search(line))
                elif mode == "throttled":
                    if _ALWAYS_PRINT_RE.search(line):
                        # important lines - print immediately
                        if suppressed:
                            print(f"(suppressed {suppressed} lines)", flush=True)
                            suppressed = 0
                        last_print = time.monotonic()
                        show_now = True
                    else:
                        now = time.monotonic()
                        if now - last_print >= throttle_sec:
                            if suppressed:
                                print(f"(suppressed {suppressed} lines)", flush=True)
                                suppressed = 0
                            last_print = now
                            show_now = True
                        else:
                            suppressed += 1
                            show_now = False
                # mode "all": show every line

                if show_now:
                    print(line, end="", flush=True)

            if mode == "throttled" and suppressed:
                print(f"(suppressed {suppressed} lines)", flush=True)

        returncode = result.wait()
        stdout = "".join(out_lines)
        stderr = ""  # merged into stdout during streaming

    # non-streaming path - buffered, prints at end
    else:
        stdout, stderr = result.communicate()
        returncode = result.returncode
        if stdout:
            log.info(stdout)

    log.info(f"Command return code: {returncode}")

    if returncode != 0:
        # If streaming, errors may be in stdout
        err_text = stderr or stdout
        if err_text:
            log.error(err_text)
        raise RuntimeError(
            f"Command failed with exit code {returncode}: {shlex.join(map(str, command))}"
        )

    return stdout, stderr, returncode

Nipype Integration

A wrapper module to generate a Nipype interface.

GearContextInterfaceBase

Bases: SimpleInterface

Factory class for generating Nipype interfaces from Flywheel gear manifests.

This class dynamically constructs a Nipype interface based on a gear's manifest.json, allowing for seamless integration into Nipype workflows.

Source code in fw_gear/utils/wrapper/nipype.py
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
class GearContextInterfaceBase(SimpleInterface):
    """Factory class for generating Nipype interfaces from Flywheel gear manifests.

    This class dynamically constructs a `Nipype` interface based on a gear's
    `manifest.json`, allowing for seamless integration into `Nipype` workflows."""

    @staticmethod
    def get_class_name(manifest: t.Dict = None) -> t.AnyStr:
        """Returns the class name from the gear name in manifest.

        Example: 'my-gear' -> 'MyGearContextInterface'"""
        gn = manifest.get("name")
        gn_clean_cap = "".join([x.capitalize() for x in re.split("[^a-zA-Z0-9]", gn)])
        return f"{gn_clean_cap}ContextInterface"

    @classmethod
    def factory(cls, manifest: t.Dict = None) -> type:
        """Returns a new GearContextInterface instance for the given manifest."""
        class_name = cls.get_class_name(manifest)
        attrs = {
            "_run_interface": cls._run_interface,
            "_manifest": manifest,
            "input_spec": GearContextInterfaceInputSpec,
            "output_spec": GearContextInterfaceOutputSpecBase.factory(manifest),
        }
        interface = type(class_name, (cls,), attrs)

        return interface

    def _run_interface(self, runtime):
        outputs = self._outputs().get()
        config = self.inputs.config_dict.get("config")
        inputs = self.inputs.config_dict.get("inputs")

        # Parse config_dict
        for k, v in outputs.items():
            if k.startswith("config_"):
                self._results[k] = config.get(k.split("config_")[1])
            elif k.startswith("inputs_"):
                input_key = k.split("inputs_")[1]
                base_type = inputs[input_key].get("base")
                if base_type == "context":
                    self._results[k] = inputs[input_key].get("value")
                if base_type == "file":
                    self._results[k] = inputs[input_key].get("location").get("path")

        return runtime

    def __getstate__(self):
        # To give a hand to pickle dump those dynamically built classes
        if self.__class__.__name__ not in GLOBALS:  # needed for pickle
            GLOBALS[self.__class__.__name__] = self.__class__
        return (self.__dict__, self.__class__._manifest)

    def __setstate__(self, state):
        # To give a hand to pickle load those dynamically built classes
        if self.__class__.__name__ in GLOBALS:  # needed for pickle
            self.__class__ = GLOBALS[self.__class__.__name__]
        else:
            self.__class__ = self.__class__.factory(state[1])
        self.__dict__.update(state[0])
factory(manifest=None) classmethod

Returns a new GearContextInterface instance for the given manifest.

Source code in fw_gear/utils/wrapper/nipype.py
150
151
152
153
154
155
156
157
158
159
160
161
162
@classmethod
def factory(cls, manifest: t.Dict = None) -> type:
    """Returns a new GearContextInterface instance for the given manifest."""
    class_name = cls.get_class_name(manifest)
    attrs = {
        "_run_interface": cls._run_interface,
        "_manifest": manifest,
        "input_spec": GearContextInterfaceInputSpec,
        "output_spec": GearContextInterfaceOutputSpecBase.factory(manifest),
    }
    interface = type(class_name, (cls,), attrs)

    return interface
get_class_name(manifest=None) staticmethod

Returns the class name from the gear name in manifest.

Example: 'my-gear' -> 'MyGearContextInterface'

Source code in fw_gear/utils/wrapper/nipype.py
141
142
143
144
145
146
147
148
@staticmethod
def get_class_name(manifest: t.Dict = None) -> t.AnyStr:
    """Returns the class name from the gear name in manifest.

    Example: 'my-gear' -> 'MyGearContextInterface'"""
    gn = manifest.get("name")
    gn_clean_cap = "".join([x.capitalize() for x in re.split("[^a-zA-Z0-9]", gn)])
    return f"{gn_clean_cap}ContextInterface"

GearContextInterfaceInputSpec

Bases: BaseInterfaceInputSpec

Defines the input specification for a GearContextInterface.

Attributes:

Name Type Description
config_dict Dict

The Flywheel config.json as a dictionary.

Source code in fw_gear/utils/wrapper/nipype.py
61
62
63
64
65
66
67
68
69
class GearContextInterfaceInputSpec(BaseInterfaceInputSpec):
    """Defines the input specification for a `GearContextInterface`.

    Attributes:
        config_dict (traits.Dict): The Flywheel `config.json` as a dictionary.
    """

    # TODO: consider adding context as input
    config_dict = traits.Dict(mandatory=True, desc="Gear config.json as dictionary")

GearContextInterfaceOutputSpecBase

Factory class for generating Nipype output specifications from Flywheel gear manifests.

This class dynamically builds Nipype output specifications by extracting parameter definitions from the manifest.json of a Flywheel gear.

Source code in fw_gear/utils/wrapper/nipype.py
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
class GearContextInterfaceOutputSpecBase:
    """Factory class for generating Nipype output specifications from Flywheel gear manifests.

    This class dynamically builds `Nipype` output specifications by extracting
    parameter definitions from the `manifest.json` of a Flywheel gear.
    """

    @staticmethod
    def get_outputspec_attr(manifest: t.Dict = None) -> t.Dict[str, ContextType]:
        """Returns OutputSpec attributes."""
        attrs = {}
        for k, v in manifest.get("config").items():
            traits_obj = get_traits_object(v["type"], v.get("description"))
            if traits_obj:
                # prefixing with "config_" to avoid collision with inputs
                attrs[f"config_{k}"] = traits_obj

        for k, v in manifest.get("inputs").items():
            if v["base"] in ["file", "context"]:
                traits_obj = get_traits_object(v["base"], v.get("description"))
                # prefixing with "inputs_" to avoid collision with config
                if traits_obj:
                    attrs[f"inputs_{k}"] = traits_obj

        return attrs

    @staticmethod
    def get_class_name(manifest: t.Dict = None) -> t.AnyStr:
        """Return class name from gear name in manifest.

        Example: 'my-gear' -> MyGearContextInterface
        """
        gn = manifest.get("name")
        gn_clean_cap = "".join([x.capitalize() for x in re.split("[^a-zA-Z0-9]", gn)])
        return f"{gn_clean_cap}ContextInterfaceOutputSpec"

    @classmethod
    def factory(cls, manifest):
        """Returns an OutputSpec class based on manifest.json"""
        class_name = cls.get_class_name(manifest)
        attrs = {"_manifest": manifest, "__module__": __name__}
        traits_attrs = cls.get_outputspec_attr(manifest)
        attrs.update(traits_attrs)
        output_spec = type(class_name, (cls, TraitedSpec), attrs)
        # if class_name not in GLOBALS:  # needed for pickle
        #     GLOBALS[class_name] = output_spec
        return output_spec

    def __getstate__(self):
        # To give a hand to pickle dump those dynamically built classes
        if self.__class__.__name__ not in GLOBALS:  # needed for pickle
            GLOBALS[self.__class__.__name__] = self.__class__
        return (self.__dict__, self.__class__._manifest)

    def __setstate__(self, state):
        # To give a hand to pickle load those dynamically built classes
        if self.__class__.__name__ in GLOBALS:  # needed for pickle
            self.__class__ = GLOBALS[self.__class__.__name__]
        else:
            self.__class__ = self.__class__.factory(state[1])
        self.__dict__.update(state[0])
factory(manifest) classmethod

Returns an OutputSpec class based on manifest.json

Source code in fw_gear/utils/wrapper/nipype.py
108
109
110
111
112
113
114
115
116
117
118
@classmethod
def factory(cls, manifest):
    """Returns an OutputSpec class based on manifest.json"""
    class_name = cls.get_class_name(manifest)
    attrs = {"_manifest": manifest, "__module__": __name__}
    traits_attrs = cls.get_outputspec_attr(manifest)
    attrs.update(traits_attrs)
    output_spec = type(class_name, (cls, TraitedSpec), attrs)
    # if class_name not in GLOBALS:  # needed for pickle
    #     GLOBALS[class_name] = output_spec
    return output_spec
get_class_name(manifest=None) staticmethod

Return class name from gear name in manifest.

Example: 'my-gear' -> MyGearContextInterface

Source code in fw_gear/utils/wrapper/nipype.py
 98
 99
100
101
102
103
104
105
106
@staticmethod
def get_class_name(manifest: t.Dict = None) -> t.AnyStr:
    """Return class name from gear name in manifest.

    Example: 'my-gear' -> MyGearContextInterface
    """
    gn = manifest.get("name")
    gn_clean_cap = "".join([x.capitalize() for x in re.split("[^a-zA-Z0-9]", gn)])
    return f"{gn_clean_cap}ContextInterfaceOutputSpec"
get_outputspec_attr(manifest=None) staticmethod

Returns OutputSpec attributes.

Source code in fw_gear/utils/wrapper/nipype.py
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
@staticmethod
def get_outputspec_attr(manifest: t.Dict = None) -> t.Dict[str, ContextType]:
    """Returns OutputSpec attributes."""
    attrs = {}
    for k, v in manifest.get("config").items():
        traits_obj = get_traits_object(v["type"], v.get("description"))
        if traits_obj:
            # prefixing with "config_" to avoid collision with inputs
            attrs[f"config_{k}"] = traits_obj

    for k, v in manifest.get("inputs").items():
        if v["base"] in ["file", "context"]:
            traits_obj = get_traits_object(v["base"], v.get("description"))
            # prefixing with "inputs_" to avoid collision with config
            if traits_obj:
                attrs[f"inputs_{k}"] = traits_obj

    return attrs

get_traits_object(datatype, description=None)

Returns the corresponding traits object for a given input datatype.

Parameters:

Name Type Description Default
datatype str

The data type to map to a traits object.

required
description Optional[str]

A description for the trait. Defaults to None.

None

Returns:

Type Description
Union[None, ContextType]

Union[None, ContextType]: The corresponding traits object or None if the datatype is unsupported.

Raises:

Type Description
TypeError

If the datatype is not recognized.

Source code in fw_gear/utils/wrapper/nipype.py
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
def get_traits_object(
    datatype: str, description: Optional[str] = None
) -> t.Union[None, ContextType]:
    """Returns the corresponding `traits` object for a given input
    datatype.

    Args:
        datatype (str): The data type to map to a `traits` object.
        description (Optional[str], optional): A description for the trait. Defaults to None.

    Returns:
        Union[None, ContextType]: The corresponding `traits` object or None if the datatype is unsupported.

    Raises:
        TypeError: If the datatype is not recognized.

    """
    if datatype == "boolean":
        return traits.Bool(desc=description)
    elif datatype in ("string", "context"):
        return traits.Str(desc=description)
    elif datatype == "integer":
        return traits.Int(desc=description)
    elif datatype == "number":
        return traits.Float(desc=description)
    elif datatype == "array":
        return traits.List(desc=description)
    elif datatype == "file":
        return File(desc=description)
    else:
        raise TypeError(
            f"{datatype} type is not supported. Please only use valid datatype."
        )

HPC Integration

Set up tools for HPC integration using Singularity.