API Reference
Core Modules
GearContext
Context for gear runtime.
- Provides methods for interacting with gear runtime objects such as:
config.json(Runtime configuration)manifest.json(Defined configuration and gear metadata).metadata.json(Hierarchy metadata to be updated at end of gear run)
- Configures gear logging
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
gear_path | Optional[str] | Path to gear directory, default behavior will use | '/flywheel/v0' |
manifest_path | Optional[str] | Path to the gear's manifest.json file, defaults to | '/flywheel/v0/manifest.json' |
config_path | Optional[str] | Path to the gear's config.json file, defaults to | '/flywheel/v0/config.json' |
log_metadata | bool | Whether to log .metadata.json to log upon write, defaults to True. | True |
clean_on_error | bool | Whether to clean output directory upon exit with error, defaults to False. | False |
Source code in fw_gear/context.py
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 | |
client property
Wrapper around self.get_client().
output_dir property
Get the absolute path to the output directory.
Returns:
| Type | Description |
|---|---|
Path | pathlib.Path: The absolute path to outputs. |
work_dir property
Get the absolute path to a work directory.
Returns:
| Type | Description |
|---|---|
Path | pathlib.Path: The absolute path to work. |
get_client()
Get the SDK client, if an api key input exists or CLI client exists.
Returns:
| Type | Description |
|---|---|
Optional[Client] | flywheel.Client: The Flywheel SDK client. |
Raises:
| Type | Description |
|---|---|
GearContextError | If the Flywheel SDK is not installed, no API key is found in |
Source code in fw_gear/context.py
172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 | |
get_context_value(name)
Get the context input for name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name | str | The name of the input. | required |
Returns:
| Name | Type | Description |
|---|---|---|
dict | Optional[dict] | The input context value, or None if not found. |
Source code in fw_gear/context.py
252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 | |
init_logging(default_config_name=None, update_config=None)
Configures logging via fw_gear.logging.configure_logging.
If no default_config_name is provided, will get debug from the configuration options. If debug is False or not defined in the gear configuration options, default_config_name will be set to info.
If update_config is not provided, manifest['custom']['log_config'] will be used (if defined in the manifest).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
default_config_name | str | A string, 'info' or 'debug', indicating the default template to use. (Defaults to 'info'). | None |
update_config | dict | A dictionary containing the keys, sub-keys, and values of the templates to update. (Defaults to None). | None |
Source code in fw_gear/context.py
96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 | |
is_fw_context()
Return True if the gear is running in a Flywheel gear context, False otherwise.
Source code in fw_gear/context.py
165 166 167 168 169 170 | |
log_config()
Print the configuration and input files to the logger.
Source code in fw_gear/context.py
208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 | |
open_output(name, mode='w', **kwargs)
Open the named output file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name | str | The name of the output. | required |
mode | str | The open mode (Default value = 'w'). | 'w' |
**kwargs | dict | Keyword arguments for | {} |
Returns:
| Name | Type | Description |
|---|---|---|
File | IO | The file object. |
Source code in fw_gear/context.py
238 239 240 241 242 243 244 245 246 247 248 249 250 | |
Config
Basic Config class akin to Manifest class.
This class is mainly used as a helper for the gear config create function of the python-cli, however it may have unforeseen programmatic usages.
There is some confusing nomenclature here. The gear configuration is stored in a file named config.json, within the config.json file, there are at three keys: * config: configuration options for the gear, often booleans, integers, floats, or short strings. * inputs: Gear inputs, usually in the form of files, longer text. * destination: Information on where the gear is being run within the flywheel hierarchy.
This class represents the config.json file, meaning that is has attributes which store the configuration options, inputs, and destination. That means that the class Config has a config attribute (configuration options) and the constructor accepts a config dictionary that contains a config (configuration options) key.
Source code in fw_gear/config.py
15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 | |
destination property writable
Get gear run destination.
inputs property writable
Get provided inputs.
job property writable
Get provided job info when show-job set to true.
opts property writable
Get provided configuration values.
__init__(client, config, path=Path.cwd())
Load or generate config.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
client | Optional[Client] | Optional flywheel SDK client. | required |
config | Optional[dict] | Config dictionary with any of the following keys: 'config','inputs','destination'. Defaults to None. | required |
path | Path | Path to existing configuration or path to directory in which to create config. Defaults to Path.cwd(). | cwd() |
Raises:
| Type | Description |
|---|---|
ConfigValidationError |
|
Source code in fw_gear/config.py
42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | |
__str__()
Return string representation.
Source code in fw_gear/config.py
378 379 380 381 382 383 384 385 | |
add_input(name, val, type_='file', file_=None)
Add an input to the config.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name | str | name of input | required |
val | str | input value, file path, api-key, context | required |
type_ | str | file, api-key, or context. Defaults to "file". | 'file' |
file_ | Optional[Any] | File object to set | None |
Raises:
| Type | Description |
|---|---|
ValueError | When file doesn't exist. |
NotImplementedError | When type is not file or api-key |
Source code in fw_gear/config.py
154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 | |
default_config_from_manifest(manifest) classmethod
Create a default config.json from a manifest file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
manifest | Union[Path, Manifest] | Path to manifest or instantiated Manifest object. | required |
Raises:
| Type | Description |
|---|---|
ValueError | When there is a problem parsing the manifest. |
Returns:
| Name | Type | Description |
|---|---|---|
Config | Config | new config class with a default configuration. |
Source code in fw_gear/config.py
220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 | |
get_destination_container()
Returns the destination container.
Source code in fw_gear/config.py
352 353 354 355 356 357 | |
get_destination_parent()
Returns the parent container of the destination container.
Source code in fw_gear/config.py
359 360 361 362 363 364 365 | |
get_input(name)
Get raw input object by name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name | str | Name of input. | required |
Source code in fw_gear/config.py
249 250 251 252 253 254 255 | |
get_input_file_object(name)
Get the specified input file object from config.json
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name | str | The name of the input. | required |
Returns:
| Name | Type | Description |
|---|---|---|
dict | Optional[dict] | The input dictionary, or None if not found. |
Source code in fw_gear/config.py
296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 | |
get_input_file_object_value(name, key)
Get the value of the input file metadata from config.json.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name | str | The name of the input. | required |
key | str | The name of the file container metadata. | required |
Raises:
| Type | Description |
|---|---|
ValueError | if the input exists, but is not a file. |
Returns:
| Type | Description |
|---|---|
Any | Union[str, list, dict]: The value of the specified file metadata, or None if not found. |
Source code in fw_gear/config.py
312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 | |
get_input_filename(name)
Get the the filename of given input file. Sourced from the 'inputs' field in the manifest.json
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name | str | The name of the input. | required |
Raises:
| Type | Description |
|---|---|
ValueError | if the input exists, but is not a file. |
Returns:
| Name | Type | Description |
|---|---|---|
str | Optional[str] | The filename to the input file if it exists, otherwise None. |
Source code in fw_gear/config.py
276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 | |
get_input_path(name)
Get input filepath by name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name | str | Input name | required |
Raises:
| Type | Description |
|---|---|
ValueError | If found input is not a file. |
Returns:
| Type | Description |
|---|---|
Optional[Path] | t.Optional[Path]: Input filepath if found |
Source code in fw_gear/config.py
257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 | |
open_input(name, mode='r', **kwargs)
Open the named input file, derived from the 'inputs' field in the manifest.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name | str | The name of the input. | required |
mode | str | The open mode (Default value = 'r'). | 'r' |
**kwargs | dict | Keyword arguments for | {} |
Raises:
| Type | Description |
|---|---|
ValueError | If input |
FileNotFoundError | If the path in the config.json for input |
Returns:
| Type | Description |
|---|---|
IO | The file object. |
Source code in fw_gear/config.py
329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 | |
to_json(path=None)
Dump config to a file in config.json format.
Source code in fw_gear/config.py
368 369 370 371 372 373 374 375 376 | |
update_destination(dest)
Perform dictionary on destination dictionary.
Source code in fw_gear/config.py
150 151 152 | |
update_opts(vals)
Perform dictionary update on configuration values.
Source code in fw_gear/config.py
146 147 148 | |
Manifest
A class to host a manifest and a methods around it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
manifest | Path - like or dict | Path to a manifest.json file or a manifest dictionary. If none is provided, try to load manifest.json from the current working directory. | None |
Attributes:
| Name | Type | Description |
|---|---|---|
manifest | Dotty | Dictionary (dotty-enhanced version to be exact) of the manifest.json file. |
Raises:
| Type | Description |
|---|---|
FileNotFoundError | If no manifest.json found |
Source code in fw_gear/manifest.py
19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 | |
author property
Returns manifest author.
config property
Returns manifest config.
description property
Returns manifest description.
gear_spec_schema property
Returns the json schema of the gear spec.
inputs property
Returns manifest inputs.
label property
Returns manifest label.
license property
Returns manifest license.
name property
Returns manifest name.
schema property
Returns the json schema definition of the manifest.json file.
source property
Returns manifest source.
url property
Returns manifest url.
version property
Returns manifest version.
__getitem__(dotty_key)
Return any value of the manifest by passing the dotty-dict key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dotty_key | str | A string representing a key or nested key (e.g. custom.gear-builder.image) | required |
Source code in fw_gear/manifest.py
122 123 124 125 126 127 128 129 | |
derive_config_schema(full=True)
Returns json schema for config from manifest.
Adapted from https://gitlab.com/flywheel-io/public/gears/-/blob/master/gears/generator.py#L66.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
full | bool | Whether to derive a "full" config schema, to be used when validating a real config that has been populated. Set "full" to false if you are just validating the default values in a config. | True |
Source code in fw_gear/manifest.py
317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 | |
get_docker_image_name_tag()
Returns docker image tag from either locations.
Look first at 'custom.gear-builder.image'. If not defined there, look at 'custom.docker-image' and log a warning.
Source code in fw_gear/manifest.py
184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 | |
get_manifest_from_file(path=None) staticmethod
Returns the dictionary representation of the manifest.json at path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | Path - like | Path to manifest. If None, look for manifest.json in current directory. | None |
Returns:
| Type | Description |
|---|---|
dict | The manifest as a dictionary |
Source code in fw_gear/manifest.py
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 | |
get_value(dotty_key)
Returns value found at dotty_key (e.g. 'custom.gear-builder.image') if any.
More on dotty-dict notation at here
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dotty_key | str | A string representing a key or nested key (e.g. custom.gear-builder.image) | required |
Source code in fw_gear/manifest.py
159 160 161 162 163 164 165 166 167 168 | |
is_valid()
Returns True if manifest is valid, False otherwise.
Source code in fw_gear/manifest.py
211 212 213 214 215 216 217 | |
to_json(path, validate=True, ensure_ascii=True)
Save as json file to path.
Source code in fw_gear/manifest.py
170 171 172 173 174 175 176 177 178 179 180 181 182 | |
validate(validate_classification=False)
Validates manifest.
Raises:
| Type | Description |
|---|---|
ManifestValidationError | If errors are found in the manifest. |
Source code in fw_gear/manifest.py
201 202 203 204 205 206 207 208 209 | |
Metadata
Class to store and interact with Flywheel metadata via .metadata.json or SDK call.
Source code in fw_gear/metadata.py
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 | |
__init__(context=None, name_override='', version_override='')
Initialize metadata class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context | Optional[GearContext] | GearContext from which to populate gear info (inputs, config, destination, ...) | None |
name_override | str | Optional gear name to use, overrides manifest value from GearContext. Defaults to "". | '' |
version_override | str | Optional gear version to use, overrides manifest value from GearContext. Defaults to "". | '' |
Source code in fw_gear/metadata.py
71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | |
add_file_tags(file_, tags)
Add tag(s) to a file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_ | Any | File Object | required |
tags | str or Iterable[str] | Tags to add. | required |
Source code in fw_gear/metadata.py
401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 | |
add_gear_info(top_level, cont_, **kwargs)
Add arbitrary gear info to a particular file/container.
Add predefined info on gear configuration, as well as custom free-form information to a given file/container.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
top_level | str | Top level info key under "info" in which to store the data, i.e. "qc.result" stores info under "info.qc.result". | required |
cont_ | t.Any] | File/container to store information on | required |
kwargs | Any | Custom information to add under gear info. | {} |
Source code in fw_gear/metadata.py
530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 | |
add_or_update_gear_info(top_level_keys, cont_info, **kwargs)
Modify container info section with current job run info including info about the gear.
Source code in fw_gear/metadata.py
496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 | |
add_qc_result(file_, name, state, **data)
Add a pass/fail/na QC results to a file and update .metadata.json.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_ | Any | File object | required |
name | str | QC result name | required |
state | str | pass, fail, or na | required |
data | custom data | {} |
Source code in fw_gear/metadata.py
346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 | |
add_qc_result_to_analysis(name, state, **data)
Add a pass/fail/na QC results to the destination analysis container via update .metadata.json.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name | str | QC result name | required |
state | str | pass, fail, or na | required |
data | custom data | {} |
Source code in fw_gear/metadata.py
363 364 365 366 367 368 369 370 371 372 373 374 375 376 | |
add_qc_result_via_sdk(cont_, name, state, **data)
Add a pass/fail/na QC results to a container/file and update the container/file info via HTTP methods.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cont_ | Any | Flywheel Container object | required |
name | str | QC result name | required |
state | str | pass, fail, or na | required |
data | custom data | {} |
Source code in fw_gear/metadata.py
379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 | |
clean()
Clean metadata before writing.
Source code in fw_gear/metadata.py
281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 | |
log()
Log representation of metadata.
Source code in fw_gear/metadata.py
300 301 302 303 304 305 306 | |
modify_container_file_info(file_, **info)
Modify file info with HTTP requests.
Wrapper for modify_container_file_info SDK method call. Using set method (add on) to update.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_ | File | File Object | required |
info | Dict[Any] | Info dictionary that will be used to update file info. | {} |
Source code in fw_gear/metadata.py
241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 | |
modify_container_info(cont_id, **info)
Update metadata for the given container type in the hierarchy.
Wrapper for modify_container_info SDK method call. Using set method (add on) to update.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cont_id | str | Container ID that will be used to update container info | required |
info | Dict[Any] | Info dictionary that will be used to update the container info. | {} |
Source code in fw_gear/metadata.py
262 263 264 265 266 267 268 269 270 271 272 273 274 275 | |
pull_job_info()
Pull job info from GearContext if present.
Get job input files, configuration, and job id.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name | str | Gear name override. | required |
version | str | Gear version override. | required |
Source code in fw_gear/metadata.py
423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 | |
update_container(container_type, deep=True, **kwargs)
Update metadata for the given container type in the hierarchy.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
container_type | str | The container type (e.g. session or acquisition). | required |
deep | bool | Perform a deep (recursive) update on subdictionaries. Default: True | True |
**kwargs | dict | Update arguments | {} |
Source code in fw_gear/metadata.py
129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 | |
update_file_metadata(file_, container_type, deep=True, **kwargs)
Update file metadata by name.
Update a file by name in the hierarchy at the given container type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_ | Any | File name (str), SDK file (flywheel.FileEntry) or dictionary from config.json | required |
container_type | str | Type of parent container. | required |
deep | bool | Perform a deep (recursive) update on subdictionaries. Default: True | True |
**kwargs | dict | Update arguments | {} |
Source code in fw_gear/metadata.py
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 | |
update_zip_member_count(path, container_type=None)
Update metadata with zip-member count.
Add the zip member count for files in a given directory the metadata
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path | Path | directory or file. | required |
container_type | Optional[str] | Container type found files should be associated with. | None |
Source code in fw_gear/metadata.py
201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 | |
write(directory, fail_on_validation=False, log_meta=True)
Write metadata to .metadata.json in the given directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
directory | Path | Directory in which to write .metadata.json | required |
fail_on_validation | bool | Fail if engine metadata schema validation has errors. Default: False | False |
log_meta | bool | Also log cleaned metadata. Default: True | True |
Source code in fw_gear/metadata.py
308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 | |
File
Module for class represent Flywheel file object.
File dataclass
Class representing a context agnostic Flywheel file.
The SDK and config.json represent files differently. This object holds all the metadata associated with a file in a predictable manner and allows for converting from SDK or config.json file representations.
Attributes:
| Name | Type | Description |
|---|---|---|
name | str | File name. |
parent_type | str | Container type of file's parent. |
modality | str | File modality. |
type | str | Flywheel file type. |
mimetype | str | File mimetype. |
classification | Dict[str, List[str]] | Classification dictionary. |
tags | List[str] | List of tags. |
info | dict | Custom information dict. |
_local_path | Optional[Path] | Local path to file |
parents | Dict[str, str] | File parents. |
zip_member_count | Optional[int] | File zip member count. |
version | Optional[int] | File version. |
file_id | Optional[str] | File id. |
size | Optional[int] | File size in bytes. |
fw_path(str) | Optional[int] | Path to file in Flywheel. |
Source code in fw_gear/file.py
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 | |
fw_path property
Lazy-load the Flywheel path to the file.
local_path property
Lazy-load the local path by downloading the file on first access.
__del__()
Clean up the temporary file if the source is 'sdk'.
Source code in fw_gear/file.py
221 222 223 224 225 226 227 228 | |
download_file_to_temp(file_) staticmethod
Downloads a file to a temporary directory and returns the path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_ | FileOutput | FW file object. | required |
Returns:
| Name | Type | Description |
|---|---|---|
Path | Path | Local path to the downloaded file. |
Source code in fw_gear/file.py
197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 | |
from_config(file_, context) classmethod
Returns a File object from a GearContext.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_ | dict | Config.json dictionary representing the file. | required |
Source code in fw_gear/file.py
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 | |
from_sdk(file_, context) classmethod
Returns a File from an SDK "file".
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_ | FileOutput | SDK "file" object | required |
Source code in fw_gear/file.py
116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 | |
Logging
Module for default logging configuration and helpers.
This module utilizes an optionally provided configuration dictionary to update default logging configuration.
Alternatives to updating the default logging templates can be >>> log_level = logging.INFO # logging.DEBUG, etc. >>> log_format = '[%(levelname)8s] %(message)s' >>> log_dtfmt = '' >>> logging.basicConfig(level=log_level, format=log_format, datefmt=log_dtfmt)
Or, >>> logging.config.dictConfig(config)
For a dictionary logging configuration, config, as below.
For more in-depth information on python logging, see https://docs.python.org/3.6/library/logging.html.
ErrFilter
Bases: Filter
Class with single boolean function for filtering error log records.
Source code in fw_gear/logging/__init__.py
51 52 53 54 55 56 57 58 | |
OutFilter
Bases: Filter
Class with single boolean function for filtering non-error log records.
Source code in fw_gear/logging/__init__.py
41 42 43 44 45 46 47 48 | |
configure_logging(default_config_name='info', update_config=None)
Configures logging with default settings or a dictionary of log settings.
Configures the logging for the gear-toolkit module using one of either two default json templates (info, debug -- see below). Whichever one is used can be updated with an optional update_config dictionary.
Logging sent to DEBUG, ERROR, and CRITICAL will be filtered and sent to stderr.
Logging sent to INFO and WARNING will be filtered and sent to stdout.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
default_config_name | str | A string, 'info' or 'debug', indicating the default template to use. Defaults to 'info'. | 'info' |
update_config | dict | A dictionary containing the keys, subkeys, and values of the templates to update. Defaults to | None |
Example
.. code-block:: python
config = {
"formatters": {
"gear": {
"format": "[TEST] %(levelname)4s %(message)s"
}
}
}
configure_logging(update_config = config)
will update the ['formatters']['gear']['format'] value in the info. Default configuration below.
Note
Default logging configuration dictionaries are stored in defaults/<prefix>.json.
info:
.. code-block:: json
{
"version": 1,
"filters": {
"out_filter": {
"()": "PlaceHolder",
"param": "noshow"
},
"err_filter": {
"()": "PlaceHolder",
"param": "noshow"
}
},
"formatters": {
"gear": {
"format":"[%(asctime)s - %(levelname)s - %(name)s] %(message)s",
"datefmt": "%Y%m%d"
}
},
"handlers": {
"gear": {
"level":"INFO",
"formatter": "gear",
"class":"logging.StreamHandler"
}
},
"loggers": {
"": {
"handlers": ["gear"],
"level":"INFO"
}
},
"disable_existing_loggers": false
}
debug:
.. code-block:: json
{
"version": 1,
"filters": {
"out_filter": {
"()": "PlaceHolder",
"param": "noshow"
},
"err_filter": {
"()": "PlaceHolder",
"param": "noshow"
}
},
"formatters": {
"gear": {
"format": "[%(asctime)s - %(levelname)s - %(name)s:%(lineno)d] %(message)s",
"datefmt": "%Y%m%d"
}
},
"handlers": {
"gear": {
"level":"DEBUG",
"formatter": "gear",
"class":"logging.StreamHandler"
}
},
"loggers": {
"": {
"handlers": ["gear"],
"level":"DEBUG"
}
},
"disable_existing_loggers": false
}
Source code in fw_gear/logging/__init__.py
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 | |
Invocation Schema
Flywheel Gear Invocation Schema Generator.
derive_invocation_schema(manifest)
Creates an invocation schema from a gear manifest.
This can be used to validate the files and configuration offered to run a gear.
Source code in fw_gear/invocation_schema.py
65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 | |
format_json(d)
Convenience method to pretty-print JSON.
Source code in fw_gear/invocation_schema.py
22 23 24 | |
get_master_gear_schema()
Return gear spec schema.
Source code in fw_gear/invocation_schema.py
44 45 46 47 48 49 50 51 52 53 | |
get_master_schema()
Returns the master schema. This will load it from disk and cache in memory.
Source code in fw_gear/invocation_schema.py
33 34 35 36 37 38 39 40 41 | |
isolate_config_invocation(invocation)
Given an invocation schema, isolate just the config portion.
Useful to validate configuration options separately from files.
Source code in fw_gear/invocation_schema.py
170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 | |
isolate_file_invocation(invocation, input_name)
Given an invocation schema, isolate just a specific file.
Useful to validate a single input.
Source code in fw_gear/invocation_schema.py
153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 | |
load_json_from_file(path)
Loads a JSON file from disk and returns the parsed map.
Source code in fw_gear/invocation_schema.py
27 28 29 30 | |
validate_invocation(manifest, invocation)
Validates a configuration against a gear's manifest.
Expect a jsonschema.exceptions.ValidationError error to be raised on failure.
Source code in fw_gear/invocation_schema.py
187 188 189 190 191 192 193 194 | |
validate_manifest(manifest)
Validates a gear manifest against the master schema.
Expect a jsonschema.exceptions.ValidationError error to be raised on failure.
Source code in fw_gear/invocation_schema.py
56 57 58 59 60 61 62 | |
Utils Module
Core Utilities
Utility functions for fw-gears, including data processing and SDK helpers.
convert_nan_in_dict(data)
Convert NaN values in a dictionary to None.
Source code in fw_gear/utils/utils_helpers.py
41 42 43 44 | |
deep_merge(base, **update)
Recursive merging of update dict on base dict.
Instead of updating only top-level keys, deep_merge recurse down to perform a "nested" update.
Source code in fw_gear/utils/utils_helpers.py
47 48 49 50 51 52 53 54 55 56 57 | |
trim(obj)
Trim object for printing.
Source code in fw_gear/utils/utils_helpers.py
60 61 62 | |
trim_lists(obj)
Replace a long list with a representation.
List/Arrays greater than 5 in length will be replaced with the first two items followed by ... then the last two items
Source code in fw_gear/utils/utils_helpers.py
65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 | |
install_requirements(req_file)
Install requirements from a file programmatically
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
req_file | str | Path to requirements file | required |
Raises:
| Type | Description |
|---|---|
SystemExit | If there was an error from pip |
Source code in fw_gear/utils/utils_helpers.py
97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 | |
SDK Helpers
get_container_from_ref(client, ref)
Returns the container from its reference.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
client | Client | Authenticated Flywheel SDK client | required |
ref | dict | A dictionary with type and id keys defined. | required |
Returns:
| Name | Type | Description |
|---|---|---|
Container | A flywheel container. |
Source code in fw_gear/utils/sdk_helpers.py
13 14 15 16 17 18 19 20 21 22 23 24 25 | |
get_parent(client, container)
Returns parent container of container input.
Source code in fw_gear/utils/sdk_helpers.py
28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 | |
setup_gear_run(client, gear_name, gear_args)
Setup gear run for a specified gear with provided gear arguments.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
client | Client | Authenticated Flywheel SDK client. | required |
gear_name | str | Name of the gear to run. | required |
gear_args | dict | Dictionary of gear inputs and configuration. | required |
Raises:
| Type | Description |
|---|---|
ValueError | If the specified gear does not exist. |
ValueError | If a required input / configuration for the gear is missing. |
Returns:
| Name | Type | Description |
|---|---|---|
Tuple | Tuple[GearDoc, dict, dict] | A tuple containing an object of the gear document, input dictionary, and configuration dictionary. |
Source code in fw_gear/utils/sdk_helpers.py
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 | |
Context Utilities
report_open_fds(sockets_only)
Decorator which reports on number of open file descriptors before and after a function.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sockets_only | bool | Report on only sockets, or all open FDs. | required |
Source code in fw_gear/utils/contextutils.py
94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 | |
report_usage_stats(interval=1.0, save_output=False, plot=True, save_dir='/flywheel/v0/output', cpu_usage=True, disk_usage=True, mem_usage=True, net_usage=True, disk_monitor_dirs=None, output_format=None)
Decorator to report on resource usage during a function.
Optional configuration to:
- Change the update interval,
- Turn off certain usage measurements
- Adjust outputs save location
- Adjust output types
- Adjust which disk locations are monitored for disk usage.
Source code in fw_gear/utils/contextutils.py
133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 | |
sdk_delete_404_handler(client)
Ignore 404 errors on retries of deletes.
Source code in fw_gear/utils/contextutils.py
76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 | |
sdk_post_retry_handler(client)
Patch the SDK session object to retry on specific errors
Safe to use on
- updating container info
- updating file classification
- adding notes
- adding tags
Not safe to use on
- Creating containers
- Uploading files
Source code in fw_gear/utils/contextutils.py
41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | |
Archive/ZIP Management
Modules for zip manager utilities.
get_config_from_zip(zipfile_path, search_str='_config\\.json')
Extracts and returns configuration data from a JSON file within a ZIP archive.
This function searches for a file within zipfile_path whose name matches the search_str pattern and returns its contents as a Python dictionary.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
zipfile_path | str | Absolute path to the ZIP file to be searched. | required |
search_str | str | Regular expression pattern for identifying the target JSON file. Defaults to '_config.json'. | '_config\\.json' |
Returns:
| Type | Description |
|---|---|
Optional[Dict] | dict or None: A dictionary containing the extracted JSON data if found, |
Optional[Dict] | otherwise None. |
Example
config = get_config_from_zip('/flywheel/v0/inputs/ZIP/file.zip')
Source code in fw_gear/utils/archive/zip_manager.py
39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | |
unzip_archive(zipfile_path, output_dir, dry_run=False)
Extracts the contents of a ZIP archive to a specified directory.
This function extracts the contents of zipfile_path into output_dir. Extraction is performed only if dry_run is set to False.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
zipfile_path | str | Absolute path to the ZIP file. | required |
output_dir | str | Absolute path to the directory where the extracted files will be placed. | required |
dry_run | bool | If True, the extraction is skipped, allowing for debugging without modifying files. | False |
Examples:
>>> unzip_archive('/flywheel/v0/inputs/ZIP/file.zip', '/flywheel/v0/work/')
>>> unzip_archive('/flywheel/v0/inputs/ZIP/file.zip', '/flywheel/v0/work/', dry_run=True)
Source code in fw_gear/utils/archive/zip_manager.py
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | |
zip_info(zipfile_path)
Retrieves a list of relative file paths stored in a ZIP archive.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
zipfile_path | str | Absolute path to the ZIP archive. | required |
Returns:
| Type | Description |
|---|---|
List[str] | List[str]: A sorted list of relative file paths contained in the ZIP archive. |
Example
file_info = zip_info('/path/to/zipfile.zip')
Source code in fw_gear/utils/archive/zip_manager.py
141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 | |
zip_output(root_dir, source_dir, output_zip_filename, dry_run=False, exclude_files=None)
Compresses a directory into a ZIP archive.
This function creates a ZIP file containing the contents of source_dir relative to root_dir. The resulting ZIP archive is saved at output_zip_filename.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root_dir | str | The root directory to zip relative to. | required |
source_dir | str | Subdirectory within | required |
output_zip_filename | str | Full path of the output ZIP archive. | required |
dry_run | bool | If True, skips actual compression for debugging. | False |
exclude_files | Optional[List[str]] | List of file paths to exclude from the ZIP archive. Defaults to None. | None |
Raises:
| Type | Description |
|---|---|
FileNotFoundError | If |
Examples:
>>> zip_output('/flywheel/v0/work', 'gear_output', 'output.zip')
>>> zip_output('/flywheel/v0/work', 'gear_output', 'output.zip', dry_run=True)
.. code-block:: python
zip_output(
'/flywheel/v0/work', 'gear_output', 'output.zip',
exclude_files=['sub_dir1/file1.txt', 'sub_dir2/file2.txt']
)
Source code in fw_gear/utils/archive/zip_manager.py
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 | |
FreeSurfer License
Utilities for installing the Freesurfer license file in the expected location for algorithm execution.
check_project_for_license(context)
Retrieves the FreeSurfer license text from a Flywheel project metadata.
This function checks the project's info field for a license key under either FREESURFER-LICENSE or FREESURFER_LICENSE.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context | GearContext | The gear context containing configuration and inputs. | required |
Returns:
| Type | Description |
|---|---|
Optional[str] | Optional[str]: The extracted license text if found, otherwise None. |
Source code in fw_gear/utils/licenses/freesurfer.py
192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 | |
find_license_info(context)
Retrieves the FreeSurfer license text from available sources.
It checks for the license in the following order: 1) Input file (freesurfer_license_file). 2) Config parameter (freesurfer_license_key). 3) Flywheel project metadata (FREESURFER_LICENSE).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context | GearContext | The gear context containing configuration and inputs. | required |
Returns:
| Type | Description |
|---|---|
Optional[str] | Optional[str]: The extracted license text, or None if not found. |
Source code in fw_gear/utils/licenses/freesurfer.py
106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 | |
get_fs_license_path(fs_license_path=None)
Determines the path where the FreeSurfer license should be installed.
If a path is provided, ensures it ends with 'license.txt'. Otherwise, it attempts to infer the path from the $FREESURFER_HOME environment variable or by locating the FreeSurfer installation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
fs_license_path | Optional[PathLike] | Custom path for the license file. Defaults to None. | None |
Returns:
| Name | Type | Description |
|---|---|---|
Path | Path | The resolved path for the FreeSurfer license file. |
Raises:
| Type | Description |
|---|---|
RuntimeError | If the FreeSurfer installation path cannot be determined. |
Source code in fw_gear/utils/licenses/freesurfer.py
63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 | |
install_freesurfer_license(context, fs_license_path=None)
Installs the FreeSurfer license file in the expected location.
The license text is obtained using one of the following methods, in order of priority:
1) Provided as an input file (freesurfer_license_file in the manifest). 2) Retrieved from the "freesurfer_license_key" config parameter. 3) Extracted from a Flywheel project's info metadata (FREESURFER_LICENSE).
Once retrieved, the license file is written to the FreeSurfer top-level directory, enabling FreeSurfer usage within the gear.
See How to include a Freesurfer license file... <https://docs.flywheel.io/user/compute/gears/freesurfer/user_including_a_freesurfer_license_file_to_run_a_freesurfer_or_fmriprep_gear/>_
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context | GearContext | The gear context with core functionality. | required |
fs_license_path | str | The path where the license should be installed, typically | None |
Raises:
| Type | Description |
|---|---|
FileNotFoundError | If the license cannot be found. |
Examples:
>>> from fw_gear.utils.licenses.freesurfer import install_freesurfer_license
>>> install_freesurfer_license(context, '/opt/freesurfer/license.txt')
Source code in fw_gear/utils/licenses/freesurfer.py
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | |
isolate_key_name(context)
Identifies the relevant FreeSurfer license key from the gear configuration.
This function searches for a predefined set of FreeSurfer license key names within the gear's configuration and returns the first matching key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context | GearContext | The gear context containing configuration. | required |
Returns:
| Type | Description |
|---|---|
Optional[str] | Optional[str]: The matched license key name if found, otherwise None. |
Source code in fw_gear/utils/licenses/freesurfer.py
135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 | |
read_input_license(context)
Reads and returns the FreeSurfer license text from an input file.
This function retrieves the license file specified as freesurfer_license_file in the gear's inputs and returns its contents as a formatted string.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context | GearContext | The gear context containing configuration and inputs. | required |
Returns:
| Type | Description |
|---|---|
Optional[str] | Optional[str]: The license text as a formatted string, or None if the file is not found. |
Example
license_text = read_input_license(context) if license_text: print("License successfully retrieved.")
Source code in fw_gear/utils/licenses/freesurfer.py
166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 | |
write_license_info(fs_license_path, license_info)
Writes the FreeSurfer license text to the specified file.
This function ensures that the target directory exists before writing the license text to the specified path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
fs_license_path | Path | Path to the FreeSurfer license file. | required |
license_info | str | The license text to be written. | required |
Raises:
| Type | Description |
|---|---|
OSError | If unable to write the license file. |
Source code in fw_gear/utils/licenses/freesurfer.py
224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 | |
Command Wrapper Module
Command Execution
Wrapper for executing shell commands using Python's subprocess library.
This module provides a streamlined interface for constructing and executing command-line processes. It simplifies handling subprocess calls by dynamically building command-line arguments from configuration parameters (e.g., config.json), managing environment variables, and providing structured error handling.
Features: - Builds command-line arguments dynamically from key-value parameter mappings. - Executes shell commands with logging, error handling, and real-time output streaming. - Supports dry-run mode for debugging without executing commands. - Captures stdout and stderr, raising an exception for non-zero exit codes.
Examples:
>>> command = ["ls"]
>>> param_list = {"l": True, "a": True, "h": True}
>>> command = build_command_list(command, param_list)
>>> command
["ls", "-l", "-a", "-h"]
>>> exec_command(command)
Another example using parameters with values:
>>> command = ["du"]
>>> params = {"a": True, "human-readable": True, "max-depth": 3}
>>> command = build_command_list(command, params)
>>> command
["du", "-a", "--human-readable", "--max-depth=3"]
>>> params = {"dir1": ".", "dir2": "/tmp"}
>>> command = build_command_list(command, params, include_keys=False)
>>> command
["du", ".", "/tmp"]
>>> exec_command(command)
These examples demonstrate how command-line parameters can be dynamically constructed and executed from gear configuration options.
build_command_list(command, param_list, include_keys=True)
Constructs a command-line argument list for subprocess execution.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
command | List[str] | Base command (e.g., | required |
param_list | Dict[str, any] | Dictionary of key-value pairs representing command-line arguments. - Boolean values ( | required |
include_keys | bool | Whether to include parameter keys in the command list. Defaults to True. | True |
Returns:
| Type | Description |
|---|---|
List[str] | List[str]: A formatted command list suitable for |
Example
command = ["du"] params = {"a": True, "human-readable": True, "max-depth": 3} command = build_command_list(command, params) command ["du", "-a", "--human-readable", "--max-depth=3"]
Source code in fw_gear/utils/wrapper/command.py
92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 | |
exec_command(command, dry_run=False, environ=None, shell=False, stream=False, stream_mode=None, throttle_sec=1.0, logfile=None)
Executes a shell command using the subprocess module with flexible output handling.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
command | List[str] | List of command-line arguments, starting with the command itself. | required |
dry_run | bool | If True, logs the command without executing it. Returns ("", "", 0). Defaults to False. | False |
environ | Optional[Dict[str, str]] | Additional environment variables to merge with the current environment. Defaults to None. | None |
shell | bool | Run via system shell (required for redirects/pipes). When True, only literal tokens (">", ">>", "<", "1>", "1>>", "2>", "2>>", "2>&1", "1>&2") are left unquoted; all other arguments are safely quoted. Defaults to False. | False |
stream | bool | Enable live streaming of output line-by-line (deadlock-safe). When True, stderr is merged into stdout. Defaults to False. | False |
stream_mode | Optional[str] | Controls what is printed during streaming: - "all": Print every line in real-time (default) - "filter_only": Only print lines matching important patterns (errors, warnings, etc.) - "throttled": Print important lines immediately, rate-limit others Full output is always captured regardless of mode. Defaults to "all". | None |
throttle_sec | float | Minimum time in seconds between printing non-important lines in "throttled" mode. Defaults to 1.0. | 1.0 |
logfile | Optional[Union[str, PathLike]] | Path to append the full unfiltered output stream. Works with all streaming modes. Defaults to None. | None |
Returns:
| Type | Description |
|---|---|
Tuple[str, str, int] | Tuple[str, str, int]: A tuple containing: - stdout (str): Standard output from the command. When stream=True, contains merged stdout+stderr. - stderr (str): Standard error output. Empty string ("") when stream=True. - returncode (int): Command exit status (0 for success). |
Raises:
| Type | Description |
|---|---|
RuntimeError | If the command returns a non-zero exit code, or if the logfile cannot be opened for writing. |
Notes
- Streaming is automatically disabled when shell redirection tokens are detected
- Important lines are detected via regex pattern (configurable via EXEC_ALWAYS_PRINT_RE env var)
- In "throttled" mode, a final suppressed line count is printed at the end
- The logfile path is validated before command execution (fail-fast behavior)
Example
command = ["du"] params = {"a": True, "human-readable": True, "max-depth":3} command = build_command_list(command, params) params = {"dir1":".","dir2":"/tmp"} command = build_command_list(command, params, include_keys=False) exec_command(command)
1) Basic run (no live streaming)
stdout, stderr, rc = exec_command(command)
2) Live stream only important lines; keep full transcript
stdout, stderr, rc = exec_command( ... command, stream=True, stream_mode="filter_only", logfile="du.stream.log" ... )
3) If you truly need shell redirection (no streaming)
cmd2 = command + [">>", "du.out.log", "2>&1"] stdout, stderr, rc = exec_command(cmd2, shell=True, stream=False)
Source code in fw_gear/utils/wrapper/command.py
143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 | |
Nipype Integration
A wrapper module to generate a Nipype interface.
GearContextInterfaceBase
Bases: SimpleInterface
Factory class for generating Nipype interfaces from Flywheel gear manifests.
This class dynamically constructs a Nipype interface based on a gear's manifest.json, allowing for seamless integration into Nipype workflows.
Source code in fw_gear/utils/wrapper/nipype.py
135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 | |
factory(manifest=None) classmethod
Returns a new GearContextInterface instance for the given manifest.
Source code in fw_gear/utils/wrapper/nipype.py
150 151 152 153 154 155 156 157 158 159 160 161 162 | |
get_class_name(manifest=None) staticmethod
Returns the class name from the gear name in manifest.
Example: 'my-gear' -> 'MyGearContextInterface'
Source code in fw_gear/utils/wrapper/nipype.py
141 142 143 144 145 146 147 148 | |
GearContextInterfaceInputSpec
Bases: BaseInterfaceInputSpec
Defines the input specification for a GearContextInterface.
Attributes:
| Name | Type | Description |
|---|---|---|
config_dict | Dict | The Flywheel |
Source code in fw_gear/utils/wrapper/nipype.py
61 62 63 64 65 66 67 68 69 | |
GearContextInterfaceOutputSpecBase
Factory class for generating Nipype output specifications from Flywheel gear manifests.
This class dynamically builds Nipype output specifications by extracting parameter definitions from the manifest.json of a Flywheel gear.
Source code in fw_gear/utils/wrapper/nipype.py
72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 | |
factory(manifest) classmethod
Returns an OutputSpec class based on manifest.json
Source code in fw_gear/utils/wrapper/nipype.py
108 109 110 111 112 113 114 115 116 117 118 | |
get_class_name(manifest=None) staticmethod
Return class name from gear name in manifest.
Example: 'my-gear' -> MyGearContextInterface
Source code in fw_gear/utils/wrapper/nipype.py
98 99 100 101 102 103 104 105 106 | |
get_outputspec_attr(manifest=None) staticmethod
Returns OutputSpec attributes.
Source code in fw_gear/utils/wrapper/nipype.py
79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | |
get_traits_object(datatype, description=None)
Returns the corresponding traits object for a given input datatype.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
datatype | str | The data type to map to a | required |
description | Optional[str] | A description for the trait. Defaults to None. | None |
Returns:
| Type | Description |
|---|---|
Union[None, ContextType] | Union[None, ContextType]: The corresponding |
Raises:
| Type | Description |
|---|---|
TypeError | If the datatype is not recognized. |
Source code in fw_gear/utils/wrapper/nipype.py
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 | |
HPC Integration
Set up tools for HPC integration using Singularity.