Automatically Extending a Parameter Study

Automatically Extending a Parameter Study#

Warning

The waves build feature is considered experimental. The feature should be treated like an alpha feature whose design, behavior, and interface may change without notice. This feature is documented to solicit user feedback on preferred behavior and usefulness.

Warning

This tutorial is in a draft outline state. It is provided as reference and will be fleshed out in future releases. Be sure to check back in on this tutorial or watch the Changelog for updates!

References#

Environment#

SCons and WAVES can be installed in a Conda environment with the Conda package manager. See the Conda installation and Conda environment management documentation for more details about using Conda.

Note

The SALib and numpy versions may not need to be this strict for most tutorials. However, Tutorial: Sensitivity Study uncovered some undocumented SALib version sensitivity to numpy surrounding the numpy v2 rollout.

  1. Create the tutorials environment if it doesn’t exist

    $ conda create --name waves-tutorial-env --channel conda-forge waves 'scons>=4.6' matplotlib pandas pyyaml xarray seaborn 'numpy>=2' 'salib>=1.5.1' pytest

    PS > conda create --name waves-tutorial-env --channel conda-forge waves scons matplotlib pandas pyyaml xarray seaborn numpy salib pytest

  2. Activate the environment

    $ conda activate waves-tutorial-env

    PS > conda activate waves-tutorial-env

Some tutorials require additional third-party software that is not available for the Conda package manager. This software must be installed separately and either made available to SConstruct by modifying your system’s PATH or by modifying the SConstruct search paths provided to the waves.scons_extensions.add_program() method.

Directory Structure#

  1. Create and change to a new project root directory to house the tutorial files if you have not already done so. For example

$ mkdir -p ~/waves-tutorials
$ cd ~/waves-tutorials
$ pwd
/home/roppenheimer/waves-tutorials

PS > New-Item $HOME\waves-tutorials -ItemType "Directory"
PS > Set-Location $HOME\waves-tutorials
PS > Get-Location

Path
----
C:\Users\roppenheimer\waves-tutorials

Note

If you skipped any of the previous tutorials, run the following commands to create a copy of the necessary tutorial files.

$ pwd
/home/roppenheimer/waves-tutorials
$ waves fetch --overwrite --tutorial 8 && mv tutorial_08_data_extraction_SConstruct SConstruct
WAVES fetch
Destination directory: '/home/roppenheimer/waves-tutorials'

PS > Get-Location

Path
----
C:\Users\roppenheimer\waves-tutorials

PS > waves fetch --overwrite --tutorial 8 && Move-Item tutorial_08_data_extraction_SConstruct SConstruct -Force
WAVES fetch
Destination directory: 'C:\Users\roppenheimer\waves-tutorials'

  1. Download and copy the tutorial_08_data_extraction.scons file to a new file named tutorial_extend_study.scons with the WAVES Command-Line Utility fetch subcommand.

$ pwd
/home/roppenheimer/waves-tutorials
$ waves fetch --overwrite tutorials/tutorial_08_data_extraction.scons && cp tutorial_08_data_extraction.scons tutorial_extend_study.scons
WAVES fetch
Destination directory: '/home/roppenheimer/waves-tutorials'

PS > Get-Location

Path
----
C:\Users\roppenheimer\waves-tutorials

PS > waves fetch --overwrite tutorials\tutorial_08_data_extraction.scons && Copy-Item tutorial_08_data_extraction.scons tutorial_extend_study.scons
WAVES fetch
Destination directory: 'C:\Users\roppenheimer\waves-tutorials'

SConscript#

A diff against the tutorial_08_data_extraction.scons file from Tutorial 08: Data Extraction is included below to help identify the differences between the two parameter generators and workflows.

waves-tutorials/tutorial_extend_study.scons

--- /home/runner/work/waves/waves/build/docs/tutorials_tutorial_08_data_extraction.scons
+++ /home/runner/work/waves/waves/build/docs/tutorials_tutorial_extend_study.scons
@@ -13,7 +13,7 @@
 
 import waves
 
-from modsim_package.python.rectangle_compression_cartesian_product import parameter_schema
+from modsim_package.python.rectangle_compression_sobol_sequence import parameter_schema
 
 # Inherit the parent construction environment
 Import("env")
@@ -24,6 +24,11 @@
 build_directory = pathlib.Path(Dir(".").abspath)
 workflow_name = build_directory.name
 parameter_study_file = build_directory / "parameter_study.h5"
+extended_parameter_study_file = build_directory / "extended_parameter_study.h5"
+simulation_constants = {
+    "global_seed": 1,
+    "displacement": -0.01,
+}
 
 # Collect the target nodes to build a concise alias for all targets
 workflow = []
@@ -31,10 +36,11 @@
 
 # Comment used in tutorial code snippets: marker-2
 
-# Parameter Study with Cartesian Product
-parameter_generator = waves.parameter_generators.CartesianProduct(
+# Parameter Study with Sobol Sequence
+parameter_generator = waves.parameter_generators.SobolSequence(
     parameter_schema(),
-    previous_parameter_study=parameter_study_file,
+    previous_parameter_study=extended_parameter_study_file,
+    scramble=False,
 )
 parameter_generator.write(output_file=parameter_study_file)
 
@@ -43,7 +49,7 @@
 # Parameterized targets must live inside current simulation_variables for loop
 for set_name, parameters in parameter_generator.parameter_study_to_dict().items():
     set_path = pathlib.Path(set_name)
-    simulation_variables = parameters
+    simulation_variables = {**parameters, **simulation_constants}
 
     # Comment used in tutorial code snippets: marker-4
 
@@ -159,6 +165,22 @@
 
 # Comment used in tutorial code snippets: marker-6
 
+# Post-processing
+post_processing_source = [
+    pathlib.Path(set_name) / "rectangle_compression_datasets.h5"
+    for set_name in parameter_generator.parameter_study_to_dict()
+]
+workflow.extend(
+    env.PythonScript(
+        target=[extended_parameter_study_file.name],
+        source=["#/modsim_package/python/tutorial_extend_study.py", parameter_study_file.name, *post_processing_source],
+        subcommand_options=(
+            "--input-file ${SOURCES[2:].abspath} --parameter-study-file ${SOURCES[1].abspath}"
+            " --output-file ${TARGETS[0].abspath}"
+        ),
+    )
+)
+
 # Collector alias based on parent directory name
 env.Alias(workflow_name, workflow)
 env.Alias(f"{workflow_name}_datacheck", datacheck)

This post-processing workflow should look nearly identical to the post-processing tasks introduced in Tutorial 09: Post-Processing. The difference in this tutorial is that there are two parameter study files to track. The configuration time parameter study is written to parameter_study.h5 as before, but the previous_parameter_study now reads from a new file, extended_parameter_study.h5, which does not exist until the workflow is run at least once.

A post-processing task is written to generate the extended_parameter_study.h5 as a final step of the workflow. As shown in the next section where the post-processing file is introduced, this task reads the parameter_study.h5 file and the associated simulation results, performs some analysis, and either writes new parameter sets into the extended_parameter_study.h5 file or simply creates a copy of the parameter_study.h5 file. Because the configuration time parameter study reads from extended_parameter_study.h5, the next time the workflow is executed, it will pick up any new parameter sets and configure the new sets as part of the workflow. This process of performing new work continues until the the post-processing task no longer recommends new parameter sets.

Both the configuration time parameter study definition found in the SConscript file and the post-processing parameter study extension take advantage of the WAVES feature that preserves set name assignments of the original parameter sets. At each re-execution of the workflow, parameter sets that have already been seen preserve their original set names and set contents. Those set-named output files already exist, so re-work is minimized to the newest parameter sets.

The WAVES build subcommand was written to help automate the process of re-execution. The command is written to continue re-submitting the normal scons command until SCons reports that there is no new work to complete, as when the post-processing script no longer recommends new parameter sets, or until the maximum number of iterations has been reached.

Post-processing script#

  1. In the waves-tutorials/modsim_package/python directory, create a file called tutorial_extend_study.py using the contents below.

waves-tutorials/modsim_package/python/tutorial_extend_study.py

"""Read simulation output, process data, recommend new parameter sets, write updated parameter study."""

import argparse
import copy
import pathlib
import shutil
import typing

import waves

from modsim_package.python.rectangle_compression_sobol_sequence import parameter_schema


def main(
    input_files: typing.Sequence[pathlib.Path],
    parameter_study_file: pathlib.Path,
    output_file: pathlib.Path | None = None,
) -> None:
    """Read simulation output, process data, recommend new parameter sets, write updated parameter study."""
    set_count = len(input_files)
    if set_count < 8:
        new_schema = copy.deepcopy(parameter_schema())
        new_schema["num_simulations"] = set_count + 2
        new_generator = waves.parameter_generators.SobolSequence(
            new_schema,
            previous_parameter_study=parameter_study_file,
            scramble=False,
        )
        new_generator.write(output_file=output_file)
    else:
        shutil.copyfile(parameter_study_file, output_file)


def get_parser() -> argparse.ArgumentParser:
    """Return parser for CLI options."""
    script_name = pathlib.Path(__file__)

    prog = f"python {script_name.name} "
    cli_description = "Read Xarray Datasets, process data, recommend new parameter sets, write updated parameter study."
    parser = argparse.ArgumentParser(description=cli_description, prog=prog)
    required_named = parser.add_argument_group("required named arguments")
    required_named.add_argument(
        "-i",
        "--input-file",
        nargs="+",
        type=pathlib.Path,
        required=True,
        help="The Xarray Dataset file(s)",
    )
    required_named.add_argument(
        "-p",
        "--parameter-study-file",
        type=pathlib.Path,
        help="An optional h5 file with a WAVES parameter study Xarray Dataset (default: %(default)s)",
    )
    parser.add_argument(
        "-o",
        "--output-file",
        type=pathlib.Path,
        help=("The parameter study with new parameter sets."),
    )
    return parser


if __name__ == "__main__":
    parser = get_parser()
    args = parser.parse_args()
    main(
        input_files=args.input_file,
        output_file=args.output_file,
        parameter_study_file=args.parameter_study_file,
    )

This post-processing script is a stripped down version of the post_processing.py script introduced in Tutorial 09: Post-Processing. The tutorial_extend_study.py script simply checks the number of simulations and increases the size of the parameter study to a hardcoded value. This simple implementation is intended only to demonstrate the workflow and task structure of an automatically extending workflow.

In real projects, the purpose of extending a parameter is to perform an optimization or uncertainty quantification workflow. A post-processing script more similar to the post_processing.py script of Tutorial 09: Post-Processing or Tutorial: Sensitivity Study would perform some kind of sensitivity analysis or uncertainty quantification and recommend new parameter sets to meet an objective function. Such a workflow may also require several post-processing steps, such as training a reduced-order or surrogate model which is used to investigate a larger parameter space before recommending parameter sets to investigate with the original, higher cost simulations.

In all these examples, the outcome of the post-processing step should be new parameter sets that are added to the original parameter study. On re-execution of the workflow, the new parameter sets are read into the workflow definition and the workflow picks up with the new parameter sets without re-computing the old ones.

SConstruct#

A diff against the SConstruct file from Tutorial 08: Data Extraction is included below to help identify the changes made in this tutorial.

waves-tutorials/SConstruct

--- /home/runner/work/waves/waves/build/docs/tutorials_tutorial_08_data_extraction_SConstruct
+++ /home/runner/work/waves/waves/build/docs/tutorials_tutorial_extend_study_SConstruct
@@ -99,7 +99,8 @@
     "tutorial_05_parameter_substitution.scons",
     "tutorial_06_include_files.scons",
     "tutorial_07_cartesian_product.scons",
-    "tutorial_08_data_extraction.scons",
+    "tutorial_07_sobol_sequence.scons",
+    "tutorial_extend_study.scons",
 ]
 for workflow in workflow_configurations:
     workflow_build_directory = build_directory / pathlib.Path(workflow).stem

Build Targets#

  1. Build the new targets

$ pwd
/home/roppenheimer/waves-tutorials
$ waves build --max-iterations=4 tutorial_extend_study --jobs=4

Output Files#

$ pwd
/home/roppenheimer/waves-tutorials
$ tree build/tutorial_extend_study -d
build/tutorial_extend_study
|-- parameter_set0
|-- parameter_set1
|-- parameter_set2
|-- parameter_set3
|-- parameter_set4
|-- parameter_set5
|-- parameter_set6
`-- parameter_set7

8 directories