Metadata-Version: 2.4
Name: failureflags
Version: 1.1.0
Summary: Failure Flags is a Python SDK for building application-level chaos experiments and reliability tests using the Gremlin Fault Injection platform.
Author-email: Jeff Nickoloff <jeff.nickoloff@gremlin.com>
Project-URL: Code, https://github.com/gremlin/failure-flags-python
Project-URL: Bug Tracker, https://github.com/gremlin/failure-flags-python/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# failure-flags

Failure Flags is a python SDK for building application-level chaos experiments and reliability tests using the Gremlin Fault Injection platform. This library works in concert with Gremlin-Lambda, a Lambda Extension; or Gremlin-Sidecar, a container sidecar agent. This architecture minimizes the impact to your application code, simplifies configuration, and makes adoption painless.

Just like feature flags, Failure Flags are safe to add to and leave in your application. Failure Flags will always fail safe if it cannot communicate with its sidecar or its sidecar is misconfigured.

Take three steps to run an application-level experiment with Failure Flags:

1. Instrument your code with this SDK
2. Configure and deploy your code along side one of the Failure Flag sidecars
3. Run an Experiment with the console, API, or command line

## Instrumenting Your Code

You can get started by adding failureflags to your package dependencies:

```sh
pip install failureflags
```

Then instrument the part of your application where you want to inject faults. 

```python
from failureflags import FailureFlag

...

FailureFlag(name: 'flagname', labels: {}).invoke()

...
```

The best spots to add a failure flag are just before or just after a call to one of your network dependencies like a database or other network service. Or you can instrument your request handler and affect the way your application responses to its callers. Here's a simple Lambda example:

```python
# Change 1: Bring in the failureflags module
from failureflags import FailureFlag

import os
import logging
import time
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all

logger = logging.getLogger()
logger.setLevel(logging.INFO)
patch_all()

def lambda_handler(event, context):
    start = time.time()

    # Change 2: add a FailureFlag to your code
    FailureFlag("http-ingress", {}, debug=True).invoke()

    end = time.time()
    return {
        'statusCode': 200,
        'headers': {
            'Content-Type': 'application/json'
        },
        'body': {
            'processingTime': f"{start - end}",
            'isActive': active,
            'isImpacted': impacted
        }
    }
```

## Enabling the SDK in your Environment

*Don't forget to enable the SDK by setting the FAILURE_FLAGS_ENABLED environment variable to `true`, `yes`, or `1`!* Any other value, including `false`, and any unset variable leaves the SDK short-circuited, and no attempt to fetch experiments will be made.

The rest of the SDK's configuration comes from the environment too, and every variable is optional:

| Variable | Default | Description |
| --- | --- | --- |
| `FAILURE_FLAGS_ENABLED` | unset | Set to `true`, `yes`, or `1` (case-insensitive) to enable the SDK. Anything else disables it. |
| `FAILURE_FLAGS_ENDPOINT` | `http://localhost:5032/experiment` | The full sidecar URL. Takes precedence over the host and port variables below. Must be `http` or `https`; anything else is ignored. |
| `GREMLIN_SIDECAR_HOST` | `localhost` | The sidecar host, matching the sidecar's own configuration namespace. |
| `GREMLIN_SIDECAR_PORT` | `5032` | The sidecar port. The sidecar reads this same variable as a *listen address*, so `6032`, `:6032`, `0.0.0.0:6032`, and `localhost:6032` are all accepted and all mean port 6032. Only the port is used: a listen address says what the sidecar binds, not where to reach it. A port outside 1..65535 falls back to the default. |
| `FAILURE_FLAGS_TIMEOUT_MS` | `1` | The fetch deadline in **milliseconds**. The sidecar is a co-process on loopback, so this is deliberately tight. |

The `endpoint` and `timeout` keyword arguments to `FailureFlag` override the corresponding variables. Mind the units: `timeout` is in seconds (`timeout=.005`), `FAILURE_FLAGS_TIMEOUT_MS` is in milliseconds.

Every variable is read on each call, so a `FailureFlag` constructed at import time still picks up configuration that the process sets later.

### Changed since 1.0.3

`FAILURE_FLAGS_ENABLED` used to enable the SDK whenever the variable was *present*, whatever its value. Setting it to `false`, as the install docs tell proxy-mode users to do, left the SDK live and injecting faults. The value is now parsed, so `false`, `no`, `0`, and `""` all disable the SDK.

Three effect-processing changes bring this SDK back in line with the Go and Node SDKs. Each one used to be a silent no-op, and every one of them changes when a fault actually fires:

- **An experiment with no `rate` is now applied.** An absent or `null` rate means 1.0. Previously the experiment was fetched and reported as active, but nothing was injected, so Gremlin recorded an experiment the application never felt. A `rate` that is present but is not a number in 0..1 is still skipped.
- **A fractional latency is now applied.** JSON has one number type, so `{"latency": 1000}` and `{"latency": 1000.0}` are the same instruction. The SDK accepted only whole numbers, so a fractional latency did nothing, and a fractional `ms` inside a `latency` object reported impact while sleeping zero. Both forms now work, as do numeric strings.
- **A latency clause that resolves to no delay is no longer reported as impact.** `{"latency": {}}`, a negative delay, and a non-numeric `ms` used to return "impacted" after sleeping zero. Infinite and `NaN` delays are rejected outright rather than hanging the caller.

One consequence: `FailureFlag.enabled` is now a read-only property backed by the environment. If you were disabling a flag by assigning to it, patch the environment instead:

```python
# no longer works: raises AttributeError
flag.enabled = False

# do this instead
with unittest.mock.patch.dict(os.environ, {"FAILURE_FLAGS_ENABLED": "false"}):
    flag.invoke()
```

## Extensibility

You can always bring your own behaviors and effects by providing a behavior function. Here's another Lambda example that writes the experiment data to the console instead of changing the application behavior:

```python
# Change 1: Bring in the failureflags module
from failureflags import FailureFlag, defaultBehavior

import os
import logging
import time
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all

logger = logging.getLogger()
logger.setLevel(logging.INFO)
patch_all()

def customBehavior(ff, experiments):
    logger.debug(experiments)
    return defaultBehavior(ff, experiments)

def lambda_handler(event, context):
    start = time.time()

    # Change 2: add a FailureFlag to your code
    FailureFlag("http-ingress", {}, debug=True, behavior=customBehavior, timeout=.005).invoke()

    end = time.time()
    return {
        'statusCode': 200,
        'headers': {
            'Content-Type': 'application/json'
        },
        'body': {
            'processingTime': f"{start - end}",
            'isActive': active,
            'isImpacted': impacted
        }
    }
```

### Providing Metadata to Custom Behaviors

The default effect chain included with the Failure Flags SDK is aware of well-known effect properties including, "latency" and "exception." The user can extend or replace that functionality and use the same properties, or provide their own. For example, suppose a user wants to use a "random jitter" effect that the Standard Chain does not provide. Suppose they wanted to inject a random amount of jitter up to some maximum. They could implement that small extension and make up their own Effect property called, "my-jitter" that specifies that maximum. The resulting Effect Statement would look like:

```json
{ "my-jitter": 500 }
```

They might also combine this with parts of the default chain:

```json
{
  "latency": 1000,
  "my-jitter": 500
}
```

## Alternatively, use it like a Feature Flag

Sometimes you need even more manual control. For example, in the event of an experiment you might not want to make some API call or need to rollback some transaction. In most cases the Exception effect can help, but the `invoke` function also returns a boolean to indicate if there was an experiment. You can use that to create branches in your code like you would for any feature flag.

```python
...
active, impacted, experiments = FailureFlag("myFlag", {}).invoke()
if active and impacted:
  // if there is a running experiment then do this
else:
  // if there is no experiment then do this
...
```

### Pulling the Experiment and Branching Manually

If you want to work with lower-level Experiment data you can use `fetch` directly.

## Building Experiments: Targeting with Selectors

Experiments match specific invocations of a Failure Flag based on its name, and the labels you provide. Experiments define Selectors that the Failure Flags engine uses to determine if an invocation matches. Selectors are simple key to list of values maps. The basic matching logic is every key in a selector must be present in the Failure Flag labels, and at least one of the values in the list for a selector key must match the value in the label.

## Effects and Examples

Once you've instrumented your code and deployed your application with the sidecar you're ready to run an Experiment. None of the work you've done so far describes the Effect during an experiment. You've only marked the spots in code where you want the opportunity to experiment. Gremlin Failure Flags Experiments take an Effect parameter. The Effect parameter is a simple JSON map. That map is provided to the Failure Flags SDK if the application is targeted by a running Experiment. The Failure Flags SDK will process the map according to the default behavior chain or the behaviors you've provided. Today the default chain provides both latency and error Effects.

### Introduce Flat Latency

This Effect will introduce a constant 2000 millisecond delay.

```json
{ "latency": 2000 }
```

### Introduce Minimum Latency with Some Maximum Jitter

This Effect will introduce between 2000 and 2200 milliseconds of latency where there is a pseudo-random uniform probability of any delay between 2000 and 2200.

```json
{
  "latency": {
    "ms": 2000,
    "jitter": 200
  }
}
```

### Throw an Error

This Effect will cause Failure Flags to throw a ValueError with the provided message. This is useful if your application uses Errors with well-known messages.

```json
{ "exception": "this is a custom message" }
```

If your app uses custom error types or other error condition metadata then use the object form of exception. This Effect will cause the SDK to import http.client module and raise an http.client.ImproperConnectionState exception:

```json
{
  "exception": {
    "message": "this is a custom message",
    "module": "http.client",
    "className": "ImproperConnectionState"
  }
}
```

If `module` is omitted the SDK will assume `builtins`. If `className` is omitted the SDK will assume `ValueError`. `name` works as an alias for `className`, so the cross-language error metadata form (`{"message": ..., "name": ...}`) does what you would expect here; `className` wins if you provide both. If the named class cannot be imported the SDK raises a `ValueError` carrying your `message` rather than nothing at all.

### Combining the Two for a "Delayed Exception"

Many common failure modes eventually result in an exception being thrown, but there will be some delay before that happens. Examples include network connection failures, or degradation, or other timeout-based issues.

This Effect Statement will cause a Failure Flag to pause for a full 2 seconds before throwing an exception/error a message, "Custom TCP Timeout Simulation"

```json
{
  "latency": 2000,
  "exception": {
    "message": "Custom TCP Timeout Simulation",
    "className": "TimeoutError"
  }
}
```

