# Creating robust applications by creating test cases
This example shows you how you can easily create (integration) test cases for Burr your project.

With Burr it is easy to foster a test-driven development mindset when developing actions, which are the building blocks of your Burr
application.

Note: writing test cases for GenAI projects can be tricky. The same LLM API calls
can result in different outputs. This means that 'exact' equality tests may not work,
and you'll need to resort to more fuzzy tests like checking for the presence of certain
words or phrases, or using LLMs to grade the output, etc. We aren't opinionated on how you
do this, but in any case, you'll need to write a test case to exercise things, and this
is what we're showing you how to do here.


## Motivation

With LLMs non-deterministic behavior can impact your application's behavior. It is important to create test cases to
ensure that your application is robust and behaves as expected; it is easy to tweak a prompt and break
a particular use case. This example shows you how you can create test cases
for your project from real traces, since part of the struggle is to create test cases that are representative of the
real-world behavior of your application -- Burr can make this process easier.

## Steps
1. Build your project with Burr.
2. Run your Burr application with tracking/persistence.
3. Find a trace that you want to create a test case for.
4. Note the partition_key, app_id, and sequence_id of the trace. You can find these in the UI.
5. Create a test case using the `burr-test-case create` command passing the project name, partition_key, app_id, and sequence_id as arguments.
6. Choose what you're going to test and how. E.g. exact match, similarity, LLM Grader, etc.
7. Iterate on the test case until it is robust and representative of the real-world behavior of your application.
8. Profit.

For step (5) above, this is the corresponding command:
```bash
%%sh
burr-test-case create  \
  --project-name "SOME_NAME" \
  --partition-key "SOME_KEY" \
  --app-id "SOME_ID" \
  --sequence-id 0 \
  --target-file-name /tmp/test-case.json
```
See [the notebook](./notebook.ipynb) for example usage.

## Testing Actions
In `test_application.py` you'll find examples tests for a simple action
that is found in `application.py`.

## Notebook
The notebook also shows how things work. <a target="_blank" href="https://colab.research.google.com/github/dagworks-inc/burr/blob/main/examples/test-case-creation/notebook.ipynb">
  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>

## Future Work
We see many more improvements here:

1. Annotating data in the UI to make it easier to pull out.
2. Automatically suggesting tests cases for you to add.
3. Data export / integration with evaluation tools.
4. etc. Please let us know what you need!
