Environment: CliffWalking-v1. Maximize cumulative episode reward.
Resolved settings (override documentation examples): {"id": "CliffWalking-v1", "kwargs": {"render_mode": null, "is_slippery": false}, "max_episode_steps": 500}
Observation space: Discrete(48)
Action space: Discrete(4)
Episode limit: 500 steps. You receive observations only, not info/action masks.
Installed Gymnasium environment documentation:
Cliff walking involves crossing a gridworld from start to goal while avoiding falling off a cliff.

## Description
The game starts with the player at location [3, 0] of the 4x12 grid world with the
goal located at [3, 11]. If the player reaches the goal the episode ends.

A cliff runs along [3, 1..10]. If the player moves to a cliff location it
returns to the start location.

The player makes moves until they reach the goal.

Adapted from Example 6.6 (page 132) from Reinforcement Learning: An Introduction
by Sutton and Barto [<a href="#cliffwalk_ref">1</a>].

The cliff can be chosen to be slippery (disabled by default) so the player may move perpendicular
to the intended direction sometimes (see <a href="#is_slippy">`is_slippery`</a>).

With inspiration from:
[https://github.com/dennybritz/reinforcement-learning/blob/master/lib/envs/cliff_walking.py](https://github.com/dennybritz/reinforcement-learning/blob/master/lib/envs/cliff_walking.py)

## Action Space
The action shape is `(1,)` in the range `{0, 3}` indicating
which direction to move the player.

- 0: Move up
- 1: Move right
- 2: Move down
- 3: Move left

## Observation Space
There are 3 x 12 + 1 possible states. The player cannot be at the cliff, nor at
the goal as the latter results in the end of the episode. What remains are all
the positions of the first 3 rows plus the bottom-left cell.

The observation is a value representing the player's current position as
current_row * ncols + current_col (where both the row and col start at 0).

For example, the starting position can be calculated as follows: 3 * 12 + 0 = 36.

The observation is returned as an `int()`.

## Starting State
The episode starts with the player in state `[36]` (location [3, 0]).

## Reward
Each time step incurs -1 reward, unless the player stepped into the cliff,
which incurs -100 reward.

## Episode End
The episode terminates when the player enters state `[47]` (location [3, 11]).

## Information

`step()` and `reset()` return a dict with the following keys:
- "p" - transition proability for the state.

As cliff walking is not stochastic, the transition probability returned always 1.0.

## Arguments

```python
import gymnasium as gym
gym.make('CliffWalking-v1')
```

## References
<a id="cliffwalk_ref"></a>[1] R. Sutton and A. Barto, “Reinforcement Learning:
An Introduction” 2020. [Online]. Available: [http://www.incompleteideas.net/book/RLbook2020.pdf](http://www.incompleteideas.net/book/RLbook2020.pdf)

## Version History
- v1: Add slippery version of cliffwalking
- v0: Initial version release

Evaluation contract (takes precedence over documentation examples):
Maximize the arithmetic mean of undiscounted cumulative episode rewards across the fixed search seed panel. Higher is better, including when all rewards are negative.
Early stopping is disabled; maximize mean search reward throughout the generation budget.
Held-out episodes are for reporting only and never guide selection or stopping.
Wall-clock evaluation limit: 10 seconds per episode, per seed, for the worker to return its episode result. Policy initialization and environment steps must fit this limit. Docker startup, queueing and final process cleanup are outside this limit. This is not a total budget across all seeds.
Call limit: 10 seconds per policy call (act/reset/close), also subject to the episode limit. Keep all computation bounded.
A timeout is an evaluation failure, not a truncated successful episode or a zero reward. Search-time failures enter the repair loop, within the repair budget.
The policy receives observations, not rewards or info/action masks; episode state must be reset between episodes. Evaluation feedback reaches the LLM between proposals.
