{% include "_privacy_meta.html" %} {% include "_dashboard_scripts.html" %} {% set page_subtitle = "Experiments" %} {% set nav_current = "experiments" %} {% include "_header.html" %}
{% if message %}
{{ message }}
{% endif %} {% if promote_error %} {% endif %} {% if not enabled %}

No experiment is running.

How to enable one

Set [experiment].enabled = true in the config this dashboard loaded, then restart the dashboard process.

{% elif error %}
{{ error }}
{% elif report %}

{{ experiment.id }}

Primary metric: {{ report.primary_metric }} · Assigned users: {{ report.n_assigned }}

{% if report.exposure_conditional %} Scored on users who saw a recommendation (exposure). {% else %} Scored on assigned users who had later events (intent-to-treat). {% endif %} Mixture confidence interval uses α={{ experiment.alpha }}.

{% if thompson and thompson.champion and thompson.challenger %}

Now testing {{ thompson.challenger }} vs champion {{ thompson.champion }}.

Pair volume: {{ thompson.pair_impressions }} / {{ thompson.min_impressions }} impressions before challenger rotation.

{% for arm in thompson.arms %} {% endfor %}
Thompson conversion
Recipe Impressions Conversions CVR P(best)
{{ arm.name }} {% if arm.role != "parked" %} ({{ arm.role }}) {% endif %} {{ arm.impressions }} {{ arm.conversions }} {{ "%.1f"|format(arm.cvr_pct) }}% {{ "%.0f"|format(arm.p_best * 100) }}%
{% endif %} {% if promoted_variant %}

Promoted variant: {{ promoted_variant }} (100% of serve traffic until you resume the split).

{% endif %}
{% for item in report.comparisons %} {% endfor %}
Variant comparison
Treatment {{ lift_label }} Mixture CI Treatment users Control users Status
{{ item.treatment.name }} vs {{ item.control.name }} {{ "%.4f"|format(item.difference) }} [{{ "%.4f"|format(item.ci_low) }}, {{ "%.4f"|format(item.ci_high) }}] {{ item.treatment.n_users }} {{ item.control.n_users }} {% if item.decided %}{{ item.winner }}{% else %}undecided{% endif %}
{% if report.guardrails %}

Guardrails

{% for item in report.guardrails %} {% endfor %}
Guardrails
Variant Fallback rate Top-item share Distinct items OK
{{ item.variant }} {{ "%.3f"|format(item.fallback_rate) }} {{ "%.3f"|format(item.top_item_share) }} {{ item.distinct_items }} {% if item.ok %}yes{% else %}{{ item.failures|join(", ") }}{% endif %}
{% endif %} {% if can_ship and ship_variant and not promoted_variant %}

This sends 100% of serve traffic to {{ ship_variant }} until you resume the split.

{% elif not promoted_variant and ship_blocked %} {% if experiment.allocation != "thompson" or ship_blocked != ("undecided",) %}

Promote is blocked: {{ ship_blocked|join(", ") }}.

{% endif %} {% endif %}
{% endif %}