{% if ai_influence_enabled %} {% endif %} {% if bws_enabled %} {% endif %} {% if mace_enabled %} {% endif %} {% if embedding_viz_enabled %} {% endif %}

Agreement

Full report →

Loading…

{{ ui_lang.admin_section_system_info|default('System Information') }}

{{ ui_lang.admin_th_user_id|default('User ID') }} {{ ui_lang.admin_th_phase|default('Phase') }} {{ ui_lang.admin_th_annotations|default('Annotations') }} {{ ui_lang.admin_th_working_time|default('Working Time') }} {{ ui_lang.admin_th_avg_time_per_annotation|default('Avg Time/Annotation') }} {{ ui_lang.admin_th_speed_per_hour|default('Speed (per hour)') }} {{ ui_lang.admin_th_completion_pct|default('Completion %') }} {{ ui_lang.admin_th_max_instances|default('Max Instances') }} {{ ui_lang.admin_th_last_activity|default('Last Activity') }}
{{ ui_lang.admin_th_instance_id|default('Instance ID') }} {{ ui_lang.admin_th_text_preview|default('Text Preview') }} {{ ui_lang.admin_th_annotations|default('Annotations') }} {{ ui_lang.admin_th_completion_pct|default('Completion %') }} {{ ui_lang.admin_th_most_frequent_label|default('Most Frequent Label') }} {{ ui_lang.admin_th_disagreement|default('Disagreement') }} {{ ui_lang.admin_th_avg_time|default('Avg Time') }} {{ ui_lang.admin_th_annotators|default('Annotators') }} {{ ui_lang.admin_th_num_ai_used|default('Num AI Used') }}

{{ ui_lang.admin_section_devices|default('Annotator Devices') }}

{{ ui_lang.admin_th_user_id|default('User ID') }} {{ ui_lang.admin_th_last_device|default('Last Device') }} {{ ui_lang.admin_th_mobile_visits|default('Mobile Visits') }} {{ ui_lang.admin_th_tablet_visits|default('Tablet Visits') }} {{ ui_lang.admin_th_desktop_visits|default('Desktop Visits') }} {{ ui_lang.admin_th_pocket_visits|default('Mobile UI Visits') }} {{ ui_lang.admin_th_last_seen|default('Last Seen') }}

{{ ui_lang.admin_section_ai_usage|default('AI Assistance Usage') }}

{{ ui_lang.admin_section_quality|default('Quality Indicators') }}

{{ ui_lang.admin_th_user_id|default('User ID') }} {{ ui_lang.admin_th_instances|default('Instances') }} {{ ui_lang.admin_th_avg_time_s|default('Avg Time (s)') }} {{ ui_lang.admin_th_interactions|default('Interactions') }} {{ ui_lang.admin_th_changes|default('Changes') }} {{ ui_lang.admin_th_ai_requests|default('AI Requests') }} {{ ui_lang.admin_th_ai_accept_rate|default('AI Accept Rate') }} {{ ui_lang.admin_th_suspicion|default('Suspicion') }}
{% if embedding_viz_enabled %}

Loading...

INSTANCE PREVIEW

Hover over a point to see details

SELECTION QUEUE

Use lasso or box selection to select points

Selected items will be prioritized for annotation

CONTROLS

{% endif %} {% if bws_enabled %}

Loading BWS scoring data...

{{ ui_lang.admin_section_bws_scores|default('BWS Item Scores') }}

Compute item scores from best-worst annotations. Counting is the default method (no dependencies). Bradley-Terry and Plackett-Luce require the choix package.

{{ ui_lang.admin_th_rank|default('Rank') }} {{ ui_lang.admin_th_item_id|default('Item ID') }} {{ ui_lang.admin_th_text|default('Text') }} {{ ui_lang.admin_th_score|default('Score') }} {{ ui_lang.admin_th_best_count|default('Best Count') }} {{ ui_lang.admin_th_worst_count|default('Worst Count') }} {{ ui_lang.admin_th_appearances|default('Appearances') }}
Click "Generate Scores" to compute BWS scores
{% endif %}

How fast is the work going?

Active time is measured from each annotator's own interaction events, with gaps longer than five minutes capped so an idle tab does not count as work. Only saved items count. When the review workflow is on, first-pass approval is the share of an annotator's reviewed items that were finished without being sent back, and rework is the share sent back at least once.

Loading...

{% if ai_influence_enabled %}

Did AI assistance change the answers?

A random share of each annotator's items (ai_support.holdout) showed no AI assistance. Each scheme compares those answers with the assisted ones: how often they match the model's own answer, how often they match gold where there is one, and how much annotators agree within each group. Higher model agreement with assistance, without higher gold accuracy, is the signature of anchoring.

Loading...

{% endif %} {% if mace_enabled %}

Loading MACE data...

{{ ui_lang.admin_section_competence|default('Annotator Competence') }}

MACE estimates each annotator's reliability (0 = random guessing, 1 = always correct).

{{ ui_lang.admin_th_annotator|default('Annotator') }} {{ ui_lang.admin_th_competence|default('Competence') }} {{ ui_lang.admin_th_reliability|default('Reliability') }}
Click "Run MACE" to compute scores

{{ ui_lang.admin_section_predicted_labels|default('Predicted Labels') }}

{{ ui_lang.admin_mace_predicted_help|default("MACE's best estimate of the true label for each item, weighted by annotator competence.") }}

{{ ui_lang.admin_mace_run_predictions|default('Run MACE to see predictions.') }}

{% endif %}

{{ ui_lang.admin_section_system_config|default('System Configuration') }}