reading the records

Live

{{ liveBoard.live.length }} running · refreshed {{ refreshed || 'never' }}

nothing is running under the root right now

recent
recordkindstatelast seen
{{ entry.name }} {{ entry.path }} {{ entry.kind }} {{ stateOf(entry) }} {{ entry.status.last_seen ? ago(entry.status.last_seen) : '' }}

The address bar carries the record, the tab and the open plot, so a link shares exactly one view and a reload keeps it. A running record opens on its monitor tab and is followed as it changes. Every run as a table, with its params and its best value.

Runs

{{ tableSorted.length }} of {{ tableRows.length }} records
compare the two ticked tick two rows to compare them

Every run and sweep point under the root: the params of its manifest, the best value of the checkpoint monitor with its turn, the last evaluation values, the turns, the time and the device. Sort by a column, filter by name, path or param=value, tick two rows to compare them.

record{{ sortDesc ? '▼' : '▲' }} state {{ key }}{{ sortDesc ? '▼' : '▲' }} best{{ sortDesc ? '▼' : '▲' }} {{ key }}{{ sortDesc ? '▼' : '▲' }} {{ tableUnit }}s time device
{{ row.name }} {{ row.path }} {{ stateOf(row) }} {{ fmt((row.params || {})[key]) }} {{ fmt((row.last || {})[key]) }} {{ row.turns }} {{ row.seconds ? ms(row.seconds * 1000) : '' }} {{ row.device || '' }}

{{ tableLoaded ? 'no run under the root' : 'reading the records' }}

Compare

The history series of both records overlaid, the first solid and the second dashed, one chart per metric with the sets as lines; below, the difference between their resolved.yaml files.

open this page from the runs table with two rows ticked, or write ?a=path&b=path

reading both records

one of the records does not exist

{{ path }}

sweep {{ sweepState }} {{ stopText(record) }}
{{ sweep.objective.monitor }} {{ sweep.objective.mode || 'min' }} at {{ sweep.objective.at || 'best' }} strategy {{ shortUri(sweep.manifest.strategy) }} {{ sweep.points.length }} of {{ sweep.manifest.total || '?' }} points started {{ ago(sweep.manifest.started) }}

The points of the sweep from their manifests and sweep.json: finished ones scored, running ones with the best value so far from their history. Every point is drawn across the whole space, the failed ones included; drag on any axis to narrow the charts and the table together. The explorer picks its own axes, every param is shown against the objective, and the objective curves of the running or ticked points sit below with the diff of two ticked points.

scored {{ sweepStats.scored }} of {{ sweepStats.planned }} planned
failed {{ sweepStats.failed }} {{ sweepStats.failed ? Math.round(100 * sweepStats.failed / sweepStats.total) + '% of the points' : 'none' }}
best {{ objectiveName }} {{ fmt(sweep.best.objective.value) }} point {{ sweep.best.id }}, {{ sweepUnit }} {{ sweep.best.objective.turn }}
spread {{ (100 * sweepStats.spread).toFixed(1) }}% {{ fmt(sweepStats.low) }} to {{ fmt(sweepStats.high) }}
every point across the space drag on an axis to narrow, click a line to open the point
{{ objectiveName }} by rank best {{ fmt(scoreExtent[objectiveMode === 'min' ? 0 : 1]) }} worst {{ fmt(scoreExtent[objectiveMode === 'min' ? 1 : 0]) }} never scored
{{ brushed.length }} of {{ sweepRows.length }} points {{ note.text }} × drag on an axis above to narrow everything below, and the table
explorer {{ exploreChart.shown }} points, {{ exploreChart.ylabel }} against {{ exploreChart.xlabel }}, colored by {{ explore.color }}
best so far
{{ fmt(sweep.best.objective.value) }} at {{ sweepUnit }} {{ sweep.best.objective.turn }} {{ key }} = {{ fmt(value) }} open point {{ sweep.best.id }}
plot id{{ sortDesc ? '▼' : '▲' }} {{ key }}{{ sortDesc ? '▼' : '▲' }} status {{ sweepUnit }}s{{ sortDesc ? '▼' : '▲' }} {{ sweep.objective.monitor || 'objective' }}{{ sortDesc ? '▼' : '▲' }}
{{ point.id ?? '' }} {{ fmt(point.values[key]) }} {{ stateOf(point) }} stop {{ point.turns }} open

no point has started yet

where the points died {{ sweepStats.failed }} of {{ sweepStats.total }} never finished a turn
{{ entry.key }}
{{ level.label }} {{ level.failed }}/{{ level.total }}

A level that fails about as often as every other level points at the node or the environment, not at the params.

{{ objectiveName }} over the sweep order is the search still finding anything
{{ sweep.objective.monitor || 'objective' }} of {{ selected.length ? 'the ticked points' : 'the running points' }}
difference between {{ selected[0] }} and {{ selected[1] }}
{{ line }}

reading resolved.yaml of both

{{ record.manifest && record.manifest.name || path }}

{{ kind }} {{ state }} {{ stopText(record) }} {{ path }}
started {{ ago(record.manifest.started) }} last seen {{ ago(record.status.last_seen) }} {{ elapsed }} device {{ record.device.device }} kalfa {{ record.manifest.version }} {{ record.host.hostname }} · pid {{ record.host.pid }} {{ turns }} {{ turnLabel }}s
{{ key }} = {{ fmt(value) }}
objective {{ fmt(record.sweep.objective.value) }} at {{ turnLabel }} {{ record.sweep.objective.turn }} · point {{ record.sweep.id }} {{ pointObjective }}

The record as it runs: the {{ turnLabel }} and batch bars, the checkpoint monitor across the sets, the loss every optimizer minimizes, the batch loss per step, the latest evaluation values and the tail of stdout, refreshed as the record changes or on the footer's schedule; a finished record shows its last state. The stop button in the header asks the record to end after its current turn.

progress {{ state }} {{ progressInfo.done }} {{ turnLabel }}s
{{ monitorKey }} = {{ fmt(progressInfo.monitorValue) }} {{ key }} = {{ fmt(value) }} at step {{ progressInfo.step }} {{ entry.name }} minimizes {{ entry.current }}

nothing recorded yet, the first {{ turnLabel }} is running

latest {{ turnLabel }}
serieslastbest
{{ row.set }}/{{ row.name }} {{ row.kind }} {{ fmt(row.last) }} {{ fmt(row.min) }} @{{ row.minTurn }} / {{ fmt(row.max) }} @{{ row.maxTurn }}
rules fired
{{ logs.name }}all
{{ logs.lines.slice(-14).join('\n') }}

The last {{ turnLabel }} of history.jsonl per series with its kind and the lego behind it (from resolved.yaml), the loss every optimizer minimized with the weights of its terms as the rules left them, the switches and the other effects of the rules over the {{ turnLabel }}s, the learning rates, the best {{ turnLabel }} by the checkpoint monitor, the checkpoints and the calibrations of the record.

best {{ turnLabel }}
{{ fmt(record.best.value) }} {{ record.best.monitor }} ({{ record.best.mode }}) at {{ turnLabel }} {{ record.best.turn }}
run
status{{ record.run.status }}
wall time{{ ms(record.run.ms) }}
error{{ record.run.error }}
training
{{ entry.name }} minimizes {{ entry.current || 'its loss' }} ({{ shortUri(entry.uri) }}) = {{ entry.parts.join(' + ') }}
learning rate {{ fmt(entry.lr) }} at the last {{ turnLabel }}
rule effects, as history.jsonl carries them
{{ row.target }}
learning rates
{{ row.key }}
rules fired
resumed from
{{ record.resume.resume_from }}
{{ record.resume.checkpoint }}
checkpoints and exports
{{ item.name }}{{ fmtBytes(item.size) }} {{ ago(item.modified) }}
calibrations
{{ name }}{{ text(note) }}
latest turn
serieskindlegolastminmaxtrend
{{ row.set }}/{{ row.name }} {{ row.kind }} {{ shortUri(row.uri) }} → {{ row.output }} {{ fmt(row.last) }} {{ fmt(row.min) }} @{{ row.minTurn }} {{ fmt(row.max) }} @{{ row.maxTurn }}

no turn recorded yet

The predictions of the test set, or of another set when the run wrote one: a target per chip, the prediction against the truth with the y = x line on a sample of 2,000 rows or on every row, the residual histogram, the distributions of the prediction and the truth over the same bins with their ratio below, and the largest errors over every row, in the original units; the calibration flags are counted on the rows drawn. The filter is a pandas query over the columns of the file, the targets, the pred_ and raw_ wires, the flags and the data.spectators columns; R², rmse, mae, the histogram and the largest errors are all computed over the rows it keeps. Expand opens a chart large with its settings: scales, line width, points, the bins of the histogram.

{{ name }} · a sample of 2,000 all rows {{ count(predictionsData.rows) }} rows · {{ count(predictionsData.sample.length) }} drawn {{ count(predictionsData.rows) }} of {{ count(predictionsData.total) }} rows · {{ count(predictionsData.sample.length) }} drawn
filter clear carried: {{ predictionsData.carried.join(", ") }}

the query did not run: {{ predictionsData.error }}

no predictions file in the record yet

reading the predictions

the predictions have no target and prediction pair to draw

The fitted plan under fitted/preprocessors: every field with its chain and the columns it produces, and what each preprocessor learned on the train set (a scaler's mean and scale per column, an encoder's categories, an imputer's statistic), read from the pickles.

reading the fitted preprocessors

no fitted preprocessors in the record

Every file of the record with its size and time. A name opens the file here: text (the first 400 KB), images and PDFs; a weights file or a table says it is not shown. Save downloads any file as it is.

files · {{ filesData.files.length }} files, {{ fmtBytes(filesData.total) }}
namesizemodified
{{ item.name }} {{ fmtBytes(item.size) }}{{ ago(item.modified) }} save

reading the files

{{ fileView.name }} first {{ fmtBytes(400000) }} of {{ fmtBytes(fileView.size) }} save

{{ fileView.text }}

{{ fileView.text }}

The event stream of tezgah (events.jsonl): every node from its start to its end on one time axis, indented by block; a node still running is orange, a failed one red.

reading the events

no node event yet

nodes over time · {{ eventsData.total }} events, {{ ms(timeline.total) }} from the first start to the last end
{{ row.node }}
{{ row.ms !== undefined ? ms(row.ms) : '' }}
{{ metricCharts.length }} metrics

One chart per metric, one line per set, drawn from history.jsonl once per turn; the axis says epoch when a turn is one pass over the train loader and turn when the run trains by steps. Drag to zoom, click a legend entry to hide a line, the toolbar downloads; expand opens a large view. Dashed lines mark the {{ turnLabel }}s a rule fired.

nothing to draw yet

{{ steps.lines.length }} updates

One line of steps.jsonl per optimizer update: loss/<optimizer> is the batch loss that update stepped on (not the running mean of the turn), lr/<optimizer> its learning rate at that step, grad_norm/<optimizer> the gradient norm before clipping under grad_clip. Dashed lines mark the {{ turnLabel }}s, labelled along the top, and the tooltip names the {{ turnLabel }} of a step; a long run is drawn with at most 2000 points per line, the extremes of every bucket kept, so spikes stay visible.

no step written yet

Every report model from architecture.json, written at the end of the run, drawn as a schematic: one block per node of the config with its input pins on the left and its output pins on the right, every wire with its width, the losses and the optimizers beside the outputs. A block clicked opens in place: the layers of a node chained left to right with the tensor shapes on the wires between them, the nodes of another model with their wires, the features of the input one per line; an inner block opens the same way, a second click closes it; hover lists what a block holds. Drag a block to move it, drag the upright part of a wire to bend it elsewhere, the wheel zooms, the background pans; the arrangement is kept in this browser per record, fit and reset undo it, save writes the png as arranged, expand opens it large. The module text is the torch repr.

no architecture.json in the record; runs write it since this version

module text (plots/architecture_text.txt) open save
{{ moduleText === undefined ? 'reading' : moduleText }}

The data block from data.json: the source, every transform with its call and the columns it added or removed, the split, the fit on train (dashed wires from valid and test: applied, never fitted), the feeds and the loaders. A block clicked opens in place with everything it holds, the columns named one per line; a second click closes it; hover lists a few. The tables below list the calls and the columns of every stage.

open in the large view

stages before the split
stagerowscolumnstransformaddedremoved
{{ stage.stage }}{{ count(stage.rows) }}{{ stage.columns }}
transforms after the split
settransforms
{{ entry.set }}
{{ shortUri(call.uri) }} {{ paramsText(call.params) }}

no data.json in the record

The files under plots/, written when training ended or by kalfa plots: click an image for the large view, a text plot opens in a viewer, save downloads the file as it is.

The per {{ turnLabel }} sample images under samples/ written by sample_writer; play walks through the {{ turnLabel }}s, the gallery lists them all.

{{ record.samples[frame] }}
{{ name }} open save
{{ texts[textKey(name)] === undefined ? 'reading' : texts[textKey(name)] }}

nothing written yet

resolved.yaml, the config as the run read it: includes expanded, aliases resolved to URIs, the command line overrides annotated.

resolved.yaml
{{ record.resolved || 'no resolved.yaml' }}

The identity of the record (manifest.json, host.json, device.json, git.json), every node of the run with its time from run.json and the seconds of every {{ turnLabel }} from history.jsonl.

manifest
{{ key }}{{ text(value) }}
nodes of the run
{{ node.node }} {{ ms(node.ms) }}
{{ node.status }}
{{ turnLabel }}s

{{ epochs.rows.length }} {{ turnLabel }}s · {{ ms(epochs.total * 1000) }} in total · {{ ms(epochs.mean * 1000) }} per {{ turnLabel }}

{{ turnLabel }}secondsglobal step
{{ row.turn }} {{ row.seconds === null ? '' : fmt(row.seconds) }} {{ row.step ?? '' }}
host
{{ key }}{{ text(value) }}
device
{{ key }}{{ text(value) }}
git
{{ key }}{{ text(value) }}

The last 60 events of events.jsonl, what tezgah started and finished and how long each node took; the timeline tab draws them all.

timekindnodemsnote
{{ clock(event.t) }}{{ event.kind }}{{ event.path }} {{ event.ms !== undefined ? ms(event.ms) : '' }}{{ event.error || event.status || '' }}

no event yet

The tail of stdout.txt and stderr.txt as the run wrote them (tezgah tees both into the record): the last 300 lines, refreshed while the record runs.

{{ name }} last {{ logs.lines.length }} of {{ logs.total }} lines
{{ logs.lines.join('\n') }}

{{ record.logs.length ? 'empty so far' : 'no log file in the record' }}

kalfa describe over the record: the same analysis the command prints, read from resolved.yaml under the contract the run was compiled by, computed when the tab opens.

{{ describeText }}

reading the config under the record's contract

{{ path }}

reading the record, or there is no record at this path