reading the records

Live

{{ liveBoard.live.length }} running · refreshed {{ refreshed || 'never' }}

nothing is running under the root right now

recent
recordkindstatelast seen
{{ entry.name }} {{ entry.path }} {{ entry.kind }} {{ stateOf(entry) }} {{ entry.status.last_seen ? ago(entry.status.last_seen) : '' }}

The address bar carries the record, the tab and the open plot, so a link shares exactly one view and a reload keeps it. A running record opens on its monitor tab and is followed as it changes. Every run as a table, with its params and its best value.

Runs

{{ tableSorted.length }} of {{ tableRows.length }} records
compare the {{ selected.length }} ticked tick two or more rows to compare them
columns {{ tableColumns.params.length + tableColumns.metrics.length }} of {{ tableColumns.allParams.length + tableColumns.allMetrics.length }}
params last values

Every run and sweep point under the root: the params of its manifest, the best value of the checkpoint monitor with its turn, the last evaluation values, the turns, the time and the device. Sort by a column. The filter takes words, each one a comparison of a param, a last value, best, turns or seconds (lr<0.01, val/rmse<=0.3, width=64, model!=mlp) or a text found in the name, the path or a param=value; a row stays when every word holds. The chips keep the states you pick, the columns menu picks what the table and the CSV carry (kept in this browser), and ticking two or more rows (or the first 50 shown) draws every series of them together.

record{{ sortDesc ? '▼' : '▲' }} state {{ key }}{{ sortDesc ? '▼' : '▲' }} best{{ sortDesc ? '▼' : '▲' }} {{ key }}{{ sortDesc ? '▼' : '▲' }} {{ tableUnit }}s time device
{{ row.name }} {{ row.path }} {{ stateOf(row) }} {{ fmt((row.params || {})[key]) }} {{ fmt((row.last || {})[key]) }} {{ row.turns }} {{ row.seconds ? ms(row.seconds * 1000) : '' }} {{ row.device || '' }}

{{ tableLoaded ? 'no run under the root' : 'reading the records' }}

Compare

{{ compareRuns.length }} records
{{ run.name }} {{ run.path }} ×

The history series of every record together, one chart per series with a line per record, the colour and the dash as in the list above; the chips keep one set, the filter the series whose name holds it, and expand opens a chart large. Below, the params that differ between the records, and for two records the difference of their resolved.yaml files. A record that is running redraws as it goes.

tick two or more rows in the runs table or points of a sweep, or write ?runs=path,path,...

reading the records

{{ path }}

sweep {{ sweepState }} {{ stopText(record) }} the report of {{ reportStatus.written }} {{ reportLine() }}
{{ sweep.objective.monitor }} {{ sweep.objective.mode || 'min' }} at {{ sweep.objective.at || 'best' }} strategy {{ shortUri(sweep.manifest.strategy) }} {{ sweep.points.length }} of {{ sweep.manifest.total || '?' }} points started {{ ago(sweep.manifest.started) }}

The points of the sweep from their manifests and sweep.json: finished ones scored, running ones with the best value so far from their history. Every point is drawn across the whole space, the failed ones included; drag on any axis to narrow the charts and the table together. The explorer picks its own axes, every param is shown against the objective, and the objective curves of the running or ticked points sit below with the diff of two ticked points.

scored {{ sweepStats.scored }} of {{ sweepStats.planned }} planned
failed {{ sweepStats.failed }} {{ sweepStats.failed ? Math.round(100 * sweepStats.failed / sweepStats.total) + '% of the points' : 'none' }}
best {{ objectiveName }} {{ fmt(sweep.best.objective.value) }} point {{ sweep.best.id }}, {{ sweepUnit }} {{ sweep.best.objective.turn }}
spread {{ (100 * sweepStats.spread).toFixed(1) }}% {{ fmt(sweepStats.low) }} to {{ fmt(sweepStats.high) }}
every point across the space drag on an axis to narrow, click a line to open the point
{{ objectiveName }} by rank best {{ fmt(scoreExtent[objectiveMode === 'min' ? 0 : 1]) }} worst {{ fmt(scoreExtent[objectiveMode === 'min' ? 1 : 0]) }} never scored
{{ brushed.length }} of {{ sweepRows.length }} points {{ note.text }} × drag on an axis above to narrow everything below, and the table
explorer {{ exploreChart.shown }} points, {{ exploreChart.ylabel }} against {{ exploreChart.xlabel }}, colored by {{ explore.color }}
best so far
{{ fmt(sweep.best.objective.value) }} at {{ sweepUnit }} {{ sweep.best.objective.turn }} {{ key }} = {{ fmt(value) }} open point {{ sweep.best.id }}
plot id{{ sortDesc ? '▼' : '▲' }} {{ key }}{{ sortDesc ? '▼' : '▲' }} status {{ sweepUnit }}s{{ sortDesc ? '▼' : '▲' }} {{ sweep.objective.monitor || 'objective' }}{{ sortDesc ? '▼' : '▲' }}
{{ point.id ?? '' }} {{ fmt(point.values[key]) }} {{ stateOf(point) }} stop {{ point.turns }} open

no point has started yet

why the points failed {{ sweepStats.failed }} of {{ sweepStats.total }} points, grouped by the first error of each
pointserrorids
{{ group.points.length }} {{ group.reason }} and {{ group.points.length - 40 }} more

A point is repaired in place with kalfa repair CONFIG --id N --record ROOT once the cause is fixed; without --id every failed point is repaired.

where the points died {{ sweepStats.failed }} of {{ sweepStats.total }} failed or lost
{{ entry.key }}
{{ level.label }} {{ level.failed }}/{{ level.total }}

A level that fails about as often as every other level points at the node or the environment, not at the params.

{{ objectiveName }} over the sweep order is the search still finding anything
{{ overlayKey || 'a series' }} of {{ selected.length ? 'the ' + selected.length + ' ticked points' : 'the running points' }}
every series of the ticked

tick points in the table, or the best ones here; without a tick the running points are drawn

difference between {{ selected[0] }} and {{ selected[1] }}
{{ line }}

reading resolved.yaml of both

{{ record.manifest && record.manifest.name || path }}

{{ kind }} {{ state }} {{ stopText(record) }} {{ path }}
started {{ ago(record.manifest.started) }} last seen {{ ago(record.status.last_seen) }} {{ elapsed }} device {{ record.device.device }} kalfa {{ record.manifest.version }} {{ record.host.hostname }} · pid {{ record.host.pid }} {{ turns }} {{ turnLabel }}s
{{ key }} = {{ fmt(value) }}
{{ divergence.key }} is {{ divergence.value }} from {{ turnLabel }} {{ divergence.turn }} on {{ failures[0].node || 'failed' }}: {{ failures[0].type ? failures[0].type + ': ' : '' }}{{ failures[0].message }} no heartbeat for more than ten minutes and no run.json: the process was killed or its machine went away
objective {{ fmt(record.sweep.objective.value) }} at {{ turnLabel }} {{ record.sweep.objective.turn }} · point {{ record.sweep.id }} {{ pointObjective }}

The record as it runs: the {{ turnLabel }} and batch bars, the checkpoint monitor across the sets, the loss every optimizer minimizes, the batch loss per step, the latest evaluation values and the tail of stdout, refreshed as the record changes or on the footer's schedule; a finished record shows its last state. The stop button in the header asks the record to end after its current turn.

progress {{ state }} {{ progressInfo.done }} {{ turnLabel }}s
{{ monitorKey }} = {{ fmt(progressInfo.monitorValue) }} {{ key }} = {{ fmt(value) }} at step {{ progressInfo.step }} {{ entry.name }} minimizes {{ entry.current }}

latest {{ turnLabel }}
serieslastbest
{{ row.set }}/{{ row.name }} {{ row.kind }} {{ fmt(row.last) }} {{ fmt(row.min) }} @{{ row.minTurn }} / {{ fmt(row.max) }} @{{ row.maxTurn }}
rules fired
{{ logs.name }}all
{{ logs.lines.slice(-14).join('\n') }}

The last {{ turnLabel }} of history.jsonl per series with its kind and the lego behind it (from resolved.yaml), the loss every optimizer minimized with the weights of its terms as the rules left them, the switches and the other effects of the rules over the {{ turnLabel }}s, the learning rates, the best {{ turnLabel }} by the checkpoint monitor, the checkpoints and the calibrations of the record.

failure {{ record.failure ? 'failure.json' : 'run.json' }}
{{ item.node || 'before the run started' }} {{ item.path }}
{{ item.type ? item.type + ': ' : '' }}{{ item.message }}
traceback
{{ item.traceback }}

Once the cause is fixed, the run goes on in its own directory with kalfa repair CONFIG --record {{ path }}; a sweep point with kalfa repair CONFIG --id N --record ROOT.

repairs
#stagefromstartedstatus
{{ item.attempt }}{{ item.stage }} {{ item.from }} {{ turnLabel }} {{ item.turn }} {{ ago(item.started) }} {{ item.status }}

The files of every earlier attempt are under attempts/ in the record.

best {{ turnLabel }}
{{ fmt(record.best.value) }} {{ record.best.monitor }} ({{ record.best.mode }}) at {{ turnLabel }} {{ record.best.turn }}
run
status{{ record.run.status }}
wall time{{ ms(record.run.ms) }}
error{{ failures[0].message }}
training
{{ entry.name }} minimizes {{ entry.current || 'its loss' }} ({{ shortUri(entry.uri) }}) = {{ entry.parts.join(' + ') }}
learning rate {{ fmt(entry.lr) }} at the last {{ turnLabel }}
rule effects, as history.jsonl carries them
{{ row.target }}
learning rates
{{ row.key }}
rules fired
resumed from
{{ record.resume.resume_from }}
{{ record.resume.checkpoint }}
checkpoints and exports
{{ item.name }}{{ fmtBytes(item.size) }} {{ ago(item.modified) }}
calibrations
{{ name }}{{ text(note) }}
latest turn
serieskindlegolastminmaxtrend
{{ row.set }}/{{ row.name }} {{ row.kind }} {{ shortUri(row.uri) }} → {{ row.output }} {{ fmt(row.last) }} {{ fmt(row.min) }} @{{ row.minTurn }} {{ fmt(row.max) }} @{{ row.maxTurn }}

no turn recorded yet

The predictions of the test set, or of another set when the run wrote one: a target per chip, the prediction against the truth with the y = x line on a sample of 2,000 rows or on every row, the residual histogram, the distributions of the prediction and the truth over the same bins with their ratio below, and the largest errors over every row, in the original units; the calibration flags are counted on the rows drawn. Every target is shown as a regression until you switch it to classification, which draws the ROC of a score column with its AUC and working points, the score by class and the confusion of the predicted classes. The filter runs over the columns of the file, the targets, the pred_ and raw_ wires, the flags and the data.spectators columns, with comparisons, and, or, not, arithmetic and lists (say pt > 500 and site in ['s1', 's2']); every number here is computed over the rows it keeps. Expand opens a chart large with its settings: scales, line width, points; on a histogram the x range and the bins are counted again over the rows inside the range (evenly on the log axis under log x, the rows outside counted above the chart) and y can show the fraction of rows. Closing it leaves the page as it was, 40 bins over the whole range.

{{ name }} · a sample of 2,000 all {{ predictionsData ? count(predictionsData.rows) + ' ' : '' }}rows {{ count(predictionsData.rows) }} rows · {{ count(predictionsData.drawn) }} drawn {{ count(predictionsData.rows) }} of {{ count(predictionsData.total) }} rows · {{ count(predictionsData.drawn) }} drawn
filter clear carried: {{ predictionsData.carried.join(", ") }}
{{ currentPair.target }} as the choice is kept in this browser for every record with the target {{ currentPair.target }}

the filter did not run: {{ predictionsData.error }}

no predictions file in the record yet

reading the predictions

{{ currentPair.target }} or {{ currentPair.pred }} is not numeric, so the regression view has nothing to compute; switch to classification

the predictions have no target and prediction pair to draw

The fitted plan under fitted/preprocessors: every field with its chain and the columns it produces, and what each preprocessor learned on the train set (a scaler's mean and scale per column, an encoder's categories, an imputer's statistic), read from the pickles.

reading the fitted preprocessors

no fitted preprocessors in the record

Every file of the record with its size and time. A name opens the file here: text (the first 400 KB), images and PDFs; a weights file or a table says it is not shown. Save downloads any file as it is.

files · {{ filesData.files.length }} files, {{ fmtBytes(filesData.total) }}
namesizemodified
{{ item.name }} {{ fmtBytes(item.size) }}{{ ago(item.modified) }} save

reading the files

{{ fileView.name }} first {{ fmtBytes(400000) }} of {{ fmtBytes(fileView.size) }} save

{{ fileView.text }}

{{ fileView.text }}

The event stream of tezgah (events.jsonl): every node from its start to its end on one time axis, indented by block; a node still running is orange, a failed one red. A loop is one bar over all its iterations, and the table below sums every node inside it over the iterations: where the time of a turn goes. Every iteration as its own row is one click away.

reading the events

no node event yet

nodes over time · {{ eventsData.total }} events, {{ ms(timeline.total) }} from the first start to the last end
{{ row.loop ? `${row.node} ×${row.count}` : row.node }}
{{ row.ms !== undefined ? ms(row.ms) : '' }}
inside the loops every node of an iteration, summed over the iterations that ended
nodelooprunstotalmeanlongestshare of the loop
{{ entry.name }} {{ entry.failed }} failed {{ entry.loop }} {{ count(entry.count) }} {{ ms(entry.total) }} {{ ms(entry.mean) }} {{ ms(entry.longest) }} {{ entry.share === null ? '' : (100 * entry.share).toFixed(1) + ' %' }}
{{ metricCharts.length }} metrics

One chart per metric, one line per set, drawn from history.jsonl once per turn; the axis says epoch when a turn is one pass over the train loader and turn when the run trains by steps. Drag to zoom, click a legend entry to hide a line, the toolbar downloads; expand opens a large view. Dashed lines mark the {{ turnLabel }}s a rule fired.

nothing to draw yet

{{ steps.lines.length }} updates

One line of steps.jsonl per optimizer update: loss/<optimizer> is the batch loss that update stepped on (not the running mean of the turn), lr/<optimizer> its learning rate at that step, grad_norm/<optimizer> the gradient norm before clipping under grad_clip. Dashed lines mark the {{ turnLabel }}s, labelled along the top, and the tooltip names the {{ turnLabel }} of a step; a long run is drawn with at most 2000 points per line, the extremes of every bucket kept, so spikes stay visible.

no step written yet

Every report model from architecture.json, written at the end of the run, drawn as a schematic: one block per node of the config with its input pins on the left and its output pins on the right, every wire with its width, the losses and the optimizers beside the outputs. A block clicked opens in place: the layers of a node chained left to right with the tensor shapes on the wires between them, the nodes of another model with their wires, the features of the input one per line; an inner block opens the same way, a second click closes it; hover lists what a block holds. Drag a block to move it, drag the upright part of a wire to bend it elsewhere, the wheel zooms, the background pans; the arrangement is kept in this browser per record, fit and reset undo it, save writes the png as arranged, expand opens it large. The module text is the torch repr.

no architecture.json in the record; runs write it since this version

module text (plots/architecture_text.txt) open save
{{ moduleText === undefined ? 'reading' : moduleText }}

The data block from data.json: the source, every transform with its call and the columns it added or removed, the split, the fit on train (dashed wires from valid and test: applied, never fitted), the feeds and the loaders. A block clicked opens in place with everything it holds, the columns named one per line; a second click closes it; hover lists a few. The tables below list the calls and the columns of every stage.

open in the large view

stages before the split
stagerowscolumnstransformaddedremoved
{{ stage.stage }}{{ count(stage.rows) }}{{ stage.columns }}
transforms after the split
settransforms
{{ entry.set }}
{{ shortUri(call.uri) }} {{ paramsText(call.params) }}

no data.json in the record

The files under plots/, written when training ended or by kalfa plots: click an image for the large view, a text plot opens in a viewer, save downloads the file as it is.

The per {{ turnLabel }} sample images under samples/ written by sample_writer; play walks through the {{ turnLabel }}s, the gallery lists them all.

{{ record.samples[frame] }}
{{ name }} open save
{{ texts[textKey(name)] === undefined ? 'reading' : texts[textKey(name)] }}

nothing written yet

resolved.yaml, the config as the run read it: includes expanded, aliases resolved to URIs, the command line overrides annotated.

resolved.yaml
{{ record.resolved || 'no resolved.yaml' }}

The identity of the record (manifest.json, host.json, device.json, git.json), every node of the run with its time from run.json and the seconds of every {{ turnLabel }} from history.jsonl.

manifest
{{ key }}{{ text(value) }}
nodes of the run
{{ node.node }}
{{ node.error }}
{{ ms(node.ms) }}
{{ node.status }}
{{ turnLabel }}s

{{ epochs.rows.length }} {{ turnLabel }}s · {{ ms(epochs.total * 1000) }} in total · {{ ms(epochs.mean * 1000) }} per {{ turnLabel }}

{{ turnLabel }}secondsglobal step
{{ row.turn }} {{ row.seconds === null ? '' : fmt(row.seconds) }} {{ row.step ?? '' }}
host
{{ key }}{{ text(value) }}
device
{{ key }}{{ text(value) }}
git
{{ key }}{{ text(value) }}

The last 60 events of events.jsonl, what tezgah started and finished and how long each node took; the timeline tab draws them all.

timekindnodemsnote
{{ clock(event.t) }}{{ event.kind }}{{ event.path }} {{ event.ms !== undefined ? ms(event.ms) : '' }}{{ event.error || event.status || '' }}

no event yet

The tail of stdout.txt and stderr.txt as the run wrote them (tezgah tees both into the record), refreshed while the record runs. A progress bar shows as its last state, the way a terminal draws it; the view follows the end unless you scroll up.

{{ name }} last {{ logs.lines.length }} of {{ logs.total }} lines the whole file
{{ logs.lines.join('\n') }}

{{ record.logs.length ? 'empty so far' : 'no log file in the record' }}

kalfa describe over the record: the same analysis the command prints, read from resolved.yaml under the contract the run was compiled by, computed when the tab opens.

{{ describeText }}

reading the config under the record's contract

{{ path }}

reading the record, or there is no record at this path