TRACKMANIARL / SYSTEM GUIDE / 04
Compose a value-based agent
Choose representation and value components independently. The same learner trains the composition.
01 Build a representation
02 Choose the value representation
03 Train with the shared DiscreteValueLearner
Frame adapter
Restore sequence axes.
Sensor encoder
Encode each frame
independently.
Temporal core
Identity, GRU or Mamba
Recurrent execution
Value head + strategy
Scalar Q · fixed quantiles (QR-DQN)
Random support (IQN) · learned fractions (FQF)
Action values
Scalar or quantile
values for each action
Current action
Evaluate the selected
action distribution.
Double-Q target
Online selects
Target evaluates
Loss + update
Regress values
Priorities + target sync
FQF
Uses a separate fraction optimizer and a
target-side fraction proposal network.
RECURRENT STATE
Burn-in rebuilds context. Actor hidden state
belongs to an episode, not the checkpoint.
Download editable diagram