When Offline RL Cannot Evaluate Teaching: A Diagnostic Case Study

The two-policy tutoring system. The high-level pedagogical policy (this paper) selects knowledge-graph nodes; the low-level interaction policy (future work) executes teaching turns with the student.

Overview

Picture an intelligent tutoring system with two levels. A small pedagogical policy decides what to teach next over a knowledge graph, and a foundation-model interaction layer handles each teaching turn with the student. This paper is about the high-level policy. We train it as a Decision Transformer on top of a Deep Knowledge Tracing world model, try two reward functions grounded in the learning sciences, and run every comparison through a four-diagnostic screening protocol before believing it.

The result is a negative one, on purpose. The diagnostics caught every single comparison between a pedagogically-grounded policy and the immediate-correctness baseline, on both datasets. The failure reasons range from algebraic degeneracy in a reward formula to results that depend entirely on the world model's calibration. So the paper makes no positive claim about either reward. Its contribution is the screening protocol itself, plus a concrete list of the data infrastructure a real evaluation would need.

Setup

Real students only appear as logged data, from two primary-school math platforms. XES3G5M is the primary dataset: 18,066 students, 5.1M interactions, and 1,175 knowledge components in a four-level hierarchy, where the 865 leaf KCs form the action space. Junyi Academy (16.2M interactions, 1,326 exercises) is the cross-dataset reproduction target.

We fit a K=5 ensemble of Deep Knowledge Tracing LSTMs to each log. It reaches test AUC 0.851 on XES3G5M but only validation AUC 0.762 on Junyi, and that gap ends up mattering a lot later. The ensemble gets wrapped as a Gym-like environment: the state is the reference member's hidden vector plus an ensemble-disagreement scalar, and transitions sample synthetic correctness labels from the ensemble mean.

World-model training loop diagram — The world-model loop: the Decision Transformer picks a topic, the DKT ensemble returns predicted correctness, a belief update, and an uncertainty estimate, and the reward is computed under one of the candidate shapes.

Offline trajectories (10,000 per dataset, horizon 50) come from an ε-myopic-greedy behavior policy (ε = 0.5) rolled out in the world model. Decision Transformers (~1.13M parameters) train on them with a MOPO pessimism penalty swept over two values of λ.

Notice what this pipeline chains together: real students, then a world model, then synthetic trajectories, then a policy, then off-policy evaluation back on real test states. Every link introduces distribution shift. That is exactly why the evaluation needs screening.

Reward shapes

The standard reward in RL-for-tutoring is immediate correctness: teach whatever the student is most likely to already get right.

r^{\text{IMM}}(s_t, a_t) := \bar{p}_t(a_t) = \frac{1}{K}\sum_{i=1}^{K} \hat{p}_t^{(i)}(a_t).

The learning sciences have spent thirty years arguing this is the wrong target. So we built two alternatives from distinct traditions.

From Preparation for Future Learning, a preparedness-gain reward: the belief change a teaching action causes on the taught topic's sibling neighborhood.

r^{\text{PFL-GAIN}}(s_t, a_t, s_{t+1}) := \frac{1}{|\mathcal{N}(a_t)|}\sum_{k \in \mathcal{N}(a_t)} \bigl[\, \bar{p}_{t+1}(k) - \bar{p}_t(k) \,\bigr].

From Productive Failure and the desirable-difficulty literature, a boundary-of-capability reward that targets moderate difficulty.

r^{\text{PF-DIFF}}(s_t, a_t) := -\bigl|\, r^{\text{IMM}}(s_t, a_t) - p^{\star} \,\bigr|, \qquad p^{\star} = 0.5.

A fourth candidate, SIB-MEAN (predicted correctness averaged over the sibling neighborhood), serves as a neighborhood-smoothed baseline next to IMM.

The screening protocol

Four diagnostics wrap every comparison. None of them is exotic. Each is standard offline-RL or empirical practice, just applied as a screen at the reward-design stage instead of after results are already reported.

Cumulative-return correlation precondition. If a candidate reward's return-to-go ranks trajectories the same way the baseline's does, return conditioning produces a mechanically equivalent policy no matter what the formula says. This takes five minutes to compute, before any training.
Action-overlap check. A policy whose actions are out-of-distribution under the logged data gets inflated FQE scores through circular Q-bootstrapping. We compare each policy's argmax distribution against the empirical teacher.
Pairwise inter-policy diagnostic. Catches the case where differently-rewarded policies each look plausible on their own but are actually the same policy.
Cross-dataset check. A comparison has to survive a dataset switch before it counts as evidence about the policies rather than the world model.

Behind the diagnostics sit three off-policy estimators (FQE, WIS, DualDICE), each rank-validated on a tractable HMM toy environment before use, and a 20-seed FQE bootstrap sized to match policy-class variance. The seed budget was not arbitrary: a 5-seed pilot returned t = 1.34 on a comparison where the 20-seed bootstrap returned t = +5.41.

What the diagnostics caught

The correlation precondition fired before a single policy was trained:

Candidate reward	Corr. with IMM return	Verdict
SIB-MEAN	+0.9956	Fails (positive collapse)
PF-DIFF (p* = 0.5)	−0.9959	Fails (negative collapse)
PF-DIFF (p* = 0.85)	−0.9325	Borderline
PFL-GAIN	−0.1305	Passes (orthogonal)

SIB-MEAN collapses into the baseline because XES3G5M's curriculum tree groups siblings by similarity, not prerequisite direction. PF-DIFF at p* = 0.5 turns out to be algebraically anti-correlated with the baseline under myopic-greedy behavior. We did not anticipate that consequence of the formula; the check caught it before training did.

Two-panel FQE bootstrap results — 20-seed FQE bootstrap on real test states (±1σ). A: on XES3G5M, DT(PF-diff) lands in the empirical teacher's band, which the action-overlap diagnostic exposes as Q-bootstrap inflation. B: on Junyi, the PF-diff vs. Imm ranking inverts, and the five differently-rewarded DTs collapse to a single policy (99.6%+ pairwise argmax agreement).

Three catches stand out.

The teacher-matching mirage. On XES3G5M, DT(PF-DIFF) reached the empirical teacher's FQE band (2.714 ± 0.235 vs. 2.679 ± 0.025; the +0.036 difference is below the ~0.10 minimum detectable effect at 20 seeds). Impressive, until the action-overlap check: 0% argmax agreement with the teacher, 100% zero-overlap on top-5 action sets, and the teacher's median probability at the DT's chosen action was below uniform. This is Q-bootstrap inflation on actions the logged data never covers, and it is worse here than in CQL, the canonical example of the pathology.

The sign flip. On Junyi, the same PF-diff vs. Imm comparison inverted (Welch t went from +9.86 to −48.30). The ranking was measuring the world model's calibration regime, not teaching quality.

The policy collapse. The pairwise diagnostic showed all five differently-rewarded Junyi policies were effectively one policy. The bootstrap had been measuring FQE evaluation noise on a single underlying policy. Two isolation experiments (filtering trajectories to the top-10% return slice; dropping the MOPO penalty 4×) left the policies unchanged at 99.2% argmax agreement. The load-bearing structural cause, on both datasets, is action-space flatness.

The data infrastructure a faithful evaluation needs

Each catch points at a missing piece of infrastructure. The paper names these as prerequisites for the next attempt:

A directed prerequisite graph. PFL's transfer construct needs edges a → b; both benchmarks only expose similarity-curated hierarchies.
A transfer-aligned action structure. Hierarchical actions (stay-on-node vs. advance) instead of flat 865-way or 1,326-way argmaxes.
A post-intervention transfer assessment. This is PFL's actual metric, and it is absent from every offline pipeline.
A behavioral anchor outside synthetic trajectories. Real-teacher logs, so evaluation has a fulcrum outside the world model.

Takeaway

Offline RL for education inherits every known OPE failure mode, and the standard benchmarks cannot currently support the comparisons the field wants to make. The encouraging part: cheap diagnostics applied before training (correlation preconditions, action-overlap checks, cross-dataset replication) catch results that would otherwise ship as positive findings. The screening protocol is the contribution. The list of catches is the spec for what to build next.