3D-DELTA Learned low-rank expert-steering head for MoE LLMs β€” research & ops dashboard Β· updated 2026-07-12

Paper PDF Full report Repo
Mean steering gain (seed-0, OLMoE)
+.101
vs SteerMoE +.025 β€” but see stability
Seed-mean gain (each seed's own argmax)
+.059 Β± .033
seeds 0/1/2: +.101 / +.053 / +.022
Transfer of seed-0's cells to new seeds
βˆ’.045 / βˆ’.032
the chosen cells do not transfer
Qwen3-30B steering (valid, unpublished)
+.042
Inco +.108 Β· 216/216 cells @ N=800
Live code after cleanup
11.8k
lines, down from 23.5k (βˆ’49.6%)
⚠ The headline is fragile β€” this gates everything.

+.101 / Inco +.288 is a seed-0, in-sample argmax result. Full per-seed sweeps (jobs 9104633/34) show each seed's own-argmax mean is +.059 Β± .033, and seed-0's winning cells go net negative on seeds 1–2. The paper's "+.21 Β± .11" rescue number compares N=600 steered cells against N=1499 baselines (invalid). Fix: consistent-N grids + validation-split cell selection β†’ Results tab.

⏰ BlackboxNLP 2026: abstract is COMPILED β€” submit before (Jul 17 AoE).

The 2-page non-archival abstract is written, compiled, and live: read the PDF, then submit on the OpenReview direct track (extended abstract, non-anonymized). Full checklist + verified track rules β†’ Venues tab.

βœ” Mixtral steering unblocked (head_dim fix); Qwen3 sweep valid.

Both failed Mixtral sweeps died on a vLLM 0.8.5 Γ— transformersβ‰₯4.54 head_dim=None crash (not OOM as the paper says) β€” fixed via hf_overrides; the re-run is queued (10169111). Qwen3's valid sweep (+.042) is promoted into results/QWEN3_combined_fix_36cell/.

πŸ–₯ Compute status (updated 2026-07-12, night).

12 cluster jobs live on liu32_1378 (fairshare 0.10, GPU queue ~630 deep β€” single-GPU jobs schedule fastest): OLMoE seed 0–3 N=800 grids (10169107–110, 1 GPU each) Β· Mixtral (10169111, the one remaining 2Γ—A100 job) Β· Qwen3-B4 resubmitted as 1Γ—A100-80 (10182888) Β· full pipelines on 1Γ—A40 each: Qwen1.5-MoE-A2.7B (10182886), Granite-3.1-3B (10182923), Phi-mini-MoE (10182924), DeepSeek-MoE-16B (10182987, capture-hook path), Granite-3.1-1B (10182988) Β· MMLU capability check (10182895). All dead robinjia_1822 account lines fixed repo-wide; gate-patcher now also handles router-named gates (Granite). Colab plan B for the seed grids stays live: colab/seed_sweep_colab.ipynb. Plan C β€” Modal (2026-07-12): account created, academic-credit application filed (answer ~1 week, up to $10k); the OLMoE seed grids + MMLU were re-lanned to the near-empty V100 pool (fp16, jobs 10185546–550); modal/sweep_app.py runs any sweep by preset β€” first $30 free credit is earmarked for --preset qwen3-b4 on A100-80 (~$25), Mixtral fits a single H200 (~$30). All delta heads are committed under deltas/ so containers need nothing from the cluster.

Where the project actually stands

Solid: the method and pipeline. One command per model (capture → (k, agg) sweep → train → 36-cell A×D inference → finalize), idempotent, now 12k lines lighter with the Mixtral blocker, the FaithEval-CF parser gap, per-cell dataset re-downloads, and the per-forward CPU→GPU delta copies all fixed. The learning-side result is robust: last-token pooling, k=64 wins on OLMoE (0.946), Qwen3 (0.976), Mixtral (0.836). Qwen3 steering works end-to-end (+.042 mean, +.108 Inco) — real evidence the recipe transfers beyond OLMoE.

Fragile: the headline OLMoE number. The AΓ—D grid argmax overfits the grid per seed: gains are real per-seed (+.02…+.10 mean) but the specific cells β€” and their magnitude β€” don't replicate. This is fixable (validation-split cell selection is standard practice and partly computable offline from existing outputs) and even publishable as a finding about selection stability, but the current paper text overstates it.

Novelty (lit sweep of 53 verified papers): the claim survives narrowed β€” "a learned low-rank inference-time faithfulness-steering head for frozen MoE LLMs, drop-in replacing SteerMoE's hand statistic." Must cite Router Lens/CEFT (EMNLP 2025) and MASCing (concurrent, Apr 2026); never claim "first learned expert steering." β†’ Research tab.

Collaboration: actively seeking a faculty collaborator / advisor for the full-scale version of this work β€” the method, codebase, and evaluation harness are ready for a lab-scale run (contact below).

Model status matrix

Legend: done action needed blocked parked

Model expansion β€” verified plan (2026-07-12)

Every candidate was verified against the pinned stacks (vLLM 0.8.5 registry + transformers 4.54.1, read from the venv source, not docs) for: size vs A40-48/A100-80, both-stack support, output_router_logits capture path, router scoring (softmax required β€” sigmoid routers break the clamp semantics), and the vLLM MoE-block attribute names our patcher hooks.

ModelSizeWhy it adds signalStatus
Qwen1.5-MoE-A2.7B (top-4 / 60+shared)29 GBGated shared-expert topology β€” absent from our current set; zero code changequeued 10182886
Granite-3.1-3B-A800M (top-8 / 40)6.6 GBNew vendor family, dense-routing regime, cheapest full grid; zero code changequeued 10182923
Phi-mini-MoE (Jun 2025, top-2 / 16)15 GBNewest viable release; SlimMoE distilled MoE β€” a compression data point; same PhiMoE class as Phi-3.5queued 10182924
DeepSeek-MoE-16B (top-6 / 64+2 shared)33 GBCanonical fine-grained+shared paper architecture; softmax βœ“; capture via new gate pre-hooks (unit-tested); patcher now indexes by MoE-block ordinal (dense first layer)queued 10182987
Granite-3.1-1B-A400M (top-8 / 32)2.7 GBExtra scale point β€” smallest MoE in the tablequeued 10182988

Verified NOT viable (don't burn time): Moonlight-16B & DeepSeek-V2/V3-style (sigmoid routers), ERNIE-4.5-21B / SmallThinker / Ling / LFM2 / Granite-4.0 / Nemotron-3-Nano / FlexOlmo / JetMoE (absent from vLLM 0.8.5 registry), Hunyuan-A13B / Jamba-Mini / Qwen3-Next / GRIN / Mixtral-8×22B (don't fit our GPUs), MiniCPM-MoE (needs 2 code tweaks, low traction). End-state generalization table: OLMoE · Qwen3-30B · Mixtral · Qwen1.5-MoE · Granite-3.1 (3B+1B) · Phi-mini · DeepSeek-MoE — seven families, 1.3B→47B, spanning shared-expert, fine-grained, distilled, and dense-first routing designs.

Session log

  • Jul 11 β€” repo cleanup (23.5k β†’ 11.8k live lines, 13 bugs fixed), truth audit (seed fragility + valid Qwen3 sweep discovered), 53-paper lit sweep, venue verification, this dashboard.
  • Jul 12 β€” 12 jobs queued (seed grids, Mixtral, Qwen3-B4, 5 new-model pipelines, MMLU check); model candidates verified against the pinned stacks; DeepSeek capture hooks + ordinal-indexing fix; BlackboxNLP abstract written and compiled to PDF; CEFT + MASCing deep-read playbook.
  • Security (still open for you): the git remote URL embeds an old personal-access token β€” rotate it and run git remote set-url origin https://github.com/Nikelroid/moe-steering-3d-delta.git. Pushes use MOE_TOKEN via askpass.

Steering gain Ξ” per benchmark β€” OLMoE-1B-7B (seed-0 grid)

Ξ” = best-steered cell minus each method's own no-steer baseline. SteerMoE/adsrivatsa quoted from their papers. Hover any bar for exact values; full table below.

Seed stability β€” the decisive analysis

Each seed picks its own best cell Seed-0's cells applied to all seeds
Why the paper's "+.21 Β± .11" is invalid

Partial warm-up jobs ran 6 cells at full N (Inco N=1499); the later 36-cell jobs ran at N=600. The sweep resumes past existing cells, so each seed's grid mixes sample sizes. The paper's number takes an any-N argmax β€” e.g. seed-1's "best" Inco cell is an N=600 cell at 0.822 compared against an N=1499 baseline of 0.526. Same-N argmax gives +.169 Β± .094 (Inco); the honest transfer test gives β‰ˆ0. Seed-1's grid is also incomplete (32–33/36 cells).

Qwen3-30B-A3B β€” valid steering sweep (unpublished until now)

Baseline β†’ best-steered accuracy per benchmark (Combined head, 216/216 cells, N=800, job 9110683, Jun 2). The earlier ~1%-accuracy runs in results/_superseded/ were an fp16/thinking-mode artifact β€” never cite them. The B4-winner Qwen3 sweep was never re-run.

FEVER drives the Inconsistent win (seed-0)

Best Ξ” on FaithEval-Inconsistent by training mix. SQuAD+FEVER alone beats the three-way mix β€” FEVER's claim-vs-evidence pairs are a conflict-detection task. Caveat: single-seed; the ablation inherits the stability question.

Learning-side generalization (held-out ranking acc.)

Same recipe (last-token pooling, k=64) wins across three MoE families, 7B→47B params, 8→128 experts. This result is robust and traceable to results/b4_kagg_sweep/; regenerate cross_model_summary/ for Qwen3/Mixtral provenance.

Paper ↔ data consistency (from the truth audit)

  • fixed paper/table_main.csv contradicted the paper (pre-fix Inco 0.112 "false floor") β€” deleted.
  • fixed Garbage Qwen3 dirs quarantined to results/_superseded/; valid run promoted to results/QWEN3_combined_fix_36cell/.
  • open main.tex L186–187/L329–331: replace the contaminated "+.21 Β± .11 / +.13 Β± .09" with consistent-N numbers once re-runs land.
  • open main.tex L327: Mixtral failure is a code crash (now fixed), not "out-of-memory" β€” correct the wording.
  • open Held-out 0.976/0.836 have no backing file in results/ β€” regenerate cross_model_summary.
  • fixed README synced to the honest state (stability note, Qwen3 +.042, Mixtral unblocked).
Live Python before β†’ after
23.5k β†’ 11.8k
lines; βˆ’49.6% in one pass
Deleted outright (dead)
~4.6k
zero runtime references, audit-verified
Archived (provenance)
7.0k
archive/ β€” per-token variant, GPT-OSS, notes
Bugs fixed
13
2 high Β· 6 medium Β· 5 perf/portability

What was removed and why

ItemLinesWhy it was safe
src/steermoe3d/modelling/ (5 files)4,052Dead by design β€” LLM_REGISTRATION=steermoe3d registers nothing; the live method monkey-patches vLLM's stock router gate at runtime. Nothing imported these copies.
src/toksteermoe/ β†’ archive/4,496Per-token steering variant, no SLURM/script launcher; direction explicitly ruled out.
GPT-OSS modelling + launchers β†’ archive/~2,600Pinned env (transformers 4.54.1) cannot load gpt_oss; the pipeline preflight hard-exits. Never ran.
scripts/orchestrator.py478Superseded by run_model_pipeline.py; only its output state file is read elsewhere.
src/device.py, src/visualization/heatmap.py, patch_*.py, llm_steering_3d.*, root report.html~1,000No importers/invokers; one-shot generators; report lives in docs/.
Stale session notes β†’ archive/notes/β€”CONTEXT.txt, SESSION_WRAPUP.md, MEETING_BRIEF.md (May state, superseded by this page).

Bugs fixed this session

Still-open engineering work

  • P1 Consolidate the remaining 2 modelling trees (activation, steermoe β€” 5 archs Γ— 2 modes, ~95% identical per pair). Target: one src/modelling/<arch>.py + a MoERouterHook strategy (capture | manual | delta3d), or β€” better β€” port both legacy modes onto the proven runtime gate-patcher and delete vendored modelling entirely. Needs a GPU node for verification; don't do blind.
  • P2 Batched capture in generate.py (2 forwards/sample today; ~2–4Γ— speedup) β€” only for future captures; never regenerate existing caches (numerics drift).
  • P2 Layer-alignment ablation: Ξ” trains on decoder-layer output hidden_states[l+1] but applies at the MoE gate input. It works, but nobody has measured the aligned variant.
  • P2 One pinned dependency source of truth (currently requirements.txt + 2 pinned variants + environment.yml + pyproject).
  • P2 sacct-based job tracking in wait_for_jobs (squeue-presence heuristic can still mis-fire past the 90s grace on a very congested queue).

How to run (current entry points)

# full pipeline for one model (idempotent; skips existing outputs)
python scripts/run_model_pipeline.py --model allenai/OLMoE-1B-7B-0125-Instruct --tag OLMOE --num-experts-per-tok 8

# Mixtral steering re-run (now unblocked; head_dim fix is in the driver)
sbatch --export=ALL,MODEL_NAME=mistralai/Mixtral-8x7B-Instruct-v0.1,NUM_EXPERTS_PER_TOK=2,DELTA_PATH=/scratch1/$USER/3d_delta/deltas/MIXTRAL_combined.pt,EXP_ID=MIXTRAL_combined_fix2,INFERENCE_DIR=/scratch1/$USER/3d_delta/inference/MIXTRAL_combined_fix2 \
  --array=0-35 slurm/steermoe3d/olmoe_faithfulness.slurm

# consistent-N seed grids (write to FRESH dirs; do not resume the contaminated ones)
#   train:  slurm/steermoe3d/train_olmoe_seeds.slurm   (seeds already trained: OLMOE_combined_seed{1,2}.pt)
#   sweep:  same array as above with DELTA_PATH=…seed1.pt, INFERENCE_DIR=…seed1_N800_cells, and a fixed --max-examples

TO-DO β€” ranked

Ordering rule: everything that changes the paper's central claim first, then reviewer-proofing, then upgrades. P0 Β· gates submission P1 Β· reviewer-proofing P2 Β· upgrades

NOT-TO-DO β€” settled questions and known traps

Each entry cost real time to learn. Re-litigating them burns weeks.

Novelty verdict

The claim survives, narrowed: "a learned low-rank inference-time faithfulness-steering head for frozen MoE LLMs that drop-in replaces SteerMoE's hand-engineered statistic." Nothing in 53 verified papers learns a hidden-state-conditioned low-rank expert-steering head for faithfulness on frozen MoEs. Two papers force the narrowing:

  • THREAT-high Router Lens + CEFT (arXiv:2508.19594, EMNLP 2025 Main) β€” learns context-faithful expert identification via router tuning on SQuAD/HotpotQA/NQ, on OLMoE and Mixtral, then fine-tunes those experts. Differences to defend: they update weights, we keep the model frozen; no inference-time mask rule; no FaithEval/SteerMoE grid. Must cite; ideally add as a baseline.
  • THREAT-high MASCing (arXiv:2604.27818, Apr 2026, concurrent) β€” learned sparse expert (de)activation mask added to routing logits at inference, no retraining… for safety/jailbreak, via LSTM-surrogate optimization per behavior. Same intervention point, different objective + architecture. Position as concurrent work.

Framing point that helps us: the optimization-over-routing literature exists mostly on the attack side (RouteHijack, LΒ³, Misrouter, F-SOUR) β€” we bring it to the faithfulness/defense side with a parametric head.

Competitor playbook β€” deep-reads of CEFT & MASCing (2026-07-12)

CEFT / Router Lens (EMNLP 2025)MASCing (Apr 2026, cs.CR)3D-DELTA
ObjectiveContext faithfulnessSafety (jailbreak / content)Context faithfulness
Model weightsModified twice (router-tune β†’ expert FT)Frozen βœ“Frozen βœ“
InterventionNone at inference (weight editing)Static additive gate bias, thresholded LΓ—E matrixInference-time mask from a hidden-state-conditioned low-rank head
Learned artifactTuned router + expert FFNs (0.5B params on OLMoE)Dense LΓ—E matrix per behavior, optimized vs LSTM surrogate~200k-param head, one per training mix, contrastive hinge
Input-conditioned?n/aNo (static; names "dynamic input-dependent masks" as future work β€” that's us)Yes (head reads pooled hidden state)
ModelsOLMoE-0924, DS-V2-Lite, MiniCPM-MoE, Mixtral7 incl. Mixtral, Phi-3.5, Qwen1.5-MoE, Qwen3-30B-2507, GPT-OSS, Hunyuan, DeepSeek-16BOLMoE-0125, Qwen3-30B, Mixtral (+5 families queued)
Seeds / error barsNone anywhereNone; grid tuned on the eval metric itself4-seed consistent-N CIs + validation-split selection (in progress) β€” our rigor moat
Held-out generalityTrains on each benchmark's own splitBehavior-level labels, no test split describedTrains once on SQuAD+HQ+FEVER, evaluates on 6 held-out benchmarks
Capability checkMMLU: FFT βˆ’18.4, CEFT βˆ’6.9, RT βˆ’2.5 (OLMoE)MMLU+GSM8K 5-shot: avg βˆ’4.1Frozen model β‡’ measure it, likely ~0 (job queued)

Adoptables now on the roadmap (merged from both papers, full details in the to-do list): NQ-Swap + ConFiQA-MC eval harness (the only shared-benchmark surface vs CEFT β€” note they use OLMoE-0924; matching it exactly makes the table clean) Β· MMLU/GSM8K capability table (our free win) Β· soft additive bias vs hard clamp ablation (MASCing's cleanest experiment: continuous 83.9% vs discrete 69.0% β€” predicts our soft mode may beat clamping) Β· per-layer Οƒβ„“ gate-logit scaling (candidate fix for the L0 concentration) Β· destructive-masking causal control (CEFT Fig. 3: βˆ’73% when masking their experts vs small drop for random) Β· router-tuning + CAD/CFP baseline rows with their published numbers Β· MASCing's own recipe re-run on our labels as a learned-baseline row (their code is public and simple).

The Pareto framing for any head-to-head: CEFT will beat us on raw EM (it trains in-distribution: NQ-Swap 90.5 vs base 28.1) β€” so the comparison table must carry three qualifier columns where we dominate: trainable params at deployment (0) Β· MMLU retention (~100%) Β· train-data access (held-out). Both papers' weaknesses (no seeds, no validation-split selection) are exactly the rigor work we're already doing β€” say so explicitly in the paper.

Baselines reviewers will demand

  1. ITI + CAA on OLMoE's residual stream, same 6 benchmarks β€” the dense-steering control everyone will name (SpARE optional).
  2. Prompting + CAD/AdaCAD β€” training-free controls; AxBench showed prompting is the bar to clear. If we don't beat a system prompt, the pitch becomes composability/capability-preservation.
  3. CEFT (or its router-tuning stage) or expert-pruning (NAEE, ACL 2024) at matched compute β€” isolates "learned selection" from "any adaptation."

Top upgrade actions (impact Γ· effort)

  1. Capability-degradation table (MMLU/GSM8K/TruthfulQA under the mask) β€” RASA made this the MoE-safety norm; cheap, defuses the #1 review.
  2. Multi-seed CIs framed via the steering-reliability literature (2407.12404, 2505.22637) β€” turns the fragility into a rigor selling point.
  3. Input-conditional Ξ” (CAST-style gate on the pooled hidden state) β€” answers "static-global is crude," differentiates from MASCing.
  4. Cross-model Ξ” transfer via linear activation-space maps (2503.04429) β€” upgrades "the metric transfers" to "the intervention transfers."
  5. Loss ablation: pairwise hinge vs BiPO/DPO-style vs InfoNCE.

Literature map (verified entries; filter + search)

All Threats Baselines Opportunities Cite

Deadline table verified on official CFPs, 2026-07-11 Β· all AoE

Recommended plan

  1. This week β€” BlackboxNLP 2026 (due Jul 17 ): the 2-page non-archival extended abstract is written and compiled (PDF) β€” review and submit on OpenReview. Not the 8-page archival track (that consumes the work). Buys expert interpretability-community review + a Budapest poster (Oct 29) at zero eligibility cost. The multi-seed + Mixtral jobs run in parallel through July–September.
  2. Main shot β€” ARR October cycle (due Oct 12 ) β†’ commit to NAACL 2027 (San Francisco, Jun 2027). 8-page long with 3 steered models + honest multi-seed CIs. No anonymity period β€” arXiv anytime. Optional parallel: a NeurIPS 2026 workshop (papers ~Aug 29, all non-archival β€” list posting mid-July; look for MechInterp).
  3. Backups: mixed ARR reviews β†’ revise + resubmit early-2027 cycle β†’ ACL 2027 (Japan). NAACL reject β†’ COLM 2027 (~Mar 31 est.) or ICML 2027 (~Jan 27 est.). Terminal fallback: TMLR (rolling, claims-based).

Skip: AAAI-27 (abstract Jul 21 β€” results won't be ready; weak audience fit) and the ARR Aug 3 β†’ EACL cycle (forces submission before the multi-seed story lands) unless everything solidifies unusually fast.

BlackboxNLP 2026 β€” submission package (due Jul 17 )

The 2-page non-archival abstract is written AND compiled: PDF here Β· tex source. It leads with the learned-head method, the OLMoE table (+.101 / Inco +.288), the Qwen3-30B transfer (+.042), the FEVER-attribution and layer-migration mechanistic findings, and β€” as a deliberate strength β€” the selection-stability analysis (grid-argmax overfits per seed; validation-split selection proposed), positioned against CEFT and MASCing. In-progress work (4-seed CIs, MMLU-under-mask, new MoE families) is stated as running, which the WIP track explicitly welcomes.

Track rules, verified on the official call page: extended abstracts are "2 pages + references", "need not be anonymized", non-archival ("will not be included in the proceedings") β€” for "work in progress" or cross-submissions. Dual submission allowed. ACL template βœ“ (used). Direct deadline Jul 17 is consistent across the site; note the site's two pages disagree downstream (news page: ARR-commit Aug 16 / notify Aug 27 / camera Sept 10 Β· call page: Aug 28 / Sept 8 / Sept 20) β€” irrelevant for the direct track.

  1. Read the compiled PDF; edit wording if you like (tex + acl.sty are in paper/; recompile with module load texlive && pdflatex).
  2. Submit on the BlackboxNLP OpenReview direct track β†’ extended abstract (non-archival) before Jul 17 AoE. Non-anonymized is correct.
  3. If any job lands before Jul 17 (MMLU capability check is fastest), drop the real number into the corresponding sentence.

Watch items

  • NeurIPS 2026 workshop list β€” notifications were due Jul 11 (today); check mid–late July; papers ~Aug 29, non-archival by policy.
  • ICLR 2027 CFP β€” posts ~Aug; est. abstract ~Sept 18 / full ~Sept 23–24. Mutually exclusive with the Oct ARR cycle β€” choose by early Sept based on how the multi-seed grids look.
  • SoCal NLP Symposium 2026 β€” USC-local, non-archival; CFP ~Sept.
  • ACL 2027 feeding cycle + COLM 2027 CFP β€” check Dec–Jan.