Research category
Reusable training recipes, SFT methods, data selection, distillation, and optimization practice.
5 papers
Trajectory2Gym constructs executable tool-use interactions to cover underrepresented multi-step dependencies and supervises only decisions justified by the agent's visible history. Structural-support selection and parent-normalized SFT improve the mean score across Tau1, Tau2, and BFCL by 16.9–29.8 percentage points across five Qwen3 scales.
A-RSI monitors reward variation and accuracy in training logs to refresh a multimodal task pool when its learning signal declines. Model-authored programs generate questions with independently checkable answers without external training examples or human-written labels, raising the average score across eleven benchmarks from 57.77 to 61.05 with 800 fresh tasks.
GraphForge synthesizes training tasks for working agents from real-file workspaces, using an evidence graph to ground both task requirements and verification rubrics. It checks executability and revises tasks before collecting trajectories, improving downstream agent performance through fine-tuning.
BigBang is a general-purpose 35B-A3B model post-trained on verifiable frontier tasks generated by an adversarial, self-evolving generator-critic framework. Calibrated with held-out research tasks, the pipeline iteratively improves task difficulty and evaluation quality, yielding broad gains in scientific research, reasoning, coding, and tool use.
DataPrep-Bench is the first unified, downstream-grounded benchmark for evaluating how LLMs, agents, and data-centric workflows construct supervised training data and predict candidate datasets' downstream utility. It also introduces a skill-guided construction agent and the Distributional Alignment Score, which outperforms existing quality, diversity, and heuristic evaluators across most tested domains.
3 papers
LoCo identifies overthinking states in self-generated reasoning traces, regenerates shorter verified continuations from the same prefixes, and uses DPO on these local preference pairs to teach when to conclude. Trained only on mathematics, it cuts completion length by 41.7–45.4% on math and 23.4–36.3% on code while maintaining or improving accuracy.
This paper identifies SwiGLU's near-quadratic amplification on large positive inputs as a source of outliers and numerical instability in low-precision large-scale LLM pre-training. It proposes Power Linear Unit (PowLU), a rational-power activation that preserves adaptive nonlinearity while stabilizing spike regions, with scaling-law and Ling-model experiments showing competitive performance and improved training scalability.
This paper shows that Warmup-Stable-Only pre-training, which keeps the learning rate constant after warmup, consistently improves downstream SFT performance over decay-based schedules across 1B and 8B models. Its loss-landscape analysis attributes the gain to flatter minima that preserve model adaptability.
8 papers
This paper studies on-policy distillation at the data-minimal limit and finds that training on a single query recovers most of the gain from full-data OPD across domains and model families. It explains this through state coverage and student-teacher alignment, showing that 16 semantically distinct queries can match full-data training while OPD is data-overfed but algorithm-starved.
This controlled study separates rollout policy, token-level KL direction, and learning rate in strong-to-weak distillation. Forward KL is robust to rollout source, reverse KL favors student-generated rollouts, and learning rate has a stronger effect on forgetting and update sparsity than rollout policy.
This paper shows that effective warm-up for on-policy distillation depends on teacher-compatible chain-of-thought supervision rather than answer correctness alone. It proposes Simple-OPD, a plug-and-play initialization recipe that uses near-saturation LoRA warm-up on teacher-generated reasoning traces to balance in-domain adaptation and out-of-distribution generalization.
This paper identifies a low-KL agreement trap in on-policy distillation, where a teacher locally agrees with a student's corrupted rollout and provides little corrective supervision. It proposes KAT, an adaptive online termination rule that filters these uninformative suffixes, improving mathematical reasoning accuracy while reducing average rollout length by 59.73%.
This paper proposes BRTS, which samples multiple teacher trajectories and selects supervision by prioritizing correctness and then alignment with the student's current behavior. An auxiliary teacher-context branch and ground-truth-conditioned recovery improve OPD on challenging mathematical reasoning benchmarks.
This paper proposes TrOPD to stabilize on-policy distillation under large teacher-student distribution mismatch by restricting reverse-KL supervision to reliable regions. It handles outliers through clipping, masking, or forward-KL estimation and uses teacher-prefix off-policy guidance to steer exploration back toward trustworthy regions.
This paper identifies three failure modes in sampled-token OPD: imbalanced token supervision, unreliable teacher guidance on student-generated prefixes, and tokenizer or special-token mismatch. It combines teacher top-K local support matching with top-p rollout sampling and special-token masking, improving stability and outperforming standard OPD by 19.8%.
This paper studies white-box knowledge distillation for generative LLMs and proposes MiniLLM, replacing the standard forward KLD objective with reverse KLD to avoid overestimating low-probability teacher regions. The method derives an effective optimization procedure and improves instruction-following quality, calibration, exposure bias, and long-text generation across model families from 120M to 13B parameters.
Paper
Project
Code
Hugging Face
Paper