LLM Compass At a Glance

A knowledge compass for LLM training algorithms — this single page covers every algorithm: one sentence, the core formula, and when to use it. Build a full mental map in five minutes; click "Details →" for derivations, pseudocode, and tuning notes.

How to use this compass → · Notation →

Supervised Fine-Tuning

SFT

Supervised fine-tuning on instruction–response pairs turns a base model into an instruction follower.

$${\mathcal{L}}_{\text{SFT}}=-{\mathbb{E}}_{(x,y)}[\sum _t\log {\pi }_{\theta }(y_t\mid x,y_{<t})]$$

When to use: the first step of all post-training. Details →

Key engineering sub-topics: Full Fine-Tuning · Data Construction · Chat Template · Sequence Packing · Loss Masking

LoRA & Variants

LoRA

Freeze $W_0$ and train only the low-rank update $\mathrm{\Delta }W=BA$; trainable parameters drop below 1%, and the adapter merges back at inference with zero overhead.

$$h=W_{0}x+\frac{\alpha }{r}BAx$$

When to use: the default for memory-constrained fine-tuning. Details →

QLoRA

Store the frozen base in 4-bit NF4, dequantize on the fly for compute; the LoRA adapter trains as usual.

$$h=\mathrm{dequant}(W_0^{\text{NF4}})x+\frac{\alpha }{r}BAx$$

When to use: fine-tuning large models on a single GPU / very low memory, trading training speed. Details →

DoRA

Decompose weights into magnitude × direction: train the magnitude directly, update the direction via LoRA — learning dynamics closer to full fine-tuning.

$$W=m\cdot \frac{W_0+BA}{\Vert W_0+BA{\Vert }_c}$$

When to use: chasing quality at low rank. Details →

AdaLoRA

Parameterize the update in SVD form $\mathrm{\Delta }W=P\mathrm{\Lambda }Q$ and prune singular values by importance, allocating the rank budget to the modules that need it.

When to use: tight parameter budgets with uneven module importance. Details →

rsLoRA

Change the scaling factor from $\alpha /r$ to $\alpha /\sqrt{r}$ so the update scale no longer decays at high rank.

$$h=W_{0}x+\frac{\alpha }{\sqrt{r}}BAx$$

When to use: a one-line free win at rank ≥ 64. Details →

LoRA+

Give the $B$ matrix a much larger learning rate (${\eta }_B=\lambda {\eta }_A$, $\lambda \approx 16$), correcting the inherently asymmetric gradient scales of the two matrices.

When to use: a free speedup for any LoRA run. Details →

PiSSA

Initialize $B,A$ from the principal singular components of $W_0$ (freezing the residual), so training starts along the most important directions.

When to use: faster convergence; pairs with quantization to shrink quantization error. Details →

Preference Optimization (DPO Family)

DPO

Solve the KL-constrained RLHF objective in closed form, collapsing "train an RM + run RL" into a single classification loss.

$${\mathcal{L}}_{\text{DPO}}=-\mathbb{E}[\log \sigma (\beta \log \frac{{\pi }_{\theta }(y_w|x)}{{\pi }_{\text{ref}}(y_w|x)}-\beta \log \frac{{\pi }_{\theta }(y_l|x)}{{\pi }_{\text{ref}}(y_l|x)})]$$

When to use: the default alignment method when paired preference data is available. Details →

IPO

Under deterministic preferences DPO pushes the reward gap to infinity; IPO switches to a squared loss that pulls the gap toward a fixed target.

$${\mathcal{L}}_{\text{IPO}}=\mathbb{E}\left[{(\log \frac{{\pi }_{\theta }(y_w|x){\pi }_{\text{ref}}(y_l|x)}{{\pi }_{\theta }(y_l|x){\pi }_{\text{ref}}(y_w|x)}-\frac{1}{2\tau })}^2\right]$$

When to use: when DPO clearly overfits the preference data. Details →

KTO

No paired data required — one sample plus a good/bad label; the loss borrows loss aversion from prospect theory.

When to use: only binary feedback (thumbs up/down) is available. Details →

ORPO

SFT loss + odds-ratio penalty: a single stage that "learns to answer + aligns preferences" with no reference model.

$${\mathcal{L}}_{\text{ORPO}}={\mathcal{L}}_{\text{SFT}}(y_w)-\lambda \log \sigma (\log \frac{{\text{odds}}_{\theta }(y_w|x)}{{\text{odds}}_{\theta }(y_l|x)})$$

When to use: skipping the two-stage SFT → DPO pipeline. Details →

SimPO

Reference-free: the implicit reward becomes the length-normalized average logprob, plus a target margin $\gamma$.

$${\mathcal{L}}_{\text{SimPO}}=-\mathbb{E}[\log \sigma (\frac{\beta }{|y_w|}\log {\pi }_{\theta }(y_w|x)-\frac{\beta }{|y_l|}\log {\pi }_{\theta }(y_l|x)-\gamma )]$$

When to use: tight memory, or plagued by length inflation. Details →

CPO

Approximate the reference with a uniform prior to get an upper bound of the DPO loss, plus an SFT term to prevent chosen-probability collapse.

$${\mathcal{L}}_{\text{CPO}}=-\mathbb{E}[\log \sigma (\beta \log {\pi }_{\theta }(y_w|x)-\beta \log {\pi }_{\theta }(y_l|x))]-\mathbb{E}[\log {\pi }_{\theta }(y_w|x)]$$

When to use: reference-free training that must preserve generation quality (e.g., translation). Details →

RLHF / Reinforcement Learning

The shared RL objective:

$$\underset{{\pi }_{\theta }}{max}{\mathbb{E}}_{y\sim {\pi }_{\theta }}[r(x,y)]-\beta {\mathbb{D}}_{\text{KL}}[{\pi }_{\theta }\parallel {\pi }_{\text{ref}}]$$

Reward Model

Train a scoring model on preference data with the Bradley-Terry loss; it supplies the reward for RL.

$${\mathcal{L}}_{\text{RM}}=-\mathbb{E}[\log \sigma (r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l))]$$

When to use: the prerequisite of RLHF; its quality caps the RL ceiling. Details →

PPO

Clip the importance-sampling ratio to bound each policy update; advantages via GAE (requires training a critic).

$${\mathcal{L}}_{\text{PPO}}=-{\mathbb{E}}_t[min({\rho }_t{A}_t,\mathrm{clip}({\rho }_t,1\pm ϵ)A_t)]$$

When to use: classic RLHF; ample resources and token-level credit assignment needed. Details →

GRPO

Drop the critic: sample a group of responses per prompt; the group-standardized reward is the advantage.

$${\hat{A}}_i=\frac{r_i-\mathrm{mean}(\{r_j\})}{\mathrm{std}(\{r_j\})}$$

When to use: the current mainstream for reasoning RL (the DeepSeek-R1 recipe). Details →

RLOO

Back to REINFORCE: each response uses the mean reward of the other $k-1$ as its baseline — unbiased, no critic.

$$\mathrm{\nabla }\mathcal{J}=\frac{1}{k}\sum _i(r_i-\frac{1}{k-1}\sum _{j\ne i}{r}_j)\mathrm{\nabla }\log {\pi }_{\theta }(y_i|x)$$

When to use: the simplest unbiased group-sampling option. Details →

REINFORCE++

REINFORCE plus PPO's stabilizers (token-level KL, clip, global-batch advantage normalization) — no group sampling needed.

When to use: lightweight option when the sampling budget is one response per prompt. Details →

Agent & Skills

Tool Use Training

Teach the model to issue schema-correct function calls and digest the results; mostly SFT, with preference optimization refining call decisions. Details →

Agent Skills

Package procedures and tool knowledge into "instructions + scripts + resources" skill bundles — loaded on demand, no weight changes. Details →

Agentic RL

Extend the episode from single-turn generation to multi-turn "generate → execute → observe" loops, optimizing the policy against task-outcome rewards. Details →

Multi-Agent

Planner / executor / reviewer agents collaborate; orchestration patterns and credit assignment are the core problems. Details →