Vizuara AI Labs · RL from AI feedback (v2)

SLM‑500M RLAIF

The 500M instruction model improved against a learned reward model (best-of-N). Ask a question or give an instruction.

500M

parameters

4,000

pref pairs

6.7→10.1

reward lift

N=4

best-of

Validation metrics along the 500M lineage

Each perplexity is measured on that stage's own validation set, so read the trend as 'how well the model fits its own stage's data', not as one curve on one dataset. DPO and RLAIF optimize preferences rather than likelihood, so they log preference margin and reward instead of perplexity. Click a stage to open that model.

Base

ppl 7.91

pretrain val

→

QA SFT

ppl 5.41

QA val

→

Instruct

ppl 6.69

instruction val

→

DPO

margin 88.0%

preference val, no ppl

RLAIF

reward 6.7→10.1

RM reward, no ppl

→

RAFT on DPO

ppl 1.95

RAFT val

RAFT on RLAIF

ppl 2.01

RAFT val

instruction or question

optional: text to work on (attached as TEXT)

ready

The response will appear here.

What this is RLAIF v2

The 500M instruction model aligned with RLAIF: a Bradley-Terry reward model trained on the pairs, then best-of-N reward-weighted fine-tuning, on 4,000 AI-feedback preference pairs spanning closed-book QA and instruction-following failure modes (wrong figures, invented citations, broken format constraints, ignored instructions), prompts held out of every SFT set. Lineage: base → QA SFT → instruction SFT → RLAIF v2.

Served scale-to-zero on Modal, so the first request may take ~20–60s while the model wakes.

HuggingFace → SLM Arena 21 → 125M base → 500M base →