A language model that is not built the way language models are built

Four departures from standard practice

Every large language model in production shares four design decisions. PSCR-Q makes different ones. The table is not a list of optimizations — these are structural substitutions, and each one removes a component the field treats as a given.

The consequences are concrete. A flat projection over this vocabulary would require roughly 23.5 terabytes of parameters; the decode instead costs 1.04 million, or half a percent of the model. Comparable models spend far more of themselves on their output embedding: 31% for GPT-2 Small, 24% for Pythia-160M, 21% for SmolLM2-135M. Nearly the entire parameter budget here is transformer rather than bookkeeping.

Structural position — where a character sits inside a word — is a supervised output here rather than something the model must infer. The zone head that predicts it runs at 96.6% on held-out text at the end of base training. That number is the structural classifier, not character prediction; predicting the character itself is reported separately below.

It writes, and it holds a thread

Two checkpoints, same sampler, nothing selected and nothing edited. First, at 49% of base training:

Three conversational turns, correct quotation marks, correct line breaks between speakers, and the third turn picking up the first turn's construction. A character-level model produces that one character at a time, choosing from a thousand possibilities at every step, with nothing telling it where a word ends or where a speaker changes.

The same opening at step 100,000, temperature 0.8, first and only run:

Register holds and it tracks the prompt. A scientific opening stays scientific, a commercial opening stays commercial, across entirely different vocabulary, at the same checkpoint:

The content is invented, and in the first of those two it is circular rather than merely wrong. That is expected of a model this size at this stage and is not the point — the point is that the form is correct before the facts are. Grammar, register, coreference and sentence structure arrive first; knowledge accumulates behind them.

It knows more than it can say

A base model continues text; it cannot yet answer a question. But its preference between candidate answers can be read directly, and that reaches knowledge generation cannot produce. Each probe is one half of a matched pair — identical options, identical framing, only the subject changes. A model responding to word frequency rather than meaning scores identically on both halves, so the difference between them is the measurement.

Ask the capital of France and it prefers Paris. Ask the capital of Japan, same three options, and the preference moves to Tokyo. That shift is not vocabulary; it is the model conditioning on what was asked.

The noise floor is measured, not assumed. Twelve control probes use invented subjects — an invented country, an invented animal — where no answer can be known, and the largest shift any of them produces is 0.503. A family counts only if its shift is larger than that. Measured at the end of base training:

70 of 122 probes correct, against 40.7 expected by chance, and 20 of 61 families clear the strictest control. Every capital-city set, every plural and past-tense pair, and every materials pair clears. Arithmetic does not move: asked two different sums, the model picks the same frequent number, Twelve, for both. Authors and inventors do not move either. That profile is what a web-text corpus produces, and it says which capabilities the next training phase has to supply.

More capable per token, at every budget measured

Comparing a partly-trained model against a finished one proves little. EleutherAI publishes 154 intermediate checkpoints for each of its Pythia models, which allows the controlled question instead: at the same number of tokens seen, which model scores higher?

Identical scoring code, identical normalization, and every item of four tasks — HellaSwag, ARC-Easy, PIQA and WinoGrande, 15,523 items in all. The character-to-token conversion is measured on the corpus (4.43 characters per token), not assumed. Pythia's score at each budget is interpolated between its published checkpoints at 4.2, 6.3 and 8.4 billion tokens.

Ahead of both at all five matched budgets measured, by +0.022 to +0.030 — including against a model with twice the parameter count. That last comparison removes the obvious objection: whatever produces the difference, it is not simply having more parameters.

The margin holds through the end of base training: +0.030 against Pythia-160M at 5.8 billion tokens and +0.028 at 7.2 billion. Pythia-410M's 4,000-step checkpoint, at 8.4 billion tokens, scores 0.443 — still below PSCR-Q's final 0.466. Base training ended at 7.2 billion tokens, so there is no matched comparison beyond that.

An earlier curve, scored on only the first 500 items of each task while training was running (steps 49,000 to 126,500, 2.2 to 5.6 billion tokens), was ahead of both models at all 29 budgets. The two item sets give different scores on identical weights, so they are not mixed here.

The control

A benchmark can flatter an architecture without measuring anything it learned. So the same model was instantiated with random weights and scored identically:

The untrained control scores below every Pythia checkpoint in the comparison, including the earliest of each. Chance on this four-task set is 0.375, and the control sits 0.001 above it. So the margin above Pythia is not something the scoring hands this architecture for free; it is 0.091 of measured learning.

Against fully trained Pythia-160M the honest word is level, not ahead. Its final checkpoint scores 0.457 and PSCR-Q's final scores 0.466. The difference, 0.009, is about 1.2 standard errors on a four-task mean over 15,523 items, so it is not a measured advantage. What is defensible is the budget: PSCR-Q reaches that level on 2.4% of the tokens Pythia-160M consumed. Pythia-410M, fully trained, scores 0.516 and is clearly ahead.

Six zero-shot tasks at the end of base training

Every item in each split — 21,848 items in all — scored by the same code for every model, baselines included, rather than quoted from a paper:

On the mean PSCR-Q is level with GPT-2 Small and Pythia-160M, about half a point below each. The mean hides a split: PSCR-Q is ahead on HellaSwag and PIQA and behind on ARC-Easy and LAMBADA. On the same items Pythia-410M scores 0.461 and SmolLM2-135M 0.486; both are well ahead. ARC-Challenge sits below chance for every model in the table; its questions were chosen because simple word-statistics methods answer them wrongly.

Grammar, reading and compression

Three further benchmarks at the final checkpoint, against Pythia-160M and GPT-2 Small scored by the same code, and against the same architecture with random weights:

On grammar PSCR-Q sits between the two: ahead of Pythia-160M, behind GPT-2 Small. Within BLiMP it leads Pythia-160M on syntax (0.770 against 0.750) and semantics (0.813 against 0.732) and trails on morphology (0.901 against 0.933). On science questions that come with a supporting passage it is ahead of both, and stays ahead when scores are normalised for answer length (0.708 against 0.688 and 0.660). On compressing Wikipedia text it is behind both. PSCR-Q's figure counts every bit, including 0.070 bits per byte spent on where words end.

On the same items Pythia-410M scores 0.831, 0.814 and 0.854, and SmolLM2-135M 0.824, 0.842 and 0.893; both are ahead of PSCR-Q on all three.

Taken together, against Pythia-160M and GPT-2 Small: ahead when the answer is supported by a passage in front of it, behind on ARC-Easy, which asks for facts without one. That gap is one reason the plan for instruction tuning includes a search tool.

It can see inside a word

A subword model receives strawberry as two or three opaque integers. The letters are not in its input, which is the documented reason such models fail at counting letters and spelling. PSCR-Q receives ten characters, each at its own coordinate. If character-native representation buys anything, it should buy this — and unlike the comparisons above, the mechanism is not in dispute.

One thousand items per task, words drawn from the training corpus, each used once, the correct answer forced to a uniform position. All three models see byte-identical items.

The two that move, move a long way. At the end of base training, counting runs 0.732 against 0.590 for Pythia and spelling 0.827 against 0.406. Both start below chance with random weights — this architecture has a systematic bias against the correct answers before it is trained, so the gain over its own baseline is larger than the gain over chance: +0.68 on counting and +0.70 on spelling. Nothing here is handed to it by the scoring.

Single checkpoints move around on these tasks, so the figures above are the final ones rather than the best. Across the ten checkpoints scored from step 61,000 to the end, counting ranged 0.674–0.742 and spelling 0.487–0.827. Every one of those values is above Pythia on both tasks, which is the claim; the exact margin is not stable to three decimals and is not reported as though it were.

Reversal and containment sit at or near chance for every model including this one. Those are reported because they are part of the same battery, not because they support anything.

Two results we withdrew

Running an untrained control against every benchmark cost one claim outright and invalidated three measurements. Both are recorded here because a page that only reports what worked is not a technical record.

Noise robustness: no effect

The prediction was that corrupting characters would damage a subword model faster than a character model, and an earlier version of this page reported 92.3% retention against 82.7% for Pythia at 20% corruption. That figure was one task and one corruption type out of six such cells. Aggregated across all six, at the end of base training, it is 90.9% against 89.4% — and that gap is smaller than PSCR-Q's own spread: across eleven checkpoints from step 56,000 to the end, its figure ranged from 89.2% to 93.0%. No effect.

The control explains why the metric looked promising. Retention is accuracy under noise divided by that model's own clean score, and a model whose output does not depend on its input retains everything. Measured: the untrained model retains 99.2%. Corrupting text cannot damage a judgement that was never reading it. Normalizing instead by the signal above chance, PSCR-Q retains 71.3% against Pythia's 67.4%, with its own checkpoints ranging from 63.2% to 76.0% — again inside its own spread.

Three character tasks that measure nothing

Of the eight tasks in the character battery, three — first letter, last letter, and word length — turned out to be unmeasurable as constructed. Their distractors are drawn from the same word as the answer, so a strategy that ignores letter identity entirely can score well above chance on one and far below it on the others.

The control settles two of them. By the end of base training, training has moved last-letter from 0.251 to 0.128 and word-length from 0.223 to 0.114: a model that learns gets worse at both than one with random weights. That is an inverted task, not a weak model.

First-letter fails differently, and it is worth being specific because an earlier version of this page was going to report it as a success. It moves in the right direction from the untrained baseline, but it will not sit still: 0.263 at step 85,000, 0.378 at 95,000, 0.521 at 100,000, 0.270 at 103,000, 0.614 at the final checkpoint, against an untrained 0.208. The spread across consecutive checkpoints is wider than the whole distance from the control, so no single value from it means anything. All three columns are excluded from the table above and from every figure on this page.

Base training, start to finish

Four P100 GPUs on a 40-gigacharacter corpus. The held-out split is drawn after the training split in a single ordered stream, so overlap is impossible by construction rather than by assurance.

Character prediction is reported by position in the word, because the four positions behave nothing alike and a single averaged figure hides that. Measured over 204,244 positions drawn from 100 held-out windows at the end of base training:

Held-out perplexity fell from 5.85 at the first evaluation to 2.0327 at step 162,500, the last of 325 evaluations. It did not descend smoothly: it improves for a stretch, holds flat for a few thousand steps, then resumes lower.

Word-initial prediction — choosing the first character of the next word, which requires context rather than continuing a spelling — moved far more slowly than the rest: on the same 204,244 positions it went from 35.8% at step 103,000 to 36.9% at the end, and has been flat since about step 143,000. At 16.3% of all characters, it is the clearest remaining weakness of the base model.

Where the work stands

Base training is complete. The two phases that follow are the ones that change what the model can be asked to do.

Structure that can be read

In a conventional transformer a weight is a floating-point number and nothing else is true about it. Here every attention weight occupies a position on the PSCR number line, so quantities exist that have no analogue elsewhere — where a weight sits among the whole-number shapes, and how far it still is from the circle.

Most interpretability research is spent discovering structure inside a trained network. Some of this structure is present by construction and can be read while the model runs, and reading it has already produced an event no loss curve would have shown.

Between steps 92,000 and 95,000 the entire attention-weight population crossed the monogon, N = 1, together: the share above it went from 0.45% to 99.4% in three thousand steps. Held-out perplexity reached a local minimum, 2.0972, at the end of that window and did not get below it again until step 102,500. One event is not a mechanism, and we are not claiming the crossing caused the pause. But it is a quantity that exists only because the weights live on a line, and it was logged without anyone looking for it.

At the end of base training, none of the 51,380,224 attention weights sits on a whole number: every one is an imaginary shape, iN, between whole numbers, and 99.998% of them lie between N = 1 and N = 2, just below the digon, where a weight is zero.

Whether that can be developed into something stronger — a model whose own processing is inspectable as it works — is an open question, and one of the reasons this is being built at all.