PSCR-Q shares almost nothing with a conventional model below the transformer blocks. It has no learned subword vocabulary, no flat output projection, no free-floating weights, and no separate representation of structure. Each of those was replaced with geometry. What follows is what that produced — measured at 207 million parameters, on four 2016-generation GPUs, 46% of the way through base training.
Every large language model in production shares four design decisions. PSCR-Q makes different ones. The table is not a list of optimizations — these are structural substitutions, and each one removes a component the field treats as a given.
| Component | Conventional | PSCR-Q |
|---|---|---|
| Vocabulary | Subword units learned from corpus frequency | Exact rational coordinates on a mathematical line |
| Output layer | One flat projection over the vocabulary | Factorized decode; one quantity computed, not predicted |
| Weights | Free values, constrained by clipping | Indices on the line, bounded by a theorem |
| Structure | Learned implicitly, never supervised | Part of the coordinate; explicitly supervised |
The consequences are concrete. A flat projection over this vocabulary would require roughly 23.5 terabytes of parameters; the decode instead costs 1.04 million, or half a percent of the model. Conventional models of this size spend closer to forty percent. Nearly the entire parameter budget is transformer rather than bookkeeping.
Structural position — where a character sits inside a word — is a supervised output here rather than something the model must infer. It runs at 96.3% accuracy, and it is the signal that three earlier generations of this architecture were never able to learn at all.
The mathematical framework, the addressing scheme and the encoder construction are not documented here. This page reports what the architecture does, not how it is built. Technical discussion is available on request.
At 46% of base training, sampled from a single prompt with no selection:
Three conversational turns, correct quotation marks, correct line breaks between speakers, and the third turn picking up the first turn's construction. A character-level model produces that one character at a time, choosing from a thousand possibilities at every step, with nothing telling it where a word ends or where a speaker changes.
Register holds too, and it changes with the prompt. Given a scientific opening the model stays scientific; given a business opening it stays commercial, across entirely different vocabulary:
The content is invented. That is expected of a model this size at this stage and is not the point — the point is that the form is correct before the facts are. Grammar, register, coreference and dialogue structure arrive first; knowledge accumulates behind them.
A base model continues text; it cannot yet answer a question. But its preference between candidate answers can be read directly, and that reaches knowledge generation cannot produce. Each probe is one half of a matched pair — identical options, identical framing, only the subject changes. A model responding to word frequency rather than meaning scores identically on both halves, so the difference between them is the measurement.
Ask what colour grass is and it prefers green. Ask what colour snow is, same three options, and the preference moves to white. That shift is not vocabulary; it is the model conditioning on what was asked.
| Domain | Effect size | Above noise |
|---|---|---|
| Language — plurals, tense, opposites | +0.853 | 7 / 7 |
| Geography — capitals, rivers, currencies | +0.615 | 10 / 10 |
| Everyday — tools, occupations, clothing | +0.419 | 5 / 6 |
| Physical — colour, state, material | +0.338 | 5 / 10 |
| Science — elements, gases, mechanics | +0.180 | 3 / 6 |
| Biology — habitat, anatomy, diet | +0.169 | 3 / 10 |
| Culture — authors, inventors | +0.046 | 2 / 5 |
| Arithmetic and counting | +0.021 | 1 / 7 |
71 of 122 probes correct against a chance rate of 41. Thirty-six of sixty-one families clear a noise floor set by twelve controls using invented subjects — invented countries, invented animals, invented words — so the threshold is measured rather than assumed.
Every geography and every language pair clears. Five separate capital-city sets, continents, oceans, rivers, currencies, irregular plurals, irregular past tenses. Arithmetic does not move at all: the model picks a frequent number regardless of the question. That profile is exactly what a web-text corpus should produce, and it says precisely which capabilities the next training phases have to supply.
Comparing a partly-trained model against a finished one proves little. EleutherAI publishes 154 intermediate checkpoints for each of its Pythia models, which allows the controlled question instead: at the same number of tokens seen, which model scores higher?
Identical scoring code, identical evaluation items, identical normalization. The character-to-token conversion is measured on the corpus, not assumed.
| Tokens seen | PSCR-Q 207M | Pythia 160M | Pythia 410M |
|---|---|---|---|
| 2.66 B | 0.479 | 0.425 | 0.417 |
| 2.77 B | 0.486 | 0.426 | 0.420 |
| 3.19 B | 0.492 | 0.433 | 0.430 |
| 3.33 B | 0.478 | 0.435 | 0.433 |
Ahead of both at every matched budget measured, across twenty-two checkpoints — including a model with twice the parameter count. That last comparison matters most, because it removes the obvious objection: whatever is producing the difference, it is not simply having more parameters.
Pythia trained on a 2020 corpus; PSCR-Q trains on a 2024 one that is more heavily filtered. Data quality is worth real points and is not controlled for here. The parameter objection is answered; the data objection is not.
Corrupt a character and a subword model's tokenization shatters — a familiar word becomes several unrelated units, and every representation downstream of it changes. A character-level model sees one wrong character inside an otherwise familiar pattern. The question is therefore about a slope: how fast does each model fall apart?
Tested on short-form question answering, where a fixed corruption rate does the most damage because there is least surrounding context to recover from:
| Corruption at 20% of characters | PSCR-Q 207M | Pythia 160M |
|---|---|---|
| Mixed — case, transposition, deletion, typos | 92.3% | 82.7% |
| Keyboard typos — adjacent-key substitution | 93.5% | 83.7% |
Accuracy retained as a fraction of each model's own clean score, so the comparison is unaffected by which model scored higher to begin with. Nearly ten points of advantage, consistent across both corruption types, and it has grown as training has continued. The effect is strongest where context is shortest; on long-form tasks with several hundred characters of surrounding text, both models recover and the difference closes.
Four P100 GPUs on a 40-gigacharacter corpus. The held-out split is drawn after the training split in a single ordered stream, so overlap is impossible by construction rather than by assurance.
| Metric | Current | Previous generation, final |
|---|---|---|
| Held-out perplexity, per character | 2.14 | no valid measurement |
| Word-interior accuracy | 85.1% | 68.4% |
| Word-initial accuracy | 36.0% | 27.3% |
| Structural position accuracy | 96.3% | — |
Held-out perplexity has fallen at all 103 evaluations to date, without a single reversal. Word-initial prediction — choosing the first character of the next word, which requires understanding context rather than continuing a spelling — passed the previous generation's final value within the first two thousand steps, against 130,000 steps and three phases.
Everything on this page is a waypoint. Base training is under half complete, and the two phases that follow are the ones that change what the model can be asked to do.
46% of the way through a 32-gigacharacter run on four P100 GPUs. Held-out perplexity has fallen at every evaluation, and word-initial prediction — the contextual signal — is still climbing.
A 3.6-gigacharacter code corpus spanning thirteen languages, drawn from permissively licensed sources, alongside a technical prose corpus. The probe results say precisely which capabilities this phase has to supply.
Instruction tuning teaches format rather than facts, which is why the knowledge probes matter: knowledge the model already ranks correctly is knowledge that becomes reachable once it learns that a question expects an answer.
Every figure here comes from a logged evaluation against a held-out split that cannot overlap the training data, with baselines scored by the same code on the same items. The benchmarks re-run at every checkpoint.
In a conventional transformer a weight is a floating-point number and nothing else is true about it. Here every weight occupies a position within a mathematical structure, so quantities exist that have no analogue elsewhere — a class a weight can move between as it trains, and a measure of how far it still has to travel.
Most interpretability research is spent discovering structure inside a trained network. Some of this structure is present by construction and can be read while the model runs. Doing so has already surfaced one property of the architecture that no loss curve or benchmark would have revealed, and a clean organization of behaviour by layer depth that nobody imposed.
Whether that can be developed into something stronger — a model whose own processing is inspectable as it works — is an open question, and one of the reasons this is being built at all.
Base training in progress at 207M parameters, 46% complete. Domain tuning to follow on a 3.6-gigacharacter code corpus spanning thirteen languages, then instruction tuning. Get in touch →