Quantization: Smaller Models and What Is Lost

A model is a large collection of numbers, and each number is stored with a certain precision. Quantization stores those numbers with fewer bits. The file gets smaller, the memory needed to run it drops, and the arithmetic can sometimes run faster. What it costs is accuracy, and that cost depends on the task in ways a single benchmark score will not reveal.

What precision means here

Each weight is stored in a format that uses a fixed number of bits. Fewer bits give fewer distinct values to choose from. A weight that was held precisely is rounded to the nearest value the smaller format can represent. Quantization methods differ in how they choose those values, whether they share one scale across a block of weights or use separate scales for smaller groups, and whether certain sensitive parts of the model stay at higher precision.

Rounding is a good starting picture. The error on any single weight is small. What matters is how those small errors accumulate through many layers and many steps of generation, and that accumulation is hard to predict from the arithmetic alone.

Quantization is also one option among several. A smaller model at full precision, a larger model that is aggressively quantized, and a mid-sized model at moderate precision can all fit the same memory budget. The choice among them is an empirical question about your tasks, and it is worth framing it that way early.

Memory and speed

The memory needed for the weights is roughly the number of parameters multiplied by the bits per weight, divided by eight to give bytes. Moving to fewer bits reduces that figure in proportion. The total memory use is larger than the weights alone, though. A conversation holds a cache of earlier tokens, the runtime needs buffers, and there is overhead, so the saving on the whole process is smaller than the arithmetic suggests.

Some runtimes can also quantize the cache that holds the conversation, which is a separate decision with its own trade-offs. The cache grows with the length of the context, so a model that runs comfortably with short prompts can run short of memory on long documents even at the same weight precision. Check memory use at the context lengths you expect, not only at the start of a conversation.

Speed gains come mainly from moving less data for each token. Generating text is often limited by how quickly weights can be read from memory, so smaller weights can help. The size of the gain depends heavily on the hardware and the software that runs the model. A general claim about speed tells you little about your machine, so measure throughput where the model will actually run.

Where quality goes

The loss is uneven. Tasks with a lot of redundancy, such as summarizing a familiar kind of document, often survive reduction well. Tasks that depend on precision can suffer more. Exact figures, long blocks of code where one wrong token breaks a function, uncommon languages or unusual symbols, and long outputs where small drift compounds are all places to look.

Errors can also be subtle. A quantized model may write fluent text with a slightly higher rate of wrong facts, and a casual read will not catch that. This is why the useful question is never simply whether the model got smaller. The useful question is what fails, how often, and on which work.

The relationship between bits and quality is not a straight line. The point where losses begin to matter depends on the model, the method and the task. It has to be measured for each combination, because a result borrowed from another setup may not hold for yours.

Measuring the loss on your own tasks

Start with a small evaluation set drawn from real inputs, with a clear pass criterion for each item. Run the full-precision model and the quantized one on the same inputs with the same settings. Compare pass rates, then read the failures side by side, because a count can hide what kind of mistake is being made.

Perplexity, a measure of how well a model predicts a reference text, is useful for tracking change during development. It tells you little about whether a particular answer is correct, so do not use it as the final judgment. Repeat the comparison over several sampled runs, since one run can mislead in either direction. Look for categories where the smaller version fails more often, not only for an overall average that might hide a serious gap in one area.

Choosing a level and practicing the comparison

Choosing a level is a trade-off across memory, speed and quality. A common approach is to pick the highest precision that fits your hardware with comfortable headroom, then test one or two lower levels to see where your own failures begin to rise. Write the decision down with its evidence: the test set, the pass rates, the hardware and the workload it was measured on. A decision without that record is hard to revisit when the model or the hardware changes.

Some quantization methods use a small calibration set of text to choose their values. If that set looks nothing like your domain, the chosen values may suit the calibration text better than your work. Ask which set was used and whether it matters for your case.

For practice, take one task you run regularly and assemble twenty real inputs from it. Run them through two precision levels and record for each output whether it passed. Then write one paragraph saying which level you would use, and name the one check you would repeat first if the model or the hardware changed. The paragraph is worth more than the numbers, because it shows whether you understand why the result came out as it did.