What is LLM determinism and why is it missing at temperature zero?

LLM determinism is the property that the same prompt, sent multiple times with the same parameters, always produces the same text output. Setting temperature to zero, which in theory removes random sampling and forces the model to always pick the most probable token, is not enough to guarantee this in practice: anyone calling the same API with the same prompt often gets different answers across separate requests. The cause is not only leftover sampling, but how inference actually runs on GPUs: the kernels that compute a model's numerical reductions, normalization, matrix multiplication, attention, are not batch invariant. The numerical result for one request changes depending on which other requests the inference server processes together in the same batch at that moment. Because floating point addition is not associative, a different order of operations produces different rounding, which propagates through the model's layers until it flips the chosen token, even at temperature zero.

See also: Confidence Calibration · How well an AI model's stated confidence actually matches its real probability of being right.

The technical cause, not just sampling

An AI lab documented this mechanism in a September 2025 technical post (Thinking Machines Lab, "Defeating Nondeterminism in LLM Inference"), showing that server load, meaning how many other requests share the batch at a given instant, determines the reduction strategy the kernels use and therefore the final numerical value, before the model even picks a token. The team built batch invariant kernels for normalization, matmul and attention, achieving 1,000 bitwise identical runs on the same prompt, at a measured time cost of about 61.5% more (26→42 s in the source's vLLM test). The "seed" some APIs expose reduces part of the residual sampling but does not touch this cause, which is numerical and tied to the compute infrastructure, not to the random number generator.

It is not just the server: the whole stack changes

In the month after that publication, a series of experiments on real agentic workloads documented the second layer of the problem: with identical weights and prompts, the chosen tokens also change with the implementation. The attention backend selected by the serving engine, the degree of quantization applied to the weights, that of the KV cache, the tensor parallelism degree and the GPU generation each produce divergence that is repeatable bit for bit, not noise: the same tool call against a network router succeeds on one backend and fails on another, targeting the wrong interface, and an 8-bit community build of the weights proved more faithful to the reference than the official reduced-precision release (Level1Techs, August 2026). Anyone carrying a demo into production changes stacks, not just machines.

An enterprise example

A company uses an LLM to automatically classify customer support tickets into billing categories, and needs to reproduce the exact same output for the same ticket months later during an audit, to show the classification followed a consistent rule. The team set temperature to zero believing this alone would guarantee reproducibility, and discovered in production that the same ticket, reprocessed twice minutes apart, can receive different classifications: not because of a policy or model change, but because server load at the moment of the second request differed from the first.

Why it matters for decision makers

When determinism is a real requirement, for audit, regulatory compliance, or automated tests that compare against an expected output, temperature zero alone is not a guarantee worth writing into a contract or an internal policy. There are three real mitigations and none is perfect: use providers offering batch invariant kernels (available in vLLM and SGLang, at a throughput cost), cache and store the answer obtained instead of regenerating it on every request when consistency over time matters, or accept the non-determinism and shift verification from a single expected output to a distribution of acceptable outputs, the kind of discipline a well built eval should already apply. Anyone signing a reproducibility commitment with a client or an auditor needs to know this before signing, not discover it after the first failed audit.