Can LLMs do randomness?

I enjoyed that.

When you asked it to choose by picking a random number between 1 and 4, it skewed the results heavily to 2 and 3. It could have interpreted your instructions to mean literally between 1 and 4 (not inclusive).

could you use structured output to make a more efficient estimator for the logits based?

Author here. Yeah totally agreed. The more rigorous way to do this would be to use a fixed seed and temp and in a local model setting and then sample the logprobs and then analyse that data.

I had an hour to kill and did this experiment.

I was gonna say floating point errors might contribute especially at fp16 and fp8, but those are technically deterministic.

Congratulations, this was all a test to see if there were anyone on HN with any knowledge of how LLMs work, and you gave the correct answer.

The core of an LLM is completely deterministic. The randomness seen in LLM output is purely the result of post processing the output of the pure neural net part of the LLM, which exists explicitly to inject randomness into the generation process.

This what the “temperature” parameter of an LLM controls. Setting the temperature of an LLM to 0 effectively disables that randomness, but the result is a very boring output that’s likely to end up caught in a never ending loop of useless output.

You're right, although tests like this have been done many times locally as well. This issue comes from the fact that RL usually kills the token prediction variance, disproportionately narrowing it to 2-3 likely choices in the output distribution even in cases where uncertainty calls for hundreds. This is also a major factor behind fixed LLM stereotypes and -isms. Base models usually don't exhibit that behavior and have sufficient randomness.

Agreed. These tests should be performed on local models.

That's an interrogation as old as time

> floating point operations like addition are not commutative

maybe you meant associative? Floating point addition is commutative: a+b is always equal to b+a for any values of a and b. It is not associative, though: a+(b+c) is in general different to (a+b)+c, think what happens if a is tiny and b,c are huge, for example.

Cool experiment! My intuition suggests you would get a better result if you let the LLM generate tokens for a while before giving you an answer. Could be another experiment idea to see what kind of instructions lead to better randomness. (And to extend this, whether these instructions help humans better generate random numbers too.)

In the summary at the top it says you used 0-10 but then for the prompt it says 1-10. I had assumed the summary was incorrect but I guess it's the prompt that's wrong?

I think randomness needs to be better defined. In the article it seems to be that randomness should be an evenly distributed type of event occurences. I agree that it is very unintuitive for us as, I believe, we assume randomness to be any sequence of event that doesn't follow any known/recognizable pattern. Show a section of the Fibonacci to a 10 yo kid and they will most likely find the sequence of numbers to be random (maybe they will note that it is always increasing, but that's it). Even in this article the fact that o1 always throws "heads" could indicate that it "knows" what randomness is, and is then just being random by throwing only heads.

I personnaly would define ideal randomness as a behavior that is fundamentally uncomputable and/or cannot be expressed as a mathematical function. If this definition holds than the question cannot apply to LLMs as they are a just (big) mathematical function.

Veritasium did a video on this. Most people guess 37 when asked to pick between 1-100

That's expected behavior when you run LLM locally with a fixed seed and temperature at zero

Author here. I know it’s silly. I understand to some extent how they work. I was just doing this for fun. Took about 1hr for everything and it all started when a friend asked me whether we can use them for a coin toss.

The algorithms are not deterministic: they output a probability distribution over next tokens, which is then sampled. That’s why clicking “retry” gives you a different answer. An LM could easily (in principle) compute a 50/50 distribution when asked to flip a coin.

"You can actually read up on how these things work."

While you can definitely read about how some parts of a very complex neural network function, it's very challenging to understand the underlying patterns.

That's why even the people who invented components of these networks still invest in areas like mechanistic interpretability, trying to develop a model of how these systems actually operate. See https://www.transformer-circuits.pub/2022/mech-interp-essay (Chris Olah)

Deterministic with a random seed?

Yes, but sometimes asking dumb questions is the first step to asking smart questions. And OP's investigation does raise some questions to me at least.

1. Give a model a context with some # of actually random numbers and then ask it to generate the next random number. How random is that number? Repeat N times, graph the results, is there anything interesting about the results?

2. I remember reading about how brains/etc are kinda edge-balanced chaotic systems. So if a model is bad at outputting random numbers (ie: needs a very high temperature for the experiment from step 1 to produce a good distribution of random numbers) What if anything does that tell us about the model?

3. Can we add a training step/fine-tuning step that makes the model better at the experiment from step #2? What effect does that have on its benchmarks?

I'm not an ML researcher, so maybe this is still nonsense.

The algorithms are definitely not deterministic. That said I agree with your general point that experimenting on LLMs as if they're black boxes with unknown internals is silly.

EDIT: I'm seeing another poster saying "Deterministic with a random seed?" That's a good point--all the non-determinism comes from the seed, which isn't particularly critical to the algorithm. One could easily make an LLM deterministic by simply always using the same seed.