Ask HN: LLMs and Information Theory

You know, it really takes about tens years of thinking about something to really begin to understand it at a fundamental level (which should tell you something about the true pace of human understanding.)

Shannon said his part over fifty years ago now, and we’re still trying to interpret that.

Let’s see, what might LLMs confirm, deny, or help us ponder?

- the correlation between the volume of training materials and the size and accuracy of their output

- the proficiency and deviation of various training techniques

There is so much more of course, it’s questions like yours that get things rolling.