One of the reasons I’m a bit bearish in AI is related to this, there still seem to be some basic tests that AI fail when trying to figure out, is it combining vectors in its training data, or actually understanding. In places where the training data is robust, it feels like understanding, but hit any place where it’s sparse and it stays around the problem.
ChatGPT used to not be able to generate an image of a full glass of wine(you can search YouTube for a long video of this). Probably because there had never been a full glass of wine in its training data.
They fixed that, maybe altering how they label things to include factors they hadn’t thought of? Maybe they literally watched the video and just added images of full glasses of wine? But there was a big article that went on to say “the problem has been solved, they think now”
So i went into chatGPT and started saying “generate me an image with x% full glass of wine, and it doesn’t do it.
Occasionally it gets close but with no consistency. And this is a problem that is very logical/mathematical, I could develop a script to understand the nature of the task easily. weird.
anonym29 · 3h ago
Have you tried with frontier models from other providers, like Anthropic, Alphabet, xAI, etc? I am finding the capabilities with this specific test to be vary quite wildly among "leading" models.
ChatGPT used to not be able to generate an image of a full glass of wine(you can search YouTube for a long video of this). Probably because there had never been a full glass of wine in its training data.
They fixed that, maybe altering how they label things to include factors they hadn’t thought of? Maybe they literally watched the video and just added images of full glasses of wine? But there was a big article that went on to say “the problem has been solved, they think now”
So i went into chatGPT and started saying “generate me an image with x% full glass of wine, and it doesn’t do it.
Occasionally it gets close but with no consistency. And this is a problem that is very logical/mathematical, I could develop a script to understand the nature of the task easily. weird.