"The new diffusion-based image generator works by first using a digital encoder (that has been trained on publicly available datasets) to create the static that will ultimately make the picture. This requires a small amount of energy. Then, a liquid crystal screen known as a spatial light modulator (SLM) imprints this pattern onto a laser beam. The beam is then passed through a second decoding SLM, which turns the pattern in the laser into the final image."
Ok, I'm going to go read the paper now, but I have to say, this sounds like absolute nonsense.
kbelder · 1d ago
Ok, I don't understand all of the paper. I do like that fact that it's combining methods from physics and computer science, because there are probably many more discoveries to be made in those kinds of permutations.
But it seems, if I'm interpreting the paper correctly, that the real gain is that there is a static decoder filter that is generated for one particular prompt, and then it may be reused over and over at minimal cost, to generate butterfly after butterfly. Transforming the diffuser from a general-purpose generator to a specialized, more-efficient one that needs to be rebuilt if a different type of image is desired.
dekhn · 1d ago
To me the simplest place to start is Fourier Optics. A carefully set up collection of lenses, illuminators, samples, and cameras can compute (well, physically evaluate) the fourier transorm of a 2D image. https://en.wikipedia.org/wiki/Fourier_optics
(so you can for example do convolutions)
From what I can see in this area, the physical set up mainly computes forward passes by passing light through the SLM, while the computer computes the backward pass (using a mathematical equation that approximates the forward optics and computes its derivative). The nice thing is that SLMs are simply programmed to have a specific intensity pattern, it doesn't need a whole manufacturing pass to update the SLM.
Ok, I'm going to go read the paper now, but I have to say, this sounds like absolute nonsense.
But it seems, if I'm interpreting the paper correctly, that the real gain is that there is a static decoder filter that is generated for one particular prompt, and then it may be reused over and over at minimal cost, to generate butterfly after butterfly. Transforming the diffuser from a general-purpose generator to a specialized, more-efficient one that needs to be rebuilt if a different type of image is desired.
From what I can see in this area, the physical set up mainly computes forward passes by passing light through the SLM, while the computer computes the backward pass (using a mathematical equation that approximates the forward optics and computes its derivative). The nice thing is that SLMs are simply programmed to have a specific intensity pattern, it doesn't need a whole manufacturing pass to update the SLM.