Reference implementations are unmaintained and buggy.
For example https://github.com/huggingface/transformers/issues/27961 OpenAI's tokenizer for CLIP is buggy, it's a reference implementation, it isn't the one they used for training, and the problems with it go unsolved and get copied endlessly by other projects.
What about Flux? They don't say it was used for training, it wasn't, there are bugs with it that break cudagraphs or similar that aren't that impactful. On the other hand, it uses CLIP reference, and CLIP reference is buggy, so this is buggy...
liuliu · 12h ago
Congrats on finding a bug!
However, the keyword here is training / inference divergence. Unfortunately, nobody is going to spend multi-million to retrain a model, so our reimplementation needs to be bug-to-bug correct to use the trained weights properly. That's why the reference implementations are essential because it is from the original model trainers so you have the best "bet" on matching the training code properly.
To give you some concrete example of bugs we needs to maintain:
1. In SDXL, they use OpenClipG for text encoding, but wrongfully uses 0 as padding tokens (corresponding to symbol "!") whereas even for OpenClipG its own training, the endoftext token was used as padding token. However, if you switching SDXL to use endoftext token as padding token, due to training / inference divergence, you get subpar generated images.
2. In FLUX, we mainly use T5 as text encoder. However, T5 usually used as encoder with mask to exactly the same input length, to avoid extended impact of padding tokens. In FLUX, we don't apply mask for T5 text encoding, hence intuitively causing padding token to take more effect than it should. Again, "fixing" this bug without retraining you will get subpar generated images.
There are many examples like this, some are easier to fix some are not (HiDream uses a different ODE solver that is different than what we usually do for rectified flow, hence you need to negate its prediction to be compatible with existing samplers, but this is "easier to fix").
TL;DR: Yes, there are bugs in software, but we better to maintain bug-to-bug compatibility than trying to "fix" it, hence highlight the importance of a "done" reference implementation, rather than a usual "active" implementations in software industry otherwise.
I think if you read the issue carefully you would understand that the CLIP implementation in transformers and as published by OpenAI is wrong and does not match their trained model code; and that doing the fix I suggest, empirically for me and in theory, improves results.
42lux · 17h ago
You can disable clip l on flux without a loss in quality. You are also making an elephant out of a fly. CLIP is used everywhere.
doctorpangloss · 13h ago
Consider another interpretation: CLIP L in Flux can be disabled without a loss in quality because the way it is used is buggy!
electroglyph · 14h ago
It shouldn't take a lot of effort to fix a tokenizer...
doctorpangloss · 10h ago
People are a little too blinded by the insight porn of matching buggy behavior to just read and comprehend the issue. They can’t engage with the simpler and more pornographic insight porn that the reference implementations are buggy and do not match the trained artifacts.
The dataset is for trying out fine-tuning of the diffusion model. It's a reimplementation of SD3 by writing the code from scratch again, but the weights are taken from HuggingFace due to hardware constraints on my part.
jatins · 8h ago
> It's a reimplementation of SD3 by writing the code from scratch again, but the weights are taken from HuggingFace due to hardware constraints on my part.
Could you clarify what you mean by this part -- if the weights are taken from HF then what's the implementation for?
MoonGhost · 5h ago
My guess the weights from HF are used as initial state for the model because full training is too expensive. Then small dataset is used to train in further for short time. Which is fine tuning. Together it shows that model is 1) compatible 2) trainable. In theory it can be trained from scratch on big dataset. I didn't look in the code yet so the questions are: 1) can it be trained in parallel? 2) resources required for training?
Anyway, I may try to train it on limited specialized dataset...
elbear · 59m ago
The model consists of its architecture which is expressed as code, and its knowledge, which is gained through training.
montebicyclelo · 4h ago
> if the weights are taken from HF then what's the implementation for
The weights are essentially a bunch of floating point numbers, (grouped into tensors). The code says what operations to do with the weights. E.g. say you load matrix W from the weights, you could do `y = W @ x`, or `y = W.T @ x`, or `y = W @ W @ x` etc.
reedlaw · 20h ago
So this implements SD3 inference and fine-tuning?
Dwedit · 14h ago
Does using pure PyTorch improve performance on non-NVIDIA cards in any way? Or is PyTorch so highly optimized for CUDA that no other GPU vendors have a chance?
jwitthuhn · 13h ago
Pytorch also runs great on apple silicon, though it is hard to directly compare because Apple's high end GPUs can't compute anywhere near as much as nvidia's high end stuff.
e: I'll also add that pytorch does still have one oddity on apple silicon which is that it considers each tensor to be 'owned' by a particular device, either a cpu or gpu. Macs have unified memory but pytorch will still do a full copy when you 'move' data between the cpu and gpu because it just wasn't built for unified memory.
brcmthrowaway · 13h ago
Does pytorch work on AS out of the box? Or do you need some apple specific package
thom · 10h ago
`uv pip install torch` just works, set your default device to `mps:0`, enjoy the RAM. Depends what you're doing though - some stuff isn't implemented, so if you're trying to fit a Gamma/Beta/Student-T distribution you're out of luck.
chickenzzzzu · 14h ago
It is possible to run ML workloads on for example AMD devices via Vulkan. With newer extensions like cooperative matrix, and maybe also in the future some scheduling magic exposed by the driver through a new extension, the remaining single digit percent gap CUDA has will evaporate.
3abiton · 14h ago
It seems to be the case, although pytorch rocm is coming around slowly. Very slowly, if you get it working that is.
VeejayRampay · 14h ago
I believe pytorch works nicely with rocm, but I don't know if it's nicely to the point where it's "on par"
albert_e · 21h ago
Sounds like a great resources for learners.
Just wondering aloud --
Is there a tutorial/explainer by any chance that a beginner could use to follow along and learn how this is done.
Great resource Jeremy Howard is awesome, I have been waiting to take this course and follow along because anything older than a year in Deep Learning is already outdated. I hope they release a new version.
whiplash451 · 16h ago
I don’t think this is true. The fast.ai class covers a lot of fundamentals that are still valid and useful today.
This adds connections between the parameters of q, k, and v whereas the original doesn't, unless my very tired brain is missing something.
smus · 10h ago
Nope, they all depend on x and the same is true in this scenario
godelski · 8h ago
It is actually really common practice. It is a single linear layer so there's no connection intranodes. The reason to do this is because it is a bit less computationally intensive.
tldr: linear layers have an associative property
NoelJacob · 19h ago
So, that's Stable Diffusion without license constraints, is it?
Sharlin · 18h ago
No, the inference/training algorithms, being math, are not copyrightable. OP just wrote another implementation. What's copyrighted are the models, which OP did not train from scratch (having neither the training material nor the compute to do that).
Zambyte · 14h ago
> What's copyrighted are the models
Has this actually been tested yet? Or are we still at the stage of AI companies trying to pretend this into reality?
bravesoul2 · 12h ago
If the models are copyright protected then presumably they obeyed license on the upstream dependencies they included (i.e. the training data).
bredren · 6h ago
Is upstream dependency licensure necessary to establish copyright? For example, I Need a Haircut was still a unique work regardless of the rights to sample Alone Again.
dheera · 13h ago
I mean, if you take a match to a blank CD-ROM, or shoot neutrinos at a USB drive, there is a very small chance that you get the SD weights stored on them
Zambyte · 9h ago
You can say that about literally any digital information. This isn't really interesting in the context of the copyright status of AI models.
bravesoul2 · 12h ago
Oh :( wasn't what I thought it would be. Wondered why it wasn't more blown up on HN!
echelon · 17h ago
We should be specific when we say "models".
The code outlining the network vs. the resultant weights. (Also vs. any training, inference, fine tuning, misc support code, etc.)
The theoretical diagram of how the code networks and modules are connected is math. But an implementation of that in code is copyrightable.
Afaik, the weights are still a grey area. Whereas code is code and is copyrightable.
Weights are not produced by humans. They are the result of an automated process and are not afforded copyright protection. But this hasn't been tested in court.
If OpenAI GPT 4o weights leak, I think the whole world could use it for free. You'd just have to write the code to run them yourself.
bravesoul2 · 12h ago
I use model architecture for the code/math and weights for the weights to avoid confusion!
Then there are hyperparameters which are also needed to be known to use the weights with the model architecture.
MoonGhost · 5h ago
> I use model architecture for the code/math
Code is copyrightable and math is not. What about 'architecture'?
vrighter · 15h ago
which means he is still in full violation of their license
ineedasername · 14h ago
When I think of SD 3.5 (or any version) I think of the portion that results from training, i.e., the weights. The code seems less important? I mean as far as output quality is concerned, or performance. But I'm honestly not sure, and not trying to judge these efforts on that basis.
refulgentis · 17h ago
I'm embarrassed to ask: can someone elaborate on, say, what we have now that we didn't have before the repo existed?
I have studiously avoided making models, though I've been adjacent to their output for years now... I think the root of my confusion is I kinda assumed there was already PyTorch based scripts for inference / training. (I assumed _at least_ inference scripts were released with models, and kinda figured fine-tuning / training ones were too)
So then I'm not sure if I'm just looking at a clean room / dirty room rewrite of those. Or maybe everyone is using "PyTorch" but it's usually calling into CUDA/C/some proprietary thingy that is much harder to grok than a pure PyTorch impl?
Anyways, these arent great guesses, so I'll stop myself here. :)
rockemsockem · 13h ago
I believe this is the main piece
> with minimal dependencies
I haven't tried running SD 3.5 specifically, but it's built on hugging face libraries which I personally always find to be a mess of dependencies that make it really hard to setup without the exact configuration the original developers used (which is often not provided in enough detail to actually work). This makes it pretty hard to run certain models especially if it's a few months/years after the original release.
For example this appears to be the requirements for the stability AI reference implementation for SD3.5 and there are no versions specified and it includes "transformers" which is just an enormous library.
Ah, tyvm, that maps well onto my knowledge set, I have a ONNX inference wrapper written in Dart. However, I have never been able to leverage transformers.js ONNX demo code, i.e. have a reference to port to Dart.
IIRC it is written in an abstraction layer that supports a transformers-like API surface. This also makes it opaque to figure out what you're actually passing to the model, adding a Python dep mess on top of that...woo boy.
_tqr3 · 16h ago
Stability AI, creators of Stable Diffusion models release their products under own Stability AI Community License which is not "free" like MIT license. You are not allowed to modify the weights in certain ways.
This package is basically running the model (inference) and maybe fine tuning it using existing AI weights. A great way to learn but still could run into same licensing issue.
refulgentis · 15h ago
You can't finetune SD 3.5!?
I thought the community license stuff was about keeping people from using it in prod and charging for it without Stability getting at least a small taste.
This sucks.
I haven't been keeping up with gooner squad on Civit, but I did have some understanding SD was less popular, but I thought it was just because 3.5 came far too long after Flux with too little, if any, quality increase to be worth building new scaffolding for.
fc417fc802 · 8h ago
> You can't finetune SD 3.5!?
They don't want you finetuning it in specific ways that might make them look bad by association.
b0a04gl · 21h ago
does the DiT here actually capture cross-token attention the same way as full SD 3.5 or is it simplified for clarity?
caycep · 19h ago
How usable is the original academic source available from Ludwig Maximilian University CompViz group?
vergessenmir · 19h ago
Is there any notable properties of this implementation, are some parts slower, faster etc
eapriv · 19h ago
I find it hilarious that “from scratch” now somehow means “in PyTorch”.
monsieurbanana · 19h ago
If any "from scratch" post doesn't start with linking to a Primitive Technology video, I'm closing the tab
mkoubaa · 16h ago
Unless the author was raised by chimps I'm out
0cf8612b2e1e · 15h ago
Not fusing heavier elements from hydrogen? I’m out.
chairmansteve · 17h ago
Yeah. Should have done it in assembly.
mardifoufs · 15h ago
Pytorch is a pretty basic building block when you get to some degree of model complexity. It wouldn't really be interesting to implement autograd or some other things pytorch provides imo when the goal is to show a reimplantation of something as "higher" level as SD. It's similar to how I don't mind it when someone doesn't reimplement an OS, or a JavaScript engine when writing a web app from scratch.
And there's been a recent surge in abstractions over pytorch, and even standalone packages for models that you are just expected to import and use as an API (which are very useful, don't get me wrong!). So it's nice to see an implementation that doesn't have 10 different dependencies that each abstract over something pytorch does.
eapriv · 6h ago
> It wouldn't really be interesting
Andrej Karpathy did exactly that, and I think it’s quite interesting.
squircle · 21h ago
Although I'm leaning heavily away from being passionate about software development, this is a cool project, and its freaken awesome how anyone can now reinvent the wheel from first principles.
nothrowaways · 10h ago
Pure pytorch?
CamperBob2 · 20h ago
Add a Hugging Face Token in get_checkpoints.py before running the script.
Can you be a bit more specific here? It's not clear what such a token is, what it takes to get one, or where it would be placed in get_checkpoints.py.
einsteinx2 · 19h ago
> what such a token is
An API token from Hugging Face
> what it takes to get one
You generate them in your Hugging Face account
> where it would be placed in get_checkpoints.py.
Line 59 in the empty quotes where it says token = “”
Dwedit · 14h ago
Leaving off the "API" part from "API Token" causes confusion, since AI models tokenize all text into "tokens" before running the model. It's using the same word to describe two very different things.
einsteinx2 · 14h ago
Yep totally. Fwiw I’m not part of the project I just saw the comment and figured I’d try and help.
CamperBob2 · 18h ago
Ah, I see it now, thanks.
That's the kind of thing that, stylistically speaking, it's good to define at the very top of the module.
einsteinx2 · 14h ago
Agreed. I’m not part of the project I just saw your comment and figured I’d try and help.
hkon · 17h ago
now do it in minecraft
theturtle · 20h ago
Cool. Can it still make images of Anne Hathaway leading a herd of blue giraffes on the Moon?
IncreasePosts · 18h ago
Seems difficult, as there are no known portraits of Anne Hathaway
The minRF project is very easy to start with training small diffusion models with rectified flow: https://github.com/cloneofsimo/minRF
Also, the reference implementation of SD 3.5 is actually minimalistic too: https://github.com/Stability-AI/sd3-ref
For example https://github.com/huggingface/transformers/issues/27961 OpenAI's tokenizer for CLIP is buggy, it's a reference implementation, it isn't the one they used for training, and the problems with it go unsolved and get copied endlessly by other projects.
What about Flux? They don't say it was used for training, it wasn't, there are bugs with it that break cudagraphs or similar that aren't that impactful. On the other hand, it uses CLIP reference, and CLIP reference is buggy, so this is buggy...
However, the keyword here is training / inference divergence. Unfortunately, nobody is going to spend multi-million to retrain a model, so our reimplementation needs to be bug-to-bug correct to use the trained weights properly. That's why the reference implementations are essential because it is from the original model trainers so you have the best "bet" on matching the training code properly.
To give you some concrete example of bugs we needs to maintain:
1. In SDXL, they use OpenClipG for text encoding, but wrongfully uses 0 as padding tokens (corresponding to symbol "!") whereas even for OpenClipG its own training, the endoftext token was used as padding token. However, if you switching SDXL to use endoftext token as padding token, due to training / inference divergence, you get subpar generated images.
2. In FLUX, we mainly use T5 as text encoder. However, T5 usually used as encoder with mask to exactly the same input length, to avoid extended impact of padding tokens. In FLUX, we don't apply mask for T5 text encoding, hence intuitively causing padding token to take more effect than it should. Again, "fixing" this bug without retraining you will get subpar generated images.
There are many examples like this, some are easier to fix some are not (HiDream uses a different ODE solver that is different than what we usually do for rectified flow, hence you need to negate its prediction to be compatible with existing samplers, but this is "easier to fix").
TL;DR: Yes, there are bugs in software, but we better to maintain bug-to-bug compatibility than trying to "fix" it, hence highlight the importance of a "done" reference implementation, rather than a usual "active" implementations in software industry otherwise.
(I maintain the most complete reimplementation of SoTA media generation models in Swift: https://github.com/drawthingsai/draw-things-community/tree/m.... So I tend to think that I know one or two about "reimplementation from scratch".)
The training dataset is very small, only including fashion-related pictures: https://github.com/yousef-rafat/miniDiffusion/tree/main/data...
Could you clarify what you mean by this part -- if the weights are taken from HF then what's the implementation for?
Anyway, I may try to train it on limited specialized dataset...
The weights are essentially a bunch of floating point numbers, (grouped into tensors). The code says what operations to do with the weights. E.g. say you load matrix W from the weights, you could do `y = W @ x`, or `y = W.T @ x`, or `y = W @ W @ x` etc.
e: I'll also add that pytorch does still have one oddity on apple silicon which is that it considers each tensor to be 'owned' by a particular device, either a cpu or gpu. Macs have unified memory but pytorch will still do a full copy when you 'move' data between the cpu and gpu because it just wasn't built for unified memory.
Just wondering aloud --
Is there a tutorial/explainer by any chance that a beginner could use to follow along and learn how this is done.
tldr: linear layers have an associative property
Has this actually been tested yet? Or are we still at the stage of AI companies trying to pretend this into reality?
The code outlining the network vs. the resultant weights. (Also vs. any training, inference, fine tuning, misc support code, etc.)
The theoretical diagram of how the code networks and modules are connected is math. But an implementation of that in code is copyrightable.
Afaik, the weights are still a grey area. Whereas code is code and is copyrightable.
Weights are not produced by humans. They are the result of an automated process and are not afforded copyright protection. But this hasn't been tested in court.
If OpenAI GPT 4o weights leak, I think the whole world could use it for free. You'd just have to write the code to run them yourself.
Then there are hyperparameters which are also needed to be known to use the weights with the model architecture.
Code is copyrightable and math is not. What about 'architecture'?
I have studiously avoided making models, though I've been adjacent to their output for years now... I think the root of my confusion is I kinda assumed there was already PyTorch based scripts for inference / training. (I assumed _at least_ inference scripts were released with models, and kinda figured fine-tuning / training ones were too)
So then I'm not sure if I'm just looking at a clean room / dirty room rewrite of those. Or maybe everyone is using "PyTorch" but it's usually calling into CUDA/C/some proprietary thingy that is much harder to grok than a pure PyTorch impl?
Anyways, these arent great guesses, so I'll stop myself here. :)
> with minimal dependencies
I haven't tried running SD 3.5 specifically, but it's built on hugging face libraries which I personally always find to be a mess of dependencies that make it really hard to setup without the exact configuration the original developers used (which is often not provided in enough detail to actually work). This makes it pretty hard to run certain models especially if it's a few months/years after the original release.
For example this appears to be the requirements for the stability AI reference implementation for SD3.5 and there are no versions specified and it includes "transformers" which is just an enormous library.
https://github.com/Stability-AI/sd3.5/blob/main/requirements...
IIRC it is written in an abstraction layer that supports a transformers-like API surface. This also makes it opaque to figure out what you're actually passing to the model, adding a Python dep mess on top of that...woo boy.
This package is basically running the model (inference) and maybe fine tuning it using existing AI weights. A great way to learn but still could run into same licensing issue.
I thought the community license stuff was about keeping people from using it in prod and charging for it without Stability getting at least a small taste.
This sucks.
I haven't been keeping up with gooner squad on Civit, but I did have some understanding SD was less popular, but I thought it was just because 3.5 came far too long after Flux with too little, if any, quality increase to be worth building new scaffolding for.
They don't want you finetuning it in specific ways that might make them look bad by association.
And there's been a recent surge in abstractions over pytorch, and even standalone packages for models that you are just expected to import and use as an API (which are very useful, don't get me wrong!). So it's nice to see an implementation that doesn't have 10 different dependencies that each abstract over something pytorch does.
Andrej Karpathy did exactly that, and I think it’s quite interesting.
Can you be a bit more specific here? It's not clear what such a token is, what it takes to get one, or where it would be placed in get_checkpoints.py.
An API token from Hugging Face
> what it takes to get one
You generate them in your Hugging Face account
> where it would be placed in get_checkpoints.py.
Line 59 in the empty quotes where it says token = “”
That's the kind of thing that, stylistically speaking, it's good to define at the very top of the module.