Releasing open weights for FLUX.1 Krea

Hi. Thanks for this. What is your goal of doing so? From a business standpoint. Or is it purely altruistic?

Describing it as "Octopus DJ with no fingers" got rid of the hands for me, but interestingly, also removed every anthropomorphized element of the octopus, so that it was literally just an octopus spinning turntables.

The model looks incredible!

Regarding this part: > Since flux-dev-raw is a guidance distilled model, we devise a custom loss to finetune the model directly on a classifier-free guided distribution.

Could you go more into detail on the specific loss used for this and any other possible tips for finetuning this that you might have? I remember the general open source ai art community had a hard time with finetuning the original distilled flux-dev so I'm very curious about that.

thanks for doing this!

what does " designed to be compatible with FLUX architecture" mean and why is that important?

Quick napkin math assuming bfloat16 format : 1B * 16 bits = 16B bits = 2GB. Since it's a 12B parameter model, you get around ~24GB. Downcasting to bfloat16 from float32 comes with pretty minimal performance degradation, so we uploaded the weights in bfloat16 format.

That's a good ballpark for something quantized to 8 bits per parameter. But you can 2x/4x that for 16 and 32 bit.

A parameter can be any size float. Lots of downloadable models are FP8 (8 bits per parameter), but it appears this model is FP16 (16 bits per parameter)

Often, the training is done in FP16 then quantized down to FP8 or FP4 for distribution.

We actually have a GitHub repository to help with inference code.

Check this out: https://github.com/krea-ai/flux-krea

Let me see if we can add more details on the blog post and thanks for the flag!

We have not added a separate RTX accelerated version for FLUX.1 Krea, but the model is fully compatible with existing FLUX.1 dev codebase. I don't think we made a separate onnx export for it though. Doing 4~8 bit quantized version with SVDQuant would be a nice follow up so that the checkpoint is more friendly for consumer grade hardware.

You can find details about the license here: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/mai...

In a nutshell, it follows the same license as BFL Flux-dev model.

for the love of the game, not for corporate greed

I love owls. Photorealism was one of the focus areas for training because "AI look" (e.g. plastic skin) was biggest complaint for FLUX.1 model series. Photorealism was achieved with both careful curation of finetuning and preference dataset.

Thanks!

From a business point of view, there are many use-cases. Here's a list in no particular order:

- You can quickly generate assets that can be used _alongside_ more traditional tools such as Adobe Photoshop, After Effects, or Maya/Blender/3ds Max. I've seen people creating diffuse maps for 3D using a mix of diffusion models and manual tweaking with Photoshop.

- Because this model is compatible with the FLUX architecture, we've also seen people personalizing the model to keep products or characters consistent across shots. This is useful in e-commerce and fashion industry. We allow easy training in our website — we labeled it Krea 1 — to do this, but the idea with this release is to encourage people with local rigs and more powerful GPUs to be able to tweak with LoRAs themselves too.

- Then I've seen fascinating use-cases such as UI/UX designers who prompt the model to create icons, illustrations, and sometimes even whole layouts that then they use as a reference (like Pinterest) to refine their designs on Figma. This reminds me of people who have a raster image and then vectorize it manually using the pen tool in Adobe Illustrator.

We also have seen big companies using it for both internal presentations and external ads across marketing teams and big agencies like Publicis.

EDIT: Then there's a more speculative use-case that I have in mind: Generating realistic pictures of food.

While many restaurants have people who either make illustrations of their menu items and others have photographers, the big tail of restaurants do not have the means/expertise to do this. The idea we have from the company perspective is to make it as easy as snapping a few pictures of all your dishes and being able to turn all your menu (in this case) into a set of professional-looking pictures that accurately represent your menu.

Thank you! Glad you find it helpful. The model is focused on photorealism so it should be able to generate most realistic scenes. Although, I think using 3D engines would be more suitable for typical cases for robotics training since it gives you ground truth data on objects, location, etc.

One interesting use case would be if you are focusing on a robotics task that would require perception of realistic scenes.

Hey thanks! I’ll ask Sangwu to hop here to answer this and give a more research-oriented answer

yeah data really is everything that was the number one lesson from this whole project

Haha thanks! Do you happen to have a use-case for it?

We used two types of datasets for post-training. Supervised finetuning data and preference data used for RLHF stage. You can actually use less than < 1M samples to significantly boost the aesthetics. Quality matters A LOT. Quantity helps with generalisation and stability of the checkpoints though.

Yeah, there are still imperfections. But, it’s surprising to us how much the quality can be improved without the need of a whole pre-training (re-) run.

Yup... We have the following: https://www.krea.ai/pricing

Though we wanted to keep this technical blogpost free from marketing fluff, but maybe we over-did it.

However, sometimes it's hard to give an exact price per image, as it depends on resolution, number of steps, whether a LoRA is being used or not, etc.

Copying and pasting Sangwu’s answer:

We used two types of datasets for post-training. Supervised finetuning data and preference data used for RLHF stage. You can actually use less than < 1M samples to significantly boost the aesthetics. Quality matters A LOT. Quantity helps with generalisation and stability of the checkpoints though.

It's because https://www.krea.ai/blog/flux-krea-open-source-release contains this:

  <link rel="canonical" href="https://www.krea.ai/blog/new-krea">

Our software follows canonical links when it finds them.

I've fixed the link above now (and rolled back the clock on the submission, to make up for lost time) but you might want to fix this for future pages.