However, the keyword here is training / inference divergence. Unfortunately, nobody is going to spend multi-million to retrain a model, so our reimplementation needs to be bug-to-bug correct to use the trained weights properly. That's why the reference implementations are essential because it is from the original model trainers so you have the best "bet" on matching the training code properly.

To give you some concrete example of bugs we needs to maintain:

1. In SDXL, they use OpenClipG for text encoding, but wrongfully uses 0 as padding tokens (corresponding to symbol "!") whereas even for OpenClipG its own training, the endoftext token was used as padding token. However, if you switching SDXL to use endoftext token as padding token, due to training / inference divergence, you get subpar generated images.

2. In FLUX, we mainly use T5 as text encoder. However, T5 usually used as encoder with mask to exactly the same input length, to avoid extended impact of padding tokens. In FLUX, we don't apply mask for T5 text encoding, hence intuitively causing padding token to take more effect than it should. Again, "fixing" this bug without retraining you will get subpar generated images.

There are many examples like this, some are easier to fix some are not (HiDream uses a different ODE solver that is different than what we usually do for rectified flow, hence you need to negate its prediction to be compatible with existing samplers, but this is "easier to fix").

TL;DR: Yes, there are bugs in software, but we better to maintain bug-to-bug compatibility than trying to "fix" it, hence highlight the importance of a "done" reference implementation, rather than a usual "active" implementations in software industry otherwise.

(I maintain the most complete reimplementation of SoTA media generation models in Swift: https://github.com/drawthingsai/draw-things-community/tree/m.... So I tend to think that I know one or two about "reimplementation from scratch".)

Could you clarify what you mean by this part -- if the weights are taken from HF then what's the implementation for?

tldr: linear layers have an associative property

Has this actually been tested yet? Or are we still at the stage of AI companies trying to pretend this into reality?

The code outlining the network vs. the resultant weights. (Also vs. any training, inference, fine tuning, misc support code, etc.)

The theoretical diagram of how the code networks and modules are connected is math. But an implementation of that in code is copyrightable.

Afaik, the weights are still a grey area. Whereas code is code and is copyrightable.

Weights are not produced by humans. They are the result of an automated process and are not afforded copyright protection. But this hasn't been tested in court.

If OpenAI GPT 4o weights leak, I think the whole world could use it for free. You'd just have to write the code to run them yourself.

IIRC it is written in an abstraction layer that supports a transformers-like API surface. This also makes it opaque to figure out what you're actually passing to the model, adding a Python dep mess on top of that...woo boy.

I thought the community license stuff was about keeping people from using it in prod and charging for it without Stability getting at least a small taste.

This sucks.

I haven't been keeping up with gooner squad on Civit, but I did have some understanding SD was less popular, but I thought it was just because 3.5 came far too long after Flux with too little, if any, quality increase to be worth building new scaffolding for.

Andrej Karpathy did exactly that, and I think it’s quite interesting.

That's the kind of thing that, stylistically speaking, it's good to define at the very top of the module.