I honestly think of the T5 model family to sort of be the real beginning of this open model craze - I know BERT was already popular for classification etc, but T5 was the first sort of generally useful model, was exceptionally simple to fine-tune, and is still in use today (t5 base is still averaging over a million downloads a month on huggingface), has tons of variants and sort of kickstarted this whole community. US labs get a lot of flack but Google has been super supportive and open in a lot of ways that has pushed this whole endeavor forward, even if I feel like they've sort of declined in transparency in recent years with their open models.

Couldn't agree more and can only recommend T5 as a base to anyone. It's amazing to get started, whether as a learning resource or for real (albeit very tailored) applications. Especially the BigScience fine tunes are such a great starting point and I, as a total layman, have learned a lot, especially concerning how a model can be optimised via all manner of methods since even mt0 is small enough to where one can do multiple runs with wildly different outcomes in quick succession. Quantise, prune vocab, try different approaches to sourcing training data, retrain dozens of times, it's all pleasantly possible on consumer hardware [0] and surprising how much you can squeeze in functionality-wise. How does latency change vs memory usage, what affects format reliability, how languages and scripts affect training and the efficiency equation, etc. are all quite exciting to learn.

Understand why T5Gemma is no longer under Apache-2.0 and honestly, have not seen that much advantage when testing that vs T0 in my experiments either way, but still, there are good reasons why plain old T5 and its descendants are still popular, licensing being among them.

Gemma team also has very consistently interesting models, especially like DiffusionGemma. Ironic, as (beside 2.5 Pro), I have never warmed up to the Gemini series of models but rate Gemma models far higher than e.g. Qwen in direct competition. In any case, thanks to the teams behind these for making as much possible.

[0] As in proper consumer hardware, not a cluster of DGX Sparks or Mac Studios solely for experiments that sometimes are asserted as being consumer grade...

There are some issue using T5 for regular people since the tokenizer has some issues. mT5 though, that one is pretty solid to start with.

Example: https://stackoverflow.com/questions/75851029/t5-fine-tuned-m...

There was also CodeT5 which I thought could inspire some source-to-source transpiling or other tricks.

Google had done much to lay the framework for current models (and is currently doing good stuff with Gemma). But after ChatGPT took off I don’t think Google continued to release any big models freely before meta released its models (maybe it was because Google didn’t have any models at all during the time)

yeah they were probably thinking, "we really gave away the transformer recipe for free?"

Thanks!

When is Gemma5 coming out? :-)

Hey, with the changes at Deepmind, is the Gemma project still ongoing?

They just said they work on Gemma at Deepmind. That stands to reason that as of time of writing, we should assume the answer is yes.

What?

I'm just beginning to enjoy the release of Gemma E4B why are you giving me a cold shower here telling me there may be no Gemma 5?

Here's hoping the supreme leader of the US does a speech like Chairman Xi.

And yet it's not called eleuther.cpp or bert.cpp. And the famous subreddit isn't called r/localbert but r/locallama.

In fairness, people also don’t say “let me ask llama” or “llama told me…” like they do with “GPT”.

Colloquialisms don’t establish importance, just popularity.

I see no contradiction. GPT as synonym for AI and llama as symbol for open weights models. The original comment said llama had set something off in the open weights community. And I say you see that with llama.cpp, llamaindex, r/localllama. No one bothered to update them to qwen.cpp or qwenindex. And they were never localgpt or gpt.cpp.

Culture and language use says a lot about importance of ideas. Regardless of what was first or what some scholar sees as historically more important.

Oh fuck, tjwebbnorfolk's got demands. Time to start working nights.

It's already been trained. GDM said it was going to be released and it hasn't yet. Not sure why so much snark

Don’t ask engineers at big companies for release dates, they either don’t know or are not at liberty to say.

Don't even ask me at a small company for release dates

Just forget about the concept of a release date.

Bucket crab.

"Oh no, a company and its people in it are inconvenienced by what people want them to do, don't even bother vocalizing your desires!"

They already announced it for release in April. Why am I being attacked here

It's not you; the culture around here is not what it used to be.

So did Picard.