r/singularity • u/Snoo26837 ▪️ It's here • 12h ago
AI Muse wonder surpassed fable 5.1 on artificial analysis dashboard
195
u/Snoo26837 ▪️ It's here 12h ago edited 12h ago
Gemini
45
u/runaway-devil 12h ago
Not sure why its even there since most of the user base still don't have access to it
17
11
14
u/BriefImplement9843 12h ago
still topping human blind voting benchmarks. tops lmarena by a wide margin.
sonnet 5.5 is completely benchmaxxed. it's #35 on lmarena...lol.
5
u/no_witty_username 8h ago
Google busy building AI systems that predicting protein folding and trying to cure human diseases... I think they just dont care for agents and llm's honestly, seem like a priority thing versus capability.
3
12
u/the_real_ms178 12h ago
Why does Meta still keep using the old Muse Spark 1.1 via Meta.ai? Other labs are faster with their rollout and more generous even at the free tier.
6
33
40
u/Starks 12h ago
Google is just going to keep falling further behind when they don't even take a chance and release a model. It's as if they have nothing to immediately iterate with afterwards.
26
u/whoknowsifimjoking 12h ago
I mean, they seem to have a decent model in Gemini 4 Carbon, so I don't think they are necessarily falling behind on internal models, but it's absolutely infuriating as a user that they don't fucking release anything.
5
u/Concurrency_Bugs 11h ago
The key is they "seem to" have a decent model. We'll never know because no one ever gets to use their models except flash models
5
u/PeterPawn 12h ago
Agree. Business wise it's probably good to sell compute at a premium instead of models at a discount. As long as they keep iterating in the background..
2
1
u/___positive___ 3h ago
because nobody else has internal models. did the 700 math papers come from google carbon??? anthropic is set to release a model next month and openai in december.
7
u/Over-Dragonfruit5939 12h ago
Idk why they don’t. It doesn’t even have to be the best model, just release it already. I find their User Interface on the web to be the best of all of the ai companies with their images and graphs they include in thumbnails. Now they’re just lacking in the models reasoning abilities. I always found that Gemini is still among the best in stem subjects out of all of them.
15
u/PugilisticCat 10h ago
As a user of several of these products, they are almost completely fungible. Google correctly identifies that there is no reason to rush the frontier at the time being, given that it is not profitable and fundamentally doesn't change anything.
Google is content with trailing and dropping servicable models every few months since they have an actual path to revenue.
4
u/SanDiegoDude 10h ago
Still the best multimedia captioning game in town and its not even close. Gemini has its uses - anything to do with video or audio. All that Youtube data!
5
u/Ok_Buddy_Ghost 10h ago
no you don't understand, you have to launch a model every 3 months and your model has to top every chart or your company is DEAD!!
5
u/PugilisticCat 10h ago
Yeah I have been browsing these AI subs over the past few weeks (more like reddit has been forcing them down my throat), and it's clear to me that most people in these subs either lack experience or fundamentally misunderstand how companies deliver value. A lot of magical thinking in here.
-2
u/ozone6587 9h ago
So ironic that you think Google is fumbling releases on purpose and you call everyone else inexperienced. Every time I ask Google bootlickers this they never answer or dodge the question:
If this is 4D chess by Google then why didn't they release Gemini 3.5 Pro "in 1 month" after Google I/O as promised? What was the strategy there? Please cite sources rather than pull a reason out of your ass.
5
u/PugilisticCat 9h ago
I don't think that they are "fumbling releases on purpose", I think that they have just not prioritized them in the way that other companies have, resulting in these coming out more slowly.
The release delay of 3.5 was bad, but what did it change, exactly? The fundamental problems with all of this still exists.
Lol it's funny you also mention sources on this sub like it's anything other that pure speculation or cherry picking of poor benchmarking metrics.
-1
u/ozone6587 9h ago
How am I cherry picking anything? Occam's Razor boy. I see the intelligence gap and the performance in benchmarks and harnesses and naturally conclude OpenAI and Anthropic are leagues ahead.
You see all this as evidence of some higher tier 4D chess strategy with absolutely no evidence. We are not arguing at the same level. You are violating the Occam's Razor principle by ascribing some grander strategy to Google's fumble.
They are behind in almost all benchmarks for the models they have actually released. To say **I'M** cherry-picking requires a level of stupidity that is hard to comprehend.
5
u/PugilisticCat 9h ago
You see all this as evidence of some higher tier 4D chess strategy with absolutely no evidence.
I literally told you I don't think them fumbling releases they do plan is a strategy.
see the intelligence gap and the performance in benchmarks and harnesses and naturally conclude OpenAI and Anthropic are leagues ahead.
And this has achieved what for them, exactly? They have a better model that can only be effectively differentiated by the most obsessed users, and what do they win for it? Mountains of debt?
My point is that all of these frontier models are so insanely far from financial feasibility that it makes no sense to rush to the front of the line there.
Google has a path to revenue that does not include the debt laden approach, and I believe that they are taking it. This is integrations into their existing offerings, and providing hardware while people are still buying shovels.
I think that if there is a reality where these models actually show sustainable returns, they will choose to pivot there.
1
u/Tkins 7h ago
I don't use Google AI products very often, but I can still recognize they have the second highest AI userbase in the world after OpenAI and are increasing their market share.
They must be doing something right while not needing to pump out the top model.
People also forget the SOTA models are not used by many people. Anthropic said that Fable only accounted for 11% of their usage. I think it was 6% of revenue or something.
3
u/junglebunglerumble 9h ago
Dude calm down. The poster never said it was 4D chess by Google or that they hadn't had missteps. Their point was they aren't playing the same game as Anthropic or OpenAI and don't need to come out with a top line frontier model every 3 months like the two main players do. They can afford to rethink their models and decide not to release them if they want to try a different approach, because it isn't as important to them as it is to companies who are entirely reliant on frontier models as their entire product. Especially when Google's most visible AI focus currently is through AI mode inside Google search anyway, which has a >billion active monthly users and has no need for a frontier model.
Ironically, Google (AI mode) and Microsoft (Copilot and Office integrations) are big players who have actually managed to incorporate LLMs into existing profitable products despite not having top of the line models themselves. They simply don't need to be constantly pushing the boundaries of these models themselves and can be more focussed with what models they do develop, because Google/Microsoft/Apple's existing ecosystems are their product, not a LLM model
1
u/asfsdgwe35r3asfdas23 9h ago
This sounds like the people that do not do exercise during the year, and then try to get fit a month before the summer. You cannot train a frontier model when it is convenient to you, either you keep up with the progress or you are left behind.
3
u/TieBackground453 8h ago
Compute is continually improving. In several years it will be notably cheaper and more efficient to do the training. Doing the training now when it is more expensive and effective methods are less well established only lets you serve models for a loss for a few more years. In 5 years, everyone will still know who Google is and will be very happy to use their model if it is one of the best.
•
u/asfsdgwe35r3asfdas23 6m ago
And google will lack 5 years of knowledge about developing LLMs, so how is going to train a competitive model? Training LLMs is not trivial, it is extremely complex task, very few people know all the tricks to train a good model.
2
u/TicoTheCow 4h ago
What's the benefit of staying at the forefront? They can swoop in whenever they want really
1
u/Thorteris 10h ago
They’ll release a new checkpoint when gemini 4 releases and be right back near the top
16
u/Super-Award-2244 12h ago
Is Sonnet really better than fable and Astra?
11
u/MartinLik3Gam3 AGI 2027 ASI 2028 Singularity 2030 12h ago
It's good but the AA result is a little flawed since it used way way more compute than Astra to achieve that kind of result
9
u/gavinderulo124K 12h ago
At max probably yes but at max it uses so many tokens that its way more expensive abd takes ages to finish.
22
u/New_World_2050 12h ago
its a newer generation of sonnet than the fable generation and from my use, its a spectacular model.
13
u/FateOfMuffins 12h ago
https://epoch.ai/eci?view=graph&tab=leaderboard
AA is updated much faster but I'd trust Epoch ECI over it by far.
2
u/SwimmerOld6155 12h ago
How useful is a benchmark at that resolution? The difference between using e.g. 5.4 and Astra is night and day, yet they're only separated by 9 points? Even 5.5 vs 5.6 is noticeable at a separation of 3 points. Just looking at the y-axis it's calibrated way too low.
Shows they're similar in capability, though.
8
u/FateOfMuffins 12h ago
I believe it was estimated that 5 points of ECI roughly correlated to a doubling of the METR time horizon (which we can no longer measure) so this can be a proxy.
So 5.4 vs Astra would be about 4x time horizon here
4
u/SwimmerOld6155 12h ago
ahhh ok, so it's not a linear scale, gotcha. seems about right then
3
u/tiger_ace 10h ago
it literally says ECI is linear in the FAQ at the bottom but i agree with your confusion
i think it feels like a 5 point difference is a step function or more
1
u/SwimmerOld6155 10h ago edited 10h ago
ah, I did not see that, cheers
and agree, I don't think 5 points in a scale of like 150 should be immediately obvious when you use it lol. and to have Gemini Flash separated from Astra by 10 points is a bit nuts.
1
u/tiger_ace 9h ago
lol yeah i know what you mean: i also saw 3.7 flash and i was like okay wtf is going on
but if we think about astra being like 2 step functions / generations better then it seems to be coherent
1
u/XxDarkSasuke69xX 12h ago
Yes, it's on par pretty much and is pretty cheap in comparison. Definitely way more worth than Fable in terms of quality/price, no question on that.
1
u/BriefImplement9843 12h ago edited 12h ago
no. on lmarena it's on par with astra at #32. not even close to fable though. heavily benchmaxxed model. astra is also not very good outside math/coding, which is why the score is so low. fable on the other hand is good at everything and is only under opus 5.5 and gemini argon.
10
u/FateOfMuffins 12h ago
OK need to see the cost and performance of the other reasoning levels and Wonder Full cause Pareto curve wise it actually doesn't look so good for Wonder.
The "cheaper" Wonder is basically around same score as 6.1 Sol, 6 Astra and Gemini Argon, but more expensive than Astra, more than 2x expensive than Argon (which admittedly has the 50% discount), and 6x the cost of 6.1 Sol
7
6
u/Immediate_Simple_217 11h ago
I like the fact that Mark Zuckerberg didn't try to market it as a "too dangerous to release" model and just spilled at Google's face lol. Google didn't see this coming. If there is someone who can be compared do Google is Meta. They have tons of users personal data and own more than half of internet's social media.
8
u/duerra 12h ago
You could not pay me to use Muse. I absolutely, 100% do not trust any AI service coming out of Meta.
6
u/Smile_Clown 11h ago
I agree, although this is at odds with my geenral distastewith anti-fanboy attitude. Like grok, I should hate it, but I set that aside. For meta, I am not setting it aside.
they are too willing to pay billions in fines for "ask for permission later".
2
u/duerra 9h ago
I'd say they have an uphill battle but the fact is that they are probably in a good position to capture the "social audience", ie the masses and give them exposure to more AI capabilities than most have ever seen to this point. But I know people in security at Meta and even they are 😬
4
3
u/CryMoreT_T 10h ago
Anthropic and Openai fans (imagine being a fan of a company lol) on their way to comment benchmaxxed (models been out for an hour and they haven't tried it)
5
u/junglebunglerumble 8h ago
Yeah it's really weird, plus the constant over-dramatic reactions to every model release (people have legit been posting that OpenAI are finished since Opus 5.5, and were posting the Anthropic were finished when Astra came out - they'll never learn).
Meanwhile in the real world, the biggest tech players are actually busy incorporating LLMs into their existing massively profitable products (Apple Intelligence in iPhones, Copilot in MS Office, AI Mode in Google, Muse inside Instagram/WhatsApp/Facebook etc), yet the drama queens that clog up this sub are just obsessed with who comes top each week on benchmarks
2
u/Mistuv 9h ago
Space lizard fans (imagine being a fan of an alien) on their way to be amazed Muse Wonder, sometimes just falls off a cliff on random new benchmarks. Surely nothing suspicious about that.
-1
u/CryMoreT_T 9h ago edited 9h ago
Did the OpenAI fanboy get upset? Don't need to be a fan of Zuckerburg to know to first use something before criticizing it.
sometimes just falls off a cliff on random new benchmarks.
Source? Considering it just came out 4hrs ago
2
u/Bitter-College8786 12h ago
I used Muse Spark 1.3 but it didn't "feel" so extremely intelligent for its benchmarks. Hope Muse Wonder is better.
1
1
1
1
1
1
u/Gotisdabest 4h ago
Damn, that means in real world usage it's... Still trash, almost certainly. Meta benchmaxxes it like crazy particularly for the AA.
People have gone to giving credibility to AA like this benchmark didn't very recently have a configuration which basically said that Gemini 3.8 flash, muse spark and fucken gpt astra were very close models.
And what they did to solve that wasn't some radical overhaul in as much as tweaking the numbers to make astra score better.
They still use dubious benchmarks like the original HLE too.
I'll be very sceptical till we see actual usage and results. It's probably a fable distill benchmaxxed even past the already customary Claude benchmaxxing.
1
-1
u/benjaminovich 11h ago
I don't care about a result coming from whatever hodge-podge methodology these bench-marks have turned into. Meta suddenly becoming frontier is very hard to believe, and anyone who ever takes Meta on their word for anything is a sucker.
If it really turns out to be true during actual usage, I'll accept it of course, but it's gotta be able to do my laundry, cook me dinner and wank me off to sleep, before I would ever personally consider giving Meta that level of access to my pc.
0
u/BothYou243 11h ago
sonnet 5.5 did so, is a bigger deal I guess, but it thwy open source muse wonder, then then then it'll be fire
0
11h ago
[deleted]
2
u/DelphiTsar 10h ago
Chip makers yes, cloud providers no.
Cloud providers Costs + Depreciation is raising faster than revenue
0
0
113
u/JobKnown5009 12h ago
Meta's AI fighting at the border wasn't on my bingo card this year.