$/M tokens is stupid

[u/lordmairtis]

https://medium.com/@zdengineering/anthropic-is-misleading-us-639b61fe01f0

Maybe I'm late to the party, or idk, but truly, Opus 5.5 has double the $/MT price as Sonnet 5.5, almost identical scores on the test, costs almost the same to do the test. GPT 6.1 Sol maybe (?) has barely worse results, same $/MT, tasks cost multitudes less. So great, we now need to measure this for ourselves too!

And don't even get me started on simple tasks. It is "intelligent", but if you use the larger model accidentally, simple prompts will eat your limits for lunch. We need proxies, routers and all that just to make sure Opus doesn't get easy questions, but then again, we need to check whether the model used is actually cheaper for the task, or just by the published numbers....

</rant>

Feels like the only way forward is really just running some simpler task for all new models and check if a new model does better or the one we are using already, and for how much... does not guarantee better and/or cheaper results on the actual work still.

[u/skidanscours]

Don't forget that they all have their own tokenizer too.  So the same text input is about 30% more tokens on Anthropic's models than OpenAI.

[u/lordmairtis → u/skidanscours]

okay, but Opus 5.5 doing the same task as Sonnet 5.5 for about the same price? costs double the $/Token.

i mean i would be very happy if this was the case for all tasks, big models using less tokens for easy work, perfect. it's just nem consistently the case

[u/DonutHoles4Ever → u/lordmairtis]

End of the day they want more money for less. Obviously a orchestrator would help route better but who would add that unless competition has it and people rather pay for that.

[u/Ctbhatia]

the tokenizer is what breaks the $/M comparison, the same text is a different token count per provider, so the published number says little until you count on your own workload. price it per finished task and include the cache hit rate.

[u/YoungSilent232]

Everything, from quality to how many tokens are output to cost per task per model is specific to your individual usecase, don’t rely on published cost per task for anything tbh.

And if you’re on subscription rather than enterprise jobs, it’s also not just cost per task but it’s actually % of weekly usage per task. So none of these benchmark cost numbers even matter when trying to compare different providers

[u/WishboneOk305 → u/YoungSilent232]

yeah its up to the user themselves to do a CBA, if the user is a hedge funds with unlimited funding for a simple binary answer(buy/sell) theyll just use the most powerful and expensive model cost be damned.

[u/PhilosophyforOne]

And still somehow Anthropic gives me like 3x as much usage on the 200/m max sub.

[u/Reasonable_Swing_503 → u/PhilosophyforOne]

Yeah codex gpt 6 Astra is in a pretty fucked situation now.

[u/Lazy--Expression]

I use both Codex and Claude code. Claude delivers about 5 times as much work as Codex before reaching usage limits.

[u/one-wandering-mind]

Also fine to have max comparisons, but how about alao the default or reccomended thinkkng level. 

[u/MrSnowden → u/[deleted]]

I expect the same thing and I think they are going there now.  Use JEV style tools for quick answers, lower tier models for fast response and then escalate to higher models.  Our proprietary harness does that now (mostly to redirect traffic to OS locally hosted models). But I feel that will get built into the core tools. 

[u/lordmairtis → u/MrSnowden]

low tier model for fast response is nonsense, many low tier models are really slow. Luna can think about anwers for minutes and cost 5 cents still