r/OpenAI • u/lordmairtis • 1d ago

News $/M tokens is stupid

https://medium.com/@zdengineering/anthropic-is-misleading-us-639b61fe01f0

Maybe I'm late to the party, or idk, but truly, Opus 5.5 has double the $/MT price as Sonnet 5.5, almost identical scores on the test, costs almost the same to do the test. GPT 6.1 Sol maybe (?) has barely worse results, same $/MT, tasks cost multitudes less. So great, we now need to measure this for ourselves too!

And don't even get me started on simple tasks. It is "intelligent", but if you use the larger model accidentally, simple prompts will eat your limits for lunch. We need proxies, routers and all that just to make sure Opus doesn't get easy questions, but then again, we need to check whether the model used is actually cheaper for the task, or just by the published numbers....

</rant>

Feels like the only way forward is really just running some simpler task for all new models and check if a new model does better or the one we are using already, and for how much... does not guarantee better and/or cheaper results on the actual work still.

75 Upvotes

91% Upvoted

13 comments sorted by

View all comments

-1

u/[deleted] 1d ago

[deleted]

1

u/MrSnowden 1d ago

I expect the same thing and I think they are going there now.  Use JEV style tools for quick answers, lower tier models for fast response and then escalate to higher models.  Our proprietary harness does that now (mostly to redirect traffic to OS locally hosted models). But I feel that will get built into the core tools. 

1

u/lordmairtis 1d ago

low tier model for fast response is nonsense, many low tier models are really slow. Luna can think about anwers for minutes and cost 5 cents still