I have served as a CTO of the tech venture I co-founded, for the better part of three years. The foremost selling point of our product, and it is our customers who insist upon it, is quality and reliability. In a heavily regulated domain like the one we work in, these are not so much virtues as preconditions: an error does not merely inconvenience, it can prove fatal.
This left us, from the very first day, precisely one option: to restrict LLM-generated code, the risks of doing otherwise being too grave to contemplate.
Under the internal framework that governs their use, no decision-making, no brainstorming, no review, no deployment, and no other critical activity is ever delegated to an LLM. They are confined to menial, repetitive labour. Were we to dismiss the LLMs tomorrow, we should deliver the same output, only later. That, in its entirety, is what they contribute.
An approach such as this does not endear one to the AI boosters, as you may imagine. Over the years I have received no shortage of abuse for it: much of it frankly ad hominem; and on an uncomfortable number of occasions, the objection on offer was misogyny, unadorned: a description of my body, rendered in the register of technical disagreement. It arrives not as an insult, but as an opinion, often from those who are ill-informed about software development themselves.
The other threats are a ritual rodomontade: that I would be "left behind", that I was "behind the curve", that I was "losing the game". They are comical in the present day, for not one of them has manifested, and events have proved rather the contrary.
In my own business circle, executives from adjacent companies have come to me (reluctantly, which is the honest word for it) to ask for help with the technology, their software problems having eaten into the goodwill of their customers. (Spoiler alert: the LLMs are the reason.) I have my own enterprise to mind, and the hourly consultation fee is a rounding error upon my finances. What draws me is the opportunity to observe disaster at scale, first-hand. And it is worse than anything I had imagined, especially given my stance on LLMs.
In recent months alone I have been involved with at least half a dozen companies in which LLMs have produced an overwhelming proliferation of defects. The most alarming are failures that even a junior programmer would instinctively avoid. The recurring ones include, though they are not limited to:
- A farrago of dead, redundant and convoluted code, and with it, a needlessly compounded population of bugs, with no module having been written to be reused.
- Functionality where nobody knows how any of it works, least of all the very teams who implemented the features themselves.
- A categorical indifference to product and design specifications, and an end-user experience in ruins, because it is the most human part of any tech product, and its validation was entrusted to a cloud service whose singular talent is the appearance of a plausible answer.
- Gaps in accessibility (a big deal, considering they wish to sell this product in markets and domains where it is mandated to a certain level).
- Failures in data handling and security, in ways usually expected of LLMs.
To me, the whole exercise is a refutation of anyone who claims that LLMs are fit to manage code without a human reading through a line of it. The rudiments of the technology, context limits and degradation, are enough to make it evident that these instruments are hopeless at any realistic scale, which is to say at anything that could be a viable business; and no quantity of compute remedies this, contrary to what LLM marketing tends to claim.
The firms I have advised have spent more time, more capital and more resources to build less product than I have shipped with a more measured approach, and every hour of that excess went into cat-and-mouse games with the most basic issues that even the slightest human oversight would have fixed. Cognitive debt is the gravest of the perils here, and the discipline that holds it in check is simple: if you expect to scale & ship something useful, never write code faster than you can reasonably grasp.
Those who went all-in on LLMs, recklessly, are unwilling to face the consequences of decisions born of gross ignorance. My foot comes down every time I am asked "why not use Claude Doopus Table Five Point Something to solve it?", and it comes down firmly: if that is your conception of a solution, I am the wrong person to consult, and it's my sign to walk away from a sinking ship.
Had actual professionals been responsible for output of this calibre, the scrutiny would have been vocal and plentiful. But it appears that lighting money on fire is more forgivable when the handiwork is an AI-psychosis in the management. What it amounts to is a grotesque tolerance: for disregarding the labour that builds the business, and for disregarding the very users who pay for it.
For the entire debacle, one of them blamed his engineers, calling them "underperforming" for not "using enough tokens", saying that "some of them use EVEN less than the team average" (My most profound felicitations to this vainglorious bean-counter who just discovered basic arithmetic).
And the rot runs through every level of these organisations. When I descend to the bottom of an issue I take the account of every party to it; and speaking with the members of the UX teams directly, I could see the meticulous care they had put into getting the experience right: care that was discarded because management preferred whatever the LLM saw fit to emit, and because "perhaps Claude made a better judgement writing the code".
Despite fruitful efforts that made themselves felt on the balance sheet, for some executives the verdict is unchanged: "Great, we can go back to using Claude again." One of the grander swindles the LLM companies have brought off is persuading decision-makers that the most expensive model is the price of not being "left behind", a notion parroted by some of the loudest voices in the industry, software development included.
If over-reliance on LLMs is money set on fire, paying for the most expensive model is gasoline poured over the blaze.
If there is anything I look forward to, should this madness ever subside, it is the return of a little sense. That is one way of looking ahead of the curve: the same curve on which everyone, quite unsuccessfully, insisted I would be left behind.
----
EDIT: While there's understandably, ample scepticism in the comments suggesting this was generated by an LLM, this is very unapologetically, my original style of writing that was lost to not sounding "LLM like". I have responded to comments engaging this aspect in good faith.
As for the rest: You have the liberty to opinionate about it. But your lack of vocabulary isn't an invitation for arrogant certitude about the originality of my thoughts, and I stand sans obligation to address your rudimentary observations.