r/cscareerquestions • u/FearlessPark4588 • 7h ago

Statistical rises in PR counts in the AI era are actually nonsense

Let me explain. My org heavily pressured us to get PR counts up. So, engineers did that by splitting up work into smaller items. It's actually more work --more context switching-- to review 2 smaller items than one item that is effectively the same thing.

We all know lines of code, PR counts, tickets closed are all proxies for actual productivity that moves businesses forward, and our work is too abstract to try and measure the real desired outcome (ostensibly: higher profits, more sales, etc) so they use this very rigid quantifiers.

32 Upvotes
  • permalink
  • reddit

83% Upvoted

16 comments sorted by

32

u/pm_me_github_repos AI/ML Research Engineer 6h ago

I don’t think businesses should be evaluating on PR counts, LOC, etc.

But I’m not reviewing your AI-generated 10k diff PR. Break that into targeted PRs if you want a real review.

4

u/GargantuanCake 6h ago

They absolutely shouldn't. That's something that's been known for as long as software engineering has been a thing. Line of code is the worst possible metric you can measure productivity by as 50,000 lines of dogshit unmaintainable code that constantly throws errors is considered better than 100 lines of good, functional code with no problems and 100% test coverage. The problem is that the business idiots have no idea what it is that engineers actually do and are looking for easy ways to quantize it anyway. Lines of code, commits, changes, and pull requests are easy numbers to measure so they're always trying to use those even though as metrics for productivity they're worthless.

2

u/FlipFlapFlooperooni 5h ago

Precisely.

After a year of measuring AI impact at my org, you'd think the VPs above me would get that when I say "token usage is way up (obviously), LoC is way up, PRs (and approvals) are way up, and usage of the internal MCP server is steadily increasing" that they'd all be able to see how NONE of those things correlate to higher feature output, better margins or more business value delivered/generated (that's what my dashboard clearly indicates).

Instead it's the same dumb monthly conversation of these guys blankly staring at me asking: "if more things, then why not more money?"

It's even more fun to try to express to them that it's their obsession with ass-backwards planning processes and endless hand-wringing about what "the right thing" is with zero tolerance for failure that actually prevents this newly found "AI efficiency" (as they call it) from netting meaningful business results.

1

u/GargantuanCake 4h ago

Well you see it's newer so it must be better and Dario said it would make our developers 10x more efficient so that means 10x more profits right?

We gave everybody unlimited Claude tokens and profits did not 10x explain immediately.

10

u/CapableHerring 7h ago

When companies start tracking arbitrary and meaningless metrics like PR count, or lines of code, the engineers start optimizing their workflow to that meaningless metric.

This isn't unique to the AI-era. Companies have used PR counts as a performance metric in the past, and the exact thing you're experiencing happens. Another one is using PR comments as a metric, which just means every PR starts getting a bunch of tiny nitpicks. When companies use lines of code, suddenly everyone starts writing extremely verbose code, making new multi-hundred line files to achieve what would've been done in 10 lines.

Unfortunately, this isn't a battle you're going to win. Your leadership has decided this is a performance metric they want to track. It's a sign of a deteriorating culture. Might be time to stat dusting off that resume.

My previous company went through something similar. It was after we just hired a new CTO, so we could already tell they were tightening the belt. They bought a new product that tracked PR's, PR comments, confluence posts, Jira progress/comments, etc. Leadership was adamant that it wouldn't be used as an individual performance tracking tool, they were just going to use it at the aggregate level to understand what teams were spending time on.

I'll give you 1 guess how that turned out.

Culture went to shit, very, very quickly. I saw the writing on the wall and GTFO as soon as I had another offer lined up. Not long after I left the company did a massive 25% layoff.

4

u/FearlessPark4588 7h ago

They just want proof that AI is doing something for them and they don't know how to show it besides easily quantifiable dashboards.

6

u/CapableHerring 7h ago

It's a problem as old as time. Leadership has always struggled with this, there aren't easily quantifiable metrics to show how well a SWE is doing. Even shit like story points don't mean anything, it's an awful metric to say if one SWE is better than another.

So they just pick something stupid, and we game the system. Leadership will read whatever message they want out of it. Before AI, they'd use those metrics to demonstrate how some BS company culture overhaul resulted in massive productivity boons.

1

u/FearlessPark4588 6h ago

They have to prove to someone (eg: the board) that it's doing something. Everyone has an owner.

5

u/ViralSubCatch 6h ago

https://en.wikipedia.org/wiki/Perverse_incentive

1

u/Perfect-Job6155 6h ago

Wikipedia link really says it all, classic case of measuring the wrong thing and getting exactly that

2

u/fsk 2h ago

A better explanation is Goodhart's Law.

Example: Counting number of pull requests might be a way to measure employee productivity. Once employees know that's the metric you're using to evaluate them, they're going to flood the system with pull requests.

Example: If you're the first person ever to do leetcode-style interviews, you might be able to find good workers other people are overlooking. Once the entire industry is using leetcode interviews and every candidate is practicing, it ceases to be effective.

2

u/Jcampuzano2 5h ago

PR count is basically a productivity vanity metric. If people are rewarded for the number, they’ll optimize the number instead of the actual work.

3

u/AssignmentMammoth696 5h ago

Companies are going to have to make a decision. Either you 100% trust the AI output, or the bottleneck will always be the review/QA process. There is no inbetween.

2

u/jsdodgers 5h ago

we've been splitting up our PRs for over a decade to make them as small as possible, and it's infinitely easier to review

0

u/the_pwnererXx 6h ago

I'm making 200 PRs a month, avg 500 loc