r/programming • u/Ok_Stomach6651 • 8h ago

Optimizing Our E2E Pipeline

https://slack.engineering/speedup-e2e-testing/
30 Upvotes

65% Upvoted

16 comments sorted by

67

u/ebol4anthr4x 8h ago

This was one of the least useful articles I've ever read. It borders on blogspam. I don't know whether to feel good or bad that such a large and valuable company was making such basic mistakes in their CI pipelines for so long. tl;dr: They were rebuilding and persisting the artifacts to S3 on every single push, even if nothing changed, so they switched to using changeset conditionals in their CI configuration.

23

u/SoilMassive6850 7h ago

Things like this is why I'm becoming more and more convinced giving devs too many resources is a bad thing.

Where I work a new (usually internal) service is given 1 vCPU, 4GB of ram and 100GB of storage by default, and before allocating more there are basic sanity checks to justify it. It's not that we're even struggling for compute resources, but 90% of the time this filter just helps catch dumb shit before it becomes tech debt because those early issues would likely be even worse if scaled up. Of course we allocate more when theres cause but most often it's unnecessary.

3

u/ebol4anthr4x 6h ago

Things were better when we spoke of RAM in terms of MB.

3

u/bzbub2 7h ago

the three dot notation example is an interesting tidbit. otherwise, it is not super forthcoming and the conclusions are sort of odd (unclear relation to test flakiness..??)

6

u/ChemicalRascal 4h ago

This is not blogspam. Blogspam has a very specific definition, a blog post that does fuck all else beyond linking to another source. As an example, if I had a blog article that said "Slack just dropped info on how they decreased build times! <url>", that's blogspam.

7

u/ChemicalRascal 4h ago

And yeah, this article isn't amazing, but it's fine. Not every article needs to be a deeply informative technical masterpiece. This isn't so bad that it needs to be removed.

1

u/AD7GD 2h ago

It's a shame, because the same basic problem becomes a bottleneck in any automatic merge strategy, and a survey of methods would be valuable to people hitting that problem.

My contribution: bazel isn't that bad if you have AI to set it up and debug it for you ;-)

53

u/chucker23n 7h ago

This work was not possible without contributions from:

Anirudh Janga, Josh Cartmell, Arminé Iradian, Anupama Jasthi, Matt Jennings, Zack Weeden, John Long, Issac Gerges, Andrew MacDonald, Vani Anantha and Dave Harrington

They needed 11 developers to figure out that when a PR doesn’t change the front-end, it doesn’t need to be rebuilt, nor uploaded to Amazon?

16

u/JoustyMe 7h ago

You know those jokes "how many X are needed to screw in a light bulb?"
So now we have "how many developers are needed to get managment to care about Tech Debt?"

-14

u/Ok_Stomach6651 5h ago

Thanks for your input

9

u/bodiam 3h ago

What a strange blogpost. Github provides this feature out of the box:

on: 
   push:      
      branches :      
      - main      
      paths :      
      -  'src/frontend/**'

Not really clear to me why they couldn't use this approach.

3

u/qmunke 40m ago

They wanted the build to run, just not for the whole repo. This is why everyone who says to just use monorepos needs to understand there actually needs to be someone managing the things - which is reasonable at huge shops but hard to justify for those of us without a dedicated devx team

2

u/bodiam 35m ago

Yes, but that's exactly what the above trigger does; only run when the frontend has changed. And I'm a big fan of mono repos, it prevents a whole class of problems from ever occurring, it just looks like their solution is a bit custom.

5

u/Flashynuff 6h ago

Geez a whole article about optimizing a pipeline and not a single trace/query screenshot. Let’s see the 📉!

2

u/ComoddifiedCraic 5h ago

Not frontend focused but runccc.dev solves this with a static analyser and runtime instrumentation. Ran against a massive enterprise test suite which took hours to run now takes less than a minute for any small changes