Crunch — it’s more than just games
Every few months the topic of crunch does the rounds. We get a few shallow pontificating articles from non-developers on whether video games need crunch before the whole topic is lost again to the content waterfall.
The game dev perspective has been covered extensively by Jim Sterling, thank god for him, and i’m not keen to retread that ground. Watch here and here if you want a quick primer.
What I want to talk about is how crunch is deeply ingrained in all of the tech industry, not just games. I've experience it in nearly every development project I've ever worked and i’m a web developer. I’m going to talk through how and why its so prevalent and proselytise a little on the DevOps movement that’s trying to kill crunch for good.
What is crunch? Crunch is the practice of coercing workers into gruelling often extreme overtime so that a company can meet a deadline for a product.
Crunch is rarely, if ever, paid as overtime or at all. Developers are often asked to donate their time to make up for the project running late. It can run on for months, even years at a time. Companies often employ shameless tactics like applying social pressure and emotional manipulation to force people into a corner where they can’t say no.
It’s called crunch because that’s how it feels, like being squeezed in a pressure cooker of anxiety.
Crunch is incredibly stressful, people have breakdowns, it accelerates burnout and often permanently damages the relationships between development staff and upper management.
In my opinion crunch is a response to the iron triangle of project management. Every project aims to deliver a good product so quality is non-optional and gets placed at the centre of the triangle.
The three points around the outside are optional and as long as you prioritise two of them you can attain a quality product.
Quality gets sacrificed if you try to prioritise all three.
To put it another way, you can have a project run on time and on budget if you sacrifice scope. Or a project can be in scope and on budget but will run over for time. Or it can be on time and in scope but will cost much more than anticipated.
Immediately you should see the problem here, few if any companies are happy to delay an announced launch date. Projects are sold with fixed budgets and once expectations are set for scope they are very difficult to change.
The problems don’t stop there either. In The Mythical Man-Month Fred Brooks coined Brooks’s Law which roughly states that adding more developers onto a project that is already running late will only make it run even later.
It’s complex but potentially the most important reason for this is that each new developer takes time to learn how to contribute to a project, usually measured in months. While learning how to contribute they will reduce the overall productivity of the team due to mentoring and skill sharing sessions.
For some tasks you cannot apply a divide and conquer strategy, these are non-divisible tasks. The excerpt of the book below illustrates a perfect example of a non-divisible task.
“The bearing of a child takes nine months, no matter how many [people] are assigned.” — Fred Brooks.
It is crucial to the success of a project that all non-divisible tasks be identified early and given proper allotment of resources.
Unfortunately this seldom happens and represents rework. It’s often the case that project manager has included the same non-divisible task as a subtask of two separate tickets, or worse a hero attempts to work faster than another engineer in order to “get the glory” of fixing a high profile issue.
Other significant overheads that are their own entire subject include; team communication, meeting ceremonies and coordination of effort.
How does crunch happen? Imagine you’re a development manager and you know a deadline is coming.
You know you are going to be late but you still have features left to finish. You’re aware of Brooks’s law so you know you can’t add more developers because it’s too late in the game. You have a fixed budget that is already accounted for and you can’t go to upper management to delay as it will reflect poorly on you and your team which would result in a reduced budget for future projects. You can’t even drop scope because you’ve already committed to specific features for the product.
You’re screwed. You can’t beat the triangle and you know it.
You just need a little more time, but time has to come from somewhere so you walk into the main project space at lunch on Friday and ask the team if they can work weekends until the project is done. They all groan and you mention getting pizza.
Two weeks later you ask them to stay until 10pm everyday.
Months pass of this, you’re all exhausted, the team is straining, you’ve eaten nothing but takeout pizza in so long you’ve started ordering broccoli toppings in an effort to see a vegetable. Somehow you still need more time. You ask everyone to start coming in at 5am.
You’re screwed. You can’t beat the triangle and you know it.
After months of crunch the project squeaks passed it’s deadline and in the next engineering bulletin you get one faint line of praise. Two days after launch you have a major outage.
You sacrificed quality without even realising it.
Why do we put up with it? In the UK it’s often written into our contracts.
When I worked at the now defunct Silkwood Valley we were given contracts 3 months into the founding of the company which stated that we would sometimes be required to put in additional hours, which we would not be paid for, which we would not receive holiday in lieu for and which were non-optional. I did not sign this contract and threatened to walk if they didn’t remove the stipulation, in the end they buckled but it was a short lived success. The company went bust less than 6 months later.
Other companies I’ve worked for, that I’m legally not allowed to name, either due to my employment contract or subsequent non-disclosure agreements, have had the same stipulations and didn’t budge at all when requested.
It isn’t just contracts though, there is a social aspect to this as well. When a project runs late there is a lot of shame directed at developers from management and a lot of internalised shame at promising something we haven’t delivered.
I’ve been on multiple teams where my team-lead has been publicly chewed out for a project running late, projects that on more than one occasion had been dropped on the team at last minute. On one occasion, the project had been dropped on us that very morning.
Hero culture is hard to disassemble once it’s become a part of company culture and most don’t want too. I’ve worked with project managers who have actively encouraged heroics as a side channel to reach otherwise unattainable deadlines.
If you don’t know what hero culture is unfortunately it’s not comic book related.
Hero culture or Heroics are when a company expects its staff to go above and beyond their contracted hours to fix problems or investigate outages. It’s work outside of business time without remuneration. It often involves a full weekend debugging production issue, late nights fixing catastrophic merges, cluster meltdowns, etc.
Sounds kinda like crunch doesn’t it?
That’s because it is crunch. Instead of an entire team it’s one or two developers working crunch style overtime for a specific issue. The trick here is middle management will praise the heroes publicly which becomes their mechanism to recruit more heroes. More developers looking to prove themselves. Sometimes there’s even a non-monetary reward like a pizza party or a bottle of champagne.
A good company will run an incident postmortem and learn how to prevent the issue in the future by adjusting processes. A bad company will accept this as a cost of doing business or find someone to blame and fire them.
You can see why this kind of crunch happens though. Even with a blameless culture if my code causes an outage in production I feel responsible. It’s also notoriously difficult to build stable always online services. Perpetual motion doesn’t exist, code will throw, fans burn out, clusters go down and ISP’s have cut off buildings accidentally. Something is always going to happen and a lot of companies prioritise customer experience over an engineers sleep.
What can we do? Culture is the biggest tool to fight crunch. You need to make it socially unacceptable to ask for unpaid overtime. You need to enforce repercussions for managers who seek to weaponize our sense of responsibility. You need to disassemble the idea that it is possible to beat the triangle and have either flexible deadlines or flexible feature delivery.
The most effective way to achieve a culture that inherently rejects crunch, is to embrace the DevOps mindset.
The Phoenix Project brought us an easily digestible idea of positive change through continual improvement of daily work and tighter, faster feedback loops to continually realign ourselves to what benefits the customer most.
The DevOps handbook codified steps to achieve the transformation talked about in The Phoenix Project, the focus was heavily on bringing the business into ops and getting ops and dev to FINALLY talk to each as different forms of the same function.
The Unicorn Project refined it all into 5 ideals that any business can take to boost engineering effectiveness and throw crunch into the trash where all ineffective tools belong.
The Five Ideals are:
- Locality and Simplicity
- Focus, Flow, and Joy
- Improvement of Daily Work
- Psychological Safety
- Customer Focus
Read all three books, ideally starting with The Phoenix Project moving to The DevOps Handbook then read The Unicorn Project. They have life changing advice for grassroots as well as the rarer top down movements.
Let me leave you with this.
Any managerial tool that sacrifices the ideals listed above is destructive, increases engineering dissatisfaction and reduces quality. A tool that breaks multiple ideals at once is guaranteed to result in code and infrastructure that is difficult at best to maintain.
Crunch violates at least four.
