🔗 Eliminating SPOFs: the importance of documentation

After a highly intense year at work, Sarah has gone on a much-needed two-week holiday. Although Louise, her CTO, had assured her she should take the opportunity to recharge, Sarah had told her team she would be reachable by phone in an emergency.

Sarah lands in Vietnam on Monday evening; she is jetlagged. She goes straight to her hotel, and when she goes to bed, she realises she must have left her phone in the taxi. She knows she probably can't get it back, so she decides to sleep and will deal with it tomorrow.

Sarah works for CalUp, an early-stage startup in the Netherlands. They plan to gear up for a seed round later in the year, and stability will be important for their investors.

In Amsterdam, Louise is making a coffee in the office when her phone buzzes. She sees several emails coming through to the company's support email. At the same time, she notices monitoring alerts coming through on Slack. She panics. The payment integration solution used within the application is failing.

She speaks with the other developers who realise an environment variable needs updating after the payment provider released a new API version. Nobody knows which of the 25 variables are safe to modify or what the new value should be. They have got this incorrect in the past, resulting in data loss.

After a tense discussion with the team, they decided to place an alert notifying users that transactions may fail. Louise emails their customers to apologise for a temporary issue with payment processing. Eight hours later, after many complaints from customers, Louise receives an email from Sarah explaining about her lost phone. She is then able to walk the team through the update required.

This is the world of SPOFs (Single points of failure), where crucial information for your company lives in the head of one individual.