Samir Desai’s Post

On September 7th 2023, Square / CashApp had a major outage for 12+ hours. By September 11th, they were able to fully recover and did a pretty detailed post-mortem on everything that went wrong. For Cash App, there were already calls from merchants for compensation for lost sales, so even if it was resolved quickly, the aftereffects may be lasting. It was hopeful that Square had the ability to enable STIP (stand-in processing) where transactions are processed as-is & reconciled with the network later, but there’s inherent risk with STIP as transactions can’t be pre-evaluated for riskiness or for the user’s ability to pay for a transaction. As news of the outage started hitting the public, the stock tanked, sending it to its lowest level in the past 12 months. Meanwhile while many neobanks were secretly thrilled about new user influx, Chimers knew that rooting for another’s demise was only a recipe for future failure. In fact, OG Chimers were uncomfortably familiar with this situation. Back in 2019, a database issue with our processor led to a similar kind of outage. It was especially tough because we were so integrated with our processor everything broke, including customer support & the app, not just payments. Major props to our engineers & our CTO who were up late every night in a "war room" trying to figure out the issue & the fix because we knew how important Chime was to members. A team of Chimers even booked an emergency flight to Utah to resolve datacenter issue directly. It was thanks to Engineering that we were able to resolve issues as quickly as we did (and thanks to everyone else who chipped in to support member experience in trying to allay our members’ concerns). Unfortunately, as systems came back up, transactions came back *before* balances so it temporarily seemed like balances were negative. Combined with a lack of customer support, some folks thought we rug pulled. While we moved quickly, if it's your bank account, any outage is unacceptable. Folks won't leave right then, but they build a backup so if it occurs again, they switch. It will likely be the same was Square so even if retention holds, it's not the full story. It's the next outage that matters. So it was important to not just get repayment right but also make sure it never happened again. StratFin actually worked across the organization to estimate losses & fairly compensate. And the whole team overhauled every system to lower dependencies & increase redundancy. It's a constant focus now at Chime. When Chime had its outage, opportunistic competitors were quick to jump in with "Join us if you want reliability!" The only silver lining was schadenfreude when they too had outages. Not CashApp though. They knew it could happen to them and/or were above petty competition. Either way, they did the right thing. That's a wrap! If you liked this post, please repost for reach and follow me or check out interspace.samir.xyz for more fintech, tech and stratfin nerdery.