In the past 6 months, I’ve started working to support a software system with complexity much like the kind described in this article. When I started and saw it in all its glory, my naive engineer brain thought no way this can work. The system was very large, patchwork through outsourcing/turnover, and poorly documented both at the code and UX level. There were many critical bugs in the code and new features were constantly being introduced. Yet as the months went by, it continued functioning at scale as it had done for so many years. Many things were wrong and did break constantly but a combination of operators being careful/resourceful, support staff making in DB changes/hotfixes to recover from fail states 24/7, and just so much human labor in the system allowed it to work and scale across many clients. So many things could fail in a day, and even though the system was so complex, people made it work and it was very educational to me.

Now, seeing that up close as a loner engineer gave me nightmares and inspired me in developing the technology for my startup. I’m just one guy so I limit complexity, write as little code as possible, and protect myself like a paranoid person. Basically, I see the primary value I can deliver from an engineering perspective is solving my domain as much as possible while limiting the scope as much as possible. Then when scaling, the costs/risks presented in this article are delayed as much as possible. It’s not always possible because some domains touch the real world too much but I see competitors in my field not heed this, and they’re starting to topple over in my opinion.

Definitely some good reminders in there!

> But the converse: that successful outcomes are also the result of gambles; is not widely appreciated.

This statement however I would challenge: There is the whole ‘build fast and break shit’ culture that bets exactly on that outcome.

Aww, missed opportunity to have fun with virtual hosts as https://why.complexsystems.fail/ is currently nxdomain

I would have put up something that returned an <ol> akin to:

1. Cert expiry

2. DNS (or, is the current state of affairs a meta joke?!)

3. Firefall Rules