2.8 KiB
2.8 KiB
Work
A few learnings from launch morning:
- At our highest traffic points, Web server and DB server performance were never even close to hitting any limits whether CPU, memory, disk space, etc. I feel somewhat justified in my scale-up decisions before Bradley steam-rolled them with his own. But I don't regret going higher than needed versus an alternative of not going high enough and having to scale-up in the middle of launch day.
- Caching was a big win, at least in the short term--perhaps we fall off the horse on the "no pre-emptive optimization" horse and should cache rarely-changing data much more often?
- Our code felt sloppily crafted. What I mean is:
- Routes like the Orders API bulk preview route were grossly inefficient. While it was (apparently) created to allow doing multiple previews at once, it appears that zero thought was put into actually taking advantage of doing this in bulk and/or parallel
- Caching was not carefully implemented (ie, not implemented at all in obvious places)
- We have a ridiculous web of interconnected microservices. Discounts, orders, products, subscriptions (and more)--none of them are stand-alone services. They all depend on each others, it seems. I believe there has not be careful planning around the boundaries of these APIs and, unless replatforming supplants them completely, we need some major overhaul or at least strongly-worded guided on incremental improvements as we maintain these services. See this thread as an example where a bug arose because Orders API was not calculating tax in preview routes the same way Subscriptions API was--absolutely inexcusable (to be really dramatic) that two services have to calculate tax at all (instead of a single one doing so for use cases) or at the very least don't use a shared codebase for doing so.
- I see a couple of potential causes for this sloppiness:
- Some of our most experienced senior engineers (who also had the most senior insight into what Commerce and Payments was doing) leaving shortly before launch (Nate Merritt and Timothy Baumgartner). This left us not only short-handed, but also not knowing what we didn't know in some cases. I can't point to specific problems that I could draw a direct from to them leaving, but it feels like some problems may have simply been caught and never appeared as an issue long before launch.
- Time: teams were working up to the last minute on things. We had little time to optimize, refactor and improve code.
- Legacy design decisions: my impression (not quite a fully educated one) is that we regret a lot of design choices around Subscriptions API (and subscribable products specifically). This has a lot to do with the "web of interconnected services" I mentioned above.
- A better pla