From 9fc75cb187a7cade251314d3f1273bdb96089275 Mon Sep 17 00:00:00 2001 From: benjaminramey Date: Tue, 22 Oct 2024 10:14:14 -0500 Subject: [PATCH] vault backup: 2024-10-22 10:14:13 --- Daily Notes/2024/10/2024-10-22 (Tuesday).md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/Daily Notes/2024/10/2024-10-22 (Tuesday).md b/Daily Notes/2024/10/2024-10-22 (Tuesday).md index f4fcb56..ea367e2 100644 --- a/Daily Notes/2024/10/2024-10-22 (Tuesday).md +++ b/Daily Notes/2024/10/2024-10-22 (Tuesday).md @@ -3,10 +3,12 @@ A few learnings from launch morning: 1. At our highest traffic points, Web server and DB server performance were _never even close_ to hitting any limits whether CPU, memory, disk space, etc 2. Caching was a big win, at least in the short term 3. Our code felt sloppily crafted. What I mean is: - 1. Routes like the Orders API bulk preview route were _grossly_ inefficient. While it was (apparently) created to allow doing multiple previews at once, it appear that *zero* thought was put into actually taking advantage of doing this in bulk and/or parallel - 2. Caching was not carefully implement (ie, not implement at all in obvious places) - 3. We have a ridiculous web of interconnected microservices. Discounts, orders, products, subscriptions--none of them are stand-alone services. They all depend on each others, it seems. I believe there has not be careful planning around the boundaries of these APIs and, unless replatforming supplants them completely, we need some major overhaul or at least strongly-worded guided on incremental improvements as we maintain these services -4. I see a couple of potential causes for this "carelessness": - 1. Our most experienced senior engineer (who also had the most senior insight into what Commerce was doing) leaving two months before launch (Nate Merritt) - 2. + 1. Routes like the Orders API bulk preview route were _grossly_ inefficient. While it was (apparently) created to allow doing multiple previews at once, it appears that *zero* thought was put into actually taking advantage of doing this in bulk and/or parallel + 2. Caching was not carefully implemented (ie, not implemented at all in obvious places) + 3. We have a ridiculous web of interconnected microservices. Discounts, orders, products, subscriptions (and more)--none of them are stand-alone services. They all depend on each others, it seems. I believe there has not be careful planning around the boundaries of these APIs and, unless replatforming supplants them completely, we need some major overhaul or at least strongly-worded guided on incremental improvements as we maintain these services. See [this thread](https://faithlife.slack.com/archives/C03R24VCERG/p1729552403256199) as an example where a bug arose because Orders API was not calculating tax in preview routes the same way Subscriptions API was--absolutely inexcusable (to be really dramatic) that two services have to calculate tax _at all_ (instead of a single one doing so for use cases) or _at the very least_ don't use a shared codebase for doing so. +4. I see a couple of potential causes for this sloppiness: + 1. Some of our most experienced senior engineers (who also had the most senior insight into what Commerce and Payments was doing) leaving shortly before launch (Nate Merritt and Timothy Baumgartner). This left us not only short-handed, but also not knowing what we didn't know in some cases. I can't point to specific problems that I could draw a direct from to them leaving, but it feels like some problems may have simply been caught and never appeared as an issue long before launch. + 2. Time: teams were working up to the last minute on things. We had little time to optimize, refactor and improve code. + 3. Legacy design decisions: my impression (not quite a fully educated one) is that we regret a lot of design choices around Subscriptions API (and subscribable products specifically). This has a lot to do with the "web of interconnected services" I mentioned above. +5. # Meetings