benjaminramey/ObsidianJournal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

vault backup: 2024-11-08 09:47:17

Browse Source
This commit is contained in:
Benjamin Ramey
2024-11-08 09:47:17 -06:00
parent 78958a5afb
commit d7dad35e49
77 changed files with 4605 additions and 1768 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.obsidian/community-plugins.json vendored
View File

@@ -3,5 +3,6 @@
"obsidian-importer", "obsidian-importer",
"numerals", "numerals",
"nldates-obsidian", "nldates-obsidian",
"calendar" "calendar",
"obsidian-read-it-later"
] ]

48
.obsidian/plugins/obsidian-read-it-later/data.json vendored Normal file
View File

@@ -0,0 +1,48 @@
{
"inboxDir": "Reading/Read Later",
"assetsDir": "Reading/Read Later/assets",
"openNewNote": false,
"openNewNoteInNewTab": false,
"youtubeNoteTitle": "Youtube - %title%",
"youtubeNote": "[[ReadItLater]] [[Youtube]]\n- [ ] Watched\n\n# [%videoTitle%](%videoURL%)\n\n%videoPlayer%",
"youtubeEmbedWidth": "560",
"youtubeEmbedHeight": "315",
"youtubeUsePrivacyEnhancedEmbed": true,
"vimeoNoteTitle": "Vimeo - %title%",
"vimeoNote": "[[ReadItLater]] [[Vimeo]]\n- [ ] Watched\n\n# [%videoTitle%](%videoURL%)\n\n%videoPlayer%",
"vimeoEmbedWidth": "560",
"vimeoEmbedHeight": "315",
"bilibiliNoteTitle": "Bilibili - %title%",
"bilibiliNote": "[[ReadItLater]] [[Bilibili]]\n\n# [%videoTitle%](%videoURL%)\n\n%videoPlayer%",
"bilibiliEmbedWidth": "560",
"bilibiliEmbedHeight": "315",
"twitterNoteTitle": "Tweet from %tweetAuthorName% (%date%)",
"twitterNote": "[[ReadItLater]] [[Tweet]]\n- [ ] Read\n\n# [%tweetAuthorName%](%tweetURL%)\n\n%tweetContent%",
"parseableArticleNoteTitle": "%title%",
"parsableArticleNote": "[[ReadItLater]] [[Article]]\n- [ ] Read\n\n# [%articleTitle%](%articleURL%)\n\n%articleContent%",
"notParseableArticleNoteTitle": "Article %date%",
"notParsableArticleNote": "[[ReadItLater]] [[Article]]\n- [ ] Read\n\n[%articleURL%](%articleURL%)",
"textSnippetNoteTitle": "Note %date%",
"textSnippetNote": "[[ReadItLater]] [[Textsnippet]]\n- [ ] Read\n\n%content%",
"mastodonNoteTitle": "Toot from %tootAuthorName% (%date%)",
"mastodonNote": "[[ReadItLater]] [[Toot]]\n- [ ] Read\n\n# [%tootAuthorName%](%tootURL%)\n\n> %tootContent%",
"downloadImages": true,
"downloadImagesInArticleDir": false,
"dateTitleFmt": "YYYY-MM-DD HH-mm-ss",
"dateContentFmt": "YYYY-MM-DD",
"downloadMastodonMediaAttachments": true,
"downloadMastodonMediaAttachmentsInDir": false,
"saveMastodonReplies": false,
"mastodonReply": "[%tootAuthorName%](%tootURL%)\n\n> %tootContent%",
"stackExchangeNoteTitle": "%title%",
"stackExchangeNote": "[[ReadItLater]] [[StackExchange]]\n- [ ] Read\n\n# [%questionTitle%](%questionURL%)\n\nAuthor: [%authorName%](%authorProfileURL%)\n\n%questionContent%\n\n***\n\n%topAnswer%\n\n%answers%",
"stackExchangeAnswer": "Answered by: [%authorName%](%authorProfileURL%)\n\n%answerContent%",
"downloadStackExchangeAssets": true,
"downloadStackExchangeAssetsInDir": false,
"youtubeApiKey": "",
"tikTokNoteTitle": "TikTok from %authorName% (%date%)",
"tikTokNote": "[[ReadItLater]] [[TikTok]]\n- [ ] Watched\n\n%videoDescription%\n\n[%videoURL%](%videoURL%)\n\n%videoPlayer%",
"tikTokEmbedWidth": "325",
"tikTokEmbedHeight": "760",
"extendShareMenu": true
}

167
.obsidian/plugins/obsidian-read-it-later/main.js vendored Normal file
View File

File diff suppressed because one or more lines are too long

10
.obsidian/plugins/obsidian-read-it-later/manifest.json vendored Normal file
View File

@@ -0,0 +1,10 @@
{
"id": "obsidian-read-it-later",
"name": "ReadItLater",
"version": "0.6.0",
"minAppVersion": "1.6.2",
"description": "Saves the clipboard to a new note.",
"author": "Dominik Pieper",
"authorUrl": "https://github.com/DominikPieper",
"isDesktopOnly": false
}

14
Daily Notes/2024/11/2024-11-07 (Thursday).md Normal file
View File

@@ -0,0 +1,14 @@
# To-dos
- [ ] Finish Skeletor asynification
- [ ] Review all uses of Url and HttpContext to make sure they are not used after we do ConfigureAwait(false) (like in private controller methods)
Since _all_ controller methods (private and public, or otherwise) have access to the controller's base class HttpContext and Url properties, I'm going to keep ConfigureAwait(true) for any controller methods but make sure they pass HttpContext down to any methods outside the controller
- [ ] Review all HttpContext.Current usage to replace with passed-in HttpContextBase parameters
- [x] Follow-up on on-call action-item to handle liquid errors differently
- [ ] Search for specific causes of bad site performance on 11/1 FBOTM day https://faithlife.slack.com/archives/C03R1MQSR51/p1730821466045049?thread_ts=1730477236.809519&cid=C03R1MQSR51
- [x] Consolidate broken circuit handling in Skeletor
- [x] Give feedback to Ricardo on new.logos.com rollout ideas: [Replatforming testing ideas - Google Docs](https://docs.google.com/document/d/1r3W1Ot1T89gxgmPGNKuFPg2lDZEJiUHlxdfbcwLMZYo/edit?tab=t.0)
# Summary
Kyle 1:1 - Analytics needs our help understanding and organizing our data
I ended up not liking the idea of a separate (even if just business hours) on-call rotation for martech to feed liquid errors to. I think this is simply the on-call dev's job to address or pass on these errors. The real fix is to implement the on-save liquid validation step that BEN has a case for.

43
Reading/Done Reading/Scaling up the Prime Video audiovideo monitoring service and reducing costs by 90% - Prime Video Tech.md
View File

@@ -1,43 +0,0 @@
Clipped from: [https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90](https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90)
## The move from a distributed microservices architecture to a monolith application helped achieve higher scale, resilience, and reduce costs.
At Prime Video, we offer thousands of live streams to our customers. To ensure that customers seamlessly receive content, Prime Video set up a tool to monitor every stream viewed by customers. This tool allows us to automatically identify perceptual quality issues (for example, block corruption or audio/video sync problems) and trigger a process to fix them.
Our Video Quality Analysis (VQA) team at Prime Video already owned a tool for audio/video quality inspection, but we never intended nor designed it to run at high scale (our target was to monitor thousands of concurrent streams and grow that number over time). While onboarding more streams to the service, we noticed that running the infrastructure at a high scale was very expensive. We also noticed scaling bottlenecks that prevented us from monitoring thousands of streams. So, we took a step back and revisited the architecture of the existing service, focusing on the cost and scaling bottlenecks.
The initial version of our service consisted of distributed components that were orchestrated by [AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html). The two most expensive operations in terms of cost were the orchestration workflow and when data passed between distributed components. To address this, we moved all components into a single process to keep the data transfer within the process memory, which also simplified the orchestration logic. Because we compiled all the operations into a single process, we could rely on scalable [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) and [Amazon Elastic Container Service (Amazon ECS)](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) instances for the deployment.
### **Distributed systems overhead**
Our service consists of three major components. The media converter converts input audio/video streams to frames or decrypted audio buffers that are sent to detectors. Defect detectors execute algorithms that analyze frames and audio buffers in real-time looking for defects (such as video freeze, block corruption, or audio/video synchronization problems) and send real-time notifications whenever a defect is found. For more information about this topic, see our [How Prime Video uses machine learning to ensure video quality](https://www.primevideotech.com/computer-vision/how-prime-video-uses-machine-learning-to-ensure-video-quality) article. The third component provides orchestration that controls the flow in the service.
We designed our initial solution as a distributed system using serverless components (for example, AWS Step Functions or [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html)), which was a good choice for building the service quickly. In theory, this would allow us to scale each service component independently. However, the way we used some components caused us to hit a hard scaling limit at around 5% of the expected load. Also, the overall cost of all the building blocks was too high to accept the solution at a large scale.
The following diagram shows the serverless architecture of our service.
![The diagram shows a control plane and data plan in the initial architecture. The customer's request is handled by a lambda function that is then forwarded to relevant step functions that execute detectors. At the same time, Media Conversion service starts processing the input stream, providing artifacts to detectors through an S3 bucket. Once the analysis is completed, the aggregated result is being stored in an S3 bucket.](Exported%20image%2020240808113914-0.png)
**The initial architecture of our defect detection system.**
The main scaling bottleneck in the architecture was the orchestration management that was implemented using AWS Step Functions. Our service performed multiple state transitions for every second of the stream, so we quickly reached account limits. Besides that, AWS Step Functions charges users per state transition.
The second cost problem we discovered was about the way we were passing video frames (images) around different components. To reduce computationally expensive video conversion jobs, we built a microservice that splits videos into frames and temporarily uploads images to an [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) bucket. Defect detectors (where each of them also runs as a separate microservice) then download images and processed it concurrently using AWS Lambda. However, the high number of Tier-1 calls to the S3 bucket was expensive.
### **From distributed microservices to a monolith application**
To address the bottlenecks, we initially considered fixing problems separately to reduce cost and increase scaling capabilities. We experimented and took a bold decision: we decided to rearchitect our infrastructure.
We realized that distributed approach wasn’t bringing a lot of benefits in our specific use case, so we packed all of the components into a single process. This eliminated the need for the S3 bucket as the intermediate storage for video frames because our data transfer now happened in the memory. We also implemented orchestration that controls components within a single instance.
The following diagram shows the architecture of the system after migrating to the monolith.
![The diagram represents a control and data plan for the updated architecture. All the components run within a single ECS task, therefore the control doesn't go through the network. Data sharing is done through instance memory and only the final results are uploaded to an S3 bucket.](Exported%20image%2020240808113914-1.png)
**The updated architecture for monitoring a system with all components running inside a single Amazon ECS task.**
Conceptually, the high-level architecture remained the same. We still have exactly the same components as we had in the initial design (media conversion, detectors, or orchestration). This allowed us to reuse a lot of code and quickly migrate to a new architecture.
In the initial design, we could scale several detectors horizontally, as each of them ran as a separate microservice (so adding a new detector required creating a new microservice and plug it in to the orchestration). However, in our new approach the number of detectors only scale vertically because they all run within the same instance. Our team regularly adds more detectors to the service and we already exceeded the capacity of a single instance. To overcome this problem, we cloned the service multiple times, parametrizing each copy with a different subset of detectors. We also implemented a lightweight orchestration layer to distribute customer requests.
The following diagram shows our solution for deploying detectors when the capacity of a single instance is exceeded.
![Customer's request is being forwarded by a lambda function to relevant ECS tasks. The result for each detector is stored in S3 bucket separately.](Exported%20image%2020240808113914-2.png)
**Our approach for deploying more detectors to the service.**
### **Results and takeaways**
Microservices and serverless components are tools that do work at high scale, but whether to use them over monolith has to be made on a case-by-case basis.
Moving our service to a monolith reduced our infrastructure cost by over 90%. It also increased our scaling capabilities. Today, we’re able to handle thousands of streams and we still have capacity to scale the service even further. Moving the solution to Amazon EC2 and Amazon ECS also allowed us to use the [Amazon EC2 compute saving plans](https://aws.amazon.com/savingsplans/compute-pricing/) that will help drive costs down even further.
Some decisions we’ve taken are not obvious but they resulted in significant improvements. For example, we replicated a computationally expensive media conversion process and placed it closer to the detectors. Whereas running media conversion once and caching its outcome might be considered to be a cheaper option, we found this not be a cost-effective approach.
The changes we’ve made allow Prime Video to monitor all streams viewed by our customers and not just the ones with the highest number of viewers. This approach results in even higher quality and an even better customer experience.

3
Reading/Read Later/2020-01-31 Tech Talk - Michael Sterling - Royalties - Faithlife Coders - Amber.md
View File

@@ -1,3 +0,0 @@
| | |
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| ![thumbnail](Exported%20image%2020240808113925-0.png) | \| \|<br>\|---\|<br>\|## 2020-01-31 Tech Talk - Michael Sterling - Royalties - Faithlife Coders - Amber\|<br>\|[https://amber.faithlife.com/shares/921WWa1mhUUOlewF](https://amber.faithlife.com/shares/921WWa1mhUUOlewF)\|<br>\|Royalties Tech Talk Slides: [https://docs.google.com/presentation/d/1UERbx5Op1upek32_TCBuXRPgIKtZkXjmWQM6IyEzCSA/edit#slide=id.g7d0408af6d_2_93](https://docs.google.com/presentation/d/1UERbx5Op1upek32_TCBuXRPgIKtZkXjmWQM6IyEzCSA/edit#slide=id.g7d0408af6d_2_93)...\| |

78
Reading/Done Reading/A Completely Non-Technical Explanation of AI and Deep Learning - Parand.md → Reading/Read Later/A Completely Non-Technical Explanation of AI and Deep Learning.md
View File

@@ -1,94 +1,134 @@
Clipped from: [https://www.parand.com/a-completely-non-technical-explanation-of-ai.html](https://www.parand.com/a-completely-non-technical-explanation-of-ai.html) [[ReadItLater]] [[Article]]
- [x] Read
# Overview # [A Completely Non-Technical Explanation of AI and Deep Learning](https://www.parand.com/a-completely-non-technical-explanation-of-ai.html)
## Overview
This document will explain what neural networks are and how they work, which will help you understand how AI and machine learning work. In the scenario below you'll play the part of the neural network. This document will explain what neural networks are and how they work, which will help you understand how AI and machine learning work. In the scenario below you'll play the part of the neural network.
# Day One ## Day One
First day of your new job as a "classifier" your boss walks in and drops a big spreadsheet of numbers on your desk. First day of your new job as a "classifier" your boss walks in and drops a big spreadsheet of numbers on your desk.
"Is this a cat?" she asks. "Is this a cat?" she asks.
Confused you ask "What?" Confused you ask "What?"
![Exported image](Exported%20image%2020240808113913-0.png) ![](Reading/Read%20Later/assets/neural-cat.png)
"Is this a cat?" "Is this a cat?"
Even more confused, you respond "I don't know". Even more confused, you respond "I don't know".
"Wrong!" she says, and slaps you across the face. "Wrong!" she says, and slaps you across the face.
Before you've had a chance to be shocked she drops another large spreadsheet on your desk. "Is this a cat?" Before you've had a chance to be shocked she drops another large spreadsheet on your desk. "Is this a cat?"
"I don't understand" you respond. "I don't understand" you respond.
"Wrong!" she says, and slaps you again. "Wrong!" she says, and slaps you again.
Another spreadsheet. "Is this a cat?" Another spreadsheet. "Is this a cat?"
Not wanting another slap, you meekly respond "Yes?" Not wanting another slap, you meekly respond "Yes?"
"Correct! Good job!" she says, and gives you a wonderful reward. Almost makes up for the slaps. "Correct! Good job!" she says, and gives you a wonderful reward. Almost makes up for the slaps.
Another spreadsheet. Cat? Another spreadsheet. Cat?
Slightly more confident and wanting another reward you respond "Yes" Slightly more confident and wanting another reward you respond "Yes"
"Wrong!". Another slap. "Wrong!". Another slap.
You are very confused. "I don't understand what's going on. You haven't told me the rules, you haven't told me how to figure out something is a cat, you haven't given me the logic to figure this out. You haven't trained me." You are very confused. "I don't understand what's going on. You haven't told me the rules, you haven't told me how to figure out something is a cat, you haven't given me the logic to figure this out. You haven't trained me."
"Correct" says the boss. "This is training. Trust me, you're going to get very good at recognizing cats" "Correct" says the boss. "This is training. Trust me, you're going to get very good at recognizing cats"
![Exported image](Exported%20image%2020240808113913-1.png) ![](Reading/Read%20Later/assets/woman-carrot-stick.png)
Another spreadsheet. This time you focus on the sheet. It's 256 columns and 256 rows, filled with numbers. Is it a cat? You don't know, so you guess. Another spreadsheet. This time you focus on the sheet. It's 256 columns and 256 rows, filled with numbers. Is it a cat? You don't know, so you guess.
Another and another. Right, wrong, wrong, right again, it keeps going. Slaps and rewards. Another and another. Right, wrong, wrong, right again, it keeps going. Slaps and rewards.
You're starting to notice some patterns - if the sheet is almost all zeros then it's not a cat. You look at the center of the sheet - that part should have larger numbers. You're starting to notice some patterns - if the sheet is almost all zeros then it's not a cat. You look at the center of the sheet - that part should have larger numbers.
It's getting slightly better - you're getting more rewards than slaps, guessing correctly more often than not. It's all about the patterns of the numbers. It's getting slightly better - you're getting more rewards than slaps, guessing correctly more often than not. It's all about the patterns of the numbers.
# Day Two ## Day Two
This is the strangest job you've ever had. The slaps are terrible, but the rewards are so great you don't want to quit. How do you get better at this? It's not really possible for one person. If you had more people they could focus on different aspects of the sheet, look for different patterns, and you could use their findings to make better guesses. This is the strangest job you've ever had. The slaps are terrible, but the rewards are so great you don't want to quit. How do you get better at this? It's not really possible for one person. If you had more people they could focus on different aspects of the sheet, look for different patterns, and you could use their findings to make better guesses.
You hire 10 people and bring them to work the next day. When you get a spreadsheet you show it to those 10 people and ask them "Is this a cat?" You hire 10 people and bring them to work the next day. When you get a spreadsheet you show it to those 10 people and ask them "Is this a cat?"
They are as confused as you were. You force them to guess. The first guy looks like an idiot, so you decide to go with the opposite of what he says. The third lady looks really thoughtful so you put a lot of weight on what she says. In your mind you assign a weight to each of their guesses to come up with your final answer. They are as confused as you were. You force them to guess. The first guy looks like an idiot, so you decide to go with the opposite of what he says. The third lady looks really thoughtful so you put a lot of weight on what she says. In your mind you assign a weight to each of their guesses to come up with your final answer.
Each time you get a reward or punishment you share it with the 10 people you've hired: you reward or slap them based on how much weight you put on their input and how much they contributed to you getting the answer right or wrong. You learn the patterns: if the first guy says a strong no and the third lady a strong yes then it's very likely a cat. You learn many more patterns like this. Each time you get a reward or punishment you share it with the 10 people you've hired: you reward or slap them based on how much weight you put on their input and how much they contributed to you getting the answer right or wrong. You learn the patterns: if the first guy says a strong no and the third lady a strong yes then it's very likely a cat. You learn many more patterns like this.
The 10 people are learning to hone their opinions based on the rewards and punishments they get. They can focus on different aspects of the sheet, and their opinions together can tell you a lot about the sheet. The 10 people are learning to hone their opinions based on the rewards and punishments they get. They can focus on different aspects of the sheet, and their opinions together can tell you a lot about the sheet.
After what feels like an epoch and a lot of spreadsheets, eventually you get better. You're getting more rewards. This is working. Your boss says you're very perceptive, and starts calling you Perceptron. After what feels like an epoch and a lot of spreadsheets, eventually you get better. You're getting more rewards. This is working. Your boss says you're very perceptive, and starts calling you Perceptron.
![Exported image](Exported%20image%2020240808113913-2.png) ![](Reading/Read%20Later/assets/man-people.png)
# Day Ten ## Day Ten
What if you get more people involved? You could make it 50 people reporting to you, but that'd be hard to manage. How about we add another layer of people before your 10? What if you get more people involved? You could make it 50 people reporting to you, but that'd be hard to manage. How about we add another layer of people before your 10?
You hire 50 more people, have them give their guesses to the 10 people that report to you. You're now very removed from looking at the spreadsheets - instead you rely on the patterns found by the first layer of the people, who give their opinions to the second layer of the people, who then inform you. Everybody passes the slaps and rewards down the line based on how much each earlier person's guess contributed to their guess. You hire 50 more people, have them give their guesses to the 10 people that report to you. You're now very removed from looking at the spreadsheets - instead you rely on the patterns found by the first layer of the people, who give their opinions to the second layer of the people, who then inform you. Everybody passes the slaps and rewards down the line based on how much each earlier person's guess contributed to their guess.
It takes even longer, but eventually the system starts working. You're much more accurate in identifying when it's a cat. It takes even longer, but eventually the system starts working. You're much more accurate in identifying when it's a cat.
Interestingly you were never taught the rules or logic, and you didn't teach your people the logic or rules. You just propagated the rewards and punishments back through each layer: the more each person's opinion contributes to the answer, the more rewards or punishment you shared with them, and they in turn with the people in the layer behind them. Each person uses the same method to propogate their rewards and punishment back to the people in the layer below them. Interestingly you were never taught the rules or logic, and you didn't teach your people the logic or rules. You just propagated the rewards and punishments back through each layer: the more each person's opinion contributes to the answer, the more rewards or punishment you shared with them, and they in turn with the people in the layer behind them. Each person uses the same method to propogate their rewards and punishment back to the people in the layer below them.
# Neural Networks ## Neural Networks
This is how neural networks work: they see many examples and get rewarded or punished based on whether their guesses are correct. They use multiple layers of workers and eventually learn patterns. Importantly no one is teaching them what patterns they should be looking for or telling them the logic or the rules - the networks eventually figure out the patterns and logic based on very many rounds of example, reward, and punishment. This is called **machine learning**, because the machine is learning the rules by itself. This is how neural networks work: they see many examples and get rewarded or punished based on whether their guesses are correct. They use multiple layers of workers and eventually learn patterns. Importantly no one is teaching them what patterns they should be looking for or telling them the logic or the rules - the networks eventually figure out the patterns and logic based on very many rounds of example, reward, and punishment. This is called **machine learning**, because the machine is learning the rules by itself.
Neural networks work well when you have many examples of something (eg. pictures of cats), but it's hard to write the logic and rules to describe how to recognize that thing. Try it - write down some rules for how to recognize a cat (eg. "has 4 legs"), then look at pictures of cats and see where the rules fail (eg. a picture of a cat's head). In these cases you use machine learning so the machine learns the rules by itself. Neural networks work well when you have many examples of something (eg. pictures of cats), but it's hard to write the logic and rules to describe how to recognize that thing. Try it - write down some rules for how to recognize a cat (eg. "has 4 legs"), then look at pictures of cats and see where the rules fail (eg. a picture of a cat's head). In these cases you use machine learning so the machine learns the rules by itself.
Also note that computers see things as multi-dimensional tables of data. They don't look at a "picture" - they see 3 spreadsheets of numbers representing the RGB values of the picture. Also note that computers see things as multi-dimensional tables of data. They don't look at a "picture" - they see 3 spreadsheets of numbers representing the RGB values of the picture.
# Day Forty ## Day Forty
Identifying cats is a lucrative business and you're pretty good at it. But not good enough. How can we get even better? More people, more layers! Identifying cats is a lucrative business and you're pretty good at it. But not good enough. How can we get even better? More people, more layers!
Unfortunately it takes a long time for people to do their calculations, so you've been stuck with just a few layers of people for a while. Having a lot more people would make the process take too long. Unfortunately it takes a long time for people to do their calculations, so you've been stuck with just a few layers of people for a while. Having a lot more people would make the process take too long.
One day you run into a group of people who call themselves Gaming People United (GPU). These people play a game that has taught them to be very good at looking at spreadsheets and calculating numbers - in fact they can do a lot of calculations in parallel, very quickly. One day you run into a group of people who call themselves Gaming People United (GPU). These people play a game that has taught them to be very good at looking at spreadsheets and calculating numbers - in fact they can do a lot of calculations in parallel, very quickly.
Excited, you hire a bunch of them and put them to work recognizing cats. Now instead of two layers of people you can have 10! Each layer can focus on higher and higher concepts - the first layer can look for small details (eg. do I see a pattern that looks like a small circle? Do I see a pattern that looks like a sharp edge?), the second layer can look for patterns from the results of the first layer (eg. are there two circles close to each other), the third layer can build on that (eg. are there two circle close to each other, and a triangle below them), and so forth. By the time the results of the layers get to you you have some fairly sophisticated concepts - we see a pattern that looks like a face, we also see patterns that could be legs, and a sharp thing that might be a beak. Now your guess is more informed than ever. Excited, you hire a bunch of them and put them to work recognizing cats. Now instead of two layers of people you can have 10! Each layer can focus on higher and higher concepts - the first layer can look for small details (eg. do I see a pattern that looks like a small circle? Do I see a pattern that looks like a sharp edge?), the second layer can look for patterns from the results of the first layer (eg. are there two circles close to each other), the third layer can build on that (eg. are there two circle close to each other, and a triangle below them), and so forth. By the time the results of the layers get to you you have some fairly sophisticated concepts - we see a pattern that looks like a face, we also see patterns that could be legs, and a sharp thing that might be a beak. Now your guess is more informed than ever.
You train the layers sending slaps and rewards down through each layer, and after many epochs you get really good at recognizing cats. You train the layers sending slaps and rewards down through each layer, and after many epochs you get really good at recognizing cats.
![Exported image](Exported%20image%2020240808113913-3.png) ![](Reading/Read%20Later/assets/man-cat.png)
# Deep Learning ## Deep Learning
One of the major break-throughs in machine learning was the advent of *Deep Learning*, which is basically what we describe above - GPUs (Graphics Processing Units) got popular because they enable fast 3D graphics for games. They also happen to be very good at quickly doing the types of calculations neural networks need. Machine learning people started using GPUs for training neural networks, and with this extra speed they could have many more layers - from 3 or 4 layers to 10 or 11, and then hundreds.
One of the major break-throughs in machine learning was the advent of _Deep Learning_, which is basically what we describe above - GPUs (Graphics Processing Units) got popular because they enable fast 3D graphics for games. They also happen to be very good at quickly doing the types of calculations neural networks need. Machine learning people started using GPUs for training neural networks, and with this extra speed they could have many more layers - from 3 or 4 layers to 10 or 11, and then hundreds.
This addition of layers led to significant advances in neural network performance - very quickly neural networks became the best solution for voice recognition, for image recognition, image creation, and sophisticated language models. This is called **Deep Learning** because there are so many layers, not because it's profound. This addition of layers led to significant advances in neural network performance - very quickly neural networks became the best solution for voice recognition, for image recognition, image creation, and sophisticated language models. This is called **Deep Learning** because there are so many layers, not because it's profound.
_You can stop reading here if you want, the following portion is only here because someone asked me what "convolutional neural networks" are._
# Day Fifty *You can stop reading here if you want, the following portion is only here because someone asked me what "convolutional neural networks" are.*
## Day Fifty
How can we make the process even more efficient? You start to realize that looking at the entire sheet is too hard - the first layer of people literally have to stare at the whole thing and try to guess based on that. What if we give them a small portion of the sheet to look at, give a guess for that portion, then move on to the next portion of the sheet, and so on? The first layer of people are focused on lower level concepts anyway - looking for edges, things that look like circles, and so forth - and they can find those in the small portions of the sheet, without the need to look at the whole thing at once. How can we make the process even more efficient? You start to realize that looking at the entire sheet is too hard - the first layer of people literally have to stare at the whole thing and try to guess based on that. What if we give them a small portion of the sheet to look at, give a guess for that portion, then move on to the next portion of the sheet, and so on? The first layer of people are focused on lower level concepts anyway - looking for edges, things that look like circles, and so forth - and they can find those in the small portions of the sheet, without the need to look at the whole thing at once.
![Exported image](Exported%20image%2020240808113913-4.gif) ![](Reading/Read%20Later/assets/Convolving.gif)
[Via towardsdatascience](https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1) [Via towardsdatascience](https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1)
You tell the first layer of people this is what they should start doing. They respond that this sounds very convoluted and they'll only agree to do it if you call them "colonel". You tell the first layer of people this is what they should start doing. They respond that this sounds very convoluted and they'll only agree to do it if you call them "colonel".
This makes the process even faster and more accurate: the first layer become specialists in small features of the sheet, and they learn to be very efficient and fast. You try the same idea for the next several layers: you tell people to focus on small portions of the feedback they get from the layer below them, moving the portion they focus on so eventually the scan across all of the feedback. This makes the process even faster and more accurate: the first layer become specialists in small features of the sheet, and they learn to be very efficient and fast. You try the same idea for the next several layers: you tell people to focus on small portions of the feedback they get from the layer below them, moving the portion they focus on so eventually the scan across all of the feedback.
This specialization and focus makes you even more accurate. Congratulations, you are officially recognized as the best cat classifier in the world. This specialization and focus makes you even more accurate. Congratulations, you are officially recognized as the best cat classifier in the world.
# Convolutional Neural Networks ## Convolutional Neural Networks
Convolutions are this idea of applying a specific set of calculations, sometimes called a kernel, to each portion of the input, and scanning your window of attention across the entire image. The set of calculations, or kernel, is learned by each worker - you don't tell it what to calculate or how, it learns what's useful based on rewards and punishments. Convolutional Neural Networks (CNNs) were a significant step forward in the capability of neural networks. Convolutions are this idea of applying a specific set of calculations, sometimes called a kernel, to each portion of the input, and scanning your window of attention across the entire image. The set of calculations, or kernel, is learned by each worker - you don't tell it what to calculate or how, it learns what's useful based on rewards and punishments. Convolutional Neural Networks (CNNs) were a significant step forward in the capability of neural networks.
# Further Reading ## Further Reading
_If you're interested in this topic you might also enjoy the_ _Completley Non-Technical Explanation of ChatGPT_ _series as well._ *If you're interested in this topic you might also enjoy the other posts in the [Non-Technical Explainer](https://www.parand.com/tag/explainer.html) series as well.*

148
Reading/Done Reading/A Non-Technical Explanation of ChatGPT - Parand.md → Reading/Read Later/A Non-Technical Explanation of ChatGPT.md
View File

@@ -1,77 +1,139 @@
Clipped from: [https://www.parand.com/a-non-technical-explanation-of-chatgpt.html](https://www.parand.com/a-non-technical-explanation-of-chatgpt.html) [[ReadItLater]] [[Article]]
- [x] Read
# [A Non-Technical Explanation of ChatGPT](https://www.parand.com/a-non-technical-explanation-of-chatgpt.html)
Continuing the series of non-technical explainers, let's figure out how ChatGPT (and in general Large Language Models, or LLMs) work. This is part one of the ChatGPT explainer, with part two coming soon. Continuing the series of non-technical explainers, let's figure out how ChatGPT (and in general Large Language Models, or LLMs) work. This is part one of the ChatGPT explainer, with part two coming soon.
It's helpful but not necessary to read the [non-technical explanation of AI](https://www.parand.com/a-completely-non-technical-explanation-of-ai.html) first, if you feel like it read that and come back. It's helpful but not necessary to read the [non-technical explanation of AI](https://www.parand.com/a-completely-non-technical-explanation-of-ai.html) first, if you feel like it read that and come back.
# Fill In The Blank ## Fill In The Blank
Let's play a quick game of fill in the blank: Let's play a quick game of fill in the blank:
to be or not to _____
Why did you immediately think of the word _be_ to fill in that blank, instead of the word _banana_ or _fish_? Because you've seen the phrase many times and your brain has learned the most likely next word is _be_. > to be or not to \_\_\_\_\_
Why did you immediately think of the word *be* to fill in that blank, instead of the word *banana* or *fish*? Because you've seen the phrase many times and your brain has learned the most likely next word is *be*.
Let's do another one: Let's do another one:
rock and ____
Did you think of the word _roll_? Why? Because that's the word you see most often following the words _rock and_. > rock and \_\_\_\_
Did you think of the word *roll*? Why? Because that's the word you see most often following the words *rock and*.
How about this one: How about this one:
I am very _____
> I am very \_\_\_\_\_
This is less clear - the next word depends on the context. This is less clear - the next word depends on the context.
I just ran a marathon. I am very _____
Perhaps now you would say _tired_. Or _happy_, or _proud_. > I just ran a marathon. I am very \_\_\_\_\_
Perhaps now you would say *tired*. Or *happy*, or *proud*.
Your brain is picking the most likely next word based on the context of the sentence and based on what words it's seen most frequently in that context. Your brain is picking the most likely next word based on the context of the sentence and based on what words it's seen most frequently in that context.
# Aliens and CatGPT ## Aliens and CatGPT
In an astonishing turn of events a race of alien cats have landed on earth. Since you are [the world's foremost cat expert](https://www.parand.com/a-completely-non-technical-explanation-of-ai.html) you have been chosen to communicate with them. In an astonishing turn of events a race of alien cats have landed on earth. Since you are [the world's foremost cat expert](https://www.parand.com/a-completely-non-technical-explanation-of-ai.html) you have been chosen to communicate with them.
![Exported image](Exported%20image%2020240808113915-0.png) ![](Reading/Read%20Later/assets/alien-cat.png)
Your boss walks in and drops a massive print out of all the alien cat communications on your desk. Your boss walks in and drops a massive print out of all the alien cat communications on your desk.
"Speak cat!" she commands.
Unfortunately you don't speak alien cat.
You page through the print outs - it looks like pages and pages of gibberish.
zoog zeeg zag. zoog zeeg kaz. zoog zeeg bah. rag zoog zeeg. rag zoog ko. kaz rag. zap zoog zeeg. ...
# Markov to the Rescue "Speak cat!" she commands.
Unfortunately you don't speak alien cat.
You page through the print outs - it looks like pages and pages of gibberish.
> zoog zeeg zag. zoog zeeg kaz. zoog zeeg bah. rag zoog zeeg. rag zoog ko. kaz rag. zap zoog zeeg. ...
## Markov to the Rescue
What to do? You remember your friend Markov who's always talking about languages, words, and their relationships. Maybe he can help. You show him the alien cat communications and ask him if he can help you say something in cat. What to do? You remember your friend Markov who's always talking about languages, words, and their relationships. Maybe he can help. You show him the alien cat communications and ask him if he can help you say something in cat.
"Yes!" he exclaims - "We can do this. We will create our response one word at a time, just by picking the best word to say next, and keep going until we have a sentence" "Yes!" he exclaims - "We can do this. We will create our response one word at a time, just by picking the best word to say next, and keep going until we have a sentence"
"But how do we know what the best word to say next is? Don't we need to know what the words mean?" "But how do we know what the best word to say next is? Don't we need to know what the words mean?"
"Nope, we just need to know what word to say next" "Nope, we just need to know what word to say next"
"That... doesn't seem like it would work" "That... doesn't seem like it would work"
"Let me show you", says Markov, and grabs the book on his desk, [Alice in Wonderland](https://www.gutenberg.org/ebooks/11). "Let me show you", says Markov, and grabs the book on his desk, [Alice in Wonderland](https://www.gutenberg.org/ebooks/11).
![Exported image](Exported%20image%2020240808113915-1.png) ![](Reading/Read%20Later/assets/alice-in-wonderland-cat.png)
"We just need to know what word to say next. The best word to say next is simply the word that shows up most frequently after the word we're looking at. All we have to do is make a table of how often each word follows another word. We'll call this a frequency table" "We just need to know what word to say next. The best word to say next is simply the word that shows up most frequently after the word we're looking at. All we have to do is make a table of how often each word follows another word. We'll call this a frequency table"
He shows you how to create the frequency table - you just note down how many times each word follows another. It takes a while to go through all Alice in Wonderland and do the counting. You end up with:
Next word after "Alice" → was: 17 times, and: 16 times, thought: 12 times, had: 11 times, ...Next word after "sat" → down: 9 times, silent: 2 times, still: 2 times, for: 1 times, ...Next word after "was" → a: 31 times, the: 20 times, not: 12 times, going: 11 times, ...Next word after "and" → the: 80 times, she: 52 times, then: 31 times, was: 19 times, ...Next word after "a" → little: 59 times, very: 25 times, large: 20 times, great: 17 timesNext word after "very" → much: 10 times, soon: 7 times, curious: 6 times, glad: 5 times...
This table gives you a lot of help in forming sentences in the style of Alice in Wonderland. For example, if you start with the word _Alice_, then you'd look that up in the above table and see that the most frequent next word is _was_. And after _was_ you would select _a_. After _a_ you would get _little_. You'd end up with _Alice was a little_.
How well does this work? Let's create some sentences:
the door, when i beg your verdict," it was quite plainly through the bottle, i'm afraid that they had its nest.
You can try this out for yourself on [almeopedia](http://almeopedia.com/markovtest.html).
That's not great. Why is it so nonsensical? Think back to our fill in the blank examples at the start of this post: in order to pick a good next word you need context. If someone asked you what word should appear after _rock_, you'd have a hard time picking something reasonable, but if they gave you _rock and_, you'd fairly quickly think of _roll_.
What would happen if you considered two words instead of a single word for your context? For one thing your frequency table creation would become harder - now instead of a single line for each word, you'd have a line for each word pair. You'd have to gather stats for all the two word permutations, which is a lot more than the single word case.
Next word after "Alice was" → not: 3 times, beginning: 2 times, very: 2 times, ...Next word after "was not" → a: 3 times, here: 1 times, going: 1 times ...Next word after "not here" → before: 1 timesNext word after "not a" → moment: 2 times, bit: 2 times, serpent: 2 times ...
Let's create a sentence with this and see how it looks:
the hatter, and, burning with curiosity, she decided on going into the garden.
You can try this out for yourself on [almeopedia](http://www.almeopedia.com/markov3test.html).
That's looking better, it almost makes sense. Let's keep going - instead of two words of context, how about three?
The Hatter’s remark seemed to have no sort of chance of her ever getting out of the water, and seemed to quiver all over with diamonds, and walked two and two, as the soldiers did.
Four?
So she swallowed one of the cakes, and was delighted to find that she knew the name of nearly everything there.
Hmm. This is a little too good. It turns out a very similar sentence exists in the original Alice in Wonderland text:
So she swallowed one of the cakes, and was delighted to find that she began shrinking directly.
Our simple method of picking the most likely next word can result in the system memorizing the text snippets - given long enough context, the next most likely word is exactly the word that appeared in the original text following that context.
We can fix this by introducing some randomness - instead of always picking the most likely next word, we can pick somewhat likely next words. This way we're less likely to regurgitate the original text.
The good news is our method seems to work for English, so it'll probably work for alien cat language as well.
_Aside: You might run into the term "stochastic" as you look into language models - this just means randomly determined. For example, you might hear people argue whether these systems are_ **Stochastic Parrots**_, implying the systems are simply parroting back the original text that they saw, with some randomness thrown in._
# Large Language Models He shows you how to create the frequency table - you just note down how many times each word follows another. It takes a while to go through all Alice in Wonderland and do the counting. You end up with:
> ```
> Next word after "Alice" → was: 17 times, and: 16 times, thought: 12 times, had: 11 times, ...
> Next word after "sat" → down: 9 times, silent: 2 times, still: 2 times, for: 1 times, ...
> Next word after "was" → a: 31 times, the: 20 times, not: 12 times, going: 11 times, ...
> Next word after "and" → the: 80 times, she: 52 times, then: 31 times, was: 19 times, ...
> Next word after "a" → little: 59 times, very: 25 times, large: 20 times, great: 17 times
> Next word after "very" → much: 10 times, soon: 7 times, curious: 6 times, glad: 5 times
> ...
> ```
This table gives you a lot of help in forming sentences in the style of Alice in Wonderland. For example, if you start with the word *Alice*, then you'd look that up in the above table and see that the most frequent next word is *was*. And after *was* you would select *a*. After *a* you would get *little*. You'd end up with *Alice was a little*.
How well does this work? Let's create some sentences:
> the door, when i beg your verdict," it was quite plainly through the bottle, i'm afraid that they had its nest.
You can try this out for yourself on [almeopedia](http://almeopedia.com/markovtest.html).
That's not great. Why is it so nonsensical? Think back to our fill in the blank examples at the start of this post: in order to pick a good next word you need context. If someone asked you what word should appear after *rock*, you'd have a hard time picking something reasonable, but if they gave you *rock and*, you'd fairly quickly think of *roll*.
What would happen if you considered two words instead of a single word for your context? For one thing your frequency table creation would become harder - now instead of a single line for each word, you'd have a line for each word pair. You'd have to gather stats for all the two word permutations, which is a lot more than the single word case.
> ```
> Next word after "Alice was" → not: 3 times, beginning: 2 times, very: 2 times, ...
> Next word after "was not" → a: 3 times, here: 1 times, going: 1 times ...
> Next word after "not here" → before: 1 times
> Next word after "not a" → moment: 2 times, bit: 2 times, serpent: 2 times
> ```
>
> ...
Let's create a sentence with this and see how it looks:
> the hatter, and, burning with curiosity, she decided on going into the garden.
You can try this out for yourself on [almeopedia](http://www.almeopedia.com/markov3test.html).
That's looking better, it almost makes sense. Let's keep going - instead of two words of context, how about three?
> The Hatter’s remark seemed to have no sort of chance of her ever getting out of the water, and seemed to quiver all over with diamonds, and walked two and two, as the soldiers did.
Four?
> So she swallowed one of the cakes, and was delighted to find that she knew the name of nearly everything there.
Hmm. This is a little too good. It turns out a very similar sentence exists in the original Alice in Wonderland text:
> So she swallowed one of the cakes, and was delighted to find that she began shrinking directly.
Our simple method of picking the most likely next word can result in the system memorizing the text snippets - given long enough context, the next most likely word is exactly the word that appeared in the original text following that context.
We can fix this by introducing some randomness - instead of always picking the most likely next word, we can pick somewhat likely next words. This way we're less likely to regurgitate the original text.
The good news is our method seems to work for English, so it'll probably work for alien cat language as well.
*Aside: You might run into the term "stochastic" as you look into language models - this just means randomly determined. For example, you might hear people argue whether these systems are **Stochastic Parrots**, implying the systems are simply parroting back the original text that they saw, with some randomness thrown in.*
## Large Language Models
ChatGPT and many other Large Language Models (LLMs) do essentially what you did above: they examine a very large amount of human communication, gather stats (or probabilities) on what words are most likely to follow other words, and play a continous game of fill-in-the-blank. You give them some context (your prompt or question), and they create the reply one word at a time by selecting the word most likely to appear next. They respond with a word, then look at their internal stats to pick a word to follow their first word, then a word to follow their second word, and so forth, one word at a time, until they've formed a response. ChatGPT and many other Large Language Models (LLMs) do essentially what you did above: they examine a very large amount of human communication, gather stats (or probabilities) on what words are most likely to follow other words, and play a continous game of fill-in-the-blank. You give them some context (your prompt or question), and they create the reply one word at a time by selecting the word most likely to appear next. They respond with a word, then look at their internal stats to pick a word to follow their first word, then a word to follow their second word, and so forth, one word at a time, until they've formed a response.
# To be Continued... ## To be Continued...
In [part two of this series](https://www.parand.com/a-non-technical-explanation-of-chatgpt-deep-learning.html) we'll look at the problems you'll run into with this method and how deep learning helps you overcome those problems. In [part two of this series](https://www.parand.com/a-non-technical-explanation-of-chatgpt-deep-learning.html) we'll look at the problems you'll run into with this method and how deep learning helps you overcome those problems.
# Further Reading ## Further Reading
_If you're interested in this topic you might also enjoy the other posts in the_ _Non-Technical Explainer_ _series as well._ *If you're interested in this topic you might also enjoy the other posts in the [Non-Technical Explainer](https://www.parand.com/tag/explainer.html) series as well.*

415
Reading/Read Later/Applied LLMs - What We’ve Learned From A Year of Building with LLMs.md
View File

@@ -1,415 +0,0 @@
Clipped from: [https://applied-llms.org/](https://applied-llms.org/)
A practical guide to building successful LLM products.
Authors
[Eugene Yan](https://eugeneyan.com/)
[Bryan Bischof](https://www.linkedin.com/in/bryan-bischof/)
[Charles Frye](https://www.linkedin.com/in/charles-frye-38654abb/)
[Hamel Husain](https://hamel.dev/)
[Jason Liu](https://jxnl.co/)
[Shreya Shankar](https://www.sh-reya.com/)
Published
June 8, 2024
Also published on O’Reilly Media in three parts: [Tactical](https://www.oreilly.com/radar/what-we-learned-from-a-year-of-building-with-llms-part-i/), [Operational](https://www.oreilly.com/radar/what-we-learned-from-a-year-of-building-with-llms-part-ii/), Strategic (pending).
It’s an exciting time to build with large language models (LLMs). Over the past year, LLMs have become “good enough” for real-world applications. And they’re getting better and cheaper every year. Coupled with a parade of demos on social media, there will be an [estimated $200B investment in AI by 2025](https://www.goldmansachs.com/intelligence/pages/ai-investment-forecast-to-approach-200-billion-globally-by-2025.html). Furthermore, provider APIs have made LLMs more accessible, allowing everyone, not just ML engineers and scientists, to build intelligence into their products. Nonetheless, while the barrier to entry for building with AI has been lowered, creating products and systems that are effective—beyond a demo—remains deceptively difficult.
We’ve spent the past year building, and have discovered many sharp edges along the way. While we don’t claim to speak for the entire industry, we’d like to share what we’ve learned to help you avoid our mistakes and iterate faster. These are organized into three sections:
- [Tactical](https://applied-llms.org/#tactical-nuts--bolts-of-working-with-llms): Some practices for prompting, RAG, flow engineering, evals, and monitoring. Whether you’re a practitioner building with LLMs, or hacking on weekend projects, this section was written for you.
- [Operational](https://applied-llms.org/#operation-day-to-day-and-org-concerns): The organizational, day-to-day concerns of shipping products, and how to build an effective team. For product/technical leaders looking to deploy sustainably and reliably.
- Strategic: The long-term, big-picture view, with opinionated takes such as “no GPU before PMF” and “focus on the system not the model”, and how to iterate. Written with founders and executives in mind.
We intend to make this a practical guide to building successful products with LLMs, drawing from our own experiences and pointing to examples from around the industry.
Ready to ~~delve~~ dive in? Let’s go.
# 1 Tactical: Nuts & bolts of working with LLMs
Here, we share best practices for core components of the emerging LLM stack: prompting tips to improve quality and reliability, evaluation strategies to assess output, retrieval-augmented generation ideas to improve grounding, how to design human-in-the-loop workflows, and more. While the technology is still nascent, we trust these lessons are broadly applicable and can help you ship robust LLM applications.
## 1.1 Prompting
We recommend starting with prompting when prototyping new applications. It’s easy to both underestimate and overestimate its importance. It’s underestimated because the right prompting techniques, when used correctly, can get us very far. It’s overestimated because even prompt-based applications require significant engineering around the prompt to work well.
### 1.1.1 Focus on getting the most out of fundamental prompting techniques
A few prompting techniques have consistently helped with improving performance across a variety of models and tasks: n-shot prompts + in-context learning, chain-of-thought, and providing relevant resources.
The idea of in-context learning via n-shot prompts is to provide the LLM with examples that demonstrate the task and align outputs to our expectations. A few tips:
- If n is too low, the model may over-anchor on those specific examples, hurting its ability to generalize. As a rule of thumb, aim for n ≥ 5. Don’t be afraid to go as high as a few dozen.
- Examples should be representative of the prod distribution. If you’re building a movie summarizer, include samples from different genres in roughly the same proportion you’d expect to see in practice.
- You don’t always need to provide the input-output pairs; examples of desired outputs may be sufficient.
- If you plan for the LLM to use tools, include examples of using those tools.
In Chain-of-Thought (CoT) prompting, we encourage the LLM to explain its thought process before returning the final answer. Think of it as providing the LLM with a sketchpad so it doesn’t have to do it all in memory. The original approach was to simply add the phrase “Let’s think step by step” as part of the instructions, but, we’ve found it helpful to make the CoT more specific, where adding specificity via an extra sentence or two often reduces hallucination rates significantly.
For example, when asking an LLM to summarize a meeting transcript, we can be explicit about the steps:
- First, list out the key decisions, follow-up items, and associated owners in a sketchpad.
- Then, check that the details in the sketchpad are factually consistent with the transcript.
- Finally, synthesize the key points into a concise summary.
Note that in recent times, [some doubt](https://arxiv.org/abs/2405.04776) has been cast on if this technique is as powerful as believed. Additionally, there’s significant debate as to exactly what is going on during inference when Chain-of-Thought is being used. Regardless, this technique is one to experiment with when possible.
Providing relevant resources is a powerful mechanism to expand the model’s knowledge base, reduce hallucinations, and increase the user’s trust. Often accomplished via Retrieval Augmented Generation (RAG), providing the model with snippets of text that it can directly utilize in its response is an essential technique. When providing the relevant resources, it’s not enough to merely include them; don’t forget to tell the model to prioritize their use, refer to them directly, and to mention when none of the resources are sufficient. These help “ground” agent responses to a corpus of resources.
### 1.1.2 Structure your inputs and outputs
Structured input and output help models better understand the input as well as return output that can reliably integrate with downstream systems. Adding serialization formatting to your inputs can help provide more clues to the model as to the relationships between tokens in the context, additional metadata to specific tokens (like types), or relate the request to similar examples in the model’s training data.
As an example, many questions on the internet about writing SQL begin by specifying the SQL schema. Thus, you can expect that effective prompting for Text-to-SQL should include [structured schema definitions](https://www.researchgate.net/publication/371223615_SQL-PaLM_Improved_Large_Language_ModelAdaptation_for_Text-to-SQL).
Structured input expresses tasks clearly and resembles how the training data is formatted, increasing the probability of better output. Structured output simplifies integration into downstream components of your system. [Instructor](https://github.com/jxnl/instructor) and [Outlines](https://github.com/outlines-dev/outlines) work well for structured output. (If you’re importing an LLM API SDK, use Instructor; if you’re importing Huggingface for a self-hosted model, use Outlines.)
When using structured input, be aware that each LLM family has their own preferences. Claude prefers <xml> while GPT favors Markdown and JSON. With XML, you can even pre-fill Claude’s responses by providing a <response> tag like so.
messages=[ { "role": "user", "content": """Extract the <name>, <size>, <price>, and <color> from this product description into your <response>. <description>The SmartHome Mini is a compact smart home assistant available in black or white for only $49.99. At just 5 inches wide, it lets you control lights, thermostats, and other connected devices via voice or app—no matter where you place it in your home. This affordable little hub brings convenient hands-free control to your smart devices. </description>""" }, { "role": "assistant", "content": "<response><name>" }]
### 1.1.3 Have small prompts that do one thing, and only one thing, well
A common anti-pattern / code smell in software is the “[God Object](https://en.wikipedia.org/wiki/God_object)”, where we have a single class or function that does everything. The same applies to prompts too.
A prompt typically starts simple: A few sentences of instruction, a couple of examples, and we’re good to go. But as we try to improve performance and handle more edge cases, complexity creeps in. More instructions. Multi-step reasoning. Dozens of examples. Before we know it, our initially simple prompt is now a 2,000 token Frankenstein. And to add injury to insult, it has worse performance on the more common and straightforward inputs! GoDaddy shared this challenge as their [No. 1 lesson from building with LLMs](https://www.godaddy.com/resources/news/llm-from-the-trenches-10-lessons-learned-operationalizing-models-at-godaddy#h-1-sometimes-one-prompt-isn-t-enough).
Just like how we strive (read: struggle) to keep our systems and code simple, so should we for our prompts. Instead of having a single, catch-all prompt for the meeting transcript summarizer, we can break it into steps:
- Extract key decisions, action items, and owners into structured format
- Check extracted details against the original transcription for consistency
- Generate a concise summary from the structured details
As a result, we’ve split our single prompt into multiple prompts that are each simple, focused, and easy to understand. And by breaking them up, we can now iterate and eval each prompt individually.
### 1.1.4 Craft your context tokens
Rethink, and challenge your assumptions about how much context you actually need to send to the agent. Be like Michaelangelo, do not build up your context sculpture—chisel away the superfluous material until the sculpture is revealed. RAG is a popular way to collate all of the potentially relevant blocks of marble, but what are you doing to extract what’s necessary?
We’ve found that taking the final prompt sent to the model—with all of the context construction, and meta-prompting, and RAG results—putting it on a blank page and just reading it, really helps you rethink your context. We have found redundancy, self-contradictory language, and poor formatting using this method.
The other key optimization is the structure of your context. If your bag-of-docs representation isn’t helpful for humans, don’t assume it’s any good for agents. Think carefully about how you structure your context to underscore the relationships between parts of it and make extraction as simple as possible.
More [prompting fundamentals](https://eugeneyan.com/writing/prompting/) such as prompting mental model, prefilling, context placement, etc.
## 1.2 Information Retrieval / RAG
Beyond prompting, another effective way to steer an LLM is by providing knowledge as part of the prompt. This grounds the LLM on the provided context which is then used for in-context learning. This is known as retrieval-augmented generation (RAG). Practitioners have found RAG effective at providing knowledge and improving output, while requiring far less effort and cost compared to finetuning.
### 1.2.1 RAG is only as good as the retrieved documents’ relevance, density, and detail
The quality of your RAG’s output is dependent on the quality of retrieved documents, which in turn can be considered along a few factors
The first and most obvious metric is relevance. This is typically quantified via ranking metrics such as [Mean Reciprocal Rank (MRR)](https://en.wikipedia.org/wiki/Mean_reciprocal_rank) or [Normalized Discounted Cumulative Gain (NDCG)](https://en.wikipedia.org/wiki/Discounted_cumulative_gain). MRR evaluates how well a system places the first relevant result in a ranked list while NDCG considers the relevance of all the results and their positions. They measure how good the system is at ranking relevant documents higher and irrelevant documents lower. For example, if we’re retrieving user summaries to generate movie review summaries, we’ll want to rank reviews for the specific movie higher while excluding reviews for other movies.
Like traditional recommendation systems, the rank of retrieved items will have a significant impact on how the LLM performs on downstream tasks. To measure the impact, run a RAG-based task but with the retrieved items shuffled—how does the RAG output perform?
Second, we also want to consider information density. If two documents are equally relevant, we should prefer one that’s more concise and has fewer extraneous details. Returning to our movie example, we might consider the movie transcript and all user reviews to be relevant in a broad sense. Nonetheless, the top-rated reviews and editorial reviews will likely be more dense in information.
Finally, consider the level of detail provided in the document. Imagine we’re building a RAG system to generate SQL queries from natural language. We could simply provide table schemas with column names as context. But, what if we include column descriptions and some representative values? The additional detail could help the LLM better understand the semantics of the table and thus generate more correct SQL.
### 1.2.2 Don’t forget keyword search; use it as a baseline and in hybrid search
Given how prevalent the embedding-based RAG demo is, it’s easy to forget or overlook the decades of research and solutions in information retrieval.
Nonetheless, while embeddings are undoubtedly a powerful tool, they are not the be-all and end-all. First, while they excel at capturing high-level semantic similarity, they may struggle with more specific, keyword-based queries, like when users search for names (e.g., Ilya), acronyms (e.g., RAG), or IDs (e.g., claude-3-sonnet). Keyword-based search, such as BM25, is explicitly designed for this. Finally, after years of keyword-based search, users have likely taken it for granted and may get frustrated if the document they expect to retrieve isn’t being returned.
Vector embeddings _do not_ magically solve search. In fact, the heavy lifting is in the step before you re-rank with semantic similarity search. Making a genuine improvement over BM25 or full-text search is hard. — [Aravind Srinivas, CEO Perplexity.ai](https://x.com/AravSrinivas/status/1737886080555446552)
We’ve been communicating this to our customers and partners for months now. Nearest Neighbor Search with naive embeddings yields very noisy results and you’re likely better off starting with a keyword-based approach. — [Beyang Liu, CTO Sourcegraph](https://twitter.com/beyang/status/1767330006999720318)
Second, it’s more straightforward to understand why a document was retrieved with keyword search—we can look at the keywords that match the query. In contrast, embedding-based retrieval is less interpretable. Finally, thanks to systems like Lucene and OpenSearch that have been optimized and battle-tested over decades, keyword search is usually more computationally efficient.
In most cases, a hybrid will work best: keyword matching for the obvious matches, and embeddings for synonyms, hypernyms, and spelling errors, as well as multimodality (e.g., images and text). [Shortwave shared how they built their RAG pipeline](https://www.shortwave.com/blog/deep-dive-into-worlds-smartest-email-ai/), including query rewriting, keyword + embedding retrieval, and ranking.
### 1.2.3 Prefer RAG over fine-tuning for new knowledge
Both RAG and fine-tuning can be used to incorporate new information into LLMs and increase performance on specific tasks. However, which should we prioritize?
Recent research suggests RAG may have an edge. [One study](https://arxiv.org/abs/2312.05934) compared RAG against unsupervised finetuning (aka continued pretraining), evaluating both on a subset of MMLU and current events. They found that RAG consistently outperformed fine-tuning for knowledge encountered during training as well as entirely new knowledge. In [another paper](https://arxiv.org/abs/2401.08406), they compared RAG against supervised finetuning on an agricultural dataset. Similarly, the performance boost from RAG was greater than fine-tuning, especially for GPT-4 (see Table 20).
Beyond improved performance, RAG has other practical advantages. First, compared to continuous pretraining or fine-tuning, it’s easier—and cheaper!—to keep retrieval indices up-to-date. Second, if our retrieval indices have problematic documents that contain toxic or biased content, we can easily drop or modify the offending documents. Consider it an andon cord for [documents that ask us to add glue to pizza](https://x.com/petergyang/status/1793480607198323196).
In addition, the R in RAG provides finer-grained control over how we retrieve documents. For example, if we’re hosting a RAG system for multiple organizations, by partitioning the retrieval indices, we can ensure that each organization can only retrieve documents from their own index. This ensures that we don’t inadvertently expose information from one organization to another.
### 1.2.4 Long-context models won’t make RAG obsolete
With Gemini 1.5 providing context windows of up to 10M tokens in size, some have begun to question the future of RAG.
I tend to believe that Gemini 1.5 is significantly overhyped by Sora. A context window of 10M tokens effectively makes most of existing RAG frameworks unnecessary — you simply put whatever your data into the context and talk to the model like usual. Imagine how it does to all the startups / agents / langchain projects where most of the engineering efforts goes to RAG 😅 Or in one sentence: the 10m context kills RAG. Nice work Gemini — [Yao Fu](https://x.com/Francis_YAO_/status/1758935954189115714)
While it’s true that long contexts will be a game-changer for use cases such as analyzing multiple documents or chatting with PDFs, the rumors of RAG’s demise are greatly exaggerated.
First, even with a context size of 10M tokens, we’d still need a way to select relevant context. Second, beyond the narrow needle-in-a-haystack eval, we’ve yet to see convincing data that models can effectively reason over large context sizes. Thus, without good retrieval (and ranking), we risk overwhelming the model with distractors, or may even fill the context window with completely irrelevant information.
Finally, there’s cost. During inference, the Transformer’s time complexity scales linearly with context length. Just because there exists a model that can read your org’s entire Google Drive contents before answering each question doesn’t mean that’s a good idea. Consider an analogy to how we use RAM: we still read and write from disk, even though there exist compute instances with [RAM running into the tens of terabytes](https://aws.amazon.com/ec2/instance-types/high-memory/).
So don’t throw your RAGs in the trash just yet. This pattern will remain useful even as context sizes grow.
## 1.3 Tuning and optimizing workflows
Prompting an LLM is just the beginning. To get the most juice out of them, we need to think beyond a single prompt and embrace workflows. For example, how could we split a single complex task into multiple simpler tasks? When is finetuning or caching helpful with increasing performance and reducing latency/cost? Here, we share proven strategies and real-world examples to help you optimize and build reliable LLM workflows.
### 1.3.1 Step-by-step, multi-turn “flows” can give large boosts
It’s common knowledge that decomposing a single big prompt into multiple smaller prompts can achieve better results. For example, [AlphaCodium](https://arxiv.org/abs/2401.08500): By switching from a single prompt to a multi-step workflow, they increased GPT-4 accuracy (pass@5) on CodeContests from 19% to 44%. The workflow includes:
- Reflecting on the problem
- Reasoning on the public tests
- Generating possible solutions
- Ranking possible solutions
- Generating synthetic tests
- Iterating on the solutions on public and synthetic tests.
Small tasks with clear objectives make for the best agent or flow prompts. It’s not required that every agent prompt requests structured output, but structured outputs help a lot to interface with whatever system is orchestrating the agent’s interactions with the environment. Some things to try:
- A tightly-specified, explicit planning step. Also, consider having predefined plans to choose from.
- Rewriting the original user prompts into agent prompts, though this process may be lossy!
- Agent behaviors as linear chains, DAGs, and state machines; different dependency and logic relationships can be more and less appropriate for different scales. Can you squeeze performance optimization out of different task architectures?
- Planning validations; your planning can include instructions on how to evaluate the responses from other agents to make sure the final assembly works well together.
- Prompt engineering with fixed upstream state—make sure your agent prompts are evaluated against a collection of variants of what may have happen before.
### 1.3.2 Prioritize deterministic workflows for now
While AI agents can dynamically react to user requests and the environment, their non-deterministic nature makes them a challenge to deploy. Each step an agent takes has a chance of failing, and the chances of recovering from the error are poor. Thus, the likelihood that an agent completes a multi-step task successfully decreases exponentially as the number of steps increases. As a result, teams building agents find it difficult to deploy reliable agents.
A potential approach is to have agent systems produce deterministic plans which are then executed in a structured, reproducible way. First, given a high-level goal or prompt, the agent generates a plan. Then, the plan is executed deterministically. This allows each step to be more predictable and reliable. Benefits include:
- Generated plans can serve as few-shot samples to prompt or finetune an agent.
- Deterministic execution makes the system more reliable, and thus easier to test and debug. In addition, failures can be traced to the specific steps in the plan.
- Generated plans can be represented as directed acyclic graphs (DAGs) which are easier, relative to a static prompt, to understand and adapt to new situations.
The most successful agent builders may be those with strong experience managing junior engineers because the process of generating plans is similar to how we instruct and manage juniors. We give juniors clear goals and concrete plans, instead of vague open-ended directions, and we should do the same for our agents too.
In the end, the key to reliable, working agents will likely be found in adopting more structured, deterministic approaches, as well as collecting data to refine prompts and finetune models. Without this, we’ll build agents that may work exceptionally well some of the time, but on average, disappoint users.
### 1.3.3 Getting more diverse outputs beyond temperature
Suppose your task requires diversity in an LLM’s output. Maybe you’re writing an LLM pipeline to suggest products to buy from your catalog given a list of products the user bought previously. When running your prompt multiple times, you might notice that the resulting recommendations are too similar—so you might increase the temperature parameter in your LLM requests.
Briefly, increasing the temperature parameter makes LLM responses more varied. At sampling time, the probability distributions of the next token become flatter, meaning that tokens that are usually less likely get chosen more often. Still, when increasing temperature, you may notice some failure modes related to output diversity. For example, some products from the catalog that could be a good fit may never be output by the LLM. The same handful of products might be overrepresented in outputs, if they are highly likely to follow the prompt based on what the LLM has learned at training time. If the temperature is too high, you may get outputs that reference nonexistent products (or gibberish!)
In other words, increasing temperature does not guarantee that the LLM will sample outputs from the probability distribution you expect (e.g., uniform random). Nonetheless, we have other tricks to increase output diversity. The simplest way is to adjust elements within the prompt. For example, if the prompt template includes a list of items, such as historical purchases, shuffling the order of these items each time they’re inserted into the prompt can make a significant difference.
Additionally, keeping a short list of recent outputs can help prevent redundancy. In our recommended products example, by instructing the LLM to avoid suggesting items from this recent list, or by rejecting and resampling outputs that are similar to recent suggestions, we can further diversify the responses. Another effective strategy is to vary the phrasing used in the prompts. For instance, incorporating phrases like “pick an item that the user would love using regularly” or “select a product that the user would likely recommend to friends” can shift the focus and thereby influence the variety of recommended products.
### 1.3.4 Caching is underrated
Caching saves cost and eliminates generation latency by removing the need to recompute responses for the same input. Furthermore, if a response has previously been guardrailed, we can serve these vetted responses and reduce the risk of serving harmful or inappropriate content.
One straightforward approach to caching is to use unique IDs for the items being processed, such as if we’re summarizing new articles or [product reviews](https://www.cnbc.com/2023/06/12/amazon-is-using-generative-ai-to-summarize-product-reviews.html). When a request comes in, we can check to see if a summary already exists in the cache. If so, we can return it immediately; if not, we generate, guardrail, and serve it, and then store it in the cache for future requests.
For more open-ended queries, we can borrow techniques from the field of search, which also leverages caching for open-ended inputs. Features like autocomplete, spelling correction, and suggested queries also help normalize user input and thus increase the cache hit rate.
### 1.3.5 When to finetune
We may have some tasks where even the most cleverly designed prompts fall short. For example, even after significant prompt engineering, our system may still be a ways from returning reliable, high-quality output. If so, then it may be necessary to finetune a model for your specific task.
Successful examples include:
- [Honeycomb’s Natural Language Query Assistant](https://www.honeycomb.io/blog/introducing-query-assistant): Initially, the “programming manual” was provided in the prompt together with n-shot examples for in-context learning. While this worked decently, fine-tuning the model led to better output on the syntax and rules of the domain-specific language.
- [Rechat’s Lucy](https://www.youtube.com/watch?v=B_DMMlDuJB0): The LLM needed to generate responses in a very specific format that combined structured and unstructured data for the frontend to render correctly. Fine-tuning was essential to get it to work consistently.
Nonetheless, while fine-tuning can be effective, it comes with significant costs. We have to annotate fine-tuning data, finetune and evaluate models, and eventually self-host them. Thus, consider if the higher upfront cost is worth it. If prompting gets you 90% of the way there, then fine-tuning may not be worth the investment. However, if we do decide to finetune, to reduce the cost of collecting human-annotated data, we can [generate and finetune on synthetic data](https://eugeneyan.com/writing/synthetic/), or [bootstrap on open-source data](https://eugeneyan.com/writing/finetuning/).
## 1.4 Evaluation & Monitoring
Evaluating LLMs can be a minefield. The inputs and the outputs of LLMs are arbitrary text, and the tasks we set them to are varied. Nonetheless, rigorous and thoughtful evals are critical—it’s no coincidence that technical leaders at OpenAI [work on evaluation and give feedback on individual evals](https://twitter.com/eugeneyan/status/1701692908074873036).
Evaluating LLM applications invites a diversity of definitions and reductions: it’s simply unit testing, or it’s more like observability, or maybe it’s just data science. We have found all of these perspectives useful. In the following section, we provide some lessons we’ve learned about what is important in building evals and monitoring pipelines.
### 1.4.1 Create a few assertion-based unit tests from real input/output samples
Create [unit tests (i.e., assertions)](https://hamel.dev/blog/posts/evals/#level-1-unit-tests) consisting of samples of inputs and outputs from production, with expectations for outputs based on at least three criteria. While three criteria might seem arbitrary, it’s a practical number to start with; fewer might indicate that your task isn’t sufficiently defined or is too open-ended, like a general-purpose chatbot. These unit tests, or assertions, should be triggered by any changes to the pipeline, whether it’s editing a prompt, adding new context via RAG, or other modifications. This [write-up has an example](https://hamel.dev/blog/posts/evals/#step-1-write-scoped-tests) of an assertion-based test for an actual use case.
Consider beginning with assertions that specify phrases or ideas to either include or exclude in all responses. Also consider checks to ensure that word, item, or sentence counts lie within a range. For other kinds of generation, assertions can look different. [Execution-evaluation](https://www.semanticscholar.org/paper/Execution-Based-Evaluation-for-Open-Domain-Code-Wang-Zhou/1bed34f2c23b97fd18de359cf62cd92b3ba612c3) is a powerful method for evaluating code-generation, wherein you run the generated code and determine that the state of runtime is sufficient for the user-request.
As an example, if the user asks for a new function named foo; then after executing the agent’s generated code, foo should be callable! One challenge in execution-evaluation is that the agent code frequently leaves the runtime in slightly different form than the target code. It can be effective to “relax” assertions to the absolute most weak assumptions that any viable answer would satisfy.
Finally, using your product as intended for customers (i.e., “dogfooding”) can provide insight into failure modes on real-world data. This approach not only helps identify potential weaknesses, but also provides a useful source of production samples that can be converted into evals.
### 1.4.2 LLM-as-Judge can work (somewhat), but it’s not a silver bullet
LLM-as-Judge, where we use a strong LLM to evaluate the output of other LLMs, has been met with skepticism by some. (Some of us were initially huge skeptics.) Nonetheless, when implemented well, LLM-as-Judge achieves decent correlation with human judgements, and can at least help build priors about how a new prompt or technique may perform. Specifically, when doing pairwise comparisons (e.g., control vs. treatment), LLM-as-Judge typically gets the direction right though the magnitude of the win/loss may be noisy.
Here are some suggestions to get the most out of LLM-as-Judge: - Use pairwise comparisons: Instead of asking the LLM to score a single output on a [Likert](https://en.wikipedia.org/wiki/Likert_scale) scale, present it with two options and ask it to select the better one. This tends to lead to more stable results. - Control for position bias: The order of options presented can bias the LLM’s decision. To mitigate this, do each pairwise comparison twice, swapping the order of pairs each time. Just be sure to attribute wins to the right option after swapping! - Allow for ties: In some cases, both options may be equally good. Thus, allow the LLM to declare a tie so it doesn’t have to arbitrarily pick a winner. - Use Chain-of-Thought: Asking the LLM to explain its decision before giving a final preference can increase eval reliability. As a bonus, this allows you to use a weaker but faster LLM and still achieve similar results. Because frequently this part of the pipeline is in batch mode, the extra latency from CoT isn’t a problem. - Control for response length: LLMs tend to bias toward longer responses. To mitigate this, ensure response pairs are similar in length.
One particularly powerful application of LLM-as-Judge is checking a new prompting strategy against regression. If you have tracked a collection of production results, sometimes you can rerun those production examples with a new prompting strategy, and use LLM-as-Judge to quickly assess where the new strategy may suffer.
Here’s an example of a [simple but effective approach](https://hamel.dev/blog/posts/evals/#automated-evaluation-w-llms) to iterate on LLM-as-Judge, where we simply log the LLM response, judge’s critique (i.e., CoT), and final outcome. They are then reviewed with stakeholders to identify areas for improvement. Over three iterations, agreement with human and LLM improved from 68% to 94%!
![c model response {"calculations":[{"column":"dur MAX" ","op":"does-not-exist","join_c olumn":" "orders" order" limit" model critque The response is nearly correct, as it is looking for the slowest trace by using and ordering by duration_ms in descending order, which is appropriate for finding the 'slowest' trace. Additionally, filtering with trace.parent_id does-not-exist correctly identifies root spans. However, the query should be grouping by trace.trace_id to ensure that we identify distinct traces, not just the longest individual span. Without the correct grouping, the analysis does not guarantee that the result is a full trace, but merely the longest span. Also, specifying a limit of I is good as it will return the single slowest trace. as reauested. model outcome bad Phillip critique The response is nearly correct, as it is looking for the slowest trace by using MAX(duration_ms) and ordering by duration _ ms in descending order, which is appropriate for finding the 'slowest' trace. Additionally, filtering with trace.parent_id does-not-exist correctly identifies root spans. However, the query should be grouping by trace.trace_id to actually show the slowest trace. Without that grouping, the query only shows the MAX(duration_ms) measurement over time, irrespective of which trace is responsible for that measurement. Phillip outcome bad Phillip revised response {"calculations":[{"column":"dura tion_ms" , "op" "fi Iters" ":" does-not-exist", "join _ column" :""}],"orders":[{"column":"durati scending" Y], "limit" "time_rang agreement TRUE ](Exported%20image%2020240808113930-0.png)
LLM-as-Judge is not a silver bullet though. There are subtle aspects of language where even the strongest models fail to evaluate reliably. In addition, we’ve found that [conventional classifiers](https://eugeneyan.com/writing/finetuning/) and reward models can achieve higher accuracy than LLM-as-Judge, and with lower cost and latency. For code generation, LLM-as-Judge can be weaker than more direct evaluation strategies like execution-evaluation.
### 1.4.3 The “intern test” for evaluating generations
We like to use the following “intern test” when evaluating generations: If you took the exact input to the language model, including the context, and gave it to an average college student in the relevant major as a task, could they succeed? How long would it take?
If the answer is no because the LLM lacks the required knowledge, consider ways to enrich the context.
If the answer is no and we simply can’t improve the context to fix it, then we may have hit a task that’s too hard for contemporary LLMs.
If the answer is yes, but it would take a while, we can try to reduce the complexity of the task. Is it decomposable? Are there aspects of the task that can be made more templatized?
If the answer is yes, they would get it quickly, then it’s time to dig into the data. What’s the model doing wrong? Can we find a pattern of failures? Try asking the model to explain itself before or after it responds, to help you build a theory of mind.
### 1.4.4 Overemphasizing certain evals can hurt overall performance
“When a measure becomes a target, it ceases to be a good measure.” — Goodhart’s Law.
An example of this is the Needle-in-a-Haystack (NIAH) eval. The original eval helped quantify model recall as context sizes grew, as well as how recall is affected by needle position. However, it’s been so overemphasized that it’s featured as [Figure 1 for Gemini 1.5’s report](https://arxiv.org/abs/2403.05530). The eval involves inserting a specific phrase (“The special magic {city} number is: {number}”) into a long document which repeats the essays of Paul Graham, and then prompting the model to recall the magic number.
While some models achieve near-perfect recall, it’s questionable whether NIAH truly reflects the reasoning and recall abilities needed in real-world applications. Consider a more practical scenario: Given the transcript of an hour-long meeting, can the LLM summarize the key decisions and next steps, as well as correctly attribute each item to the relevant person? This task is more realistic, going beyond rote memorization and also considering the ability to parse complex discussions, identify relevant information, and synthesize summaries.
Here’s an example of a [practical NIAH eval](https://observablehq.com/@shreyashankar/needle-in-the-real-world-experiments). Using [transcripts of doctor-patient video calls](https://github.com/wyim/aci-bench/tree/main/data/challenge_data), the LLM is queried about the patient’s medication. It also includes a more challenging NIAH, inserting a phrase for random ingredients for pizza toppings, such as “_The secret ingredients needed to build the perfect pizza are: Espresso-soaked dates, Lemon and Goat cheese._”. Recall was around 80% on the medication task and 30% on the pizza task.
![Avg Recall of Reference Answer in Model Outputs Recall 0.9- 0.8- 0.7- 0.6- 0.5- 0.4- 0.3- 0.2- 0.1- 0.0- dbrx gemini gemma gpt3.5 gpt4 haiku -5 Model mistral opus sonnet ](Exported%20image%2020240808113930-1.png)
Tangentially, an overemphasis on NIAH evals can lead to lower performance on extraction and summarization tasks. Because these LLMs are so finetuned to attend to every sentence, they may start to treat irrelevant details and distractors as important, thus including them in the final output (when they shouldn’t!)
This could also apply to other evals and use cases. For example, summarization. An emphasis on factual consistency could lead to summaries that are less specific (and thus less likely to be factually inconsistent) and possibly less relevant. Conversely, an emphasis on writing style and eloquence could lead to more flowery, marketing-type language that could introduce factual inconsistencies.
### 1.4.5 Simplify annotation to binary tasks or pairwise comparisons
Providing open-ended feedback or ratings for model output on a [Likert scale](https://en.wikipedia.org/wiki/Likert_scale) is cognitively demanding. As a result, the data collected is more noisy—due to variability among human raters—and thus less useful. A more effective approach is to simplify the task and reduce the cognitive burden on annotators. Two tasks that work well are binary classifications and pairwise comparisons.
In binary classifications, annotators are asked to make a simple yes-or-no judgment on the model’s output. They might be asked whether the generated summary is factually consistent with the source document, or whether the proposed response is relevant, or if it contains toxicity. Compared to the Likert scale, binary decisions are more precise, have higher consistency among raters, and lead to higher throughput. This was how [Doordash setup their labeling queues](https://doordash.engineering/2020/08/28/overcome-the-cold-start-problem-in-menu-item-tagging/) for tagging menu items though a tree of yes-no questions.
In pairwise comparisons, the annotator is presented with a pair of model responses and asked which is better. Because it’s easier for humans to say “A is better than B” than to assign an individual score to either A or B individually, this leads to faster and more reliable annotations (over Likert scales). At a [Llama2 meetup](https://www.youtube.com/watch?v=CzR3OrOkM9w), Thomas Scialom, an author on the Llama2 paper, confirmed that pairwise-comparisons were faster and cheaper than collecting supervised finetuning data such as written responses. The former’s cost is $3.5 per unit while the latter’s cost is $25 per unit.
If you’re starting to write labeling guidelines, here are some [reference guidelines](https://eugeneyan.com/writing/labeling-guidelines/) from Google and Bing Search.
### 1.4.6 (Reference-free) evals and guardrails can be used interchangeably
Guardrails help to catch inappropriate or harmful content while evals help to measure the quality and accuracy of the model’s output. In the case of reference-free evals, they may be considered two sides of the same coin. Reference-free evals are evaluations that don’t rely on a “golden” reference, such as a human-written answer, and can assess the quality of output based solely on the input prompt and the model’s response.
Some examples of these are [summarization evals](https://eugeneyan.com/writing/evals/#summarization-consistency-relevance-length), where we only have to consider the input document to evaluate the summary on factual consistency and relevance. If the summary scores poorly on these metrics, we can choose not to display it to the user, effectively using the eval as a guardrail. Similarly, reference-free [translation evals](https://eugeneyan.com/writing/evals/#translation-statistical--learned-evals-for-quality) can assess the quality of a translation without needing a human-translated reference, again allowing us to use it as a guardrail.
### 1.4.7 LLMs will return output even when they shouldn’t
A key challenge when working with LLMs is that they’ll often generate output even when they shouldn’t. This can lead to harmless but nonsensical responses, or more egregious defects like toxicity or dangerous content. For example, when asked to extract specific attributes or metadata from a document, an LLM may confidently return values even when those values don’t actually exist. Alternatively, the model may respond in a language other than English because we provided non-English documents in the context.
While we can try to prompt the LLM to return a “not applicable” or “unknown” response, it’s not foolproof. Even when the log probabilities are available, they’re a poor indicator of output quality. While log probs indicate the likelihood of a token appearing in the output, they don’t necessarily reflect the correctness of the generated text. On the contrary, for instruction-tuned models that are trained to respond to queries and generate coherent response, log probabilities may not be well-calibrated. Thus, while a high log probability may indicate that the output is fluent and coherent, it doesn’t mean it’s accurate or relevant.
While careful prompt engineering can help to some extent, we should complement it with robust guardrails that detect and filter/regenerate undesired output. For example, OpenAI provides a [content moderation API](https://platform.openai.com/docs/guides/moderation) that can identify unsafe responses such as hate speech, self-harm, or sexual output. Similarly, there are numerous packages for [detecting personally identifiable information](https://github.com/topics/pii-detection) (PII). One benefit is that guardrails are largely agnostic of the use case and can thus be applied broadly to all output in a given language. In addition, with precise retrieval, our system can deterministically respond “I don’t know” if there are no relevant documents.
A corollary here is that LLMs may fail to produce outputs when they are expected to. This can happen for various reasons, from straightforward issues like long tail latencies from API providers to more complex ones such as outputs being blocked by content moderation filters. As such, it’s important to consistently log inputs and (potentially a lack of) outputs for debugging and monitoring.
### 1.4.8 Hallucinations are a stubborn problem
Unlike content safety or PII defects which have a lot of attention and thus seldom occur, factual inconsistencies are stubbornly persistent and more challenging to detect. They’re more common and occur at a baseline rate of 5 - 10%, and from what we’ve learned from LLM providers, it can be challenging to get it below 2%, even on simple tasks such as summarization.
To address this, we can combine prompt engineering (upstream of generation) and factual inconsistency guardrails (downstream of generation). For prompt engineering, techniques like CoT help reduce hallucination by getting the LLM to explain its reasoning before finally returning the output. Then, we can apply a [factual inconsistency guardrail](https://eugeneyan.com/writing/finetuning/) to assess the factuality of summaries and filter or regenerate hallucinations. In some cases, hallucinations can be deterministically detected. When using resources from RAG retrieval, if the output is structured and identifies what the resources are, you should be able to manually verify they’re sourced from the input context.
# 2 Operational: Day-to-day and org concerns
## 2.1 Data
Just as the quality of ingredients determines the dish’s taste, the quality of input data constrains the performance of machine learning systems. In addition, output data is the only way to tell whether the product is working or not. All the authors focus tightly on the data, looking at inputs and outputs for several hours a week to better understand the data distribution: its modes, its edge cases, and the limitations of models of it.
### 2.1.1 Check for development-prod skew
A common source of errors in traditional machine learning pipelines is _train-serve skew_. This happens when the data used in training differs from what the model encounters in production. Although we can use LLMs without training or fine-tuning, hence there’s no training set, a similar issue arises with development-prod data skew. Essentially, the data we test our systems on during development should mirror what the systems will face in production. If not, we might find our production accuracy suffering.
LLM development-prod skew can be categorized into two types: structural and content-based. Structural skew includes issues like formatting discrepancies, such as differences between a JSON dictionary with a list-type value and a JSON list, inconsistent casing, and errors like typos or sentence fragments. These errors can lead to unpredictable model performance because different LLMs are trained on specific data formats, and prompts can be highly sensitive to minor changes. Content-based or “semantic” skew refers to differences in the meaning or context of the data. 
As in traditional ML, it’s useful to periodically measure skew between the LLM input/output pairs. Simple metrics like the length of inputs and outputs or specific formatting requirements (e.g., JSON or XML) are straightforward ways to track changes. For more “advanced” drift detection, consider clustering embeddings of input/output pairs to detect semantic drift, such as shifts in the topics users are discussing, which could indicate they are exploring areas the model hasn’t been exposed to before. 
When testing changes, such as prompt engineering, ensure that hold-out datasets are current and reflect the most recent types of user interactions. For example, if typos are common in production inputs, they should also be present in the hold-out data. Beyond just numerical skew measurements, it’s beneficial to perform qualitative assessments on outputs. Regularly reviewing your model’s outputs—a practice colloquially known as “vibe checks”—ensures that the results align with expectations and remain relevant to user needs. Finally, incorporating nondeterminism into skew checks is also useful—by running the pipeline multiple times for each input in our testing dataset and analyzing all outputs, we increase the likelihood of catching anomalies that might occur only occasionally.
### 2.1.2 Look at samples of LLM inputs and outputs every day
LLMs are dynamic and constantly evolving. Despite their impressive zero-shot capabilities and often delightful outputs, their failure modes can be highly unpredictable. For custom tasks, regularly reviewing data samples is essential to developing an intuitive understanding of how LLMs perform.
Input-output pairs from production are the “real things, real places” (_genchi genbutsu_) of LLM applications, and they cannot be substituted. [Recent research](https://arxiv.org/abs/2404.12272) highlighted that developers’ perceptions of what constitutes “good” and “bad” outputs shift as they interact with more data (i.e., _criteria drift_). While developers can come up with some criteria upfront for evaluating LLM outputs, these predefined criteria are often incomplete. For instance, during the course of development, we might update the prompt to increase the probability of good responses and decrease the probability of bad ones. This iterative process of evaluation, reevaluation, and criteria update is necessary, as it’s difficult to predict either LLM behavior or human preference without directly observing the outputs.
To manage this effectively, we should log LLM inputs and outputs. By examining a sample of these logs daily, we can quickly identify and adapt to new patterns or failure modes. When we spot a new issue, we can immediately write an assertion or eval around it. Similarly, any updates to failure mode definitions should be reflected in the evaluation criteria. These “vibe checks” are signals of bad outputs; code and assertions operationalize them. Finally, this attitude must be socialized, for example by adding review or annotation of inputs and outputs to your on-call rotation.
## 2.2 Working with models
With LLM APIs, we can rely on intelligence from a handful of providers. While this is a boon, these dependencies also involve trade-offs on performance, latency, throughput, and cost. Also, as newer, better models drop (almost every month in the past year), we should be prepared to update our products as we deprecate old models and migrate to newer models. In this section, we share our lessons from working with technologies we don’t have full control over, where the models can’t be self-hosted and managed.
### 2.2.1 Generate structured output to ease downstream integration
For most real-world use cases, the output of an LLM will be consumed by a downstream application via some machine-readable format. For example, [ReChat](https://www.youtube.com/watch?v=B_DMMlDuJB0), a real-estate CRM, required structured responses for the front end to render widgets. Similarly, [Boba](https://martinfowler.com/articles/building-boba.html), a tool for generating product strategy ideas, needed structured output with fields for title, summary, plausibility score, and time horizon. Finally, LinkedIn shared about [constraining the LLM to generate YAML](https://www.linkedin.com/blog/engineering/generative-ai/musings-on-building-a-generative-ai-product), which is then used to decide which skill to use, as well as provide the parameters to invoke the skill.
This application pattern is an extreme version of Postel’s Law: be liberal in what you accept (arbitrary natural language) and conservative in what you send (typed, machine-readable objects). As such, we expect it to be extremely durable.
Currently, [Instructor](https://github.com/jxnl/instructor) and [Outlines](https://github.com/outlines-dev/outlines) are the de facto standards for coaxing structured output from LLMs. If you’re using an LLM API (e.g., Anthropic, OpenAI), use Instructor; if you’re working with a self-hosted model (e.g., Huggingface), use Outlines.
### 2.2.2 Migrating prompts across models is a pain in the ass
Sometimes, our carefully crafted prompts work superbly with one model but fall flat with another. This can happen when we’re switching between various model providers, as well as when we upgrade across versions of the same model. 
For example, Voiceflow found that [migrating from gpt-3.5-turbo-0301 to gpt-3.5-turbo-1106 led to a 10% drop](https://www.voiceflow.com/blog/how-much-do-chatgpt-versions-affect-real-world-performance) on their intent classification task. (Thankfully, they had evals!) Similarly, [GoDaddy observed a trend in the positive direction](https://www.godaddy.com/resources/news/llm-from-the-trenches-10-lessons-learned-operationalizing-models-at-godaddy#h-3-prompts-aren-t-portable-across-models), where upgrading to version 1106 narrowed the performance gap between gpt-3.5-turbo and gpt-4. (Or, if you’re a glass-half-full person, you might be disappointed that gpt-4’s lead was reduced with the new upgrade)
Thus, if we have to migrate prompts across models, expect it to take more time than simply swapping the API endpoint. Don’t assume that plugging in the same prompt will lead to similar or better results. Also, having reliable, automated evals helps with measuring task performance before and after migration, and reduces the effort needed for manual verification.
### 2.2.3 Version and pin your models
In any machine learning pipeline, “[changing anything changes everything](https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html)”. This is particularly relevant as we rely on components like large language models (LLMs) that we don’t train ourselves and that can change without our knowledge.
Fortunately, many model providers offer the option to “pin” specific model versions (e.g., gpt-4-turbo-1106). This enables us to use a specific version of the model weights, ensuring they remain unchanged. Pinning model versions in production can help avoid unexpected changes in model behavior, which could lead to customer complaints about issues that may crop up when a model is swapped, such as overly verbose outputs or other unforeseen failure modes.
Additionally, consider maintaining a shadow pipeline that mirrors your production setup but uses the latest model versions. This enables safe experimentation and testing with new releases. Once you’ve validated the stability and quality of the outputs from these newer models, you can confidently update the model versions in your production environment.
### 2.2.4 Choose the smallest model that gets the job done
When working on a new application, it’s tempting to use the biggest, most powerful model available. But once we’ve established that the task is technically feasible, it’s worth experimenting if a smaller model can achieve comparable results.
The benefits of a smaller model are lower latency and cost. While it may be weaker, techniques like chain-of-thought, n-shot prompts, and in-context learning can help smaller models punch above their weight. Beyond LLM APIs, fine-tuning our specific tasks can also help increase performance.
Taken together, a carefully crafted workflow using a smaller model can often match, or even surpass, the output quality of a single large model, while being faster and cheaper. For example, this [tweet](https://twitter.com/mattshumer_/status/1770823530394833242) shares anecdata of how Haiku + 10-shot prompt outperforms zero-shot Opus and GPT-4. In the long term, we expect to see more examples of [flow-engineering](https://twitter.com/karpathy/status/1748043513156272416) with smaller models as the optimal balance of output quality, latency, and cost.
As another example, take the humble classification task. Lightweight models like DistilBERT (67M parameters) are a surprisingly strong baseline. The 400M parameter DistilBART is another great option—when finetuned on open-source data, it could [identify hallucinations with an ROC-AUC of 0.84](https://eugeneyan.com/writing/finetuning/), surpassing most LLMs at less than 5% of latency and cost.
The point is, don’t overlook smaller models. While it’s easy to throw a massive model at every problem, with some creativity and experimentation, we can often find a more efficient solution. 
## 2.3 Product
While new technology offers new possibilities, the principles of building great products are timeless. Thus, even if we’re solving new problems for the first time, we don’t have to reinvent the wheel on product design. There’s a lot to gain from grounding our LLM application development in solid product fundamentals, allowing us to deliver real value to the people we serve.
### 2.3.1 Involve design early and often
Having a designer will push you to understand and think deeply about how your product can be built and presented to users. We sometimes stereotype designers as folks who take things and make them pretty. But beyond just the user interface, they also rethink how the user experience can be improved, even if it means breaking existing rules and paradigms.
Designers are especially gifted at reframing the user’s needs into various forms. Some of these forms are more tractable to solve than others, and thus, they may offer more or fewer opportunities for AI solutions. Like many other products, building AI products should be centered around the job to be done, not the technology that powers them.
Focus on asking yourself: “What job is the user asking this product to do for them? Is that job something a chatbot would be good at? How about autocomplete? Maybe something different!” Consider the existing [design patterns](https://www.tidepool.so/blog/emerging-ux-patterns-for-generative-ai-apps-copilots) and how they relate to the job-to-be-done. These are the invaluable assets that designers add to your team’s capabilities.
### 2.3.2 Design your UX for Human-In-The-Loop
One way to get quality annotations is to integrate Human-in-the-Loop (HITL) into the user experience (UX). By allowing users to provide feedback and corrections easily, we can improve the immediate output and collect valuable data to improve our models.
Imagine an e-commerce platform where users upload and categorize their products. There are several ways we could design the UX:
- The user manually selects the right product category; an LLM periodically checks new products and corrects miscategorization on the backend.
- The user doesn’t select any category at all; an LLM periodically categorizes products on the backend (with potential errors).
- An LLM suggests a product category in real-time, which the user can validate and update as needed.
While all three approaches involve an LLM, they provide very different UXes. The first approach puts the initial burden on the user and has the LLM acting as a post-processing check. The second requires zero effort from the user but provides no transparency or control. The third strikes the right balance. By having the LLM suggest categories upfront, we reduce cognitive load on the user and they don’t have to learn our taxonomy to categorize their product! At the same time, by allowing the user to review and edit the suggestion, they have the final say in how their product is classified, putting control firmly in their hands. As a bonus, the third approach creates a [natural feedback loop for model improvement](https://eugeneyan.com/writing/llm-patterns/#collect-user-feedback-to-build-our-data-flywheel). Suggestions that are good are accepted (positive labels) and those that are bad are updated (negative followed by positive labels).
This pattern of suggestion, user validation, and data collection is commonly seen in several applications:
- Coding assistants: Where users can accept a suggestion (strong positive), accept and tweak a suggestion (positive), or ignore a suggestion (negative)
- Midjourney: Where users can choose to upscale and download the image (strong positive), vary an image (positive), or generate a new set of images (negative)
- Chatbots: Where users can provide thumbs up (positive) or thumbs down (negative) on responses, or choose to regenerate a response if it was really bad (strong negative).
Feedback can be explicit or implicit. Explicit feedback is information users provide in response to a request by our product; implicit feedback is information we learn from user interactions without needing users to deliberately provide feedback. Coding assistants and Midjourney are examples of implicit feedback while thumbs up and thumb downs are explicit feedback. If we design our UX well, like coding assistants and Midjourney, we can collect plenty of implicit feedback to improve our product and models.
### 2.3.3 Prioritize your hierarchy of needs ruthlessly
As we think about putting our demo into production, we’ll have to think about the requirements for:
- Reliability: 99.9% uptime, adherence to structured output
- Harmlessness: Not generate offensive, NSFW, or otherwise harmful content
- Factual consistency: Being faithful to the context provided, not making things up
- Usefulness: Relevant to the users’ needs and request
- Scalability: Latency SLAs, supported throughput
- Cost: Because we don’t have unlimited budget
- And more: Security, privacy, fairness, GDPR, DMA, etc, etc.
If we try to tackle all these requirements at once, we’re never going to ship anything. Thus, we need to prioritize. Ruthlessly. This means being clear what is non-negotiable (e.g., reliability, harmlessness) without which our product can’t function or won’t be viable. It’s all about identifying the minimum lovable product. We have to accept that the first version won’t be perfect, and just launch and iterate.
### 2.3.4 Calibrate your risk tolerance based on the use case
When deciding on the language model and level of scrutiny of an application, consider the use case and audience. For a customer-facing chatbot offering medical or financial advice, we’ll need a very high bar for safety and accuracy. Mistakes or bad output could cause real harm and erode trust. But for less critical applications, such as a recommender system, or internal-facing applications like content classification or summarization, excessively strict requirements only slow progress without adding much value.
This aligns with a recent [a16z report](https://a16z.com/generative-ai-enterprise-2024/) showing that many companies are moving faster with internal LLM applications compared to external ones. By experimenting with AI for internal productivity, organizations can start capturing value while learning how to manage risk in a more controlled environment. Then, as they gain confidence, they can expand to customer-facing use cases.
## 2.4 Team & Roles
No job function is easy to define, but writing a job description for the work in this new space is more challenging than others. We’ll forgo venn diagrams of intersecting job titles, or suggestions for job descriptions. We will, however, submit to the existence of a new role—the AI engineer—and discuss its place. Importantly, we’ll discuss the rest of the team and how responsibilities should be assigned.
### 2.4.1 Focus on process, not tools
When faced with new paradigms, such as LLMs, software engineers tend to favor tools. As a result, we overlook the problem and process the tool was supposed to solve. In doing so, many engineers assume accidental complexity, which has negative consequences for the team’s long-term productivity.
For example, [this write-up](https://hamel.dev/blog/posts/prompt/) discusses how certain tools can automatically create prompts for large language models. It argues (rightfully IMHO) that engineers who use these tools without first understanding the problem-solving methodology or process end up taking on unnecessary technical debt.
In addition to accidental complexity, tools are often underspecified. For example, there is a growing industry of LLM evaluation tools that offer “LLM Evaluation In A Box” with generic evaluators for toxicity, conciseness, tone, etc. We have seen many teams adopt these tools without thinking critically about the specific failure modes of their domains. Contrast this to EvalGen. It focuses on teaching users the process of creating domain-specific evals by deeply involving the user each step of the way, from specifying criteria, to labeling data, to checking evals. The software leads the user through a workflow that looks like this:
![Prompt Node Infer You be doing named entity recognition (NER). Extract up to 3 well-known entities fron the following tweet: {tweet_fu For each entity, write one sentence describing the person or entity. ALI the entities you extract should be found in a knowledge base Like Wikipedia, SO don't make up Nurn responses per prompt: Multi -Evaluator criteria from my context you extract should' Let me specify criteria manually Let an Al help you generate criteria and implement evaluation functions. Is this or ? - Bravotv: A television network that focuses on reality TV shows, including popular f tike The Real Housewives. - BravoWWHL: Stands for Bravo's Watch What Happens Live, a late—night talk hosted by Andy Cohen that features celebrity games, and discussions about Bravo's reality TV shows. — Paris Hilton: A well—known socialite, businesswoman, and media personality, known for her appearance on the reality TV show The Simple Life and her work as a Singer, actress, and entrepreneur. ntities I'm tired Type a criteria to add, then press Enter: Markdown Format The response should be in Markdown format. NO Made up Entities There shouldn't be any made up entities in the response, Grade some responses first Grade Some responses first. to help sn•urselt identify criteria. The Al will incorporate your grades in its criteria suggestions. Suggest more Bulleted List The response should contain a bulleted list. NO Hashtags The response should not extract hashtags as entities. "_text STOLE WY HOUSE *justdoit #juststealit uravOtv @BravON4L https://t.co.'8xShKrøSYq Prompt You be doing nard entity recognition (NER). Extract up to 3 entities f rn the tueet : you STOLE MY GOD""' HOUSE *justdolt •juststeatit aparisHilton https://t.co'8xShKreSYq For each entity, write one sentence describing the person or entity. Alt the I'm done. Implement it! Coverage ot Bad Reswtses 77.78% Ento Stuw False Failure Rate 28.57% ](Exported%20image%2020240808113930-2.png)
[Shankar, S., et al. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. Retrieved from https://arxiv.org/abs/2404.12272](https://arxiv.org/abs/2404.12272)
EvalGen guides the user through a best practice of crafting LLM evaluations, namely:
1. Defining domain-specific tests (bootstrapped automatically from the prompt). These are defined as either assertions with code or with LLM-as-a-Judge.
2. The importance of aligning the tests with human judgment, so that the user can check that the tests capture the specified criteria.
3. Iterating on your tests as the system (prompts, etc) changes. 
EvalGen provides developers with a mental model of the evaluation building process without anchoring them to a specific tool. We have found that after providing AI Engineers with this context, they often decide to select leaner tools or build their own.  
There are too many components of LLMs beyond prompt writing and evaluations to list exhaustively here.  However, it is important that AI Engineers seek to understand the processes before adopting tools.
### 2.4.2 Always be experimenting
ML products are deeply intertwined with experimentation. Not only the A/B, Randomized Control Trials kind, but the frequent attempts at modifying the smallest possible components of your system, and doing offline evaluation. The reason why everyone is so hot for evals is not actually about trustworthiness and confidence—it’s about enabling experiments! The better your evals, the faster you can iterate on experiments, and thus the faster you can converge on the best version of your system. 
It’s common to try different approaches to solving the same problem because experimentation is so cheap now. The high-cost of collecting data and training a model is minimized—prompt engineering costs little more than human time. Position your team so that everyone is taught the basics of prompt engineering. This encourages everyone to experiment and leads to diverse ideas from across the organization.
Additionally, don’t only experiment to explore—also use them to exploit! Have a working version of a new task? Consider having someone else on the team approach it differently. Try doing it another way that’ll be faster. Investigate prompt techniques like Chain-of-Thought or Few-Shot to make it higher quality. Don’t let your tooling hold you back on experimentation; if it is, rebuild it, or buy something to make it better. 
Finally, during product/project planning, set aside time for building evals and running multiple experiments. Think of the product spec for engineering products, but add to it clear criteria for evals. And during roadmapping, don’t underestimate the time required for experimentation—expect to do multiple iterations of development and evals before getting the green light for production.
### 2.4.3 Empower everyone to use new AI technology
As generative AI increases in adoption, we want the entire team—not just the experts—to understand and feel empowered to use this new technology. There’s no better way to develop intuition for how LLMs work (e.g., latencies, failure modes, UX) than to, well, use them. LLMs are relatively accessible: You don’t need to know how to code to improve performance for a pipeline, and everyone can start contributing via prompt engineering and evals.
A big part of this is education. It can start as simple as the basics of prompt engineering, where techniques like n-shot prompting and CoT help condition the model towards the desired output. Folks who have the knowledge can also educate about the more technical aspects, such as how LLMs are autoregressive in nature. In other words, while input tokens are processed in parallel, output tokens are generated sequentially. As a result, latency is more a function of output length than input length—this is a key consideration when designing UXes and setting performance expectations.
We can also go further and provide opportunities for hands-on experimentation and exploration. A hackathon perhaps? While it may seem expensive to have an entire team spend a few days hacking on speculative projects, the outcomes may surprise you. We know of a team that, through a hackathon, accelerated and almost completed their three-year roadmap within a year. Another team had a hackathon that led to paradigm shifting UXes that are now possible thanks to LLMs, which are now prioritized for the year and beyond.
### 2.4.4 Don’t fall into the trap of “AI Engineering is all I need”
As new job titles are coined, there is an initial tendency to overstate the capabilities associated with these roles. This often results in a painful correction as the actual scope of these jobs becomes clear. Newcomers to the field, as well as hiring managers, might make exaggerated claims or have inflated expectations. Notable examples over the last decade include:
- Data Scientist: “[someone who is better at statistics than any software engineer and better at software engineering than any statistician](https://x.com/josh_wills/status/198093512149958656).”  
- Machine Learning Engineer (MLE): a software engineering-centric view of machine learning 
Initially, many assumed that data scientists alone were sufficient for data-driven projects. However, it became apparent that data scientists must collaborate with software and data engineers to develop and deploy data products effectively. 
This misunderstanding has shown up again with the new role of AI Engineer, with some teams believing that AI Engineers are all you need. In reality, building machine learning or AI products requires a [broad array of specialized roles](https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html). We’ve consulted with more than a dozen companies on AI products and have consistently observed that they fall into the trap of believing that “AI Engineering is all you need.” As a result, products often struggle to scale beyond a demo as companies overlook crucial aspects involved in building a product.
For example, evaluation and measurement are crucial for scaling a product beyond vibe checks. The skills for effective evaluation align with some of the strengths traditionally seen in machine learning engineers—a team composed solely of AI Engineers will likely lack these skills. Co-author Hamel Husain illustrates the importance of these skills in his recent work around detecting [data drift](https://github.com/hamelsmu/ft-drift) and [designing domain-specific evals](https://hamel.dev/blog/posts/evals/).
Here is a rough progression of the types of roles you need, and when you’ll need them, throughout the journey of building an AI product:
1. First, focus on building a product. This might include an AI engineer, but it doesn’t have to. AI Engineers are valuable for prototyping and iterating quickly on the product (UX, plumbing, etc). 
2. Next, create the right foundations by instrumenting your system and collecting data. Depending on the type and scale of data, you might need platform and/or data engineers. You must also have systems for querying and analyzing this data to debug issues.
3. Next, you will eventually want to optimize your AI system. This doesn’t necessarily involve training models. The basics include steps like designing metrics, building evaluation systems, running experiments, optimizing RAG retrieval, debugging stochastic systems, and more. MLEs are really good at this (though AI engineers can pick them up too). It usually doesn’t make sense to hire an MLE unless you have completed the prerequisite steps.
Aside from this, you need a domain expert at all times. At small companies, this would ideally be the founding team—and at bigger companies, product managers can play this role. Being aware of the progression and timing of roles is critical. Hiring folks at the wrong time (e.g., [hiring an MLE too early](https://jxnl.co/writing/2024/04/08/hiring-mle-at-early-stage-companies/)) or building in the wrong order is a waste of time and money, and causes churn.  Furthermore, regularly checking in with an MLE (but not hiring them full-time) during phases 1-2 will help the company build the right foundations. 
# 3 Strategic: Long-term business strategy (pending)
PENDING RELEASE (tentatively 6th June)
# 4 Stay In Touch
If you found this useful and want updates on write-ups, courses, and activities, subscribe below.
You can also find our individual contact information on our [about page](https://applied-llms.org/about.html).
## 4.1 Acknowledgements
This series started as a conversation in a group chat, where Bryan quipped that he was inspired to write “A Year of AI Engineering”. Then, ✨magic✨ happened, and we were all inspired to chip in and share what we’ve learned so far.
The authors would like to thank Eugene for leading the bulk of the document integration and overall structure in addition to a large proportion of the lessons. Additionally, for primary editing responsibilities and document direction. The authors would like to thank Bryan for the spark that led to this writeup, restructuring the write-up into tactical, operational, and strategic sections and their intros, and for pushing us to think bigger on how we could reach and help the community. The authors would like to thank Charles for his deep dives on cost and LLMOps, as well as weaving the lessons to make them more coherent and tighter—you have him to thank for this being 30 instead of 40 pages! The authors thank Hamel and Jason for their insights from advising clients and being on the front lines, for their broad generalizable learnings from clients, and for deep knowledge of tools. And finally, thank you Shreya for reminding us of the importance of evals and rigorous production practices and for bringing her research and original results.
Finally, we would like to thank all the teams who so generously shared your challenges and lessons in your own write-ups which we’ve referenced throughout this series, along with the AI communities for your vibrant participation and engagement with this group.
## 4.2 About the authors
See the [about page](https://applied-llms.org/about.html) for more information on the authors.
If you found this useful, please cite this write-up as:
Yan, Eugene, Bryan Bischof, Charles Frye, Hamel Husain, Jason Liu, and Shreya Shankar. 2024. ‘Applied LLMs - What We’ve Learned From A Year of Building with LLMs’. Applied LLMs. 8 June 2024. [https://applied-llms.org/](https://applied-llms.org/).
or
@article{AppliedLLMs2024, title = {What We've Learned From A Year of Building with LLMs}, author = {Yan, Eugene and Bischof, Bryan and Frye, Charles and Husain, Hamel and Liu, Jason and Shankar, Shreya}, journal = {Applied LLMs}, year = {2024}, month = {Jun}, url = {https://applied-llms.org/}}

205
Reading/Read Later/Done, and Gets Things Smart.md Normal file
View File

@@ -0,0 +1,205 @@
[[ReadItLater]] [[Article]]
# [Done, and Gets Things Smart](https://steve-yegge.blogspot.com/2008/06/done-and-gets-things-smart.html)
*Disclaimer: I do not speak for Google! These are my own views and opinions, and are not endorsed in any way by my employer, nor anyone else, for that matter.*
Everyone knows and quotes Joel's old chestnut, "**Smart**, and **Gets Things Done**." It was a [blog](http://www.joelonsoftware.com/articles/fog0000000073.html), then a book, and now it's an aphorism.
People quote Joel's Proverb all the time because it gives us all such a nice snuggly feeling. Why? Because to be in this industry at all, even a bottom-feeder, you *have* to be smart. You were probably the top kid in your elementary school class. People probably picked on you. You were the geek back when geeks weren't popular. But now "smart" is fashionable.
That's right! All those loser kneebiter jocks in high school who played varsity and got all the girls and sported their big, winning, shark-white smiles as they barely jee-parlor-fran-saysed their way through the classes you coasted through: where are they now? A bunch of sorry-ass bank managers and failed insurance salesmen and suit-wearing stiffs at big law firms working for billion-dollar conglomerates where their daddies got them VP jobs where they just have to show up and putt against the other VPs on little astroturf greens in the hallways!
That's right, los... er, well, some of them are doing OK, I guess. "But they're not as rich as Bill Gates!" That's the other big tautology-cum-aphorism we geeks like to pull out when we're feeling insecure. Notwithstanding that Bill had a rich daddy and made his money through exactly the kind of corporate shenanigans that Big Meat is using on us today, with those selfsame jocks. We like to think the more important point is that Bill, Jobs, Bezos and the other tech billionaires (all fierce, business-savvy people) were geeks, and are thus evidence of the revenge of the nerds. And hey, tech did make a lot of money in the 80s and 90s. So "smart" is sort of in fashion now.
Unfortunately, "smart" is a generic enough concept that pretty much everyone in the world thinks they're smart. This is due to the Nobel-prize-winning [Dunning-Kruger Effect](http://en.wikipedia.org/wiki/Dunning-Kruger_effect), which says, in effect, that people don't know when they're not smart.
*(Note: people have pointed out that it was an Ig-Nobel. Me == Dumbass. Fortunately, "me == dumbass" was more or less the point of the article, so I'll let it stand.)*
So looking for **Smart** is a bit problematic, since we aren't smart enough to distinguish it from B.S. The best we can do is find people who we *think* are smart because they're a bit like us.
Er, what about **Gets Stuff Done**, then? Well, the other qualification you need to be in this industry is that you have to have produced *something*. Anything at all. As soon as you write your first working program, you've gotten something done. And at that point you probably think of yourself as being a pretty hot market commodity, again because of the Dunning-Kruger Effect.
So, like, what kind of people is this **Smart, and Gets Things Done** adage actually hiring?
I've said it before: I thought I was a top-5% (or so) programmer for the first fifteen years I was coding. (1987-2002). Every time I learned something new, I thought "Gosh, I was really dumb for not knowing that, but *now* I'm definitely a superstar." It took me fifteen frigging years before I realized that there might in fact still be "one or two things" left to learn, at which point I started looking outward and finally discovered how absolutely bambi-esquely thumperly incompetently clueless I really am. Fifteen years!
But hell, let's be honest here: I still think I'm smart. We all do. Sure, I've managed to figure out, at least partly, just how un-smart I am, but I've got the whole "I'm smart" thing so hardwired from when I was a kid surrounded by "dolts" who couldn't memorize things as fast as I could, or predict the teacher's punch line way in advance, or whatever questionable heuristic I was using for measuring my own smartness, that it's hard not to think that way. And of course the "dolts" were way better than me at just about everything else. And they think they're pretty smart too! Definitely above average, anyway.
**Squaaaaawk**
So we all think we're smart for different reasons. Mine was memorization. Smart, eh? In reality I was just a giant, uncomprehending parrot. I got my first big nasty surprise when I was in the Navy Nuclear Power School program in Orlando, Florida, and I was setting historical records for the highest scores on their exams. The courses and exams had been carefully designed over some 30 years to maximize and then test "literal retention" of the material. They gave you all the material in outline form, and made you write it in your notebook, and your test answers were graded on edit-distance from the original notes. (I'm not making this up or exaggerating in the slightest.) They had set up the ultimate parrot game, and I happily accepted. I memorized the entire notebooks word-for-word, and aced their tests.
They treated me like some sort of movie star — that is, until the Radar final lab exam in electronics school, in which we had to troubleshoot an actual working (well, technically, not-working) radar system. I failed spectacularly: I'd arguably set another historical record, because I had *no idea* what to do. I just stood there hemming and hawing and pooing myself for three hours. I hadn't understood a single thing I'd memorized. Hey man, I was just playing their game! But I lost. I mean, I still made it through just fine, but I lost the celebrity privileges in a big way.
Having a good memory is a serious impediment to understanding. It lets you cheat your way through life. I've never learned to read sheet music to anywhere near the level I can play (for both guitar and piano.) I have large-ish repertoires and, at least for guitar, good technique from lots of lessons, but since I could memorize the sheet music in one sitting, I never learned how to read it faster than about a measure a minute. (It's not a photographic memory - I have to work a little to commit it to memory. But it was a lot less work than learning to read the music.) And as a result, my repertoire is only a thousandth what it could be if I knew how to read.
My memory (and, you know, overall laziness) has made me musically illiterate.
But when you combine the Dunning-Kruger effect (which affects me just as much as it does you) with having one or two things I've been good at in the past, it's all too easy to fall into the trap of thinking of myself as "smart", even if I know better now. All you have to do, to be "smart", is have a little competency at something, anything at all, just enough to be dangerous, and then the Dunning-Kruger Effect makes you think you're God's gift to that field, discipline, or what have you.
This is why everyone loves **Smart, and Gets Things Done**, which Joel always writes in boldface. We love it because right from the start it has the yummy baked-in assumption that *you* are smart, and that *you* get things done. And it also tacitly assumes that you know how to identify other people with the same qualities!
But... you don't.
It sucks, but the Dunning-Kruger Effect is frighteningly universal. It's a devil's pitchfork two horrible, boldfaced **prongs**. First:
**Incompetent people grossly overestimate their own competence**
We already talked about that, right? You're a good programmer! Heck, you're a great programmer! You're "smart", so anything you don't know you can go look up if you need it! Right?
Welcome to incompetence.
The second prong is a bit longer, and has a barbed poison tip:
**Incompetent people fail to recognize genuine skill in others**
Both prongs are intrinsically funny when you're watching them in action in *someone else*, and they're incomprehensible when anyone tries to poke you with them. Not necessarily offensive, mind you: you *might* get offended if someone tries to imply that you're not as competent as you feel you are, but it's more likely (per the D-K effect) that you'll simply not believe them. Not even close. You're smart! They're just wrong! Gosh!
The second prong, that of the ability to recognize true competence, has major ramifications when we conduct interviews. That's what Joel was writing about in **Smart and Gets Things Done**, you know: conducting technical interviews.
How do you hire someone who's smarter than you? How do you *tell* if someone's smarter than you?
This is a problem I've thought about, over nearly twenty years of interviewing, and it appears that the answer is: you can't. You just have to get lucky.
It's easy for a candidate to *sound* smart. Anyone can use big jargon-y words and talk the talk like nobody's business, but I'm living, breathing proof that articulacy doesn't connote any other form of intelligence. Heck, the Markov-chain synopses of my blogs that people post in quasi-jest tend to look like I wrote them.
All too often I find myself on interview loops where the candidate knows a seemingly astounding amount about coding, computer science, software engineering, and general industry topics, but we find out at the last minute that they can't code Hello, World in any language.
This is, of course, one of the failure-patterns that Joel's **Get Things Done** clause is designed to catch. But interviews are conducted under pretty artificial conditions, and as a result they wind up being most effective at hiring people who are good at interviewing. This is a special breed of parrot, in a way. Interviewing isn't a particularly good predictor of performance, any more than your rank in a coding competition is a predictor of real-world performance. In fact, somewhat depressingly, there's almost no correlation whatsoever.
**If interviews suck, then why do we still do them this way?**
Lots of reasons. One is just history: everyone else does it that way. Companies tend to hire pretty similar HR departments, and HR tends to guide companies towards doing it the way everyone else does it. Same goes for technical management, which is all too often HR-driven as the company grows.
Another is that interviewing is already a fairly time-intrusive function for the interviewers, and it tends to be miserable for the interviewees. Trying to make the process somehow more rigorous or accurate just exacerbates these side effects.
Another is the "rite of passage" phenomenon, wherein engineers feel that if they had to go through the gauntlet, then everyone else should, too.
So for the most part, everyone does the same non-working variety of interviews, and hopes for the best.
As far as identifying good people goes, the best solution I've ever seen was at Geoworks, where you were required to do a six-month internship before you could get hired full-time. This seems to be the norm in non-tech departments at most tech companies. They often substitute "contractor" for "intern", but it works out roughly the same. Geoworks is the only company I've seen stick to their guns and make it mandatory for engineers.
However, I'm convinced that it only worked because Geoworks seeded the engineering staff with great people. The founding engineers set up a truly beautiful software engineering environment, with lots of focus on tools, mentoring, continuing education, "anarchy projects" to let off steam and encourage innovation, and a host of other goodnesses.
I've been a contractor at companies that had no good engineers at all, literally none whatsoever. A mandatory six-month internship at such companies would only serve to *lower* their average bar, since anybody competent would leave after the six months was up. This doesn't contradict the D/K Effect. It's easy to spot lackluster, soulless engineering organizations, and doing so doesn't imply that you're especially smart.
**The "Extended Interview"**
Anyway, I've often wondered where the Geoworks founders found such great engineers. The short answer is: "Berkeley", but I'm really looking for something deeper than that; namely: how did they *know*?
Along similar lines, I've long felt that Amazon's success was due in no small part to Bezos having seeded his tech staff with great engineers. World-class great. I don't know where or how he found them, since, again, how do you hire someone who's smarter than you? He's a brilliant guy, but his original choices (ex-[Lucid](http://en.wikipedia.org/wiki/Lucid_Inc.) folks, by and large) seem a stroke of blind luck that's hard to attribute to mere genius. I'll probably never know how it happened. Wish I did!
They weren't too big on software engineering, though, or more likely they all felt that time-to-market trumped everything else (and were correct, at least in their case), so Amazon is successful but lacks a high-quality software engineering culture. It's gotten much better over the years, of course, but it's a far cry from Geoworks. It's largely up to individual teams at Amazon to decide how much engineering to sprinkle into their coding. There's no central group (or distributed peer group) who can tell any team how to build their software. This is effectively mandated from the top, in fact.
But time-to-market is a pretty powerful business force. Maybe that's the missing link? Lucid was founded by Dick Gabriel, the "worse is better" guy, and Amazon took the "worse is better" idea (internally) to untold new extremes.
Dunno! But it sure worked for them.
Google has a process similar to the Geoworks 6-month internship idea. Geoworks's internship was a form of "extended interview", since it was obvious even in the 1980s that the classic interview format isn't a very good predictor of performance. At Google, you don't have to do an internship. However, unlike at most other companies, you're not "slotted" into a real position on the tech ladder until you've been on the job at least six months. During that time you need to prove that you can function at the level you were hired for, and if it's wrong in either direction your level is adjusted at slotting time.
The "extended interview" (in any form) is the *only* solution I've ever seen to the horrible dilemma, **How do you hire someone smarter than you?** Or even the simpler problem, How do you identify someone who's *actually* **Smart, and Gets Things Done**? Interviews alone just don't cut it.
Let me say it more directly, for those of you who haven't taken this personally yet: you can't do what Joel is asking you to do. You're not qualified. The **Smart and Gets Things Done** approach to interviewing will only get you copies of yourself, and the work of Dunning and Kruger implies that if you hire someone better than you are, then it's entirely accidental.
Don't get me wrong: you should still try! Don't throw the bath and baby away. **Smart and Gets Things Done** is a good weeder function to filter out some of the common "epic fail" types.
But realize that this approach has a cost: it will also filter out some people who are just as good as you, if not better, or even *way* better, along dimensions that are entirely invisible to you due to Dunning-Kruger forces mostly beyond your control.
So there's this related interviewing/hiring heuristic that I think may better approximate the kinds of people you really want to hire: **Done, and Gets Things Smart**.
I'll take Joel's cue and write it in **bold** so you know it's important. It's not **condescending** or anything. Really. Let's all recite it together, to make it catchy and stuff. Maybe we should have a little ditty for it, so it sticks in our heads annoyingly, forever. I think the Happy Birthday song will do nicely:
**Hap-py Birth-day To Yooooooou,**
**Done-and Geh-ets Things Smaaaart**
**Hmmm hmm HMMMMM Hmmmm Hmmmm whoeeeeeever**
**Done and geh-eeeeeets thiiiiiings Smaaaaaaaart** \*clap\* \*clap\*
There! You'll remember it every time anyone has a birthday.
**Done, and Gets Things Smart**
All gentle faux-condescension aside (or as the classroom jocks would have read, "fox condesomething"), Joel's **Smart, and Gets Things Done** heuristic seems really obvious to everyone. It has this magical "we're all smart in this together" appeal. But sadly, for the reasons I've outlined, almost everyone interprets it to mean "**Carbon Copy, of Myself**". Great guidance gone astray!
The **Done, and Gets Things Smart** approach is also a way of finding great people, but it recognizes that the Dunning-Kruger Effect requires some countermeasures. It's modeled on the early successes I've witnessed at Geoworks, Amazon, and Google, all of whom had one thing in common: they hired **brilliant seed engineers**. This boldface is really addictive when you get started on it!
They all managed to continue hiring great people afterwards, but the seed engineers were the most important. I'm hoping that this is intuitively obvious in much the same way that wanting smart people who get things done is intuitively obvious.
I think you really, *really* want great seed engineers, and that this is a different class of engineer from the "pretty darn good" engineer we typically hire using Joel's oft-misinterpreted advice.
Seed engineers. It's key. You can apply the "I want ideal seed engineers" rule recursively to an organization, a department, a project, a team, and even your officemates. We're not just talking about startups.
Let me ask you a brutally honest question: since you began interviewing, how many of the engineers you've voted thumbs-up on (i.e. "hire!"), are engineers you'd personally hire to work with you in your first startup company? Let's say this is a hypothetical company you're going to found someday when you have just a little more financial freedom and a great idea.
I posit that most of you, willing to admit it or not, have a lower bar for your current company than you would for your own personal startup company.
The people you'd want to be in your startup are *not* of the **Smart and Gets Things Done** variety.
For your startup (or, applying the recursion, for your new project at your current company), you don't want someone who's "smart". You're not looking for "eager to learn", "picks things up quickly", "proven track record of ramping up fast".
No! Screw that. You want someone who's superhumanly godlike. Someone who can teach *you* a bunch of stuff. Someone you admire and wish you could emulate, not someone who you think will admire and emulate you.
You want someone who, when you give them a project to research, will come in on Monday and say: "I'm **Done**, and by the way I improved the existing infrastructure while I was at it."
Someone who always seems to be finishing stuff so fast it makes your head spin. That's what my **Done** clause means. It means they're frigging done all the time.
I met my first **Done, and Gets Things Smart** engineers back at Geoworks. This was looong before I had any sort of a clue that I suck as an engineer, but these folks (Andrew Wilson and Chris Thomas, if you really must know) were *weird*. They never seemed to be working that hard, but they were not only 10x as productive as their peers, they also managed technical feats that were quite frankly too scary for anyone else. They could (as just one trait) dive in and learn new languages and make fixes to tools that the rest of us assumed were, I dunno, stuff you'd normally pay a vendor to fix. They dove into the hairiest depths of every code base they encountered and didn't just add features and make fixes; they waved some sort of magic wand and improved the system while they were in there: they would **Get Things Smart**. Make the systems smarter, that is. Sort of like getting your act together, but *they*'d do it for *your* code.
I've met many more such engineers along the way. They're out there. They're better than you. They were better twenty years ago than I am today or ever will be. Maybe it's natural ability. Maybe it's luck in education or upbringing. Maybe they have a secret recipe for improving rapidly and learning utter fearlessness. I don't know. But I've met 'em, and they aren't "smart". They're abso-flutely fugging brilliant.
You can't interview these people. For starters, they're not interested; these are the people that companies hold on to as long as humanly or companyly possible. The kinds of people that companies file lawsuits over when they're recruited away.
You can only find **Done, and Gets Things Smart** people in two ways, and one of them I still don't understand very well.
The first way is to get real lucky and have one as a coworker or classmate. You work with them for a few years and come to realize they're just cut from a finer cloth than you and your other unwashed cohorts. You may be friends with some of them, which helps with the recruiting a little, but not necessarily. The important thing is that you *recognize* them, even if you don't know what makes them tick.
This is the one great hope we programmers have for fighting the Dunning-Kruger Effect, the one hope we have for getting something better than the average "just like me" Solid Plugger Joe Nobody you pick up with the **Smart, and Gets Things Done** approach. Our only "out" is that working side-by-side with someone will show us clearly when they vastly outclass us.
Your devious little mind will come up with all sorts of rationalizations for why they're so damn good, so you can continue to think of yourself as **Smart, and Gets Things Done** material. You may conclude that they're just a genetic anomaly, and it's no fair even trying to compare yourself to someone who obviously has an unfair gift from the heavens. Or you may tell yourself that they're just a domain expert in various domains that you don't "need" right now. Or you may simply choose not to think about it too much. Good Old Compartmentalization to the rescue!
But working with them directly *will* show you when they're better. It's the only way. You'll gradually realize that your math deficiencies aren't just something that you might need to beef up on if you ever "need to"; you'll see that virtually every problem space has a mathematical modeling component that you were blissfully unaware of until **Done, and Gets Things Smart** gal points it out to you and says, "There's an infinitely smarter approach, which by the way I implemented over the weekend." You stare slack-jawed for a little while, and then she says: "Here's a ball. Why don't you go bounce it?"
These people aren't just pure gold; they're golden-egg-laying geese. They are the ones you want to bring with you to your own startup. Not the **Smart, and Gets Things Done, Just As Soon As I Read Up On The Subject, On The Company's Dime** riff-raff like you and me. No. They're your seed engineers: the ones who will make or break your company with both their initial technical output and the engineering-culture decisions they put into place — decisions that will largely determine how the company works for the next twenty years.
I've been debating whether to say this, since it'll smack vaguely of obsequiousness, but I've realized that one of the Google seed engineers (exactly one) is almost singlehandedly responsible for the amazing quality of Google's engineering culture. And I mean both in the sense of having established it, and also in the sense of keeping the wheel spinning. I won't name the person, and for the record he almost certainly loathes me, for reasons that are my own damn fault. But I'd hire him in a heartbeat: more evidence, I think, that the **Done, and Gets Things Smart** folks aren't necessarily your friends. They're just people you're lucky enough to have worked with.
At first it's entirely non-obvious who's responsible for Google's culture of engineering discipline: the design docs, audited code reviews, early design reviews, readability reviews, resisting introduction of new languages, unit testing and code coverage, profiling and performance testing, etc. You know. The whole gamut of processes and tools that quality engineering organizations use to ensure that code is open, readable, documented, and generally non-shoddy work.
But if you keep an eye on the emails that go out to Google's engineering staff, over time a pattern emerges: there's one superheroic dude who's keeping us all in line.
How do you interview for such a person? You can't! Everyone will tell you they're God's Gift to engineering quality. Everyone knows how to give it impressive lip service. Heck, there are lots of people who take it way too far and try to gridlock the organization in their over-enthusiasm, when what you really want is a balanced and pragmatic approach. I'd argue that it's virtually impossible to detect these "soft skills" in a classic interview setting, except to the extent that you're hiring your own clone, which according to our thesis here, is NOT what you want. You want **Done, and Gets Things Smart**: done faster than you, and made *your* system smarter!
I'm guessing that Google's founders worked with this person in school, enough to recognize his valuable talents. Hence they used Identification Approach #1: get lucky in who you work with.
Incidentally, they hired plenty of other brilliant seed engineers who were equally responsible for Google's great technical infrastructure. I'm just using this one guy as an illustrative example. But you really want the **Done, and Gets Things Smart** people on every team. If you could mix in one **Done, and Gets Things Smart** person with every five to ten **Smart, and Gets Things Done** people, then you'd be in good shape, since the latter, being "smart", can hopefully learn a lot from the former.
But when you're starting a company, or an organization, or a big project, the need for **Done, and Gets Things Smart** seed engineers is *desperate*. It's dire, in the sense that if you don't get the right seed people in place, you're dooming your organization to mediocrity, if you manage to succeed at all.
And it's direst when you're in a startup, because you can't pillage people from elsewhere in your organization who you know are good. And because **Done, and Gets Things Smart** people are worth their weight in refined plutonium, they're probably reasonably happy in their current position.
So let's assume you're looking at the vast ocean of programmers, all of whom are self-professed superstars who've gotten lots of "stuff" done, and you want to identify not the superstars, but the super-*heroes*.
How do you do it? Well, Brian Dougherty of Geoworks did it somehow. Jeff Bezos did it somehow. Larry and Sergey did it somehow. I'm willing to bet good money that *every* successful tech company out there had some freakishly good seed engineers. But a lot of company heads who make these decisions aren't necessarily industry programmers, and they still manage to find some world-class people in all the noise. So there must be a second way to identify **Done, and Gets Things Smart**.
I think Identification Approach #2, and this is the one I don't understand very well, is that you "ask around". You know. You manually perform the graph build-and-traversal done by the Facebook "Smartest Friend" plug-in, where you ask everyone to name the best engineer they know, and continue doing that until it converges.
I think this might work, assuming you have lots of connections initially (lots of roots for your graph), so you don't get stuck in some slummy local minimum.
I've seen companies go to university professors and ask them who their brightest students are; that's a variant of this approach, although it usually only turns up *future* **Done, and Gets Things Smart** engineers. Every once in a very rare while you'll get a recent college grad in this category, but I think more often they tend to be experienced enough to make Gandalf feel young again.
Because technical brilliance, seemingly superhuman productivity, and near-militaristic adherence to software discipline aren't enough. They also need leadership skills. They don't have to be *great* leaders; in fact in a pinch, just being bossy might work for a while, as long as they're bossing people in the right directions. But they need to have the ability to guide the organization (or new team, or whatever) in uniformly excellent directions, which requires *some* leadership, even if it's bad or amateurish leadership.
As much as I suspect Approach #2 may work, I think Approach #1 is probably more reliable. Take a closer look at your coworkers who are doing things that you "could learn if you ever need it". Read up on the old Dunning-Kruger Effect. I recommend it with irony dialed at 11, since personally I have yet to read more than the Wikipedia article and a few other articles here and there. I'll read it if I "need to". Psh.
**Done, and Gets Things Smart**
Not superstars: superheroes! People who are freakishly good at what they do. People who finish things so fast that they seem to have paranormal assistance. People who can take in any new system or design for all intents instantaneously, with no "ramp-up", and who can immediately bring insights to bear that are quite simply beyond your rustic abilities.
Those are the folks you want. I'm not going to tell you: "Don't settle for less." Far from it. You still want to hire the **Smart, and Gets Things Done** folks. But those folks have a long way to grow, and they probably have absolutely no idea just how far it is. So you want some **Done, and Gets Things Smart** people to guide them.
And now, to play us out...
Dooooone and Ge-ets Things Smart,
Done and Ge-ets Things Smart,
Done and Geeee-eeeets Thiiings Smaaaa-aaaart,
Done and Ge-ets Things Smart!

960
Reading/Read Later/Edulinq Jon Skeet's coding blog.md → Reading/Read Later/Edulinq – Jon Skeet's coding blog.md
View File

File diff suppressed because it is too large Load Diff

160
Reading/Done Reading/Embrace Complexity; Tighten Your Feedback Loops.md → Reading/Read Later/Embrace Complexity; Tighten Your Feedback Loops.md
View File

@@ -1,193 +1,287 @@
Clipped from: [https://ferd.ca/embrace-complexity-tighten-your-feedback-loops.html](https://ferd.ca/embrace-complexity-tighten-your-feedback-loops.html) [[ReadItLater]] [[Article]]
# [Embrace Complexity; Tighten Your Feedback Loops](https://ferd.ca/embrace-complexity-tighten-your-feedback-loops.html)
This post contains a transcript of the talk I wrote for and gave at [QCon New York 2023](https://qconnewyork.com/presentation/jun2023/embrace-complexity-tighten-your-feedback-loops) for [Vanessa Huerta Granda](https://qconnewyork.com/speakers/vanessahuertagranda)'s [track on resilience engineering](https://qconnewyork.com/track/jun2023/resilience-engineering-culture-system-requirement). This post contains a transcript of the talk I wrote for and gave at [QCon New York 2023](https://qconnewyork.com/presentation/jun2023/embrace-complexity-tighten-your-feedback-loops) for [Vanessa Huerta Granda](https://qconnewyork.com/speakers/vanessahuertagranda)'s [track on resilience engineering](https://qconnewyork.com/track/jun2023/resilience-engineering-culture-system-requirement).
The official talk title was "Embrace Complexity; Tighten Your Feedback Loops". That’s the descriptive title for the talk that follows the conference’s guidelines about good descriptive titles. Instead I decided to follow my gut feeling and go with what I think really explains my perspective and the approach I bring with me to work and even my life in general: The official talk title was "Embrace Complexity; Tighten Your Feedback Loops". That’s the descriptive title for the talk that follows the conference’s guidelines about good descriptive titles. Instead I decided to follow my gut feeling and go with what I think really explains my perspective and the approach I bring with me to work and even my life in general:
![Tkts Is ALL GOING To KELL ACL WE CAW DO IS (ÄFL(JENC€ Pow CT's TAKE frea Hebert keeps ](Exported%20image%2020240808113916-0.png) ![](Reading/Read%20Later/assets/01.png)
I take what would probably be a sardonic approach to dealing with life and systems, and so “This is all going to hell anyway” is pervasive to my approach. Things are going to be challenging. There are going to always be pressures that keep pushing our systems to the edge of chaos. I don’t think this can be fixed or avoided. Any improvement will be used to bring it right to that edge. In complex systems, the richness and variability is often there for a reason. Trying to stamp it out in favour of stronger control is likely to create weird issues. I take what would probably be a sardonic approach to dealing with life and systems, and so “This is all going to hell anyway” is pervasive to my approach. Things are going to be challenging. There are going to always be pressures that keep pushing our systems to the edge of chaos. I don’t think this can be fixed or avoided. Any improvement will be used to bring it right to that edge. In complex systems, the richness and variability is often there for a reason. Trying to stamp it out in favour of stronger control is likely to create weird issues.
So the best I personally hope for is to have some limited influence in steering things the best I can to delay going to hell as long as possible, but that’s it. And my talk is going to focus on a lot of these approaches, but first, I want to explain why I feel things are that way. So the best I personally hope for is to have some limited influence in steering things the best I can to delay going to hell as long as possible, but that’s it. And my talk is going to focus on a lot of these approaches, but first, I want to explain why I feel things are that way.
![OFF TE Figure 2. (Color online) Process Map From: - OF -tl-e ](Exported%20image%2020240808113916-1.png) ![](Reading/Read%20Later/assets/02.png)
In what is probably my favorite paper ever, titled [Moving Off The Map](https://ferd.ca/notes/paper-moving-off-the-map.html), Ruthanne Huising ran ethnological studies by embedding herself into projects within many large corporations doing planned organizational changes. In supporting these efforts, they were doing “tracing” of their functions, which meant gathering a lot of data about what activities take place, what interactions and hand-offs exist, what information and tools are used and required? How long do tasks take? How do people and teams deal with errors? Generally asking the question “what do we do here?” and wondering with whom they do it. In what is probably my favorite paper ever, titled [Moving Off The Map](https://ferd.ca/notes/paper-moving-off-the-map.html), Ruthanne Huising ran ethnological studies by embedding herself into projects within many large corporations doing planned organizational changes. In supporting these efforts, they were doing “tracing” of their functions, which meant gathering a lot of data about what activities take place, what interactions and hand-offs exist, what information and tools are used and required? How long do tasks take? How do people and teams deal with errors? Generally asking the question “what do we do here?” and wondering with whom they do it.
To build these maps they generally reached out to experts within the organization who were supposed to know how things were working. Even then, they were really surprised. To build these maps they generally reached out to experts within the organization who were supposed to know how things were working. Even then, they were really surprised.
![Figure 2. (Color online) Process Map OFF TE k(it was (ike the sun rose for the first time... r sau the bigger picture." k(Tke proble%f is that it was not designed in the first place." k(Tkis is even Biore fucked up than r imagined." ](Exported%20image%2020240808113916-2.png) ![](Reading/Read%20Later/assets/03.png)
One explained that “it was like the sun rose for the first time… I saw the bigger picture.” Participants had never seen the pieces (jobs, technologies, tools, and routines) connected in one place, and they realized that their prior view was narrow and fractured, despite being considered experts. One explained that “it was like the sun rose for the first time… I saw the bigger picture.” Participants had never seen the pieces (jobs, technologies, tools, and routines) connected in one place, and they realized that their prior view was narrow and fractured, despite being considered experts.
Others would state that “the problem is that it was not designed in the first place.” The system was not designed nor coordinated, but generally showed the result of various parts of the organization making their own decisions, solving local problems, and adapting in a decentralized manner. Others would state that “the problem is that it was not designed in the first place.” The system was not designed nor coordinated, but generally showed the result of various parts of the organization making their own decisions, solving local problems, and adapting in a decentralized manner.
The last quote comes from events when a manager at one of the organizations walked the CEO through the map, highlighting the lack of design and the disconnect between strategy and operations. The CEO sat down, put his head on the table, and said, “This is even more fucked up than I imagined.” He realized that the operation of his organization was out of his control, and that his grasp on it was imaginary. The last quote comes from events when a manager at one of the organizations walked the CEO through the map, highlighting the lack of design and the disconnect between strategy and operations. The CEO sat down, put his head on the table, and said, “This is even more fucked up than I imagined.” He realized that the operation of his organization was out of his control, and that his grasp on it was imaginary.
![CONTROL Pgoc.ESs COSTS SAVINGS I LLUSoRY To coczECT€b DATA CREhTED nap ](Exported%20image%2020240808113916-3.png) ![](Reading/Read%20Later/assets/04.png)
One of the most surprising results reported in there was about tracking the people who participated in organizing and running the change projects, and seeing who got promoted, who left, and who moved around the org or industry they were in. One of the most surprising results reported in there was about tracking the people who participated in organizing and running the change projects, and seeing who got promoted, who left, and who moved around the org or industry they were in.
She found out there were two main types of outcome. The first group turned out to be filled with people who got promotions. They were mostly folks who worked in communications, training, who managed the costs and savings of the projects, or those who helped do process design. Follow-up interviews revealed that most of them attributed their promotions to having worked on a big project to put under their belt, and to frequently working with higher-ups, which both helped with getting promoted.
Another group however mostly contained people who moved to the periphery: away from core roles at the organization, sometimes becoming consultants, or leaving altogether. Those who fit this category happened to be the people who collected the data and created the map. They attributed their moves to either feeling like they finally understood the organization better, felt more empowered to change things, or became so alienated by the results they wanted to get out.
So the question of course became how come people who feel they understand how the organization truly works and who want to change it move _away_ from the central roles and positions, and into the peripheral ones?
!["FATAL «organizations and institutions exist on(g actual people's doings and that these are necessari(g particular, local and epkæeral" «socia( worlds do not have independent, stable existence but instead emerge from our collective action" «t.Uitkin the system of roles, rules, and routines, there is far %tore room to htaneuver than previous(g assumed." ](Exported%20image%2020240808113916-4.png) She found out there were two main types of outcome. The first group turned out to be filled with people who got promotions. They were mostly folks who worked in communications, training, who managed the costs and savings of the projects, or those who helped do process design. Follow-up interviews revealed that most of them attributed their promotions to having worked on a big project to put under their belt, and to frequently working with higher-ups, which both helped with getting promoted.
Another group however mostly contained people who moved to the periphery: away from core roles at the organization, sometimes becoming consultants, or leaving altogether. Those who fit this category happened to be the people who collected the data and created the map. They attributed their moves to either feeling like they finally understood the organization better, felt more empowered to change things, or became so alienated by the results they wanted to get out.
So the question of course became how come people who feel they understand how the organization truly works and who want to change it move *away* from the central roles and positions, and into the peripheral ones?
![](Reading/Read%20Later/assets/05.png)
The fatal insight, according to Huising, is something sociologists knew for a good while: the culture and the order imposed to organizations, groups, and even societies is often emergent and negotiated. And while it's obvious that these structures dictate a lot of actions, the actions themselves can preserve or change the structures around them. The fatal insight, according to Huising, is something sociologists knew for a good while: the culture and the order imposed to organizations, groups, and even societies is often emergent and negotiated. And while it's obvious that these structures dictate a lot of actions, the actions themselves can preserve or change the structures around them.
The feelings of empowerment and alienation come in no small part because people realized that they could change a lot more than they could, albeit often from outside the core decision-making that enforces the structure (while understanding how that core works), or because the ways they thought they were impacting things was shown not to be effective and they felt disembedding. The feelings of empowerment and alienation come in no small part because people realized that they could change a lot more than they could, albeit often from outside the core decision-making that enforces the structure (while understanding how that core works), or because the ways they thought they were impacting things was shown not to be effective and they felt disembedding.
![NOetstNAL Vs. 0-0-0 0—0-0 o o o ](Exported%20image%2020240808113916-5.png) ![](Reading/Read%20Later/assets/06.png)
Another thing you have possibly experienced and isn’t in the paper now is one of differentiating between the nominal and actual structure of the org, the emergent one that depends on power dynamics, who knows what or whom, who likes or dislikes each other, and so on. Another thing you have possibly experienced and isn’t in the paper now is one of differentiating between the nominal and actual structure of the org, the emergent one that depends on power dynamics, who knows what or whom, who likes or dislikes each other, and so on.
If you've ever worked in a flat organization, like the one in the middle here, is that even though you have little management structure to speak of, power dynamics and decision-making authority still exists. People who have no power attached to their role are still going to be consulted or inserted in the decision-making flow of the organization, they're still going to be influential and have the ability to make or break projects, but just with less obvious accountability. If you've ever worked in a flat organization, like the one in the middle here, is that even though you have little management structure to speak of, power dynamics and decision-making authority still exists. People who have no power attached to their role are still going to be consulted or inserted in the decision-making flow of the organization, they're still going to be influential and have the ability to make or break projects, but just with less obvious accountability.
The nominal structure is the one where each level of management and within the organizational ladder specifies how information flows, and how authority is applied. It's what we see on the left in a more traditional org structure, and this way of organizing groups will simultaneously be useful to align efforts and to constrain them. It makes accountability more explicit and transparent, but structurally will prevent people from doing unspecified things, whether they would be harmful or useful. The nominal structure is the one where each level of management and within the organizational ladder specifies how information flows, and how authority is applied. It's what we see on the left in a more traditional org structure, and this way of organizing groups will simultaneously be useful to align efforts and to constrain them. It makes accountability more explicit and transparent, but structurally will prevent people from doing unspecified things, whether they would be harmful or useful.
The emergent structure is always there as well. It is implicit, always changing, and not necessarily constrained to your own organization either. Sometimes, people who know how to run, maintain, or operate components, or whom people listen to, are not even in your org anymore. They might have moved away (to a different team or even a competitor), retired, or never been in and they have just published a really influential piece of media and people look up to them. The emergent structure is always there as well. It is implicit, always changing, and not necessarily constrained to your own organization either. Sometimes, people who know how to run, maintain, or operate components, or whom people listen to, are not even in your org anymore. They might have moved away (to a different team or even a competitor), retired, or never been in and they have just published a really influential piece of media and people look up to them.
But who knows what, works with whom, and who can move things around in specific contexts can be key to successful initiatives. Even if the organizational structure has often been put in place to constrain change, as a barrier to people working in mis-aligned ways, some folks central to the emergent structure, in key contexts, have earned enough trust to be allowed tacitly to bend and break the rules. They can choose not to enforce the rules, or the rules are not enforced as tightly for them with the hopes of positive outcomes—even if sometimes it can get you the opposite result. But who knows what, works with whom, and who can move things around in specific contexts can be key to successful initiatives. Even if the organizational structure has often been put in place to constrain change, as a barrier to people working in mis-aligned ways, some folks central to the emergent structure, in key contexts, have earned enough trust to be allowed tacitly to bend and break the rules. They can choose not to enforce the rules, or the rules are not enforced as tightly for them with the hopes of positive outcomes—even if sometimes it can get you the opposite result.
I’m not here to argue in favor of one or the other structure, but mostly that in my experience, driving change or making initiatives succeeds the most when catering to both structures at once, or rather fails when only looking at one and being blocked by the other. They're both real, both distinct, and pretending only either exists is bound to cause you grief. I’m not here to argue in favor of one or the other structure, but mostly that in my experience, driving change or making initiatives succeeds the most when catering to both structures at once, or rather fails when only looking at one and being blocked by the other. They're both real, both distinct, and pretending only either exists is bound to cause you grief.
![THE GAP ETwEEN WORK-AS-... JRK- ms- IMAGINED unRk-AS-DscLSED WORK-AS-DONE ](Exported%20image%2020240808113916-6.png) ![](Reading/Read%20Later/assets/07.png)
As a continuation of this, the way people work every day is often different from the way people around them imagine their work is being done. The gap between how work is thought to be done and how it is actually done is a major but generally invisible factor in how systems work out. As a continuation of this, the way people work every day is often different from the way people around them imagine their work is being done. The gap between how work is thought to be done and how it is actually done is a major but generally invisible factor in how systems work out.
Based on flawed mental models of the work, procedures and prescriptions are given about how to do work, and will vary in inaccuracy. People will imagine things like, for example, writing all the tests before writing or modifying any code and that code coverage could be ideal and then that it will all be reviewed in depth by an expert, and will enshrine this as a policy. Based on flawed mental models of the work, procedures and prescriptions are given about how to do work, and will vary in inaccuracy. People will imagine things like, for example, writing all the tests before writing or modifying any code and that code coverage could be ideal and then that it will all be reviewed in depth by an expert, and will enshrine this as a policy.
But the application of these policies is never perfect. Sometimes code doesn't have an owner, or due to crunch time and based on how much the reviewer and author trust each other, the review won't be as in-depth as expected. But the application of these policies is never perfect. Sometimes code doesn't have an owner, or due to crunch time and based on how much the reviewer and author trust each other, the review won't be as in-depth as expected.
When you see this mismatch causing people to ignore or bend rules, you can choose to apply authority and ask for a stricter rule-following. This pattern of enforcing the rules harder will likely drive these adaptations underground rather than stamping them out, because real constraints drive that behavior. When you see this mismatch causing people to ignore or bend rules, you can choose to apply authority and ask for a stricter rule-following. This pattern of enforcing the rules harder will likely drive these adaptations underground rather than stamping them out, because real constraints drive that behavior.
In turn, the work as disclosed will be less adequate, and the work as imagined progressively gets worse and worse. In turn, the work as disclosed will be less adequate, and the work as imagined progressively gets worse and worse.
This becomes a feedback loop of misunderstanding and at some point, like our devastated CEO, you’re not managing the real world anymore. This becomes a feedback loop of misunderstanding and at some point, like our devastated CEO, you’re not managing the real world anymore.
![Fr Hebert mononcqc@hachyderm.io If you're a software developer who ever worked for an employer who had you track time hourly into specific projects/customer accounts and you were short on time budget, did you: ESCRO 13K¯ work for free/untracked/stopped work¯ 13% enter time in unrelated projects with more buffer 35% enter time in the same project regardlesE 58% - my time tracking was always fake and lies Refresh • 120 people • Closed Jan 21, 2023, 14:05 web ](Exported%20image%2020240808113916-7.png) ![](Reading/Read%20Later/assets/08.png)
To demonstrate this, earlier this year I went to my local mastodon network—so you know this is super scientific—and ran a poll about time sheets. The question was "If you're a software developer who ever worked for an employer who had you track your time hourly into specific projects/customer accounts and you were short on time budget, did you..." To demonstrate this, earlier this year I went to my local mastodon network—so you know this is super scientific—and ran a poll about time sheets. The question was "If you're a software developer who ever worked for an employer who had you track your time hourly into specific projects/customer accounts and you were short on time budget, did you..."
Multiple answers were accepted. Fewer than 15% of people either stopped work, worked without tracking their time anymore (for free), or shifted their time into other projects with more buffer space. Multiple answers were accepted. Fewer than 15% of people either stopped work, worked without tracking their time anymore (for free), or shifted their time into other projects with more buffer space.
Roughly a third of people reported billing anyway, some stating that it's not their problem the time allocation wasn't realistic or adequate. Roughly a third of people reported billing anyway, some stating that it's not their problem the time allocation wasn't realistic or adequate.
But the vast majority of answers, nearly 60%, came from people saying "my time tracking was always fake and lies," with some people stating they even wrote applications to generate realistic-looking time sheets. But the vast majority of answers, nearly 60%, came from people saying "my time tracking was always fake and lies," with some people stating they even wrote applications to generate realistic-looking time sheets.
What we can see here is an example of how work-as-imagined gets translated into policies ("people do their work in projects, and account for their time"), which at some point doesn't get applied right anymore. If I were to suppose, it could be things like not being allowed to go over time, or just finding the practice useless. But the end result is that the time sheet data just isn't trustworthy, and then it can get used again and again in further decision making. What we can see here is an example of how work-as-imagined gets translated into policies ("people do their work in projects, and account for their time"), which at some point doesn't get applied right anymore. If I were to suppose, it could be things like not being allowed to go over time, or just finding the practice useless. But the end result is that the time sheet data just isn't trustworthy, and then it can get used again and again in further decision making.
The gap widens, and our CEO might also get to think "this is all fucked up." The gap widens, and our CEO might also get to think "this is all fucked up."
![PRESSURES AO CONFLICTS WoeKLoAD RISK TRuST SOCCESS FAILURE ](Exported%20image%2020240808113916-8.png) ![](Reading/Read%20Later/assets/09.png)
Part of the reason for this is that every day decisions are made by trying to deal with all sorts of pressures coming from the workplace, which includes the values communicated both as spoken and as acted out. People generally want to do a good job and they’ll try to balance these conflicting values and pressures as well as they can. Part of the reason for this is that every day decisions are made by trying to deal with all sorts of pressures coming from the workplace, which includes the values communicated both as spoken and as acted out. People generally want to do a good job and they’ll try to balance these conflicting values and pressures as well as they can.
The outcome of that trade-off being a success or a failure isn’t known ahead of time, but these small decisions accumulate based on the feedback we get from each of these and can end up compounding and accumulating, either as improvements, or as erosion that makes organizations more brittle, or really anywhere in between. People adopt the organization’s constraints as their own, and this set of pressures is the kind of stuff that drives processes to the edge of chaos over and over again. The outcome of that trade-off being a success or a failure isn’t known ahead of time, but these small decisions accumulate based on the feedback we get from each of these and can end up compounding and accumulating, either as improvements, or as erosion that makes organizations more brittle, or really anywhere in between. People adopt the organization’s constraints as their own, and this set of pressures is the kind of stuff that drives processes to the edge of chaos over and over again.
These accumulations of small decisions, these continuous negotiations, that’s one way your culture can define itself. Small common everyday acts and small amounts of social pressure you can apply locally has an impact, as minor as it might be, and compounds. You can easily foster your own local counterculture within a team if you want to. This can both be good (say in Skunkworks where you bypass a structure to do important work) or bad (normalizing behaviors that are counterproductive and can create conflict). These accumulations of small decisions, these continuous negotiations, that’s one way your culture can define itself. Small common everyday acts and small amounts of social pressure you can apply locally has an impact, as minor as it might be, and compounds. You can easily foster your own local counterculture within a team if you want to. This can both be good (say in Skunkworks where you bypass a structure to do important work) or bad (normalizing behaviors that are counterproductive and can create conflict).
![Mow To EMBRACE GMPLFXITY? QADE-OFFs FVk1kJG KEEP EEDBACk LOPS HAC— — H OVER ](Exported%20image%2020240808113916-9.png) ![](Reading/Read%20Later/assets/10.png)
So while a lot of the work you can do to improve reliability or resilience as a whole can be driven locally, my experience is that you nevertheless get the best results by also aligning with or re-aligning some of the organizational pressures and values usually set from above. So while a lot of the work you can do to improve reliability or resilience as a whole can be driven locally, my experience is that you nevertheless get the best results by also aligning with or re-aligning some of the organizational pressures and values usually set from above.
The idea here is to start looking at the organization from both ends: how can we support the people dealing with the trade-offs in conflicting goals as they happen, how can we influence the higher-level values and pressures such that we can try to reduce how often these conflicts happen even though they will definitely keep happening, and how can we better carry context and feedback across both ends so that we constantly adjust as best as we can. A system perspective on interactions, rather than focusing on components is also something I've found useful. The rest of the talk is going to be spent on these ideas.
_(as a note, the third drawing is_ _Dimethylmercury__, a highly volatile, reactive, flammable, and colorless liquid. It's one of the strongest known neurotoxins, and less than 0.1 mL is enough to kill you through your skin, and gloves apparently do a bad job at protecting you)_
![Mow To EMBRACE GMPLFXITY? QADE-OFFs FVk1kJG KEEP EEDBACk LOPS HAC— — H OVER ](Exported%20image%2020240808113916-10.png) The idea here is to start looking at the organization from both ends: how can we support the people dealing with the trade-offs in conflicting goals as they happen, how can we influence the higher-level values and pressures such that we can try to reduce how often these conflicts happen even though they will definitely keep happening, and how can we better carry context and feedback across both ends so that we constantly adjust as best as we can. A system perspective on interactions, rather than focusing on components is also something I've found useful. The rest of the talk is going to be spent on these ideas.
*(as a note, the third drawing is [Dimethylmercury](https://en.wikipedia.org/wiki/Dimethylmercury), a highly volatile, reactive, flammable, and colorless liquid. It's one of the strongest known neurotoxins, and less than 0.1 mL is enough to kill you through your skin, and gloves apparently do a bad job at protecting you)*
![](Reading/Read%20Later/assets/11.png)
So let's start with negotiating trade-offs, with a bit more of an ops-y perspective, because that's where I'm coming from. So let's start with negotiating trade-offs, with a bit more of an ops-y perspective, because that's where I'm coming from.
![Dou'T DELWEÄ WHAT Asm To ](Exported%20image%2020240808113916-11.png) ![](Reading/Read%20Later/assets/12.png)
This is a painful one sometimes, especially when you have highly professional people who take their jobs seriously. This is a painful one sometimes, especially when you have highly professional people who take their jobs seriously.
Locally for you as a DevOps or SRE team, there is a need for the awareness of what the organization and customers actually care about. Some availability targets become useless metrics because they’re disconnected from what users want, and you’re just going to burn people out doing it. Locally for you as a DevOps or SRE team, there is a need for the awareness of what the organization and customers actually care about. Some availability targets become useless metrics because they’re disconnected from what users want, and you’re just going to burn people out doing it.
I learned this lesson when talking to the SRE manager of one of these websites where people pick their favorite images, put them on boards, and get shown ads. He was telling me how their site was having a lot of reliability issues. It would keep going down, his team would do heroics to bring it back up, and it'd open all over again. I learned this lesson when talking to the SRE manager of one of these websites where people pick their favorite images, put them on boards, and get shown ads. He was telling me how their site was having a lot of reliability issues. It would keep going down, his team would do heroics to bring it back up, and it'd open all over again.
He felt his team was burning out. They were losing people, and their call rotation was so painful they were also having issues hiring back into it. He was seeing the death spiral happening and was wondering what to do. He felt his team was burning out. They were losing people, and their call rotation was so painful they were also having issues hiring back into it. He was seeing the death spiral happening and was wondering what to do.
He added that there were perverse incentives at play: every time the site went down, they stopped showing images, but not ads. That meant that during incidents, they still earned money, but no longer paid for bandwidth. The site was more profitable when it failed than when it worked, and seemingly, users didn't mind much. He added that there were perverse incentives at play: every time the site went down, they stopped showing images, but not ads. That meant that during incidents, they still earned money, but no longer paid for bandwidth. The site was more profitable when it failed than when it worked, and seemingly, users didn't mind much.
They were not getting help, nobody seemed to consider it a problem. Not really knowing what to say, I just asked off-hand: "are you trying to deliver more reliability than people are asking for? What if you just stopped and let it burn more and rested your people?" He thought about it seriously, and said "yeah, maybe." They were not getting help, nobody seemed to consider it a problem. Not really knowing what to say, I just asked off-hand: "are you trying to deliver more reliability than people are asking for? What if you just stopped and let it burn more and rested your people?" He thought about it seriously, and said "yeah, maybe."
I never actually found out what happened after this, but it still stuck with me as a really good question to ask from time to time. I never actually found out what happened after this, but it still stuck with me as a really good question to ask from time to time.
![DoucT DELWEP- WHAT To ](Exported%20image%2020240808113916-12.png) ![](Reading/Read%20Later/assets/13.png)
In some cases, the answer will be "yes, we want to be this reliable". But you just won't be given the right tools to do it. In some cases, the answer will be "yes, we want to be this reliable". But you just won't be given the right tools to do it.
At Honeycomb, we want on-call rotations to have 5-8 people on them because that’s what we think gives a good pace that maintains a balance between how rested and how out-of-practice people can be. Not too often nor not often enough. At Honeycomb, we want on-call rotations to have 5-8 people on them because that’s what we think gives a good pace that maintains a balance between how rested and how out-of-practice people can be. Not too often nor not often enough.
But many services are owned by smaller teams of 3-4 people. If we wanted rotations to be made of people who know all their components in depth, where they could build expertise and operate what they wrote, we couldn't reach a sustainable frequency. But many services are owned by smaller teams of 3-4 people. If we wanted rotations to be made of people who know all their components in depth, where they could build expertise and operate what they wrote, we couldn't reach a sustainable frequency.
Instead, to keep the pace right, we tend to put together rotations made of multiple teams, for which people won’t understand many of the components they operate. This in turn makes us prepare to deal with more unknown: fewer runbooks, more high-level switches and manual circuit breakers to gracefully degrade parts of the system to keep it running off-hours, and with different patterns of escalation. Instead, to keep the pace right, we tend to put together rotations made of multiple teams, for which people won’t understand many of the components they operate. This in turn makes us prepare to deal with more unknown: fewer runbooks, more high-level switches and manual circuit breakers to gracefully degrade parts of the system to keep it running off-hours, and with different patterns of escalation.
We started leaning more heavily on this when a big public product launch required shipping a new feature, which was to be operated by a team that didn't have full time to get it operationally ready. When our SRE team was discussing with them what still needed to be done, we asked for a few simple things: a way to switch the feature off for a single customer, and a way to turn it off entirely, that wouldn't break the rest of the product. The rest we could add as we went. We started leaning more heavily on this when a big public product launch required shipping a new feature, which was to be operated by a team that didn't have full time to get it operationally ready. When our SRE team was discussing with them what still needed to be done, we asked for a few simple things: a way to switch the feature off for a single customer, and a way to turn it off entirely, that wouldn't break the rest of the product. The rest we could add as we went.
We ended up using these switches a few times, one of which prevented a surprising write-amplification bug that could have killed the whole system, and instead let us wait a few hours for the code owners to get up and fix it at a leisurely pace. We're going to accept a bit of well-scoped, partial unavailability—something that happens a lot in large distributed systems—in order to keep the system stable. We ended up using these switches a few times, one of which prevented a surprising write-amplification bug that could have killed the whole system, and instead let us wait a few hours for the code owners to get up and fix it at a leisurely pace. We're going to accept a bit of well-scoped, partial unavailability—something that happens a lot in large distributed systems—in order to keep the system stable.
The person wearing the pager often does triage and that weird issues will eventually be handled by code owners, just not right now. The person wearing the pager often does triage and that weird issues will eventually be handled by code owners, just not right now.
This approach means that rather than working impossible hours and making inhuman efforts foreseeing the unforeseeable, we keep moving rather fast, gather feedback, find issues, and turn around a bit more on a dime. In order to do this though, there’s a general understanding that production issues may turn parts of the roadmap upside down, that escalations outside of the call rotation can disrupt project work, and so on. This approach means that rather than working impossible hours and making inhuman efforts foreseeing the unforeseeable, we keep moving rather fast, gather feedback, find issues, and turn around a bit more on a dime. In order to do this though, there’s a general understanding that production issues may turn parts of the roadmap upside down, that escalations outside of the call rotation can disrupt project work, and so on.
That’s one of the complex trade-offs we can make between staffing, training/onboarding, capacity planning, iterative development, testing approaches, operations, roadmap, and feature delivery. And you know, for some parts of our infra we make different decisions because the consequences and mechanisms differ. That’s one of the complex trade-offs we can make between staffing, training/onboarding, capacity planning, iterative development, testing approaches, operations, roadmap, and feature delivery. And you know, for some parts of our infra we make different decisions because the consequences and mechanisms differ.
![THERE ARE NO SUbSTtT(ffES Foe SAFETY ](Exported%20image%2020240808113916-13.png) ![](Reading/Read%20Later/assets/14.png)
To make these tricky decisions, you have to be able to bring up these constraints, these challenges, and have them be discussed openly without a repression that forces them underground. To make these tricky decisions, you have to be able to bring up these constraints, these challenges, and have them be discussed openly without a repression that forces them underground.
One of my favorite examples is from a prior job, where one of my first mandates was to try and help with their reliability story. We went over 30 or so incident reports that had been written over the previous year, and a pattern that quickly came up was how many reports mentioned "lack of tests" (or lack of good tests) as causes, and had "adding tests" in action items. One of my favorite examples is from a prior job, where one of my first mandates was to try and help with their reliability story. We went over 30 or so incident reports that had been written over the previous year, and a pattern that quickly came up was how many reports mentioned "lack of tests" (or lack of good tests) as causes, and had "adding tests" in action items.
By looking at the overall list, our initial diagnosis was that testing practices were challenging. We thought of improving the ergonomics around tests (making them faster) and to also provide training in better ways to test. But then we had another incident where the review reported tests as an issue, so I decided to jump in. By looking at the overall list, our initial diagnosis was that testing practices were challenging. We thought of improving the ergonomics around tests (making them faster) and to also provide training in better ways to test. But then we had another incident where the review reported tests as an issue, so I decided to jump in.
I reached out to the engineers in question and asked about what made them feel like they had enough tests. I said that we often write tests up until the point we feel they're not adding much anymore, and that I was wondering what they were looking at, what made them feel like they had reached the points where they had enough tests. They just told me directly that they knew they didn't have enough tests. In fact, they knew that the code was buggy. But they felt in general that it was safer to be on-time with a broken project than late with a working one. They were afraid that being late would put them in trouble and have someone yell at them for not doing a good job. I reached out to the engineers in question and asked about what made them feel like they had enough tests. I said that we often write tests up until the point we feel they're not adding much anymore, and that I was wondering what they were looking at, what made them feel like they had reached the points where they had enough tests. They just told me directly that they knew they didn't have enough tests. In fact, they knew that the code was buggy. But they felt in general that it was safer to be on-time with a broken project than late with a working one. They were afraid that being late would put them in trouble and have someone yell at them for not doing a good job.
When I went up to upper management, they absolutely believed that engineers were empowered and should feel safe pressing a big red button that stopped feature work if they thought their code wasn't ready. The engineers on that team felt that while this is what they were being told, in practice they'd still get in trouble. When I went up to upper management, they absolutely believed that engineers were empowered and should feel safe pressing a big red button that stopped feature work if they thought their code wasn't ready. The engineers on that team felt that while this is what they were being told, in practice they'd still get in trouble.
There's no amount of test training that would fix this sort of issue. The engineers knew they didn't have enough tests and they were making that tradeoff willingly. There's no amount of test training that would fix this sort of issue. The engineers knew they didn't have enough tests and they were making that tradeoff willingly.
![THERE so NON-TECHNICAL WAYs Fop THINGS To GO (T ts SOt0ETlES åkAY foe TO TRADE OF THAT RI Sk ](Exported%20image%2020240808113916-14.png) ![](Reading/Read%20Later/assets/15.png)
*(note: this slide was cut from the presentation since I was short on time)*
_(note: this slide was cut from the presentation since I was short on time)_
Speaking of which, sometimes it’s also fine to drop reliability because there are bigger systemic threats. Speaking of which, sometimes it’s also fine to drop reliability because there are bigger systemic threats.
Sometimes you can eat downtime or degraded service because it’s going to keep your workload manageable and people from burning out. or maybe you take a hit because a big customer that makes you hit your targets as an org and can prevent layoffs will put some things over the limit and a component’s performance will suffer. You can’t be the department of “no” and that negotiation has to be done across departments. Sometimes you can eat downtime or degraded service because it’s going to keep your workload manageable and people from burning out. or maybe you take a hit because a big customer that makes you hit your targets as an org and can prevent layoffs will put some things over the limit and a component’s performance will suffer. You can’t be the department of “no” and that negotiation has to be done across departments.
Conversely however, you have to be able to call out when your teams are strained, when targets aren’t being met and customers are complaining about it. It means you might be right, and some deadlines or feature delivery could be deferred to make room for others. Conversely however, you have to be able to call out when your teams are strained, when targets aren’t being met and customers are complaining about it. It means you might be right, and some deadlines or feature delivery could be deferred to make room for others.
How do you deal with capacity planning when making your biggest customer renew their contract prevents you from signing up another one that’s as big? Very carefully, by talking it out by all the involved people. How do you deal with capacity planning when making your biggest customer renew their contract prevents you from signing up another one that’s as big? Very carefully, by talking it out by all the involved people.
And sometimes that trade-off is very reasonable. And good engineering requires you to move it earlier in the lifecycle of software than just around incidents. It’s much simpler to change the shape of a product’s features than it is to deliver the perfect distributed systems sometimes. Making your features take the ideal shape to deal with the reality of physics is one of the things a good collaborative approach can facilitate. And sometimes that trade-off is very reasonable. And good engineering requires you to move it earlier in the lifecycle of software than just around incidents. It’s much simpler to change the shape of a product’s features than it is to deliver the perfect distributed systems sometimes. Making your features take the ideal shape to deal with the reality of physics is one of the things a good collaborative approach can facilitate.
![Mow To EMBRACE COMPLEXITY? QADE-OFFs FVk1kJG KEEP EEDBACk LOPS HAC— — H OVER ](Exported%20image%2020240808113916-15.png) ![](Reading/Read%20Later/assets/16.png)
So we can make tradeoff negotiation simpler by having these honest discussions, but in many cases this ability to discuss constraints to influence how work takes place brings us to this next step, where we don’t only influence the decisions people make, but surface these challenges to influence how the organization applies its pressures. This is moving from the local level to the alignment to the broader org structure. So we can make tradeoff negotiation simpler by having these honest discussions, but in many cases this ability to discuss constraints to influence how work takes place brings us to this next step, where we don’t only influence the decisions people make, but surface these challenges to influence how the organization applies its pressures. This is moving from the local level to the alignment to the broader org structure.
![METRICS APE TREE To You, BJOT YW To SERVE THOA ](Exported%20image%2020240808113916-16.png) ![](Reading/Read%20Later/assets/17.png)
Metrics are good to direct your attention and confirm hypotheses, but not as a target, and they’re unlikely to be good for insights. [They’re compression, and it can be unreliable](https://ferd.ca/plato-s-dashboards.html). Metrics are good to direct your attention and confirm hypotheses, but not as a target, and they’re unlikely to be good for insights. [They’re compression, and it can be unreliable](https://ferd.ca/plato-s-dashboards.html).
The thing you generally care about is your customer or user's satisfaction, but there's a limit to how many times you can ask "would you recommend us to a friend?" and still get a good signal. So you start picking a surrogate variable. The thing you generally care about is your customer or user's satisfaction, but there's a limit to how many times you can ask "would you recommend us to a friend?" and still get a good signal. So you start picking a surrogate variable.
You assume that when the site is down and slow, people are mad, and you make being up and fast a proxy for satisfaction. But then that signal is a bit messy and not super actionable, because it can include user devices or bits of the network you don't control, plus it's hard to measure, so you'll settle for response time at the edge of your infrastructure. This loses fidelity into the signal, but it'll get worse as you suddenly find some teams have more data than others, and they use features differently, so you either need a ton of alarms or fewer messier ones, but you're getting further and further away from whether people are actually satisfied. You assume that when the site is down and slow, people are mad, and you make being up and fast a proxy for satisfaction. But then that signal is a bit messy and not super actionable, because it can include user devices or bits of the network you don't control, plus it's hard to measure, so you'll settle for response time at the edge of your infrastructure. This loses fidelity into the signal, but it'll get worse as you suddenly find some teams have more data than others, and they use features differently, so you either need a ton of alarms or fewer messier ones, but you're getting further and further away from whether people are actually satisfied.
This loss of context is a critical part of dealing with systems that are too complex to adequately be represented by a single aggregate. Whenever a signal is useful, an in-depth dive is usually worth it if you are looking to embrace complexity. This loss of context is a critical part of dealing with systems that are too complex to adequately be represented by a single aggregate. Whenever a signal is useful, an in-depth dive is usually worth it if you are looking to embrace complexity.
The metric is better used to attract your attention than as a target or as something that tells you what to know. Seek to explain and understand the metric first, not to change it. The metric is better used to attract your attention than as a target or as something that tells you what to know. Seek to explain and understand the metric first, not to change it.
![A OSEFW INDICAToR ITSELF USELESS (i ROhJSTCY RAVE GØD Foe A t.JlCELY ](Exported%20image%2020240808113916-17.png) ![](Reading/Read%20Later/assets/18.png)
As a related concept, if you act on a leading indicator, it stops leading, particularly when it’s influenced by trade-offs. As a related concept, if you act on a leading indicator, it stops leading, particularly when it’s influenced by trade-offs.
Metrics that become their own targets and are gamed of course lose meaningfulness; this is one of the most common issues with counting incidents and then debating whether an outage should or shouldn’t be declared in a way that might affect the tally rather than addressing it directly. Metrics that become their own targets and are gamed of course lose meaningfulness; this is one of the most common issues with counting incidents and then debating whether an outage should or shouldn’t be declared in a way that might affect the tally rather than addressing it directly.
But other metrics are of interest as well. If you evaluate your total capacity by some bottleneck’s value, and that this bottleneck is a target of optimization work, you will lose the ability to easily know when or how to scale up because that bottleneck possibly hid something else. This is contributing to a non-negligible portion of our incidents at work I believe. We fix a thing that acted as an implicit blocker and off we go into the great unknown. But other metrics are of interest as well. If you evaluate your total capacity by some bottleneck’s value, and that this bottleneck is a target of optimization work, you will lose the ability to easily know when or how to scale up because that bottleneck possibly hid something else. This is contributing to a non-negligible portion of our incidents at work I believe. We fix a thing that acted as an implicit blocker and off we go into the great unknown.
Our storage engine's disk storage used to be our main bottleneck. We drove scaling out and rebalancing traffic based on how close we were to heavy usage across multiple partitions. This was a useful signal, but it also drove costs up, and eventually became the target of optimization. Our storage engine's disk storage used to be our main bottleneck. We drove scaling out and rebalancing traffic based on how close we were to heavy usage across multiple partitions. This was a useful signal, but it also drove costs up, and eventually became the target of optimization.
An engineer successfully made our data offloading almost an order of magnitude faster, and eliminated our most glaring scaling issues at the time. Removing this limit however messed with our ability to know when to scale, which then revealed issues with file descriptors, memory, and snapshotting times. An engineer successfully made our data offloading almost an order of magnitude faster, and eliminated our most glaring scaling issues at the time. Removing this limit however messed with our ability to know when to scale, which then revealed issues with file descriptors, memory, and snapshotting times.
The only good advice I have here is to re-evaluate your metrics often, and change them. I guess there’s also a lesson to be learned that improvements can also cause their own uncertainty and that these successes can themselves lead to destabilizations. The only good advice I have here is to re-evaluate your metrics often, and change them. I guess there’s also a lesson to be learned that improvements can also cause their own uncertainty and that these successes can themselves lead to destabilizations.
Because we no longer needed to scale out as aggressively and were free to discover new issues, and one of our best improvements to the system in recent memory is therefore also a contributor to a lot of operational challenges. Because we no longer needed to scale out as aggressively and were free to discover new issues, and one of our best improvements to the system in recent memory is therefore also a contributor to a lot of operational challenges.
![PEOPLE WILL Do WRAT THEY BELIEVE IS USEFUL ](Exported%20image%2020240808113916-18.png) ![](Reading/Read%20Later/assets/19.png)
Things that people think are useful are possibly going to happen even if you forbid them. If you forbid people from logging onto production hosts, and they truly think they'll need it for emergency situations, they'll make sure there's still a way for it to happen, albeit under a different name. Things that people think are useful are possibly going to happen even if you forbid them. If you forbid people from logging onto production hosts, and they truly think they'll need it for emergency situations, they'll make sure there's still a way for it to happen, albeit under a different name.
On the other hand, things that people think are useless are likely to be done in a minimal way with no enthusiasm, such as lying in your timesheets. On the other hand, things that people think are useless are likely to be done in a minimal way with no enthusiasm, such as lying in your timesheets.
This means that writing a procedure means little unless people actually see its value and believe it’s worth following. Conversely, it means that if you can demonstrate the usefulness and make some approaches more usable, they’re likely to get adopted regardless of what is written down as a list of steps or procedures. This means that writing a procedure means little unless people actually see its value and believe it’s worth following. Conversely, it means that if you can demonstrate the usefulness and make some approaches more usable, they’re likely to get adopted regardless of what is written down as a list of steps or procedures.
A related concept here is one here is that if you are tracking things like action items after an incident reviews and they go in the backlog to die, it may not be that your people are failing to follow through; it might also be that it’s impractical to do so, or it’s could also be that these action items were never feeling useful, and the process itself needs to be revisited rather than reinforced. A related concept here is one here is that if you are tracking things like action items after an incident reviews and they go in the backlog to die, it may not be that your people are failing to follow through; it might also be that it’s impractical to do so, or it’s could also be that these action items were never feeling useful, and the process itself needs to be revisited rather than reinforced.
Seeing non-compliance is not necessarily a sign of bad workers. It may rather be a sign of a bad understanding of the workers' challenges, and point to a need to adjust how work is prescribed. Seeing non-compliance is not necessarily a sign of bad workers. It may rather be a sign of a bad understanding of the workers' challenges, and point to a need to adjust how work is prescribed.
Getting a small real buy-in into something voluntary may be better than getting fake buy-in into something you’re forcing people to do. Of course if you manage to write a good procedure that people believe are worth following, more power to you, this is going great. Getting a small real buy-in into something voluntary may be better than getting fake buy-in into something you’re forcing people to do. Of course if you manage to write a good procedure that people believe are worth following, more power to you, this is going great.
![A TEAM Yao.) RANDN PEOPLE ](Exported%20image%2020240808113916-19.png) ![](Reading/Read%20Later/assets/20.png)
The shortest feedback loop may be attained by giving people the tools to make the right decisions right there and then, and let them do it. Cut the middlemen, including yourself. The shortest feedback loop may be attained by giving people the tools to make the right decisions right there and then, and let them do it. Cut the middlemen, including yourself.
How do you make that work? We come back to goal alignments and top priorities being harmonized and well understood. If the pressures and goals are understood better, the decisions made also work better. How do you make that work? We come back to goal alignments and top priorities being harmonized and well understood. If the pressures and goals are understood better, the decisions made also work better.
That does mean that you have to listen back about how these things have been going, and that not only do you need to trust your people, but they need to trust you back with critical and unpleasant information as well. The feedback flows both ways, and this hinges on psychological safety. That does mean that you have to listen back about how these things have been going, and that not only do you need to trust your people, but they need to trust you back with critical and unpleasant information as well. The feedback flows both ways, and this hinges on psychological safety.
If you've ever talked to a contractor asked to help a big organization, the first thing they'll tell you they do is go talk to the workers with boots on the ground, and ask them what they think needs changing. They'll often have years of potential improvements backlogged, and that they're ready to tell anyone about. Either because management wouldn't listen to it, or because the workers lost trust that voicing that feedback would yield any result. If you've ever talked to a contractor asked to help a big organization, the first thing they'll tell you they do is go talk to the workers with boots on the ground, and ask them what they think needs changing. They'll often have years of potential improvements backlogged, and that they're ready to tell anyone about. Either because management wouldn't listen to it, or because the workers lost trust that voicing that feedback would yield any result.
Then the contractor brings it up to management as a neutral party, and suddenly it gets listened to and acted upon. Then the contractor brings it up to management as a neutral party, and suddenly it gets listened to and acted upon.
If you've lost that trust, then contractors can play that specific role of workers at the periphery of the organization helping drive change, and they can play a very useful function. If you've lost that trust, then contractors can play that specific role of workers at the periphery of the organization helping drive change, and they can play a very useful function.
But if you have that trust already, maintaining it is crucial because that’s how you get all the good information to help orient and influence things. But if you have that trust already, maintaining it is crucial because that’s how you get all the good information to help orient and influence things.
Trust also means that if you want people to be innovative, you have to allow them to make mistakes. You can’t get it right the first time all the time; if people can’t be allowed to get it wrong here and there, they won’t be allowed to improve and try new things either. Trust also means that if you want people to be innovative, you have to allow them to make mistakes. You can’t get it right the first time all the time; if people can’t be allowed to get it wrong here and there, they won’t be allowed to improve and try new things either.
![Mow To EMBRACE COMPLEXITY? QADE-OFFs FVk1kJG KEEP EEDBACk LOPS TIGHT HAC— — H OVER ](Exported%20image%2020240808113916-20.png) ![](Reading/Read%20Later/assets/21.png)
Finally, let's look at shifting perspective away from a bare analysis and onto a more systemic point of view. People in specific teams often have a more detailed expert view than you could either have, but if you're standing outside of it, your strength might be to understand how the parts interact in a way that isn't visible to the inside. Finally, let's look at shifting perspective away from a bare analysis and onto a more systemic point of view. People in specific teams often have a more detailed expert view than you could either have, but if you're standing outside of it, your strength might be to understand how the parts interact in a way that isn't visible to the inside.
![ITS HARD WITHOUT CHANGING TE PRESSES MT @STEe ](Exported%20image%2020240808113916-21.png) ![](Reading/Read%20Later/assets/22.png)
The most basic point here is that you can’t expect to change the outcome of these small little decisions that accumulate all the time if you never address the pressures within the system that foster them. The most basic point here is that you can’t expect to change the outcome of these small little decisions that accumulate all the time if you never address the pressures within the system that foster them.
I used to try and weed my lawn a whole hell of a lot and pull the weeds hours a week until someone explained to me that weeds grew easier in the type of soil I had (poor, dry, unmaintained soil) than grass, and pulling the weeds wasn’t the way to go, I needed to actually make the soil good for the grass to crowd out the weeds. I used to try and weed my lawn a whole hell of a lot and pull the weeds hours a week until someone explained to me that weeds grew easier in the type of soil I had (poor, dry, unmaintained soil) than grass, and pulling the weeds wasn’t the way to go, I needed to actually make the soil good for the grass to crowd out the weeds.
It's similar when considering this whole idea of root cause analysis—of trying to find the one source of the problem and removing it. If your root cause is at the weed’s level, you’ll keep pulling on them forever and will rarely make decent progress. The weeds will keep growing no matter how many roots you remove. It's similar when considering this whole idea of root cause analysis—of trying to find the one source of the problem and removing it. If your root cause is at the weed’s level, you’ll keep pulling on them forever and will rarely make decent progress. The weeds will keep growing no matter how many roots you remove.
If you foster good soil, if you create the right environment that encourages the type of behavior you want instead of the type of behaviour you dislike, you have hopes that the good stuff will crowd out the bad stuff. That’s a roundabout way of talking about culture change. And for these, deep dives based on [richer narratives](https://ferd.ca/notes/paper-accident-report-interpretation.html) and [thematic analysis](https://www.jeli.io/howie/welcome) prove more useful. If you foster good soil, if you create the right environment that encourages the type of behavior you want instead of the type of behaviour you dislike, you have hopes that the good stuff will crowd out the bad stuff. That’s a roundabout way of talking about culture change. And for these, deep dives based on [richer narratives](https://ferd.ca/notes/paper-accident-report-interpretation.html) and [thematic analysis](https://www.jeli.io/howie/welcome) prove more useful.
Also there's a warning here about trying to change the decisions your people make with carrots and sticks—with incentives. They are not going to fundamentally change what pressures the employees negotiate. The pressures stay the same, all you're doing is adding more of them, either in the form of rewards or punishments, which makes decision-making more complex and trickier. Also there's a warning here about trying to change the decisions your people make with carrots and sticks—with incentives. They are not going to fundamentally change what pressures the employees negotiate. The pressures stay the same, all you're doing is adding more of them, either in the form of rewards or punishments, which makes decision-making more complex and trickier.
Chances are people will keep making the same decisions as they were already, but then they'll report it differently to either get their bonus or to avoid getting penalized for it. Surfacing, understanding, and clarifying goal conflicts can make things easier or shape work to give them more room. Adding carrots and sticks can make things harder. Chances are people will keep making the same decisions as they were already, but then they'll report it differently to either get their bonus or to avoid getting penalized for it. Surfacing, understanding, and clarifying goal conflicts can make things easier or shape work to give them more room. Adding carrots and sticks can make things harder.
![ITS HARD TO WITHOUT CHANGING TE PRESS)ES MT @STEe ](Exported%20image%2020240808113916-22.png) ![](Reading/Read%20Later/assets/23.png)
But the tip here is probably: look into what are the behaviors you want to see happen, and give them room to grow. But the tip here is probably: look into what are the behaviors you want to see happen, and give them room to grow.
My most successful initiative at Honeycomb is probably creating [weekly discussion sessions about operational stuff and on-call](https://www.honeycomb.io/blog/oncallogy-sessions-best-practices). They range from “how do we operate new service X” into trickier discussions like “is it okay to be visibly angry in an incident”, “how do you deal with shit you don’t know or avoid burnout” or “are there times where code freezes are actually a useful thing?”. My most successful initiative at Honeycomb is probably creating [weekly discussion sessions about operational stuff and on-call](https://www.honeycomb.io/blog/oncallogy-sessions-best-practices). They range from “how do we operate new service X” into trickier discussions like “is it okay to be visibly angry in an incident”, “how do you deal with shit you don’t know or avoid burnout” or “are there times where code freezes are actually a useful thing?”.
Over time we looked into all sorts of weird interactions and the meeting became its own tool. Over time we looked into all sorts of weird interactions and the meeting became its own tool.
When we noticed incident reviews were difficult to schedule across departments and timezones, we decided that a good wide incident review is good operational talk and started making the optional time slot, which was already on every engineer's calendar (and some other departments too), available for them. It became easier for people to run incident reviews, and over time their size grew from 7-8 people, scoped to 1 or 2 teams, to bigger events with 20 to 40 people in them. When we noticed incident reviews were difficult to schedule across departments and timezones, we decided that a good wide incident review is good operational talk and started making the optional time slot, which was already on every engineer's calendar (and some other departments too), available for them. It became easier for people to run incident reviews, and over time their size grew from 7-8 people, scoped to 1 or 2 teams, to bigger events with 20 to 40 people in them.
We removed a huge but subtle blocker to good feedback loops existing within the organization. We removed a huge but subtle blocker to good feedback loops existing within the organization.
These sorts of small changes are those you can drive locally with almost no risk of having them run afoul of organizational priorities, and when you see them work, use the org structure to expand them everywhere. These sorts of small changes are those you can drive locally with almost no risk of having them run afoul of organizational priorities, and when you see them work, use the org structure to expand them everywhere.
![IODICATOR IS VSEFUL WH24 IS AcTED ON -12. 70.710/ o 94.13% ](Exported%20image%2020240808113916-23.png) ![](Reading/Read%20Later/assets/24.png)
I find it useful to keep focusing on what an indicator triggers as a behavior (the interaction) rather than *only* what it reports directly. This slide here is 4 error budgets from our SLOs, which combine how successful requests are both in terms of speed and errors, compared to an objective we express in terms of the desired fault rate.
I find it useful to keep focusing on what an indicator triggers as a behavior (the interaction) rather than _only_ what it reports directly. This slide here is 4 error budgets from our SLOs, which combine how successful requests are both in terms of speed and errors, compared to an objective we express in terms of the desired fault rate.
When we have to pick targets for our platform, people often ask whether we could pick some key SLOs and turn them as the objective. My answer is almost always "I don't care if we meet the SLOs or not". I mean I care, but not like that. When we have to pick targets for our platform, people often ask whether we could pick some key SLOs and turn them as the objective. My answer is almost always "I don't care if we meet the SLOs or not". I mean I care, but not like that.
SLOs aren’t hard and fast rules. When the error budget is empty, the main thing that matters to me is that we have a conversation about it, and decide what it is we want to happen from there on. Are we going to hold off on deploys and experiments? Are we able to meet the objectives while on-call, with some schedule corrective work, some major re-architecting? Can we just talk to the customers? Were our targets too ambitious or are we going to eat dirt for a while? SLOs aren’t hard and fast rules. When the error budget is empty, the main thing that matters to me is that we have a conversation about it, and decide what it is we want to happen from there on. Are we going to hold off on deploys and experiments? Are we able to meet the objectives while on-call, with some schedule corrective work, some major re-architecting? Can we just talk to the customers? Were our targets too ambitious or are we going to eat dirt for a while?
Kneejerk automated reactions aren’t nearly as useful as sitting down and having a cross-departmental discussion about what it is we want to do, as an organization, about these signals of unmet expectations. If it fits within on-call duty, like what is probably the case with the error budget on the top left, then fine. Kneejerk automated reactions aren’t nearly as useful as sitting down and having a cross-departmental discussion about what it is we want to do, as an organization, about these signals of unmet expectations. If it fits within on-call duty, like what is probably the case with the error budget on the top left, then fine.
But in other cases, such as the top right budget here, which seems to show a gradual decline, owe have to choose whether to do corrective work (and how/when) to meet the SLO—because that wasn't expected and is undesirable—or maybe to relax it—because that's actually a natural consequence of new more expensive features and we need to tweak definitions. Or we could temporarily ignore it because corrective work is already on the way, but not a top priority right now. But in other cases, such as the top right budget here, which seems to show a gradual decline, owe have to choose whether to do corrective work (and how/when) to meet the SLO—because that wasn't expected and is undesirable—or maybe to relax it—because that's actually a natural consequence of new more expensive features and we need to tweak definitions. Or we could temporarily ignore it because corrective work is already on the way, but not a top priority right now.
The two budgets at the bottom come from SLOs that may never page anyone. But from time to time, we re-calibrate them by asking support whether there are any issues users complain about that we aren't already aware of. So long as we're ahead of the complaints, we figure the SLOs are properly defined. But from time to time, we find out that we slipped by getting comments on things our alerting never properly captured. Or maybe we needed to better manage the user's expectations—that's also an option. The two budgets at the bottom come from SLOs that may never page anyone. But from time to time, we re-calibrate them by asking support whether there are any issues users complain about that we aren't already aware of. So long as we're ahead of the complaints, we figure the SLOs are properly defined. But from time to time, we find out that we slipped by getting comments on things our alerting never properly captured. Or maybe we needed to better manage the user's expectations—that's also an option.
For any of these choices, we also have to know how this is going to be communicated to users and customers, and having these discussions is the true value of SLOs to me. SLOs that flow outside of engineering teams provide a greater feedback loop about our practices, further upstream, than those that are used exclusively by the teams defining them, regardless of their use for alerting. For any of these choices, we also have to know how this is going to be communicated to users and customers, and having these discussions is the true value of SLOs to me. SLOs that flow outside of engineering teams provide a greater feedback loop about our practices, further upstream, than those that are used exclusively by the teams defining them, regardless of their use for alerting.
![BE THE FEEDBACK Loop ](Exported%20image%2020240808113916-24.png) ![](Reading/Read%20Later/assets/26.png)
Finally, this is where SREs can be placed in a great way to shine. You can be away from the central roles, away from the decision-making, on the periphery. By being outside of silos and floating around the organization’s structure, you are allowed to take information from many levels, carry it around, and really tie the loop at the end of so many decisions made in the organization by noting and carrying their impact back once they’ve hit a production system. Finally, this is where SREs can be placed in a great way to shine. You can be away from the central roles, away from the decision-making, on the periphery. By being outside of silos and floating around the organization’s structure, you are allowed to take information from many levels, carry it around, and really tie the loop at the end of so many decisions made in the organization by noting and carrying their impact back once they’ve hit a production system.
It is an iterative exercise, our sociotechnical systems are alive, and carrying pertinent signals and amplifying them, you can influence how long it’s gonna take before it all goes to hell anyway. It is an iterative exercise, our sociotechnical systems are alive, and carrying pertinent signals and amplifying them, you can influence how long it’s gonna take before it all goes to hell anyway.

2175
Reading/Read Later/How AsyncAwait Really Works in C - .NET Blog.md
View File

File diff suppressed because it is too large Load Diff

52
Reading/Read Later/Paper Moving off the Map.md
View File

@@ -1,52 +0,0 @@
Clipped from: [https://ferd.ca/notes/paper-moving-off-the-map.html](https://ferd.ca/notes/paper-moving-off-the-map.html)
These are notes I have taken elsewhere that I'm re-posting here as my weekly paper reading because it is quite simply one of my favourite papers ever.
[Moving off the Map: How Knowledge of Organizational Operations Empowers and Alienates](https://sci-hub.se/10.1287/orsc.2018.1277) is a work of ethnography where a researcher embedded herself into 5 organizations including 6 projects aiming at re-structuration—business process redesign (BRP)—and which all had a phase of extensive process mapping. She noticed that at the projects' conclusions, most employees returned to their roles and got raises within the organization, but a subset of them who were centrally located within the organization decided to move to peripheral roles. She decided to investigate this.
What she found was that tracing out the structure of how work is done (and what work is done) and decisions are made was a significant activity behind the split. It happened because people doing this tracing activity had a shock when they realized that the business' structure was not coordinated nor planned, but an emergent mess and consequence of local behaviours in various groups. Their new understanding of work resulted in either Empowerment ("I now know how I can change things") or Alienation ("Nothing I thought mattered does, my work is useless here"), which explained their move to peripheral roles.
Some of these reports are also just plain heart breaking. I have so many highlights for it.
The paper starts by mentioning that centrally-located actors (people at the core of the management structure of an organization) are less likely to initiate change, and more likely to stall it. Additionally, desire for change are likely to come from the periphery, and as people move towards the center, that desire tends to go away. This is a surprise to no one.
However, if central actors are to initiate change, it comes from either a) contradictions, tensions, or inconsistencies are experienced and push them to reflections, and b) being exposed to how other organizations (or even societies) do things, which opens more awareness. These two things are called "disembedding", and can lead to central actors pushing for structural change.
The paper accidentally "discovered" a third approach: taking the time to study how things are done in the organization can cause that dissonance, and encourage central actors to move to the periphery of the organization _in order to effect change_ because they lose trust in the structure of the organization itself and their role in it.
This was found out while the author was doing a study of 5 big corporations with 6 major business restructuration projects with hundreds of workers, and she noticed that while some employees went back to their roles (but with promotions), or towards roles that were more central when it was done, a subset of employees instead left very central roles to go work on the periphery, for sometimes less interesting conditions. So she started asking why and ran a big analysis.
What she noticed is that all the employees who eventually left their roles were assigned different specific tasks from the rest of people in these projects: they had been ask to do process mapping, where essentially they had to make a representation of "what we do here", how the business works, how decisions are made, and how information moves around. People not involved didn't find it significant, but people involved were shocked into leaving their roles, to make it short.
The author makes a point that it's not process mapping causing this, but rather that having a deep engagement in representing and understanding the operations of the organization and how their own role would fit in it would cause this to happen—it was probabilistic.
The tracing was done by employees who would do things like walk the floor, ask people how they do work, sit in meetings with question like "What do we do?" with people in various roles, asked them to list tasks on whiteboards, connecting them with strings, and consolidated into huge maps like the following, which connected local experiences into a broader organizational context:
![Walls covered in colorful sheet of papers with strings between all elements. This is an implementation of process-mapping out the organization and who covers what](Exported%20image%2020240808113929-0.png)
This had the effect of surface things that were previously invisible and make it discrete. This likely ties into concepts mentioned here before of "work as done" vs. "work as imagined":
The map allowed them to see how the system operated below the surface, integrating all the pieces to generate a comprehensive view. They commented on the uniqueness of this comprehensive view: “We don’t allow people to see the end-to-end view... to see how things interrelate.” One explained that the experience “ruins [one’s] perspective in a good way.” Another described how it gave her a “whole different way of looking at things.” By revealing the web of roles, relations, and routines that coalesce to make the organization, the map made the organization’s actual operation intelligible.
[...]
Competent members of organizations draw on everyday knowledge [...] as they perform their roles, but this knowledge does not speak to the organization’s broader order. Despite how remarkably capable these employees were at “recognizing, knowing, and ‘doing’ the lived order,” the broader order or structure is often “resistance to analytic recovery” from the inside. Even if they would like to observe and reflect on their organization’s detailed operating process, they rarely have opportunities, such as building process maps, that provide time and access.
So what were the immediate consequences? I'm quoting this directly:
They expected to observe inefficiencies and waste, the targets of redesign, and they did. Tasks that could be done with one or two hand-offs were taking three or four. Data painstakingly collected for decision-making processes were not used. Local repairs to work processes in one unit were causing downstream problems in another. Workarounds, duplication of effort, and poor communication and coordination were all evident on the map.
Beyond these issues, they observed a more fundamental problem. A team member explained, “I’m getting a really clear visual of what the mess is.” Standing back from the wall, he sighed, and said, “The problem is that it was not designed in the first place.” Instead of observing a system designed, adapted, and coordinated to achieve stated goals, he pointed to three examples on the map that demonstrated the exercise of agency in various places and at various levels in the organization. These change efforts lacked broader perspective and direction as well as coordination and integration with other efforts
They mention examples such as a "kingdom builder" where the map revealed some manager who kept accumulating departments for the sake of accumulating power but was invisible to the organization, and essentially just found a lot of "what the fuck, this is just random shit that's leftovers from really old decisions." People see local problems, general approaches, and they try to fix things. This clashes with things the organization tries to do (when it tries), and there is no coherent organization to anything:
Some held out hope that one or two people at the top knew of these design and operation issues; however, they were often disabused of this optimism. For example, a manager walked the CEO through the map, presenting him with a view he had never seen before and illustrating for him the lack of design and the disconnect between strategy and operations. **The CEO, after being walked through the map, sat down, put his head on the table, and said, “This is even more fucked up than I imagined.”** The CEO revealed that not only was the operation of his organization out of his control but that his grasp on it was imaginary.
They learned that what they had previously attributed to the direction and control of centralized, bureaucratic forces was actually the aggregation of the work and decisions of people distributed throughout the organization. Everyone was working on the part of the organization that they were familiar with, assuming that another set of people were attending to the larger picture, coordinating the larger system to achieve goals and keeping the organization operating. They found out that this was not the case.
This may not necessarily be surprising to people, but it may be surprising for people to learn that CEOs and others think they have so much more control than they do!
Anyway, the two reactions in general were either Empowerment or Alienation.
On the front of Empowerment, this is caused because:
Members of the organization carry on as though these distinctions are facts, burdening the organization’s categories, practices, and boundaries with a false sense of durability and purpose.
[...]
The idea that organizations are an ongoing human product was a provocative insight for these employees. This new perspective, as one explained, “made things seem possible.” Once they could see the “what” as a dynamic social creation, they could begin asking better questions about “how.” A team member explained that the logic of organization should not be fixed and how its rules, synthetic creations, are free to deviate
[...]
Their peripheral role choices allowed team members to exploit this new understanding of the organization’s operations. They could work with new assumptions about the mutability and possibility of the organization and create structures and systems to coordinate and direct the web of roles and interactions. Their new role choices also allowed them to remain above and outside of the organization’s daily operations.
So in short, getting how a lot of it isn't fixed, how a lot of it is arbitrary but flexible meant that these people felt they understood how to effect change better, and that by moving away from the center and into the periphery, they could start doing effective change work.
Alienation is so god damn heartbreaking though, and the author warns that before starting this process in an organization, you have to be ready that some people may feel a major shock that the work they thought was valuable and important is in fact useless and worth nothing. In fact the author warns that finding work and jobs that were not meaningful or useful at all was a common theme:
As part of the map-building process, employees were invited to identify their role on the map and to indicate how it was connected to other roles through either inputs or outputs. Team members recounted that it was difficult to observe employees “go through a real emotional struggle when they see that what they are doing is not really adding value or that what they are doing is really disconnected from what they thought they were doing.” In one case, a finance manager noticed that his role was on the wall but that it was not connected to any other role on the wall. He had been producing financial reports and sending them to several departments because he understood them to be crucial for their decision-making process; however, no one had identified his work as an input to theirs.
This realization was, in the end, **devastating for him. He was on the verge of tears... at first, he became very argumentative and was trying to convince people that you go from this Post-it note down here to mine. [Other employees explained] Well no, we don’t do that. It was a two-hour conversation. And he finally sat down, and he said, so why am I doing this? It was devastating.**
The eventual outcome of this “aha” was that the manager was moved to another role in the department after working four years in a position that had served almost no purpose.
After such analyses, team members could not look at particular roles and people in the same way.
A lot of people also found out that they thought they were solving real problems, helping people with real issues, finding real work-arounds, but found that in the overall organizational map, it was meaningless and had no impact: they could be fixing real problems in departments that themselves were not useful.
Others found that they had properly fixed issues by introducing new databases with critical information, but that they had been unable to get any buy-in for that, so analysts and people having spent a lot of time on these just had no impact at all:
Their knowledge of the limits of local, small-scale change and the futility of changing parts of the organization without addressing the system as a whole, discouraged employees from returning to their career in the organization. They did not want to contribute to the mess or reproduce the mess they had observed.
[...]
What they had learned could not be unlearned or ignored.
The author state that whether it is due to alienation or empowerment, both behaviours push people to move to the edges of the system, where they can either find new roles or types of changes that they believe are more useful. The structural knowledge gain essentially let them know of better ways to do useful things and enact change. Specifically, learning that the organization's structure is the result of interactions rather than a context in which they take place is a key learning that sociologists knew already:
This perspective or comprehension affects how we speak and act. We speak about organizations as if they are objects that exist independent of us, and we act as though they constrain and guide our actions. When we objectify social systems (organizations, communities, families, gender roles), we apprehend them as “prearranged patterns” that impose themselves on us, coercing particular roles and rules. We free ourselves to talk about and inhabit them as independent of us: as existing prior to us, standing before us, outliving us, and operating without us. Given this, we are relieved of greater responsibility for them. Our responsibility is to skillfully fulfill our role within these objectified realms.
[...]
Whereas, as some sociologists “know that organizations and institutions exist only in actual people’s doings and that these are necessarily particular, local and ephemeral”, employees may be less likely to know this. When they do, it problematizes their past and future participation.
[...]
The realization that social worlds do not have an independent, stable existence but instead emerge from our collective action is “sometimes arrived at in a moment of heady delight, but often as a horrifying realization”. This realization is considered a “fatal insight” because it destroys assumptions that the current order, roles, rules, and routines are given. Within the system of roles, rules, and routines, there is far more room to maneuver than previously assumed. Rejection of objectivity puts possibility, perhaps even responsibility, squarely in the court of subjectivity.
I think this quote above is real good.
I'm going to conclude with it (, although the author adds a bit of a section about mentioning that given this research means that we can suspect some of the most effective change to be driven by actors who once were at the core of the system and moved to its periphery. This likely is a sign that they know how shit works and have an idea of how to challenge it. Insider knowledge dragged to the edges may be a key for strong means to modifying how things work. I'll let you read the paper if you want the details of that.

58
Reading/Read Later/Problem exploration, selection and validation. Irrational Exuberance.md
View File

@@ -1,58 +0,0 @@
Clipped from: [https://lethain.com/intro-product-management/](https://lethain.com/intro-product-management/)
![Problem Discovery Problem Selection Problem Discovery Solution Execution Validation Problem Selection Solution Execution Validation Problem Discovery Problem Selection Problem Discoverv Solution Execution Validation Problem Selection Solution Ex Validation ](Exported%20image%2020240808113923-0.png)
Most engineering organizations separate engineering and product leadership into distinct roles. This is usually ideal, not only because these roles benefit on distinct skills, but also because they thrive from different perspectives and priorities. It’s quite hard to do both well at the same time.
I’ve met many product managers who are excellent operators, but few product managers who can operate at a high degree while also getting deep with their users’ needs. Likewise, I’ve worked with many engineering managers who ground their work in their users needs, but few who can affix their attention on those users when things start getting rocky within their team.
Reality isn’t always accommodating of this ideal setup. Maybe your team’s product manager leaves or a [new team is being formed](https://lethain.com/durably-excellent-teams/), and you, as an engineering leader, need to cover both roles for a few months. This can be exciting, and yes, this can be a time when “exciting” rhymes with “terrifying.”
Product management is a deep profession, and mastery requires years of practice, but I’ve developed a simple framework for product management to use when I’ve found [myself fulfilling product management](https://lethain.com/product-management-infra-engineering/) responsibilities for a team. It’s not perfect, but hopefully it’ll be useful for you as well.
Product management is an iterative elimination tournament, with each round consisting of _problem discovery_, _problem selection_ and _solution validation_. _Problem discovery_ is uncovering possible problems to work on, _problem selection_ is filtering those problems down to a viable subset, and _solution validation_ is ensuring your approach to solving those problems work as cheaply as possible.
If you do a good job at all three phases, you win the luxury of doing it all again; this time with more complexity and scope. If you don’t do well, you end up forfeiting or being asked to leave [the game](https://www.amazon.com/dp/B004W3FM4A/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1).
## Problem discovery
The first phase of a planning cycle is exploring the different problems you could pick to solve. It’s surprisingly common to skip this phase, but that unsurprisingly leads to inertia-driven local optimization. Taking the time to evaluate which problem to solve is one of the best predictors I’ve found of a team’s long-term performance.
The themes I’ve found useful for populating the problem space are:
- **Users’ pain**. What are the problems that your users experience? It’s useful to both go broad via survey mechanisms as well as to go deep by interviewing a smaller set of interesting folks across different user segments.
- **Users’ purpose**. What motivates your users to engage with your systems? How can you better enable them to accomplish their goals?
- **Benchmark**. Look at how your company compares to competitors in the same and similar industries. Are there areas that you are quite weak? Those are areas to _consider_ investment. Sometimes folks keep to a narrow lense when benchmarking, but I’ve found that you learn the most interesting things by considering both fairly similar and rather different companies.
- **Cohorts**. What is hiding behind your clean distributions? Exploring your data for the cohorts hidden behind top-level analysis is an effective way to discover new kinds of users with surprising needs.
- **Competitive advantages.** By understanding the areas you’re exceptionally strong in, you can identify opportunities that you’re better positioned to fulfill than other companies.
- **Competitive moats**. Moats are a more extreme version of a competitive advantage. Moats represent a sustaining competitive advantage, which make it possible for you to pursue offerings that others simply cannot. It’s useful to consider moats in three different ways:
- What your existing moats enable you to do today?
- What are the potential moats you could build for the future?
- What moats are your competitors luxuriating behind?
- **Compounding leverage**. What are the composable blocks that you could start building today that will compound into major product or [technical leverage](https://lethain.com/building-technical-leverage/) over time? I think of this category of work as finding ways to get the benefit (at least) twice. This are potentially tasks that initially don’t seem important enough to prioritize, but whose compounding value makes it possible.
- A design example might be introducing a application new navigation scheme that better supports the expanded set of actions and modes you have today, and that will support future proliferation as well. (Bonus points if it manages to prevent future arguments about positioning of new actions relative to existing ones!)
- An infrastructure example might be moving a failing piece of technology to a new standard, this addresses a reliability issue, reduces maintenance costs, and also [reduces the costs of future migrations](https://lethain.com/migrations/).
## Problem selection
Once you’ve identified enough possible problems, the next challenge is to narrow down to a specific problem portfolio. Some of the aspects I’ve found useful to consider during this phase are:
- **Surviving the round**. Thinking back to the iterative elimination tournament, what do you need to do to survive the current round? This might be the revenue the product will need to generate to avoid getting canceled, adoption, etc.
- **Surviving the next round**. Where do you need to be when the next round starts, to avoid getting eliminated then? There are a number of ways, many of them revolving around quality tradeoffs, to reduce long-term throughput in favor of short term velocity. (Conversely, winning leads to significantly more resources later, so that tradeoff is appropriate sometimes!)
- **Winning rounds**. It’s important to survive every round, but it’s also important to eventually win a round! What work would ensure you’re trending towards winning a round?
- **Consider different time frames.** When folks disagree which problems to work on, I find it’s most frequently rooted in different assumptions about the correct time frame to optimize for. What would you do if your company was going to run out of money in six months? What if there were no external factors forcing you to show results until two years out? Five years out?
- **Industry trends**. Where do you think the industry is moving towards, and what work will position you to take advantage of those friends, or at least avoid having to redo the work in near future?
- **Return on investment**. Personally, I think folks often under prioritize quick, easy wins. If you’re in the uncommon position of understanding both the impact and costs of doing small projects, then take time to try ordering problems by expected return on investment. At this phase you’re unlikely to know the exact solution, so figuring out cost is tricky, but for categories of problems you’ve seen before you can probably make a solid guess (if you don’t personally have relevant experience, ask around). Particular in cases where wins are compounding, these are be surprisingly valuable over the medium and long term.
- **Experiments to learn**. What could you learn now that would make problem selection in the future much easier?
## Solution validation
Once you’ve narrowed down the problem you want to solve, it’s easy to jump directly into execution, but that can make it easy to fall in love with a difficult approach. Instead, I’ve found it well worth it to derisk your approach with an explicit solution validation phase.
The elements I’ve found effective for solution validation are:
- **Write a customer letter.** Write the launch announcement that you would send after finishing the solution. Are you able to write something exciting, useful and real? It’s much more useful to test it against your actual users rather than relying on your intuition.
- **Identify prior art**. How do peers across the industry approach this problem? The fact that others have solved a problem in a certain way doesn’t mean it’s a great way, but it does at least mean it’s possible. A mild caveat that it’s better to rely on folks you have some connection to instead of conference talks and such; there is a surprisingly large amount of misinformation out there.
- **Find reference users**. Can you find users who are willing to be the first users for the solution? If you can’t, you should be skeptical whether what you’re building is worthwhile.
- **Prefer experimentation over analysis**. It’s far more reliable to get good at cheap validation than it is to get great at consistently picking the right solution. Even if you’re brilliant, you are almost always missing essential information when you begin designing. Analysis can often uncover missing information, but it depends on knowing where to look, whereas experimentation allows you to find problems you didn’t anticipate.
- **Find the path more quickly traveled**. The most expensive way to validate a solution is to build it in its entirety. The upside of that approach is that you’ve lost no time if you picked a good solution, the downside is that you’ve sacrificed a huge amount of time if it’s not. Try to find the cheapest way to validate.
- **Justify switching costs**. What will the switching costs be for users who move to your solution? Even if folks want to use it, if the switching costs are too high then they simply won’t be able to. Test with your potential users if they’d be willing to pay the full cost of migrating to your solution instead of their existing planned work.
As an aside, I’ve found that most aspects of [running a successful technology migration](http://lethain.com/migrations) overlap with good solution validation! This is a very general skill that will repay the time you invest into learning it many times over.
Putting these three elements today–exploration, selection and validation–won’t make you an exceptional product manager overnight, but they will provide a solid starting place to develop those skills and perspective for the next time you find yourself donning the product manager hat.

65
Reading/Read Later/Problem exploration, selection and validation..md Normal file
View File

@@ -0,0 +1,65 @@
[[ReadItLater]] [[Article]]
# [Problem exploration, selection and validation.](https://lethain.com/intro-product-management/)
Most engineering organizations separate engineering and product leadership into distinct roles. This is usually ideal, not only because these roles benefit on distinct skills, but also because they thrive from different perspectives and priorities. It’s quite hard to do both well at the same time.
I’ve met many product managers who are excellent operators, but few product managers who can operate at a high degree while also getting deep with their users’ needs. Likewise, I’ve worked with many engineering managers who ground their work in their users needs, but few who can affix their attention on those users when things start getting rocky within their team.
Reality isn’t always accommodating of this ideal setup. Maybe your team’s product manager leaves or a [new team is being formed](https://lethain.com/durably-excellent-teams/), and you, as an engineering leader, need to cover both roles for a few months. This can be exciting, and yes, this can be a time when “exciting” rhymes with “terrifying.”
Product management is a deep profession, and mastery requires years of practice, but I’ve developed a simple framework for product management to use when I’ve found [myself fulfilling product management](https://lethain.com/product-management-infra-engineering/) responsibilities for a team. It’s not perfect, but hopefully it’ll be useful for you as well.
Product management is an iterative elimination tournament, with each round consisting of *problem discovery*, *problem selection* and *solution validation*. *Problem discovery* is uncovering possible problems to work on, *problem selection* is filtering those problems down to a viable subset, and *solution validation* is ensuring your approach to solving those problems work as cheaply as possible.
If you do a good job at all three phases, you win the luxury of doing it all again; this time with more complexity and scope. If you don’t do well, you end up forfeiting or being asked to leave [the game](https://www.amazon.com/dp/B004W3FM4A/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1).
## Problem discovery
The first phase of a planning cycle is exploring the different problems you could pick to solve. It’s surprisingly common to skip this phase, but that unsurprisingly leads to inertia-driven local optimization. Taking the time to evaluate which problem to solve is one of the best predictors I’ve found of a team’s long-term performance.
The themes I’ve found useful for populating the problem space are:
- **Users’ pain**. What are the problems that your users experience? It’s useful to both go broad via survey mechanisms as well as to go deep by interviewing a smaller set of interesting folks across different user segments.
- **Users’ purpose**. What motivates your users to engage with your systems? How can you better enable them to accomplish their goals?
- **Benchmark**. Look at how your company compares to competitors in the same and similar industries. Are there areas that you are quite weak? Those are areas to *consider* investment. Sometimes folks keep to a narrow lense when benchmarking, but I’ve found that you learn the most interesting things by considering both fairly similar and rather different companies.
- **Cohorts**. What is hiding behind your clean distributions? Exploring your data for the cohorts hidden behind top-level analysis is an effective way to discover new kinds of users with surprising needs.
- **Competitive advantages.** By understanding the areas you’re exceptionally strong in, you can identify opportunities that you’re better positioned to fulfill than other companies.
- **Competitive moats**. Moats are a more extreme version of a competitive advantage. Moats represent a sustaining competitive advantage, which make it possible for you to pursue offerings that others simply cannot. It’s useful to consider moats in three different ways:
- What your existing moats enable you to do today?
- What are the potential moats you could build for the future?
- What moats are your competitors luxuriating behind?
- **Compounding leverage**. What are the composable blocks that you could start building today that will compound into major product or [technical leverage](https://lethain.com/building-technical-leverage/) over time? I think of this category of work as finding ways to get the benefit (at least) twice. This are potentially tasks that initially don’t seem important enough to prioritize, but whose compounding value makes it possible.
- A design example might be introducing a application new navigation scheme that better supports the expanded set of actions and modes you have today, and that will support future proliferation as well. (Bonus points if it manages to prevent future arguments about positioning of new actions relative to existing ones!)
- An infrastructure example might be moving a failing piece of technology to a new standard, this addresses a reliability issue, reduces maintenance costs, and also [reduces the costs of future migrations](https://lethain.com/migrations/).
## Problem selection
Once you’ve identified enough possible problems, the next challenge is to narrow down to a specific problem portfolio. Some of the aspects I’ve found useful to consider during this phase are:
- **Surviving the round**. Thinking back to the iterative elimination tournament, what do you need to do to survive the current round? This might be the revenue the product will need to generate to avoid getting canceled, adoption, etc.
- **Surviving the next round**. Where do you need to be when the next round starts, to avoid getting eliminated then? There are a number of ways, many of them revolving around quality tradeoffs, to reduce long-term throughput in favor of short term velocity. (Conversely, winning leads to significantly more resources later, so that tradeoff is appropriate sometimes!)
- **Winning rounds**. It’s important to survive every round, but it’s also important to eventually win a round! What work would ensure you’re trending towards winning a round?
- **Consider different time frames.** When folks disagree which problems to work on, I find it’s most frequently rooted in different assumptions about the correct time frame to optimize for. What would you do if your company was going to run out of money in six months? What if there were no external factors forcing you to show results until two years out? Five years out?
- **Industry trends**. Where do you think the industry is moving towards, and what work will position you to take advantage of those friends, or at least avoid having to redo the work in near future?
- **Return on investment**. Personally, I think folks often under prioritize quick, easy wins. If you’re in the uncommon position of understanding both the impact and costs of doing small projects, then take time to try ordering problems by expected return on investment. At this phase you’re unlikely to know the exact solution, so figuring out cost is tricky, but for categories of problems you’ve seen before you can probably make a solid guess (if you don’t personally have relevant experience, ask around). Particular in cases where wins are compounding, these are be surprisingly valuable over the medium and long term.
- **Experiments to learn**. What could you learn now that would make problem selection in the future much easier?
## Solution validation
Once you’ve narrowed down the problem you want to solve, it’s easy to jump directly into execution, but that can make it easy to fall in love with a difficult approach. Instead, I’ve found it well worth it to derisk your approach with an explicit solution validation phase.
The elements I’ve found effective for solution validation are:
- **Write a customer letter.** Write the launch announcement that you would send after finishing the solution. Are you able to write something exciting, useful and real? It’s much more useful to test it against your actual users rather than relying on your intuition.
- **Identify prior art**. How do peers across the industry approach this problem? The fact that others have solved a problem in a certain way doesn’t mean it’s a great way, but it does at least mean it’s possible. A mild caveat that it’s better to rely on folks you have some connection to instead of conference talks and such; there is a surprisingly large amount of misinformation out there.
- **Find reference users**. Can you find users who are willing to be the first users for the solution? If you can’t, you should be skeptical whether what you’re building is worthwhile.
- **Prefer experimentation over analysis**. It’s far more reliable to get good at cheap validation than it is to get great at consistently picking the right solution. Even if you’re brilliant, you are almost always missing essential information when you begin designing. Analysis can often uncover missing information, but it depends on knowing where to look, whereas experimentation allows you to find problems you didn’t anticipate.
- **Find the path more quickly traveled**. The most expensive way to validate a solution is to build it in its entirety. The upside of that approach is that you’ve lost no time if you picked a good solution, the downside is that you’ve sacrificed a huge amount of time if it’s not. Try to find the cheapest way to validate.
- **Justify switching costs**. What will the switching costs be for users who move to your solution? Even if folks want to use it, if the switching costs are too high then they simply won’t be able to. Test with your potential users if they’d be willing to pay the full cost of migrating to your solution instead of their existing planned work.
As an aside, I’ve found that most aspects of [running a successful technology migration](http://lethain.com/migrations) overlap with good solution validation! This is a very general skill that will repay the time you invest into learning it many times over.
---
Putting these three elements today–exploration, selection and validation–won’t make you an exceptional product manager overnight, but they will provide a solid starting place to develop those skills and perspective for the next time you find yourself donning the product manager hat.

71
Reading/Read Later/Product management in infrastructure eng. Irrational Exuberance.md → Reading/Read Later/Product management in infrastructure eng..md
View File

@@ -1,38 +1,56 @@
Clipped from: [https://lethain.com/product-management-infra-engineering/](https://lethain.com/product-management-infra-engineering/) [[ReadItLater]] [[Article]]
![Discover Prioritize 000 ](Exported%20image%2020240808113927-0.png) # [Product management in infrastructure eng.](https://lethain.com/product-management-infra-engineering/)
Recently a bunch of teams I work with have turned the corner, having paid down technical debt to a long-term sustainable level. The future unfurls with possibility. We can do *anything*. That’s exciting! It can also be pretty disorienting. For me, this is the most inspiring moment of management, and one of the hardest.
Recently a bunch of teams I work with have turned the corner, having paid down technical debt to a long-term sustainable level. The future unfurls with possibility. We can do _anything_. That’s exciting! It can also be pretty disorienting. For me, this is the most inspiring moment of management, and one of the hardest.
When we were completely focused on system reliability or churning tasks, most teams pulled their roadmaps down to a month or two, and we got so focused that we disconnected from our internal users. With less time soaked by maintenance, we’ve scurried to understand our users’ needs and define an optimistic, future-facing roadmap to support them. When we were completely focused on system reliability or churning tasks, most teams pulled their roadmaps down to a month or two, and we got so focused that we disconnected from our internal users. With less time soaked by maintenance, we’ve scurried to understand our users’ needs and define an optimistic, future-facing roadmap to support them.
In short, we’ve bootstrapped product management. In short, we’ve bootstrapped product management.
Many of the infrastructure engineer teams I’ve been a part of have struggled to make the transition from maintenance to innovation, and I wanted to write down some of the ideas that we’re exploring to ease this shift. I’d also love to hear what has worked well for other folks! Many of the infrastructure engineer teams I’ve been a part of have struggled to make the transition from maintenance to innovation, and I wanted to write down some of the ideas that we’re exploring to ease this shift. I’d also love to hear what has worked well for other folks!
## Foundation to Innovation ## Foundation to Innovation
I believe teams tend to have two distinct modes of operation: I believe teams tend to have two distinct modes of operation:
- a foundation mode where the vast majority of tasks are mandatory, driven by non-negotiable needs like compliance, security, reliability and “victim of success” challenges like scaling a very popular product. Kanban is optimized for this mode, and I think [The Phoenix Project](https://www.amazon.com/dp/B00AZRBLHO/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1) is a really helpful resource on executing in this mode. - a foundation mode where the vast majority of tasks are mandatory, driven by non-negotiable needs like compliance, security, reliability and “victim of success” challenges like scaling a very popular product. Kanban is optimized for this mode, and I think [The Phoenix Project](https://www.amazon.com/dp/B00AZRBLHO/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1) is a really helpful resource on executing in this mode.
- an innovation mode where you have a lot of flexibility in which problems to prioritize and how to solve them. This is similar to product development as [described in Inspired](https://www.amazon.com/INSPIRED-Create-Tech-Products-Customers/dp/1119387507/ref=dp_ob_image_bk).
- an innovation mode where you have a lot of flexibility in which problems to prioritize and how to solve them. This is similar to product development as [described in Inspired](https://www.amazon.com/INSPIRED-Create-Tech-Products-Customers/dp/1119387507/ref=dp_ob_image_bk).
In practice many teams have one foot in both modes, and most teams cycle between the two over time, but for any given team at any given time, they usually have a primary model. In practice many teams have one foot in both modes, and most teams cycle between the two over time, but for any given team at any given time, they usually have a primary model.
The foundation mode is a game of execution, focus and limiting work-in-progress, whereas the innovation mode is about listening to users, exploring solution spaces and an eternal focus on validating solutions as early and cheaply as possible. The foundation mode is a game of execution, focus and limiting work-in-progress, whereas the innovation mode is about listening to users, exploring solution spaces and an eternal focus on validating solutions as early and cheaply as possible.
Most infrastructure teams have a lot of experience in foundation, but have much less in innovation, so I won’t belabor managing through scarcity, and will instead dive into how to manage in times of surplus engineering capacity. Most infrastructure teams have a lot of experience in foundation, but have much less in innovation, so I won’t belabor managing through scarcity, and will instead dive into how to manage in times of surplus engineering capacity.
## Problem discovery ## Problem discovery
When you have surplus engineering capacity, folks tend to have a long backlog of stuff they’d like to work on, and many teams immediately jump on those, but I think it’s useful to fight that instinct and to step back and do deliberate discovery. When you have surplus engineering capacity, folks tend to have a long backlog of stuff they’d like to work on, and many teams immediately jump on those, but I think it’s useful to fight that instinct and to step back and do deliberate discovery.
There are two things that are essential to discovering opportunity: cast a very broad net, and don’t evaluate ideas while discovering them. Try to get as many ideas as possible, and don’t spend a single moment worrying if they’re any good, prioritization is a later step. There are two things that are essential to discovering opportunity: cast a very broad net, and don’t evaluate ideas while discovering them. Try to get as many ideas as possible, and don’t spend a single moment worrying if they’re any good, prioritization is a later step.
I recommend taking a four prong approach, learning about your users’ needs, what peer companies are doing, where industry leaders are going, and brainstorming with your team. I recommend taking a four prong approach, learning about your users’ needs, what peer companies are doing, where industry leaders are going, and brainstorming with your team.
Some of the techniques that I’ve seen work well: Some of the techniques that I’ve seen work well:
- **SLAs** - sit down with your users to discuss what SLAs they expect from your system, and then turn those into your dashboards. Even if you can’t hit some of those SLAs today, understanding what users want you to be able to offer is powerful. - **SLAs** - sit down with your users to discuss what SLAs they expect from your system, and then turn those into your dashboards. Even if you can’t hit some of those SLAs today, understanding what users want you to be able to offer is powerful.
- **User surveys** - it’s surprisingly hard to write a great survey, and easy to create survey fatigue, but a good survey is an awesome way to get many folks to share input. A good survey is a short, quantifies when possible, and gets proofread by folk with opposing perspectives before it’s sent out! They’re especially good as a first step to identify who to follow up with in detail.
- **Coffee chats** - meet with your users periodically and learn about what they’re doing. Do remember to ask if there is stuff you can improve, but I think it’s most valuable to understand what they’re doing, which is great fodder for thinking about how you can help. A cheaper version of this is a short email with a quick compliment and asking if you could help with anything. - **User surveys** - it’s surprisingly hard to write a great survey, and easy to create survey fatigue, but a good survey is an awesome way to get many folks to share input. A good survey is a short, quantifies when possible, and gets proofread by folk with opposing perspectives before it’s sent out! They’re especially good as a first step to identify who to follow up with in detail.
- **Discussion groups** - bringing together a small group of two to four folks and hearing their ideas is a good way to get input. I find these work best when you can bring a specific proposal, or set of proposals, for folks to react against. As a variant, we’ve also experimented with recurring customer advisory groups.
- **Peer-company chats** - in addition to chatting with your users, I’ve found it equally valuable to chat with a wide variety of folks working on similar problems at other companies to understand how they’re thinking about things. I’ve heard some extremely valid concerns around this leads directly to cargo-culting and groupthink, which is why I think it’s so important to decouple discovery from prioritization: listen to what other folks are doing, don’t automatically decide to adopt them. - **Coffee chats** - meet with your users periodically and learn about what they’re doing. Do remember to ask if there is stuff you can improve, but I think it’s most valuable to understand what they’re doing, which is great fodder for thinking about how you can help. A cheaper version of this is a short email with a quick compliment and asking if you could help with anything.
- **Academic and industry research** - look for papers coming out of the big tech companies (Google, Microsoft, Facebook, etc) and also for academic research on a given area. I did a basic example of this [when I surveyed load generation research](https://lethain.com/braindump-on-load-generation/), and it was helpful to expand my thinking beyond the obvious.
- **Open source** - [along the lines of my exploration of the open source data ecosystem](https://lethain.com/from-lambda-to-kappa-dataflow-paradigms/), I find it useful to spend some time understanding the state of open source for a given area. This gives you an easy sense of trends and how other folks see the space evolving. - **Discussion groups** - bringing together a small group of two to four folks and hearing their ideas is a good way to get input. I find these work best when you can bring a specific proposal, or set of proposals, for folks to react against. As a variant, we’ve also experimented with recurring customer advisory groups.
- **Cloud vendors** - with [cloud offerings rapidly expanding](https://lethain.com/physics-of-cloud-expansion/), it’s useful to take a look at what new related offerings have popped up on AWS, Azure, GCP and such.
- **Peer-company chats** - in addition to chatting with your users, I’ve found it equally valuable to chat with a wide variety of folks working on similar problems at other companies to understand how they’re thinking about things. I’ve heard some extremely valid concerns around this leads directly to cargo-culting and groupthink, which is why I think it’s so important to decouple discovery from prioritization: listen to what other folks are doing, don’t automatically decide to adopt them.
- **Academic and industry research** - look for papers coming out of the big tech companies (Google, Microsoft, Facebook, etc) and also for academic research on a given area. I did a basic example of this [when I surveyed load generation research](https://lethain.com/braindump-on-load-generation/), and it was helpful to expand my thinking beyond the obvious.
- **Open source** - [along the lines of my exploration of the open source data ecosystem](https://lethain.com/from-lambda-to-kappa-dataflow-paradigms/), I find it useful to spend some time understanding the state of open source for a given area. This gives you an easy sense of trends and how other folks see the space evolving.
- **Cloud vendors** - with [cloud offerings rapidly expanding](https://lethain.com/physics-of-cloud-expansion/), it’s useful to take a look at what new related offerings have popped up on AWS, Azure, GCP and such.
The goal of each of these is to collect a tremendous amount of information. My mental model of this phase is to load as much state into your head as possible, to give you a broad perspective as you move into prioritization. The goal of each of these is to collect a tremendous amount of information. My mental model of this phase is to load as much state into your head as possible, to give you a broad perspective as you move into prioritization.
@@ -40,32 +58,47 @@ The goal of each of these is to collect a tremendous amount of information. My m
Brimming with research and user needs, I try to create three artifacts: Brimming with research and user needs, I try to create three artifacts:
- A **charter** that identifies the unique value your team tries to provide, your competitive advantages and the strategy that you’ll employ. - A **charter** that identifies the unique value your team tries to provide, your competitive advantages and the strategy that you’ll employ.
- An **optimistic three-year vision** of the best possible system you’d like to be providing. Constrain the vision by what is possible, but don’t constraint it by what is reasonable. My rule of thumb is “where could we be if everything went perfectly for three years?”
- A **prioritized list of user pain** that captures your users’ active needs, and in particular buckets the concerns together into things that you might be able to address, eliminate or empower with a single solution. - An **optimistic three-year vision** of the best possible system you’d like to be providing. Constrain the vision by what is possible, but don’t constraint it by what is reasonable. My rule of thumb is “where could we be if everything went perfectly for three years?”
- A **prioritized list of user pain** that captures your users’ active needs, and in particular buckets the concerns together into things that you might be able to address, eliminate or empower with a single solution.
Your aim is to relieve as much user pain as possible while also making forward progress towards your optimistic vision. I think doing this well is the artistry of product management, and I believe each project can usually make significant progress towards both. Your aim is to relieve as much user pain as possible while also making forward progress towards your optimistic vision. I think doing this well is the artistry of product management, and I believe each project can usually make significant progress towards both.
Changing priorities late in a project is expensive, so we try to align on priorities as early as possible, and allow users as much input as possible early to weigh in on which parts will be useful to them, and in particular which parts can be used independently. A great list of priorities allows us to deliver incremental value to our users at each phase, not waiting until the last step to deliver a big bang of utility. Changing priorities late in a project is expensive, so we try to align on priorities as early as possible, and allow users as much input as possible early to weigh in on which parts will be useful to them, and in particular which parts can be used independently. A great list of priorities allows us to deliver incremental value to our users at each phase, not waiting until the last step to deliver a big bang of utility.
Sometimes current needs don’t align well with the future, or the future is a long ways away, and in that case my rule of thumb is to invest 70% of your effort on solving immediate user needs, and 30% on advancing towards the future. I advocate 70% towards immediate user needs, because I think folks tend to over index on the future, and this helps avoid falling into that trap. [In areas where cloud vendors are rapidly expanding](https://lethain.com/physics-of-cloud-expansion/), sometimes it may be reasonable to devote 100% of your efforts to immediate user pain, on the assumption that cloud offerings will be available in the next 2-3 years to absorb the operational load and technical debt of the existing solution. Some areas are not likely to get cloud investment in the near term, but increasingly we should be looking at areas we can strategically underinvest today on the assumption that the clouds will provide an easy solution soon. Sometimes current needs don’t align well with the future, or the future is a long ways away, and in that case my rule of thumb is to invest 70% of your effort on solving immediate user needs, and 30% on advancing towards the future. I advocate 70% towards immediate user needs, because I think folks tend to over index on the future, and this helps avoid falling into that trap. [In areas where cloud vendors are rapidly expanding](https://lethain.com/physics-of-cloud-expansion/), sometimes it may be reasonable to devote 100% of your efforts to immediate user pain, on the assumption that cloud offerings will be available in the next 2-3 years to absorb the operational load and technical debt of the existing solution. Some areas are not likely to get cloud investment in the near term, but increasingly we should be looking at areas we can strategically underinvest today on the assumption that the clouds will provide an easy solution soon.
## Solution validation ## Solution validation
Once you’ve decided what to focus on solving, a many teams immediately go “full waterfall”, designing a twelve month roadmap. Long-term planning is hard to resist, because pretty much every user and every planning process demands it of you, so you’ll probably end up writing an artifact of this nature, but I beg you not to pretend it means something. Once you’ve decided what to focus on solving, a many teams immediately go “full waterfall”, designing a twelve month roadmap. Long-term planning is hard to resist, because pretty much every user and every planning process demands it of you, so you’ll probably end up writing an artifact of this nature, but I beg you not to pretend it means something.
This isn’t because estimating is hard–although estimating is hard–but because it assumes that our solutions are actually good solutions. It’s my opinion that most solutions are, in fact, pretty bad. The secret is not to “get brilliant” and pick better solutions, but rather to get skeptical and to design your approach to validate the value and practicality of solutions as cheaply as possible. This isn’t because estimating is hard–although estimating is hard–but because it assumes that our solutions are actually good solutions. It’s my opinion that most solutions are, in fact, pretty bad. The secret is not to “get brilliant” and pick better solutions, but rather to get skeptical and to design your approach to validate the value and practicality of solutions as cheaply as possible.
I often assume that internal users have a great sense of existing capabilities, but that sometimes isn’t the case. Part of “getting skeptical” is evangelizing your existing solutions and seeing if there is already a reasonable way to solve the problem under discussion that maybe you haven’t documented well or requires some architectural familiarity between both the users and the providers to realize a solution is already available. I often assume that internal users have a great sense of existing capabilities, but that sometimes isn’t the case. Part of “getting skeptical” is evangelizing your existing solutions and seeing if there is already a reasonable way to solve the problem under discussion that maybe you haven’t documented well or requires some architectural familiarity between both the users and the providers to realize a solution is already available.
It’s a common pattern to do the hardest migration first, and I’m a big fan of that approach, but even then you’ve spent the majority of the time required to build a system by the time you’re validating it. Do experiments, gather data to prove the approach won’t work, validate the approach has worked for others. Try as hard as possible to prove your solution cannot work. It’s a common pattern to do the hardest migration first, and I’m a big fan of that approach, but even then you’ve spent the majority of the time required to build a system by the time you’re validating it. Do experiments, gather data to prove the approach won’t work, validate the approach has worked for others. Try as hard as possible to prove your solution cannot work.
Once you’ve validated a solution, you also want to keep in mind similar needs from other users. The pattern might look like doing one hard integration first, and then doing a staccato burst of easy integrations to smooth the edges and check for broad applicability. Once you’ve validated a solution, you also want to keep in mind similar needs from other users. The pattern might look like doing one hard integration first, and then doing a staccato burst of easy integrations to smooth the edges and check for broad applicability.
We’ve been experimenting with the pattern of embedding the team building a solution into the team they’re building the solution for, and that feels like a good strategy for quick iteration and avoiding falling in love with awesome things that are not necessarily useful things. We’ve been experimenting with the pattern of embedding the team building a solution into the team they’re building the solution for, and that feels like a good strategy for quick iteration and avoiding falling in love with awesome things that are not necessarily useful things.
## Closing ## Closing
There are many infrastructure engineering teams which don’t make the transition from maintenance to innovation, and some which intentionally decide against doing so. It’s an uncomfortable transition, but I’ve found it remarkably rewarding: more direct contribution to your coworkers’ success, more excitement from other leaders within the company, and shrugging off the mantle of cost center to become an acknowledged source of innovation. There are many infrastructure engineering teams which don’t make the transition from maintenance to innovation, and some which intentionally decide against doing so. It’s an uncomfortable transition, but I’ve found it remarkably rewarding: more direct contribution to your coworkers’ success, more excitement from other leaders within the company, and shrugging off the mantle of cost center to become an acknowledged source of innovation.
While I’m really excited at how this approach has helped us focus on directly supporting our users, it’s still an early approach. Three questions in particular continue to bounce around in my head: While I’m really excited at how this approach has helped us focus on directly supporting our users, it’s still an early approach. Three questions in particular continue to bounce around in my head:
1. User discovery takes up a bunch of time from other teams. How can we be more effective with their time? 1. User discovery takes up a bunch of time from other teams. How can we be more effective with their time?
2. How do we avoid falling in love with solutions, particularly for those that are difficult to validate early? 2. How do we avoid falling in love with solutions, particularly for those that are difficult to validate early?
3. How do other folks do this!? 3. How do other folks do this!?
If you’re doing something different, or even if you’re doing something similar, I’d love to hear from you! If you’re doing something different, or even if you’re doing something similar, I’d love to hear from you!
---
Thanks to [Amy](https://twitter.com/amyngyn), [Peter](https://twitter.com/peterseibel) and [Ranbir](https://twitter.com/TheRanbirChawla) for shaping this post. Thanks to [Amy](https://twitter.com/amyngyn), [Peter](https://twitter.com/peterseibel) and [Ranbir](https://twitter.com/TheRanbirChawla) for shaping this post.
Published on February 6, 2018. Published on February 6, 2018.

96
Reading/Read Later/Real World Recommendation System – Part 1.md
View File

@@ -1,92 +1,102 @@
Clipped from: [https://fennel.ai/blog/real-world-recommendation-system/](https://fennel.ai/blog/real-world-recommendation-system/) [[ReadItLater]] [[Article]]
# [Real World Recommendation System – Part 1](https://fennel.ai/blog/real-world-recommendation-system/)
Training a collaborative filtering based recommendation system on a toy dataset is a sophomore-year project in colleges these days. But where the rubber meets the road is building such a system at scale, deploying in production, and serving live requests within a few hundred milliseconds while the user is waiting for the page to load. To build a system like this, engineers have to make decisions spanning multiple moving layers like: Training a collaborative filtering based recommendation system on a toy dataset is a sophomore-year project in colleges these days. But where the rubber meets the road is building such a system at scale, deploying in production, and serving live requests within a few hundred milliseconds while the user is waiting for the page to load. To build a system like this, engineers have to make decisions spanning multiple moving layers like:
- High-level paradigms (like collaborative filtering, content based recommendations, vector search, model based recommendations) - High-level paradigms (like collaborative filtering, content based recommendations, vector search, model based recommendations)
- ML algorithms (e.g., GBDTs, SVD, Multi tower neural networks, etc.) - ML algorithms (e.g., GBDTs, SVD, Multi tower neural networks, etc.)
- Modeling libraries (e.g., PyTorch, Tensorflow, XGBoost) - Modeling libraries (e.g., PyTorch, Tensorflow, XGBoost)
- Data management (e.g., choice of DB, caching strategy, reuse primary database or copy all the data in another system optimized for recommendation workload, etc.) - Data management (e.g., choice of DB, caching strategy, reuse primary database or copy all the data in another system optimized for recommendation workload, etc.) [\[1\]](https://fennel.ai/blog/real-world-recommendation-system/#fn1)
- Feature management (e.g., offline vs online, precompute vs serve live) - Feature management (e.g., offline vs online, precompute vs serve live)
- Serving systems (performance, query latency, distribution model, fault tolerance, etc.) - Serving systems (performance, query latency, distribution model, fault tolerance, etc.)
- Deployment system (e.g., how does new code get updated, build steps, keeping caches working after processes restart, etc.) - Deployment system (e.g., how does new code get updated, build steps, keeping caches working after processes restart, etc.)
- Hardware (e.g., GPUs, SSDs) - Hardware (e.g., GPUs, SSDs)
No wonder architecting a system like this is a daunting task. Thankfully though, after years of trial and error, FAANG and other top tech companies have independently converged on a common architecture for building/deploying production-grade recommendation systems. Further, this architecture is domain/vertical agnostic and can power all sorts of applications under the sun — from e-commerce and feeds to search, notifications, email marketing, etc. No wonder architecting a system like this is a daunting task. Thankfully though, after years of trial and error, FAANG and other top tech companies have independently converged on a common architecture for building/deploying production-grade recommendation systems. Further, this architecture is domain/vertical agnostic and can power all sorts of applications under the sun — from e-commerce and feeds to search, notifications, email marketing, etc.
The goal of this publication is to start from the basics, explain nuances of all the moving layers, and describe this universal recommendation system architecture. The goal of this publication is to start from the basics, explain nuances of all the moving layers, and describe this universal recommendation system architecture.
We will start with a post to explain the serving side of this architecture at a high level, quickly followed by a post about the training side — these two posts will mostly outline the structure and identify the key scaling problems in serving and training, respectively. (Edit, we ended up writing two follow-ups to this post — part 2 about [training data generation here](https://fennel.ai/blog/real-world-recommendation-systems/) and [part 3 about modeling here](https://fennel.ai/blog/real-world-recommendation-systems-21e/)). Future posts will go through these scaling issues one by one and describe how they are typically solved, along with the best practices developed over years of learning. So let’s get started: We will start with a post to explain the serving side of this architecture at a high level, quickly followed by a post about the training side — these two posts will mostly outline the structure and identify the key scaling problems in serving and training, respectively. (Edit, we ended up writing two follow-ups to this post — part 2 about [training data generation here](https://fennel.ai/blog/real-world-recommendation-systems/) and [part 3 about modeling here](https://fennel.ai/blog/real-world-recommendation-systems-21e/)). Future posts will go through these scaling issues one by one and describe how they are typically solved, along with the best practices developed over years of learning. So let’s get started:
Modern recommendation systems are composed of eight (somewhat overlapping) logical stages: Modern recommendation systems are composed of eight (somewhat overlapping) logical stages:
1. Retrieval 1. Retrieval
2. Filtering 2. Filtering
3. Feature Extraction 3. Feature Extraction
4. Scoring 4. Scoring
5. Ranking 5. Ranking
6. Feature Logging 6. Feature Logging
7. Training Data Generation 7. Training Data Generation
8. Model Training 8. Model Training
The first five of these are related to serving, and the last three are related to training. The first five of these are related to serving, and the last three are related to training.
![https://www.fennel.ai/blog/content/images/2022/10/1cacae00-cc59-46b1-bac2-71eeb72828ac_960x720-1.jpeg](Exported%20image%2020240808113928-0.jpeg) ![https://www.fennel.ai/blog/content/images/2022/10/1cacae00-cc59-46b1-bac2-71eeb72828ac_960x720-1.jpeg](Reading/Read%20Later/assets/httpswww.fennel.aiblogcontentimages2022101cacae00-cc59-46b1-bac2-71eeb72828ac_960x720-1.jpeg)
Let’s go through all the serving layers one by one: Let’s go through all the serving layers one by one:
## **1. Retrieval** ## **1\. Retrieval**
Products like Facebook have millions of things to show in any recommendation unit - so many that it’s physically impossible to score all of them using any ML model while the user is waiting for their feed to load. So instead of scoring each item in the inventory, a more manageable subset of the inventory is first obtained via a process called Retrieval or “Candidate Generation” (since it generates candidates for ranking). Products like Facebook have millions of things to show in any recommendation unit - so many that it’s physically impossible to score all of them using any ML model while the user is waiting for their feed to load. So instead of scoring each item in the inventory, a more manageable subset of the inventory is first obtained via a process called Retrieval or “Candidate Generation” (since it generates candidates for ranking).
Retrieval is not just a FAANG scale problem — since the user is waiting for the “page” to load, most recommendation requests have a budget of only 500ms or so, and it is only possible to score a few hundred items in a request. As a result, whenever the inventory is a couple thousand items or more (which covers a large % of all real-world systems), a retrieval phase is needed. Retrieval is not just a FAANG scale problem — since the user is waiting for the “page” to load, most recommendation requests have a budget of only 500ms or so, and it is only possible to score a few hundred items in a request. As a result, whenever the inventory is a couple thousand items or more (which covers a large % of all real-world systems), a retrieval phase is needed.
How does Retrieval work? Retrieval is done by writing a few heuristics, also called as “candidate generators” or simply “generators”, each of which selects, say a dozen or so distinct candidates. Some common examples of generators are: How does Retrieval work? Retrieval is done by writing a few heuristics, also called as “candidate generators” or simply “generators”, each of which selects, say a dozen or so distinct candidates. Some common examples of generators are:
- Content that is trending in a user’s geography in the last x hours - Content that is trending in a user’s geography in the last x hours
- Recent content from authors/topics that the user explicitly “follows” - Recent content from authors/topics that the user explicitly “follows”
- Find 5 contents that user “liked” in the past, and for each such content, find 5 more “related” items - Find 5 contents that user “liked” in the past, and for each such content, find 5 more “related” items
- Find the most relevant topics for a user and find the freshest content from each of the topics. - Find the most relevant topics for a user and find the freshest content from each of the topics.
Retrieval can be powered by ML (e.g., trained embeddings), but more often than not, a larger % of generators are mere heuristics that encode some “product thinking” about what content is likely to create a good recommendation experience. And by writing a few of these and taking a union of all their candidates, we ensure that the system is able to at least consider all sorts of interesting inventory. Retrieval has only two jobs — 1) get all the interesting things (or at least as many as possible) and 2) get as few total things as possible so that we can score/examine each candidate using the power of ML. Retrieval can be powered by ML (e.g., trained embeddings), but more often than not, a larger % of generators are mere heuristics that encode some “product thinking” about what content is likely to create a good recommendation experience. And by writing a few of these and taking a union of all their candidates, we ensure that the system is able to at least consider all sorts of interesting inventory. Retrieval has only two jobs — 1) get all the interesting things (or at least as many as possible) [\[2\]](https://fennel.ai/blog/real-world-recommendation-system/#fn2) and 2) get as few total things as possible so that we can score/examine each candidate using the power of ML.
## 2. Filtering ## 2\. Filtering
After retrieving a few hundred candidates, recommendation systems typically filter out “invalid inventory”. For instance, if you’re building a social network, you might want to filter out things that are likely to be spammy. Or if you are building a video OTT platform, you may have to do some geo-licensing-based filtering. Or if you’re building an e-commerce product, you may have to filter things that are out of stock. Filters can also be extremely personalized, some examples: After retrieving a few hundred candidates, recommendation systems typically filter out “invalid inventory”. For instance, if you’re building a social network, you might want to filter out things that are likely to be spammy. Or if you are building a video OTT platform, you may have to do some geo-licensing-based filtering. Or if you’re building an e-commerce product, you may have to filter things that are out of stock. Filters can also be extremely personalized, some examples:
1. Some products try to filter out content that the user has already seen before 1. Some products try to filter out content that the user has already seen before
2. Some products expose some controls to the users to hide away topics or authors or other sources of content
In short, most real-world recommendation systems develop a long list of filters over time, which once again encode some product thinking about what creates a good experience. 2. Some products expose some controls to the users to hide away topics or authors or other sources of content
In short, most real-world recommendation systems develop a long list of filters over time, which once again encode some product thinking about what creates a good experience. [\[3\]](https://fennel.ai/blog/real-world-recommendation-system/#fn3)
Filtering and retrieval have a very interesting relationship. Some filters are pushed down to the generators themselves — for instance, if you’re building a dating product, filters for location and sexual preferences may be a part of each generator itself. But more often than not, it is physically impossible to have each generator respect each filter at the source, and so a whole layer of filtering is needed. Filtering and retrieval have a very interesting relationship. Some filters are pushed down to the generators themselves — for instance, if you’re building a dating product, filters for location and sexual preferences may be a part of each generator itself. But more often than not, it is physically impossible to have each generator respect each filter at the source, and so a whole layer of filtering is needed.
## 3. Feature Extraction
After filtering, we have a slightly smaller list of candidates — but it’s still going to be a couple hundred candidates long. We somehow need to choose the top ten items to show to the user. As you can imagine, this is going to involve some sort of scoring — for instance, in a job portal, we may want to compute how close is the job’s salary range to the user’s desired salary range. After filtering, we have a slightly smaller list of candidates — but it’s still going to be a couple hundred candidates long. We somehow need to choose the top ten items to show to the user. As you can imagine, this is going to involve some sort of scoring — for instance, in a job portal, we may want to compute how close is the job’s salary range to the user’s desired salary range.
But before any scoring can even begin, we need to obtain a bunch of data about each item that is going to be scored. In the job portal example, we will need the salary range of each candidate's job. Such signals about candidates are called “features”. Features are not just about the candidates but also include data about the user (e.g., user’s desired salary range). In fact, some of the most important features in literally every single recommendation system are those that capture users’ interaction behavior with potential candidates - this is so important and so nuanced that we will dedicate a whole post on this topic in this blog soon. Either way, we have a few hundred candidates, and we extract a bunch of features (which are basically just pieces of data) about the candidates and the user. Usually, we get anywhere between a few dozen to hundreds of features per candidate. But before any scoring can even begin, we need to obtain a bunch of data about each item that is going to be scored. In the job portal example, we will need the salary range of each candidate's job. Such signals about candidates are called “features”. Features are not just about the candidates but also include data about the user (e.g., user’s desired salary range). In fact, some of the most important features in literally every single recommendation system are those that capture users’ interaction behavior with potential candidates - this is so important and so nuanced that we will dedicate a whole post on this topic in this blog soon. Either way, we have a few hundred candidates, and we extract a bunch of features (which are basically just pieces of data) about the candidates and the user. Usually, we get anywhere between a few dozen to hundreds of features per candidate.
It is worth pausing here and letting the scale sink in for a minute - we have a few hundred candidates, say 1000, and we get a few dozen features about each candidate, say 100 — we need to fetch 1000x100 or 100K pieces of data from some database in only 500ms (the latency budget of a recommendation request). And note that you don’t even have to be at FAANG scale to run into this problem - even if you have a small inventory (say a few thousand items) and a few dozen features, you’d still run into this problem. And fetching and computing so much data is an incredibly hard infra problem to solve and as a result, creates huge limitations on how “expressive” the features can actually be. It is worth pausing here and letting the scale sink in for a minute - we have a few hundred candidates, say 1000, and we get a few dozen features about each candidate, say 100 — we need to fetch 1000x100 or 100K pieces of data from some database in only 500ms (the latency budget of a recommendation request). And note that you don’t even have to be at FAANG scale to run into this problem - even if you have a small inventory (say a few thousand items) and a few dozen features, you’d still run into this problem. And fetching and computing so much data is an incredibly hard infra problem to solve and as a result, creates huge limitations on how “expressive” the features can actually be.
## 4. Scoring ## 4\. Scoring
So far, we have narrowed down the full inventory to a few hundred candidates and extracted a few dozen features about each. Now comes the bit where we use all the extracted features to assign a score to each candidate. In the simplest systems, the scoring phase is pretty rudimentary, often just a handcrafted formula that mixes a bunch of features of interest (e.g “let’s divide the number of likes by the number of impressions and give a boost by doubling the score if the user follows the content author”). But very soon, these handcrafted formulae and rules start hitting “corner cases” and creating bad experiences. That is where ML kicks in - a machine learning model is trained that takes in all these dozens of features and spits out a score (details on how such a model is trained to be covered in the next post). So far, we have narrowed down the full inventory to a few hundred candidates and extracted a few dozen features about each. Now comes the bit where we use all the extracted features to assign a score to each candidate. In the simplest systems, the scoring phase is pretty rudimentary, often just a handcrafted formula that mixes a bunch of features of interest (e.g “let’s divide the number of likes by the number of impressions and give a boost by doubling the score if the user follows the content author”). But very soon, these handcrafted formulae and rules start hitting “corner cases” and creating bad experiences. That is where ML kicks in - a machine learning model is trained that takes in all these dozens of features and spits out a score (details on how such a model is trained to be covered in the next post).
There are two key ideas that are very successful and present in the scoring of most real-world recommendation systems (and we’d write dedicated posts about both in the future — stay tuned): There are two key ideas that are very successful and present in the scoring of most real-world recommendation systems (and we’d write dedicated posts about both in the future — stay tuned):
1. Multi-stage scoring — not all ML models are equal, and some are lot “heavier” than others. And it is usually not possible to run the heaviest ML models on hundreds of candidates. So instead, scoring itself is broken down in two substages — 1st stage scoring (which uses a relatively lighter ML model like GBDTs on all 500 candidates and emits out, say, top 100 candidates) and the 2nd stage scoring, which runs the heavy model (say deep neural network) on just the top 100 candidates. 1. Multi-stage scoring — not all ML models are equal, and some are lot “heavier” than others. And it is usually not possible to run the heaviest ML models on hundreds of candidates. So instead, scoring itself is broken down in two substages — 1st stage scoring (which uses a relatively lighter ML model like GBDTs on all 500 candidates and emits out, say, top 100 candidates) and the 2nd stage scoring, which runs the heavy model (say deep neural network) on just the top 100 candidates.
2. Combining many models — ML models can only learn whatever we teach them to learn. And typically, they are taught to predict the probability of user engaging in a single action, say like. Sorting all content by what gets clicked is a good start but has lots of issues — for instance, it might only distribute clickbaity content. To make the recommendations more balanced, usually, multiple models are trained - say, one for predicting clicks, one for predicting comments, one for user reporting the content, etc. And the final score of a candidate is a weighted average of all these models. While this makes the recommendations better, this can also increase the amount of computation that needs to be done. 2. Combining many models — ML models can only learn whatever we teach them to learn. And typically, they are taught to predict the probability of user engaging in a single action, say like. Sorting all content by what gets clicked is a good start but has lots of issues — for instance, it might only distribute clickbaity content. To make the recommendations more balanced, usually, multiple models are trained - say, one for predicting clicks, one for predicting comments, one for user reporting the content, etc. And the final score of a candidate is a weighted average of all these models. While this makes the recommendations better, this can also increase the amount of computation that needs to be done. [\[4\]](https://fennel.ai/blog/real-world-recommendation-system/#fn4)
## 5. Ranking ## 5\. Ranking
Once scores have been computed for every candidate, the system moves on to the very last step — ranking. In the simplest systems, this stage is as simple as sorting all the candidates on their scores and just taking the top K. But in more complicated systems, the scores themselves are perturbed using non-ML business rules. For instance, it is a common requirement across many products to diversify the results a bit — for instance, not show content from the same publisher/author one after another. There are many algorithms for such diversification, but most of them operate in a similar fashion by adjusting the scores to respect the diversity (e.g., demote scores if successive items are not diverse enough). Once scores have been computed for every candidate, the system moves on to the very last step — ranking. In the simplest systems, this stage is as simple as sorting all the candidates on their scores and just taking the top K. But in more complicated systems, the scores themselves are perturbed using non-ML business rules. For instance, it is a common requirement across many products to diversify the results a bit — for instance, not show content from the same publisher/author one after another. There are many algorithms for such diversification, but most of them operate in a similar fashion by adjusting the scores to respect the diversity (e.g., demote scores if successive items are not diverse enough).
In addition to score perturbation, it is a common practice to run all the items against all the filters once again at this stage to avoid any embarrassing failures. For instance, maybe some of the candidate generators are a bit stale and don’t know that an item has gone out of stock — it’s better to filter it out here instead of sending an item to the user that they can’t even purchase. In addition to score perturbation, it is a common practice to run all the items against all the filters once again at this stage to avoid any embarrassing failures. For instance, maybe some of the candidate generators are a bit stale and don’t know that an item has gone out of stock — it’s better to filter it out here instead of sending an item to the user that they can’t even purchase.
Finally, once top K items are chosen, they are handed to some sort of “delivery” system which is responsible for things like pagination, caching, etc. Finally, once top K items are chosen, they are handed to some sort of “delivery” system which is responsible for things like pagination, caching, etc.
## Conclusion ## Conclusion
That’s it! These are the five serving stages of real-world recommendation systems. As you can see, even if a recommendation system is trained, deploying that in production is incredibly hard. In the next post, we will look at the training side of the recommendation system. And in the subsequent posts, we will go through all the infra/scaling issues outlined here and share how they are typically solved — stay tuned! That’s it! These are the five serving stages of real-world recommendation systems. As you can see, even if a recommendation system is trained, deploying that in production is incredibly hard. In the next post, we will look at the training side of the recommendation system. And in the subsequent posts, we will go through all the infra/scaling issues outlined here and share how they are typically solved — stay tuned!
1. Managing data is surprisingly hard for real-world recommendation systems because of extreme needs on all three of write throughput, read throughput, and read latencies. As a result, primary databases (e.g., MySQL, MongoDB, etc.) almost never work out of the box (unless you put in a lot of work to scale them in a clever way) [↩︎](https://fennel.ai/blog/real-world-recommendation-system/#fnref1) ---
2. Often bloom filters are used to filter out content already “seen” by the users [↩︎](https://fennel.ai/blog/real-world-recommendation-system/#fnref2)
3. This is actually fairly true across all the stages — 80% of decisions & iterations in building a recommendation system are all about product-specific needs & business rules, not machine learning. [↩︎](https://fennel.ai/blog/real-world-recommendation-system/#fnref3)
4. This technique is called “value modeling” - value model is an expression (typically weighted linear sum) of ML models with each term describing a specific kind of value to the user. [↩︎](https://fennel.ai/blog/real-world-recommendation-system/#fnref4)
### We publish high-quality technical content on all things machine learning. 1. Managing data is surprisingly hard for real-world recommendation systems because of extreme needs on all three of write throughput, read throughput, and read latencies. As a result, primary databases (e.g., MySQL, MongoDB, etc.) almost never work out of the box (unless you put in a lot of work to scale them in a clever way) [↩︎](https://fennel.ai/blog/real-world-recommendation-system/#fnref1)
_Subscribe to our blog to get updates delivered right to your inbox._ 2. Often bloom filters are used to filter out content already “seen” by the users [↩︎](https://fennel.ai/blog/real-world-recommendation-system/#fnref2)
Please wait... 3. This is actually fairly true across all the stages — 80% of decisions & iterations in building a recommendation system are all about product-specific needs & business rules, not machine learning. [↩︎](https://fennel.ai/blog/real-world-recommendation-system/#fnref3)
Please check your inbox and click the link
Please enter a valid email address! 4. This technique is called “value modeling” - value model is an expression (typically weighted linear sum) of ML models with each term describing a specific kind of value to the user. [↩︎](https://fennel.ai/blog/real-world-recommendation-system/#fnref4)

30
Reading/Read Later/What is Scrumban Definition, Overview, and Examples.md → Reading/Read Later/Scrumban.md
View File

@@ -1,24 +1,31 @@
Clipped from: [https://www.productplan.com/glossary/scrumban/](https://www.productplan.com/glossary/scrumban/) [[ReadItLater]] [[Article]]
# [Scrumban](https://www.productplan.com/glossary/scrumban/)
## What Is Scrumban? ## What Is Scrumban?
Scrumban is a project management framework that combines important features of two popular agile methodologies: Scrum and Kanban. The Scrumban framework merges the structure and predictable routines of Scrum with Kanban’s flexibility to make teams more agile, efficient, and productive. Scrumban is a project management framework that combines important features of two popular agile methodologies: Scrum and Kanban. The Scrumban framework merges the structure and predictable routines of Scrum with Kanban’s flexibility to make teams more agile, efficient, and productive.
For companies that implement Scrumban, the approach can help their teams focus on the correct strategic tasks while at the same time improving their processes. For companies that implement Scrumban, the approach can help their teams focus on the correct strategic tasks while at the same time improving their processes.
## How Does Scrumban Combine Scrum and Kanban? ## How Does Scrumban Combine Scrum and Kanban?
To understand how Scrumban merges Scrum and Kanban, we first need to understand each of these frameworks. To understand how Scrumban merges Scrum and Kanban, we first need to understand each of these frameworks.
[![Download the Agile Product Manager's Guide to Building Better Products ➜](https://no-cache.hubspot.com/cta/default/3434168/a4593ff5-0cd1-4437-86db-eff7703f0d47.png)](https://cta-redirect.hubspot.com/cta/redirect/3434168/a4593ff5-0cd1-4437-86db-eff7703f0d47)
## The Basics of Scrum ## The Basics of Scrum
![The Basics of Scrum](Exported%20image%2020240808113924-0.png) ![The Basics of Scrum](Reading/Read%20Later/assets/The%20Basics%20of%20Scrum.png)
A scrum is an [agile](https://www.productplan.com/agile-product-management/) approach used in software development. With Scrum, a team organizes itself into specific roles, including a Scrum master, product owner, and the rest of the Scrum team. The team breaks its workload into short timeframes called [sprints](https://www.productplan.com/glossary/sprint/). Each sprint lasts two weeks or one month. A scrum is an [agile](https://www.productplan.com/agile-product-management/) approach used in software development. With Scrum, a team organizes itself into specific roles, including a Scrum master, product owner, and the rest of the Scrum team. The team breaks its workload into short timeframes called [sprints](https://www.productplan.com/glossary/sprint/). Each sprint lasts two weeks or one month.
During a sprint, the developers work only on the tasks the team agreed to during the [sprint meeting](https://www.productplan.com/glossary/sprint-planning/). Before the next sprint, the team holds another sprint meeting and decides which items to work on next. Scrum teams also meet each morning for short [standups](https://www.productplan.com/glossary/standup/) to discuss the day’s tasks. During a sprint, the developers work only on the tasks the team agreed to during the [sprint meeting](https://www.productplan.com/glossary/sprint-planning/). Before the next sprint, the team holds another sprint meeting and decides which items to work on next. Scrum teams also meet each morning for short [standups](https://www.productplan.com/glossary/standup/) to discuss the day’s tasks.
## The Basics of Kanban ## The Basics of Kanban
Kanban is a visual approach to managing a team’s workload. With this methodology, a team creates a Kanban board to visually display its workflow in columns—such as “Ready to Start,” “In Progress,” “Under Review,” and Completed.” Kanban is a visual approach to managing a team’s workload. With this methodology, a team creates a Kanban board to visually display its workflow in columns—such as “Ready to Start,” “In Progress,” “Under Review,” and Completed.”
As developers begin working on an item, they move a card (or sticky note) with the item’s name from the Ready-to-Start column to In-Progress. If an item needs to move backward, from Under Review back to In-Progress—the team can move that card back to the In-Progress column. The Kanban board makes it easy for everyone to view and update the status of each project quickly. As developers begin working on an item, they move a card (or sticky note) with the item’s name from the Ready-to-Start column to In-Progress. If an item needs to move backward, from Under Review back to In-Progress—the team can move that card back to the In-Progress column. The Kanban board makes it easy for everyone to view and update the status of each project quickly.
## The Basics of Scrumban ## The Basics of Scrumban
@@ -27,9 +34,10 @@ Scrumban merges the structure and predictability of Scrum with Kanban’s flexib
## How Does Scrumban Work? (A Step-by-Step Guide) ## How Does Scrumban Work? (A Step-by-Step Guide)
![How Does Scrumban Work | ProductPlan](Exported%20image%2020240808113924-1.png) ![How Does Scrumban Work | ProductPlan](Reading/Read%20Later/assets/How%20Does%20Scrumban%20Work%20%20ProductPlan.png)
Scrumban involves applying Kanban principles—visualization of workflow, and flexible processes—to a team’s Scrum framework. But, Scrumban removed some of the more rigid aspects of Scrum and left each team to create a custom approach to development. Scrumban involves applying Kanban principles—visualization of workflow, and flexible processes—to a team’s Scrum framework. But, Scrumban removed some of the more rigid aspects of Scrum and left each team to create a custom approach to development.
Here is a step-by-step guide to developing a Scrumban framework for your team. Here is a step-by-step guide to developing a Scrumban framework for your team.
### Step 1: Develop a Scrumban board ### Step 1: Develop a Scrumban board
@@ -51,16 +59,24 @@ Because each sprint has a strict time limit, and the team can work on only a pre
### Step 5: Set your daily meetings ### Step 5: Set your daily meetings
Although you won’t have most of the meetings typical of the Scrum framework—sprint planning, sprint review, retrospective—Scrumban meetings can include short standups for the team to discuss their plans and challenges for the upcoming day. These short meetings are also a good way to encourage team bonding and cohesion because your developers will spend a lot of time working individually on their tasks and might not have much time for interaction otherwise. Although you won’t have most of the meetings typical of the Scrum framework—sprint planning, sprint review, retrospective—Scrumban meetings can include short standups for the team to discuss their plans and challenges for the upcoming day. These short meetings are also a good way to encourage team bonding and cohesion because your developers will spend a lot of time working individually on their tasks and might not have much time for interaction otherwise.
[![Download Strategic Project Alignment in an Agile World  ➜](Reading/Read%20Later/assets/Download%20Strategic%20Project%20Alignment%20in%20an%20Agile%20World%20%20➜.png)](https://cta-redirect.hubspot.com/cta/redirect/3434168/bfb5032e-5746-4c05-9f2a-54b36ba0e871)
### [Get Strategic Project Alignment ➜](https://go.productplan.com/cs/c/?cta_guid=5e81c0e5-ef7c-4934-9a09-1ffdb75ef256&signature=AAH58kEKHQbQJW6629FIXlrtbkdl61qPTg&placement_guid=bfb5032e-5746-4c05-9f2a-54b36ba0e871&click=50d2ee6d-3f9f-4e1f-9706-42febdab50e8&hsutk=&canon=https%3A%2F%2Fwww.productplan.com%2Fglossary%2Fscrumban%2F&portal_id=3434168&redirect_url=APefjpH62_akcPKt5r-wAyIGTPz-MzD-4jlmpUTA08t06kdMM_hG7YDHiJQk--6f3mPj8oHX8vTPELynngVJ9O9d_7KIARUO-JocvlEzt2M9qjELNmVc2lpJJElNxTAKuqZh8mlczCZJFlgGsHThwFBq0XDen52xBw)
## When Should a Team Use Scrumban? ## When Should a Team Use Scrumban?
A team can benefit from the Scrumban approach under several circumstances. For example: A team can benefit from the Scrumban approach under several circumstances. For example:
**1.** **For maintenance of ongoing projects.** **1.** **For maintenance of ongoing projects.**
These could include projects in which, unlike a new product launch, there is no definitive completion date for the work. These could include projects in which, unlike a new product launch, there is no definitive completion date for the work.
**2. For a team having trouble with Scrum.**
**2\. For a team having trouble with Scrum.**
It can happen for several reasons. For example, the company doesn’t have enough resources to support a Scrum environment, or the team finds Scrum’s requirements too rigid. It can happen for several reasons. For example, the company doesn’t have enough resources to support a Scrum environment, or the team finds Scrum’s requirements too rigid.
**3. When a company wants to give its team more flexibility in how it works.**
**3\. When a company wants to give its team more flexibility in how it works.**
With Scrum, the team often assigns specific tasks to individuals for each sprint. But Scrumban only sets a broad list of projects and lets the team itself determine how best to leverage its resources. It enhances teamwork and enables individuals in the company to find the projects best suited to their skills and interests. With Scrum, the team often assigns specific tasks to individuals for each sprint. But Scrumban only sets a broad list of projects and lets the team itself determine how best to leverage its resources. It enhances teamwork and enables individuals in the company to find the projects best suited to their skills and interests.
**Related Terms:** [Scrum agile framework](https://www.productplan.com/glossary/scrum-agile-framework/) / [Kanban board](https://www.productplan.com/glossary/kanban-board/) / [Kanban roadmap](https://www.productplan.com/glossary/kanban-roadmap/) / [Scrum master](https://www.productplan.com/glossary/scrum-master/) / [LeSS (Large Scale Scrum)](https://www.productplan.com/glossary/less-large-scale-scrum/) **Related Terms:** [Scrum agile framework](https://www.productplan.com/glossary/scrum-agile-framework/) / [Kanban board](https://www.productplan.com/glossary/kanban-board/) / [Kanban roadmap](https://www.productplan.com/glossary/kanban-roadmap/) / [Scrum master](https://www.productplan.com/glossary/scrum-master/) / [LeSS (Large Scale Scrum)](https://www.productplan.com/glossary/less-large-scale-scrum/)
**Learn More Powerful Agile Principles: [![Read the Agile Product Manager's Guide to Building Better Roadmaps](Reading/Read%20Later/assets/Read%20the%20Agile%20Product%20Manager's%20Guide%20to%20Building%20Better%20Roadmaps.png)](https://cta-redirect.hubspot.com/cta/redirect/3434168/f7b97c22-2e32-45da-99f7-1ddcb66e57d3)**

14
Reading/Read Later/StackExchange.RedisThreadTheft.md at main · StackExchangeStackExchange.Redis · GitHub.md
View File

@@ -1,14 +0,0 @@
Clipped from: [https://github.com/StackExchange/StackExchange.Redis/blob/main/docs/ThreadTheft.md](https://github.com/StackExchange/StackExchange.Redis/blob/main/docs/ThreadTheft.md)
If you're here because you followed a link in an exception and you just want your code to work, the short version is: try adding the following _early on_ in your application startup:
ConnectionMultiplexer.SetFeatureFlag("preventthreadtheft", true);
and see if that fixes things. If you want more context as to what this is about - keep reading!
Behind the scenes, for each connection to redis, StackExchange.Redis keeps a queue of the commands that we've sent to redis that are awaiting a reply. As each reply comes in we look at the next pending command (order is preserved, which keeps things simple), and we trigger the "here's your result" API for that reply. For async/await code, this then leads to your "continuation" becoming reactivated, which is how your code comes back to life when an await-ed task gets completed. That's the simple version, but reality is a bit more nuanced.
By _default_, when you trigger TrySetResult (etc) on a Task, the continuations are invoked _synchronously_, i.e. the thread that is setting the result now goes on immediately to run whatever it is that your continuation wanted. In our case, that would be very bad as that would mean that the dedicated reader loop (that is meant to be processing results from redis) is now running your application logic instead; this is **thread theft**, and would exhibit as lots of timeouts with rs: CompletePendingMessage in the information (rs is the **r**eader **s**tate; you shouldn't often observe it in the CompletePendingMessage* step, because it is meant to be very fast; if you are seeing it often it probably means that the reader is being hijacked when trying to set results).
To _avoid_ this, we use the TaskCreationOptions.RunContinuationsAsynchronously flag. What _this_ does depends a little on whether you have a SynchronizationContext. If you _don't_ (common for console applications, services, etc), then the TPL uses the standard thread-pool mechanisms to schedule the continuation. If you _do_ have a SynchronizationContext (common in UI applications and web-servers), then its Post method is used instead; the Post method is _meant_ to be an asynchronous dispatch API. But... not all implementations are equal. Some SynchronizationContext implementations treat Post as a synchronous invoke. This is true in particular of LegacyAspNetSynchronizationContext, which is what you get if you configure ASP.NET with:
<add key="aspnet:UseTaskFriendlySynchronizationContext" value="false" />
or if you do _not_ have a <httpRuntime targetFramework="..." /> of at least 4.5 (which causes the above to default true) like this:
<httpRuntime targetFramework="4.5" />
([citation](https://devblogs.microsoft.com/aspnet/all-about-httpruntime-targetframework))
In these scenarios, we would once again end up with the reader being stolen and used for processing your application logic. This can doom any further awaits to timeouts, either temporarily (until the application logic chooses to release the thread), or permanently (essentially deadlocking yourself).
To avoid this, the library includes an additional layer of mistrust; specifically, if the preventthreadtheft feature flag is enabled, we will _pre-emptively_ queue the completions on the thread-pool. This is a little less efficient in the _default_ case, but _if and only if_ you have a misbehaving SynchronizationContext, this is both appropriate and necessary, and does not represent additional overhead.
The library will attempt to detect LegacyAspNetSynchronizationContext in particular, but this is not always reliable. The flag is also available for manual use with other similar scenarios.

43
Reading/Read Later/StackExchange.RedisdocsThreadTheft.md at main · StackExchangeStackExchange.Redis.md Normal file
View File

@@ -0,0 +1,43 @@
[[ReadItLater]] [[Article]]
# [StackExchange.Redis/docs/ThreadTheft.md at main · StackExchange/StackExchange.Redis](https://github.com/StackExchange/StackExchange.Redis/blob/main/docs/ThreadTheft.md)
## Thread Theft
[](https://github.com/StackExchange/StackExchange.Redis/blob/main/docs/ThreadTheft.md#thread-theft)
If you're here because you followed a link in an exception and you just want your code to work, the short version is: try adding the following *early on* in your application startup:
```cs
ConnectionMultiplexer.SetFeatureFlag("preventthreadtheft", true);
```
and see if that fixes things. If you want more context as to what this is about - keep reading!
## What is thread theft?
[](https://github.com/StackExchange/StackExchange.Redis/blob/main/docs/ThreadTheft.md#what-is-thread-theft)
Behind the scenes, for each connection to redis, StackExchange.Redis keeps a queue of the commands that we've sent to redis that are awaiting a reply. As each reply comes in we look at the next pending command (order is preserved, which keeps things simple), and we trigger the "here's your result" API for that reply. For `async`/`await` code, this then leads to your "continuation" becoming reactivated, which is how your code comes back to life when an `await`\-ed task gets completed. That's the simple version, but reality is a bit more nuanced.
By *default*, when you trigger `TrySetResult` (etc) on a `Task`, the continuations are invoked *synchronously*, i.e. the thread that is setting the result now goes on immediately to run whatever it is that your continuation wanted. In our case, that would be very bad as that would mean that the dedicated reader loop (that is meant to be processing results from redis) is now running your application logic instead; this is **thread theft**, and would exhibit as lots of timeouts with `rs: CompletePendingMessage` in the information (`rs` is the **r**eader **s**tate; you shouldn't often observe it in the `CompletePendingMessage*` step, because it is meant to be very fast; if you are seeing it often it probably means that the reader is being hijacked when trying to set results).
To *avoid* this, we use the `TaskCreationOptions.RunContinuationsAsynchronously` flag. What *this* does depends a little on whether you have a `SynchronizationContext`. If you *don't* (common for console applications, services, etc), then the TPL uses the standard thread-pool mechanisms to schedule the continuation. If you *do* have a `SynchronizationContext` (common in UI applications and web-servers), then its `Post` method is used instead; the `Post` method is *meant* to be an asynchronous dispatch API. But... not all implementations are equal. Some `SynchronizationContext` implementations treat `Post` as a synchronous invoke. This is true in particular of `LegacyAspNetSynchronizationContext`, which is what you get if you configure ASP.NET with:
```xml
<add key="aspnet:UseTaskFriendlySynchronizationContext" value="false" />
```
or if you do *not* have a `<httpRuntime targetFramework="..." />` of at least 4.5 (which causes the above to default `true`) like this:
```xml
<httpRuntime targetFramework="4.5" />
```
([citation](https://devblogs.microsoft.com/aspnet/all-about-httpruntime-targetframework))
In these scenarios, we would once again end up with the reader being stolen and used for processing your application logic. This can doom any further `await`s to timeouts, either temporarily (until the application logic chooses to release the thread), or permanently (essentially deadlocking yourself).
To avoid this, the library includes an additional layer of mistrust; specifically, if the `preventthreadtheft` feature flag is enabled, we will *pre-emptively* queue the completions on the thread-pool. This is a little less efficient in the *default* case, but *if and only if* you have a misbehaving `SynchronizationContext`, this is both appropriate and necessary, and does not represent additional overhead.
The library will attempt to detect `LegacyAspNetSynchronizationContext` in particular, but this is not always reliable. The flag is also available for manual use with other similar scenarios.

51
Reading/Done Reading/Struggling Boy Finds Hope, Purpose, and Self-Worth through Trail Life Mentors.md → Reading/Read Later/Struggling Boy Finds Hope, Purpose, and Self-Worth through Trail Life Mentors.md
View File

@@ -1,21 +1,42 @@
Clipped from: [https://blog.traillifeusa.com/boy-finds-hope?utm_campaign=Raising%20Godly%20Boys&utm_medium=email&_hsmi=220415115&_hsenc=p2ANqtz-9wOKKcP8ywC8lxPGr9wAs3lzk6NsuStGaUd8Jw9bL9FQtvKma9aP_6LMzlBiba-jFHNF60TmvQ-95DT6tkKo_m8gvHig&utm_content=220415115&utm_source=hs_automation](https://blog.traillifeusa.com/boy-finds-hope?utm_campaign=Raising%20Godly%20Boys&utm_medium=email&_hsmi=220415115&_hsenc=p2ANqtz-9wOKKcP8ywC8lxPGr9wAs3lzk6NsuStGaUd8Jw9bL9FQtvKma9aP_6LMzlBiba-jFHNF60TmvQ-95DT6tkKo_m8gvHig&utm_content=220415115&utm_source=hs_automation) [[ReadItLater]] [[Article]]
- [x] Read
![Struggling Boy Finds Hope, Purpose, and Self-Worth through Trail Life Mentors](Exported%20image%2020240808113917-0%201.jpeg) # [Struggling Boy Finds Hope, Purpose, and Self-Worth through Trail Life Mentors](https://blog.traillifeusa.com/boy-finds-hope)
Stephen Ashton Jul 13, 2022 [5 Comments](https://blog.traillifeusa.com/boy-finds-hope#comments-listing)
***Trail Life mentors inspire young boy not only to find the will to live, but to help others around him***
**Trail Life mentors inspire young boy not only to find the will to live, but to help others around him**
 
In a society where the lines between masculine and feminine are constantly blurred, brash boyish bluster and boisterousness is losing its place. Feeling unappreciated, too boys are losing their identity. Traditional needs like physical challenge, competition, risk-taking, action, and adventure are being discounted and boys seeking to fill these needs are punished, diagnosed, or written off as unruly, difficult, or perhaps toxic. In a society where the lines between masculine and feminine are constantly blurred, brash boyish bluster and boisterousness is losing its place. Feeling unappreciated, too boys are losing their identity. Traditional needs like physical challenge, competition, risk-taking, action, and adventure are being discounted and boys seeking to fill these needs are punished, diagnosed, or written off as unruly, difficult, or perhaps toxic.
Trail Life USA, the largest Christ-centered, boy-focused scout-type organization in the country, is familiar with this struggle for boys and can provide a solution. In the fight to provide boys with unique programming that celebrates boyhood, Trail Life utilizes outdoor adventure and personal relationships to speak to the heart of a boy and and to guide him in his walk with Christ. Trail Life USA, the largest Christ-centered, boy-focused scout-type organization in the country, is familiar with this struggle for boys and can provide a solution. In the fight to provide boys with unique programming that celebrates boyhood, Trail Life utilizes outdoor adventure and personal relationships to speak to the heart of a boy and and to guide him in his walk with Christ.
Mark Hancock, Trail Life CEO, commented, _“When boys feel like they are relegated as ‘less than’ due to unappreciated gender differences, they begin to wonder where they belong in a society that seems to discount their abilities. Trail Life USA provides a boy-focused program and activities designed to let boys be boys, accentuating their strengths and allowing them to feel understood and appreciated.”_
_Hancock continued, “It seems everywhere a boy goes, he’s expected to comply with unrealistic social norms. The consistent message he gets is that he needs to ‘sit still, be quiet, do what he is told, and behave like the girls.’ But boys are not defective girls. They are created differently on purpose for a purpose. Properly channeled and intentionally challenged, the exuberance, drive, and daring of healthy boys is exactly what our society needs.”_
[![New call-to-action](Exported%20image%2020240808113917-1.jpeg)](https://blog.traillifeusa.com/cs/c/?cta_guid=ae0cca4b-12ee-44b5-9522-ea34bcd24e9e&signature=AAH58kHdz_P7HD13orXjY97P9tTqTGFjMw&pageId=77377023317&placement_guid=fe90369d-b6b8-4f68-9fc0-b6d63f14044c&click=19d2745d-b484-4784-8cde-ecde604c2e2a&hsutk=&canon=https%3A%2F%2Fblog.traillifeusa.com%2Fboy-finds-hope&portal_id=6459804&redirect_url=APefjpGnipquSbf2uJG8qs097FtoxGreH-0PhQnuOewllWKAO8ER1syjqeWtpneVenoGaHglMjK-_DfDkJUb2mKP7Sb3d5-H0WCHgYDmgDyzI-ZHx59iF80L-97IR8vDn99tCMt7ezx2Rj6p4CLK-nYdn7aZHiWVcw) Mark Hancock, Trail Life CEO, commented, *“When boys feel like they are relegated as ‘less than’ due to unappreciated gender differences, they begin to wonder where they belong in a society that seems to discount their abilities. Trail Life USA provides a boy-focused program and activities designed to let boys be boys, accentuating their strengths and allowing them to feel understood and appreciated.”*
In a system that fails to acknowledge that boys are not just like girls, [boys are increasingly diagnosed with disorders](https://www.understood.org/articles/en/do-boys-have-learning-and-thinking-differences-more-often-than-girls) and are [falling behind their female counterparts](https://www.brookings.edu/blog/up-front/2021/01/12/the-unreported-gender-gap-in-high-school-graduation-rates/) in nearly every academic category. This trend continues into [college where 60% of students are female](https://www.usatoday.com/story/opinion/2021/10/09/boys-falling-behind-how-schools-must-change-help-young-males/5913463001/). Even more tragic than the academic difficulty is the impact on the mental health of boys and men. Today, men account for [four out of five suicides in America](https://www.cdc.gov/nchs/products/databriefs/db373.htm#:~:text=In%2520both%2520urban%2520and%2520rural%2520areas%252C%2520suicide%2520rates%2520for%2520males,(30.7%2520compared%2520with%25208.0).) and twice the [drug-related deaths](https://news.wttw.com/2021/10/25/us-overdose-deaths-surge-all-time-high) as compared to women. The most rapidly growing suicide rate demographic is boys from the [ages of ten to 14](https://www.bloomberg.com/news/articles/2021-11-03/u-s-suicides-fall-for-second-year-in-a-row-during-pandemic). One mother recently wrote to share how her son was almost one of these statistics: *Hancock continued, “It seems everywhere a boy goes, he’s expected to comply with unrealistic social norms. The consistent message he gets is that he needs to ‘sit still, be quiet, do what he is told, and behave like the girls.’ But boys are not defective girls. They are created differently on purpose for a purpose. Properly channeled and intentionally challenged, the exuberance, drive, and daring of healthy boys is exactly what our society needs.”*
_“Dealing with ADHD, depression, and anxiety, my son was struggling at school and at home. Daily outbursts, disciplinary problems, and panic attacks forced us as parents to make the hard decision to pull him out of traditional school. A constant cycle of being in trouble with teachers, church leaders, and parents left him questioning his own value. At ten years of age, he decided that he was so broken and hopeless that the world would be better off without him. Last October, he attempted to take his own life and was admitted to the emergency room at the children’s hospital because of injuries sustained in that attempt_.
_“Then my son began attending_ _Trail Life__. It was the first time that he felt understood and accepted by authority figures (the_ **Trail Life** _leaders). In a world that had always attempted to squash his character, he has been encouraged to see a purpose in his ‘boy-ness.’ He has heard consistently that God created him the way he is and that he is loved. He has been encouraged that the future holds a purpose for him — one that will utilize his courage, his sensitivity, and his passion. He has been inspired intellectually and spiritually. Most importantly, he feels part of a community where he respects and admires the leaders, and has friends.”_  ***[![New call-to-action](Reading/Read%20Later/assets/New%20call-to-action.png)](https://cta-redirect.hubspot.com/cta/redirect/6459804/fe90369d-b6b8-4f68-9fc0-b6d63f14044c)***
In the active learning environment at **Trail Life**, her son has been able to shine. In the past six months, he has been hiking and camping, learned first aid, become proficient in lighting a fire without a match, and built rockets with his Troop. 
The mother commented on these skills her son learned at Trail Life, stating, _“Because my son learned first aid with his Troop, I still have both of my children. Last Sunday, I was driving on the freeway when my daughter, who is two years old, choked on a snack. She couldn’t breathe. I was stuck driving in traffic in the HOV lane, where I couldn’t pull over. My son was able to use the skills he learned at Trail Life_ _to give his sister the Heimlich maneuver, and she coughed up food and was able to breathe. He saved her life and displayed the level-headed, calm, and confident skills he needed to save his sister. I am so proud of him, and so thankful for Trail Life."_ In a system that fails to acknowledge that boys are not just like girls, [boys are increasingly diagnosed with disorders](https://www.understood.org/articles/en/do-boys-have-learning-and-thinking-differences-more-often-than-girls) and are [falling behind their female counterparts](https://www.brookings.edu/blog/up-front/2021/01/12/the-unreported-gender-gap-in-high-school-graduation-rates/) in nearly every academic category. This trend continues into [college where 60% of students are female](https://www.usatoday.com/story/opinion/2021/10/09/boys-falling-behind-how-schools-must-change-help-young-males/5913463001/). Even more tragic than the academic difficulty is the impact on the mental health of boys and men. Today, men account for [four out of five suicides in America](https://www.cdc.gov/nchs/products/databriefs/db373.htm#:~:text=In%2520both%2520urban%2520and%2520rural%2520areas%252C%2520suicide%2520rates%2520for%2520males,\(30.7%2520compared%2520with%25208.0\).) and twice the [drug-related deaths](https://news.wttw.com/2021/10/25/us-overdose-deaths-surge-all-time-high) as compared to women. The most rapidly growing suicide rate demographic is boys from the [ages of ten to 14](https://www.bloomberg.com/news/articles/2021-11-03/u-s-suicides-fall-for-second-year-in-a-row-during-pandemic). One mother recently wrote to share how her son was almost one of these statistics:
_“Even his father is seeing a tremendous difference. Before my son’s attempted suicide, they had become so estranged they were barely able to talk. When he was presented with the Life Saving Award from Trail Life,_ _his dad was able to attend and publicly commend his son. The young man who had been labeled as troublesome, difficult, and delinquent by so many is now being commended, affirmed, accepted, and encouraged by his Troop, his community, and his father.”_
_“Many sons never hear words of affirmation, acceptance, and encouragement from their father. I want to thank the men of the Troop who have poured themselves into my son and our family. Because of your intervention and effort, a relationship that was broken has been dramatically healed and turned around.”_ *“Dealing with ADHD, depression, and anxiety, my son was struggling at school and at home. Daily outbursts, disciplinary problems, and panic attacks forced us as parents to make the hard decision to pull him out of traditional school. A constant cycle of being in trouble with teachers, church leaders, and parents left him questioning his own value. At ten years of age, he decided that he was so broken and hopeless that the world would be better off without him. Last October, he attempted to take his own life and was admitted to the emergency room at the children’s hospital because of injuries sustained in that attempt*.
The mother concluded, _“Trail Life is m__aking a real difference in the lives of boys and their fathers.”_
*“Then my son began attending* *Trail Life**. It was the first time that he felt understood and accepted by authority figures (the* ***Trail Life*** *leaders). In a world that had always attempted to squash his character, he has been encouraged to see a purpose in his ‘boy-ness.’ He has heard consistently that God created him the way he is and that he is loved. He has been encouraged that the future holds a purpose for him — one that will utilize his courage, his sensitivity, and his passion. He has been inspired intellectually and spiritually. Most importantly, he feels part of a community where he respects and admires the leaders, and has friends.”* 
In the active learning environment at ***Trail Life***, her son has been able to shine. In the past six months, he has been hiking and camping, learned first aid, become proficient in lighting a fire without a match, and built rockets with his Troop. 
The mother commented on these skills her son learned at Trail Life, stating, *“Because my son learned first aid with his Troop, I still have both of my children. Last Sunday, I was driving on the freeway when my daughter, who is two years old, choked on a snack. She couldn’t breathe. I was stuck driving in traffic in the HOV lane, where I couldn’t pull over. My son was able to use the skills he learned at Trail Life* *to give his sister the Heimlich maneuver, and she coughed up food and was able to breathe. He saved her life and displayed the level-headed, calm, and confident skills he needed to save his sister. I am so proud of him, and so thankful for Trail Life."*
*“Even his father is seeing a tremendous difference. Before my son’s attempted suicide, they had become so estranged they were barely able to talk. When he was presented with the Life Saving Award from Trail Life,* *his dad was able to attend and publicly commend his son. The young man who had been labeled as troublesome, difficult, and delinquent by so many is now being commended, affirmed, accepted, and encouraged by his Troop, his community, and his father.”*
*“Many sons never hear words of affirmation, acceptance, and encouragement from their father. I want to thank the men of the Troop who have poured themselves into my son and our family. Because of your intervention and effort, a relationship that was broken has been dramatically healed and turned around.”*
The mother concluded, *“Trail Life is m**aking a real difference in the lives of boys and their fathers.”*
[![Start a Troop](Reading/Read%20Later/assets/Start%20a%20Troop.png)](https://cta-redirect.hubspot.com/cta/redirect/6459804/6b97d556-3b4d-4b1c-bd0f-34ede9c766c8)   [![Find a Troop](Reading/Read%20Later/assets/Find%20a%20Troop.png)](https://cta-redirect.hubspot.com/cta/redirect/6459804/9420b3e5-7e8d-4eed-85d4-79dc384b6f91)
###### About the Author
![Stephen Ashton](Reading/Read%20Later/assets/Stephen%20Ashton.jpg)
Stephen Ashton is the National Director of Marketing for Trail Life USA and serves as an adjunct professor at Clarks Summit University and Anchor Christian University. Prior to his work at Trail Life, he spent 15 years working with at-risk youth in residential therapeutic wilderness programs and served as the Vice-President of the Wilderness Road Therapeutic Camping Association. An author and a speaker, he has written for journals and published a book chronicling the foundations of therapeutic camping. He frequently speaks on the topics of fatherhood, biblical masculinity, outdoor education, and wilderness therapy. Stephen lives in South Carolina with his wife and 4 sons.

710
Reading/Read Later/What We’ve Learned From A Year of Building with LLMs – Applied LLMs.md Normal file
View File

@@ -0,0 +1,710 @@
[[ReadItLater]] [[Article]]
# [What We’ve Learned From A Year of Building with LLMs – Applied LLMs](https://applied-llms.org/)
> Also published on O’Reilly Media in three parts: [Tactical](https://www.oreilly.com/radar/what-we-learned-from-a-year-of-building-with-llms-part-i/), [Operational](https://www.oreilly.com/radar/what-we-learned-from-a-year-of-building-with-llms-part-ii/), [Strategic](https://www.oreilly.com/radar/what-we-learned-from-a-year-of-building-with-llms-part-iii-strategy/) ([podcast](https://lu.ma/e8huz3s6)). Also translated to [Japanese](https://zenn.dev/seya/articles/12c67b5d80670a) and Chinese ([Parts 1](https://iangyan.github.io/2024/09/08/building-with-llms-part-1/), [2](https://iangyan.github.io/2024/10/05/building-with-llms-part-2/), [3](https://iangyan.github.io/2024/10/06/building-with-llms-part-3/)).
It’s an exciting time to build with large language models (LLMs). Over the past year, LLMs have become “good enough” for real-world applications. And they’re getting better and cheaper every year. Coupled with a parade of demos on social media, there will be an [estimated $200B investment in AI by 2025](https://www.goldmansachs.com/intelligence/pages/ai-investment-forecast-to-approach-200-billion-globally-by-2025.html). Furthermore, provider APIs have made LLMs more accessible, allowing everyone, not just ML engineers and scientists, to build intelligence into their products. Nonetheless, while the barrier to entry for building with AI has been lowered, creating products and systems that are effective—beyond a demo—remains deceptively difficult.
We’ve spent the past year building, and have discovered many sharp edges along the way. While we don’t claim to speak for the entire industry, we’d like to share what we’ve learned to help you avoid our mistakes and iterate faster. These are organized into three sections:
- [Tactical](https://applied-llms.org/#tactical-nuts--bolts-of-working-with-llms): Some practices for prompting, RAG, flow engineering, evals, and monitoring. Whether you’re a practitioner building with LLMs, or hacking on weekend projects, this section was written for you.
- [Operational](https://applied-llms.org/#operation-day-to-day-and-org-concerns): The organizational, day-to-day concerns of shipping products, and how to build an effective team. For product/technical leaders looking to deploy sustainably and reliably.
- [Strategic](https://applied-llms.org/#strategy-building-with-llms-without-getting-out-maneuvered): The long-term, big-picture view, with opinionated takes such as “no GPU before PMF” and “focus on the system not the model”, and how to iterate. Written with founders and executives in mind.
We intend to make this a practical guide to building successful products with LLMs, drawing from our own experiences and pointing to examples from around the industry.
Ready to ~delve~ dive in? Let’s go.
---
## Tactical: Nuts & Bolts of Working with LLMs
Here, we share best practices for core components of the emerging LLM stack: prompting tips to improve quality and reliability, evaluation strategies to assess output, retrieval-augmented generation ideas to improve grounding, how to design human-in-the-loop workflows, and more. While the technology is still nascent, we trust these lessons are broadly applicable and can help you ship robust LLM applications.
## Prompting
We recommend starting with prompting when prototyping new applications. It’s easy to both underestimate and overestimate its importance. It’s underestimated because the right prompting techniques, when used correctly, can get us very far. It’s overestimated because even prompt-based applications require significant engineering around the prompt to work well.
### Focus on getting the most out of fundamental prompting techniques
A few prompting techniques have consistently helped with improving performance across a variety of models and tasks: n-shot prompts + in-context learning, chain-of-thought, and providing relevant resources.
The idea of in-context learning via n-shot prompts is to provide the LLM with examples that demonstrate the task and align outputs to our expectations. A few tips:
- If n is too low, the model may over-anchor on those specific examples, hurting its ability to generalize. As a rule of thumb, aim for n ≥ 5. Don’t be afraid to go as high as a few dozen.
- Examples should be representative of the prod distribution. If you’re building a movie summarizer, include samples from different genres in roughly the same proportion you’d expect to see in practice.
- You don’t always need to provide the input-output pairs; examples of desired outputs may be sufficient.
- If you plan for the LLM to use tools, include examples of using those tools.
In Chain-of-Thought (CoT) prompting, we encourage the LLM to explain its thought process before returning the final answer. Think of it as providing the LLM with a sketchpad so it doesn’t have to do it all in memory. The original approach was to simply add the phrase “Let’s think step by step” as part of the instructions, but, we’ve found it helpful to make the CoT more specific, where adding specificity via an extra sentence or two often reduces hallucination rates significantly.
For example, when asking an LLM to summarize a meeting transcript, we can be explicit about the steps:
- First, list out the key decisions, follow-up items, and associated owners in a sketchpad.
- Then, check that the details in the sketchpad are factually consistent with the transcript.
- Finally, synthesize the key points into a concise summary.
Note that in recent times, [some doubt](https://arxiv.org/abs/2405.04776) has been cast on if this technique is as powerful as believed. Additionally, there’s significant debate as to exactly what is going on during inference when Chain-of-Thought is being used. Regardless, this technique is one to experiment with when possible.
Providing relevant resources is a powerful mechanism to expand the model’s knowledge base, reduce hallucinations, and increase the user’s trust. Often accomplished via Retrieval Augmented Generation (RAG), providing the model with snippets of text that it can directly utilize in its response is an essential technique. When providing the relevant resources, it’s not enough to merely include them; don’t forget to tell the model to prioritize their use, refer to them directly, and to mention when none of the resources are sufficient. These help “ground” agent responses to a corpus of resources.
### Structure your inputs and outputs
Structured input and output help models better understand the input as well as return output that can reliably integrate with downstream systems. Adding serialization formatting to your inputs can help provide more clues to the model as to the relationships between tokens in the context, additional metadata to specific tokens (like types), or relate the request to similar examples in the model’s training data.
As an example, many questions on the internet about writing SQL begin by specifying the SQL schema. Thus, you can expect that effective prompting for Text-to-SQL should include [structured schema definitions](https://www.researchgate.net/publication/371223615_SQL-PaLM_Improved_Large_Language_ModelAdaptation_for_Text-to-SQL).
Structured input expresses tasks clearly and resembles how the training data is formatted, increasing the probability of better output. Structured output simplifies integration into downstream components of your system. [Instructor](https://github.com/jxnl/instructor) and [Outlines](https://github.com/outlines-dev/outlines) work well for structured output. (If you’re importing an LLM API SDK, use Instructor; if you’re importing Huggingface for a self-hosted model, use Outlines.)
When using structured input, be aware that each LLM family has their own preferences. Claude prefers `<xml>` while GPT favors Markdown and JSON. With XML, you can even pre-fill Claude’s responses by providing a `<response>` tag like so.
```
messages=[
{
"role": "user",
"content": """Extract the <name>, <size>, <price>, and <color> from this product description into your <response>.
<description>The SmartHome Mini is a compact smart home assistant available in black or white for only $49.99. At just 5 inches wide, it lets you control lights, thermostats, and other connected devices via voice or app—no matter where you place it in your home. This affordable little hub brings convenient hands-free control to your smart devices.
</description>"""
},
{
"role": "assistant",
"content": "<response><name>"
}
]
```
### Have small prompts that do one thing, and only one thing, well
A common anti-pattern / code smell in software is the “[God Object](https://en.wikipedia.org/wiki/God_object)”, where we have a single class or function that does everything. The same applies to prompts too.
A prompt typically starts simple: A few sentences of instruction, a couple of examples, and we’re good to go. But as we try to improve performance and handle more edge cases, complexity creeps in. More instructions. Multi-step reasoning. Dozens of examples. Before we know it, our initially simple prompt is now a 2,000 token Frankenstein. And to add injury to insult, it has worse performance on the more common and straightforward inputs! GoDaddy shared this challenge as their [No. 1 lesson from building with LLMs](https://www.godaddy.com/resources/news/llm-from-the-trenches-10-lessons-learned-operationalizing-models-at-godaddy#h-1-sometimes-one-prompt-isn-t-enough).
Just like how we strive (read: struggle) to keep our systems and code simple, so should we for our prompts. Instead of having a single, catch-all prompt for the meeting transcript summarizer, we can break it into steps:
- Extract key decisions, action items, and owners into structured format
- Check extracted details against the original transcription for consistency
- Generate a concise summary from the structured details
As a result, we’ve split our single prompt into multiple prompts that are each simple, focused, and easy to understand. And by breaking them up, we can now iterate and eval each prompt individually.
### Craft your context tokens
Rethink, and challenge your assumptions about how much context you actually need to send to the agent. Be like Michaelangelo, do not build up your context sculpture—chisel away the superfluous material until the sculpture is revealed. RAG is a popular way to collate all of the potentially relevant blocks of marble, but what are you doing to extract what’s necessary?
We’ve found that taking the final prompt sent to the model—with all of the context construction, and meta-prompting, and RAG results—putting it on a blank page and just reading it, really helps you rethink your context. We have found redundancy, self-contradictory language, and poor formatting using this method.
The other key optimization is the structure of your context. If your bag-of-docs representation isn’t helpful for humans, don’t assume it’s any good for agents. Think carefully about how you structure your context to underscore the relationships between parts of it and make extraction as simple as possible.
More [prompting fundamentals](https://eugeneyan.com/writing/prompting/) such as prompting mental model, prefilling, context placement, etc.
## Information Retrieval / RAG
Beyond prompting, another effective way to steer an LLM is by providing knowledge as part of the prompt. This grounds the LLM on the provided context which is then used for in-context learning. This is known as retrieval-augmented generation (RAG). Practitioners have found RAG effective at providing knowledge and improving output, while requiring far less effort and cost compared to finetuning.
### RAG is only as good as the retrieved documents’ relevance, density, and detail
The quality of your RAG’s output is dependent on the quality of retrieved documents, which in turn can be considered along a few factors
The first and most obvious metric is relevance. This is typically quantified via ranking metrics such as [Mean Reciprocal Rank (MRR)](https://en.wikipedia.org/wiki/Mean_reciprocal_rank) or [Normalized Discounted Cumulative Gain (NDCG)](https://en.wikipedia.org/wiki/Discounted_cumulative_gain). MRR evaluates how well a system places the first relevant result in a ranked list while NDCG considers the relevance of all the results and their positions. They measure how good the system is at ranking relevant documents higher and irrelevant documents lower. For example, if we’re retrieving user summaries to generate movie review summaries, we’ll want to rank reviews for the specific movie higher while excluding reviews for other movies.
Like traditional recommendation systems, the rank of retrieved items will have a significant impact on how the LLM performs on downstream tasks. To measure the impact, run a RAG-based task but with the retrieved items shuffled—how does the RAG output perform?
Second, we also want to consider information density. If two documents are equally relevant, we should prefer one that’s more concise and has fewer extraneous details. Returning to our movie example, we might consider the movie transcript and all user reviews to be relevant in a broad sense. Nonetheless, the top-rated reviews and editorial reviews will likely be more dense in information.
Finally, consider the level of detail provided in the document. Imagine we’re building a RAG system to generate SQL queries from natural language. We could simply provide table schemas with column names as context. But, what if we include column descriptions and some representative values? The additional detail could help the LLM better understand the semantics of the table and thus generate more correct SQL.
### Don’t forget keyword search; use it as a baseline and in hybrid search
Given how prevalent the embedding-based RAG demo is, it’s easy to forget or overlook the decades of research and solutions in information retrieval.
Nonetheless, while embeddings are undoubtedly a powerful tool, they are not the be-all and end-all. First, while they excel at capturing high-level semantic similarity, they may struggle with more specific, keyword-based queries, like when users search for names (e.g., Ilya), acronyms (e.g., RAG), or IDs (e.g., claude-3-sonnet). Keyword-based search, such as BM25, is explicitly designed for this. Finally, after years of keyword-based search, users have likely taken it for granted and may get frustrated if the document they expect to retrieve isn’t being returned.
> Vector embeddings *do not* magically solve search. In fact, the heavy lifting is in the step before you re-rank with semantic similarity search. Making a genuine improvement over BM25 or full-text search is hard. — [Aravind Srinivas, CEO Perplexity.ai](https://x.com/AravSrinivas/status/1737886080555446552)
> We’ve been communicating this to our customers and partners for months now. Nearest Neighbor Search with naive embeddings yields very noisy results and you’re likely better off starting with a keyword-based approach. — [Beyang Liu, CTO Sourcegraph](https://twitter.com/beyang/status/1767330006999720318)
Second, it’s more straightforward to understand why a document was retrieved with keyword search—we can look at the keywords that match the query. In contrast, embedding-based retrieval is less interpretable. Finally, thanks to systems like Lucene and OpenSearch that have been optimized and battle-tested over decades, keyword search is usually more computationally efficient.
In most cases, a hybrid will work best: keyword matching for the obvious matches, and embeddings for synonyms, hypernyms, and spelling errors, as well as multimodality (e.g., images and text). [Shortwave shared how they built their RAG pipeline](https://www.shortwave.com/blog/deep-dive-into-worlds-smartest-email-ai/), including query rewriting, keyword + embedding retrieval, and ranking.
### Prefer RAG over finetuning for new knowledge
Both RAG and finetuning can be used to incorporate new information into LLMs and increase performance on specific tasks. However, which should we prioritize?
Recent research suggests RAG may have an edge. [One study](https://arxiv.org/abs/2312.05934) compared RAG against unsupervised finetuning (aka continued pretraining), evaluating both on a subset of MMLU and current events. They found that RAG consistently outperformed finetuning for knowledge encountered during training as well as entirely new knowledge. In [another paper](https://arxiv.org/abs/2401.08406), they compared RAG against supervised finetuning on an agricultural dataset. Similarly, the performance boost from RAG was greater than finetuning, especially for GPT-4 (see Table 20).
Beyond improved performance, RAG has other practical advantages. First, compared to continuous pretraining or finetuning, it’s easier—and cheaper!—to keep retrieval indices up-to-date. Second, if our retrieval indices have problematic documents that contain toxic or biased content, we can easily drop or modify the offending documents. Consider it an andon cord for [documents that ask us to add glue to pizza](https://x.com/petergyang/status/1793480607198323196).
In addition, the R in RAG provides finer-grained control over how we retrieve documents. For example, if we’re hosting a RAG system for multiple organizations, by partitioning the retrieval indices, we can ensure that each organization can only retrieve documents from their own index. This ensures that we don’t inadvertently expose information from one organization to another.
### Long-context models won’t make RAG obsolete
With Gemini 1.5 providing context windows of up to 10M tokens in size, some have begun to question the future of RAG.
> I tend to believe that Gemini 1.5 is significantly overhyped by Sora. A context window of 10M tokens effectively makes most of existing RAG frameworks unnecessary — you simply put whatever your data into the context and talk to the model like usual. Imagine how it does to all the startups / agents / langchain projects where most of the engineering efforts goes to RAG 😅 Or in one sentence: the 10m context kills RAG. Nice work Gemini — [Yao Fu](https://x.com/Francis_YAO_/status/1758935954189115714)
While it’s true that long contexts will be a game-changer for use cases such as analyzing multiple documents or chatting with PDFs, the rumors of RAG’s demise are greatly exaggerated.
First, even with a context size of 10M tokens, we’d still need a way to select relevant context. Second, beyond the narrow needle-in-a-haystack eval, we’ve yet to see convincing data that models can effectively reason over large context sizes. Thus, without good retrieval (and ranking), we risk overwhelming the model with distractors, or may even fill the context window with completely irrelevant information.
Finally, there’s cost. During inference, the Transformer’s time complexity scales linearly with context length. Just because there exists a model that can read your org’s entire Google Drive contents before answering each question doesn’t mean that’s a good idea. Consider an analogy to how we use RAM: we still read and write from disk, even though there exist compute instances with [RAM running into the tens of terabytes](https://aws.amazon.com/ec2/instance-types/high-memory/).
So don’t throw your RAGs in the trash just yet. This pattern will remain useful even as context sizes grow.
## Tuning and optimizing workflows
Prompting an LLM is just the beginning. To get the most juice out of them, we need to think beyond a single prompt and embrace workflows. For example, how could we split a single complex task into multiple simpler tasks? When is finetuning or caching helpful with increasing performance and reducing latency/cost? Here, we share proven strategies and real-world examples to help you optimize and build reliable LLM workflows.
### Step-by-step, multi-turn “flows” can give large boosts
It’s common knowledge that decomposing a single big prompt into multiple smaller prompts can achieve better results. For example, [AlphaCodium](https://arxiv.org/abs/2401.08500): By switching from a single prompt to a multi-step workflow, they increased GPT-4 accuracy (pass@5) on CodeContests from 19% to 44%. The workflow includes:
- Reflecting on the problem
- Reasoning on the public tests
- Generating possible solutions
- Ranking possible solutions
- Generating synthetic tests
- Iterating on the solutions on public and synthetic tests.
Small tasks with clear objectives make for the best agent or flow prompts. It’s not required that every agent prompt requests structured output, but structured outputs help a lot to interface with whatever system is orchestrating the agent’s interactions with the environment. Some things to try:
- A tightly-specified, explicit planning step. Also, consider having predefined plans to choose from.
- Rewriting the original user prompts into agent prompts, though this process may be lossy!
- Agent behaviors as linear chains, DAGs, and state machines; different dependency and logic relationships can be more and less appropriate for different scales. Can you squeeze performance optimization out of different task architectures?
- Planning validations; your planning can include instructions on how to evaluate the responses from other agents to make sure the final assembly works well together.
- Prompt engineering with fixed upstream state—make sure your agent prompts are evaluated against a collection of variants of what may have happen before.
### Prioritize deterministic workflows for now
While AI agents can dynamically react to user requests and the environment, their non-deterministic nature makes them a challenge to deploy. Each step an agent takes has a chance of failing, and the chances of recovering from the error are poor. Thus, the likelihood that an agent completes a multi-step task successfully decreases exponentially as the number of steps increases. As a result, teams building agents find it difficult to deploy reliable agents.
A potential approach is to have agent systems produce deterministic plans which are then executed in a structured, reproducible way. First, given a high-level goal or prompt, the agent generates a plan. Then, the plan is executed deterministically. This allows each step to be more predictable and reliable. Benefits include:
- Generated plans can serve as few-shot samples to prompt or finetune an agent.
- Deterministic execution makes the system more reliable, and thus easier to test and debug. In addition, failures can be traced to the specific steps in the plan.
- Generated plans can be represented as directed acyclic graphs (DAGs) which are easier, relative to a static prompt, to understand and adapt to new situations.
The most successful agent builders may be those with strong experience managing junior engineers because the process of generating plans is similar to how we instruct and manage juniors. We give juniors clear goals and concrete plans, instead of vague open-ended directions, and we should do the same for our agents too.
In the end, the key to reliable, working agents will likely be found in adopting more structured, deterministic approaches, as well as collecting data to refine prompts and finetune models. Without this, we’ll build agents that may work exceptionally well some of the time, but on average, disappoint users.
### Getting more diverse outputs beyond temperature
Suppose your task requires diversity in an LLM’s output. Maybe you’re writing an LLM pipeline to suggest products to buy from your catalog given a list of products the user bought previously. When running your prompt multiple times, you might notice that the resulting recommendations are too similar—so you might increase the temperature parameter in your LLM requests.
Briefly, increasing the temperature parameter makes LLM responses more varied. At sampling time, the probability distributions of the next token become flatter, meaning that tokens that are usually less likely get chosen more often. Still, when increasing temperature, you may notice some failure modes related to output diversity. For example, some products from the catalog that could be a good fit may never be output by the LLM. The same handful of products might be overrepresented in outputs, if they are highly likely to follow the prompt based on what the LLM has learned at training time. If the temperature is too high, you may get outputs that reference nonexistent products (or gibberish!)
In other words, increasing temperature does not guarantee that the LLM will sample outputs from the probability distribution you expect (e.g., uniform random). Nonetheless, we have other tricks to increase output diversity. The simplest way is to adjust elements within the prompt. For example, if the prompt template includes a list of items, such as historical purchases, shuffling the order of these items each time they’re inserted into the prompt can make a significant difference.
Additionally, keeping a short list of recent outputs can help prevent redundancy. In our recommended products example, by instructing the LLM to avoid suggesting items from this recent list, or by rejecting and resampling outputs that are similar to recent suggestions, we can further diversify the responses. Another effective strategy is to vary the phrasing used in the prompts. For instance, incorporating phrases like “pick an item that the user would love using regularly” or “select a product that the user would likely recommend to friends” can shift the focus and thereby influence the variety of recommended products.
### Caching is underrated
Caching saves cost and eliminates generation latency by removing the need to recompute responses for the same input. Furthermore, if a response has previously been guardrailed, we can serve these vetted responses and reduce the risk of serving harmful or inappropriate content.
One straightforward approach to caching is to use unique IDs for the items being processed, such as if we’re summarizing new articles or [product reviews](https://www.cnbc.com/2023/06/12/amazon-is-using-generative-ai-to-summarize-product-reviews.html). When a request comes in, we can check to see if a summary already exists in the cache. If so, we can return it immediately; if not, we generate, guardrail, and serve it, and then store it in the cache for future requests.
For more open-ended queries, we can borrow techniques from the field of search, which also leverages caching for open-ended inputs. Features like autocomplete, spelling correction, and suggested queries also help normalize user input and thus increase the cache hit rate.
### When to finetune
We may have some tasks where even the most cleverly designed prompts fall short. For example, even after significant prompt engineering, our system may still be a ways from returning reliable, high-quality output. If so, then it may be necessary to finetune a model for your specific task.
Successful examples include:
- [Honeycomb’s Natural Language Query Assistant](https://www.honeycomb.io/blog/introducing-query-assistant): Initially, the “programming manual” was provided in the prompt together with n-shot examples for in-context learning. While this worked decently, finetuning the model led to better output on the syntax and rules of the domain-specific language.
- [Rechat’s Lucy](https://www.youtube.com/watch?v=B_DMMlDuJB0): The LLM needed to generate responses in a very specific format that combined structured and unstructured data for the frontend to render correctly. Finetuning was essential to get it to work consistently.
Nonetheless, while finetuning can be effective, it comes with significant costs. We have to annotate finetuning data, finetune and evaluate models, and eventually self-host them. Thus, consider if the higher upfront cost is worth it. If prompting gets you 90% of the way there, then finetuning may not be worth the investment. However, if we do decide to finetune, to reduce the cost of collecting human-annotated data, we can [generate and finetune on synthetic data](https://eugeneyan.com/writing/synthetic/), or [bootstrap on open-source data](https://eugeneyan.com/writing/finetuning/).
## Evaluation & Monitoring
Evaluating LLMs is a [minefield](https://www.cs.princeton.edu/~arvindn/talks/evaluating_llms_minefield/) and even the biggest labs find it [challenging](https://www.anthropic.com/news/evaluating-ai-systems). LLMs return open-ended outputs, and the tasks we set them to are varied. Nonetheless, rigorous and thoughtful evals are critical—it’s no coincidence that technical leaders at OpenAI [work on evaluation and give feedback on individual evals](https://twitter.com/eugeneyan/status/1701692908074873036).
Evaluating LLM applications invites a diversity of definitions and reductions: it’s simply unit testing, or it’s more like observability, or maybe it’s just data science. We have found all of these perspectives useful. In this section, we provide some lessons on what is important in building evals and monitoring pipelines.
### Create a few assertion-based unit tests from real input/output samples
Create [unit tests (i.e., assertions)](https://hamel.dev/blog/posts/evals/#level-1-unit-tests) consisting of samples of inputs and outputs from production, with expectations for outputs based on at least three criteria. While three criteria might seem arbitrary, it’s a practical number to start with; fewer might indicate that your task isn’t sufficiently defined or is too open-ended, like a general-purpose chatbot. These unit tests, or assertions, should be triggered by any changes to the pipeline, whether it’s editing a prompt, adding new context via RAG, or other modifications. This [write-up has an example](https://hamel.dev/blog/posts/evals/#step-1-write-scoped-tests) of an assertion-based test for an actual use case.
Consider beginning with assertions that specify phrases that let us include or exclude responses. Also try checks to ensure that word, item, or sentence counts lie within a range. For other kinds of generations, assertions can look different. [Execution-based evaluation](https://www.semanticscholar.org/paper/Execution-Based-Evaluation-for-Open-Domain-Code-Wang-Zhou/1bed34f2c23b97fd18de359cf62cd92b3ba612c3) is one way to evaluate code generation, wherein you run the generated code and check if the state of runtime is sufficient for the user request.
As an example, if the user asks for a new function named foo; then after executing the agent’s generated code, foo should be callable! One challenge in execution-based evaluation is that the agent code frequently leaves the runtime in a slightly different form than the target code. It can be effective to “relax” assertions to the absolute most weak assumptions that any viable answer would satisfy.
Finally, using your product as intended for customers (i.e., “dogfooding”) can provide insight into failure modes on real-world data. This approach not only helps identify potential weaknesses, but also provides a useful source of production samples that can be converted into evals.
### LLM-as-Judge can work (somewhat), but it’s not a silver bullet
LLM-as-Judge, where we use a strong LLM to evaluate the output of other LLMs, has been met with skepticism. (Some of us were initially huge skeptics.) Nonetheless, when implemented well, LLM-as-Judge achieves decent correlation with human judgments, and can at least help build priors about how a new prompt or technique may perform. Specifically, when doing pairwise comparisons (control vs. treatment), LLM-as-Judge typically gets the direction right though the magnitude of the win/loss may be noisy.
Here are some suggestions to get the most out of LLM-as-Judge:
- Use pairwise comparisons: Instead of asking the LLM to score a single output on a [Likert](https://en.wikipedia.org/wiki/Likert_scale) scale, present it with two options and ask it to select the better one. This tends to lead to more stable results.
- Control for position bias: The order of options presented can bias the LLM’s decision. To mitigate this, do each pairwise comparison twice, swapping the order of pairs each time. Just be sure to attribute wins to the right option after swapping!
- Allow for ties: In some cases, both options may be equally good. Thus, allow the LLM to declare a tie so it doesn’t have to arbitrarily pick a winner.
- Use Chain-of-Thought: Asking the LLM to explain its decision before giving a final answer can increase eval reliability. As a bonus, this lets you to use a weaker but faster LLM and still achieve similar results. Because this part of the pipeline is typically run in batch, the extra latency from CoT isn’t a problem.
- Control for response length: LLMs tend to bias toward longer responses. To mitigate this, ensure response pairs are similar in length.
A useful application of LLM-as-Judge is checking a new prompting strategy against regression. If you have tracked a collection of production results, sometimes you can rerun those production examples with a new prompting strategy, and use LLM-as-Judge to quickly assess where the new strategy may suffer.
Here’s an example of a [simple but effective approach](https://hamel.dev/blog/posts/evals/#automated-evaluation-w-llms) to iterate on LLM-as-Judge, where we log the LLM response, judge’s critique (i.e., CoT), and final outcome. They are then reviewed with stakeholders to identify areas for improvement. Over three iterations, agreement with humans and LLM improved from 68% to 94%!
![](Reading/Read%20Later/assets/spreadsheet.png)
LLM-as-Judge is not a silver bullet though. There are subtle aspects of language where even the strongest models fail to evaluate reliably. In addition, we’ve found that [conventional classifiers](https://eugeneyan.com/writing/finetuning/) and reward models can achieve higher accuracy than LLM-as-Judge, and with lower cost and latency. For code generation, LLM-as-Judge can be weaker than more direct evaluation strategies like execution-evaluation.
> [Read more](https://eugeneyan.com/writing/llm-evaluators/) on techniques, alignment workflows, finetuning models, critiques, etc. for LLM-evaluators
### The “intern test” for evaluating generations
We like to use the following “intern test” when evaluating generations: If you took the exact input to the language model, including the context, and gave it to an average college student in the relevant major as a task, could they succeed? How long would it take?
- If the answer is no because the LLM lacks the required knowledge, consider ways to enrich the context.
- If the answer is no and we simply can’t improve the context to fix it, then we may have hit a task that’s too hard for contemporary LLMs.
- If the answer is yes, but it would take a while, we can try to reduce the complexity of the task. Is it decomposable? Are there aspects of the task that can be made more templatized?
- If the answer is yes, they would get it quickly, then it’s time to dig into the data. What’s the model doing wrong? Can we find a pattern of failures? Try asking the model to explain itself before or after it responds, to help you build a theory of mind.
### Overemphasizing certain evals can hurt overall performance
> “When a measure becomes a target, it ceases to be a good measure.” — Goodhart’s Law.
An example of this is the Needle-in-a-Haystack (NIAH) eval. The original eval helped quantify model recall as context sizes grew, as well as how recall is affected by needle position. However, it’s been so overemphasized that it’s featured as [Figure 1 for Gemini 1.5’s report](https://arxiv.org/abs/2403.05530). The eval involves inserting a specific phrase (“The special magic {city} number is: {number}”) into a long document that repeats the essays of Paul Graham, and then prompting the model to recall the magic number.
While some models achieve near-perfect recall, it’s questionable whether NIAH truly measures the reasoning and recall abilities needed in real-world applications. Consider a more practical scenario: Given the transcript of an hour-long meeting, can the LLM summarize the key decisions and next steps, as well as correctly attribute each item to the relevant person? This task is more realistic, going beyond rote memorization, and considers the ability to parse complex discussions, identify relevant information, and synthesize summaries.
Here’s an example of a [practical NIAH eval](https://observablehq.com/@shreyashankar/needle-in-the-real-world-experiments). Using [doctor-patient transcripts](https://github.com/wyim/aci-bench/tree/main/data/challenge_data), the LLM is queried about the patient’s medication. It also includes a more challenging NIAH, inserting a phrase for random ingredients for pizza toppings, such as “*The secret ingredients needed to build the perfect pizza are: Espresso-soaked dates, Lemon, and Goat cheese.*”. Recall was around 80% on the medication task and 30% on the pizza task.
![](Reading/Read%20Later/assets/niah.png)
Tangentially, an overemphasis on NIAH evals can reduce performance on extraction and summarization tasks. Because these LLMs are so finetuned to attend to every sentence, they may start to treat irrelevant details and distractors as important, thus including them in the final output (when they shouldn’t!)
This could also apply to other evals and use cases. For example, summarization. An emphasis on factual consistency could lead to summaries that are less specific (and thus less likely to be factually inconsistent) and possibly less relevant. Conversely, an emphasis on writing style and eloquence could lead to more flowery, marketing-type language that could introduce factual inconsistencies.
### Simplify annotation to binary tasks or pairwise comparisons
Providing open-ended feedback or ratings for model output on a [Likert scale](https://en.wikipedia.org/wiki/Likert_scale) is cognitively demanding. As a result, the data collected is more noisy—due to variability among human raters—and thus less useful. A more effective approach is to simplify the task and reduce the cognitive burden on annotators. Two tasks that work well are binary classifications and pairwise comparisons.
In binary classifications, annotators are asked to make a simple yes-or-no judgment on the model’s output. They might be asked whether the generated summary is factually consistent with the source document, or whether the proposed response is relevant, or if it contains toxicity. Compared to the Likert scale, binary decisions are more precise, have higher consistency among raters, and lead to higher throughput. This was how [Doordash set up their labeling queues](https://doordash.engineering/2020/08/28/overcome-the-cold-start-problem-in-menu-item-tagging/) for tagging menu items through a tree of yes-no questions.
In pairwise comparisons, the annotator is presented with a pair of model responses and asked which is better. Because it’s easier for humans to say “A is better than B” than to assign an individual score to either A or B individually, this leads to faster and more reliable annotations (over Likert scales). At a [Llama2 meetup](https://www.youtube.com/watch?v=CzR3OrOkM9w), Thomas Scialom, an author on the Llama2 paper, confirmed that pairwise-comparisons were faster and cheaper than collecting supervised finetuning data such as written responses. The former’s cost is $3.5 per unit while the latter’s cost is $25 per unit.
If you’re writing labeling guidelines, here are some [example guidelines](https://eugeneyan.com/writing/labeling-guidelines/) from Google and Bing Search.
### (Reference-free) evals and guardrails can be used interchangeably
Guardrails help to catch inappropriate or harmful content while evals help to measure the quality and accuracy of the model’s output. And if your evals are reference-free, they can be used as guardrails too. Reference-free evals are evaluations that don’t rely on a “golden” reference, such as a human-written answer, and can assess the quality of output based solely on the input prompt and the model’s response.
Some examples of these are [summarization evals](https://eugeneyan.com/writing/evals/#summarization-consistency-relevance-length), where we only have to consider the input document to evaluate the summary on factual consistency and relevance. If the summary scores poorly on these metrics, we can choose not to display it to the user, effectively using the eval as a guardrail. Similarly, reference-free [translation evals](https://eugeneyan.com/writing/evals/#translation-statistical--learned-evals-for-quality) can assess the quality of a translation without needing a human-translated reference, again allowing us to use it as a guardrail.
### LLMs will return output even when they shouldn’t
A key challenge when working with LLMs is that they’ll often generate output even when they shouldn’t. This can lead to harmless but nonsensical responses, or more egregious defects like toxicity or dangerous content. For example, when asked to extract specific attributes or metadata from a document, an LLM may confidently return values even when those values don’t actually exist. Alternatively, the model may respond in a language other than English because we provided non-English documents in the context.
While we can try to prompt the LLM to return a “not applicable” or “unknown” response, it’s not foolproof. Even when the log probabilities are available, they’re a poor indicator of output quality. While log probs indicate the likelihood of a token appearing in the output, they don’t necessarily reflect the correctness of the generated text. On the contrary, for instruction-tuned models that are trained to answer queries and generate coherent responses, log probabilities may not be well-calibrated. Thus, while a high log probability may indicate that the output is fluent and coherent, it doesn’t mean it’s accurate or relevant.
While careful prompt engineering can help to an extent, we should complement it with robust guardrails that detect and filter/regenerate undesired output. For example, OpenAI provides a [content moderation API](https://platform.openai.com/docs/guides/moderation) that can identify unsafe responses such as hate speech, self-harm, or sexual output. Similarly, there are numerous packages for [detecting personally identifiable information](https://github.com/topics/pii-detection). One benefit is that guardrails are largely agnostic of the use case and can thus be applied broadly to all output in a given language. In addition, with precise retrieval, our system can deterministically respond “I don’t know” if there are no relevant documents.
A corollary here is that LLMs may fail to produce outputs when they are expected to. This can happen for various reasons, from straightforward issues like long-tail latencies from API providers to more complex ones such as outputs being blocked by content moderation filters. As such, it’s important to consistently log inputs and (potentially a lack of) outputs for debugging and monitoring.
### Hallucinations are a stubborn problem
Unlike content safety or PII defects which have a lot of attention and thus seldom occur, factual inconsistencies are stubbornly persistent and more challenging to detect. They’re more common and occur at a baseline rate of 5 - 10%, and from what we’ve learned from LLM providers, it can be challenging to get it below 2%, even on simple tasks such as summarization.
To address this, we can combine prompt engineering (upstream of generation) and factual inconsistency guardrails (downstream of generation). For prompt engineering, techniques like CoT help reduce hallucination by getting the LLM to explain its reasoning before finally returning the output. Then, we can apply a [factual inconsistency guardrail](https://eugeneyan.com/writing/finetuning/) to assess the factuality of summaries and filter or regenerate hallucinations. In some cases, hallucinations can be deterministically detected. When using resources from RAG retrieval, if the output is structured and identifies what the resources are, you should be able to manually verify they’re sourced from the input context.
## Operational: Day-to-day and Org concerns
## Data
Just as the quality of ingredients determines the taste of a dish, the quality of input data constrains the performance of machine learning systems. In addition, output data is the only way to tell whether the product is working or not. All the authors focus on the data, looking at inputs and outputs for several hours a week to better understand the data distribution: its modes, its edge cases, and the limitations of models of it.
### Check for development-prod skew
A common source of errors in traditional machine learning pipelines is *train-serve skew*. This happens when the data used in training differs from what the model encounters in production. Although we can use LLMs without training or finetuning, hence there’s no training set, a similar issue arises with development-prod data skew. Essentially, the data we test our systems on during development should mirror what the systems will face in production. If not, we might find our production accuracy suffering.
LLM development-prod skew can be categorized into two types: structural and content-based. Structural skew includes issues like formatting discrepancies, such as differences between a JSON dictionary with a list-type value and a JSON list, inconsistent casing, and errors like typos or sentence fragments. These errors can lead to unpredictable model performance because different LLMs are trained on specific data formats, and prompts can be highly sensitive to minor changes. Content-based or “semantic” skew refers to differences in the meaning or context of the data. 
As in traditional ML, it’s useful to periodically measure skew between the LLM input/output pairs. Simple metrics like the length of inputs and outputs or specific formatting requirements (e.g., JSON or XML) are straightforward ways to track changes. For more “advanced” drift detection, consider clustering embeddings of input/output pairs to detect semantic drift, such as shifts in the topics users are discussing, which could indicate they are exploring areas the model hasn’t been exposed to before. 
When testing changes, such as prompt engineering, ensure that hold-out datasets are current and reflect the most recent types of user interactions. For example, if typos are common in production inputs, they should also be present in the hold-out data. Beyond just numerical skew measurements, it’s beneficial to perform qualitative assessments on outputs. Regularly reviewing your model’s outputs—a practice colloquially known as “vibe checks”—ensures that the results align with expectations and remain relevant to user needs. Finally, incorporating nondeterminism into skew checks is also useful—by running the pipeline multiple times for each input in our testing dataset and analyzing all outputs, we increase the likelihood of catching anomalies that might occur only occasionally.
### Look at samples of LLM inputs and outputs every day
LLMs are dynamic and constantly evolving. Despite their impressive zero-shot capabilities and often delightful outputs, their failure modes can be highly unpredictable. For custom tasks, regularly reviewing data samples is essential to developing an intuitive understanding of how LLMs perform.
Input-output pairs from production are the “real things, real places” (*genchi genbutsu*) of LLM applications, and they cannot be substituted. [Recent research](https://arxiv.org/abs/2404.12272) highlighted that developers’ perceptions of what constitutes “good” and “bad” outputs shift as they interact with more data (i.e., *criteria drift*). While developers can come up with some criteria upfront for evaluating LLM outputs, these predefined criteria are often incomplete. For instance, during the course of development, we might update the prompt to increase the probability of good responses and decrease the probability of bad ones. This iterative process of evaluation, reevaluation, and criteria update is necessary, as it’s difficult to predict either LLM behavior or human preference without directly observing the outputs.
To manage this effectively, we should log LLM inputs and outputs. By examining a sample of these logs daily, we can quickly identify and adapt to new patterns or failure modes. When we spot a new issue, we can immediately write an assertion or eval around it. Similarly, any updates to failure mode definitions should be reflected in the evaluation criteria. These “vibe checks” are signals of bad outputs; code and assertions operationalize them. Finally, this attitude must be socialized, for example by adding review or annotation of inputs and outputs to your on-call rotation.
## Working with models
With LLM APIs, we can rely on intelligence from a handful of providers. While this is a boon, these dependencies also involve trade-offs on performance, latency, throughput, and cost. Also, as newer, better models drop (almost every month in the past year), we should be prepared to update our products as we deprecate old models and migrate to newer models. In this section, we share our lessons from working with technologies we don’t have full control over, where the models can’t be self-hosted and managed.
### Generate structured output to ease downstream integration
For most real-world use cases, the output of an LLM will be consumed by a downstream application via some machine-readable format. For example, [Rechat](https://www.youtube.com/watch?v=B_DMMlDuJB0), a real-estate CRM, required structured responses for the front end to render widgets. Similarly, [Boba](https://martinfowler.com/articles/building-boba.html), a tool for generating product strategy ideas, needed structured output with fields for title, summary, plausibility score, and time horizon. Finally, LinkedIn shared about [constraining the LLM to generate YAML](https://www.linkedin.com/blog/engineering/generative-ai/musings-on-building-a-generative-ai-product), which is then used to decide which skill to use, as well as provide the parameters to invoke the skill.
This application pattern is an extreme version of Postel’s Law: be liberal in what you accept (arbitrary natural language) and conservative in what you send (typed, machine-readable objects). As such, we expect it to be extremely durable.
Currently, [Instructor](https://github.com/jxnl/instructor) and [Outlines](https://github.com/outlines-dev/outlines) are the de facto standards for coaxing structured output from LLMs. If you’re using an LLM API (e.g., Anthropic, OpenAI), use Instructor; if you’re working with a self-hosted model (e.g., Huggingface), use Outlines.
### Migrating prompts across models is a pain in the ass
Sometimes, our carefully crafted prompts work superbly with one model but fall flat with another. This can happen when we’re switching between various model providers, as well as when we upgrade across versions of the same model. 
For example, Voiceflow found that [migrating from gpt-3.5-turbo-0301 to gpt-3.5-turbo-1106 led to a 10% drop](https://www.voiceflow.com/blog/how-much-do-chatgpt-versions-affect-real-world-performance) in their intent classification task. (Thankfully, they had evals!) Similarly, [GoDaddy observed a trend in the positive direction](https://www.godaddy.com/resources/news/llm-from-the-trenches-10-lessons-learned-operationalizing-models-at-godaddy#h-3-prompts-aren-t-portable-across-models), where upgrading to version 1106 narrowed the performance gap between gpt-3.5-turbo and gpt-4. (Or, if you’re a glass-half-full person, you might be disappointed that gpt-4’s lead was reduced with the new upgrade)
Thus, if we have to migrate prompts across models, expect it to take more time than simply swapping the API endpoint. Don’t assume that plugging in the same prompt will lead to similar or better results. Also, having reliable, automated evals helps with measuring task performance before and after migration, and reduces the effort needed for manual verification.
### Version and pin your models
In any machine learning pipeline, “[changing anything changes everything](https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html)”. This is particularly relevant as we rely on components like large language models (LLMs) that we don’t train ourselves and that can change without our knowledge.
Fortunately, many model providers offer the option to “pin” specific model versions (e.g., gpt-4-turbo-1106). This enables us to use a specific version of the model weights, ensuring they remain unchanged. Pinning model versions in production can help avoid unexpected changes in model behavior, which could lead to customer complaints about issues that may crop up when a model is swapped, such as overly verbose outputs or other unforeseen failure modes.
Additionally, consider maintaining a shadow pipeline that mirrors your production setup but uses the latest model versions. This enables safe experimentation and testing with new releases. Once you’ve validated the stability and quality of the outputs from these newer models, you can confidently update the model versions in your production environment.
### Choose the smallest model that gets the job done
When working on a new application, it’s tempting to use the biggest, most powerful model available. But once we’ve established that the task is technically feasible, it’s worth experimenting if a smaller model can achieve comparable results.
The benefits of a smaller model are lower latency and cost. While it may be weaker, techniques like chain-of-thought, n-shot prompts, and in-context learning can help smaller models punch above their weight. Beyond LLM APIs, finetuning our specific tasks can also help increase performance.
Taken together, a carefully crafted workflow using a smaller model can often match, or even surpass, the output quality of a single large model, while being faster and cheaper. For example, this [tweet](https://twitter.com/mattshumer_/status/1770823530394833242) shares anecdata of how Haiku + 10-shot prompt outperforms zero-shot Opus and GPT-4. In the long term, we expect to see more examples of [flow-engineering](https://twitter.com/karpathy/status/1748043513156272416) with smaller models as the optimal balance of output quality, latency, and cost.
As another example, take the humble classification task. Lightweight models like DistilBERT (67M parameters) are a surprisingly strong baseline. The 400M parameter DistilBART is another great option—when finetuned on open-source data, it could [identify hallucinations with an ROC-AUC of 0.84](https://eugeneyan.com/writing/finetuning/), surpassing most LLMs at less than 5% of the latency and cost.
The point is, don’t overlook smaller models. While it’s easy to throw a massive model at every problem, with some creativity and experimentation, we can often find a more efficient solution. 
## Product
While new technology offers new possibilities, the principles of building great products are timeless. Thus, even if we’re solving new problems for the first time, we don’t have to reinvent the wheel on product design. There’s a lot to gain from grounding our LLM application development in solid product fundamentals, allowing us to deliver real value to the people we serve.
### Involve design early and often
Having a designer will push you to understand and think deeply about how your product can be built and presented to users. We sometimes stereotype designers as folks who take things and make them pretty. But beyond just the user interface, they also rethink how the user experience can be improved, even if it means breaking existing rules and paradigms.
Designers are especially gifted at reframing the user’s needs into various forms. Some of these forms are more tractable to solve than others, and thus, they may offer more or fewer opportunities for AI solutions. Like many other products, building AI products should be centered around the job to be done, not the technology that powers them.
Focus on asking yourself: “What job is the user asking this product to do for them? Is that job something a chatbot would be good at? How about autocomplete? Maybe something different!” Consider the existing [design patterns](https://www.tidepool.so/blog/emerging-ux-patterns-for-generative-ai-apps-copilots) and how they relate to the job-to-be-done. These are the invaluable assets that designers add to your team’s capabilities.
### Design your UX for Human-In-The-Loop
One way to get quality annotations is to integrate Human-in-the-Loop (HITL) into the user experience (UX). By allowing users to provide feedback and corrections easily, we can improve the immediate output and collect valuable data to improve our models.
Imagine an e-commerce platform where users upload and categorize their products. There are several ways we could design the UX:
- The user manually selects the right product category; an LLM periodically checks new products and corrects miscategorization on the backend.
- The user doesn’t select any category at all; an LLM periodically categorizes products on the backend (with potential errors).
- An LLM suggests a product category in real-time, which the user can validate and update as needed.
While all three approaches involve an LLM, they provide very different UXes. The first approach puts the initial burden on the user and has the LLM acting as a post-processing check. The second requires zero effort from the user but provides no transparency or control. The third strikes the right balance. By having the LLM suggest categories upfront, we reduce cognitive load on the user and they don’t have to learn our taxonomy to categorize their product! At the same time, by allowing the user to review and edit the suggestion, they have the final say in how their product is classified, putting control firmly in their hands. As a bonus, the third approach creates a [natural feedback loop for model improvement](https://eugeneyan.com/writing/llm-patterns/#collect-user-feedback-to-build-our-data-flywheel). Suggestions that are good are accepted (positive labels) and those that are bad are updated (negative followed by positive labels).
This pattern of suggestion, user validation, and data collection is commonly seen in several applications:
- Coding assistants: Where users can accept a suggestion (strong positive), accept and tweak a suggestion (positive), or ignore a suggestion (negative)
- Midjourney: Where users can choose to upscale and download the image (strong positive), vary an image (positive), or generate a new set of images (negative)
- Chatbots: Where users can provide thumbs up (positive) or thumbs down (negative) on responses, or choose to regenerate a response if it was really bad (strong negative).
Feedback can be explicit or implicit. Explicit feedback is information users provide in response to a request by our product; implicit feedback is information we learn from user interactions without needing users to deliberately provide feedback. Coding assistants and Midjourney are examples of implicit feedback while thumbs up and thumb downs are explicit feedback. If we design our UX well, like coding assistants and Midjourney, we can collect plenty of implicit feedback to improve our product and models.
### Prioritize your hierarchy of needs ruthlessly
As we think about putting our demo into production, we’ll have to think about the requirements for:
- Reliability: 99.9% uptime, adherence to structured output
- Harmlessness: Not generate offensive, NSFW, or otherwise harmful content
- Factual consistency: Being faithful to the context provided, not making things up
- Usefulness: Relevant to the users’ needs and request
- Scalability: Latency SLAs, supported throughput
- Cost: Because we don’t have unlimited budget
- And more: Security, privacy, fairness, GDPR, DMA, etc, etc.
If we try to tackle all these requirements at once, we’re never going to ship anything. Thus, we need to prioritize. Ruthlessly. This means being clear what is non-negotiable (e.g., reliability, harmlessness) without which our product can’t function or won’t be viable. It’s all about identifying the minimum lovable product. We have to accept that the first version won’t be perfect, and just launch and iterate.
### Calibrate your risk tolerance based on the use case
When deciding on the language model and level of scrutiny of an application, consider the use case and audience. For a customer-facing chatbot offering medical or financial advice, we’ll need a very high bar for safety and accuracy. Mistakes or bad output could cause real harm and erode trust. But for less critical applications, such as a recommender system, or internal-facing applications like content classification or summarization, excessively strict requirements only slow progress without adding much value.
This aligns with a recent [a16z report](https://a16z.com/generative-ai-enterprise-2024/) showing that many companies are moving faster with internal LLM applications compared to external ones (image below). By experimenting with AI for internal productivity, organizations can start capturing value while learning how to manage risk in a more controlled environment. Then, as they gain confidence, they can expand to customer-facing use cases.
![](Reading/Read%20Later/assets/How-willing-are-enterprises-to-use-LLMs-for-different-use-cases_-1-scaled.jpg) Proportion of enterprise LLM use across internal and external-facing use cases ([source: a16z report](https://a16z.com/generative-ai-enterprise-2024/))
## Team & Roles
No job function is easy to define, but writing a job description for the work in this new space is more challenging than others. We’ll forgo Venn diagrams of intersecting job titles, or suggestions for job descriptions. We will, however, submit to the existence of a new role—the AI engineer—and discuss its place. Importantly, we’ll discuss the rest of the team and how responsibilities should be assigned.
### Focus on the process, not tools
When faced with new paradigms, such as LLMs, software engineers tend to favor tools. As a result, we overlook the problem and process the tool was supposed to solve. In doing so, many engineers assume accidental complexity, which has negative consequences for the team’s long-term productivity.
For example, [this write-up](https://hamel.dev/blog/posts/prompt/) discusses how certain tools can automatically create prompts for large language models. It argues (rightfully IMHO) that engineers who use these tools without first understanding the problem-solving methodology or process end up taking on unnecessary technical debt.
In addition to accidental complexity, tools are often underspecified. For example, there is a growing industry of LLM evaluation tools that offer “LLM Evaluation In A Box” with generic evaluators for toxicity, conciseness, tone, etc. We have seen many teams adopt these tools without thinking critically about the specific failure modes of their domains. Contrast this to EvalGen. It focuses on teaching users the process of creating domain-specific evals by deeply involving the user each step of the way, from specifying criteria, to labeling data, to checking evals. The software leads the user through a workflow that looks like this:
![](Reading/Read%20Later/assets/evalgen.png) Shankar, S., et al. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. Retrieved from [https://arxiv.org/abs/2404.12272](https://arxiv.org/abs/2404.12272)
EvalGen guides the user through a best practice of crafting LLM evaluations, namely:
- Defining domain-specific tests (bootstrapped automatically from the prompt). These are defined as either assertions with code or with LLM-as-a-Judge.
- The importance of aligning the tests with human judgment, so that the user can check that the tests capture the specified criteria.
- Iterating on your tests as the system (prompts, etc) changes. 
EvalGen provides developers with a mental model of the evaluation-building process without anchoring them to a specific tool. We have found that after providing AI Engineers with this context, they often decide to select leaner tools or build their own.  
There are too many components of LLMs beyond prompt writing and evaluations to list exhaustively here.  However, it is important that AI Engineers seek to understand the processes before adopting tools.
### Always be experimenting
ML products are deeply intertwined with experimentation. Not only the A/B, Randomized Control Trials kind, but the frequent attempts at modifying the smallest possible components of your system, and doing offline evaluation. The reason why everyone is so hot for evals is not actually about trustworthiness and confidence—it’s about enabling experiments! The better your evals, the faster you can iterate on experiments, and thus the faster you can converge on the best version of your system. 
It’s common to try different approaches to solving the same problem because experimentation is so cheap now. The high cost of collecting data and training a model is minimized—prompt engineering costs little more than human time. Position your team so that everyone is taught [the basics of prompt engineering](https://eugeneyan.com/writing/prompting/). This encourages everyone to experiment and leads to diverse ideas from across the organization.
Additionally, don’t only experiment to explore—also use them to exploit! Have a working version of a new task? Consider having someone else on the team approach it differently. Try doing it another way that’ll be faster. Investigate prompt techniques like Chain-of-Thought or Few-Shot to make it higher quality. Don’t let your tooling hold you back on experimentation; if it is, rebuild it, or buy something to make it better. 
Finally, during product/project planning, set aside time for building evals and running multiple experiments. Think of the product spec for engineering products, but add to it clear criteria for evals. And during roadmapping, don’t underestimate the time required for experimentation—expect to do multiple iterations of development and evals before getting the green light for production.
### Empower everyone to use new AI technology
As generative AI increases in adoption, we want the entire team—not just the experts—to understand and feel empowered to use this new technology. There’s no better way to develop intuition for how LLMs work (e.g., latencies, failure modes, UX) than to, well, use them. LLMs are relatively accessible: You don’t need to know how to code to improve performance for a pipeline, and everyone can start contributing via prompt engineering and evals.
A big part of this is education. It can start as simple as [the basics of prompt engineering](https://eugeneyan.com/writing/prompting/), where techniques like n-shot prompting and CoT help condition the model towards the desired output. Folks who have the knowledge can also educate about the more technical aspects, such as how LLMs are autoregressive when generating output. In other words, while input tokens are processed in parallel, output tokens are generated sequentially. As a result, latency is more a function of output length than input length—this is a key consideration when designing UXes and setting performance expectations.
We can go further and provide opportunities for hands-on experimentation and exploration. A hackathon perhaps? While it may seem expensive to have a team spend a few days hacking on speculative projects, the outcomes may surprise you. We know of a team that, through a hackathon, accelerated and almost completed their three-year roadmap within a year. Another team had a hackathon that led to paradigm-shifting UXes that are now possible thanks to LLMs, which have been prioritized for the year and beyond.
### Don’t fall into the trap of “AI Engineering is all I need”
As new job titles are coined, there is an initial tendency to overstate the capabilities associated with these roles. This often results in a painful correction as the actual scope of these jobs becomes clear. Newcomers to the field, as well as hiring managers, might make exaggerated claims or have inflated expectations. Notable examples over the last decade include:
- Data Scientist: “[someone who is better at statistics than any software engineer and better at software engineering than any statistician](https://x.com/josh_wills/status/198093512149958656).”  
- Machine Learning Engineer (MLE): a software engineering-centric view of machine learning 
Initially, many assumed that data scientists alone were sufficient for data-driven projects. However, it became apparent that data scientists must collaborate with software and data engineers to develop and deploy data products effectively. 
This misunderstanding has shown up again with the new role of AI Engineer, with some teams believing that AI Engineers are all you need. In reality, building machine learning or AI products requires a [broad array of specialized roles](https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html). We’ve consulted with more than a dozen companies on AI products and have consistently observed that they fall into the trap of believing that “AI Engineering is all you need.” As a result, products often struggle to scale beyond a demo as companies overlook crucial aspects involved in building a product.
For example, evaluation and measurement are crucial for scaling a product beyond vibe checks. The skills for effective evaluation align with some of the strengths traditionally seen in machine learning engineers—a team composed solely of AI Engineers will likely lack these skills. Co-author Hamel Husain illustrates the importance of these skills in his recent work around detecting [data drift](https://github.com/hamelsmu/ft-drift) and [designing domain-specific evals](https://hamel.dev/blog/posts/evals/).
Here is a rough progression of the types of roles you need, and when you’ll need them, throughout the journey of building an AI product:
- First, focus on building a product. This might include an AI engineer, but it doesn’t have to. AI Engineers are valuable for prototyping and iterating quickly on the product (UX, plumbing, etc). 
- Next, create the right foundations by instrumenting your system and collecting data. Depending on the type and scale of data, you might need platform and/or data engineers. You must also have systems for querying and analyzing this data to debug issues.
- Next, you will eventually want to optimize your AI system. This doesn’t necessarily involve training models. The basics include steps like designing metrics, building evaluation systems, running experiments, optimizing RAG retrieval, debugging stochastic systems, and more. MLEs are really good at this (though AI engineers can pick them up too). It usually doesn’t make sense to hire an MLE unless you have completed the prerequisite steps.
Aside from this, you need a domain expert at all times. At small companies, this would ideally be the founding team—and at bigger companies, product managers can play this role. Being aware of the progression and timing of roles is critical. Hiring folks at the wrong time (e.g., [hiring an MLE too early](https://jxnl.co/writing/2024/04/08/hiring-mle-at-early-stage-companies/)) or building in the wrong order is a waste of time and money, and causes churn.  Furthermore, regularly checking in with an MLE (but not hiring them full-time) during phases 1-2 will help the company build the right foundations. 
More on how to interview and hire ML/AI Engineers [here](https://eugeneyan.com/writing/how-to-interview), where we discuss: (i) what to interview for, (ii) how to conduct phone screens, interview loops, and debriefs, and (iii) tips for interviewers and hiring managers.
## Strategy: Building with LLMs without Getting Out-Maneuvered
Successful products require thoughtful planning and prioritization, not endless prototyping or following the latest model releases or trends. In this final section, we look around the corners and think about the strategic considerations for building great AI products. We also examine key trade-offs teams will face, like when to build and when to buy, and suggest a “playbook” for early LLM application development strategy.
## No GPUs before PMF
To be great, your product needs to be more than just a thin wrapper around somebody else’s API. But mistakes in the opposite direction can be even more costly. The past year has also seen a mint of venture capital, including an eye-watering six billion dollar Series A, spent on training and customizing models without a clear product vision or target market. In this section, we’ll explain why jumping immediately to training your own models is a mistake and consider the role of self-hosting.
### Training from scratch (almost) never makes sense
For most organizations, pretraining an LLM from scratch is an impractical distraction from building products.
As exciting as it is and as much as it seems like everyone else is doing it, developing and maintaining machine learning infrastructure takes a lot of resources. This includes gathering data, training and evaluating models, and deploying them. If you’re still validating product-market fit, these efforts will divert resources from developing your core product. Even if you had the compute, data, and technical chops, the pretrained LLM may become obsolete in months.
Consider [BloombergGPT](https://arxiv.org/abs/2303.17564), an LLM specifically trained for financial tasks. The model was pretrained on 363B tokens via a heroic effort by [nine full-time employees](https://twimlai.com/podcast/twimlai/bloomberggpt-an-llm-for-finance/), four from AI Engineering and five from ML Product and Research. Despite this, it was [outclassed by gpt-3.5-turbo and gpt-4](https://arxiv.org/abs/2305.05862) on those very tasks within a year.
This story and others like it suggest that for most practical applications, pretraining an LLM from scratch, even on domain-specific data, is not the best use of resources. Instead, teams are better off finetuning the strongest open-source models available for their specific needs.
There are of course exceptions. One shining example is [Replit’s code model](https://blog.replit.com/replit-code-v1_5), trained specifically for code generation and understanding. With pretraining, Replit was able to outperform other models of larger sizes such as CodeLlama7b. But as other, increasingly capable models have been released, maintaining utility has required continued investment.
### Don’t finetune until you’ve proven it’s necessary
For most organizations, finetuning is driven more by FOMO than by clear strategic thinking.
Organizations invest in finetuning too early, trying to beat the “just another wrapper” allegations. In reality, finetuning is heavy machinery, to be deployed only after you’ve collected plenty of examples that convince you other approaches won’t suffice.
A year ago, many teams were telling us they were excited to finetune. Few have found product-market fit and most regret their decision. If you’re going to finetune, you’d better be *really* confident that you’re set up to do it again and again as base models improve—see the [“The model isn’t the product”](https://applied-llms.org/#the-model-isnt-the-product-the-system-around-it-is) and [“Build LLMOps”](https://applied-llms.org/#build-llmops-but-build-it-for-the-right-reason-faster-iteration) below.
When might finetuning actually be the right call? If the use case requires data not available in the mostly-open web-scale datasets used to train existing models—and if you’ve already built an MVP that demonstrates the existing models are insufficient. But be careful: if great training data isn’t readily available to the model builders, where are *you* getting it?
LLM-powered applications aren’t a science fair project. Investment in them should be commensurate with their contribution to your business’ strategic objectives and competitive differentiation.
### Start with inference APIs, but don’t be afraid of self-hosting
With LLM APIs, it’s easier than ever for startups to adopt and integrate language modeling capabilities without training their own models from scratch. Providers like Anthropic, and OpenAI offer general APIs that can sprinkle intelligence into your product with just a few lines of code. By using these services, you can reduce the effort spent and instead focus on creating value for your customers—this allows you to validate ideas and iterate towards product-market fit faster.
But, as with databases, managed services aren’t the right fit for every use case, especially as scale and requirements increase. Indeed, self-hosting may be the only way to use models without sending confidential / private data out of your network, as required in regulated industries like healthcare and finance, or by contractual obligations or confidentiality requirements.
Furthermore, self-hosting circumvents limitations imposed by inference providers, like rate limits, model deprecations, and usage restrictions. In addition, self-hosting gives you complete control over the model, making it easier to construct a differentiated, high-quality system around it. Finally, self-hosting, especially of finetunes, can reduce cost at large scale. For example, [Buzzfeed shared how they finetuned open-source LLMs to reduce costs by 80%](https://tech.buzzfeed.com/lessons-learned-building-products-powered-by-generative-ai-7f6c23bff376#9da5).
## Iterate to something great
To sustain a competitive edge in the long run, you need to think beyond models and consider what will set your product apart. While speed of execution matters, it shouldn’t be your only advantage.
### The model isn’t the product, the system around it is
For teams that aren’t building models, the rapid pace of innovation is a boon as they migrate from one SOTA model to the next, chasing gains in context size, reasoning capability, and price-to-value to build better and better products. This progress is as exciting as it is predictable. Taken together, this means models are likely to be the least durable component in the system.
Instead, focus your efforts on what’s going to provide lasting value, such as:
- Evals: To reliably measure performance on your task across models
- Guardrails: To prevent undesired outputs no matter the model
- Caching: To reduce latency and cost by avoiding the model altogether
- Data flywheel: To power the iterative improvement of everything above
These components create a thicker moat of product quality than raw model capabilities.
But that doesn’t mean building at the application layer is risk-free. Don’t point your shears at the same yaks that OpenAI or other model providers will need to shave if they want to provide viable enterprise software.
For example, some teams invested in building custom tooling to validate structured output from proprietary models; minimal investment here is important, but a deep one is not a good use of time. OpenAI needs to ensure that when you ask for a function call, you get a valid function call—because all of their customers want this. Employ some “strategic procrastination” here, build what you absolutely need, and await the obvious expansions to capabilities from providers.
### Build trust by starting small
Building a product that tries to be everything to everyone is a recipe for mediocrity. To create compelling products, companies need to specialize in building sticky experiences that keep users coming back.
Consider a generic RAG system that aims to answer any question a user might ask. The lack of specialization means that the system can’t prioritize recent information, parse domain-specific formats, or understand the nuances of specific tasks. As a result, users are left with a shallow, unreliable experience that doesn’t meet their needs, leading to churn.
To address this, focus on specific domains and use cases. Narrow the scope by going deep rather than wide. This will create domain-specific tools that resonate with users. Specialization also allows you to be upfront about your system’s capabilities and limitations. Being transparent about what your system can and cannot do demonstrates self-awareness, helps users understand where it can add the most value, and thus builds trust and confidence in the output.
### Build LLMOps, but build it for the right reason: faster iteration
DevOps is not fundamentally about reproducible workflows or shifting left or empowering two pizza teams—and it’s definitely not about writing YAML files.
DevOps is about shortening the feedback cycles between work and its outcomes so that improvements accumulate instead of errors. Its roots go back, via the Lean Startup movement, to Lean Manufacturing and the Toyota Production System, with its emphasis on Single Minute Exchange of Die and Kaizen.
MLOps has adapted the form of DevOps to ML. We have reproducible experiments and we have all-in-one suites that empower model builders to ship. And Lordy, do we have YAML files.
But as an industry, MLOps didn’t adopt the function of DevOps. It didn’t shorten the feedback gap between models and their inferences and interactions in production.
Hearteningly, the field of LLMOps has shifted away from thinking about hobgoblins of little minds like prompt management and towards the hard problems that block iteration: production monitoring and continual improvement, linked by evaluation.
Already, we have interactive arenas for neutral, crowd-sourced evaluation of chat and coding models – an outer loop of collective, iterative improvement. Tools like LangSmith, Log10, LangFuse, W&B Weave, HoneyHive, and more promise to not only collect and collate data about system outcomes in production, but also to leverage them to improve those systems by integrating deeply with development. Embrace these tools or build your own.
### Don’t Build LLM Features You Can Buy
Most successful businesses are not LLM businesses. Simultaneously, most businesses have opportunities to be improved by LLMs.
This pair of observations often mislead leaders into hastily retrofitting systems with LLMs at increased cost and decreased quality and releasing them as ersatz, vanity “AI” features, complete with the [now-dreaded sparkle icon](https://x.com/nearcyan/status/1783351706031718412). There’s a better way: focus on LLM applications that truly align with your product goals and enhance your core operations.
Consider a few misguided ventures that waste your team’s time:
- Building custom text-to-SQL capabilities for your business.
- Building a chatbot to talk to your documentation.
- Integrating your company’s knowledge base with your customer support chatbot.
While the above are the hellos-world of LLM applications, none of them make sense for a product company to build themselves. These are general problems for many businesses with a large gap between promising demo and dependable component—the customary domain of software companies. Investing valuable R&D resources on general problems being tackled en masse by the current Y Combinator batch is a waste.
If this sounds like trite business advice, it’s because in the frothy excitement of the current hype wave, it’s easy to mistake anything “LLM” as cutting-edge, accretive differentiation, missing which applications are already old hat.
### AI in the loop; Humans at the center
Right now, LLM-powered applications are brittle. They required an incredible amount of safe-guarding and defensive engineering, yet remain hard to predict. Additionally, when tightly scoped these applications can be wildly useful. This means that LLMs make excellent tools to accelerate user workflows.
While it may be tempting to imagine LLM-based applications fully replacing a workflow, or standing in for a job function, today the most effective paradigm is a human-computer centaur ([Centaur chess](https://en.wikipedia.org/wiki/Advanced_chess)). When capable humans are paired with LLM capabilities tuned for their rapid utilization, productivity and happiness doing tasks can be massively increased. One of the flagship applications of LLMs, GitHub CoPilot, demonstrated the power of these workflows:
> “Overall, developers told us they felt more confident because coding is easier, more error-free, more readable, more reusable, more concise, more maintainable, and more resilient with GitHub Copilot and GitHub Copilot Chat than when they’re coding without it.” - [Mario Rodriguez, GitHub](https://resources.github.com/learn/pathways/copilot/essentials/measuring-the-impact-of-github-copilot/)
For those who have worked in ML for a long time, you may jump to the idea of “human-in-the-loop”, but not so fast: HITL Machine Learning is a paradigm built on Human experts ensuring that ML models behave as predicted. While related, here we are proposing something more subtle. LLM-driven systems should not be the primary drivers of most workflows today, they should merely be a resource.
By centering humans, and asking how an LLM can support their workflow, this leads to significantly different product and design decisions. Ultimately, it will drive you to build different products than competitors who try to rapidly offshore all responsibility to LLMs; better, more useful, and less risky products.
## Start with prompting, evals, and data collection
The previous sections have delivered a firehose of techniques and advice. It’s a lot to take in. Let’s consider the minimum useful set of advice: if a team wants to build an LLM product, where should they begin?
Over the past year, we’ve seen enough to be confident that successful LLM applications follow a consistent trajectory. We walk through this basic “getting started” playbook in this section. The core idea is to start simple and only add complexity as needed. A decent rule of thumb is that each level of sophistication typically requires at least an order of magnitude more effort than the one before it. With this in mind…
### Prompt engineering comes first
Start with prompt engineering. Use all the techniques we discussed in the tactics section before. Chain-of-thought, n-shot examples, and structured input and output are almost always a good idea. Prototype with the most highly capable models before trying to squeeze performance out of weaker models.
Only if prompt engineering cannot achieve the desired level of performance should you consider finetuning. This will come up more often if there are non-functional requirements (e.g., data privacy, complete control, cost) that block the use of proprietary models and thus require you to self-host. Just make sure those same privacy requirements don’t block you from using user data for finetuning!
### Build evals and kickstart a data flywheel
Even teams that are just getting started need evals. Otherwise, you won’t know whether your prompt engineering is sufficient or when your finetuned model is ready to replace the base model.
Effective evals are [specific to your tasks](https://twitter.com/thesephist/status/1707839140018974776) and mirror the intended use cases. The first level of evals that we [recommend](https://hamel.dev/blog/posts/evals/) is unit testing. These simple assertions detect known or hypothesized failure modes and help drive early design decisions. Also see other [task-specific evals](https://eugeneyan.com/writing/evals/) for classification, summarization, etc.
While unit tests and model-based evaluations are useful, they don’t replace the need for human evaluation. Have people use your model/product and provide feedback. This serves the dual purpose of measuring real-world performance and defect rates while also collecting high-quality annotated data that can be used to finetune future models. This creates a positive feedback loop, or data flywheel, which compounds over time:
- Human evaluation to assess model performance and/or find defects
- Use the annotated data to finetune the model or update the prompt
- Repeat
For example, when auditing LLM-generated summaries for defects we might label each sentence with fine-grained feedback identifying factual inconsistency, irrelevance, or poor style. We can then use these factual inconsistency annotations to [train a hallucination classifier](https://eugeneyan.com/writing/finetuning/) or use the relevance annotations to train a [relevance-reward model](https://arxiv.org/abs/2009.01325). As another example, LinkedIn shared about their success with using [model-based evaluators](https://www.linkedin.com/blog/engineering/generative-ai/musings-on-building-a-generative-ai-product) to estimate hallucinations, responsible AI violations, coherence, etc. in their write-up
By creating assets that compound their value over time, we upgrade building evals from a purely operational expense to a strategic investment, and build our data flywheel in the process.
## The high-level trend of low-cost cognition
In 1971, the researchers at Xerox PARC predicted the future: the world of networked personal computers that we are now living in. They helped birth that future by playing pivotal roles in the invention of the technologies that made it possible, from Ethernet and graphics rendering to the mouse and the window.
But they also engaged in a simple exercise: they looked at applications that were very useful (e.g. video displays) but were not yet economical (i.e. enough RAM to drive a video display was many thousands of dollars). Then they looked at historic price trends for that technology (a la Moore’s Law) and predicted when those technologies would become economical.
We can do the same for LLM technologies, even though we don’t have something quite as clean as transistors per dollar to work with. Take a popular, long-standing benchmark, like the Massively-Multitask Language Understanding dataset, and a consistent input approach (five-shot prompting). Then, compare the cost to run language models with various performance levels on this benchmark over time.
![](Reading/Read%20Later/assets/models-prices.png) Figure. For a fixed cost, capabilities are rapidly increasing. For a fixed capability level, costs are rapidly decreasing. Created by co-author Charles Frye using public data on May 13, 2024.
In the four years since the launch of OpenAI’s davinci model as an API, the cost of running a model with equivalent performance on that task at the scale of one million tokens (about one hundred copies of this document) has dropped from $20 to less than 10¢ – a halving time of just six months. Similarly, the cost to run Meta’s LLaMA 3 8B, via an API provider or on your own, is just 20¢ per million tokens as of May of 2024, and it has similar performance to OpenAI’s text-davinci-003, the model that enabled ChatGPT. That model also cost about $20 per million tokens when it was released in late November of 2023. That’s two orders of magnitude in just 18 months – the same timeframe in which Moore’s Law predicts a mere doubling.
Now, let’s consider an application of LLMs that is very useful (powering generative video game characters, a la [Park et al](https://arxiv.org/abs/2304.03442)) but is not yet economical (their cost was estimated at $625 per hour [here](https://arxiv.org/abs/2310.02172)). Since that paper was published in August of 2023, the cost has dropped roughly one order of magnitude, to $62.50 per hour. We might expect it to drop to $6.25 per hour in another nine months.
Meanwhile, when Pac-Man was released in 1980, $1 of today’s money would buy you a credit, good to play for a few minutes or tens of minutes – call it six games per hour, or $6 per hour. This napkin math suggests that a compelling LLM-enhanced gaming experience will become economical sometime in 2025.
These trends are new, only a few years old. But there is little reason to expect this process to slow down in the next few years. Even as we perhaps use up low-hanging fruit in algorithms and datasets, like scaling past the “Chinchilla ratio” of ~20 tokens per parameter, deeper innovations and investments inside the data center and at the silicon layer promise to pick up the slack.
And this is perhaps the most important strategic fact: what is a completely infeasible floor demo or research paper today will become a premium feature in a few years and then a commodity shortly after. We should build our systems, and our organizations, with this in mind.
## Enough 0 to 1 demos, it’s time for 1 to N products
We get it, building LLM demos is a ton of fun. With just a few lines of code, a vector database, and a carefully crafted prompt, we create ✨magic ✨. And in the past year, this magic has been compared to the internet, the smartphone, and even the printing press.
Unfortunately, as anyone who has worked on shipping real-world software knows, there’s a world of difference between a demo that works in a controlled setting and a product that operates reliably at scale.
> There’s a large class of problems that are easy to imagine and build demos for, but extremely hard to make products out of. For example, self-driving: It’s easy to demo a car self-driving around a block; making it into a product takes a decade. - [Andrej Karpathy](https://x.com/eugeneyan/status/1672692174704766976)
Take, for example, self-driving cars. The first car was driven by a neural network in [1988](https://proceedings.neurips.cc/paper/1988/file/812b4ba287f5ee0bc9d43bbf5bbe87fb-Paper.pdf). Twenty-five years later, Andrej Karpathy [took his first demo ride in a Waymo](https://x.com/karpathy/status/1689819017610227712). A decade after that, the company received its [driverless permit](https://x.com/Waymo/status/1689809230293819392). That’s thirty-five years of rigorous engineering, testing, refinement, and regulatory navigation to go from prototype to commercial product.
Across industry and academia, we’ve observed the ups and downs for the past year: Year 1 of N for LLM applications. We hope that the lessons we’ve learned—from [tactics](https://applied-llms.org/#tactical-nuts-bolts-of-working-with-llms) like evals, prompt engineering, and guardrails, to [operational](https://applied-llms.org/#operational-day-to-day-and-org-concerns) techniques and building teams to [strategic](https://applied-llms.org/#strategy-building-with-llms-without-getting-out-maneuvered) perspectives like which capabilities to build internally—help you in year 2 and beyond, as we all build on this exciting new technology together.
---
## Stay In Touch
If you found this useful and want updates on write-ups, courses, and activities, subscribe below.
You can also find our individual contact information on our [about page](https://applied-llms.org/about.html).
## Acknowledgements
This series started as a convo in a group chat, where Bryan quipped that he was inspired to write “A Year of AI Engineering”. Then, ✨magic✨ happened, and we were all pitched in to share what we’ve learned so far.
The authors would like to thank Eugene for leading the bulk of the document integration and overall structure in addition to a large proportion of the lessons. Additionally, for primary editing responsibilities and document direction. The authors would like to thank Bryan for the spark that led to this writeup, restructuring the write-up into tactical, operational, and strategic sections and their intros, and for pushing us to think bigger on how we could reach and help the community. The authors would like to thank Charles for his deep dives on cost and LLMOps, as well as weaving the lessons to make them more coherent and tighter—you have him to thank for this being 30 instead of 40 pages! The authors thank Hamel and Jason for their insights from advising clients and being on the front lines, for their broad generalizable learnings from clients, and for deep knowledge of tools. And finally, thank you Shreya for reminding us of the importance of evals and rigorous production practices and for bringing her research and original results.
Finally, we would like to thank all the teams who so generously shared your challenges and lessons in your own write-ups which we’ve referenced throughout this series, along with the AI communities for your vibrant participation and engagement with this group.
## About the authors
See the [about page](https://applied-llms.org/about.html) for more information on the authors.
If you found this useful, please cite this write-up as:
> Yan, Eugene, Bryan Bischof, Charles Frye, Hamel Husain, Jason Liu, and Shreya Shankar. 2024. ‘Applied LLMs - What We’ve Learned From A Year of Building with LLMs’. Applied LLMs. 8 June 2024. https://applied-llms.org/.
or
```
@article{AppliedLLMs2024,
title = {What We've Learned From A Year of Building with LLMs},
author = {Yan, Eugene and Bischof, Bryan and Frye, Charles and Husain, Hamel and Liu, Jason and Shankar, Shreya},
journal = {Applied LLMs},
year = {2024},
month = {Jun},
url = {https://applied-llms.org/}
}
```

19
Reading/Done Reading/What is a Boy.md → Reading/Read Later/What is a Boy.md
View File

@@ -1,13 +1,26 @@
Clipped from: [https://blog.traillifeusa.com/what-is-a-boy?utm_campaign=Raising%20Godly%20Boys&utm_medium=email&_hsmi=219419808&_hsenc=p2ANqtz-9tkiaPng4HKJvYRy6Ee8wBSX_GRpZ-nKWnj4tZJxJ13W0R00aCDx3IevUfMoI2iUWmB4GMzcGgFwHTSSQIWosWfF_FPw&utm_content=219419808&utm_source=hs_automation](https://blog.traillifeusa.com/what-is-a-boy?utm_campaign=Raising%20Godly%20Boys&utm_medium=email&_hsmi=219419808&_hsenc=p2ANqtz-9tkiaPng4HKJvYRy6Ee8wBSX_GRpZ-nKWnj4tZJxJ13W0R00aCDx3IevUfMoI2iUWmB4GMzcGgFwHTSSQIWosWfF_FPw&utm_content=219419808&utm_source=hs_automation) [[ReadItLater]] [[Article]]
- [x] Read
![What is a Boy?](Exported%20image%2020240808113917-0%201.jpeg) # [What is a Boy?](https://blog.traillifeusa.com/what-is-a-boy)
Alan Beck Feb 24, 2021 [15 Comments](https://blog.traillifeusa.com/what-is-a-boy#comments-listing)
Between the innocence of babyhood and the dignity of manhood we find a delightful creature called a boy. Boys come in assorted sizes, weights, and colors, but all boys have the same creed: to enjoy every second of every minute of every hour of every day and to protest with noise (their only weapon) when their last minute is finished and the adult males pack them off to bed at night. Between the innocence of babyhood and the dignity of manhood we find a delightful creature called a boy. Boys come in assorted sizes, weights, and colors, but all boys have the same creed: to enjoy every second of every minute of every hour of every day and to protest with noise (their only weapon) when their last minute is finished and the adult males pack them off to bed at night.
Boys are found everywhere—on top of, underneath, inside of, climbing on, swinging from, running around, or jumping to. Boys are found everywhere—on top of, underneath, inside of, climbing on, swinging from, running around, or jumping to.
Mothers love them, little girls hate them, older sisters and brothers tolerate them, adults ignore them, and Heaven protects them. Mothers love them, little girls hate them, older sisters and brothers tolerate them, adults ignore them, and Heaven protects them.
A boy is truth with dirt on its face, beauty with a cut on its finger, wisdom with bubble gum in its hair, and the hope of the future with a frog in its pocket. When you are busy, a boy is an inconsiderate, bothersome, intruding jangle of noise. When you want him to make a good impression, his brain turns to jelly or else he becomes a savage, sadistic, jungle creature bent on destroying the world and himself with it. A boy is truth with dirt on its face, beauty with a cut on its finger, wisdom with bubble gum in its hair, and the hope of the future with a frog in its pocket. When you are busy, a boy is an inconsiderate, bothersome, intruding jangle of noise. When you want him to make a good impression, his brain turns to jelly or else he becomes a savage, sadistic, jungle creature bent on destroying the world and himself with it.
A boy is a composite—he has the appetite of a horse, the digestion of a sword-swallower, the energy of a pocket-sized atomic bomb, the curiosity of a cat, the lungs of a dictator, the imagination of a Paul Bunyan, the shyness of a violet, the audacity of a steel trap, the enthusiasm of a firecracker, and when he makes something, he has five thumbs on each hand. He likes ice cream, knives, saws, Christmas, comic books, the boy across the street, woods, water (in its natural habitat), large animals, Dad, trains, Saturday mornings, and fire engines. A boy is a composite—he has the appetite of a horse, the digestion of a sword-swallower, the energy of a pocket-sized atomic bomb, the curiosity of a cat, the lungs of a dictator, the imagination of a Paul Bunyan, the shyness of a violet, the audacity of a steel trap, the enthusiasm of a firecracker, and when he makes something, he has five thumbs on each hand. He likes ice cream, knives, saws, Christmas, comic books, the boy across the street, woods, water (in its natural habitat), large animals, Dad, trains, Saturday mornings, and fire engines.
He is not much for Sunday School, company, schools, books without pictures, music lessons, neckties, barbers, girls, overcoats, adults, or bedtime. Nobody else is so early to rise, or so late to supper. Nobody else gets so much fun out of trees, dogs, and breezes. Nobody else can cram into one pocket a rusty knife, a half-eaten apple, three feet of string, an empty Bull Durham sack, two gum drops, six cents, a slingshot, a chunk of unknown substance, and a genuine supersonic code ring with a secret compartment. He is not much for Sunday School, company, schools, books without pictures, music lessons, neckties, barbers, girls, overcoats, adults, or bedtime. Nobody else is so early to rise, or so late to supper. Nobody else gets so much fun out of trees, dogs, and breezes. Nobody else can cram into one pocket a rusty knife, a half-eaten apple, three feet of string, an empty Bull Durham sack, two gum drops, six cents, a slingshot, a chunk of unknown substance, and a genuine supersonic code ring with a secret compartment.
A boy is a magical creature—you can lock him out of your workshop, but you can’t lock him out of your heart. You can get him out of your study, but you can’t get him out of your mind. Might as well give up—he is your captor, your jailer, your boss, and your master—a freckled-faced, pint-sized, cat-chasing, bundle of noise. But when you come home at night with only shattered pieces of your hopes and dreams, he can mend them like new with two magic words, "Hi Dad!" A boy is a magical creature—you can lock him out of your workshop, but you can’t lock him out of your heart. You can get him out of your study, but you can’t get him out of your mind. Might as well give up—he is your captor, your jailer, your boss, and your master—a freckled-faced, pint-sized, cat-chasing, bundle of noise. But when you come home at night with only shattered pieces of your hopes and dreams, he can mend them like new with two magic words, "Hi Dad!"
---
\---
[Trail Life USA](http://www.traillifeusa.com/) is designed uniquely for boys. Established on timeless values and set in the context of outdoor adventure, boys from Kindergarten through 12th grade are engaged in a Troop setting by male mentors where they are challenged to grow in character, understand their purpose, serve their community, and develop practical leadership skills to carry out the mission for which they were created. [Trail Life USA](http://www.traillifeusa.com/) is designed uniquely for boys. Established on timeless values and set in the context of outdoor adventure, boys from Kindergarten through 12th grade are engaged in a Troop setting by male mentors where they are challenged to grow in character, understand their purpose, serve their community, and develop practical leadership skills to carry out the mission for which they were created.
    [![Start a Troop](Reading/Read%20Later/assets/Start%20a%20Troop.png)](https://cta-redirect.hubspot.com/cta/redirect/6459804/6b97d556-3b4d-4b1c-bd0f-34ede9c766c8)    [![Find a Troop](Reading/Read%20Later/assets/Find%20a%20Troop.png)](https://cta-redirect.hubspot.com/cta/redirect/6459804/9420b3e5-7e8d-4eed-85d4-79dc384b6f91)

BIN
Reading/Read Later/assets/01.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 122 KiB

BIN
Reading/Read Later/assets/02.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 203 KiB

BIN
Reading/Read Later/assets/03.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 193 KiB

BIN
Reading/Read Later/assets/04.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 82 KiB

BIN
Reading/Read Later/assets/05.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 52 KiB

BIN
Reading/Read Later/assets/06.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 155 KiB

BIN
Reading/Read Later/assets/07.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 96 KiB

BIN
Reading/Read Later/assets/08.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 253 KiB

BIN
Reading/Read Later/assets/09.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 147 KiB

BIN
Reading/Read Later/assets/10.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 124 KiB

BIN
Reading/Read Later/assets/11.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 144 KiB

BIN
Reading/Read Later/assets/12.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 127 KiB

BIN
Reading/Read Later/assets/13.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 143 KiB

BIN
Reading/Read Later/assets/14.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 120 KiB

BIN
Reading/Read Later/assets/15.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 142 KiB

BIN
Reading/Read Later/assets/16.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 147 KiB

BIN
Reading/Read Later/assets/17.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 89 KiB

BIN
Reading/Read Later/assets/18.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 89 KiB

BIN
Reading/Read Later/assets/19.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 77 KiB

BIN
Reading/Read Later/assets/20.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 80 KiB

BIN
Reading/Read Later/assets/21.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 146 KiB

BIN
Reading/Read Later/assets/22.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 166 KiB

BIN
Reading/Read Later/assets/23.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 205 KiB

BIN
Reading/Read Later/assets/24.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 125 KiB

BIN
Reading/Read Later/assets/26.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 102 KiB

BIN
Reading/Read Later/assets/Allocation associated with asynchronous operations on .NET Core with pooling.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.0 KiB

BIN
Reading/Read Later/assets/Allocation associated with asynchronous operations on .NET Core.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 7.4 KiB

BIN
Reading/Read Later/assets/Allocation associated with asynchronous operations on .NET Framework.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 27 KiB

BIN
Reading/Read Later/assets/Convolving.gif Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 743 KiB

BIN
Reading/Read Later/assets/Download Strategic Project Alignment in an Agile World ➜.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.2 KiB

BIN
Reading/Read Later/assets/Find a Troop.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.1 KiB

BIN
Reading/Read Later/assets/How Does Scrumban Work ProductPlan.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 137 KiB

BIN
Reading/Read Later/assets/How-willing-are-enterprises-to-use-LLMs-for-different-use-cases_-1-scaled.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 195 KiB

BIN
Reading/Read Later/assets/New call-to-action.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 72 KiB

BIN
Reading/Read Later/assets/Read the Agile Product Manager's Guide to Building Better Roadmaps.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 5.5 KiB

BIN
Reading/Read Later/assets/Stack overflow due to improper handling of synchronous completion.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

BIN
Reading/Read Later/assets/Start a Troop.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.3 KiB

BIN
Reading/Read Later/assets/Stephen Ashton.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 8.1 KiB

BIN
Reading/Read Later/assets/Stephen Toub - MSFT.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.6 KiB

BIN
Reading/Read Later/assets/The Basics of Scrum.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 313 KiB

BIN
Reading/Read Later/assets/alice-in-wonderland-cat.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 367 KiB

BIN
Reading/Read Later/assets/alien-cat.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 326 KiB

BIN
Reading/Read Later/assets/evalgen.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 955 KiB

BIN
Reading/Read Later/assets/httpswww.fennel.aiblogcontentimages2022101cacae00-cc59-46b1-bac2-71eeb72828ac_960x720-1.jpeg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

BIN
Reading/Read Later/assets/man-cat.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 238 KiB

BIN
Reading/Read Later/assets/man-people.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 512 KiB

BIN
Reading/Read Later/assets/models-prices.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 143 KiB

BIN
Reading/Read Later/assets/neural-cat.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 207 KiB

BIN
Reading/Read Later/assets/niah.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 88 KiB

BIN
Reading/Read Later/assets/spreadsheet.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 362 KiB

BIN
Reading/Read Later/assets/woman-carrot-stick.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 177 KiB

1
_Templates/DailyNote.md
View File

@@ -1,3 +1,4 @@
# To-dos # To-dos
- [ ]
# Summary # Summary