vault backup: 2024-08-08 15:47:33
This commit is contained in:
@@ -0,0 +1,94 @@
|
||||
Clipped from: [https://www.parand.com/a-completely-non-technical-explanation-of-ai.html](https://www.parand.com/a-completely-non-technical-explanation-of-ai.html)
|
||||
|
||||
# Overview
|
||||
|
||||
This document will explain what neural networks are and how they work, which will help you understand how AI and machine learning work. In the scenario below you'll play the part of the neural network.
|
||||
|
||||
# Day One
|
||||
|
||||
First day of your new job as a "classifier" your boss walks in and drops a big spreadsheet of numbers on your desk.
|
||||
"Is this a cat?" she asks.
|
||||
Confused you ask "What?"
|
||||
|
||||
|
||||
|
||||
"Is this a cat?"
|
||||
Even more confused, you respond "I don't know".
|
||||
"Wrong!" she says, and slaps you across the face.
|
||||
Before you've had a chance to be shocked she drops another large spreadsheet on your desk. "Is this a cat?"
|
||||
"I don't understand" you respond.
|
||||
"Wrong!" she says, and slaps you again.
|
||||
Another spreadsheet. "Is this a cat?"
|
||||
Not wanting another slap, you meekly respond "Yes?"
|
||||
"Correct! Good job!" she says, and gives you a wonderful reward. Almost makes up for the slaps.
|
||||
Another spreadsheet. Cat?
|
||||
Slightly more confident and wanting another reward you respond "Yes"
|
||||
"Wrong!". Another slap.
|
||||
You are very confused. "I don't understand what's going on. You haven't told me the rules, you haven't told me how to figure out something is a cat, you haven't given me the logic to figure this out. You haven't trained me."
|
||||
"Correct" says the boss. "This is training. Trust me, you're going to get very good at recognizing cats"
|
||||
|
||||
|
||||
|
||||
Another spreadsheet. This time you focus on the sheet. It's 256 columns and 256 rows, filled with numbers. Is it a cat? You don't know, so you guess.
|
||||
Another and another. Right, wrong, wrong, right again, it keeps going. Slaps and rewards.
|
||||
You're starting to notice some patterns - if the sheet is almost all zeros then it's not a cat. You look at the center of the sheet - that part should have larger numbers.
|
||||
It's getting slightly better - you're getting more rewards than slaps, guessing correctly more often than not. It's all about the patterns of the numbers.
|
||||
|
||||
# Day Two
|
||||
|
||||
This is the strangest job you've ever had. The slaps are terrible, but the rewards are so great you don't want to quit. How do you get better at this? It's not really possible for one person. If you had more people they could focus on different aspects of the sheet, look for different patterns, and you could use their findings to make better guesses.
|
||||
You hire 10 people and bring them to work the next day. When you get a spreadsheet you show it to those 10 people and ask them "Is this a cat?"
|
||||
They are as confused as you were. You force them to guess. The first guy looks like an idiot, so you decide to go with the opposite of what he says. The third lady looks really thoughtful so you put a lot of weight on what she says. In your mind you assign a weight to each of their guesses to come up with your final answer.
|
||||
Each time you get a reward or punishment you share it with the 10 people you've hired: you reward or slap them based on how much weight you put on their input and how much they contributed to you getting the answer right or wrong. You learn the patterns: if the first guy says a strong no and the third lady a strong yes then it's very likely a cat. You learn many more patterns like this.
|
||||
The 10 people are learning to hone their opinions based on the rewards and punishments they get. They can focus on different aspects of the sheet, and their opinions together can tell you a lot about the sheet.
|
||||
After what feels like an epoch and a lot of spreadsheets, eventually you get better. You're getting more rewards. This is working. Your boss says you're very perceptive, and starts calling you Perceptron.
|
||||
|
||||
|
||||
|
||||
# Day Ten
|
||||
|
||||
What if you get more people involved? You could make it 50 people reporting to you, but that'd be hard to manage. How about we add another layer of people before your 10?
|
||||
You hire 50 more people, have them give their guesses to the 10 people that report to you. You're now very removed from looking at the spreadsheets - instead you rely on the patterns found by the first layer of the people, who give their opinions to the second layer of the people, who then inform you. Everybody passes the slaps and rewards down the line based on how much each earlier person's guess contributed to their guess.
|
||||
It takes even longer, but eventually the system starts working. You're much more accurate in identifying when it's a cat.
|
||||
Interestingly you were never taught the rules or logic, and you didn't teach your people the logic or rules. You just propagated the rewards and punishments back through each layer: the more each person's opinion contributes to the answer, the more rewards or punishment you shared with them, and they in turn with the people in the layer behind them. Each person uses the same method to propogate their rewards and punishment back to the people in the layer below them.
|
||||
|
||||
# Neural Networks
|
||||
|
||||
This is how neural networks work: they see many examples and get rewarded or punished based on whether their guesses are correct. They use multiple layers of workers and eventually learn patterns. Importantly no one is teaching them what patterns they should be looking for or telling them the logic or the rules - the networks eventually figure out the patterns and logic based on very many rounds of example, reward, and punishment. This is called **machine learning**, because the machine is learning the rules by itself.
|
||||
Neural networks work well when you have many examples of something (eg. pictures of cats), but it's hard to write the logic and rules to describe how to recognize that thing. Try it - write down some rules for how to recognize a cat (eg. "has 4 legs"), then look at pictures of cats and see where the rules fail (eg. a picture of a cat's head). In these cases you use machine learning so the machine learns the rules by itself.
|
||||
Also note that computers see things as multi-dimensional tables of data. They don't look at a "picture" - they see 3 spreadsheets of numbers representing the RGB values of the picture.
|
||||
|
||||
# Day Forty
|
||||
|
||||
Identifying cats is a lucrative business and you're pretty good at it. But not good enough. How can we get even better? More people, more layers!
|
||||
Unfortunately it takes a long time for people to do their calculations, so you've been stuck with just a few layers of people for a while. Having a lot more people would make the process take too long.
|
||||
One day you run into a group of people who call themselves Gaming People United (GPU). These people play a game that has taught them to be very good at looking at spreadsheets and calculating numbers - in fact they can do a lot of calculations in parallel, very quickly.
|
||||
Excited, you hire a bunch of them and put them to work recognizing cats. Now instead of two layers of people you can have 10! Each layer can focus on higher and higher concepts - the first layer can look for small details (eg. do I see a pattern that looks like a small circle? Do I see a pattern that looks like a sharp edge?), the second layer can look for patterns from the results of the first layer (eg. are there two circles close to each other), the third layer can build on that (eg. are there two circle close to each other, and a triangle below them), and so forth. By the time the results of the layers get to you you have some fairly sophisticated concepts - we see a pattern that looks like a face, we also see patterns that could be legs, and a sharp thing that might be a beak. Now your guess is more informed than ever.
|
||||
You train the layers sending slaps and rewards down through each layer, and after many epochs you get really good at recognizing cats.
|
||||
|
||||
|
||||
|
||||
# Deep Learning
|
||||
|
||||
One of the major break-throughs in machine learning was the advent of _Deep Learning_, which is basically what we describe above - GPUs (Graphics Processing Units) got popular because they enable fast 3D graphics for games. They also happen to be very good at quickly doing the types of calculations neural networks need. Machine learning people started using GPUs for training neural networks, and with this extra speed they could have many more layers - from 3 or 4 layers to 10 or 11, and then hundreds.
|
||||
This addition of layers led to significant advances in neural network performance - very quickly neural networks became the best solution for voice recognition, for image recognition, image creation, and sophisticated language models. This is called **Deep Learning** because there are so many layers, not because it's profound.
|
||||
_You can stop reading here if you want, the following portion is only here because someone asked me what "convolutional neural networks" are._
|
||||
|
||||
# Day Fifty
|
||||
|
||||
How can we make the process even more efficient? You start to realize that looking at the entire sheet is too hard - the first layer of people literally have to stare at the whole thing and try to guess based on that. What if we give them a small portion of the sheet to look at, give a guess for that portion, then move on to the next portion of the sheet, and so on? The first layer of people are focused on lower level concepts anyway - looking for edges, things that look like circles, and so forth - and they can find those in the small portions of the sheet, without the need to look at the whole thing at once.
|
||||
|
||||
|
||||
|
||||
[Via towardsdatascience](https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1)
|
||||
You tell the first layer of people this is what they should start doing. They respond that this sounds very convoluted and they'll only agree to do it if you call them "colonel".
|
||||
This makes the process even faster and more accurate: the first layer become specialists in small features of the sheet, and they learn to be very efficient and fast. You try the same idea for the next several layers: you tell people to focus on small portions of the feedback they get from the layer below them, moving the portion they focus on so eventually the scan across all of the feedback.
|
||||
This specialization and focus makes you even more accurate. Congratulations, you are officially recognized as the best cat classifier in the world.
|
||||
|
||||
# Convolutional Neural Networks
|
||||
|
||||
Convolutions are this idea of applying a specific set of calculations, sometimes called a kernel, to each portion of the input, and scanning your window of attention across the entire image. The set of calculations, or kernel, is learned by each worker - you don't tell it what to calculate or how, it learns what's useful based on rewards and punishments. Convolutional Neural Networks (CNNs) were a significant step forward in the capability of neural networks.
|
||||
|
||||
# Further Reading
|
||||
|
||||
_If you're interested in this topic you might also enjoy the_ _Completley Non-Technical Explanation of ChatGPT_ _series as well._
|
||||
@@ -0,0 +1,77 @@
|
||||
Clipped from: [https://www.parand.com/a-non-technical-explanation-of-chatgpt.html](https://www.parand.com/a-non-technical-explanation-of-chatgpt.html)
|
||||
Continuing the series of non-technical explainers, let's figure out how ChatGPT (and in general Large Language Models, or LLMs) work. This is part one of the ChatGPT explainer, with part two coming soon.
|
||||
It's helpful but not necessary to read the [non-technical explanation of AI](https://www.parand.com/a-completely-non-technical-explanation-of-ai.html) first, if you feel like it read that and come back.
|
||||
|
||||
# Fill In The Blank
|
||||
|
||||
Let's play a quick game of fill in the blank:
|
||||
to be or not to _____
|
||||
Why did you immediately think of the word _be_ to fill in that blank, instead of the word _banana_ or _fish_? Because you've seen the phrase many times and your brain has learned the most likely next word is _be_.
|
||||
Let's do another one:
|
||||
rock and ____
|
||||
Did you think of the word _roll_? Why? Because that's the word you see most often following the words _rock and_.
|
||||
How about this one:
|
||||
I am very _____
|
||||
This is less clear - the next word depends on the context.
|
||||
I just ran a marathon. I am very _____
|
||||
Perhaps now you would say _tired_. Or _happy_, or _proud_.
|
||||
Your brain is picking the most likely next word based on the context of the sentence and based on what words it's seen most frequently in that context.
|
||||
|
||||
# Aliens and CatGPT
|
||||
|
||||
In an astonishing turn of events a race of alien cats have landed on earth. Since you are [the world's foremost cat expert](https://www.parand.com/a-completely-non-technical-explanation-of-ai.html) you have been chosen to communicate with them.
|
||||
|
||||
|
||||
|
||||
Your boss walks in and drops a massive print out of all the alien cat communications on your desk.
|
||||
"Speak cat!" she commands.
|
||||
Unfortunately you don't speak alien cat.
|
||||
You page through the print outs - it looks like pages and pages of gibberish.
|
||||
zoog zeeg zag. zoog zeeg kaz. zoog zeeg bah. rag zoog zeeg. rag zoog ko. kaz rag. zap zoog zeeg. ...
|
||||
|
||||
# Markov to the Rescue
|
||||
|
||||
What to do? You remember your friend Markov who's always talking about languages, words, and their relationships. Maybe he can help. You show him the alien cat communications and ask him if he can help you say something in cat.
|
||||
"Yes!" he exclaims - "We can do this. We will create our response one word at a time, just by picking the best word to say next, and keep going until we have a sentence"
|
||||
"But how do we know what the best word to say next is? Don't we need to know what the words mean?"
|
||||
"Nope, we just need to know what word to say next"
|
||||
"That... doesn't seem like it would work"
|
||||
"Let me show you", says Markov, and grabs the book on his desk, [Alice in Wonderland](https://www.gutenberg.org/ebooks/11).
|
||||
|
||||
|
||||
|
||||
"We just need to know what word to say next. The best word to say next is simply the word that shows up most frequently after the word we're looking at. All we have to do is make a table of how often each word follows another word. We'll call this a frequency table"
|
||||
He shows you how to create the frequency table - you just note down how many times each word follows another. It takes a while to go through all Alice in Wonderland and do the counting. You end up with:
|
||||
Next word after "Alice" → was: 17 times, and: 16 times, thought: 12 times, had: 11 times, ...Next word after "sat" → down: 9 times, silent: 2 times, still: 2 times, for: 1 times, ...Next word after "was" → a: 31 times, the: 20 times, not: 12 times, going: 11 times, ...Next word after "and" → the: 80 times, she: 52 times, then: 31 times, was: 19 times, ...Next word after "a" → little: 59 times, very: 25 times, large: 20 times, great: 17 timesNext word after "very" → much: 10 times, soon: 7 times, curious: 6 times, glad: 5 times...
|
||||
This table gives you a lot of help in forming sentences in the style of Alice in Wonderland. For example, if you start with the word _Alice_, then you'd look that up in the above table and see that the most frequent next word is _was_. And after _was_ you would select _a_. After _a_ you would get _little_. You'd end up with _Alice was a little_.
|
||||
How well does this work? Let's create some sentences:
|
||||
the door, when i beg your verdict," it was quite plainly through the bottle, i'm afraid that they had its nest.
|
||||
You can try this out for yourself on [almeopedia](http://almeopedia.com/markovtest.html).
|
||||
That's not great. Why is it so nonsensical? Think back to our fill in the blank examples at the start of this post: in order to pick a good next word you need context. If someone asked you what word should appear after _rock_, you'd have a hard time picking something reasonable, but if they gave you _rock and_, you'd fairly quickly think of _roll_.
|
||||
What would happen if you considered two words instead of a single word for your context? For one thing your frequency table creation would become harder - now instead of a single line for each word, you'd have a line for each word pair. You'd have to gather stats for all the two word permutations, which is a lot more than the single word case.
|
||||
Next word after "Alice was" → not: 3 times, beginning: 2 times, very: 2 times, ...Next word after "was not" → a: 3 times, here: 1 times, going: 1 times ...Next word after "not here" → before: 1 timesNext word after "not a" → moment: 2 times, bit: 2 times, serpent: 2 times ...
|
||||
Let's create a sentence with this and see how it looks:
|
||||
the hatter, and, burning with curiosity, she decided on going into the garden.
|
||||
You can try this out for yourself on [almeopedia](http://www.almeopedia.com/markov3test.html).
|
||||
That's looking better, it almost makes sense. Let's keep going - instead of two words of context, how about three?
|
||||
The Hatter’s remark seemed to have no sort of chance of her ever getting out of the water, and seemed to quiver all over with diamonds, and walked two and two, as the soldiers did.
|
||||
Four?
|
||||
So she swallowed one of the cakes, and was delighted to find that she knew the name of nearly everything there.
|
||||
Hmm. This is a little too good. It turns out a very similar sentence exists in the original Alice in Wonderland text:
|
||||
So she swallowed one of the cakes, and was delighted to find that she began shrinking directly.
|
||||
Our simple method of picking the most likely next word can result in the system memorizing the text snippets - given long enough context, the next most likely word is exactly the word that appeared in the original text following that context.
|
||||
We can fix this by introducing some randomness - instead of always picking the most likely next word, we can pick somewhat likely next words. This way we're less likely to regurgitate the original text.
|
||||
The good news is our method seems to work for English, so it'll probably work for alien cat language as well.
|
||||
_Aside: You might run into the term "stochastic" as you look into language models - this just means randomly determined. For example, you might hear people argue whether these systems are_ **Stochastic Parrots**_, implying the systems are simply parroting back the original text that they saw, with some randomness thrown in._
|
||||
|
||||
# Large Language Models
|
||||
|
||||
ChatGPT and many other Large Language Models (LLMs) do essentially what you did above: they examine a very large amount of human communication, gather stats (or probabilities) on what words are most likely to follow other words, and play a continous game of fill-in-the-blank. You give them some context (your prompt or question), and they create the reply one word at a time by selecting the word most likely to appear next. They respond with a word, then look at their internal stats to pick a word to follow their first word, then a word to follow their second word, and so forth, one word at a time, until they've formed a response.
|
||||
|
||||
# To be Continued...
|
||||
|
||||
In [part two of this series](https://www.parand.com/a-non-technical-explanation-of-chatgpt-deep-learning.html) we'll look at the problems you'll run into with this method and how deep learning helps you overcome those problems.
|
||||
|
||||
# Further Reading
|
||||
|
||||
_If you're interested in this topic you might also enjoy the other posts in the_ _Non-Technical Explainer_ _series as well._
|
||||
@@ -0,0 +1,193 @@
|
||||
Clipped from: [https://ferd.ca/embrace-complexity-tighten-your-feedback-loops.html](https://ferd.ca/embrace-complexity-tighten-your-feedback-loops.html)
|
||||
This post contains a transcript of the talk I wrote for and gave at [QCon New York 2023](https://qconnewyork.com/presentation/jun2023/embrace-complexity-tighten-your-feedback-loops) for [Vanessa Huerta Granda](https://qconnewyork.com/speakers/vanessahuertagranda)'s [track on resilience engineering](https://qconnewyork.com/track/jun2023/resilience-engineering-culture-system-requirement).
|
||||
The official talk title was "Embrace Complexity; Tighten Your Feedback Loops". That’s the descriptive title for the talk that follows the conference’s guidelines about good descriptive titles. Instead I decided to follow my gut feeling and go with what I think really explains my perspective and the approach I bring with me to work and even my life in general:
|
||||
|
||||
|
||||
|
||||
I take what would probably be a sardonic approach to dealing with life and systems, and so “This is all going to hell anyway” is pervasive to my approach. Things are going to be challenging. There are going to always be pressures that keep pushing our systems to the edge of chaos. I don’t think this can be fixed or avoided. Any improvement will be used to bring it right to that edge. In complex systems, the richness and variability is often there for a reason. Trying to stamp it out in favour of stronger control is likely to create weird issues.
|
||||
So the best I personally hope for is to have some limited influence in steering things the best I can to delay going to hell as long as possible, but that’s it. And my talk is going to focus on a lot of these approaches, but first, I want to explain why I feel things are that way.
|
||||
|
||||
|
||||
|
||||
In what is probably my favorite paper ever, titled [Moving Off The Map](https://ferd.ca/notes/paper-moving-off-the-map.html), Ruthanne Huising ran ethnological studies by embedding herself into projects within many large corporations doing planned organizational changes. In supporting these efforts, they were doing “tracing” of their functions, which meant gathering a lot of data about what activities take place, what interactions and hand-offs exist, what information and tools are used and required? How long do tasks take? How do people and teams deal with errors? Generally asking the question “what do we do here?” and wondering with whom they do it.
|
||||
To build these maps they generally reached out to experts within the organization who were supposed to know how things were working. Even then, they were really surprised.
|
||||
|
||||
|
||||
|
||||
One explained that “it was like the sun rose for the first time… I saw the bigger picture.” Participants had never seen the pieces (jobs, technologies, tools, and routines) connected in one place, and they realized that their prior view was narrow and fractured, despite being considered experts.
|
||||
Others would state that “the problem is that it was not designed in the first place.” The system was not designed nor coordinated, but generally showed the result of various parts of the organization making their own decisions, solving local problems, and adapting in a decentralized manner.
|
||||
The last quote comes from events when a manager at one of the organizations walked the CEO through the map, highlighting the lack of design and the disconnect between strategy and operations. The CEO sat down, put his head on the table, and said, “This is even more fucked up than I imagined.” He realized that the operation of his organization was out of his control, and that his grasp on it was imaginary.
|
||||
|
||||
|
||||
|
||||
One of the most surprising results reported in there was about tracking the people who participated in organizing and running the change projects, and seeing who got promoted, who left, and who moved around the org or industry they were in.
|
||||
She found out there were two main types of outcome. The first group turned out to be filled with people who got promotions. They were mostly folks who worked in communications, training, who managed the costs and savings of the projects, or those who helped do process design. Follow-up interviews revealed that most of them attributed their promotions to having worked on a big project to put under their belt, and to frequently working with higher-ups, which both helped with getting promoted.
|
||||
Another group however mostly contained people who moved to the periphery: away from core roles at the organization, sometimes becoming consultants, or leaving altogether. Those who fit this category happened to be the people who collected the data and created the map. They attributed their moves to either feeling like they finally understood the organization better, felt more empowered to change things, or became so alienated by the results they wanted to get out.
|
||||
So the question of course became how come people who feel they understand how the organization truly works and who want to change it move _away_ from the central roles and positions, and into the peripheral ones?
|
||||
|
||||
|
||||
|
||||
The fatal insight, according to Huising, is something sociologists knew for a good while: the culture and the order imposed to organizations, groups, and even societies is often emergent and negotiated. And while it's obvious that these structures dictate a lot of actions, the actions themselves can preserve or change the structures around them.
|
||||
The feelings of empowerment and alienation come in no small part because people realized that they could change a lot more than they could, albeit often from outside the core decision-making that enforces the structure (while understanding how that core works), or because the ways they thought they were impacting things was shown not to be effective and they felt disembedding.
|
||||
|
||||
|
||||
|
||||
Another thing you have possibly experienced and isn’t in the paper now is one of differentiating between the nominal and actual structure of the org, the emergent one that depends on power dynamics, who knows what or whom, who likes or dislikes each other, and so on.
|
||||
If you've ever worked in a flat organization, like the one in the middle here, is that even though you have little management structure to speak of, power dynamics and decision-making authority still exists. People who have no power attached to their role are still going to be consulted or inserted in the decision-making flow of the organization, they're still going to be influential and have the ability to make or break projects, but just with less obvious accountability.
|
||||
The nominal structure is the one where each level of management and within the organizational ladder specifies how information flows, and how authority is applied. It's what we see on the left in a more traditional org structure, and this way of organizing groups will simultaneously be useful to align efforts and to constrain them. It makes accountability more explicit and transparent, but structurally will prevent people from doing unspecified things, whether they would be harmful or useful.
|
||||
The emergent structure is always there as well. It is implicit, always changing, and not necessarily constrained to your own organization either. Sometimes, people who know how to run, maintain, or operate components, or whom people listen to, are not even in your org anymore. They might have moved away (to a different team or even a competitor), retired, or never been in and they have just published a really influential piece of media and people look up to them.
|
||||
But who knows what, works with whom, and who can move things around in specific contexts can be key to successful initiatives. Even if the organizational structure has often been put in place to constrain change, as a barrier to people working in mis-aligned ways, some folks central to the emergent structure, in key contexts, have earned enough trust to be allowed tacitly to bend and break the rules. They can choose not to enforce the rules, or the rules are not enforced as tightly for them with the hopes of positive outcomes—even if sometimes it can get you the opposite result.
|
||||
I’m not here to argue in favor of one or the other structure, but mostly that in my experience, driving change or making initiatives succeeds the most when catering to both structures at once, or rather fails when only looking at one and being blocked by the other. They're both real, both distinct, and pretending only either exists is bound to cause you grief.
|
||||
|
||||
|
||||
|
||||
As a continuation of this, the way people work every day is often different from the way people around them imagine their work is being done. The gap between how work is thought to be done and how it is actually done is a major but generally invisible factor in how systems work out.
|
||||
Based on flawed mental models of the work, procedures and prescriptions are given about how to do work, and will vary in inaccuracy. People will imagine things like, for example, writing all the tests before writing or modifying any code and that code coverage could be ideal and then that it will all be reviewed in depth by an expert, and will enshrine this as a policy.
|
||||
But the application of these policies is never perfect. Sometimes code doesn't have an owner, or due to crunch time and based on how much the reviewer and author trust each other, the review won't be as in-depth as expected.
|
||||
When you see this mismatch causing people to ignore or bend rules, you can choose to apply authority and ask for a stricter rule-following. This pattern of enforcing the rules harder will likely drive these adaptations underground rather than stamping them out, because real constraints drive that behavior.
|
||||
In turn, the work as disclosed will be less adequate, and the work as imagined progressively gets worse and worse.
|
||||
This becomes a feedback loop of misunderstanding and at some point, like our devastated CEO, you’re not managing the real world anymore.
|
||||
|
||||
|
||||
|
||||
To demonstrate this, earlier this year I went to my local mastodon network—so you know this is super scientific—and ran a poll about time sheets. The question was "If you're a software developer who ever worked for an employer who had you track your time hourly into specific projects/customer accounts and you were short on time budget, did you..."
|
||||
Multiple answers were accepted. Fewer than 15% of people either stopped work, worked without tracking their time anymore (for free), or shifted their time into other projects with more buffer space.
|
||||
Roughly a third of people reported billing anyway, some stating that it's not their problem the time allocation wasn't realistic or adequate.
|
||||
But the vast majority of answers, nearly 60%, came from people saying "my time tracking was always fake and lies," with some people stating they even wrote applications to generate realistic-looking time sheets.
|
||||
What we can see here is an example of how work-as-imagined gets translated into policies ("people do their work in projects, and account for their time"), which at some point doesn't get applied right anymore. If I were to suppose, it could be things like not being allowed to go over time, or just finding the practice useless. But the end result is that the time sheet data just isn't trustworthy, and then it can get used again and again in further decision making.
|
||||
The gap widens, and our CEO might also get to think "this is all fucked up."
|
||||
|
||||
|
||||
|
||||
Part of the reason for this is that every day decisions are made by trying to deal with all sorts of pressures coming from the workplace, which includes the values communicated both as spoken and as acted out. People generally want to do a good job and they’ll try to balance these conflicting values and pressures as well as they can.
|
||||
The outcome of that trade-off being a success or a failure isn’t known ahead of time, but these small decisions accumulate based on the feedback we get from each of these and can end up compounding and accumulating, either as improvements, or as erosion that makes organizations more brittle, or really anywhere in between. People adopt the organization’s constraints as their own, and this set of pressures is the kind of stuff that drives processes to the edge of chaos over and over again.
|
||||
These accumulations of small decisions, these continuous negotiations, that’s one way your culture can define itself. Small common everyday acts and small amounts of social pressure you can apply locally has an impact, as minor as it might be, and compounds. You can easily foster your own local counterculture within a team if you want to. This can both be good (say in Skunkworks where you bypass a structure to do important work) or bad (normalizing behaviors that are counterproductive and can create conflict).
|
||||
|
||||
|
||||
|
||||
So while a lot of the work you can do to improve reliability or resilience as a whole can be driven locally, my experience is that you nevertheless get the best results by also aligning with or re-aligning some of the organizational pressures and values usually set from above.
|
||||
The idea here is to start looking at the organization from both ends: how can we support the people dealing with the trade-offs in conflicting goals as they happen, how can we influence the higher-level values and pressures such that we can try to reduce how often these conflicts happen even though they will definitely keep happening, and how can we better carry context and feedback across both ends so that we constantly adjust as best as we can. A system perspective on interactions, rather than focusing on components is also something I've found useful. The rest of the talk is going to be spent on these ideas.
|
||||
_(as a note, the third drawing is_ _Dimethylmercury__, a highly volatile, reactive, flammable, and colorless liquid. It's one of the strongest known neurotoxins, and less than 0.1 mL is enough to kill you through your skin, and gloves apparently do a bad job at protecting you)_
|
||||
|
||||
|
||||
|
||||
So let's start with negotiating trade-offs, with a bit more of an ops-y perspective, because that's where I'm coming from.
|
||||
|
||||
|
||||
|
||||
This is a painful one sometimes, especially when you have highly professional people who take their jobs seriously.
|
||||
Locally for you as a DevOps or SRE team, there is a need for the awareness of what the organization and customers actually care about. Some availability targets become useless metrics because they’re disconnected from what users want, and you’re just going to burn people out doing it.
|
||||
I learned this lesson when talking to the SRE manager of one of these websites where people pick their favorite images, put them on boards, and get shown ads. He was telling me how their site was having a lot of reliability issues. It would keep going down, his team would do heroics to bring it back up, and it'd open all over again.
|
||||
He felt his team was burning out. They were losing people, and their call rotation was so painful they were also having issues hiring back into it. He was seeing the death spiral happening and was wondering what to do.
|
||||
He added that there were perverse incentives at play: every time the site went down, they stopped showing images, but not ads. That meant that during incidents, they still earned money, but no longer paid for bandwidth. The site was more profitable when it failed than when it worked, and seemingly, users didn't mind much.
|
||||
They were not getting help, nobody seemed to consider it a problem. Not really knowing what to say, I just asked off-hand: "are you trying to deliver more reliability than people are asking for? What if you just stopped and let it burn more and rested your people?" He thought about it seriously, and said "yeah, maybe."
|
||||
I never actually found out what happened after this, but it still stuck with me as a really good question to ask from time to time.
|
||||
|
||||
|
||||
|
||||
In some cases, the answer will be "yes, we want to be this reliable". But you just won't be given the right tools to do it.
|
||||
At Honeycomb, we want on-call rotations to have 5-8 people on them because that’s what we think gives a good pace that maintains a balance between how rested and how out-of-practice people can be. Not too often nor not often enough.
|
||||
But many services are owned by smaller teams of 3-4 people. If we wanted rotations to be made of people who know all their components in depth, where they could build expertise and operate what they wrote, we couldn't reach a sustainable frequency.
|
||||
Instead, to keep the pace right, we tend to put together rotations made of multiple teams, for which people won’t understand many of the components they operate. This in turn makes us prepare to deal with more unknown: fewer runbooks, more high-level switches and manual circuit breakers to gracefully degrade parts of the system to keep it running off-hours, and with different patterns of escalation.
|
||||
We started leaning more heavily on this when a big public product launch required shipping a new feature, which was to be operated by a team that didn't have full time to get it operationally ready. When our SRE team was discussing with them what still needed to be done, we asked for a few simple things: a way to switch the feature off for a single customer, and a way to turn it off entirely, that wouldn't break the rest of the product. The rest we could add as we went.
|
||||
We ended up using these switches a few times, one of which prevented a surprising write-amplification bug that could have killed the whole system, and instead let us wait a few hours for the code owners to get up and fix it at a leisurely pace. We're going to accept a bit of well-scoped, partial unavailability—something that happens a lot in large distributed systems—in order to keep the system stable.
|
||||
The person wearing the pager often does triage and that weird issues will eventually be handled by code owners, just not right now.
|
||||
This approach means that rather than working impossible hours and making inhuman efforts foreseeing the unforeseeable, we keep moving rather fast, gather feedback, find issues, and turn around a bit more on a dime. In order to do this though, there’s a general understanding that production issues may turn parts of the roadmap upside down, that escalations outside of the call rotation can disrupt project work, and so on.
|
||||
That’s one of the complex trade-offs we can make between staffing, training/onboarding, capacity planning, iterative development, testing approaches, operations, roadmap, and feature delivery. And you know, for some parts of our infra we make different decisions because the consequences and mechanisms differ.
|
||||
|
||||
|
||||
|
||||
To make these tricky decisions, you have to be able to bring up these constraints, these challenges, and have them be discussed openly without a repression that forces them underground.
|
||||
One of my favorite examples is from a prior job, where one of my first mandates was to try and help with their reliability story. We went over 30 or so incident reports that had been written over the previous year, and a pattern that quickly came up was how many reports mentioned "lack of tests" (or lack of good tests) as causes, and had "adding tests" in action items.
|
||||
By looking at the overall list, our initial diagnosis was that testing practices were challenging. We thought of improving the ergonomics around tests (making them faster) and to also provide training in better ways to test. But then we had another incident where the review reported tests as an issue, so I decided to jump in.
|
||||
I reached out to the engineers in question and asked about what made them feel like they had enough tests. I said that we often write tests up until the point we feel they're not adding much anymore, and that I was wondering what they were looking at, what made them feel like they had reached the points where they had enough tests. They just told me directly that they knew they didn't have enough tests. In fact, they knew that the code was buggy. But they felt in general that it was safer to be on-time with a broken project than late with a working one. They were afraid that being late would put them in trouble and have someone yell at them for not doing a good job.
|
||||
When I went up to upper management, they absolutely believed that engineers were empowered and should feel safe pressing a big red button that stopped feature work if they thought their code wasn't ready. The engineers on that team felt that while this is what they were being told, in practice they'd still get in trouble.
|
||||
There's no amount of test training that would fix this sort of issue. The engineers knew they didn't have enough tests and they were making that tradeoff willingly.
|
||||
|
||||
|
||||
|
||||
_(note: this slide was cut from the presentation since I was short on time)_
|
||||
Speaking of which, sometimes it’s also fine to drop reliability because there are bigger systemic threats.
|
||||
Sometimes you can eat downtime or degraded service because it’s going to keep your workload manageable and people from burning out. or maybe you take a hit because a big customer that makes you hit your targets as an org and can prevent layoffs will put some things over the limit and a component’s performance will suffer. You can’t be the department of “no” and that negotiation has to be done across departments.
|
||||
Conversely however, you have to be able to call out when your teams are strained, when targets aren’t being met and customers are complaining about it. It means you might be right, and some deadlines or feature delivery could be deferred to make room for others.
|
||||
How do you deal with capacity planning when making your biggest customer renew their contract prevents you from signing up another one that’s as big? Very carefully, by talking it out by all the involved people.
|
||||
And sometimes that trade-off is very reasonable. And good engineering requires you to move it earlier in the lifecycle of software than just around incidents. It’s much simpler to change the shape of a product’s features than it is to deliver the perfect distributed systems sometimes. Making your features take the ideal shape to deal with the reality of physics is one of the things a good collaborative approach can facilitate.
|
||||
|
||||
|
||||
|
||||
So we can make tradeoff negotiation simpler by having these honest discussions, but in many cases this ability to discuss constraints to influence how work takes place brings us to this next step, where we don’t only influence the decisions people make, but surface these challenges to influence how the organization applies its pressures. This is moving from the local level to the alignment to the broader org structure.
|
||||
|
||||
|
||||
|
||||
Metrics are good to direct your attention and confirm hypotheses, but not as a target, and they’re unlikely to be good for insights. [They’re compression, and it can be unreliable](https://ferd.ca/plato-s-dashboards.html).
|
||||
The thing you generally care about is your customer or user's satisfaction, but there's a limit to how many times you can ask "would you recommend us to a friend?" and still get a good signal. So you start picking a surrogate variable.
|
||||
You assume that when the site is down and slow, people are mad, and you make being up and fast a proxy for satisfaction. But then that signal is a bit messy and not super actionable, because it can include user devices or bits of the network you don't control, plus it's hard to measure, so you'll settle for response time at the edge of your infrastructure. This loses fidelity into the signal, but it'll get worse as you suddenly find some teams have more data than others, and they use features differently, so you either need a ton of alarms or fewer messier ones, but you're getting further and further away from whether people are actually satisfied.
|
||||
This loss of context is a critical part of dealing with systems that are too complex to adequately be represented by a single aggregate. Whenever a signal is useful, an in-depth dive is usually worth it if you are looking to embrace complexity.
|
||||
The metric is better used to attract your attention than as a target or as something that tells you what to know. Seek to explain and understand the metric first, not to change it.
|
||||
|
||||
|
||||
|
||||
As a related concept, if you act on a leading indicator, it stops leading, particularly when it’s influenced by trade-offs.
|
||||
Metrics that become their own targets and are gamed of course lose meaningfulness; this is one of the most common issues with counting incidents and then debating whether an outage should or shouldn’t be declared in a way that might affect the tally rather than addressing it directly.
|
||||
But other metrics are of interest as well. If you evaluate your total capacity by some bottleneck’s value, and that this bottleneck is a target of optimization work, you will lose the ability to easily know when or how to scale up because that bottleneck possibly hid something else. This is contributing to a non-negligible portion of our incidents at work I believe. We fix a thing that acted as an implicit blocker and off we go into the great unknown.
|
||||
Our storage engine's disk storage used to be our main bottleneck. We drove scaling out and rebalancing traffic based on how close we were to heavy usage across multiple partitions. This was a useful signal, but it also drove costs up, and eventually became the target of optimization.
|
||||
An engineer successfully made our data offloading almost an order of magnitude faster, and eliminated our most glaring scaling issues at the time. Removing this limit however messed with our ability to know when to scale, which then revealed issues with file descriptors, memory, and snapshotting times.
|
||||
The only good advice I have here is to re-evaluate your metrics often, and change them. I guess there’s also a lesson to be learned that improvements can also cause their own uncertainty and that these successes can themselves lead to destabilizations.
|
||||
Because we no longer needed to scale out as aggressively and were free to discover new issues, and one of our best improvements to the system in recent memory is therefore also a contributor to a lot of operational challenges.
|
||||
|
||||
|
||||
|
||||
Things that people think are useful are possibly going to happen even if you forbid them. If you forbid people from logging onto production hosts, and they truly think they'll need it for emergency situations, they'll make sure there's still a way for it to happen, albeit under a different name.
|
||||
On the other hand, things that people think are useless are likely to be done in a minimal way with no enthusiasm, such as lying in your timesheets.
|
||||
This means that writing a procedure means little unless people actually see its value and believe it’s worth following. Conversely, it means that if you can demonstrate the usefulness and make some approaches more usable, they’re likely to get adopted regardless of what is written down as a list of steps or procedures.
|
||||
A related concept here is one here is that if you are tracking things like action items after an incident reviews and they go in the backlog to die, it may not be that your people are failing to follow through; it might also be that it’s impractical to do so, or it’s could also be that these action items were never feeling useful, and the process itself needs to be revisited rather than reinforced.
|
||||
Seeing non-compliance is not necessarily a sign of bad workers. It may rather be a sign of a bad understanding of the workers' challenges, and point to a need to adjust how work is prescribed.
|
||||
Getting a small real buy-in into something voluntary may be better than getting fake buy-in into something you’re forcing people to do. Of course if you manage to write a good procedure that people believe are worth following, more power to you, this is going great.
|
||||
|
||||
|
||||
|
||||
The shortest feedback loop may be attained by giving people the tools to make the right decisions right there and then, and let them do it. Cut the middlemen, including yourself.
|
||||
How do you make that work? We come back to goal alignments and top priorities being harmonized and well understood. If the pressures and goals are understood better, the decisions made also work better.
|
||||
That does mean that you have to listen back about how these things have been going, and that not only do you need to trust your people, but they need to trust you back with critical and unpleasant information as well. The feedback flows both ways, and this hinges on psychological safety.
|
||||
If you've ever talked to a contractor asked to help a big organization, the first thing they'll tell you they do is go talk to the workers with boots on the ground, and ask them what they think needs changing. They'll often have years of potential improvements backlogged, and that they're ready to tell anyone about. Either because management wouldn't listen to it, or because the workers lost trust that voicing that feedback would yield any result.
|
||||
Then the contractor brings it up to management as a neutral party, and suddenly it gets listened to and acted upon.
|
||||
If you've lost that trust, then contractors can play that specific role of workers at the periphery of the organization helping drive change, and they can play a very useful function.
|
||||
But if you have that trust already, maintaining it is crucial because that’s how you get all the good information to help orient and influence things.
|
||||
Trust also means that if you want people to be innovative, you have to allow them to make mistakes. You can’t get it right the first time all the time; if people can’t be allowed to get it wrong here and there, they won’t be allowed to improve and try new things either.
|
||||
|
||||
|
||||
|
||||
Finally, let's look at shifting perspective away from a bare analysis and onto a more systemic point of view. People in specific teams often have a more detailed expert view than you could either have, but if you're standing outside of it, your strength might be to understand how the parts interact in a way that isn't visible to the inside.
|
||||
|
||||
|
||||
|
||||
The most basic point here is that you can’t expect to change the outcome of these small little decisions that accumulate all the time if you never address the pressures within the system that foster them.
|
||||
I used to try and weed my lawn a whole hell of a lot and pull the weeds hours a week until someone explained to me that weeds grew easier in the type of soil I had (poor, dry, unmaintained soil) than grass, and pulling the weeds wasn’t the way to go, I needed to actually make the soil good for the grass to crowd out the weeds.
|
||||
It's similar when considering this whole idea of root cause analysis—of trying to find the one source of the problem and removing it. If your root cause is at the weed’s level, you’ll keep pulling on them forever and will rarely make decent progress. The weeds will keep growing no matter how many roots you remove.
|
||||
If you foster good soil, if you create the right environment that encourages the type of behavior you want instead of the type of behaviour you dislike, you have hopes that the good stuff will crowd out the bad stuff. That’s a roundabout way of talking about culture change. And for these, deep dives based on [richer narratives](https://ferd.ca/notes/paper-accident-report-interpretation.html) and [thematic analysis](https://www.jeli.io/howie/welcome) prove more useful.
|
||||
Also there's a warning here about trying to change the decisions your people make with carrots and sticks—with incentives. They are not going to fundamentally change what pressures the employees negotiate. The pressures stay the same, all you're doing is adding more of them, either in the form of rewards or punishments, which makes decision-making more complex and trickier.
|
||||
Chances are people will keep making the same decisions as they were already, but then they'll report it differently to either get their bonus or to avoid getting penalized for it. Surfacing, understanding, and clarifying goal conflicts can make things easier or shape work to give them more room. Adding carrots and sticks can make things harder.
|
||||
|
||||
|
||||
|
||||
But the tip here is probably: look into what are the behaviors you want to see happen, and give them room to grow.
|
||||
My most successful initiative at Honeycomb is probably creating [weekly discussion sessions about operational stuff and on-call](https://www.honeycomb.io/blog/oncallogy-sessions-best-practices). They range from “how do we operate new service X” into trickier discussions like “is it okay to be visibly angry in an incident”, “how do you deal with shit you don’t know or avoid burnout” or “are there times where code freezes are actually a useful thing?”.
|
||||
Over time we looked into all sorts of weird interactions and the meeting became its own tool.
|
||||
When we noticed incident reviews were difficult to schedule across departments and timezones, we decided that a good wide incident review is good operational talk and started making the optional time slot, which was already on every engineer's calendar (and some other departments too), available for them. It became easier for people to run incident reviews, and over time their size grew from 7-8 people, scoped to 1 or 2 teams, to bigger events with 20 to 40 people in them.
|
||||
We removed a huge but subtle blocker to good feedback loops existing within the organization.
|
||||
These sorts of small changes are those you can drive locally with almost no risk of having them run afoul of organizational priorities, and when you see them work, use the org structure to expand them everywhere.
|
||||
|
||||
|
||||
|
||||
I find it useful to keep focusing on what an indicator triggers as a behavior (the interaction) rather than _only_ what it reports directly. This slide here is 4 error budgets from our SLOs, which combine how successful requests are both in terms of speed and errors, compared to an objective we express in terms of the desired fault rate.
|
||||
When we have to pick targets for our platform, people often ask whether we could pick some key SLOs and turn them as the objective. My answer is almost always "I don't care if we meet the SLOs or not". I mean I care, but not like that.
|
||||
SLOs aren’t hard and fast rules. When the error budget is empty, the main thing that matters to me is that we have a conversation about it, and decide what it is we want to happen from there on. Are we going to hold off on deploys and experiments? Are we able to meet the objectives while on-call, with some schedule corrective work, some major re-architecting? Can we just talk to the customers? Were our targets too ambitious or are we going to eat dirt for a while?
|
||||
Kneejerk automated reactions aren’t nearly as useful as sitting down and having a cross-departmental discussion about what it is we want to do, as an organization, about these signals of unmet expectations. If it fits within on-call duty, like what is probably the case with the error budget on the top left, then fine.
|
||||
But in other cases, such as the top right budget here, which seems to show a gradual decline, owe have to choose whether to do corrective work (and how/when) to meet the SLO—because that wasn't expected and is undesirable—or maybe to relax it—because that's actually a natural consequence of new more expensive features and we need to tweak definitions. Or we could temporarily ignore it because corrective work is already on the way, but not a top priority right now.
|
||||
The two budgets at the bottom come from SLOs that may never page anyone. But from time to time, we re-calibrate them by asking support whether there are any issues users complain about that we aren't already aware of. So long as we're ahead of the complaints, we figure the SLOs are properly defined. But from time to time, we find out that we slipped by getting comments on things our alerting never properly captured. Or maybe we needed to better manage the user's expectations—that's also an option.
|
||||
For any of these choices, we also have to know how this is going to be communicated to users and customers, and having these discussions is the true value of SLOs to me. SLOs that flow outside of engineering teams provide a greater feedback loop about our practices, further upstream, than those that are used exclusively by the teams defining them, regardless of their use for alerting.
|
||||
|
||||
|
||||
|
||||
Finally, this is where SREs can be placed in a great way to shine. You can be away from the central roles, away from the decision-making, on the periphery. By being outside of silos and floating around the organization’s structure, you are allowed to take information from many levels, carry it around, and really tie the loop at the end of so many decisions made in the organization by noting and carrying their impact back once they’ve hit a production system.
|
||||
It is an iterative exercise, our sociotechnical systems are alive, and carrying pertinent signals and amplifying them, you can influence how long it’s gonna take before it all goes to hell anyway.
|
||||
@@ -0,0 +1,43 @@
|
||||
Clipped from: [https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90](https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90)
|
||||
|
||||
## The move from a distributed microservices architecture to a monolith application helped achieve higher scale, resilience, and reduce costs.
|
||||
|
||||
At Prime Video, we offer thousands of live streams to our customers. To ensure that customers seamlessly receive content, Prime Video set up a tool to monitor every stream viewed by customers. This tool allows us to automatically identify perceptual quality issues (for example, block corruption or audio/video sync problems) and trigger a process to fix them.
|
||||
Our Video Quality Analysis (VQA) team at Prime Video already owned a tool for audio/video quality inspection, but we never intended nor designed it to run at high scale (our target was to monitor thousands of concurrent streams and grow that number over time). While onboarding more streams to the service, we noticed that running the infrastructure at a high scale was very expensive. We also noticed scaling bottlenecks that prevented us from monitoring thousands of streams. So, we took a step back and revisited the architecture of the existing service, focusing on the cost and scaling bottlenecks.
|
||||
The initial version of our service consisted of distributed components that were orchestrated by [AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html). The two most expensive operations in terms of cost were the orchestration workflow and when data passed between distributed components. To address this, we moved all components into a single process to keep the data transfer within the process memory, which also simplified the orchestration logic. Because we compiled all the operations into a single process, we could rely on scalable [Amazon Elastic Compute Cloud (Amazon EC2)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) and [Amazon Elastic Container Service (Amazon ECS)](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) instances for the deployment.
|
||||
|
||||
### **Distributed systems overhead**
|
||||
|
||||
Our service consists of three major components. The media converter converts input audio/video streams to frames or decrypted audio buffers that are sent to detectors. Defect detectors execute algorithms that analyze frames and audio buffers in real-time looking for defects (such as video freeze, block corruption, or audio/video synchronization problems) and send real-time notifications whenever a defect is found. For more information about this topic, see our [How Prime Video uses machine learning to ensure video quality](https://www.primevideotech.com/computer-vision/how-prime-video-uses-machine-learning-to-ensure-video-quality) article. The third component provides orchestration that controls the flow in the service.
|
||||
We designed our initial solution as a distributed system using serverless components (for example, AWS Step Functions or [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html)), which was a good choice for building the service quickly. In theory, this would allow us to scale each service component independently. However, the way we used some components caused us to hit a hard scaling limit at around 5% of the expected load. Also, the overall cost of all the building blocks was too high to accept the solution at a large scale.
|
||||
The following diagram shows the serverless architecture of our service.
|
||||
|
||||
|
||||
|
||||
**The initial architecture of our defect detection system.**
|
||||
The main scaling bottleneck in the architecture was the orchestration management that was implemented using AWS Step Functions. Our service performed multiple state transitions for every second of the stream, so we quickly reached account limits. Besides that, AWS Step Functions charges users per state transition.
|
||||
The second cost problem we discovered was about the way we were passing video frames (images) around different components. To reduce computationally expensive video conversion jobs, we built a microservice that splits videos into frames and temporarily uploads images to an [Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) bucket. Defect detectors (where each of them also runs as a separate microservice) then download images and processed it concurrently using AWS Lambda. However, the high number of Tier-1 calls to the S3 bucket was expensive.
|
||||
|
||||
### **From distributed microservices to a monolith application**
|
||||
|
||||
To address the bottlenecks, we initially considered fixing problems separately to reduce cost and increase scaling capabilities. We experimented and took a bold decision: we decided to rearchitect our infrastructure.
|
||||
We realized that distributed approach wasn’t bringing a lot of benefits in our specific use case, so we packed all of the components into a single process. This eliminated the need for the S3 bucket as the intermediate storage for video frames because our data transfer now happened in the memory. We also implemented orchestration that controls components within a single instance.
|
||||
The following diagram shows the architecture of the system after migrating to the monolith.
|
||||
|
||||
|
||||
|
||||
**The updated architecture for monitoring a system with all components running inside a single Amazon ECS task.**
|
||||
Conceptually, the high-level architecture remained the same. We still have exactly the same components as we had in the initial design (media conversion, detectors, or orchestration). This allowed us to reuse a lot of code and quickly migrate to a new architecture.
|
||||
In the initial design, we could scale several detectors horizontally, as each of them ran as a separate microservice (so adding a new detector required creating a new microservice and plug it in to the orchestration). However, in our new approach the number of detectors only scale vertically because they all run within the same instance. Our team regularly adds more detectors to the service and we already exceeded the capacity of a single instance. To overcome this problem, we cloned the service multiple times, parametrizing each copy with a different subset of detectors. We also implemented a lightweight orchestration layer to distribute customer requests.
|
||||
The following diagram shows our solution for deploying detectors when the capacity of a single instance is exceeded.
|
||||
|
||||
|
||||
|
||||
**Our approach for deploying more detectors to the service.**
|
||||
|
||||
### **Results and takeaways**
|
||||
|
||||
Microservices and serverless components are tools that do work at high scale, but whether to use them over monolith has to be made on a case-by-case basis.
|
||||
Moving our service to a monolith reduced our infrastructure cost by over 90%. It also increased our scaling capabilities. Today, we’re able to handle thousands of streams and we still have capacity to scale the service even further. Moving the solution to Amazon EC2 and Amazon ECS also allowed us to use the [Amazon EC2 compute saving plans](https://aws.amazon.com/savingsplans/compute-pricing/) that will help drive costs down even further.
|
||||
Some decisions we’ve taken are not obvious but they resulted in significant improvements. For example, we replicated a computationally expensive media conversion process and placed it closer to the detectors. Whereas running media conversion once and caching its outcome might be considered to be a cheaper option, we found this not be a cost-effective approach.
|
||||
The changes we’ve made allow Prime Video to monitor all streams viewed by our customers and not just the ones with the highest number of viewers. This approach results in even higher quality and an even better customer experience.
|
||||
@@ -0,0 +1,21 @@
|
||||
Clipped from: [https://blog.traillifeusa.com/boy-finds-hope?utm_campaign=Raising%20Godly%20Boys&utm_medium=email&_hsmi=220415115&_hsenc=p2ANqtz-9wOKKcP8ywC8lxPGr9wAs3lzk6NsuStGaUd8Jw9bL9FQtvKma9aP_6LMzlBiba-jFHNF60TmvQ-95DT6tkKo_m8gvHig&utm_content=220415115&utm_source=hs_automation](https://blog.traillifeusa.com/boy-finds-hope?utm_campaign=Raising%20Godly%20Boys&utm_medium=email&_hsmi=220415115&_hsenc=p2ANqtz-9wOKKcP8ywC8lxPGr9wAs3lzk6NsuStGaUd8Jw9bL9FQtvKma9aP_6LMzlBiba-jFHNF60TmvQ-95DT6tkKo_m8gvHig&utm_content=220415115&utm_source=hs_automation)
|
||||
|
||||
|
||||
|
||||
**Trail Life mentors inspire young boy not only to find the will to live, but to help others around him**
|
||||
|
||||
In a society where the lines between masculine and feminine are constantly blurred, brash boyish bluster and boisterousness is losing its place. Feeling unappreciated, too boys are losing their identity. Traditional needs like physical challenge, competition, risk-taking, action, and adventure are being discounted and boys seeking to fill these needs are punished, diagnosed, or written off as unruly, difficult, or perhaps toxic.
|
||||
Trail Life USA, the largest Christ-centered, boy-focused scout-type organization in the country, is familiar with this struggle for boys and can provide a solution. In the fight to provide boys with unique programming that celebrates boyhood, Trail Life utilizes outdoor adventure and personal relationships to speak to the heart of a boy and and to guide him in his walk with Christ.
|
||||
Mark Hancock, Trail Life CEO, commented, _“When boys feel like they are relegated as ‘less than’ due to unappreciated gender differences, they begin to wonder where they belong in a society that seems to discount their abilities. Trail Life USA provides a boy-focused program and activities designed to let boys be boys, accentuating their strengths and allowing them to feel understood and appreciated.”_
|
||||
_Hancock continued, “It seems everywhere a boy goes, he’s expected to comply with unrealistic social norms. The consistent message he gets is that he needs to ‘sit still, be quiet, do what he is told, and behave like the girls.’ But boys are not defective girls. They are created differently on purpose for a purpose. Properly channeled and intentionally challenged, the exuberance, drive, and daring of healthy boys is exactly what our society needs.”_
|
||||
|
||||
[](https://blog.traillifeusa.com/cs/c/?cta_guid=ae0cca4b-12ee-44b5-9522-ea34bcd24e9e&signature=AAH58kHdz_P7HD13orXjY97P9tTqTGFjMw&pageId=77377023317&placement_guid=fe90369d-b6b8-4f68-9fc0-b6d63f14044c&click=19d2745d-b484-4784-8cde-ecde604c2e2a&hsutk=&canon=https%3A%2F%2Fblog.traillifeusa.com%2Fboy-finds-hope&portal_id=6459804&redirect_url=APefjpGnipquSbf2uJG8qs097FtoxGreH-0PhQnuOewllWKAO8ER1syjqeWtpneVenoGaHglMjK-_DfDkJUb2mKP7Sb3d5-H0WCHgYDmgDyzI-ZHx59iF80L-97IR8vDn99tCMt7ezx2Rj6p4CLK-nYdn7aZHiWVcw)
|
||||
|
||||
In a system that fails to acknowledge that boys are not just like girls, [boys are increasingly diagnosed with disorders](https://www.understood.org/articles/en/do-boys-have-learning-and-thinking-differences-more-often-than-girls) and are [falling behind their female counterparts](https://www.brookings.edu/blog/up-front/2021/01/12/the-unreported-gender-gap-in-high-school-graduation-rates/) in nearly every academic category. This trend continues into [college where 60% of students are female](https://www.usatoday.com/story/opinion/2021/10/09/boys-falling-behind-how-schools-must-change-help-young-males/5913463001/). Even more tragic than the academic difficulty is the impact on the mental health of boys and men. Today, men account for [four out of five suicides in America](https://www.cdc.gov/nchs/products/databriefs/db373.htm#:~:text=In%2520both%2520urban%2520and%2520rural%2520areas%252C%2520suicide%2520rates%2520for%2520males,(30.7%2520compared%2520with%25208.0).) and twice the [drug-related deaths](https://news.wttw.com/2021/10/25/us-overdose-deaths-surge-all-time-high) as compared to women. The most rapidly growing suicide rate demographic is boys from the [ages of ten to 14](https://www.bloomberg.com/news/articles/2021-11-03/u-s-suicides-fall-for-second-year-in-a-row-during-pandemic). One mother recently wrote to share how her son was almost one of these statistics:
|
||||
_“Dealing with ADHD, depression, and anxiety, my son was struggling at school and at home. Daily outbursts, disciplinary problems, and panic attacks forced us as parents to make the hard decision to pull him out of traditional school. A constant cycle of being in trouble with teachers, church leaders, and parents left him questioning his own value. At ten years of age, he decided that he was so broken and hopeless that the world would be better off without him. Last October, he attempted to take his own life and was admitted to the emergency room at the children’s hospital because of injuries sustained in that attempt_.
|
||||
_“Then my son began attending_ _Trail Life__. It was the first time that he felt understood and accepted by authority figures (the_ **Trail Life** _leaders). In a world that had always attempted to squash his character, he has been encouraged to see a purpose in his ‘boy-ness.’ He has heard consistently that God created him the way he is and that he is loved. He has been encouraged that the future holds a purpose for him — one that will utilize his courage, his sensitivity, and his passion. He has been inspired intellectually and spiritually. Most importantly, he feels part of a community where he respects and admires the leaders, and has friends.”_
|
||||
In the active learning environment at **Trail Life**, her son has been able to shine. In the past six months, he has been hiking and camping, learned first aid, become proficient in lighting a fire without a match, and built rockets with his Troop.
|
||||
The mother commented on these skills her son learned at Trail Life, stating, _“Because my son learned first aid with his Troop, I still have both of my children. Last Sunday, I was driving on the freeway when my daughter, who is two years old, choked on a snack. She couldn’t breathe. I was stuck driving in traffic in the HOV lane, where I couldn’t pull over. My son was able to use the skills he learned at Trail Life_ _to give his sister the Heimlich maneuver, and she coughed up food and was able to breathe. He saved her life and displayed the level-headed, calm, and confident skills he needed to save his sister. I am so proud of him, and so thankful for Trail Life."_
|
||||
_“Even his father is seeing a tremendous difference. Before my son’s attempted suicide, they had become so estranged they were barely able to talk. When he was presented with the Life Saving Award from Trail Life,_ _his dad was able to attend and publicly commend his son. The young man who had been labeled as troublesome, difficult, and delinquent by so many is now being commended, affirmed, accepted, and encouraged by his Troop, his community, and his father.”_
|
||||
_“Many sons never hear words of affirmation, acceptance, and encouragement from their father. I want to thank the men of the Troop who have poured themselves into my son and our family. Because of your intervention and effort, a relationship that was broken has been dramatically healed and turned around.”_
|
||||
The mother concluded, _“Trail Life is m__aking a real difference in the lives of boys and their fathers.”_
|
||||
13
Reading/Done Reading/What is a Boy.md
Normal file
13
Reading/Done Reading/What is a Boy.md
Normal file
@@ -0,0 +1,13 @@
|
||||
Clipped from: [https://blog.traillifeusa.com/what-is-a-boy?utm_campaign=Raising%20Godly%20Boys&utm_medium=email&_hsmi=219419808&_hsenc=p2ANqtz-9tkiaPng4HKJvYRy6Ee8wBSX_GRpZ-nKWnj4tZJxJ13W0R00aCDx3IevUfMoI2iUWmB4GMzcGgFwHTSSQIWosWfF_FPw&utm_content=219419808&utm_source=hs_automation](https://blog.traillifeusa.com/what-is-a-boy?utm_campaign=Raising%20Godly%20Boys&utm_medium=email&_hsmi=219419808&_hsenc=p2ANqtz-9tkiaPng4HKJvYRy6Ee8wBSX_GRpZ-nKWnj4tZJxJ13W0R00aCDx3IevUfMoI2iUWmB4GMzcGgFwHTSSQIWosWfF_FPw&utm_content=219419808&utm_source=hs_automation)
|
||||
|
||||
|
||||
|
||||
Between the innocence of babyhood and the dignity of manhood we find a delightful creature called a boy. Boys come in assorted sizes, weights, and colors, but all boys have the same creed: to enjoy every second of every minute of every hour of every day and to protest with noise (their only weapon) when their last minute is finished and the adult males pack them off to bed at night.
|
||||
Boys are found everywhere—on top of, underneath, inside of, climbing on, swinging from, running around, or jumping to.
|
||||
Mothers love them, little girls hate them, older sisters and brothers tolerate them, adults ignore them, and Heaven protects them.
|
||||
A boy is truth with dirt on its face, beauty with a cut on its finger, wisdom with bubble gum in its hair, and the hope of the future with a frog in its pocket. When you are busy, a boy is an inconsiderate, bothersome, intruding jangle of noise. When you want him to make a good impression, his brain turns to jelly or else he becomes a savage, sadistic, jungle creature bent on destroying the world and himself with it.
|
||||
A boy is a composite—he has the appetite of a horse, the digestion of a sword-swallower, the energy of a pocket-sized atomic bomb, the curiosity of a cat, the lungs of a dictator, the imagination of a Paul Bunyan, the shyness of a violet, the audacity of a steel trap, the enthusiasm of a firecracker, and when he makes something, he has five thumbs on each hand. He likes ice cream, knives, saws, Christmas, comic books, the boy across the street, woods, water (in its natural habitat), large animals, Dad, trains, Saturday mornings, and fire engines.
|
||||
He is not much for Sunday School, company, schools, books without pictures, music lessons, neckties, barbers, girls, overcoats, adults, or bedtime. Nobody else is so early to rise, or so late to supper. Nobody else gets so much fun out of trees, dogs, and breezes. Nobody else can cram into one pocket a rusty knife, a half-eaten apple, three feet of string, an empty Bull Durham sack, two gum drops, six cents, a slingshot, a chunk of unknown substance, and a genuine supersonic code ring with a secret compartment.
|
||||
A boy is a magical creature—you can lock him out of your workshop, but you can’t lock him out of your heart. You can get him out of your study, but you can’t get him out of your mind. Might as well give up—he is your captor, your jailer, your boss, and your master—a freckled-faced, pint-sized, cat-chasing, bundle of noise. But when you come home at night with only shattered pieces of your hopes and dreams, he can mend them like new with two magic words, "Hi Dad!"
|
||||
---
|
||||
[Trail Life USA](http://www.traillifeusa.com/) is designed uniquely for boys. Established on timeless values and set in the context of outdoor adventure, boys from Kindergarten through 12th grade are engaged in a Troop setting by male mentors where they are challenged to grow in character, understand their purpose, serve their community, and develop practical leadership skills to carry out the mission for which they were created.
|
||||
Reference in New Issue
Block a user