- Data management (e.g., choice of DB, caching strategy, reuse primary database or copy all the data in another system optimized for recommendation workload, etc.) [[1]](https://fennel.ai/blog/real-world-recommendation-system/#fn1)
- Data management (e.g., choice of DB, caching strategy, reuse primary database or copy all the data in another system optimized for recommendation workload, etc.)
- Feature management (e.g., offline vs online, precompute vs serve live)
- Serving systems (performance, query latency, distribution model, fault tolerance, etc.)
- Deployment system (e.g., how does new code get updated, build steps, keeping caches working after processes restart, etc.)
@@ -41,7 +41,7 @@ How does Retrieval work? Retrieval is done by writing a few heuristics, also cal
- Find 5 contents that user “liked” in the past, and for each such content, find 5 more “related” items
- Find the most relevant topics for a user and find the freshest content from each of the topics.
Retrieval can be powered by ML (e.g., trained embeddings), but more often than not, a larger % of generators are mere heuristics that encode some “product thinking” about what content is likely to create a good recommendation experience. And by writing a few of these and taking a union of all their candidates, we ensure that the system is able to at least consider all sorts of interesting inventory. Retrieval has only two jobs — 1) get all the interesting things (or at least as many as possible) [[2]](https://fennel.ai/blog/real-world-recommendation-system/#fn2) and 2) get as few total things as possible so that we can score/examine each candidate using the power of ML.
Retrieval can be powered by ML (e.g., trained embeddings), but more often than not, a larger % of generators are mere heuristics that encode some “product thinking” about what content is likely to create a good recommendation experience. And by writing a few of these and taking a union of all their candidates, we ensure that the system is able to at least consider all sorts of interesting inventory. Retrieval has only two jobs — 1) get all the interesting things (or at least as many as possible) and 2) get as few total things as possible so that we can score/examine each candidate using the power of ML.
## 2. Filtering
@@ -50,7 +50,8 @@ After retrieving a few hundred candidates, recommendation systems typically filt
1. Some products try to filter out content that the user has already seen before
2. Some products expose some controls to the users to hide away topics or authors or other sources of content
In short, most real-world recommendation systems develop a long list of filters over time, which once again encode some product thinking about what creates a good experience. [[[3]](https://fennel.ai/blog/real-world-recommendation-system/#fn3)
In short, most real-world recommendation systems develop a long list of filters over time, which once again encode some product thinking about what creates a good experience.
Filtering and retrieval have a very interesting relationship. Some filters are pushed down to the generators themselves — for instance, if you’re building a dating product, filters for location and sexual preferences may be a part of each generator itself. But more often than not, it is physically impossible to have each generator respect each filter at the source, and so a whole layer of filtering is needed.
## 3. Feature Extraction
@@ -65,7 +66,7 @@ So far, we have narrowed down the full inventory to a few hundred candidates and
There are two key ideas that are very successful and present in the scoring of most real-world recommendation systems (and we’d write dedicated posts about both in the future — stay tuned):
1. Multi-stage scoring — not all ML models are equal, and some are lot “heavier” than others. And it is usually not possible to run the heaviest ML models on hundreds of candidates. So instead, scoring itself is broken down in two substages — 1st stage scoring (which uses a relatively lighter ML model like GBDTs on all 500 candidates and emits out, say, top 100 candidates) and the 2nd stage scoring, which runs the heavy model (say deep neural network) on just the top 100 candidates.
2. Combining many models — ML models can only learn whatever we teach them to learn. And typically, they are taught to predict the probability of user engaging in a single action, say like. Sorting all content by what gets clicked is a good start but has lots of issues — for instance, it might only distribute clickbaity content. To make the recommendations more balanced, usually, multiple models are trained - say, one for predicting clicks, one for predicting comments, one for user reporting the content, etc. And the final score of a candidate is a weighted average of all these models. While this makes the recommendations better, this can also increase the amount of computation that needs to be done. [[4]](https://fennel.ai/blog/real-world-recommendation-system/#fn4)
2. Combining many models — ML models can only learn whatever we teach them to learn. And typically, they are taught to predict the probability of user engaging in a single action, say like. Sorting all content by what gets clicked is a good start but has lots of issues — for instance, it might only distribute clickbaity content. To make the recommendations more balanced, usually, multiple models are trained - say, one for predicting clicks, one for predicting comments, one for user reporting the content, etc. And the final score of a candidate is a weighted average of all these models. While this makes the recommendations better, this can also increase the amount of computation that needs to be done.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
