Files
ObsidianJournal/Work/Training/Advanced Distributed Systems/Advanced Distributed Systems Design - Module 11 - Scalability and Flexibility, Monitoring and Management.md

28 lines
2.4 KiB
Markdown
Raw Permalink Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
**Naming**
Naming is challenging.  When you have web1, web2 and web3 and db1, then everything's pretty easy to keep track of.  When you get to bigger systems, naming becomes more important. Usually naming follows the service name.  When you pull out an Autonomous Component, naming is even more important.  Commonly, you use a dot notation of Service.BusinessComponent.AutonomousComponent.  But how do you name the AC?  Using the main message that is processes is helpful.   It is also helpful to prepend the name of the company too.  So, the name becomes Company.Service.BC.AC. **Monitoring Queue-Based Systems**
Error queue notifications go to admin to signal problems. Identify bottlenecks: well-named queues tell you where the bottlenecks are.  Important are:
- number of messages
- throughput
- These two numbers above give you the wait time of each message. NServiceBus calls this the CriticalTime of and endpoint.
- The context of the critical time is essential.  If we're generating PDFs, then 2 seconds is awesome.  If you're running a high-frequency trading system, then 2 second wait times will run you out of business.
**Scalability**
Traditional competing consumer:
- one queue
- more than one AC feeding off of queue
You don't scale out an entire system, but individual ACs.  You know to scale out when you see the AC continually violating or close to violating SLA. **Virtualization - Part 1**
Almost all production environments are virtualized.  We can connect our monitoring environment and our scaling processes.  We can connect to our SLA--as it's reaching its limit, we can scale out an AC. **Fault Tolerance - high level**
- Any number of active backups
- Automatic load balancing
- By hosting in a virtual environment, our data (database, queue storage, etc) is all stored on a SAN.  The SAN has snapshots and is backed up.  We have a high degree of fault tolerance **Versioning**
Make updates backwards compatible.
Test them.
Start at the back of the system and gradually move forward.  Update the database, then the server, then the messages, then the client. You can script rollouts.
- CI server builds new version
- CI installs v2 next to v1
- CI checks for error message in the queue
- CI finds some and rolls back v2 and notifies developer
This gives developers very safe exposure to the prod environment for their new code.