Scalability Fundamentals

What is Scalability?

Scalability is a system’s ability to handle increased load by adding resources. A scalable system can grow to accommodate more users, more data, or more transactions without degrading performance.

Key Insight

A system is scalable if performance remains acceptable as load increases. The goal isn’t just to handle more load—it’s to handle more load efficiently.

---

Two Approaches to Scaling

Vertical Scaling (Scale Up)

Add more power to existing machines - bigger CPU, more RAM, faster disks.

Pros:

• ✅ Simple - no code changes needed
• ✅ No distributed system complexity
• ✅ Strong consistency is easy

Cons:

• ❌ Hardware limits (can’t add infinite CPU)
• ❌ Expensive at high end
• ❌ Single point of failure
• ❌ Downtime during upgrades

Horizontal Scaling (Scale Out)

Add more machines - distribute the load across multiple servers.

Pros:

• ✅ No hardware limits (add infinite machines)
• ✅ Cost-effective (use commodity hardware)
• ✅ Built-in redundancy
• ✅ Gradual scaling

Cons:

• ❌ Distributed system complexity
• ❌ Data consistency challenges
• ❌ Code must be designed for it
• ❌ Network overhead

---

Comparison: Vertical vs Horizontal

Aspect	Vertical Scaling	Horizontal Scaling
Approach	Bigger machine	More machines
Limit	Hardware ceiling	Theoretically unlimited
Cost	Expensive at scale	Cost-effective
Complexity	Simple	Complex
Downtime	Required for upgrades	Zero-downtime possible
Failure	Single point of failure	Redundancy built-in
Code changes	Usually none	May require redesign

Industry Trend

Most large-scale systems use horizontal scaling. Companies like Google, Amazon, and Netflix run on thousands of commodity servers rather than a few supercomputers.

---

What Makes Code Horizontally Scalable?

This is where LLD meets HLD. Your class design determines whether your system can scale horizontally.

The Problem: Stateful Services

Python

stateful_service.py

class ShoppingCartService:
    """❌ NOT horizontally scalable - stores state in memory"""

    def __init__(self):
        self.carts = {}  # user_id -> cart items

    def add_item(self, user_id: str, item: str):
        if user_id not in self.carts:
            self.carts[user_id] = []
        self.carts[user_id].append(item)

    def get_cart(self, user_id: str) -> list:
        return self.carts.get(user_id, [])

# Problem: If user's next request goes to a different server,
# their cart is empty!

Java

StatefulService.java

import java.util.*;

public class ShoppingCartService {
    // ❌ NOT horizontally scalable - stores state in memory
    private Map<String, List<String>> carts = new HashMap<>();

    public void addItem(String userId, String item) {
        carts.computeIfAbsent(userId, k -> new ArrayList<>()).add(item);
    }

    public List<String> getCart(String userId) {
        return carts.getOrDefault(userId, Collections.emptyList());
    }
}

// Problem: If user's next request goes to a different server,
// their cart is empty!

The Solution: Stateless Services

The solution is to externalize all state to a shared store (Redis, database, etc.):

• No instance variables holding user/session data
• All state lives externally in Redis, database, or similar
• Any server can handle any request because they all access the same shared state

Python

stateless_service.py

class ShoppingCartService:
    """✅ Horizontally scalable - no local state"""

    def __init__(self, redis_client):
        self.redis = redis_client  # External storage

    def add_item(self, user_id: str, item: str) -> None:
        self.redis.rpush(f"cart:{user_id}", item)

    def get_cart(self, user_id: str) -> list:
        return self.redis.lrange(f"cart:{user_id}", 0, -1)

# Any server can handle any request!

Java

StatelessService.java

public class ShoppingCartService {
    private final Jedis redis;  // External storage

    public ShoppingCartService(Jedis redis) { this.redis = redis; }

    public void addItem(String userId, String item) {
        redis.rpush("cart:" + userId, item);
    }

    public List<String> getCart(String userId) {
        return redis.lrange("cart:" + userId, 0, -1);
    }
}
// Any server can handle any request!

---

Scalability Dimensions

Systems can be scaled along different dimensions:

1. Load Scalability

Handle more requests per second

2. Data Scalability

Handle more data

3. Geographic Scalability

Serve users globally with low latency

---

Design Patterns for Scalability

Pattern 1: Stateless Services

Key Principle: All state lives externally (database, cache, message queue). The service itself stores nothing between requests.

This allows you to run any number of service instances and route requests to any of them.

Pattern 2: Idempotent Operations

Key Principle: Operations that can be safely retried without side effects. Critical for distributed systems where network failures cause retries.

Implementation: Store results keyed by a unique idempotency key. Before processing, check if the key exists and return the cached result.

Pattern 3: Async Processing

Key Principle: Move slow, non-critical operations out of the request path using message queues.

Result: Users get fast responses. Slow operations (email, analytics, notifications) happen in the background without blocking.

---

Scalability Checklist for LLD

When designing classes, ask yourself:

Question	Why It Matters
Does this class store state in instance variables?	Prevents horizontal scaling
Can multiple instances run simultaneously?	Required for scaling out
Are operations idempotent?	Enables safe retries
What happens if this operation is slow?	May need async processing
Does this depend on local resources (files, memory)?	Won’t work across servers
How does this handle concurrent requests?	Thread safety concerns

---

Key Takeaways

Remember

• Scalability = handling growth efficiently
• Vertical scaling = bigger machines (simple but limited)
• Horizontal scaling = more machines (complex but unlimited)
• Stateless services are key to horizontal scaling
• External state (Redis, databases) enables shared state
• Idempotent operations enable safe retries
• Async processing offloads non-critical work

---

What’s Next?

Understanding scalability is just the beginning. Next, we’ll dive into measuring system performance:

Next up: Latency and Throughput - Learn the key metrics that define system performance.

Design Exercise

Take one of your existing classes and evaluate: Is it horizontally scalable? What changes would make it scalable?