⚖️ Load Balancing: The Traffic Director

Intermediate

A load balancer sits in front of a pool of servers and distributes incoming requests so that no single server becomes a bottleneck. It is the component that makes horizontal scaling actually work.

Why Load Balancing Exists

Without a load balancer, every client must know which server to talk to. That breaks the moment you add or remove a server and forces clients to deal with server failures directly.

Benefits at a glance:

Comparing

Without Load Balancer

Single server handles everything

With Load Balancer

Traffic distributed across a pool

Load distribution
One server gets crushed
Traffic is spread evenly
Failure handling
A dead server kills users
LB detects failure and skips it
Scaling
Adding servers is painful
LB pool grows transparently
Single point of failure
Yes — the lone server
Active-passive LB pairs remove that risk

Layer 4 vs Layer 7 Load Balancing

Load balancers operate at different layers of the network stack. This is the most important architectural decision.

Comparing

Layer 4 (Transport)

Fast, simple, network-level

Layer 7 (Application)

Smart, flexible, content-aware

Routing basis
IP + Port only
URL, headers, cookies, body
Speed
Faster — minimal inspection
Slightly slower — full request parse
SSL termination
No — passes encrypted traffic through
Yes — decrypts and re-encrypts
Content-based rules
No — every request to the same backend gets the same treatment
Yes — route /api/* to one pool, /static/* to another
Use case
Raw TCP, databases, UDP gaming
HTTP APIs, microservices, A/B testing

Load Balancing Algorithms

1. Round Robin

Requests are handed out in sequence — first to Server 1, next to Server 2, and so on, cycling back.

javascript

class RoundRobinBalancer {
  constructor(servers) {
    this.servers = servers;
    this.index = 0;
  }

  next() {
    const server = this.servers[this.index];
    this.index = (this.index + 1) % this.servers.length;
    return server;
  }
}

const lb = new RoundRobinBalancer(["s1:3001", "s2:3002", "s3:3003"]);

// Simulate 6 requests
for (let i = 1; i <= 6; i++) {
  console.log(`Request ${i} → ${lb.next()}`);
}
// Request 1 → s1:3001
// Request 2 → s2:3002
// Request 3 → s3:3003
// Request 4 → s1:3001  ← cycle restarts
// Request 5 → s2:3002
// Request 6 → s3:3003

Best for: Homogeneous servers with similar processing times.

2. Weighted Round Robin

Servers with more capacity get proportionally more requests.

javascript

class WeightedRoundRobinBalancer {
  constructor(servers) {
    // servers = [{ host, weight }]
    this.pool = servers.flatMap((s) => Array(s.weight).fill(s.host));
    this.index = 0;
  }

  next() {
    const server = this.pool[this.index];
    this.index = (this.index + 1) % this.pool.length;
    return server;
  }
}

const lb = new WeightedRoundRobinBalancer([
  { host: "s1:3001", weight: 3 }, // gets 3/6 = 50% of traffic
  { host: "s2:3002", weight: 2 }, // gets 2/6 = 33%
  { host: "s3:3003", weight: 1 }, // gets 1/6 = 17%
]);

// Pool becomes: [s1, s1, s1, s2, s2, s3]
for (let i = 1; i <= 6; i++) {
  console.log(`Request ${i} → ${lb.next()}`);
}

3. Least Connections

Route each new request to the server currently handling the fewest active connections. Excellent for long-lived connections (WebSockets, file uploads).

javascript

class LeastConnectionsBalancer {
  constructor(hosts) {
    this.servers = hosts.map((host) => ({ host, connections: 0 }));
  }

  acquire() {
    const server = this.servers.reduce((min, s) =>
      s.connections < min.connections ? s : min
    );
    server.connections++;
    return server;
  }

  release(server) {
    server.connections = Math.max(0, server.connections - 1);
  }
}

const lb = new LeastConnectionsBalancer(["s1:3001", "s2:3002", "s3:3003"]);

async function handleRequest(requestId) {
  const server = lb.acquire();
  console.log(
    `Request ${requestId} → ${server.host} (${server.connections} conns)`
  );

  // Simulate work (variable duration)
  await new Promise((r) => setTimeout(r, Math.random() * 200));

  lb.release(server);
  console.log(`Request ${requestId} done — ${server.host} freed`);
}

// Fire 5 concurrent requests
Promise.all([1, 2, 3, 4, 5].map(handleRequest));

4. IP Hash (Sticky Routing)

A hash of the client's IP address deterministically selects the server. The same client always lands on the same server — critical for session data not stored externally.

javascript

function ipToInt(ip) {
  return (
    ip.split(".").reduce((acc, oct) => (acc << 8) + parseInt(oct), 0) >>> 0
  );
}

class IpHashBalancer {
  constructor(servers) {
    this.servers = servers;
  }

  next(clientIp) {
    const hash = ipToInt(clientIp);
    return this.servers[hash % this.servers.length];
  }
}

const lb = new IpHashBalancer(["s1:3001", "s2:3002", "s3:3003"]);

console.log(lb.next("192.168.1.10")); // Always same server for this IP
console.log(lb.next("192.168.1.20")); // Always same server for this IP
console.log(lb.next("192.168.1.10")); // Same as first call ✅

⚠️ Downside: Adding or removing a server re-maps most clients. Use Consistent Hashing to fix this.

5. Consistent Hashing

Maps both servers and requests onto a virtual ring. Adding or removing a server only remaps a small fraction of keys — the rest stay put.

javascript

const crypto = require("crypto");

class ConsistentHashRing {
  constructor(servers, virtualNodes = 150) {
    this.ring = new Map();
    this.sortedKeys = [];

    for (const server of servers) {
      for (let v = 0; v < virtualNodes; v++) {
        const key = this._hash(`${server}:${v}`);
        this.ring.set(key, server);
        this.sortedKeys.push(key);
      }
    }
    this.sortedKeys.sort((a, b) => a - b);
  }

  _hash(str) {
    const hex = crypto.createHash("md5").update(str).digest("hex");
    return parseInt(hex.slice(0, 8), 16);
  }

  getServer(requestKey) {
    const hash = this._hash(requestKey);
    // Walk the ring clockwise to find the first server >= hash
    for (const ringKey of this.sortedKeys) {
      if (hash <= ringKey) return this.ring.get(ringKey);
    }
    // Wrap around to first server
    return this.ring.get(this.sortedKeys[0]);
  }

  addServer(server, virtualNodes = 150) {
    for (let v = 0; v < virtualNodes; v++) {
      const key = this._hash(`${server}:${v}`);
      this.ring.set(key, server);
      this.sortedKeys.push(key);
    }
    this.sortedKeys.sort((a, b) => a - b);
  }

  removeServer(server, virtualNodes = 150) {
    for (let v = 0; v < virtualNodes; v++) {
      const key = this._hash(`${server}:${v}`);
      this.ring.delete(key);
    }
    this.sortedKeys = this.sortedKeys.filter((k) => this.ring.has(k));
  }
}

const ring = new ConsistentHashRing(["s1:3001", "s2:3002", "s3:3003"]);

// Same request key always routes to same server
console.log(ring.getServer("user:42")); // e.g. s2:3002
console.log(ring.getServer("user:42")); // s2:3002 — consistent ✅

// Add a server — only ~25% of keys remapped
ring.addServer("s4:3004");
console.log(ring.getServer("user:42")); // may stay s2:3002 or move to s4:3004

📐 The Math of Consistent Hashing

Why does adding one server only remap ~25% of keys instead of re-shuffling everything? The math is straightforward — each server owns a contiguous arc of the ring, and adding a new server carves out a piece of one existing arc.

Remap ratio when adding one server to a ring of N:

Radd = 1N+1

Virtual nodes per server — controls variance:

σ2 = 1V

The takeaway: consistent hashing turns a N → O(N) remap problem into an O(1/N) expected remap, and adding virtual nodes flattens the distribution.

Health Checks

A load balancer is only useful if it stops sending traffic to dead servers.

javascript

class HealthAwareBalancer {
  constructor(servers, checkIntervalMs = 5000) {
    this.all = servers.map((host) => ({ host, healthy: true, failures: 0 }));
    this._startHealthChecks(checkIntervalMs);
  }

  get healthy() {
    return this.all.filter((s) => s.healthy);
  }

  next() {
    const pool = this.healthy;
    if (!pool.length) throw new Error("No healthy servers available");
    // Round-robin over healthy servers only
    this._rr = ((this._rr ?? -1) + 1) % pool.length;
    return pool[this._rr].host;
  }

  _startHealthChecks(intervalMs) {
    setInterval(async () => {
      await Promise.all(this.all.map((s) => this._check(s)));
    }, intervalMs);
  }

  async _check(server) {
    try {
      const res = await fetch(`http://${server.host}/health`, {
        signal: AbortSignal.timeout(2000),
      });
      if (res.ok) {
        server.failures = 0;
        server.healthy = true;
      } else {
        this._handleFailure(server);
      }
    } catch {
      this._handleFailure(server);
    }
  }

  _handleFailure(server) {
    server.failures++;
    if (server.failures >= 3) {
      server.healthy = false;
      console.warn(`[LB] ${server.host} marked unhealthy after 3 failures`);
    }
  }
}

Complete: Minimal HTTP Load Balancer in Node.js

This is a working Layer 7 proxy that combines round-robin, health checks, and retry logic in ~80 lines.

javascript

const http = require("http");
const { request: proxyReq } = require("http");

const SERVERS = [
  { host: "localhost", port: 3001, healthy: true, failures: 0 },
  { host: "localhost", port: 3002, healthy: true, failures: 0 },
  { host: "localhost", port: 3003, healthy: true, failures: 0 },
];

let rrIndex = 0;

function pickServer() {
  const pool = SERVERS.filter((s) => s.healthy);
  if (!pool.length) return null;
  const server = pool[rrIndex % pool.length];
  rrIndex++;
  return server;
}

function forward(req, res, server) {
  return new Promise((resolve, reject) => {
    const options = {
      hostname: server.host,
      port: server.port,
      path: req.url,
      method: req.method,
      headers: {
        ...req.headers,
        "x-forwarded-for": req.socket.remoteAddress,
        host: `${server.host}:${server.port}`,
      },
    };

    const proxy = proxyReq(options, (upstream) => {
      res.writeHead(upstream.statusCode, upstream.headers);
      upstream.pipe(res);
      resolve();
    });

    proxy.on("error", reject);
    req.pipe(proxy);
  });
}

// Main proxy server
const lb = http.createServer(async (req, res) => {
  const server = pickServer();

  if (!server) {
    res.writeHead(503);
    return res.end("No healthy servers");
  }

  try {
    await forward(req, res, server);
    server.failures = 0;
  } catch (err) {
    server.failures++;
    if (server.failures >= 3) {
      server.healthy = false;
      console.error(`[LB] ${server.host}:${server.port} removed from pool`);
    }
    res.writeHead(502);
    res.end("Bad Gateway");
  }
});

// Health check loop
setInterval(async () => {
  for (const server of SERVERS) {
    try {
      await new Promise((resolve, reject) => {
        const req = proxyReq(
          {
            hostname: server.host,
            port: server.port,
            path: "/health",
            method: "GET",
          },
          (r) => (r.statusCode === 200 ? resolve() : reject())
        );
        req.on("error", reject);
        req.setTimeout(2000, () => {
          req.destroy();
          reject();
        });
        req.end();
      });
      if (!server.healthy) {
        server.healthy = true;
        server.failures = 0;
        console.log(`[LB] ${server.host}:${server.port} recovered ✅`);
      }
    } catch {
      server.healthy = false;
    }
  }
}, 5000);

lb.listen(8080, () => console.log("[LB] Listening on :8080"));

Real-World Architecture: Multi-Tier Load Balancing

Production systems typically stack two layers of load balancing.

Algorithm Comparison

Algorithm	Best For	Downside
Round Robin	Uniform servers, short requests	Ignores server load
Weighted Round Robin	Mixed server capacities	Weights need manual tuning
Least Connections	Long-lived connections, file uploads	Requires shared state
IP Hash	Simple session stickiness	Uneven if few client IPs
Consistent Hashing	Caches, sharded data, microservices	More complex to implement
Random	Stateless services, large server pools	Can cause hot spots

Sticky Sessions vs. Stateless Design

Rule of thumb: Prefer stateless architecture backed by Redis or a database over sticky sessions. Stickiness is a workaround; statelessness is a design principle.

Common Load Balancer Tools

Tool	Layer	Use Case
Nginx	L4 + L7	Most common, easy config
HAProxy	L4 + L7	High-performance TCP/HTTP balancing
AWS ALB	L7	AWS-native, path/header routing
AWS NLB	L4	Ultra-low latency TCP/UDP
Cloudflare	L7	Global, with DDoS protection
Envoy	L4 + L7	Service mesh (Istio, Consul)
Traefik	L7	Kubernetes-native, auto-discovery

✅ Checklist Before Moving On

[ ] I can explain the difference between L4 and L7 load balancing
[ ] I know when to use Round Robin vs Least Connections
[ ] I understand why Consistent Hashing is better than IP Hash for dynamic pools
[ ] I can describe how health checks keep a pool clean
[ ] I know why stateless services scale better than sticky sessions

➡️ Next: Level 3 — Databases

⚖️ Load Balancing: The Traffic Director ​

Why Load Balancing Exists ​

Layer 4 vs Layer 7 Load Balancing ​

Load Balancing Algorithms ​

1. Round Robin ​

2. Weighted Round Robin ​

3. Least Connections ​

4. IP Hash (Sticky Routing) ​

5. Consistent Hashing ​

📐 The Math of Consistent Hashing ​

Health Checks ​

Complete: Minimal HTTP Load Balancer in Node.js ​

Real-World Architecture: Multi-Tier Load Balancing ​

Algorithm Comparison ​

Sticky Sessions vs. Stateless Design ​

Common Load Balancer Tools ​

✅ Checklist Before Moving On ​

⚖️ Load Balancing: The Traffic Director

Why Load Balancing Exists

Layer 4 vs Layer 7 Load Balancing

Load Balancing Algorithms

1. Round Robin

2. Weighted Round Robin

3. Least Connections

4. IP Hash (Sticky Routing)

5. Consistent Hashing

📐 The Math of Consistent Hashing

Health Checks

Complete: Minimal HTTP Load Balancer in Node.js

Real-World Architecture: Multi-Tier Load Balancing

Algorithm Comparison

Sticky Sessions vs. Stateless Design

Common Load Balancer Tools

✅ Checklist Before Moving On