Scaling MCP servers: horizontal, vertical and distributed

Who this is for and what you can do after reading

This is for developers who already run a single MCP server and now worry about concurrency and stability, especially teams deploying MCP in enterprise environments. After reading, you can separate horizontal scaling, vertical scaling and distributed nodes, follow how the Redis registration and heartbeat in the samples work, and know which parts of the code must change before they run.

Four ways to scale an MCP server

Enterprise MCP services often need to handle large volumes of requests with low latency. The usual approaches fall into four groups: horizontal scaling, vertical scaling, resource optimization and distributed architecture^[1].

Horizontal scaling runs several MCP server instances behind a load balancer, so requests are spread across them. It handles more simultaneous requests, and it provides fault tolerance because other instances keep serving when one fails^[1].

Vertical scaling makes a single instance handle more requests by adding CPU and memory and fine-tuning its configuration. Thread pools, request timeouts and memory limits are the examples given^[1].

Resource optimization reduces wasted work. It uses more efficient algorithms, caching and asynchronous processing to cut resource consumption and shorten response times^[1].

Distributed architecture lets several MCP nodes share the load and provide redundancy through coordination^[1]. These approaches are usually combined. Tune a single instance first, add instances horizontally, and only then coordinate nodes with each other.

Horizontal scaling: several instances behind a load balancer, state in Redis

The first difficulty in horizontal scaling is session state. The C# sample configures a Redis distributed cache with AddStackExchangeRedisCache. It reads the connection string named RedisConnection and sets the instance name prefix to MCP_^[1]. A user's state then does not have to stay tied to one instance.

The sample turns on distributed caching in the server options with EnableDistributedCaching set to true and CacheExpirationMinutes set to 60^[1]. It also registers a tool with AddMcpTool<HighPerformanceTool>() so the tool can be used across several MCP instances^[1]. The complete configuration is:

// ASP.NET Core MCP load balancing configuration
public class McpLoadBalancedStartup
{
    public void ConfigureServices(IServiceCollection services)
    {
        // Configure distributed cache for session state
        services.AddStackExchangeRedisCache(options =>
        {
            options.Configuration = Configuration.GetConnectionString("RedisConnection");
            options.InstanceName = "MCP_";
        });
        
        // Configure MCP with distributed caching
        services.AddMcpServer(options =>
        {
            options.ServerName = "Scalable MCP Server";
            options.ServerVersion = "1.0.0";
            options.EnableDistributedCaching = true;
            options.CacheExpirationMinutes = 60;
        });
        
        // Register tools
        services.AddMcpTool<HighPerformanceTool>();
    }
}

Names such as AddMcpServer and EnableDistributedCaching are written as part of the sample. Before use, confirm against the documentation of the SDK you use that these names exist and that the parameters mean what the sample assumes.

Vertical scaling: thread pools, request limits and timeouts

The typical vertical-scaling step is tuning the thread pool. The Java sample sets the core pool size to the number of available processors and the maximum to twice that number, with a comment that this is a common heuristic for I/O-bound tasks^[1]:

        // Configure thread pool for optimal performance
        int processors = Runtime.getRuntime().availableProcessors();
        int optimalThreads = processors * 2; // Common heuristic for I/O-bound tasks
        
        ExecutorService executorService = new ThreadPoolExecutor(
            processors,       // Core pool size
            optimalThreads,   // Maximum pool size 
            60L,              // Keep-alive time
            TimeUnit.SECONDS,
            new ArrayBlockingQueue<>(1000), // Request queue size
            new ThreadPoolExecutor.CallerRunsPolicy() // Backpressure strategy
        );

The pool uses a bounded queue with capacity 1000. When the queue is full, the CallerRunsPolicy makes the submitting thread run the task itself, which acts as a backpressure signal to upstream callers^[1]. This keeps requests from piling up without limit.

The same sample sets three limits: a maximum request size of 1MB, at most 100 concurrent requests, and a request timeout of 5000 milliseconds^[1]. These are sample values, not recommendations. Measure request sizes, concurrency and timeouts against your own workload before you settle on them.

Distributed nodes: registration and heartbeats through Redis

The distributed example is written in Python. When a node starts, it adds its ID to the Redis set mcp:nodes and writes its status to the hash mcp:node:{id}, connecting to redis://redis-master:6379^[1]. After startup the status changes to running, and the node has joined the cluster.

Tool registration has two parts. Every node registers the common tools. Specialized tools are assigned by taking the last hexadecimal digit of the node ID modulo 3, which spreads the three specialized tools across nodes^[1]:

    def register_tools(self):
        # Register common tools across all nodes
        self.server.register_tool(CommonTool1())
        self.server.register_tool(CommonTool2())
        
        # Register specialized tools for this node (could be based on node_id or config)
        if int(self.node_id[-1], 16) % 3 == 0:  # Simple way to distribute specialized tools
            self.server.register_tool(SpecializedTool1())
        elif int(self.node_id[-1], 16) % 3 == 1:
            self.server.register_tool(SpecializedTool2())
        else:
            self.server.register_tool(SpecializedTool3())

The heartbeat writes lastHeartbeat, load and maxLoad every 5 seconds. These hold the time of the last heartbeat, the number of active requests and the request limit. If a write fails, the loop waits one second before continuing^[1]:

    async def _heartbeat(self):
        """Periodic heartbeat to indicate node health"""
        while True:
            try:
                await self.redis.hset(
                    f"mcp:node:{self.node_id}", 
                    mapping={
                        "lastHeartbeat": int(time.time()),
                        "load": len(self.server.active_requests),
                        "maxLoad": self.server.max_concurrent_requests
                    }
                )
                await asyncio.sleep(5)  # Heartbeat every 5 seconds
            except Exception as e:
                print(f"Heartbeat error: {e}")
                await asyncio.sleep(1)

Two points need attention. First, the heartbeat calls time.time(), but the sample's imports are asyncio, aioredis, uuid and AsyncMcpServer, with no time import, so the code fails as written^[1]. Second, the sample writes the heartbeat fields to Redis, yet no code in it reads them. Load information only helps once a component reads it and uses it to route requests.

Shutdown sets the node status to stopping, stops the server, removes the node from mcp:nodes, deletes its status hash and closes the Redis connection^[1]. This keeps records of stopped nodes from remaining in the cluster.

Common problems

• Uneven load across nodes: the heartbeat records load but does not distribute requests. Load-aware routing needs a component at the entry point that reads each node's heartbeat and decides where to forward requests.
• Redis as a single point of failure: the sample connects to one Redis address and does not address Redis high availability. Plan primary-replica or cluster deployment for production.
• Timeouts and thread pools out of step: if request timeouts are long while the queue is large, queued requests expire after holding resources. Tune these values together, not one at a time.
• Code fails to run: the Python sample is missing the time import. The SDK names in both the C# and Python samples also need checking against your version.

Next steps

Security is the next topic worth reading. Once you add nodes and entry points, authentication and access control matter more^[1]. Planning them together with scaling costs less than adding them later.

Adapted from lesson material in Microsoft's open-source course *MCP for Beginners*, covering scaling and high performance in its advanced topics^[1], released under the MIT License.