Mesh LLM: distributed AI computing on iroh

Summary

Mesh LLM pools GPUs across trusted iroh-connected machines and exposes them through one OpenAI-compatible local API. It can run models locally, route requests to peers, or pipeline models that exceed a single machine's memory.

Problem

Hosted LLMs limit control over model versions, data location, hardware, privacy, and operating cost.
Many teams own separate GPUs that cannot run larger models together as one system.

Approach

Each machine runs an iroh endpoint identified by a public key, using authenticated QUIC connections with NAT traversal and relay fallback.
A gossip layer manages peer admission, trust, version compatibility, model discovery, and routing.
Requests run locally, route to a peer with the model loaded, or split a model by layer ranges across pipeline stages on multiple machines.
A plugin runtime exposes model and service capabilities through MCP, HTTP, inference, and mesh events while presenting a standard API at http://localhost:9337/v1.

Results

The catalog includes more than 40 models, ranging from 0.5B-parameter models that fit on laptops to a 235B-parameter mixture-of-experts model.
The software package is about 18 MB and supports public or private mesh deployments.
Two iroh relays in different regions provide fallback connectivity when direct peer connections fail.
No latency, throughput, cost, reliability, or model-quality benchmark results are provided in the excerpt.
The strongest demonstrated claim is functional: several modest machines can pipeline a model that no single machine can hold, while standard OpenAI clients continue using the local API.

Link

https://www.iroh.computer/blog/mesh-llm