By Evan Vega
The least intuitive finding about linking Macs for AI is that for a great many workloads
the right number of machines is one. Inside a single Mac the processor reaches unified memory at
full bandwidth with no interconnect in the path; splitting a model across two machines puts a
cable into every round trip.
Apple’s memory link makes that cable fast without making it free, so a model that already fits
in one machine runs faster on one machine. A second machine extends capacity for the narrow band
of models that will not fit alone. Novel Cognition’s decision table is at
macluster.novcog.us.com.
If the model already fits in one machine, one machine is faster. Nothing beats not needing the wire.
The build this analysis is anchored to is an M3 Ultra Mac Studio with 256 GB paired with an M4
Pro Mac mini with 48 GB. Both machines have Thunderbolt 5, so both qualify for RDMA; Apple’s
technote uses an M4 Pro Mac mini as its own worked example, listing the three RDMA interfaces such
a machine exposes.
The constraint is not the 304 GB total but the 5.3-to-1 ratio. Tensor parallelism wants
comparable shards, each shard must fit the machine holding it, and every layer ends in a
synchronisation that completes when the slowest participant completes. EXO’s topology-aware
placement weights the split by device resources and measured link quality, which mitigates this
without removing it. The published speedups were measured on matched hardware.
The honest expectation for that pair: it expands what fits rather than transforming throughput,
and for any model that already fits in 256 GB the single Studio is faster, because no interconnect
beats no interconnect. Worked through at the two-node build.
The most common error in this area is treating EXO, Apple’s MLX framework and RDMA over
Thunderbolt as competing options. They are not alternatives; they stack.
RDMA over Thunderbolt is a capability built into macOS, not software you install. Apple’s
technote TN3205 states it requires macOS 26.2 or later, an Apple silicon Mac, and Thunderbolt 5.
MLX is Apple’s open-source framework that actually runs the model, and MLX Distributed provides
the communication between machines; its RDMA backend is called jaccl. EXO, from Exo
Labs, is an orchestrator that sits on top of both, handling device discovery and deciding how to
split a model across the cluster.
Remove RDMA and a cluster still works, more slowly. Remove EXO and it still works, configured
by hand — Apple’s own research write-up drives a cluster with mlx.launch and no
orchestrator at all. Remove MLX and there is no inference left to distribute. The full breakdown
is at the three layers.
Independent figures, from Geerling’s four-machine M3 Ultra cluster: Qwen3-235B at 8-bit reached
about 32 tokens per second on the full cluster; Kimi K2 Thinking, a trillion-parameter
mixture-of-experts model with roughly 32 billion parameters active per token, ran at around 30
tokens per second. On the HPL benchmark a single M3 Ultra produced 1.3 TFLOPS against 3.7 across
all four. The cluster drew under 250 watts and idled below 10.
Vendor figures, from Exo Labs: up to 1.8× on two devices, 3.2× on four, and a 99%
latency reduction over Thunderbolt 5. Apple reports up to 4.6× faster time-to-first-token on
an M5 Ultra Mac Studio cluster running a mixture-of-experts model above 800 billion parameters.
Geerling’s loaned cluster, 1.5 TB of unified memory across four machines, came to just under
$40,000 as configured, and he stated he could not justify that himself. Against the obvious
alternatives — Nvidia’s DGX Spark and AMD’s AI Max+ 395, both capped at 128 GB — he
found a single M3 Ultra matching four maxed-out systems of either kind.
For years the accurate summary of Mac clustering was that it bought capacity rather than speed.
That described pipeline parallelism, where each machine holds a block of the model’s
layers and work passes down the line, leaving one machine busy at a time. Adding machines fitted
larger models without making generation faster.
RDMA made a second approach practical. In tensor parallelism each machine holds a
slice of every layer and all of them compute simultaneously, synchronising at each layer. That
requires hundreds of round trips per token, which is viable only at very low latency. Exo Labs
publishes up to 1.8× on two devices and 3.2× on four, and states RDMA reduces
inter-Mac latency from roughly 300 microseconds to 3; the MLX pull request behind the feature is
cited at 300 microseconds down to under 50. Those figures measure at different points in the
stack, and this analysis quotes both rather than selecting the flattering one.
The most informative published result is not a number but a direction. Testing a four-machine
cluster Apple loaned him, Jeff Geerling found EXO speeding up as nodes were added while
llama.cpp’s layer-splitting RPC approach slowed down — same hardware, same cables,
opposite slope. Detail at what it is good at.
Three failures recur in public reports. macOS Thunderbolt Bridge breaks RDMA outright, because
jaccl cannot address individual Thunderbolt ports through a bridged interface, and the
fix must be reapplied after every boot. RDMA ships disabled and is enabled only from Recovery mode
with rdma_ctl enable, once per machine, with no warning if a machine is missed. And
there are no Thunderbolt 5 switches, so the topology is a full mesh and roughly four machines is
the practical ceiling.
Geerling, who maintains the most widely used Ansible playbook for managing Macs, states plainly
that managing Linux clusters is easier, and notes that macOS cannot even run a system upgrade over
SSH. The capability is real and newly real; the operational maturity has not caught up. Field
reports are collected at field reports.
This analysis was produced under the Hidden State Drift method, which
separates what a vendor claims from what someone independent measured. The ten pre-publication
checks behind it are published at hiddenstatedrift.com/verify.
Novel Cognition is not affiliated with Apple or Exo Labs.
Part of the Frontier Watch Series: Read the previous investigation
More Coverage:
→ Read this investigation on North Denver Tribune
→ Coverage from Daily Colorado News
