More organisations are exploring local AI — running AI tools on hardware they own rather than relying entirely on cloud services. The main reasons are cost, control and data: if your organisation handles sensitive information, or your AI usage is heavy enough that cloud costs are adding up, having the compute in-house starts to make sense. The question is what that hardware looks like and what it can realistically do.

ServeTheHome recently documented a full 8-node cluster built around NVIDIA's GB10 Grace Blackwell Superchip, the processor at the heart of the DGX Spark platform. The build gives a useful indication of what can be achieved by scaling multiple GB10 systems for larger local AI workloads, including the networking, shared storage and software considerations involved.

This article summarises what ServeTheHome found, explains the technology behind the build, and discusses who this kind of infrastructure is likely to suit.

What is the GB10?

The NVIDIA GB10 Grace Blackwell Superchip combines an Arm CPU and a Blackwell GPU on a single package, with 128 GB of LPDDR5X memory shared between the two. Because the CPU and GPU draw from the same memory pool, data does not need to be copied between separate CPU and GPU memory spaces, which removes one of the bottlenecks common in conventional discrete GPU setups.

According to NVIDIA's published specifications, the GB10 delivers 1 petaFLOP of FP4 AI compute. NVIDIA's DGX Spark programme certifies boards based on the GB10 from a number of hardware partners, including ASUS, Gigabyte, Dell, Lenovo, HP, MSI and Acer. Each board includes a ConnectX-7 NIC supporting high-bandwidth networking, used in the ServeTheHome build at 100 GbE per node via the MikroTik switch described below.

Why scale to multiple nodes?

Larger AI models require more memory than a single GB10 node provides. A single node carries 128 GB of LPDDR5X. Running very large language models, particularly those with hundreds of billions of parameters, requires the model to be distributed across multiple nodes, with each node holding and processing a portion of the model.

ServeTheHome's 8-node build provided 1 TB of combined memory across the cluster (8 x 128 GB). The cluster delivered 160 Arm CPU cores and 8 petaFLOPs of aggregate FP4 AI compute. It is important to note that this is not equivalent to a single machine with 1 TB of unified memory: large model inference across a cluster works by distributing the model's layers or parameters across nodes, with the nodes communicating over a high-bandwidth interconnect during inference. Latency and throughput across that interconnect are therefore significant engineering considerations.

The networking setup

For a 2-node configuration, ServeTheHome used a direct-attach copper (DAC) cable between the two ConnectX-7 NICs. This avoids the need for a switch and allows RDMA over Converged Ethernet (RoCE), which lets the GPUs on separate nodes communicate directly without involving the host CPU for each transfer.

Scaling to 8 nodes required a switch with sufficient port density and bandwidth. The build used a MikroTik CRS804-4XQ-IN, a compact switch with four QSFP-DD ports capable of 400 GbE each. In the build, each port was configured with a breakout cable to serve four nodes, giving 16 x 100 GbE connections in total. RouterOS on this switch supports RoCE traffic.

Switch	MikroTik CRS804-4XQ-IN
Switch ports	4 x 400 GbE QSFP-DD (breakout to 4 x 100 GbE each)
Node NICs	ConnectX-7 per GB10 board
Protocol	RDMA over Converged Ethernet (RoCE)
Management	Cisco Catalyst C1300-12XT-2X (10 GbE out-of-band management)
2-node alternative	Direct-attach copper (DAC) cable, no switch required

Shared storage

The build included a QNAP NAS configured for two purposes. First, as a central repository for model weight files, so weights are not duplicated across the local NVMe drive in each node. Second, as a shared workspace for multi-node agent workflows, allowing tasks distributed across the cluster to read from and write to a common filesystem.

Storage architecture for a cluster of this kind involves trade-offs between NVMe read speed (fast, local, not shared) and NAS throughput (slower, but accessible across all nodes). The right approach depends on how frequently model weights need to be reloaded and whether the workflow is I/O-bound.

Running larger language models locally

ServeTheHome used the cluster to run Kimi K2.5 and Kimi K2.6 from Moonshot AI, which are mixture-of-experts models with a large total parameter count. These models exceed what a single node can handle, so they were distributed across the cluster.

The cluster architecture means inference involves inter-node communication for each forward pass through the model. ServeTheHome's published testing provides practical data on throughput and latency in this configuration, including results for the specific models they ran across the cluster.

Who is this relevant for?

A single GB10 node can support a wide range of local AI inference workloads, with NVIDIA positioning the platform for models up to approximately 200 billion parameters depending on model architecture, quantisation and runtime requirements. Multi-node configurations are relevant where the intended workload exceeds what a single node can handle, or where parallel inference throughput justifies the additional networking and storage infrastructure.

Organisations that may want to evaluate a multi-node GB10 setup include:

• AI and ML teams running models that cannot fit in a single node's memory and where cloud inference costs are a significant ongoing expense.
• Businesses handling data that needs to stay on-premises for compliance reasons, and where the intended models are too large for a single-node deployment.
• Research groups or technology teams who need persistent local compute for iterative model development.

A 2-node configuration (two GB10 boards, one DAC cable, no switch) is the simplest starting point and avoids the complexity of switch configuration. Adding further nodes increases both capability and infrastructure requirements.

TechVerse and GB10

We stock GB10-based systems and can supply compatible networking hardware for multi-node builds. If you are evaluating a GB10 deployment and want to discuss hardware options, networking requirements or availability, speak to the TechVerse team and we can help you identify the right configuration for your requirements.