Petals adds peer‑to‑peer inference for large language models

Petals has added a peer‑to‑peer inference layer that lets engineers run large language models on home hardware — the project’s site details the open‑source implementation and its distributed‑compute design [Petals].

The new layer creates a decentralized network where each node contributes CPU or GPU cycles to a shared inference pool. Tasks are split across peers, so a single user can query a model without provisioning a dedicated server or cloud GPU instance. The approach reduces the need for centralized infrastructure and lowers the cost barrier for running state‑of‑the‑art LLMs on consumer‑grade machines [Hacker News].

Why it matters: By spreading inference work across many modest devices, Petals cuts infrastructure expenses, making LLMs accessible to developers and researchers who lack large‑scale compute budgets. The lowered entry point expands experimentation and prototyping beyond well‑funded labs. Moreover, the network demonstrates a concrete step toward decentralized AI, where both training and deployment can occur without a single controlling entity.

The codebase is publicly available, inviting contributions that could improve routing efficiency, security, and support for additional model architectures.