Petals

• Distributed inference of large language models across volunteer-GPUs
• Fine-tuning of complex models via peer-shared resources
• Support for major open-source model families (eg. LLaMA, BLOOM)
• Integration with libraries such as Hugging Face Transformers for ease of use
• Enables inference on consumer-grade hardware by splitting model layers
• Private swarm setup option for organisations needing isolated networks
• High throughput and reduced latency compared to traditional offloading methods
• Transparency of model internals and activations for research tasks
• Community-driven contribution system where users add compute to the network
• Open-source licence; full access to codebase and community extensions

Petals is a groundbreaking decentralised AI system designed to make the power of large language models accessible without requiring massive hardware investments. Traditional large models often need hundreds of gigabytes of GPU memory and multi-GPU servers to run or fine-tune. Petals addresses this by splitting the model into many smaller blocks that are hosted across a peer-to-peer network of GPUs contributed by volunteers. A user requesting inference connects to this network and their input traverses the chain of blocks managed by many nodes, thereby enabling even modest hardware to access large-scale models. This architecture allows inference and fine-tuning of very large models in a distributed manner, and supports open-source libraries and workflows so developers and researchers can customise models, inspect internal activations, and build new applications. Because no central server is required, the system enables greater flexibility, transparency and potential cost-savings. Organisations can even form private swarms for enhanced security and control. The community-centric nature of Petals fosters collaboration, allowing anyone to contribute compute, host blocks, or run models. The system has shown promising latency and throughput improvements compared to offloading heavy models entirely to local hardware. Petals is particularly well suited for research labs, independent developers, and edge-AI scenarios where hardware is limited but access to large models is desired. With open-source licensing and active community support, Petals helps shift the barrier to entry for advanced natural-language-processing and large-model applications.

Share Your Experience: