DecentralizePy

DecentralizePy is a framework for distributed machine learning. It runs on arbitrary network topologies, including decentralized, federated, and parameter server setups. Researchers use it to study communication efficiency, privacy, and data heterogeneity.

This fork adds the tooling to emulate Privacy-Robustness-Efficiency (PRE) trade-offs. It uses Differential Privacy (DP), trimmed mean aggregation, and uniform asymmetric quantization as reference techniques.

Features

• Dependency injection through a central container
• Action and modifier system to wrap training components
• Hardware acceleration on CPU, CUDA, and ROCm
• Remote dispatch over SSH
• Stream and state transforms for quantization and compression
• A unified transport layer for binary packet exchange
• First-class metrics with a central tracker

Install

Install uv if it is not available:

curl -LsSf https://astral.sh/uv/install.sh | sh

Install just if you want to use the task recipes:

cargo install just      # via Cargo
sudo apt install just   # Debian / Ubuntu
brew install just       # macOS

Install the project in development mode. Choose one PyTorch profile. The profiles are mutually exclusive.

uv sync --extra dev --extra cpu   # CPU-only PyTorch
uv sync --extra dev --extra cuda  # CUDA (NVIDIA GPU) PyTorch
uv sync --extra dev --extra rocm  # ROCm (AMD GPU) PyTorch

Run a Job

Each run uses one JSON configuration file. Start a job on the local machine with:

uv run --no-sync src/decentralizepy/runner/dispatch_local.py config.json --seed 123

A justfile provides convenience recipes. The recipes pick a random seed for you. You can use just instead of the direct command.

just execute config.json        # run a job locally
just dispatch config.json 3 out # run three jobs on remote machines
just sync cpu                   # uv sync --extra dev,<backend>
just ci                         # ruff check, format check, and tests

The run writes results to an out/<timestamp>/ directory. Each process writes a log file and a metrics file.

Example Configurations

The configs/ directory holds example configurations. They show the PRE trade-offs in stages:

• 0_baseline.json - plain training with no mitigations
• 1_private.json - training with differential privacy
• 2_robust.json - training with trimmed mean aggregation
• 3_robust_poisoned.json - trimmed mean aggregation under a label flipping attack
• 4_quantized.json - training with uniform asymmetric quantization
• 5_combined.json - combines all techniques

Run an example locally with:

just execute configs/0_baseline.json

Remote Execution

For remote execution, configure the machines in remotes.json at the project root. The file lists the SSH hosts and their settings. See remotes.example.json for a template. The dispatch recipe reads this file and runs jobs on the listed machines.

{
  "defaults": {
    "process_count": 32,
    "work_dir": "/home/my-user/decentralizepy",
    "user": "my-user"
  },
  "remotes": [
    { "host": "my-host.example.com" },
    { "host": "my-other-host.example.com", "backend": "cuda" }
  ]
}

Each remote needs a hostname and a user. The optional backend selects the PyTorch profile (cpu, cuda, or rocm). It defaults to cpu.

Architecture

The framework splits training into modular components. A central container initializes all components with dependency injection. Each component receives only the dependencies it needs. The container is locked after initialization.

• Container manages mapping, graph, datasource, datasets, model, optimizer, criterion, tracker, transport, sharing, transforms, task, trainer, node, actions, and gpu_lock.
• Actions are event-driven hooks. They wrap or mutate components before training. Events include init, startup, pre-train, post-train, pre-exchange, post-exchange, and shutdown. The DifferentialPrivacy and LabelFlipping actions use this system.
• Datasources provide train, test, and validation datasets. Examples are CIFAR10, CIFAR100, and Camelyon17. Models can be defined next to the datasource, such as LeNet in CIFAR10.py.
• Tasks define training and inference on a batch. Implementations include MultiClassification and BinaryClassification.
• Criterions wrap loss functions and handle input and output casts. Examples are CrossEntropyLoss and BCEWithLogitsLoss.
• Trainers control the training loop with a canStep and step interface. FixedTrainer trains for a fixed number of iterations.
• Sharing exchanges and aggregates models. Implementations include PlainAverage and TrimmedMean.
• Transport sends and receives binary packets to peers. It supports per-channel communication and captures bandwidth. Implementations include TCP and Mock.
• Transforms operate on the model state or the byte stream. A state transform such as UniformAsymmetricQuantization compresses model weights. A stream transform such as ZstdCompression or FpzipCompression compresses bytes.
• Nodes coordinate sharing between training iterations. The training loop lives in the trainer. Initialization lives in the container.
• Metrics use a MetricTracker. It records named metrics in a central registry. The registry is written to <output_dir>/<uid>.metrics.json. The transport captures upload and download bandwidth. The trainer captures isolated training and exchange time.

Development

Run the code quality checks before you commit changes:

uv run --no-sync ruff check .
uv run --no-sync ruff check --fix .
uv run --no-sync ruff format .

Run the tests:

uv run --no-sync pytest tests/

Run the type checker:

basedpyright

Legacy

This project is a fork of the original DecentralizePy. The original project provides the foundation for this work. Credits and original citations follow.

Citing the Original Project

Cite the original project as:

@inproceedings{decentralizepy,
    author = {Dhasade, Akash and Kermarrec, Anne-Marie and Pires, Rafael and Sharma, Rishi and Vujasinovic, Milos},
    title = {Decentralized Learning Made Easy with DecentralizePy},
    year = {2023},
    isbn = {9798400700842},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3578356.3592587},
    doi = {10.1145/3578356.3592587},
    booktitle = {Proceedings of the 3rd Workshop on Machine Learning and Systems},
    pages = {34-41},
    numpages = {8},
    keywords = {peer-to-peer, distributed systems, machine learning, middleware, decentralized learning, network topology},
    location = {Rome, Italy},
    series = {EuroMLSys '23}
}

Built with the Original DecentralizePy

• Epidemic Learning: paper. Martijn de Vos, Sadegh Farhadkhani, Rachid Guerraoui, Anne-Marie Kermarrec, Rafael Pires, and Rishi Sharma. "Epidemic Learning: Boosting Decentralized Learning with Randomized Communication." Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
• Get More for Less in Decentralized Learning Systems: paper. Akash Dhasade, Anne-Marie Kermarrec, Rafael Pires, Rishi Sharma, Jeffrey Wigger, and Milos Vujasinovic. "Get More for Less in Decentralized Learning Systems." IEEE 43rd International Conference on Distributed Computing Systems (ICDCS), 2023.