/* hypothesizing */ If you're using it for NLP, your dataset (token ids) typically weighs much less than intermediate tensors. So, i see two scenarios here:

(1) distribute data chunks as you train using more conventional bittorrent systems (e.g. https://academictorrents.com but internal) (2) since you most likely use raw unlabeled data (e.g. just text), peers can crawl it straight from the web

Yeah, it's probably less of a concern for text tasks, where the data per example is relatively light (though there is a whole internet worth of text data...)

I mostly work with audio, where individual examples are ~2MB, so the dataset sizes get very heavy quickly.

Can you train a huge neural network without a supercomputer? Imagine you want a GPT-3-sized model, but instead of $10⁸ GPU cluster you've got support from thousands of volunteers across the world - gamers, research labs, small companies. What kind of system would you use to let them work together despite internet latency, packet loss, and hardware failures?

We at Learning@home are building just such a system. Together, we want to change large-scale deep learning from private experiments behind closed doors into a decentralized peer-to-peer activity where everyone can participate.

Let's build the BitTorrent of deep learning :)