Resources for large scale models

Related to data parallelism, model parallelism, pipeline parallelism and their variants

https://huggingface.co/transformers/master/parallelism.htmlhuggingface.co

dlatscale_draft/week03 at main · yandexdataschool/dlatscale_draftGitHub

Mixture of Experts

FastMOE: https://arxiv.org/abs/2103.13262, https://github.com/laekov/fastmoe