Large Scale Deep Learning Using the High-Performance Computing Libraries OpenMPI and DeepSpeed

import time
import math
import random
import os
def sample(num_samples):
num_inside = 0
for _ in range(num_samples):
x, y = random.uniform(-1, 1), random.uniform(-1,1)
if math.hypot(x, y) <= 1:
num_inside +=1
return num_inside
num_inside = sample(10**8)
pi = (4*num_inside)/num_samples
print(pi)Wall time: 59.8 s, GPU AWS instance: g4dn-16xlarge, 32 CPU cores, 256 GB memory per GPU, 1 instance
from mpi4py import MPI
import numpy
import sys
comm = MPI.COMM_SELF.Spawn(sys.executable,
argos= [‘’],
N = numpy.array(10**8, ‘i’)
comm.Bcast([N, MPI.INT], root=MPI.ROOT)
PI = numpy.array(0.0, ‘d’)
comm.Reduce(None, [PI, MPI.DOUBLE],
op=MPI.SUM, root=MPI.ROOT)
comm.Disconnect()Wall time: 9.79 s, AWS instance: g4dn-16xlarge, 32 CPU cores, 256 GM memory per GPU, 6 instances
#!/usr/bin/env python
from mpi4py import MPI
import numpy
comm = MPI.Comm.Get_parent()
size = comm.Get_size()
rank = comm.Get_rank()
name = MPI.Get_processor_name()
N = numpy.array(0, dtype='i')
comm.Bcast([N, MPI.INT], root=0)
h = 1.0 / N; s = 0.0
for i in range(rank, N, size):
x = h * (i + 0.5)
s += 4.0 / (1.0 + x**2)
PI = numpy.array(s * h, dtype='d')
comm.Reduce([PI, MPI.DOUBLE], None,
op=MPI.SUM, root=0)
  1. Large Scale Distributed Deep Networks. J. Dean, G. Corrado, R. Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc’Aurelio Ranzato, A. Senior, P. Tucker, Ke Yang, A. Ng. NIPS 3 December 2012
  2. Zero: Memory Optimizations Toward Training Trillion Parameter Models. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He. SC ’20: Proceedings of the International Conference for High-Performance Computing, Networking, Storage and Analysis, November 2020 Article No.: 20, Pages 1–16
  3. Wikipedia (English). Monte Carlo Method. Accessed Aug 1, 2022



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
ODSC - Open Data Science

ODSC - Open Data Science


Our passion is bringing thousands of the best and brightest data scientists together under one roof for an incredible learning and networking experience.