Cost-Effective High Performance Distributed GPU Cluster for Deep Learning Tasks

Karpov, Kirill; Kachan, Dmitry; Iushchenko, Maksim; Luzianin, Ivan; Siemens, Eduard

doi:10.25673/76925

Proceedings of International Conference on Applied Innovation in IT · 2022/03/09 · Vol. 10 · Issue 1 · pp. 1–6

Cost-Effective High Performance Distributed GPU Cluster for Deep Learning Tasks

Kirill Karpov, Dmitry Kachan, Maksim Iushchenko, Ivan Luzianin, Eduard Siemens

📄 Download PDF DOI: 10.25673/76925

Abstract

The expenses on computational resources for modern Deep Learning computing can be extremely large. However, most of them are spent on the chassis and not on the GPU units themselves. Since modern mass market

Keywords

Tensor Flow DNN Training Performance Horovod HPC High-Performance Computing

References

"RDMA Aware Networks Programming User Manual," Mellanox Technologies, Tech. Rep. Rev 1.7, 2015. [Online]. Available:.https:// network.nvidia.com/related-docs/prod software/RDMA Aware Programming user manual.pdf
W. Gropp, E. Lusk, and A. Skjellum, Using MPI: Portable Parallel Programming with the Message Passing Interface. The MIT Press, 11 1999. [Online]. Available: https://doi.org/10.7551/mitpress/7056.001.0001
"Nvidia collective communication library (nccl) documentation," Nvidia Corporation, Specification, 2020. [Online]. Available: https://docs.nvidia.com/deeplearning/sdk/nccl-developer-guide/docs/index.html
A. Sergeev and M. Del Balso, "Horovod: fast and easy distributed deep learning in tensorflow," arXiv preprint arXiv:1802.05799, 2018.
Horovod, "Horovod," https://github.com/horovod/horovod, 2017.
"Intel NUC 9 Extreme/Pro Kit, Techincal Product Specification," Intel Corporation, Specification 1, Dec. 2019. [Online]. Available:https://simplynuc.com/wp-content/uploads/2020/01/NUC9QN TechProdSpec.pdf
"Intel NUC 9 Kit, User Guide," Intel Corporation, Specification 1, May 2020. [Online]. Available: https://www.intel.com/content/dam/support/us/en/documents/intel-nuc/nuc-kits/NUC9xyQNX UserGuide.pdf
"ConnectX-5 EN Card," Mellanox Technologies, Tech. Rep., 2020. [Online]. Available: https://www.mellanox.com/files/doc-2020/pb-connectx-5-en-card.pdf
Z. Zhang, C. Chang, H. Lin, Y. Wang, R. Arora, and X. Jin, "Is network the bottleneck of distributed training?" in Proceedings of the Workshop on Network Meets AI & ML, 2020, pp. 8–13.
J. George, "Qperf 0.4.11 (1)-linux man page," Qperf (1): Measure RDMA/IP Performance.
D. Merkel et al., "Docker: lightweight linux containers for consistent development and deployment," Linux journal, vol. 2014, no. 239, p. 2, 2014.
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean,
M. Devin et al., "Tensorflow: Large-scale machine learning on heterogeneous distributed systems," arXiv preprint arXiv:1603.04467, 2016.
N. J. Gunther, Guerrilla Capacity Planning: A Tactical Approach to Planning for Highly Scalable Applications and Services, 1st ed. Springer Publishing Company, 2010.
F. Alkhoury, D. Wegener, K.-H. Sylla, and M. Mock, "Communication efficient distributed learning of neu-ral networks in big data environments using spark," in 2021 IEEE International Conference on Big Data (Big Data), 2021, pp. 3871–3877.