A fault-tolerance model for Hadoop rack-aware resource management system

Timothy Moses(1), Oladunjoye John Abiodun(2),


(1) Federal University of Lafia
(2) Federal University Wukari

Abstract


The central resource manager of Hadoop Yet Another Resource Manager (YARN) has posed a major concern to big data analysis and exploration. The central arbiter is overwhelmed whenever there are resource requests by application masters and heartbeat communication from several name nodes in the Hadoop cluster; thereby, degrading the performance of the framework. An attempt to decentralize the resource manager's responsibilities by introducing a new layer in the cluster named the Rack Unit Resource Manager (RU_RM) layer increased cluster performance but introduced a fault-tolerance concern. This work, therefore, developed a fault-tolerant model to allow for efficient and effective data analysis in the Hadoop cluster. A pseudo-distributed computation was set up with the help of the YARN Scheduler Load Simulator (SLS) and WordCount operation performed with varying input sizes. Two fault scenarios were presented and the results obtained showed that with an increase in input size (workload), the running time of the developed fault-tolerant model though slightly higher than that of the existing model, is significantly negligible when compared to the computation bottleneck incurred anytime RU_RM fails. The developed model, therefore, has good performance in the presence of failure of a unit (RU_RM) in the cluster.


Keywords


Hadoop YARN; Fault-tolerant YARN; Rack-aware resource manager; Fault-tolerance resource management

Full Text:

PDF

References


K.V. Vinod, C.M. Arun, D. Chris, A. Sharad, K. Mahadev, E. Robert, G. Thomas, L. Jason, S. Hitesh, S. Siddahart, S. Bikas, C. Carlo, O.M. Owen, R. Sanjay, R. Benjamin, and B. Eric “Apache Hadoop YARN: Yet Another Resource Negotiator”. SOCC ’13 Proceedings of the 4th annual symposium on Cloud Computing, New York, (2013) NY: ACM, 2013. http://dx.doi.org/10.1145/2523616.2523633

K. Konstantinos, A. Suresh, and C. Douglas “Advancements in YARN Resource Manager”. Encyclopedia of Big Data Technoligies: Springer International Publishing, 2018. https://doi.org/10.1007/978-3-319-63962-8_207-1

S. Shenker, and I. Stoica “Hierarchical scheduling for diverse datacentre workloads”. Proceedings of the 4th Annual Symposium on Cloud Computing, ACM, Santa Clara, California, 2013.

Apache “Apache Hadoop”. Retrieved from https://hadoop.apache.org/, on 3rd March, 2021.

A.T.H. Ibrahim, B.A. Nor, G. Abdullah, Y. Ibrar, X. Feng and U. K. Samee “MapReduce: Review and Challenges”. Springer Journal, 109(1), 389-421, 2016. http://www.doi.org/10.1145/1327452.1327492

T. Moses, H.C. Inyiama and S.O. Anigbogu “A rack-aware scalable resource management system for Hadoop YARN”. International Journal of High Performance Computing and Networking, 16(1): 1-13, 2020. http://dx.doi.org/10.1145/2523616.2523637

O. Selvitopi, G.V. Demirci, A. Turk and C. Aykanati “Locality-aware and load-balanced static task scheduling for MapReduce”. Future generation computer systems, 90: 49-61, 2018. https://doi.org/10.1016/j.future.2018.06.035

N. Maleki, H.R. Faragardi, A.M. Rahmani, M. Conti and J. Lotstead “TMaR: a two-stage MapReduce scheduler for heterogeneous environments”. Human-centric computing and information sciences, 10(42); 1-26, 2020. https://doi.org/10.1186/s13673-020-00247-5

J. Rathinaraja, and V.S. Ananthanarayana “Multi-Level per Node Combiner (MLPNC) to minimize MapReduce job latency on virtualized environment”. 33rd Association for Computing Machinery (ACM) Symposium on Applied Computing, SAC, Pau, France, 2018a. https://doi.org/10.1145/3167132.3167149

J. Rathinaraja and V.S. Ananthanarayana “Dynamic aware reduce task scheduling in MapReduce on virtualized environment”. IEEE Computer Society, Kunming, China June 13-15, 2018b. https://doi.org/10.1109/SERA.2018.8477195

K. Hu, J. Hung, H. Chen and S. Rao “Scaling Linkedln’s Hadoop YARN cluster beyond 10,000 nodes”. Linkedln engineering, 2021. https://engineering.linkedin.com/blog/2021/scaling-linkedin-s-hadoop-yarn-cluster-beyond-10-000-nodes

N. Orensa “A design framework for efficient distributed analytics on structured big data”. A thesis submitted to the College of Graduate and Postdoctoral Studies, Department of Computer Science, University of Saskatchewan (2021). https://harvest.usask.ca/bitstream/handle/10388/13511/ORENSA-THESIS-2021.pdf?sequence=1&isAllowed=y

GeeksforGeeks “Hadoop YARN architecture”. Retrieved from https://www.geeksforgeeks.org/hadoop-yarn-architecture/ on 6th June, 2022.

N.W. Ismahene, S. Boudouda and N. Zarour “A dynamic scaling approach in Hadoop YARN”. International Journal of Organization and Collective Intelligence, 12(2):1-17, 2022. https://doi.org/10.4018/IJOCI.286176

J. Rathinaraja, V.S. Ananthanarayana and P. Anand “Fine-grained data-locality aware MapReduce job scheduler in a virtualized environment”. Journal of Ambient Intelligence and Humanized Computing, 11(10) :4261-4272, 2018. https://doi.org/10.1007/s12652-020-01707-7




DOI: https://doi.org/10.36596/jcse.v4i1.651

Refbacks

  • There are currently no refbacks.


Journal of Computer Science and Engineering (JCSE)
ISSN 2721-0251 (online)
Published by : ICSE (Institute of Computer Sciences and Engineering)
Website : http://icsejournal.com/index.php/JCSE/
Email: jcse@icsejournal.com

Creative Commons License is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.