There are lots of confusion out there regarding Yarn and clear concept of it. My target here is to make it clear in a very simple words which will be completely enough for normal data scientist and Big data engineering job. Before I move into the concept, I highlight the fact that main topic here is Cluster resource manager.
Description: Cluster resource management means managing the resources of the Clusters. And by resources we mean Memory, CPU etc. Take into consideration that in separate thread I have talked about another resource manager/scheduler in HPC parts which is Slurm. In spite of the fact that functionalities are same, but they are not same and cannot be replaced.
Background
Yarn came to market as improvement to previous hadoop data-processing framework which was MapReduce data-processing model. Let’s call this old mode MapReduce version 1 (MRv1) since it was in Hadoop 1.0. So to decrease the confusion, MapReduce was both Cluster resource management and data processing in hadoop version 1. I don’t see any needs to go through the design of resource management in hadoop version 1 here as rarely being used recently.
Problems of MRv1 (in Hadoop 1.0)
Obviously this model has several problems and this was the reason that community moved to new model. But in short I highlight the main problems.
a. MapReduce data-processing provides specific programming model which cannot solve all problems in big data.
I would say this was the biggest issue, as it was not possible to use cluster for a variety of models. The problem arises when a non-MapReduce application tries to run in this framework. The application needs to conform to the MapReduce framework programming in order to run successfully. Some of the common issues faced due to this include problems with:
- Ad-hoc query
- Real-time analysis
- Message passing approach
b. Other issues was failure is cascading (node more than 4000), availability, scalability and maybe more importantly efficiency.
Both smaller and larger Hadoop clusters had never used their computational resources with optimum efficiency. In Hadoop MapReduce, the computational resources on each slave node are divided by a cluster administrator into a fixed number of map and reduce slots, which are not fungible. With the number of map and reduce slots set, a node cannot run more map tasks than map slots at any given moment, even if no reduce tasks are running. It harms the cluster utilization because when all map slots are taken (and we still want more), we cannot use any reduce slots, even if they are available, or vice versa.
Yet Another Resource Negotiator (Yarn)
YARN is a resource manager that enable several workloads (data processing engines) to process data simultaneously or better say access the same data set. YARN came up since Hadoop version 2 and it changed the hadoop platform from Single Use System (Batch Apps (MapReduce)) to Multi Purpose Platform (Batch, Interactive(Tez), Streaming(Storm), Online(Hbase), In-Memory(Spark),…). I would say the main power of the YARN is allowing other applications (open-source or proprietary) to have access to Hadoop cluster and use the data set through HDFS.
YARN takes into account all of the available compute resources on each machine in the cluster. Based on the available resources, YARN negotiates resource requests from applications running in the cluster. YARN then provides processing capacity to each application by allocating Containers. I will go through container later.
To have a better understanding of what happened in hadoop version 2 by introducing Yarn, let’s have a look at following figure.
Yarn comes as the new layer between hdfs file-system and data processing applications. So in fact YARN took over this task of cluster resource management from MapReduce in hadoop version 1, and MapReduce is streamlined to perform Data Processing only in which it is best. So in hadoop version 2 that we have Yarn, MapReduce do not do any resource management anymore and only doing data processing for whom may need it. The MapReduce here is called MapReduce2 since it works with Yarn. As can be seen, there are also different data processing models possible here like Interactive, Online, Streaming and etc.
Let’s have a look at some of the main advantages of Yarn before I go through Yarn Design:
a. We can run different applications with different data processing model as I mentioned earlier.
b. Multiple applications can run on Hadoop via YARN and all application could share common resource management. So in contrast to previous resource management in hadoop v1, There are no more fixed map-reduce slots.
c. It is backward compatible which means existing MapReduce jobs still can run on Yarn without any changes.
YARN Design
Let’s understand Yarn one time for ever. In Yarn architecture we have two type of nodes, the node that Resource Manager daemon will be installed (usually is in same server as Namenode) and node(s) that Node Manager daemon (also called Yarn client) will be installed which are slave nodes.
As I said Yarn is doing resource management job in the cluster. So he must know all the available compute resources in all Slave nodes. And based on the negotiation that he do with each application running in the cluster, decide to allocate resources.
But how he does this resource allocation? Like how he allocate the amount of Ram, CPU,… that application needs? He has a measuring unit which is called Container.
What is Container:
It is a basic unit of processing capacity in Yarn. In a simple terms it is a package of resource elements (RAM, CPU,…). Please do not get confused with Container concepts like Docker here. The figure in the bottom shows a container which encapsulated with 8 GB ram and 2 core cpu.
The following figure shows the architecture of Yarn components. Resource manager is usually installed in a separate node from slave nodes. If you already read the Hadoop installation thread in my page, I have installed resource manager at the same node as namenode. And Node Manager are all other slave nodes which basically are doing the real jobs of computation.
There are several terms that needs to be understood in the Yarn architecture:
Resource Manager: It runs as a master daemon which decides how to devote resources to competing applications. Since resource manager is a single process that has all the information, it can take better decisions for resource allocation and scheduling based on different aspects like application priority, data locality and etc.
NodeManager: It is a per-node agent that is installed in slave nodes of the cluster. It has a number of dynamically created containers. The number of containers on a node is a product of configuration parameters and the total amount of node resources. It takes care of several tasks which mainly are :
-
monitoring resource usage of individual containers
-
containers’ life-cycle management
-
keeping up-to date with the ResourceManager
-
Monitoring node health
Application Master: I would say this is one of the main difference Yarn has with previous resource manager in Hadoop 1.0. Application Manager is a daemon which comes with application when user submiting the job to the cluster. It runs in one of the slave nodes and its life-cycle is as long as application is running on the cluster. So keep in mind that each application framework that user(s) use in the cluster (written to be used in hadoop 2.0), will have its own application manager daemon in one of the slave nodes (we call it per-application application master). As an example, when we are using MapReduce framework here, it must have a specific application master implemented within in order to be able use the Yarn. That is the reason MapReduce application which is used in Yarn is called MapReduce2 in order to distinguish it from MapReduce in Hadoop 1.0. The ApplicationMaster and tasks that belong to its application run in resource containers controlled by the NodeManagers in one of the slave nodes.
So in summary application master key characteristics are:
Scalibility:Since there is an instance of an ApplicationMaster per application, the ApplicationMaster itself isn’t a common bottleneck in the cluster.
Multi-functionality: Moving all application framework specific code into the ApplicationMaster generalizes the system so that we can now support multiple frameworks such as MapReduce, MPI and Graph Processing.
And the main responsibilities of Application Master are:
Responsible to calculate total needed values of application counters and negotiate it with ResourceManager and working with the NodeManager(s) to execute and monitor the containers and their resource consumption.
Keep updating resource manager with its status and the state of container needs.
Configuration
a. Nodemanager
We can go to one of the slave nodes and check the yarn processes related to nodemanager. When the service is up and running, we can easily check the process by
[yarn@node-01 ~]$ ps aux | grep -i nodemanager
and if there is a need we can stop it like following:
[yarn@node-01 ~]$ /usr/hdp/current/hadoop-yarn-nodemanager/sbin/yarn-daemon.sh –config /etc/hadoop/conf/ stop nodemanager
The configuration related to nodemanager is in $HADOOP_CONF_DIR directory which in our case is in /etc/hadoop/conf.
Starting the service is same as above and we just need to use start at above command.
b. ResourceManager
In the node that resourcemanager has been installed, in our case is in master node that is also namenode, we can first check the services related to resource manager like this:
[yarn@hadoop-master ~]$ ps aux | grep -i resourcemanager
and then we want to stop it we can do following
[yarn@hadoop-master ~]$ /usr/hdp/current/hadoop-yarn-resourcemanager/sbin/yarn-daemon.sh –config /etc/hadoop/conf/ stop resourcemanager
We can use the same for starting the service by using start keyword.
Tunning
I will cover this part soon completely, however there is a link in the bottom from hortonworks that quite well describe how to tune the Yarn in the cluster:
HPC, Big data & information security 