MapReduce Flow in YARN

How a MapReduce job runs in YARN is different from how it used to run in MRv1. This post shows how MapReduce v2 runs internally in YARN Hadoop.

MapReduce flow - Submitting MapReduce job

First step is of course submitting the job in order to kick start the process.

For submitting the job you can use one of the following methods of the org.apache.hadoop.mapreduce.Job class-

void submit()- Submit the job to the cluster and return immediately.
boolean waitForCompletion(boolean)- Submit the job to the cluster and wait for it to finish.

When a job is submitted using one of the above mentioned methods, Job class creates an instance of JobSubmitter instance and submitJobInternal() method is called on that instance and following steps are taken.

Getting a new application ID from the resource manager for the MapReduce job.
Checking the input and output specifications of the job.
Computing the InputSplits for the job.
Setup the requisite accounting information for the org.apache.hadoop.mapreduce.filecache.DistributedCache of the job, if necessary.
Copying the job's jar and configuration to the map-reduce system directory on the distributed file-system.
Submit the job using the SubmitApplication method in YARNClient. SubmitApplication method submits a new application to YARN. It is a blocking call, it will not return ApplicationId until the submitted application is submitted successfully and accepted by the ResourceManager.

Important YARN components for running MapReduce job

Main components when running a MapReduce job in YARN are–

Client- An application submission client that submits an application.
ResourceManager- Daemon that manages the cluster resources.
ApplicationMaster- communicates with the ResourceManager to negotiate and allocate resources for future containers to run the map and reduce tasks for the submitted job.
NodeManager- Launches and monitor the resources used by the containers that run the mappers and reducers for the job. NodeManager daemon runs on each node in the cluster.

Interaction among these components is shown here-

Client<-->ResourceManager - By using YarnClient objects.

ApplicationMaster<-->ResourceManager - By using AMRMClientAsync objects, handling events asynchronously by AMRMClientAsync.CallbackHandler

ApplicationMaster<-->NodeManager - Launch containers. Communicate with NodeManagers by using NMClientAsync objects, handling container events by NMClientAsync.CallbackHandler.

Running tasks for the submitted MapReduce job

Once the job is submitted to the ResourceManager, initially a single container is negotiated for executing the application specific ApplicationMaster (Which is MRAppMaster in case of MapReduce applications). The YARN ResourceManager will then launch the ApplicationMaster on the allocated container.

Once launched ApplicationMaster performs the following tasks-

Communicate with the ResourceManager to negotiate and allocate resources for containers required to run mappers and reducers for the submitted MapReduce job.
After the containers are allocated, communicate with YARN NodeManagers to launch application containers on the nodes where the containers are allocated.
Track the progress of the tasks running on the containers.

How ApplicationMaster runs the Map and Reduce tasks

ApplicationMaster retrieves the number of input splits calculated for the job at the time of submission. While running a MapReduce job as many map tasks are created as the count of input splits and the number of reducers is calculated using the mapreduce.job.reduces property which sets the default number of reduce tasks per job.

After knowing the number of mappers and reducers required for the job ApplicationMaster has to decide should it run the tasks sequentially in the same JVM where ApplicationMaster itself is running. If it does that, it is known as running the tasks in uber mode in Hadoop.

If the job is not run as an uber task then ApplicationMaster has to negotiate with ResourceManager to get resource containers to run those map and reduce tasks.

In the resource requests to ResourceManager memory requirements and CPUs for tasks are also specified. Values used for determining memory and CPU requirements for the map and reduce tasks are in mapred-site.xml configuration file.

mapreduce.map.memory.mb- The amount of memory to request from the scheduler for each map task. Default value is 1024 MB.
mapreduce.map.cpu.vcores– The number of virtual cores to request from the scheduler for each map task. Default value is 1.
mapreduce.reduce.memory.mb– The amount of memory to request from the scheduler for each reduce task. Default value is 1024 MB.
mapreduce.reduce.cpu.vcores– The number of virtual cores to request from the scheduler for each reduce task. Default value is 1.

ResourceManager’s scheduler will allocate these containers on different nodes in the Hadoop cluster. Reduce tasks can be assigned containers on any node with no locality constraint. For map tasks scheduler tries to allocate containers on the nodes where the split data resides for data locality optimization.

Once the containers are allocated, ApplicationMaster launch those containers on the nodes by contacting the NodeManagers of those nodes. The ApplicationMaster executes the mappers and reducers in a separate jvm on the launched containers.

MapReduce flow - Task progress and completion

The running map and reduce tasks report their progress every three seconds to ApplicationMaster which can create overall job progress from the updates from these separate tasks.

Client also receives the current status from the ApplicationMaster.

ApplicationMaster also emit heartbeats to the ResourceManager to keep it informed that it is alive and still running.

If a task fails to complete, the ApplicationMaster will reschedule that task on another node in the cluster.

When all the map and reduce tasks for the job are completed ApplicationMaster changes the job status to successful. After that ApplicationMaster unregisters itself and then stops the client.

MapReduce flow in YARN

Reference- https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html

That's all for this topic MapReduce Flow in YARN. If you have any doubt or any suggestions to make please drop a comment. Thanks!

>>>Return to Hadoop Framework Tutorial Page

Related Topics

You may also like-

Tech Tutorials

Saturday, December 16, 2023