Hadoop is the solution for the big data. there are some frameworks that are used by the Hadoop.
Hadoop uses MapReduce: a programming model for data processing.
Data Processing: operations on the data, especially by a computer to retrieve,transform or classify information.
Batch Processing: as Map Reduce is a offline Batch processing. Execution of a series of program's(jobs) on a computer without any manual intervention. Jobs are set up so they cab be run to completion without human interaction. All input parameters are predefined through scripts, command-line arguments control files or job control language.
Hadoop uses parallel processing of data.
Map Reduce: there are two phases in map reduce. the map phase and reduce phase. each phase accepts input and gives output. the following are the steps:
1. The data is split into pieces( of block size) and then feeded to the map function as input , these inputs are called as input splits or simply inputs here the map task performs the mapping functionality and gives the output , as there may many number of splits, they has to combined so combined function are used and the output of the map function becomes the input of the reducer function. here after, the reducer reduces the input and produces the output. this output is stored in the HDFS.
Mapper class and Reducer class has four parameters: input key, input value, output key and output value. Note that the input key and input value of the reduce class should as same type to that of the output key and output value of the output of map function.
MapReduce has three important aspects:
1. Input data
2. MapReduce Program
3. Configuration information.
the jobs of the map reduce is divided into two tasks they are:
1. map task and 2. reduce task.
these tasks are scheduled using YARN(yet another resource negotiator) and runs on the node in the cluster. if a task fails, it will be automatically rescheduled to run on the different nodes. here hadoop divides the input into fixed size blocks called input splits or just splits. hadoop creates one map task each for split whcih runs the user defined map function for each record in the split.
Flow:
1. input is divided into small pieces (say some size, default is 128 MB) and supplied to map reduce job.
2. Hadoop creates map tasks for each input split such that parallel processing is done.
3. then the splits are combined and given as output(combining is a optional one) through the combining function.
4. this output is sent to reduce function where it reduces the data.
5. this is all done in nodes.
Data Locality Optimization:
Hadoop does its best to run the map tasks on a node where the input data resides in the HDFS, because it doesn't use valuable cluster bandwidth.
Split size should be equal to block size.(to remove overhead and dominance).
Some points to be remembered:
1. Map task writes their output to the local disk, not to HDFS.
2. Becaus ethe output is intermediate output.This intermediate output is the input for reduce tasks which produces final output. after this, final output is stored in the HDFS file.
3. for each reduce task of HDFS block of the reduce output , the first replica is stored on the local nodes with other replicas being stored on the off-rack nodes for reliability.
MapReduce can be with single reduce task, can be with multiple reduce tasks, can be with no reduce task.
Hadoop uses MapReduce: a programming model for data processing.
Data Processing: operations on the data, especially by a computer to retrieve,transform or classify information.
Batch Processing: as Map Reduce is a offline Batch processing. Execution of a series of program's(jobs) on a computer without any manual intervention. Jobs are set up so they cab be run to completion without human interaction. All input parameters are predefined through scripts, command-line arguments control files or job control language.
Hadoop uses parallel processing of data.
Map Reduce: there are two phases in map reduce. the map phase and reduce phase. each phase accepts input and gives output. the following are the steps:
1. The data is split into pieces( of block size) and then feeded to the map function as input , these inputs are called as input splits or simply inputs here the map task performs the mapping functionality and gives the output , as there may many number of splits, they has to combined so combined function are used and the output of the map function becomes the input of the reducer function. here after, the reducer reduces the input and produces the output. this output is stored in the HDFS.
Mapper class and Reducer class has four parameters: input key, input value, output key and output value. Note that the input key and input value of the reduce class should as same type to that of the output key and output value of the output of map function.
MapReduce has three important aspects:
1. Input data
2. MapReduce Program
3. Configuration information.
the jobs of the map reduce is divided into two tasks they are:
1. map task and 2. reduce task.
these tasks are scheduled using YARN(yet another resource negotiator) and runs on the node in the cluster. if a task fails, it will be automatically rescheduled to run on the different nodes. here hadoop divides the input into fixed size blocks called input splits or just splits. hadoop creates one map task each for split whcih runs the user defined map function for each record in the split.
Flow:
1. input is divided into small pieces (say some size, default is 128 MB) and supplied to map reduce job.
2. Hadoop creates map tasks for each input split such that parallel processing is done.
3. then the splits are combined and given as output(combining is a optional one) through the combining function.
4. this output is sent to reduce function where it reduces the data.
5. this is all done in nodes.
Data Locality Optimization:
Hadoop does its best to run the map tasks on a node where the input data resides in the HDFS, because it doesn't use valuable cluster bandwidth.
Split size should be equal to block size.(to remove overhead and dominance).
Some points to be remembered:
1. Map task writes their output to the local disk, not to HDFS.
2. Becaus ethe output is intermediate output.This intermediate output is the input for reduce tasks which produces final output. after this, final output is stored in the HDFS file.
3. for each reduce task of HDFS block of the reduce output , the first replica is stored on the local nodes with other replicas being stored on the off-rack nodes for reliability.
MapReduce can be with single reduce task, can be with multiple reduce tasks, can be with no reduce task.
