Menu

Install and Configure Apache Hadoop on CentOS/RHEL 7/6/5 and Ubuntu

Apache Hadoop is an open-source software framework for distributed storage and distributed processing of very large data sets on computer clusters built from commodity hardware. All the modules in Hadoop are designed with a fundamental assumption that hardware failures are common and should be automatically handled by the framework.

The core of Apache Hadoop consists of a storage part, known as Hadoop Distributed File System (HDFS), and a processing part called MapReduce. Hadoop splits files into large blocks and distributes them across nodes in a cluster. To process data, Hadoop transfers packaged code for nodes to process in parallel based on the data that needs to be processed. This approach takes advantage of data locality— nodes manipulating the data they have access to— to allow the dataset to be processed faster and more efficiently than it would be in a more conventional supercomputer architecture that relies on a parallel file system where computation and data are distributed via high-speed networking
Installing Java

Java is the primary requirement for running hadoop on any system, So make sure you have Java installed on your system using following command.
# java -version 
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
Creating Hadoop User

We recommend to create a normal (nor root) account for hadoop working. So create a system account using following command.
# adduser hadoop
# passwd hadoop
After creating account, it also required to set up key based ssh to its own account. To do this use execute following commands.
# su - hadoop
$ ssh-keygen -t rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
Lets verify key based login. Below command should not ask for password but first time it will prompt for adding RSA to the list of known hosts.
$ ssh localhost
$ exit
Downloading Hadoop

Now download hadoop 2.6.0 source archive file using below command. You can also select alternate download mirror for increasing download speed.
$ cd ~
$ wget http://apache.claz.org/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz
$ tar xzf hadoop-2.6.0.tar.gz
$ mv hadoop-2.6.0 hadoop
Configure Hadoop Pseudo-Distributed Mode
First we need to set environment variable uses by hadoop. Edit ~/.bashrc file and append following values at end of file.
export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
Now apply the changes in current running environment
$ source ~/.bashrc
Now edit $HADOOP_HOME/etc/hadoop/hadoop-env.sh file and set JAVA_HOME environment variable. Change the JAVA path as per install on your system.
export JAVA_HOME=/opt/jdk1.8.0_91/
Edit Configuration Files
Hadoop has many of configuration files, which need to configure as per requirements of your hadoop infrastructure.
$ cd $HADOOP_HOME/etc/hadoop
Edit core-site.xml
<configuration>
<property>
  <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
</property>
</configuration>
Edit hdfs-site.xml
<configuration>
<property>
 <name>dfs.replication</name>
 <value>1</value>
</property>

<property>
  <name>dfs.name.dir</name>
    <value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>

<property>
  <name>dfs.data.dir</name>
    <value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>
</configuration>
Edit mapred-site.xml
<configuration>
 <property>
  <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>
</configuration>
Edit yarn-site.xml
<configuration>
 <property>
  <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
 </property>
</configuration>
Format Namenode
Now format the namenode using following command, make sure that Storage directory is
$ hdfs namenode -format
Output Bellow : -
 = 1000
16/05/11 11:11:11 INFO namenode.FSNamesystem: fsOwner             = hadoop (auth:SIMPLE)
16/05/11 11:11:11 INFO namenode.FSNamesystem: supergroup          = supergroup
16/05/11 11:11:11 INFO namenode.FSNamesystem: isPermissionEnabled = true
16/05/11 11:11:11 INFO namenode.FSNamesystem: HA Enabled: false
16/05/11 11:11:11 INFO namenode.FSNamesystem: Append Enabled: true
16/05/11 11:11:11 INFO util.GSet: Computing capacity for map INodeMap
16/05/11 11:11:11 INFO util.GSet: VM type       = 64-bit
16/05/11 11:11:11 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
16/05/11 11:11:11 INFO util.GSet: capacity      = 2^20 = 1048576 entries
16/05/11 11:11:11 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/05/11 11:11:11 INFO util.GSet: Computing capacity for map cachedBlocks
16/05/11 11:11:11 INFO util.GSet: VM type       = 64-bit
16/05/11 11:11:11 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
16/05/11 11:11:11 INFO util.GSet: capacity      = 2^18 = 262144 entries
16/05/11 11:11:11 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
16/05/11 11:11:11 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
16/05/11 11:11:11 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
16/05/11 11:11:11 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
16/05/11 11:11:11 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
16/05/11 11:11:11 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/05/11 11:11:11 INFO util.GSet: VM type       = 64-bit
16/05/11 11:11:11 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
16/05/11 11:11:11 INFO util.GSet: capacity      = 2^15 = 32768 entries
16/05/11 11:11:11 INFO namenode.NNConf: ACLs enabled? false
16/05/11 11:11:11 INFO namenode.NNConf: XAttrs enabled? true
16/05/11 11:11:11 INFO namenode.NNConf: Maximum size of an xattr: 16384
16/05/11 11:11:11 INFO namenode.FSImage: Allocated new BlockPoolId: BP-427269761-172.31.47.17-1462965071490
16/05/11 11:11:11 INFO common.Storage: Storage directory /home/hadoop/hadoopdata/hdfs/namenode has been successfully formatted.
16/05/11 11:11:11 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/05/11 11:11:11 INFO util.ExitUtil: Exiting with status 0
16/05/11 11:11:11 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ip-172-31-47-17.us-west-2.compute.internal/172.31.47.17
************************************************************/
Start Hadoop Cluster

Lets start your hadoop cluster using the scripts provides by hadoop. Just navigate to your hadoop sbin directory and execute scripts one by one.
$ cd $HADOOP_HOME/sbin/
Now run start-dfs.sh script.
$ start-dfs.sh
Output:-
hadoop@ip-172-31-47-17:~/hadoop/sbin$ start-dfs.sh
16/05/11 11:14:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-namenode-ip-172-31-47-17.out
localhost: starting datanode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-datanode-ip-172-31-47-17.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-secondarynamenode-ip-172-31-47-17.out
16/05/11 11:14:52 WARN util.NativeCodeLoader: Unable to load native-ha
Now run start-yarn.sh script.
$ start-yarn.sh
Output:-
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-resourcemanager-ip-172-31-47-17.out
localhost: starting nodemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-ip-172-31-47-17.out
Access Hadoop Services in Browser

Hadoop NameNode started on port 50070 default. Access your server on port 50070 in your favorite web browser.
Now access port 8088 for getting the information about cluster and all applications
Access port 50090 for getting details about secondary namenode.
Access port 50075 to get details about DataNode
http://server.hackthesec.co.in:50075/
Test Hadoop Single Node Setup

Make the HDFS directories required using following commands.
$ bin/hdfs dfs -mkdir /user
$ bin/hdfs dfs -mkdir /user/hadoop
Now copy all files from local file system /var/log/httpd to hadoop distributed file system using below command
$ bin/hdfs dfs -put /var/log/httpd logs
Now browse hadoop distributed file system by opening below url in browser.
 http://server.hackthesec.co.in:50070/explorer.html#/user/hadoop/logs
Now copy logs directory for hadoop distributed file system to local file system.
$ bin/hdfs dfs -get logs /tmp/logs
$ ls -l /tmp/logs/
www.hackthesec.co.in

About Author:


I am a Linux Administrator and Security Expert with this site i can help lot's of people about linux knowladge and as per security expert i also intersted about hacking related news.TwitterFacebook

Next
Newer Post
Previous
Older Post

0 comments:

Post a Comment

 
Top