How to Set-up Hadoop Cluster

What is Hadoop Cluster and Big-data ?

Hadoop Cluster on AWS

Set-up Hadoop Cluster ?

Set-up Name Node on AWS Cloud and Data Node on Local VM

How to Launch Instance on AWS Cloud ?

How to Launch EC2 Instance on AWS Cloud

In NamNode -

To set-up hadoop cluster , we have to download two softwares — 1st JDK because Hadoop is internally configured in JAVA and second hadoop Software . So , I have downloaded both the s/w in my base OS and transfer these software to EC2 Instance using winscp Software .

Next , I have transfer these software to Root user .

Then , Install Jdk and hadoop using

rpm -ivh jdk-8u171-linux-x64.rpm — To Install JDK

rpm -ivh hadoop-1.2.1–1.x86_64.rpm — — force — To Install Hadoop

Create one directory and then format it to use it as a Namenode .

Then

cd /etc/hadoop/

In hdfs-site.xml file , we have to write one property

<?xml version=”1.0"?>
<?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>

<! — Put site-specific property overrides in this file. →

<configuration>
<property>
<name>dfs.name.dir</name>
<value>/nn</value>
</property>
</configuration>

In core-site.xml

<?xml version=”1.0"?>
<?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>

<! — Put site-specific property overrides in this file. →

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://0.0.0.0:9001</value>
</property>
</configuration>

To start the Namenode -

-> hadoop-daemon.sh start namenode

To check whether it’s started on not , we have to use jps command .

Here , No data node is connected . So we have to configure data node and attach it to Name node .

In Data Node , again we have to install jdk and hadoop software .

In Data Node —

First , we have to make one directory

• mkdir /datanode

Then ,write property in hdfs-site.xml and core-site.xml as given in figure

and then start the datanode .

• hadoop-daemon.sh start datanode

Now , Again if we check DataNode is connected to Namenode and we can check using this command -

• hadoop dfsadmin -report

Now , Namenode and Data Node is Connected . Similarly , We can attach many datanode to Namenode .

Thank you :)