Hadoop2.6集群部署

2018-07-15 约 633 字预计阅读 2 分钟次阅读

上传hadoop和jdk安装包并解压

下载地址：

1
2


http://mirror.bit.edu.cn/apache/hadoop/common
http://www.oracle.com/technetwork/java/javase/downloads/

下载并解压

1
2
3
4
5


wget http://archive.apache.org/dist/hadoop/common/hadoop-2.6.0/hadoop-2.6.1.tar.gz
tar zxf jdk-8u172-linux-x64.tar.gz
mv jdk1.8.0_172 /usr/local/
tar zxf hadoop-2.6.0.tar.gz
mv hadoop-2.6.0 /usr/local/

添加环境变量

vi ~/.bashrc

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# add java env
JAVA_HOME=/usr/local/jdk1.8.0_172
JAVA_BIN=/usr/local/jdk1.8.0_172/bin
JRE_HOME=/usr/local/jdk1.8.0_172/jre
PATH=$PATH:/usr/local/jdk1.8.0_172/bin:/usr/local/jdk1.8.0_172/jre/bin
CLASSPATH=/usr/local/jdk1.8.0_172/jre/lib:/usr/local/jdk1.8.0_172/lib:/usr/local/jdk1.8.0_172/jre/lib/charsets.jar

# add hadoop env
HADOOP_HOME=/usr/local/src/hadoop-2.6.0
export PATH=$PATH:$HADOOP_HOME/bin

source ~/.bashrc && java -version

设置主机名

1
2


vi /etc/hostname
分别为master slave1 slave2

编辑hosts

1
2
3
4


cat /etc/hosts
192.168.233.220 master
192.168.233.221 slave1
192.168.233.222 slave2

拷贝jdk到slave1和slave2

1
2


scp -rp /usr/local/jdk1.8.0_172 192.168.233.221:/usr/local/
scp -rp /usr/local/jdk1.8.0_172 192.168.233.221:/usr/local/

添加登录免密码

三台都执行

1
2


ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >~/.ssh/authorized_keys

master

1
2


ssh slave1 cat ~/.ssh/id_rsa.pub >~/.ssh/authorized_keys
ssh slave2 cat ~/.ssh/id_rsa.pub >~/.ssh/authorized_keys

slave1

1
2


ssh master cat ~/.ssh/id_rsa.pub >~/.ssh/authorized_keys
ssh slave2 cat ~/.ssh/id_rsa.pub >~/.ssh/authorized_keys

slave2

1
2


ssh slave1 cat ~/.ssh/id_rsa.pub >~/.ssh/authorized_keys
ssh master cat ~/.ssh/id_rsa.pub >~/.ssh/authorized_keys

配置hadoop

master上操作

1
2
3
4


cd /usr/local/hadoop-2.6.0/etc/hadoop/
cat slaves 
slave1
slave2

cat core-site.xml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://192.168.233.220:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/usr/local/hadoop-2.6.0/tmp</value>
    </property>
</configuration>

cat hdfs-site.xml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


<configuration>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>master:9001</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/hadoop-2.6.0/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/hadoop-2.6.0/dfs/data</value>
    </property>
    <property>
        <name>dfs.repliction</name>
        <value>3</value>
    </property>
</configuration>

cp mapred-site.xml.template mapred-site.xml

vi mapred-site.xml

1
2
3
4
5
6


<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

cat yarn-site.xml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>master:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>master:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>master:8035</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>master:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>master:8088</value>
    </property>
</configuration>

配置

1
2
3
4
5
6
7


mkdir /usr/local/hadoop-2.6.0/tmp
mkdir -p /usr/local/hadoop-2.6.0/dfs/name
mkdir -p /usr/local/hadoop-2.6.0/dfs/data
vi hadoop-env.sh
export JAVA_HOME=/usr/local/jdk1.8.0_172
vi yarn-env.sh
export JAVA_HOME=/usr/local/jdk1.8.0_172

将master配置好的hadoop拷贝到slave1和slave2

scp -rp /usr/local/hadoop-2.6.0 root@slave1:/usr/local/hadoop-2.6.0
scp -rp /usr/local/hadoop-2.6.0 root@slave2:/usr/local/hadoop-2.6.0

master 初始化namenode

./bin/hadoop namenode -format

启动集群

./sbin/start-all.sh

集群状态

使用jps查看集群状态

master状态

1
2
3
4


2322 ResourceManager
2114 SecondaryNameNode
3113 Jps
2559 NameNode

slave1状态

1
2
3


2599 jps
3222 DataNode
2347 NodeManager

slave2状态

1
2
3


2385 DataNode
2450 NodeManager
2573 Jps

监控网页

访问192.168.233.220:8088/cluster/nodes

操作命令

1
2


bin/hadoop fs -put /var/log/yum.log/
bin/hadoop fs -ls /

关闭集群

./sbin/hadoop stop-all.sh

备注

jdk env和hadoop env的另外一种设置

1
2
3
4


export JAVA_HOME=/usr/local/src/jdk1.8.0_151
export HADOOP_HOME=/usr/local/src/hadoop-2.6.1
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin

目录