柚子快報(bào)邀請(qǐng)碼778899分享：Hadoop完全分布式集群搭建

http://yzkb.51969.com/

配置hadoop集群所需要的準(zhǔn)備：a.三臺(tái)虛擬機(jī)，這里我們分別命名為master,slave1,slave2；b.jdk安裝包，Hadoop集群需要在jdk環(huán)境下運(yùn)行，3.hadoop安裝包。

開始搭建（這里我們自己從第二步開始，就是解壓jdk，Hadoop將h3cu下的jdk,hadoop解壓到/usr/local/src/目錄下）

[root@master /]#tar -zxvf /h3cu/jdk-8u161-linux-x64.tar.gz -C /usr/local/src/

[root@master /]#tar -zxvf /h3cu/hadoop-3.1.3.tar.gz -C /usr/local/src/

3進(jìn)入/usr/local/src/目錄進(jìn)行改名

[root@master src]# mv jdk1.8.0_161/ jdk

[root@master src]# mv hadoop-3.1.3/ hadoop

配置環(huán)境變量(根據(jù)要求選擇/etc/profile 或者/root/.bash_profile)

[root@master src]# vi /etc/profile

export JAVA_HOME=/usr/local/src/jdk

export PATH=$PATH:$JAVA_HOME/bin

export HADOOP_HOME=/usr/local/src/hadoop

export HDFS_NAMENODE_USER=root

export HDFS_SECONDARYNAMENODE_USER=root

export HDFS_DATANODE_USER=root

export YARN_RESOURCEMANAGER_USER=root

export YARN_NODEMANAGER_USER=root

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

[root@master src]# source /etc/profile

生成密鑰，分發(fā)密鑰（每臺(tái)虛擬機(jī)操作一次，執(zhí)行3遍）

[root@master src]# ssh-keygen -t rsa #連按3次回車

[root@master src]# ssh-copy-id master

[root@master src]# ssh-copy-id slave1

[root@master src]# ssh-copy-id slave2

分發(fā)環(huán)境變量配置文件到slave1，slave2

[root@master src]# scp /etc/profile slave1:/etc/

[root@master src]# scp /etc/profile slave2:/etc/

在slave1，slave2上使用命令：source /etc/profile使環(huán)境生效

6.分發(fā)jdk到slave1，slave2

[root@master src]# scp -r /usr/local/src/jdk/ slave1:/usr/local/src/

[root@master src]# scp -r /usr/local/src/jdk/ slave2:/usr/local/src/

在其中兩臺(tái)虛擬機(jī)上使用java -version命令出現(xiàn)一下便證明jdk分發(fā)成功

[root@slave1 /]# java -version

java version "1.8.0_161"

Java(TM) SE Runtime Environment (build 1.8.0_161-b12)

Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

8.進(jìn)入/hadoop/etc/hadoop

[root@master src]# cd /usr/local/src/hadoop/etc/hadoop/

9.開始編輯配置文件

[root@master hadoop]# vi hadoop-env.sh

# The java implementation to use. By default, this environment

# variable is REQUIRED on ALL platforms except OS X!

export JAVA_HOME=/usr/local/src/jdk

# Location of Hadoop. By default, Hadoop will attempt to determine

# this location based upon its execution path.

[root@master hadoop]# vi core-site.xml

fs.defaultFS

hdfs://master:9000

hadoop.tmp.dir

/usr/local/src/hadoop/tmp

[root@master hadoop]# vi hdfs-site.xml

dfs.replication

1

[root@master hadoop]# vi mapred-site.xml

mapreduce.framework.name

yarn

[root@master hadoop]# vi yarn-site.xml

yarn.resourcemanager.hostname

master

yarn.nodemanager.aux-services

mapreduce_shuffle

yarn.nodemanager.pmem-check-enabled

false

yarn.nodemanager.vmem-check-enabled

false

[root@master hadoop]# vi workers

master

slave1

slave2

10.分發(fā)hadoop到slave1，slave2

[root@master hadoop]# scp -r /usr/local/src/hadoop/ slave1:/usr/local/src/

[root@master hadoop]# scp -r /usr/local/src/hadoop/ slave2:/usr/local/src/

11.分發(fā)完成進(jìn)行格式化（在master上）

[root@master hadoop]# hdfs namenode -format

12.啟動(dòng)Hadoop集群

[root@master hadoop]# start-all.sh

13.最后查看節(jié)點(diǎn)

[root@master hadoop]# jps

1696 SecondaryNameNode

1413 DataNode

1238 NameNode

2550 Jps

2185 NodeManager

2015 ResourceManager

[root@slave1 /]# jps

609 NodeManager

725 Jps

493 DataNode

[root@slave2 /]# jps

418 DataNode

534 NodeManager

670 Jps

柚子快報(bào)邀請(qǐng)碼778899分享：Hadoop完全分布式集群搭建

http://yzkb.51969.com/

參考文章