转载并进行适当修改

预装环境变量

配置 SSH

为了保证远程登录管理 Hadoop 及 Hadoop 节点用户共享的安全性，Hadoop 需要配置使用 SSH 协议

  • 打开系统偏好设置-共享-远程登录-允许访问-所有用户

  • 打开<终端>，分别输入 

1
2
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >>~/.ssh/authorized_keys
  • 配置好之后，输入 ssh localhost, 登录成功即配置完成

安装及配置 Hadoop

安装 Hadoop

  • <终端>输入 
    1
    brew install hadoop

配置伪分布式 Hadoop


配置 hadoop-env.sh

1
open /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/hadoop-env.sh

将 

1
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"

修改为

1
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc="

配置 yarn-env.sh

1
oopen /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/yarn-env.sh

添加

1
YARN_OPTS="$YARN_OPTS -Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"

配置 core-site.xml

1
open /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/core-site.xml

修改为:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->
<configuration>
  <property>  
    <name>fs.defaultFS</name>             
    <value>hdfs://localhost:9000</value>          
  </property>
</configuration>

配置 hdfs-core.xml

1
open /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/hdfs-site.xml

配置为:

1
2
3
4
5
6
<configuration>
  <property>
       <name>dfs.replication</name>
       <value>1</value>
  </property>
</configuration>

配置 mapred-site.xml

1
2
cp /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/mapred-site.xml.template /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/mapred-site.xml
open /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/mapred-site.xml

配置为:

1
2
3
4
5
6
7
8
9
10
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx4096m</value>
  </property>
</configuration>

配置 yarn-site.xml

1
open /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/yarn-site.xml

配置为:

1
2
3
4
5
6
7
8
<configuration>

<!-- Site specific YARN configuration properties -->
 <property>
     <name>yarn.nodemanager.aux-services</name>
     <value>mapreduce_shuffle</value>
 </property>
</configuration>

格式化 HDFS

1
2
rm -rf /tmp/hadoop-tanjiti
hadoop namenode -format

启动

  • 启动 HDFS 

    1
    /usr/local/Cellar/hadoop/2.7.3/sbin/start-dfs.sh

  • 启动 MapReduce

    1
    /usr/local/Cellar/hadoop/2.7.3/sbin/start-yarn.sh

  • 检查启动情况 

    1
    jps

结果

1
2
3
4
5
6
7
8
9
1536
6594 NameNode
6818 SecondaryNameNode
65478 Jps
6694 DataNode
6950 ResourceManager
7051 NodeManager
1982 Launcher
15903 GradleDaemon

  • 运行 MapReduce 自带实例 
    1
    hadoop jar /usr/local/Cellar/hadoop/2.7.3/libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar pi 2 5

可视化查看

转载

文章若有纰漏请大家补充指正,谢谢~~

http://blog.xinshangshangxin.com SHANG殇