文章目录
  1. 1. 预装环境变量
  2. 2. 配置 SSH
  3. 3. 安装及配置 Hadoop
    1. 3.1. 安装 Hadoop
    2. 3.2. 配置伪分布式 Hadoop
      1. 3.2.1. 配置 hadoop-env.sh
      2. 3.2.2. 配置 yarn-env.sh
      3. 3.2.3. 配置 core-site.xml
      4. 3.2.4. 配置 hdfs-core.xml
      5. 3.2.5. 配置 mapred-site.xml
      6. 3.2.6. 配置 yarn-site.xml
    3. 3.3. 格式化 HDFS
    4. 3.4. 启动
    5. 3.5. 可视化查看
  • 转载
  • 预装环境变量

    配置 SSH

    为了保证远程登录管理 Hadoop 及 Hadoop 节点用户共享的安全性,Hadoop 需要配置使用 SSH 协议

    • 打开系统偏好设置-共享-远程登录-允许访问-所有用户

    • 打开<终端>,分别输入

    1
    2
    ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
    cat ~/.ssh/id_dsa.pub >>~/.ssh/authorized_keys
    • 配置好之后,输入 ssh localhost, 登录成功即配置完成

    安装及配置 Hadoop

    安装 Hadoop

    • <终端>输入
      1
      brew install hadoop

    配置伪分布式 Hadoop


    配置 hadoop-env.sh

    1
    open /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/hadoop-env.sh

    1
    export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"

    修改为

    1
    export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc="

    配置 yarn-env.sh

    1
    oopen /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/yarn-env.sh

    添加

    1
    YARN_OPTS="$YARN_OPTS -Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"

    配置 core-site.xml

    1
    open /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/core-site.xml

    修改为:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    <?xml version="1.0" encoding="UTF-8"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <!--
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License. See accompanying LICENSE file.
    -->

    <!-- Put site-specific property overrides in this file. -->
    <configuration>
    <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
    </property>
    </configuration>

    配置 hdfs-core.xml

    1
    open /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/hdfs-site.xml

    配置为:

    1
    2
    3
    4
    5
    6
    <configuration>
    <property>
    <name>dfs.replication</name>
    <value>1</value>
    </property>
    </configuration>

    配置 mapred-site.xml

    1
    2
    cp /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/mapred-site.xml.template /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/mapred-site.xml
    open /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/mapred-site.xml

    配置为:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    <configuration>
    <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
    </property>
    <property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx4096m</value>
    </property>
    </configuration>

    配置 yarn-site.xml

    1
    open /usr/local/Cellar/hadoop/2.7.3/libexec/etc/hadoop/yarn-site.xml

    配置为:

    1
    2
    3
    4
    5
    6
    7
    8
    <configuration>

    <!-- Site specific YARN configuration properties -->
    <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
    </property>
    </configuration>

    格式化 HDFS

    1
    2
    rm -rf /tmp/hadoop-tanjiti
    hadoop namenode -format

    启动

    • 启动 HDFS

      1
      /usr/local/Cellar/hadoop/2.7.3/sbin/start-dfs.sh
    • 启动 MapReduce

      1
      /usr/local/Cellar/hadoop/2.7.3/sbin/start-yarn.sh
    • 检查启动情况

      1
      jps

    结果

    1
    2
    3
    4
    5
    6
    7
    8
    9
    1536
    6594 NameNode
    6818 SecondaryNameNode
    65478 Jps
    6694 DataNode
    6950 ResourceManager
    7051 NodeManager
    1982 Launcher
    15903 GradleDaemon

    • 运行 MapReduce 自带实例
      1
      hadoop jar /usr/local/Cellar/hadoop/2.7.3/libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar pi 2 5

    可视化查看

    转载


    文章若有纰漏请大家补充指正,谢谢~~

    http://blog.xinshangshangxin.com SHANG殇

    文章目录
    1. 1. 预装环境变量
    2. 2. 配置 SSH
    3. 3. 安装及配置 Hadoop
      1. 3.1. 安装 Hadoop
      2. 3.2. 配置伪分布式 Hadoop
        1. 3.2.1. 配置 hadoop-env.sh
        2. 3.2.2. 配置 yarn-env.sh
        3. 3.2.3. 配置 core-site.xml
        4. 3.2.4. 配置 hdfs-core.xml
        5. 3.2.5. 配置 mapred-site.xml
        6. 3.2.6. 配置 yarn-site.xml
      3. 3.3. 格式化 HDFS
      4. 3.4. 启动
      5. 3.5. 可视化查看
  • 转载