Spark学习之路(二):Spark伪分布式安装

下载 Spark 安装包

官网下载 http://spark.apache.org/downloads.html

安装前准备

  • Java8 已安装
  • hadoop2.7.5 已安装

修改 Hadoop 配置文件

修改 Hadoop yarn-site.xml配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
vim ~/App/hadoop-2.7.3/etc/hadoop/yarn-site.xml

<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://bigdata:19888/jobhistory/logs</value>
</property>
<property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>

重启yarn服务

1
2
stop-yarn.sh 
start-yarn.sh

启动 MapReduce History Server

1
sbin/mr-jobhistory-daemon.sh start historyserver

在浏览器中打开 MapReduce history server 地址

http://bigdata:19888

Spark 安装、配置

1.解压缩 spark-2.1.0-bin-hadoop2.7.tgz

1
tar -zxvf spark-2.1.1-bin-hadoop2.7.tar -C ~/App

2.进入 conf 配置文件目录,修改 spark-env.sh

1
2
3
4
5
6
7
cd ~/App/spark-2.1.1-bin-hadoop2.7/conf
cp spark-env.sh.template spark-env.sh
vim spark-env.sh

export HADOOP_CONF_DIR=~/App/hadoop-2.7.3/etc/hadoop
export HIVE_CONF_DIR=~/App/apache-hive-2.1.1-bin/conf
export SPARK_DIST_CLASSPATH=$(~/App/hadoop-2.7.3/bin/hadoop classpath)

3.进入 conf 配置文件目录,修改 spark-defaults.conf

1
2
3
4
5
6
7
8
9
cp spark-defaults.conf.template spark-defaults.conf
vim spark-defaults.conf

spark.master=local
spark.yarn.historyServer.address=bigdata:18080
spark.history.ui.port=18080
spark.eventLog.enabled=true
spark.eventLog.dir=hdfs:///tmp/spark/events
spark.history.fs.logDirectory=hdfs:///tmp/spark/events

4.在 hdfs 上创建目录 /tmp/spark/events

1
hadoop fs –mkdir –p /tmp/spark/events

配置环境变量

1
2
3
4
5
6
vim ~/.bash_profile

export SPARK_HOME=/Users/baihe/App/spark-2.1.1-bin-hadoop2.7
export PATH=$SPARK_HOME/bin:$PATH

source ~/.bash_profile

启动

  1. 启动 Hdfs start-dfs.sh
  2. 启动 Spark ~/App/spark-2.1.1-bin-hadoop2.7/sbin/start-all.sh
  3. 启动 Spark History Server ~/App/spark-2.1.1-bin-hadoop2.7/sbin/start-history-server.sh

查看进程

1
2
3
4
5
6
7
8
9
10
➜  spark-2.1.1-bin-hadoop2.7 jps
9008 NameNode
9185 SecondaryNameNode
10484 HistoryServer
10215 Master
10281 Worker
5514
10506 Jps
9323 ResourceManager
9407 NodeManager

查看web界面

http://bigdata:8080/

spark history server

评论