8.1 安裝scala
Step1~4 下載安裝 Scala
wget http://www.scala-lang.org/files/archive/scala-2.11.6.tgz tar xvf scala-2.11.6.tgz sudo mv scala-2.11.6 /usr/local/scalaStep5 Scala使用者環境變數設定
修改~/.bashrc
sudo gedit ~/.bashrc輸入下列內容
export SCALA_HOME=/usr/local/scala export PATH=$PATH:$SCALA_HOME/binStep6 使讓~/.bashrc修改生效
source ~/.bashrc8.2 安裝Spark
Step1~3 下載安裝 Spark
wget http://apache.stu.edu.tw/spark/spark-1.4.0/spark-1.4.0-bin-hadoop2.6.tgz tar zxf spark-1.4.0-bin-hadoop2.6.tgz sudo mv spark-1.4.0-bin-hadoop2.6 /usr/local/spark/Step4 Spark使用者環境變數設定
修改~/.bashrc
sudo gedit ~/.bashrc輸入下列內容
export SPARK_HOME=/usr/local/spark export PATH=$PATH:$SPARK_HOME/binStep5 使讓~/.bashrc修改生效
source ~/.bashrc
8.4 啟動spark-shell互動介面
spark-shell8.5 設定spark-shell 顯示訊息
cd /usr/local/spark/conf cp log4j.properties.template log4j.properties修改log4j.properties
sudo gedit log4j.properties開啟gedit編輯log4j.properties,原本是INFO改為WARN
8.6 啟動Hadoop
start-all.sh8.7 本機執行Spark-shell 程式
Step1 進入spark-shell
spark-shell --master local[4]Step2 讀取本機檔案
val textFile=sc.textFile("/usr/local/spark/README.md") textFile.countStep3 讀取HDFS檔案
val textFile = sc.textFile("hdfs://master:9000/user/hduser/wordcount/input/pg5000.txt") textFile.count8.8 在Hadoop YARN執行spark-shell
Step1 在Hadoop YARN執行spark-shell
SPARK_JAR=/usr/local/spark/lib/spark-assembly-1.4.0-hadoop2.6.0.jar HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop MASTER=yarn-client /usr/local/spark/bin/spark-shellStep2 讀取本機檔案
錯誤語法
val textFile=sc.textFile("/usr/local/spark/README.md") textFile.count正確語法
val textFile=sc.textFile("file:/usr/local/spark/README.md") textFile.countStep3 讀取HDFS檔案
val textFile = sc.textFile("/user/hduser/wordcount/input/pg5000.txt") val textFile = sc.textFile("hdfs://master:9000/user/hduser/wordcount/input/pg5000.txt") textFile.count
此圖出自Spark官網 https://spark.apache.org/
以上內容節錄自這本書,很適合Python程式設計師學習Spark機器學習與大數據架構,點選下列連結查看本書詳細介紹:
Python+Spark 2.0+Hadoop機器學習與大數據分析實戰
http://pythonsparkhadoop.blogspot.tw/2016/10/pythonspark-20hadoop.html
博客來網路書店: http://www.books.com.tw/products/0010730134?loc=P_007_090
天瓏網路書店: https://www.tenlong.com.tw/items/9864341537?item_id=1023658
露天拍賣:http://goods.ruten.com.tw/item/show?21640846068139
蝦皮拍賣:https://goo.gl/IEx13P
Aws online training in india
回覆刪除Salesforce online training in india
SAS Online Training in india
Salesforce admin online training in india
Linux Online
training in India
作者已經移除這則留言。
回覆刪除Ajio Promo Codes
回覆刪除Ajio Coupons & Offers
Ajio Coupon codes
Ajio Offers Promo Codes
Ajio Offers on online shopping
Firstcry Promo Codes
Firstcry Deals & offers
Firstcry coupons codes
Firstcry Coupons Offers Promo Codes
Firstcry Offers on Kids shopping
Myntra promo codes
Myntra deals and offers
Myntra coupon codes
Myntra coupons offers promo codes
Myntra offers on online shopping
Nykaa Promo Codes
Nykaa Deals and offers
Nykaa Coupons codes
Nykaa coupons offers promo codes
Nykaa offers on online shopping
Flipkart promo codes
Flipkart deals & coupons
flipkart coupon code
flipkart coupons offer promo code
Amazon promo code
amazon offers
amazon offers and deals
amazon coupon code
回覆刪除amazon deal of the day
cleartrip promo codes
cleartrip coupon code
cleartrip offers and deals
cleartrip deals
MMT promo Codes
MMT coupon codes
Makemytrip promo codes
makemytrip offers
makemytrip deals & offers
healthkart coupon code
healthkart promo codes
healthkart deals and offers
healthkart discount offers
bigbasket promo codes
Cloud Computing Training In Noida
回覆刪除Webtrackker is IT based company in many countries. Webtrackker will provide you a real time projects based training on Cloud Computing. If you are looking for the Cloud computing training in Noida then you can join the webtrackker technology.
Cloud Computing Training In Noida , Cloud Computing Training center In Noida , Cloud Computing Training institute In Noida ,
Company Address:
Webtrackker Technology
C- 67, Sector- 63, Noida
Email: info@webtrackker.com
Website: www.webtrackker.com
http://webtrackker.com/Cloud-Computing-Training-Institutes-In-Noida.php
rpa training institute in noida
回覆刪除Blockchain training institute in Noida
click here for more information about the latest stuff oneplus c type jack
回覆刪除hadoop Training Institute in noida sector 16,
回覆刪除Java Training Institute in noida sector 16,
linux Training Institute in noida sector 16,
node.js Training Institute in noida sector 16,
openstack Training Institute in noida sector 16,
Oracle DBA Training Institute in noida sector 16,
Php Training Institute in noida sector 16,
PlSql Training Institute in Noida sector 16,
Are you looking for Distance Learning Courses in India most of the students choose and apply, Talentedgenex there are many popular courses which attract the students for having distance education. For more info visit this site:- Distance learning courses in India ,
回覆刪除Talentedgenext Way of Online Learning, Distance Education, is an increasing number of becoming popular all over the world due as it has many benefits. For further details visit in this site:- Distance Education Website,
回覆刪除Terrific article! This is the type of information that should be shared across the web. BA 2nd Year Result
回覆刪除I am really happy to say it’s an interesting post to read . I learn new information from your article , you are doing a great job . Keep it up.
回覆刪除This piece of writing posted at this ba 3rd year ka time table web site is genuinely fastidious.
回覆刪除