8.1 安裝scala
Step1~4 下載安裝 Scala
wget http://www.scala-lang.org/files/archive/scala-2.11.6.tgz tar xvf scala-2.11.6.tgz sudo mv scala-2.11.6 /usr/local/scalaStep5 Scala使用者環境變數設定
修改~/.bashrc
sudo gedit ~/.bashrc輸入下列內容
export SCALA_HOME=/usr/local/scala export PATH=$PATH:$SCALA_HOME/binStep6 使讓~/.bashrc修改生效
source ~/.bashrc8.2 安裝Spark
Step1~3 下載安裝 Spark
wget http://apache.stu.edu.tw/spark/spark-1.4.0/spark-1.4.0-bin-hadoop2.6.tgz tar zxf spark-1.4.0-bin-hadoop2.6.tgz sudo mv spark-1.4.0-bin-hadoop2.6 /usr/local/spark/Step4 Spark使用者環境變數設定
修改~/.bashrc
sudo gedit ~/.bashrc輸入下列內容
export SPARK_HOME=/usr/local/spark export PATH=$PATH:$SPARK_HOME/binStep5 使讓~/.bashrc修改生效
source ~/.bashrc
8.4 啟動spark-shell互動介面
spark-shell8.5 設定spark-shell 顯示訊息
cd /usr/local/spark/conf cp log4j.properties.template log4j.properties修改log4j.properties
sudo gedit log4j.properties開啟gedit編輯log4j.properties,原本是INFO改為WARN
8.6 啟動Hadoop
start-all.sh8.7 本機執行Spark-shell 程式
Step1 進入spark-shell
spark-shell --master local[4]Step2 讀取本機檔案
val textFile=sc.textFile("/usr/local/spark/README.md") textFile.countStep3 讀取HDFS檔案
val textFile = sc.textFile("hdfs://master:9000/user/hduser/wordcount/input/pg5000.txt") textFile.count8.8 在Hadoop YARN執行spark-shell
Step1 在Hadoop YARN執行spark-shell
SPARK_JAR=/usr/local/spark/lib/spark-assembly-1.4.0-hadoop2.6.0.jar HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop MASTER=yarn-client /usr/local/spark/bin/spark-shellStep2 讀取本機檔案
錯誤語法
val textFile=sc.textFile("/usr/local/spark/README.md") textFile.count正確語法
val textFile=sc.textFile("file:/usr/local/spark/README.md") textFile.countStep3 讀取HDFS檔案
val textFile = sc.textFile("/user/hduser/wordcount/input/pg5000.txt") val textFile = sc.textFile("hdfs://master:9000/user/hduser/wordcount/input/pg5000.txt") textFile.count
此圖出自Spark官網 https://spark.apache.org/
以上內容節錄自這本書,很適合Python程式設計師學習Spark機器學習與大數據架構,點選下列連結查看本書詳細介紹:
Python+Spark 2.0+Hadoop機器學習與大數據分析實戰
http://pythonsparkhadoop.blogspot.tw/2016/10/pythonspark-20hadoop.html
博客來網路書店: http://www.books.com.tw/products/0010730134?loc=P_007_090
天瓏網路書店: https://www.tenlong.com.tw/items/9864341537?item_id=1023658
露天拍賣:http://goods.ruten.com.tw/item/show?21640846068139
蝦皮拍賣:https://goo.gl/IEx13P
回覆刪除I have seen a lot of blogs and Info. on other Blogs and Web sites But in this Hadoop Blog Information is useful very thanks for sharing it........
Aws online training in india
回覆刪除Salesforce online training in india
SAS Online Training in india
Salesforce admin online training in india
Linux Online
training in India
React.JS online training in india
回覆刪除Python online Training in India
Oracle DBA online training in India
Java online Training in India
SAP online Training In india
Digital Marketing Online Training in India
回覆刪除Cloud Computing Online Training in India
Hadoop online training in INDIA
Javascript Training In Noida
Industrial Training in Noida
Linux Online training in India – Webtrackker Technology is providing the linux online training with 100% placement support. If you are looking for the BEST LINUX & UNIX Training Institute In india or linux online training from india, live project based LINUX & UNIX online training then you can contact to us.
回覆刪除Python online training in India, RPA Online training in India, Salesforce online training in india, AWS online training in india, Cloud Computing Online Training in India, SAS Online Training in india, Hadoop online training in INDIA, Oracle DBA online training in India, SAP online Training In india, Linux Online training in India
Its really nice and informative.. Thanks for sharing
回覆刪除Best Web Design Training Institutes in Noida
Best Hadoop Training Institutes In Noida
Best Digital Marketing Training Institute in Noida
Sap Training Institute in Noida
Best Java Training Institute in Noida
SAP SD Training Institute in Noida
Best Auto CAD Training Institute In Noida
Salesforce Training Institute in Noida
回覆刪除Salesforce Training in noida
Best Salesforce Training Institutes in Noida
Best Aws Training Institutes in Noida
best aws training in noida
aws training institute in noida
best data science training institute in delhi
python Training Institute in noida
sas Training Institute in noida
linux Training Institute in noida
servicenow scripting Training in Noida,
回覆刪除ServiceNow implementation Training in Noida
,
ServiceNow integration Training in Noida,
ServiceNow portal Training in Noida,
servicenow training institute in Noida
Best Servicenow Developer training institute in Noida
,
Best servicenow admin training institutes in Noida
作者已經移除這則留言。
回覆刪除6-week summer course in Noida - 6 weeks The summer course plays a crucial role in shaping the career of young aspiring / informatics students. This training has been specifically introduced so that students can become familiar with current industrial culture and industrial needs. Webtrackker technology offers a 6-month training program for students / graduates that includes small and large projects.
回覆刪除6-week summer course in Noida
Ajio Promo Codes
回覆刪除Ajio Coupons & Offers
Ajio Coupon codes
Ajio Offers Promo Codes
Ajio Offers on online shopping
Firstcry Promo Codes
Firstcry Deals & offers
Firstcry coupons codes
Firstcry Coupons Offers Promo Codes
Firstcry Offers on Kids shopping
Myntra promo codes
Myntra deals and offers
Myntra coupon codes
Myntra coupons offers promo codes
Myntra offers on online shopping
Nykaa Promo Codes
Nykaa Deals and offers
Nykaa Coupons codes
Nykaa coupons offers promo codes
Nykaa offers on online shopping
Flipkart promo codes
Flipkart deals & coupons
flipkart coupon code
flipkart coupons offer promo code
Amazon promo code
amazon offers
amazon offers and deals
amazon coupon code
回覆刪除amazon deal of the day
cleartrip promo codes
cleartrip coupon code
cleartrip offers and deals
cleartrip deals
MMT promo Codes
MMT coupon codes
Makemytrip promo codes
makemytrip offers
makemytrip deals & offers
healthkart coupon code
healthkart promo codes
healthkart deals and offers
healthkart discount offers
bigbasket promo codes
bigbasket coupon codes
回覆刪除bigbasket offers
bigbasket coupon and deals
pizzahut promo code
pizzahut coupon codes
pizzahut offers
pizzahut coupon and offers
hotels promo code
hotels coupon codes
hotel offers & deals
hotels discount offers
nearbuy coupon codes
nearbuy promo codes
nearbuy deals and offers
nearbuy discounts
zoomcar promo code
zoomcar coupon code
zoomcar offers on ride
zoomcar deals and offers
Each department of CAD have specific programmes which, while completed could provide you with a recognisable qualification that could assist you get a job in anything design enterprise which you would really like.
回覆刪除AutoCAD training in Noida
AutoCAD training institute in Noida
Best AutoCAD training institute in Noida
Video editing course in Noida
回覆刪除Video editing training institute in Noida- Webtrackker Technology is and IT Training institute providing the Video editing course in Noida, FCP, Final Cut Pro Training in Noida. For more call us- 8802820025.
Video editing course in Noida
Company Address:
Webtrackker Technology
C- 67, Sector- 63, Noida
Phone: 01204330760, 8802820025
Email: info@webtrackker.com
Website: http://webtrackker.com/Best-training-institute-Video-editing-FCP-course-in-Noida.php
Cloud Computing Training In Noida
回覆刪除Webtrackker is IT based company in many countries. Webtrackker will provide you a real time projects based training on Cloud Computing. If you are looking for the Cloud computing training in Noida then you can join the webtrackker technology.
Cloud Computing Training In Noida , Cloud Computing Training center In Noida , Cloud Computing Training institute In Noida ,
Company Address:
Webtrackker Technology
C- 67, Sector- 63, Noida
Email: info@webtrackker.com
Website: www.webtrackker.com
http://webtrackker.com/Cloud-Computing-Training-Institutes-In-Noida.php
rpa training institute in noida
回覆刪除Blockchain training institute in Noida
thank you your post was knowlable to me
回覆刪除rpa training institute in noida
sas training institute in noida
hadoop training institute in noida
blokchain traninig institut noida
servicenow scripting Training in Noida
回覆刪除rpa training institute in noida
sas training institute in noida
hadoop training institute in noida
blokchain traninig institute noida
click here for more information about the latest stuff oneplus c type jack
回覆刪除Like different elements of India, numerous oil and spices usually cross into making food. This effects in substances getting caught to the partitions of the filter out.
回覆刪除Visit here
http://kitchenware.ml
Best kitchen chimney installation and service
Auto clean chimney sevice in Faridabad
A model is a person with a function either to sell, show or market it industrial products (appreciably style garb in style indicates), or to serve as a visual useful resource for individuals who are creating works of artwork or to pose for images +919910294610.
回覆刪除Arvind Rawat model in Noida
Visit here
Arvind Rawat Model in Noida
Microsoft Azure Training institute in Noida,
回覆刪除AWS Training Institute in Noida sector 16,
Cloud Computing Training Institute in noida sector 16,
Data science training institute in noida sector 16,
Data Science With machine learning training Institute in Noida sector 16,
Data Science With python training Institute in Noida sector 16,
Web-designing Training Institute in Noida sector 16,
software-testing Training Institute in Noida sector 16,
Digital Marketing Training Institute in noida sector 16,
hadoop Training Institute in noida sector 16,
回覆刪除Java Training Institute in noida sector 16,
linux Training Institute in noida sector 16,
node.js Training Institute in noida sector 16,
openstack Training Institute in noida sector 16,
Oracle DBA Training Institute in noida sector 16,
Php Training Institute in noida sector 16,
PlSql Training Institute in Noida sector 16,
Python Training Institute in Noida sector 16,
回覆刪除RPA Training Institute in Noida sector 16,
Salesforce Training Institute in Noida sector 16,
Sap fico Training Institute in Noida sector 16,
ERP Sap mm Training Institute in Noida Sector 16,
Sap Training Institute in Noida Sector 16,
SAS Training Institute in Noida Sector 16,
Blue Prism Training Institute in Noida,