docker 搭建spark集群


借鑒:https://blog.csdn.net/u013705066/article/details/80030732
下面這個例子讓我感受到docker的便捷和強大。
兩行命令搭建spark集群:

1.准備一個docker-compose.yml配置文件,描述集群結構及必要參數
version: "2"
services:
  master:
    image: singularities/spark
    command: start-spark master
    hostname: master
    ports:
      - "6066:6066"
      - "7070:7070"
      - "8080:8080"
      - "50070:50070"
  worker:
    image: singularities/spark
    command: start-spark worker master
    environment:
      SPARK_WORKER_CORES: 1
      SPARK_WORKER_MEMORY: 2g
    links:
      - master

 

2.執行命令拉取spark鏡像到本地(該步驟可省略):

docker pull singularities/spark

 

3.在配置文件所在目錄啟動

docker-compose up -d

 

4.查看啟動結果



 

 瀏覽器訪問:http:/{docker_host}:50070

 











附docker基本命令:

docker build -t friendlyhello . # Create image using this directory's Dockerfile docker run -p 4000:80 friendlyhello # Run "friendlyhello" mapping port 4000 to 80 docker run -d -p 4000:80 friendlyhello # Same thing, but in detached mode docker container ls # List all running containers docker container ls -a # List all containers, even those not running docker container stop <hash> # Gracefully stop the specified container docker container kill <hash> # Force shutdown of the specified container docker container rm <hash> # Remove specified container from this machine docker container rm $(docker container ls -a -q) # Remove all containers docker image ls -a # List all images on this machine docker image rm <image id> # Remove specified image from this machine docker image rm $(docker image ls -a -q) # Remove all images from this machine docker login # Log in this CLI session using your Docker credentials docker tag <image> username/repository:tag # Tag <image> for upload to registry docker push username/repository:tag # Upload tagged image to registry docker run username/repository:tag # Run image from a registry

拷貝自官網.


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM