docker-在Kubernetes集群上搭建Hadoop集群
【摘要】 准备工作Hadoop镜像,到docker hub上拉取docker pull kubeguide/hadoop:latestKubernetes集群参考:Kubernetes-离线部署Kubernetes 1.9.0开始搭建编写好hadoop.yamlhadoop.yamlapiVersion: v1kind: ConfigMapmetadata: name: kube-hadoop-co...
准备工作
Hadoop镜像,到docker hub上拉取
docker pull kubeguide/hadoop:latest
Kubernetes集群
参考:Kubernetes-离线部署Kubernetes 1.9.0
开始搭建
编写好hadoop.yaml
hadoop.yaml
apiVersion: v1 kind: ConfigMap metadata: name: kube-hadoop-conf namespace: hadoop data: HDFS_MASTER_SERVICE: hadoop-hdfs-master HDOOP_YARN_MASTER: hadoop-yarn-master --- apiVersion: v1 kind: Service metadata: name: hadoop-hdfs-master spec: type: NodePort selector: name: hdfs-master ports: - name: rpc port: 9000 targetPort: 9000 - name: http port: 50070 targetPort: 50070 nodePort: 32007 --- apiVersion: v1 kind: ReplicationController metadata: name: hdfs-master labels: name: hdfs-master spec: replicas: 1 selector: name: hdfs-master template: metadata: labels: name: hdfs-master spec: containers: - name: hdfs-master image: kubeguide/hadoop:latest imagePullPolicy: IfNotPresent ports: - containerPort: 9000 - containerPort: 50070 env: - name: HADOOP_NODE_TYPE value: namenode - name: HDFS_MASTER_SERVICE valueFrom: configMapKeyRef: name: kube-hadoop-conf key: HDFS_MASTER_SERVICE - name: HDOOP_YARN_MASTER valueFrom: configMapKeyRef: name: kube-hadoop-conf key: HDOOP_YARN_MASTER restartPolicy: Always --- apiVersion: v1 kind: ReplicationController metadata: name: hadoop-datanode labels: app: hadoop-datanode spec: replicas: 3 selector: name: hadoop-datanode template: metadata: labels: name: hadoop-datanode spec: containers: - name: hadoop-datanode image: kubeguide/hadoop:latest imagePullPolicy: IfNotPresent ports: - containerPort: 9000 - containerPort: 50070 env: - name: HADOOP_NODE_TYPE value: datanode - name: HDFS_MASTER_SERVICE valueFrom: configMapKeyRef: name: kube-hadoop-conf key: HDFS_MASTER_SERVICE - name: HDOOP_YARN_MASTER valueFrom: configMapKeyRef: name: kube-hadoop-conf key: HDOOP_YARN_MASTER restartPolicy: Always --- apiVersion: v1 kind: Service metadata: name: hadoop-yarn-master spec: type: NodePort selector: name: yarn-master ports: - name: "8030" port: 8030 - name: "8031" port: 8031 - name: "8032" port: 8032 - name: http port: 8088 targetPort: 8088 nodePort: 32088 --- apiVersion: v1 kind: ReplicationController metadata: name: yarn-master labels: name: yarn-master spec: replicas: 1 selector: name: yarn-master template: metadata: labels: name: yarn-master spec: containers: - name: yarn-master image: kubeguide/hadoop:latest imagePullPolicy: IfNotPresent ports: - containerPort: 9000 - containerPort: 50070 env: - name: HADOOP_NODE_TYPE value: resourceman - name: HDFS_MASTER_SERVICE valueFrom: configMapKeyRef: name: kube-hadoop-conf key: HDFS_MASTER_SERVICE - name: HDOOP_YARN_MASTER valueFrom: configMapKeyRef: name: kube-hadoop-conf key: HDOOP_YARN_MASTER restartPolicy: Always --- apiVersion: v1 kind: Service metadata: name: yarn-node spec: clusterIP: None selector: name: yarn-node ports: - port: 8040 --- apiVersion: v1 kind: ReplicationController metadata: name: yarn-node labels: name: yarn-node spec: replicas: 3 selector: name: yarn-node template: metadata: labels: name: yarn-node spec: containers: - name: yarn-node image: kubeguide/hadoop:latest imagePullPolicy: IfNotPresent ports: - containerPort: 8040 - containerPort: 8041 - containerPort: 8042 env: - name: HADOOP_NODE_TYPE value: yarnnode - name: HDFS_MASTER_SERVICE valueFrom: configMapKeyRef: name: kube-hadoop-conf key: HDFS_MASTER_SERVICE - name: HDOOP_YARN_MASTER valueFrom: configMapKeyRef: name: kube-hadoop-conf key: HDOOP_YARN_MASTER restartPolicy: Always
这个yaml文件包含一个ConfigMap,5个Service,8个pod,这里需要注意的是ConfigMap中HDFS_MASTER_SERVICE和HDOOP_YARN_MASTER不要使用IP,使用HDFS service的名称,否则datanode将会连接不上namenode,出现错误【ipc.Client: Retrying connect to server: xxx:9000.】
执行创建命令
kubectl create -f hadoop.yaml
create
检查是否创建成功
查看config map
kubectl get configmap -o wide
config map
查看service
kubectl get svc -o wide
service
查看pod
kubectl get po -o wide
pod
通过浏览器访问HDFS管理界面【http://192.168.242.136:32007】
overview
查看datanode
datanodes
全部正常,搭建成功!
创建
创建完成后就可以通过浏览器看到熟悉的HDFS管理界面了
HDFS
image.png
【版权声明】本文为华为云社区用户原创内容,转载时必须标注文章的来源(华为云社区)、文章链接、文章作者等基本信息, 否则作者和本社区有权追究责任。如果您发现本社区中有涉嫌抄袭的内容,欢迎发送邮件进行举报,并提供相关证据,一经查实,本社区将立刻删除涉嫌侵权内容,举报邮箱:
cloudbbs@huaweicloud.com
- 点赞
- 收藏
- 关注作者
评论(0)