1、創(chuàng)建Headless Service服務(wù)

Headless 服務(wù)類型并不分配容器云虛擬 IP，而是直接暴露所屬 Pod 的 DNS 記錄。沒有默認負載均衡器，可直接訪問 Pod IP 地址。因此，當我們需要與集群內(nèi)真實的 Pod IP 地址進行直接交互時，Headless 服務(wù)就很有用。
其中Service的關(guān)鍵配置如下：clusterIP: None，不讓其獲取clusterIP ， DNS解析的時候直接走pod。

---
kind: Service
apiVersion: v1
metadata:
  name: ecc-spark-service
  namespace: ecc-spark-cluster
spec:
  clusterIP: None
  ports:
    - port: 7077
      protocol: TCP
      targetPort: 7077
      name: spark
    - port: 10000
      protocol: TCP
      targetPort: 10000
      name: thrift-server-tcp
    - port: 8080
      targetPort: 8080
      name: http
    - port: 45970
      protocol: TCP
      targetPort: 45970
      name: thrift-server-driver-tcp  
    - port: 45980
      protocol: TCP
      targetPort: 45980
      name: thrift-server-blockmanager-tcp    
    - port: 4040
      protocol: TCP
      targetPort: 4040
      name: thrift-server-tasks-tcp              
  selector:
    app: ecc-spark-service

EOF

Service的完全域名： ecc-spark-service.ecc-spark-cluster.svc.cluster.local
headless service的完全域名: headless-service.ecc-spark-cluster.svc.cluster.local
在容器里面ping 完全域名， service解析出的地址是clusterIP，headless service 解析出來的地址是 pod IP。

2、構(gòu)建spark集群

2.1 、創(chuàng)建spark master

spark master分為兩個部分，一個是類型為ReplicationController的主體，命名為ecc-spark-master.yaml，另一部分為一個service，暴露master的7077端口給slave使用。

#如下是把thriftserver部署在master節(jié)點，則需要暴露thriftserver端口、driver端口、
#blockmanager端口服務(wù),以提供worker節(jié)點executor與driver交互.
cat >ecc-spark-master.yaml <<EOF
kind: Deployment
apiVersion: apps/v1
metadata:
  name: ecc-spark-master
  namespace: ecc-spark-cluster
  labels:
    app: ecc-spark-master
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ecc-spark-master
  template:
    metadata:
      labels:
        app: ecc-spark-master
    spec:
      serviceAccountName: spark-cdp
      securityContext: {}
      dnsPolicy: ClusterFirst
      hostname: ecc-spark-master
      containers:
        - name: ecc-spark-master
          image: spark:3.4.1
          imagePullPolicy: IfNotPresent
          command: ["/bin/sh"]
          args: ["-c","sh /opt/spark/sbin/start-master.sh && tail -f /opt/spark/logs/spark--org.apache.spark.deploy.master.Master-1-*"]
          ports:
            - containerPort: 7077
            - containerPort: 8080
          volumeMounts:
            - mountPath: /opt/usrjars/
              name: ecc-spark-pvc
          livenessProbe:
            failureThreshold: 9
            initialDelaySeconds: 2
            periodSeconds: 15
            successThreshold: 1
            tcpSocket:
              port: 8080
            timeoutSeconds: 10
          resources:
            requests:
              cpu: "2"
              memory: "6Gi"
            limits:
              cpu: "2"
              memory: "6Gi"
         - env:
            - SPARK_LOCAL_DIRS
              value: "/odsdata/sparkdirs/"             
      volumes:
        - name: ecc-spark-pvc
          persistentVolumeClaim:
            claimName: ecc-spark-pvc-static

2.2、創(chuàng)建spark worker

在啟動spark worker腳本中需要傳入master的地址，在容器云kubernetes dns且設(shè)置了service的緣故，可以通過ecc-spark-master.ecc-spark-cluster.svc.cluster.local:7077訪問。

cat >ecc-spark-worker.yaml <<EOF
kind: Deployment
apiVersion: apps/v1
metadata:
  name: ecc-spark-worker
  namespace: ecc-spark-cluster
  labels:
    app: ecc-spark-worker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ecc-spark-worker
  template:
    metadata:
      labels:
        app: ecc-spark-worker
    spec:
      serviceAccountName: spark-cdp
      securityContext: {}
      dnsPolicy: ClusterFirst
      hostname: ecc-spark-worker
      containers:
        - name: ecc-spark-worker
          image: spark:3.4.1
          imagePullPolicy: IfNotPresent
          command: ["/bin/sh"]
          args: ["-c","sh /opt/spark/sbin/start-worker.sh spark://ecc-spark-master.ecc-spark-cluster.svc.cluster.local:7077;tail -f /opt/spark/logs/spark--org.apache.spark.deploy.worker.Worker*"]
          ports:
            - containerPort: 8081
          volumeMounts:
            - mountPath: /opt/usrjars/
              name: ecc-spark-pvc
          resources:
            requests:
              cpu: "2"
              memory: "2Gi"
            limits:
              cpu: "2"
              memory: "4Gi"
        - env:
            - SPARK_LOCAL_DIRS
              value: "/odsdata/sparkdirs/"              
      volumes:
        - name: ecc-spark-pvc
          persistentVolumeClaim:
            claimName: ecc-spark-pvc-static

EOF

2.3 構(gòu)建pyspark提交環(huán)境

import json
import flask
from flask import Flask
from concurrent.futures import ThreadPoolExecutor

app = Flask(__name__)
pool = ThreadPoolExecutor(max_workers=8)

@app.route('/')
def hello_world():  # put application's code here
    return 'Hello World!'

@app.route('/downloadCode', methods=['post'])
def download_file():
    model_id = flask.request.json.get('modelId')
    print(model_id)
    """
    異步提交任務(wù)：pool.submit()
    """
    return json.dumps(0, ensure_ascii=False)

@app.route('/modelRun', methods=['post'])
def model_run():
    """
    異步提交任務(wù)：pool.submit()
    """
    return json.dumps(0, ensure_ascii=False)

if __name__ == '__main__':
    app.run()

spark@c67e6477b2f1:/opt/spark$ python3
Python 3.8.10 (default, May 26 2023, 14:05:08) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> 

將python的調(diào)用整合到:start-master.sh 文件末尾啟動調(diào)用，便可以通過k8s暴露spark-master的F5端口實現(xiàn)http調(diào)用。

3、使用spark-operator安裝spark集群方式

可以參考阿里云文章：搭建Spark應(yīng)用文章來源地址http://www.zghlxwxcb.cn/news/detail-738765.html

到了這里，關(guān)于基于Headless構(gòu)建高可用spark+pyspark集群的文章就介紹完了。如果您還想了解更多內(nèi)容，請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！