openshift 使用curl命令訪問apiserver

openshift版本：openshift v3.6.173.0.5

使用oc(同kubectl)命令訪問apiserver資源的時候，會使用到/root/.kube/config文件中使用的配置。

使用user訪問apiserver

oc命令使用config中定義的user和證書(公鑰和私鑰)訪問apiserver。使用如下命令查看當前使用的config上下文：monitor為當前的namespace，test-openshfit-com:8443為apiserver暴露的server，system:admin為訪問apiserver使用的user名稱

# oc config current-context
monitor/test-openshfit-com:8443/system:admin

查看system:admin對應的證書(下面使用變量代替)

users:
- name: system:admin/test-openshift-com:8443
  user:
    client-certificate-data: ${CA}
    client-key-data: ${KEY}

導出證書，將下面decode出的內容分別保存到/home/ca.cert，/home/ca.key

# echo -n ${CA}|base64 --decode   #/home/ca.cert
# echo -n ${KEY}|base64 --decode  #/home/ca.key

使用如下方式即可訪問cluster范圍內的資源，該方式與oc命令的原理一樣。下面以訪問servers為例

APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
curl $APISERVER/api/v1/services --cert /home/ca.cert --key /home/ca.key --user system:admin

 

使用serviceaccount訪問apiserver

serviceaccount除了可以為pod提供secret外，還可以作為訪問apiserver資源的憑證。使用如下命令創建一個名為curltest的serviceaccount，並獲取其token

oc create serviceaccount curltest
APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
TOKEN=$(oc serviceaccounts get-token curltest)

使用如下命令進行rolebinding之后就可以查看namespaces為monitor下面的資源，但不可以查看其他namespace的資源。system:master可以看作一個超級賬戶，可參見user-facing-roles

oc policy add-role-to-user system:master -z curltest
curl $APISERVER/api/v1/namespaces/monitor/services --header "Authorization: Bearer $TOKEN"

使用下面命令查看當前rolebinding情況，可以查看當前serveraccount可進行的操作權限。注：openshift的add-role-to-user/add-cluster-role-to-user其實就是kubernetes進行rolebinding/clusterrolebinding的操作，將一個role權限賦予一個user或serviceaccount。

# oc describe rolebinding system:master
Name:                   system:master
Namespace:              monitor
Created:                5 minutes ago
Labels:                 <none>
Annotations:            <none>
Role:                   /system:master
Users:                  <none>
Groups:                 <none>
ServiceAccounts:        curltest
Subjects:               <none>
Verbs                   Non-Resource URLs       Resource Names  API Groups      Resources
[*]                     []                      []              [*]             [*]
[*]                     [*]                     []              []              []

使用如下命令進行clusterrolebinding之后就可以訪問cluster范圍內的資源，首先需要刪除先前的rolebinding

oc policy remove-role-from-user system:master -z curltest
oadm policy add-cluster-role-to-user system:master -z curltest
TOKEN=$(oc serviceaccounts get-token curltest)
curl $APISERVER/api/v1/services --header "Authorization: Bearer $TOKEN"

查看clusterrolebinding情況

# oc describe clusterrolebinding system:master
Name:                   system:masters
Created:                2 weeks ago
Labels:                 <none>
Annotations:            <none>
Role:                   /system:master
Users:                  <none>
Groups:                 system:masters
ServiceAccounts:        monitor/curltest
Subjects:               <none>
Verbs                   Non-Resource URLs       Resource Names  API Groups      Resources
[*]                     []                      []              [*]             [*]
[*]                     [*]                     []              []              []

環境清理

oadm policy remove-cluster-role-from-user system:master -z liu
oc delete sa curltest

 

下面演示pod如何使用serviceaccount訪問apiserver資源，參照在Kubernetes Pod中使用Service Account訪問API Server

首先安裝minikube和go，方法可以參見https://www.cnblogs.com/charlieroro/p/10434138.html。minikube啟動時直接使用docker驅動即可：minikube start --vm-driver=none

對client-go的操作步驟用於生成測試鏡像，可以直接下載已經打包好的鏡像(docker pull docker push woodliu268/k8s-example)來跳過下面相關操作

安裝client-go，client使用了go module方式來管理包依賴(client-go根目錄下使用go.mod和go.sum來管理包)，參見Installing client-go

export GO111MODULE=on
go mod init
go get k8s.io/client-go@master

修改client-go/examples/in-cluster-client-configuration/main.go目錄下，將panic全部修改為fmt.Println，執行如下命令編譯為可執行程序main

go build -o main main.go

Dockerfile內容如下，編譯為docker鏡像

FROM debian
COPY main /root/main
RUN chmod +x /root/main
WORKDIR /root
ENTRYPOINT ["/root/main"]

docker build -t k8s/example1:latest .

使用如下deployment創建pod，默認創建的default命名空間

# cat deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: k8s-example
spec:
  replicas: 1
  template:
    metadata:
      labels:
        run: k8s-example
    spec:
      containers:
      - name: k8s-example
        image: k8s/example1:latest
        imagePullPolicy: IfNotPresent

kubectl log -f k8s-example-7747697dbf-772df時發現有如下錯誤。說明pod使用用戶system:serviceaccount:default:default訪問apiserver的時候訪問失敗

pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope
There are 0 pods in the cluster

由於需要在cluster范圍內訪問pod資源，下面創建clusterrole和clusterrolebinding(參考Using RBAC Authorization)，並賦予system:serviceaccount:default:default list pod的權限

# cat clusterrole.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

# cat clusterrolebinding.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: User
  name: system:serviceaccount:default:default
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

重新創建deployment，查看pod日志，可以正常讀取cluster的pod信息

There are 10 pods in the cluster

 

PS：

• 使用kubectl get RESOURECE -v=NUM可以查看kubectl的與apiserver的交互，RESOURECE為pod，service等；NUM取值為6-8
• 使用oc config use-context可以設置kubeconfig文件中的current-context字段
• Service account 驗證時用戶名 system:serviceaccount:(NAMESPACE):(SERVICEACCOUNT)，被指定到組 system:serviceaccounts 和 system:serviceaccounts:(NAMESPACE)
• 應用程序可能會在如yaml模板中使用serviceaccount掛載到pod中的tls證書來訪問apiserver資源

參考：

https://kubernetes.io/docs/reference/access-authn-authz/rbac/

https://docs.openshift.com/container-platform/3.5/rest_api/index.html

https://docs.openshift.com/container-platform/3.9/admin_guide/manage_rbac.html

https://docs.openshift.com/enterprise/3.0/admin_guide/manage_authorization_policy.html

https://jimmysong.io/posts/user-authentication-in-kubernetes/