K8S实用整理(13)-创建支持ssh的pod-deployment-service.yaml

镜像为redhat5.9，安装配置有sshd。

depolyment+service.yaml如下，其中command参数为

    command:
    - '/usr/sbin/sshd'
    - -D

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: sie-ssh
  namespace: default
spec:
#  replicas: 3
  template:
    metadata:
      annotations:
        security.alpha.kubernetes.io/unsafe-sysctls: kernel.msgmnb=13107200,kernel.msgmni=256,kernel.msgmax=65536,kernel.shmmax=69719476736,kernel.sem=500 256000 250 1024
      labels:
        app: test
    spec:
      nodeSelector:
        cslckind1: test
      containers:
      - name: sie-ssh
        image: 172.28.2.2:4000/sie:20180112
        command:
        - '/usr/sbin/sshd'
        - -D
        ports:
        - containerPort: 22

---
apiVersion: v1
kind: Service
metadata:
  name: siessh-svc
  namespace: default
spec:
  selector:
    run: sie-ssh
  ports:
  - protocol: TCP
    port: 30032
    targetPort: 22

 

创建应用和service，检查结果：

daweij@master:~/stady01/ipctest$ kubectl apply -f ssh-sie.yml 
deployment "sie-ssh" created
service "siessh-svc" created
daweij@master:~/stady01/ipctest$ kubectl get service -o wide
NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE       SELECTOR
siessh-svc            ClusterIP   10.233.33.217   <none>        30032/TCP        10s       app=sie-test

daweij@master:~/stady01/ipctest$ kubectl get deployment -o wide
NAME                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINERS            IMAGES                                       SELECTOR
sie-ssh               1         1         1            1           20s       sie-ssh               172.28.2.2:4000/sie:20180112                 app=sie-test

daweij@master:~/stady01/ipctest$ kubectl get pod -o wide
NAME                                   READY     STATUS    RESTARTS   AGE       IP               NODE
sie-ssh-7cb4f57ddc-cgqbh               1/1       Running   0          49s       10.233.102.169   node1

daweij@master:~/stady01/ipctest$ ssh root@10.233.102.169
The authenticity of host '10.233.102.169 (10.233.102.169)' can't be established.
RSA key fingerprint is SHA256:wn36xY7Zpidyya2JJazfiSZ3oJBvXKQ9qMA9QuBnxrE.
Are you sure you want to continue connecting (yes/no)? no
Host key verification failed.

daweij@master:~/stady01/ipctest$ ssh root@172.28.2.211 -p 30032
ssh: connect to host 172.28.2.211 port 30032: Connection refused

daweij@master:~/stady01/ipctest$ kubectl get pod -o wide --show-labels 
NAME                                   READY     STATUS    RESTARTS   AGE       IP               NODE      LABELS
sie-ssh-7cb4f57ddc-cgqbh               1/1       Running   0          1m        10.233.102.169   node1     app=sie-test,pod-template-hash=3760913887

 

ssh可直连podid -p 22,

daweij@master:~/stady01/ipctest$ ssh root@10.233.102.169
The authenticity of host '10.233.102.169 (10.233.102.169)' can't be established.
RSA key fingerprint is SHA256:wn36xY7Zpidyya2JJazfiSZ3oJBvXKQ9qMA9QuBnxrE.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.233.102.169' (RSA) to the list of known hosts.
root@10.233.102.169's password: 
Last login: Thu Dec 22 16:04:36 from 172.20.18.70

[root@sie-ssh-7cb4f57ddc-cgqbh ~]# sysctl -a |grep -E 'kernel.msgmnb|kernel.msgmni|kernel.msgmax|kernel.shmmax|kernel.sem'
kernel.msgmax = 65536
kernel.msgmnb = 13107200
kernel.msgmni = 256
error: "Operation not permitted" reading key "kernel.unprivileged_userns_apparmor_policy"
kernel.sem = 500    256000    250    1024
kernel.sem_next_id = -1
kernel.shmmax = 69719476736
error: "Input/output error" reading key "net.ipv6.conf.all.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.default.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.eth0.stable_secret"
error: "Input/output error" reading key "net.ipv6.conf.lo.stable_secret"

 

问题来了：ssh不可连nodeip  -p 30032

why？

root@node1:~# iptables-save | grep 30032

root@node1:~# iptables-save | grep 30032
-A KUBE-SERVICES ! -s 10.233.64.0/18 -d 10.233.33.217/32 -p tcp -m comment --comment "default/siessh-svc: cluster IP" -m tcp --dport 30032 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.233.33.217/32 -p tcp -m comment --comment "default/siessh-svc: cluster IP" -m tcp --dport 30032 -j KUBE-SVC-AM2NPFNJ3LQRTONC

 

root@node1:~# iptables-save | grep 10.233.102.169  #（pod ip）
-A KUBE-SEP-EOYN3JTEJCZTK37F -s 10.233.102.169/32 -m comment --comment "default/siessh-svc:" -j KUBE-MARK-MASQ
-A KUBE-SEP-EOYN3JTEJCZTK37F -p tcp -m comment --comment "default/siessh-svc:" -m tcp -j DNAT --to-destination 10.233.102.169:22

原因：service上述使用的ClusterIP模式，修改为NodePort模式

参见http://www.cnblogs.com/DaweiJ/articles/8527625.html

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: sie-ssh
  namespace: default
spec:
#  replicas: 3
  template:
    metadata:
      annotations:
        security.alpha.kubernetes.io/unsafe-sysctls: kernel.msgmnb=13107200,kernel.msgmni=256,kernel.msgmax=65536,kernel.shmmax=69719476736,kernel.sem=500 256000 250 1024
      labels:
        app: sie-test
    spec:
      nodeSelector:
        cslckind1: test
      containers:
      - name: sie-ssh
        image: 172.28.2.2:4000/sie:20180112
        command:
        - '/usr/sbin/sshd'
        - -D
        ports:
        - containerPort: 22

---
apiVersion: v1
kind: Service
metadata:
  name: siessh-svc
  namespace: default
spec:
  type: NodePort
  selector:
    app: sie-test
  ports:
  - protocol: TCP
    nodePort: 30032
    port: 22
    targetPort: 22

再次查看：

daweij@master:~/stady01/ipctest$ kubectl get pods -o wide
NAME                                   READY     STATUS    RESTARTS   AGE       IP               NODE
sie-ssh-7cb4f57ddc-cgqbh               1/1       Running   0          13m       10.233.102.169   node1

daweij@master:~/stady01/ipctest$ kubectl get service -o wide
NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE       SELECTOR
siessh-svc            NodePort    10.233.33.217   <none>        22:30032/TCP     13m       app=sie-test

daweij@master:~/stady01/ipctest$ kubectl get deployment -o wide
NAME                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINERS            IMAGES                                       SELECTOR
sie-ssh               1         1         1            1           14m       sie-ssh               172.28.2.2:4000/sie:20180112                 app=sie-test

 

 

经测试，外部可访问：

daweij@master:~/stady01/ipctest$ ssh root@172.28.2.211 -p 30032
The authenticity of host '[172.28.2.211]:30032 ([172.28.2.211]:30032)' can't be established.
RSA key fingerprint is SHA256:wn36xY7Zpidyya2JJazfiSZ3oJBvXKQ9qMA9QuBnxrE.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[172.28.2.211]:30032' (RSA) to the list of known hosts.
root@172.28.2.211's password: 
Last login: Fri Mar  9 17:35:25 from 172.28.2.211


[root@sie-ssh-7cb4f57ddc-cgqbh ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr DE:97:EA:F5:80:F8  
          inet addr:10.233.102.169  Bcast:0.0.0.0  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:99 errors:0 dropped:0 overruns:0 frame:0
          TX packets:65 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:14335 (13.9 KiB)  TX bytes:13860 (13.5 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)