Назад | Перейти на главную страницу

При перезапуске Vault Pod переходит в аварийное состояние LoopBackOff

Мы настроили хранилище для работы в качестве модуля в кластере. В приведенном ниже файле YAML развертывания мы включили инициализацию и распечатку хранилища, которые происходят при первоначальном запуске модуля. Но когда модуль перезапускается, модуль переходит в состояние LoopBackOff, потому что хранилище повторно инициализируется. Это связано с тем, что мы включили команду инициализации и распечатки в команду жизненного цикла postStart файла развертывания. Есть ли способ, которым мы могли бы инициализировать модуль только один раз, а позже при перезапуске модуля распечатать хранилище с помощью существующих ключей?

Файл развертывания:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: vault
  name: vault
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: vault
    spec:
      containers:
      - image: vault
        name: vault
        imagePullPolicy: Always
        ports:
        - containerPort: 8200
          name: vaultport
          protocol: TCP
        securityContext:
         capabilities:
           add:
             - IPC_LOCK
        env:
        - name: VAULT_ADDR
          value: "http://0.0.0.0:8200"
        command: ["vault", "server"]
        args: 
          - "-config=/vault/config/config.hcl"
        volumeMounts:
          - name: vault-unseal
            mountPath: /vault/file/unseal.sh
            subPath: unseal.sh
          - name: vault-config
            mountPath: /vault/config/config.hcl
            subPath: config.hcl
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh", "-c", "vault operator init > /vault/file/keys.txt; sh /vault/file/unseal.sh" ]
      volumes:
      - name: vault-unseal
        configMap:
          name: vault-unseal
      - name: vault-config
        configMap:
          name: vault-config 
      imagePullSecrets:
      - name: regcred

Вывод команды kubectl describe pod:

Name:           vault-677bfd9c9c-dwsgv
Namespace:      xxx
Priority:       0
Node:           xxxxxxx-5b587f98-ljf4/10.0.0.11
Start Time:     Thu, 30 Jan 2020 06:26:21 +0000
Labels:         app=vault
                pod-template-hash=677bfd9c9c
Annotations:    <none>
Status:         Running
IP:             10.4.2.10
IPs:            <none>
Controlled By:  ReplicaSet/vault-677bfd9c9c
Containers:
  vault:
    Container ID:  xxxxxxxxxxx
    Image:         xxxxxxxxxxxxxxxx
    Image ID:      xxxxxxxxxxxxxxxxxxxxxxxxx
    Port:          8200/TCP
    Host Port:     0/TCP
    Command:
      vault
      server
    Args:
      -config=/vault/config/config.hcl
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 30 Jan 2020 06:26:26 +0000
      Finished:     Thu, 30 Jan 2020 06:26:27 +0000
    Ready:          False
    Restart Count:  1
    Environment:
      VAULT_ADDR:  http://0.0.0.0:8200
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kxfdb (ro)
      /vault/config/config.hcl from vault-config (rw,path="config.hcl")
      /vault/file from vault-data (rw)
      /vault/file/unseal.sh from vault-unseal (rw,path="unseal.sh")
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  vault-unseal:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      vault-unseal
    Optional:  false
  vault-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      vault-config
    Optional:  false
  vault-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  vault-data
    ReadOnly:   false
  default-token-kxfdb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-kxfdb
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason               Age                From                                                          Message
  ----     ------               ----               ----                                                          -------
  Normal   Scheduled            18s                default-scheduler                                             Successfully assigned xxx/xxxxxxxxxx
  Normal   Pulling              13s (x2 over 15s)  kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4  pulling image "xxxxxxxxx"
  Normal   Pulled               13s (x2 over 15s)  kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4  Successfully pulled image "xxxxxxx"
  Normal   Created              13s (x2 over 15s)  kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4  Created container
  Normal   Started              13s (x2 over 14s)  kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4  Started container
  Warning  FailedPostStartHook  12s (x2 over 14s)  kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4  Exec lifecycle hook ([/bin/sh -c vault operator init > /vault/file/keys.txt; sh /vault/file/unseal.sh]) for Container "vault" in Pod "vault-677bfd9c9c-dwsgv_xxx(6ebdc17a-4329-11ea-9fc1-4201c0a80004)" failed - error: command '/bin/sh -c vault operator init > /vault/file/keys.txt; sh /vault/file/unseal.sh' exited with 2: Error initializing: Error making API request.

URL: PUT http://0.0.0.0:8200/v1/sys/init
Code: 400. Errors:

* Vault is already initialized
An error occurred attempting to ask for an unseal key. The raw error message
is shown below, but usually this is because you attempted to pipe a value
into the unseal command or you are executing outside of a terminal (tty). You
should run the unseal command from a terminal for maximum security. If this
is not an option, the unseal key can be provided as the first argument to the
unseal command. The raw error was:  file descriptor 0 is not a terminal
An error occurred attempting to ask for an unseal key. The raw error message
is shown below, but usually this is because you attempted to pipe a value
into the unseal command or you are executing outside of a terminal (tty). You
should run the unseal command from a terminal for maximum security. If this
is not an option, the unseal key can be provided as the first argument to the
unseal command. The raw error was:  file descriptor 0 is not a terminal
An error occurred attempting to ask for an unseal key. The raw error message
is shown below, but usually this is because you attempted to pipe a value
into the unseal command or you are executing outside of a terminal (tty). You
should run the unseal command from a terminal for maximum security. If this
is not an option, the unseal key can be provided as the first argument to the
unseal command. The raw error was:  file descriptor 0 is not a terminal
Token (will be hidden):
Error authenticating: An error occurred attempting to ask for a token. The raw error message is shown below, but usually this is because you attempted to pipe a value into the command or you are executing outside of a terminal (tty). If you want to pipe the value, pass "-" as the argument to read from stdin. The raw error was: file descriptor 0 is not a terminal
, message: "Unseal Key (will be hidden): \nUnseal Key (will be hidden): \nUnseal Key (will be hidden): \nKey                Value\n---                -----\nSeal Type          shamir\nInitialized        true\nSealed             true\nTotal Shares       5\nThreshold          3\nUnseal Progress    0/3\nUnseal Nonce       n/a\nVersion            1.3.2\nHA Enabled         false\n++++++++++++ Vault Status +++++++++\nKey                Value\n---                -----\nSeal Type          shamir\nInitialized        true\nSealed             true\nTotal Shares       5\nThreshold          3\nUnseal Progress    0/3\nUnseal Nonce       n/a\nVersion            1.3.2\nHA Enabled         false\nError initializing: Error making API request.\n\nURL: PUT http://0.0.0.0:8200/v1/sys/init\nCode: 400. Errors:\n\n* Vault is already initialized\nAn error occurred attempting to ask for an unseal key. The raw error message\nis shown below, but usually this is because you attempted to pipe a value\ninto the unseal command or you are executing outside of a terminal (tty). You\nshould run the unseal command from a terminal for maximum security. If this\nis not an option, the unseal key can be provided as the first argument to the\nunseal command. The raw error was:  file descriptor 0 is not a terminal\nAn error occurred attempting to ask for an unseal key. The raw error message\nis shown below, but usually this is because you attempted to pipe a value\ninto the unseal command or you are executing outside of a terminal (tty). You\nshould run the unseal command from a terminal for maximum security. If this\nis not an option, the unseal key can be provided as the first argument to the\nunseal command. The raw error was:  file descriptor 0 is not a terminal\nAn error occurred attempting to ask for an unseal key. The raw error message\nis shown below, but usually this is because you attempted to pipe a value\ninto the unseal command or you are executing outside of a terminal (tty). You\nshould run the unseal command from a terminal for maximum security. If this\nis not an option, the unseal key can be provided as the first argument to the\nunseal command. The raw error was:  file descriptor 0 is not a terminal\nToken (will be hidden): \nError authenticating: An error occurred attempting to ask for a token. The raw error message is shown below, but usually this is because you attempted to pipe a value into the command or you are executing outside of a terminal (tty). If you want to pipe the value, pass \"-\" as the argument to read from stdin. The raw error was: file descriptor 0 is not a terminal\n"
  Normal   Killing  12s (x2 over 14s)  kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4  Killing container with id docker://vault:FailedPostStartHook
  Warning  BackOff  10s (x2 over 11s)  kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4  Back-off restarting failed container

Да, в этом случае вам нужно сделать свой сценарий более умным, чтобы сначала проверить, инициализировано ли хранилище. Вот очень простой пример с bash-скриптом.

Кроме того, обычным способом достижения этого является использование коляски (которая в конечном итоге переходит в "спящий" на неопределенный срок), но postStart hook также должен работать до тех пор, пока вы делаете свой скрипт достаточно устойчивым, учитывая, что в некоторых случаях он будет запускаться до того, как контейнер хранилища будет запущен, и в конечном итоге убьет его и перезапустит, пока они оба в конечном итоге не синхронизируются.

Я очень рекомендую это выступление Сета Варго где автоматическая инициализация выполняется с помощью более сложной программы Golang.