Skip to main content
Version: v1.26.x LTS

Troubleshooting Kubernetes environments

Troubleshooting Kubernetes environments

The following topics contain information that can help you troubleshoot problems when you encounter unexpected behavior installing and using Zowe™ containers in a Kubernetes environment.

ISSUE: /tmp directory is not writable#


readOnlyRootFilesystem SecurityContext is enabled by default in Deployment object definition. As a result, /tmp is read-only and not writable to zowe runtime user.

Recommended solution:

Adjust your component to check the TMPDIR or TMP environment variable to determine the location of the temporary directory. Zowe runtime customizes those variables and points them to /home/zowe/instance/tmp directory, which is writable.

Alternative solution:

Disabling readOnlyRootFilesystem SecurityContext is not recommended. But you can make /tmp writable by replacing it with a newly mounted volume. Here is an example of defining the /tmp volume.

apiVersion: apps/v1kind: Deploymentspec:  template:    spec:      volumes:        - name: tmp          emptyDir: {}        containers:          - name: <my-component-name>            volumeMounts:              - name: tmp                mountPath: "/tmp"

With this added to your Deployment, your component should be able to write to /tmp directory.

ISSUE: Permission denied showing in pod log#


You see error messages similar to the following one in your pod log.

cp: cannot create regular file '/home/zowe/instance/workspace/manifest.json': Permission deniedmkdir: cannot create directory '/home/zowe/instance/workspace/api-mediation': Permission deniedmkdir: cannot create directory '/home/zowe/instance/workspace/backups': Permission deniedcp: cannot create regular file '/home/zowe/instance/workspace/active_configuration.cfg': Permission denied/home/zowe/runtime/bin/internal/ line 236: /home/zowe/instance/workspace/active_configuration.cfg: Permission denied/home/zowe/runtime/bin/internal/ line 240: /home/zowe/instance/workspace/active_configuration.cfg: Permission denied/home/zowe/runtime/bin/internal/ line 241: /home/zowe/instance/workspace/active_configuration.cfg: Permission denied

It means zowe user (UID 20000) does not have write permission to your persistent volume. It's very likely the persistent volume is mounted as root user.


To solve this issue, you can modify workload files with extra initContainers step like this:

apiVersion: apps/v1kind: Deploymentspec:  template:    spec:      initContainers:        - name: update-workspace-permission          image: busybox:1.28          command: ['sh', '-c', 'OWNER=`stat -c "%u:%g" /home/zowe/instance/workspace` && PERMISSION=`stat -c "%a" /home/zowe/instance/workspace` && echo "Zowe workspace owner is ${OWNER} with ${PERMISSION} permission" && if [ "${OWNER}" != "20000:20000" -a "${PERMISSION}" != "777" ]; then chown -R 20000:20000 /home/zowe/instance/workspace; fi']          imagePullPolicy: Always          resources:            requests:              memory: "64Mi"              cpu: "10m"            limits:              memory: "128Mi"              cpu: "100m"          volumeMounts:            - name: zowe-workspace              mountPath: "/home/zowe/instance/workspace"          securityContext:            readOnlyRootFilesystem: true            allowPrivilegeEscalation: false            capabilities:              drop:                - all              add:                - CHOWN            runAsUser: 0            runAsGroup: 0

ISSUE: Deployment and ReplicaSet failed to create pod#


If you are using OpenShift and see these error messages in ReplicaSet Events:

Generated from replicaset-controllerError creating: pods "api-catalog-??????????-" is forbidden: unable to validate against any security context constraint: []

That means the Zowe ServiceAccount zowe-sa doesn't have any SecurityContextConstraint attached.


You can run this command to grant a certain level of permission, for example, privileged, to zowe-sa ServiceAccount:

oc admin policy add-scc-to-user privileged -z zowe-sa -n zowe

ISSUE: Failed to create services#


If you are using OpenShift and apply services, you may see this error:

The Service "api-catalog-service" is invalid: spec.ports[0].appProtocol: Forbidden: This field can be enabled with the ServiceAppProtocol feature gate


To fix this issue, you can simply find and comment out this line in the Service definition files:

appProtocol: https

With OpenShift, you can define a PassThrough Route to let Zowe handle TLS connections.