Tail logs from Multiple k8 pods

By: Saikrishna Madupu – Sr Devops Engineer

This article reviews how to tail logs from multiple pods via Kubernetes and Stern.

Kubernetes (K8) is a scalable container orchestrator. It is fairly lightweight to support IoT appliances and it can also handle huge business systems with hundreds of apps and hosts .

Stern is a tool for the tailing of numerous Kubernetes pods and the numerous containers that make up each pod. To facilitate faster debugging, each result is color coded.

As the query is a regular expression, the pod name can be easily filtered, and the exact id is not required. For instance, for instance omitting the deployment id. When a pod is deleted, it is removed from the tail, and when a new pod is added, it is automatically tailed.

Stern can tail all of the containers in a pod instead of having to do each one manually. You can simply specify the container flag to limit the number of containers displayed. By default, all containers are monitored.

Deploying a nginx svc:
kind: Service
apiVersion: v1
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  selector:
    app: nginx
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 80
          protocol: TCP%  

kubectl apply --filename nginx-svc.yaml -n keyva

O/p: service/nginx unchanged
deployment.apps/nginx created

we can validate and verify the svc and pods that being up and running:
kubectl get all -n keyva
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-cd55c47f5-gwtkn   1/1     Running   0          12s

NAME            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
service/nginx   ClusterIP   10.96.58.31   <none>        80/TCP    88d

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1           12s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-cd55c47f5   1         1         1       12s

kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
nginx-cd55c47f5-gwtkn   1/1     Running   0          30s

KubeCtl has limits:

Using the label selection, it is evident that kubectl can read logs from numerous pods, however this technique has a drawback.

• First, the logs from several pods are jumbled together, making it impossible to determine which log line originated from which pod.
• Second, tail mode (using –follow (-f)) is not supported.

The reason for this is that –follow streams the API server’s logs. You open a connection to the API server per pod, which opens a connection to the associated kubelet to stream logs continually. This does not scale well and results in many incoming and outgoing connections to the API server. As a result, it became a design decision to restrict the number of concurrent connections. Using Stern:

The command is fairly straightforward. Stern retrieves the logs from the given namespace for the specified application. In the case of Stern, you can view not only logs from a single Kubernetes object, such as a deployment or service, but also logs from all related objects. Example:

Stern -n keyva nginx

stern -n keyva nginx                           
+ nginx-cd55c47f5-86ql5 › nginx
+ nginx-cd55c47f5-bm55t › nginx
+ nginx-cd55c47f5-gwtkn › nginx
nginx-cd55c47f5-gwtkn nginx /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
nginx-cd55c47f5-gwtkn nginx /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
nginx-cd55c47f5-gwtkn nginx /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
nginx-cd55c47f5-gwtkn nginx 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
nginx-cd55c47f5-gwtkn nginx 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
nginx-cd55c47f5-gwtkn nginx /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx-cd55c47f5-gwtkn nginx /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
nginx-cd55c47f5-gwtkn nginx /docker-entrypoint.sh: Configuration complete; ready for start up
nginx-cd55c47f5-gwtkn nginx 2023/01/17 10:42:19 [notice] 1#1: using the "epoll" event method
nginx-cd55c47f5-gwtkn nginx 2023/01/17 10:42:19 [notice] 1#1: nginx/1.23.3
nginx-cd55c47f5-gwtkn nginx 2023/01/17 10:42:19 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 
nginx-cd55c47f5-gwtkn nginx 2023/01/17 10:42:19 [notice] 1#1: OS: Linux 5.10.124-linuxkit
nginx-cd55c47f5-gwtkn nginx 2023/01/17 10:42:19 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
nginx-cd55c47f5-gwtkn nginx 2023/01/17 10:42:19 [notice] 1#1: start worker processes
nginx-cd55c47f5-gwtkn nginx 2023/01/17 10:42:19 [notice] 1#1: start worker process 35
nginx-cd55c47f5-gwtkn nginx 2023/01/17 10:42:19 [notice] 1#1: start worker process 36
nginx-cd55c47f5-gwtkn nginx 2023/01/17 10:42:19 [notice] 1#1: start worker process 37
nginx-cd55c47f5-gwtkn nginx 2023/01/17 10:42:19 [notice] 1#1: start worker process 38
nginx-cd55c47f5-bm55t nginx /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
nginx-cd55c47f5-bm55t nginx /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
nginx-cd55c47f5-bm55t nginx /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
nginx-cd55c47f5-bm55t nginx 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
nginx-cd55c47f5-bm55t nginx 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
nginx-cd55c47f5-bm55t nginx /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx-cd55c47f5-bm55t nginx /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
nginx-cd55c47f5-bm55t nginx /docker-entrypoint.sh: Configuration complete; ready for start up
nginx-cd55c47f5-bm55t nginx 2023/01/17 10:47:26 [notice] 1#1: using the "epoll" event method
nginx-cd55c47f5-bm55t nginx 2023/01/17 10:47:26 [notice] 1#1: nginx/1.23.3
nginx-cd55c47f5-bm55t nginx 2023/01/17 10:47:26 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 
nginx-cd55c47f5-86ql5 nginx /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
nginx-cd55c47f5-86ql5 nginx /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
nginx-cd55c47f5-bm55t nginx 2023/01/17 10:47:26 [notice] 1#1: OS: Linux 5.10.124-linuxkit
nginx-cd55c47f5-bm55t nginx 2023/01/17 10:47:26 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
nginx-cd55c47f5-bm55t nginx 2023/01/17 10:47:26 [notice] 1#1: start worker processes
nginx-cd55c47f5-86ql5 nginx /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
nginx-cd55c47f5-bm55t nginx 2023/01/17 10:47:26 [notice] 1#1: start worker process 36
nginx-cd55c47f5-86ql5 nginx 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
nginx-cd55c47f5-bm55t nginx 2023/01/17 10:47:26 [notice] 1#1: start worker process 37
nginx-cd55c47f5-86ql5 nginx 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
nginx-cd55c47f5-86ql5 nginx /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx-cd55c47f5-bm55t nginx 2023/01/17 10:47:26 [notice] 1#1: start worker process 38
nginx-cd55c47f5-bm55t nginx 2023/01/17 10:47:26 [notice] 1#1: start worker process 39
nginx-cd55c47f5-86ql5 nginx /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh

If you want to use stern in Kubernetes Pods, you need to create the following ClusterRole and bind it to ServiceAccount.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: stern
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "watch", "list"]

Stern facilitates the output of custom log messages. Using the —output flag, you may utilize the following prepared templates:

It takes a custom template through the —template flag, which is subsequently compiled into a Go template and used for each log message. The following struct is passed to this Go template:

In addition to the built-in functions, the template includes the following functions:

Kubernetes can add complexity. Software programmers need logs quickly to fix problems. Set up your CLI with some aliases and get rolling to tail logs from your apps in real-time if you are using Kubernetes and have access to view logs on your Kubernetes cluster.

About the Author

Saikrishna Madupu, Sr. DevOps Engineer Sai is an IT professional with experience in DevOps Automation, Configuration Management tools, Container workloads and orchestration of those workloads via Kubernetes. A self-starter, and passionate problem-solver, with a flair for innovative design, with an ability to work towards automation whenever possible. He is an experienced Linux, Cloud data center operations and infrastructure engineer. He worked as a Devops cloud consultant in the past, helping clients in migrate on-prem applications to cloud, and holds certifications for AWS, Terraform, Gitlab, and other technologies.