---
title: debugging-crashed-kubernetes-container - Codenotary
description: MONITOR & MANAGE THE RISK EXPOSURE OF YOUR APPLICATIONS WITH TRUESBOM®
image: https://codenotary.com/hubfs/Imported_Blog_Media/unmet-volumeclaim-1.png
---

**$ protect --distro linux --machines 25 --free**

[Start now](https://apps.codenotary.com/linux)

[![cn-logo-black-nobg](https://codenotary.com/hubfs/cn-logo-black-nobg.svg)](https://codenotary.com/)

- Product
  
  #### [![AgentMon Start](https://codenotary.com/hubfs/chub_backup/AgentMon%20Start.svg) **AgentMon Start** See what your AI agents are doing on your machine. TRY NOW →](https://codenotary.com/agentmon-start)
  
  #### [![AgentMon for Enterprise](https://codenotary.com/hubfs/AgentMon%20for%20Enterprise.svg) **AgentMon** Currently monitors more \> 7 million agent interactions/day. TRY NOW →](https://codenotary.com/agentmon)
  
  #### [![AgentX](https://codenotary.com/hubfs/AgentX.svg) **AgentX** Agentic network control middleware. TRY NOW →](https://codenotary.com/agent-network-control)
  
  #### [![Autonomous Security](https://codenotary.com/hubfs/Autonomous%20Security.svg) **Autonomous Security** AI Agents keep your servers secure. TRY NOW →](https://codenotary.com/trust)
- Use Cases
  
  #### [**AI Agent Risk Monitoring** Continuous oversight of autonomous agents across every environment.](https://codenotary.com/use-cases#risk)
  
  #### [**Autonomous Security Operations** Self-healing defenses that detect, contain, and remediate threats.](https://codenotary.com/use-cases#agentops)
  
  #### [**AI Coding Governance & Performance Monitoring** AI-generated code reviewed, tracked, and held to quality standards.](https://codenotary.com/use-cases#performance)
  
  #### [**AI Tool Cost & Usage Optimization** Spend and consumption optimized across every AI service in use.](https://codenotary.com/use-cases#cost)
  
  #### [**AI Tool Security & Policy Enforcement** Approved AI usage enforced with guardrails and policy controls.](https://codenotary.com/use-cases#security#security)
  
  #### [**Shadow AI Governance** Unsanctioned AI tools discovered, surfaced, and brought under control.](https://codenotary.com/use-cases#shadowit)
- [Blog](https://codenotary.com/blog)
- [Press](https://codenotary.com/press)
- Resources
  
  #### [**Integrations** Connect with your favorite tools and platforms. LEARN MORE →](https://codenotary.com/integrations)
  
  #### [**Support** Get help from our dedicated support team. GET HELP →](https://support.codenotary.com)
  
  #### [**Success Stories** Read how customers achieve their goals. READ MORE →](https://codenotary.com/success)
  
  #### [**Learn** Access documentation and learning resources. EXPLORE →](https://codenotary.com/learn)

[Login](https://apps.codenotary.com/auth/login)

[All posts](https://codenotary.com/blog/all)

 Apr 04, 2019

# debugging-crashed-kubernetes-container

 By  [Dennis](https://codenotary.com/blog/author/dennis)  ·   2 minute read

# Debugging crashed Kubernetes container

When working with [Kubernetes](https://kubernetes.io/), you’re often amazed how easy container, service or secret configurations can be deployed with a simple ***kubectl add -f***  command. At least as long as it works. The moment your Kubernetes container doesn’t start, you are forced to start debugging and the simplicity to create resources based on community yaml files fires back.

Now you can dig deep into troubleshooting the container and trust me – that’s can be no fun at all.

Therefore, we start the post with one of the most annoying issue running your container – a **CrashLoopBackoff** error. That error can really drive you nuts.

## Start to debug

You need to know some very important commands before you can even think of debugging. Next to the common **kubectl get** command, the kubectl describe is your way into detailed information about your environment.

Sometimes a **kubectl cluster-info dump** helps getting started.

Using **kubectl get all** or **kubectl get pods**, you can find the explore your environment, but let’s go step by step.

## kubectl describe

Of course, first you need to find out what happens to the container and what’s the container (pod) name.

***kubectl get pods*** does a good job at that for the current namespace, ***kubectl get pods –namespace monitoring*** for the defined namespace and if you want only want pods of all namespace just use ***kubectl get pods –all-namespaces***.

When you know the pod name, run a kubectl describe pod/podname to find the reason of the current status. Sometimes its an simple reason, as resources are not available. Examples are not enough memory or no volumes, persistentvolumeclaims available. 

In this example, you just need to create a volume and a PersistentVolumeClaim for the container to be able to start:

![Kubernetes container missing PersistentVolumeClaim](https://static.hsstatic.net/BlogImporterAssetsUI/ex/missing-image.png)

But there are much uglier error messages like a CrashLoopBackoff that points to the inside of the pod and not the outside environment.

## CrashLoopBackoff

CrashLoopBackoff is a nasty one, as its a bucket that offers a big set of errors, all nicely hidden behind the same error condition. Our example container is called metrics-prometheus, installed with a helm install command and named metrics.

First thing is to try kubectl describe pod to get some more details. Unfortunately, most of the time, the CrashLoopBackoff isn’t showing any outside information. But maybe you’re lucky and find some error in the output.

### Log files

Lets try to get some logs

**kubectl logs pod** or **kubelet logs**

As the Kubernetes container keeps crashing, the logs command doesn’t help either as its only useful for running container.

We need to get into the Kubernetes container node. To find more information about the container we need to dig deeper using the wide parameter.

**kubectl get pods -o wide**

![kubernetes container](https://static.hsstatic.net/BlogImporterAssetsUI/ex/missing-image.png)

As we use [juju](https://kubernetes.io/docs/getting-started-guides/ubuntu/) to manage kubernetes, we can directly ssh into the node. But first we need to find the node name using the command juju status.

![juju status](https://static.hsstatic.net/BlogImporterAssetsUI/ex/missing-image.png)

## SSH into the node

The node where the crashed container was last time is called kubernetes-worker/1:

**juju ssh kubernetes-worker/1**

Now we need to find the log files of the crashed container and the log files are stored under /var/log/containers.

**sudo tail /var/log/container/metrics-prometheus ….**

Just walk through the best options and check for errors. Bingo:

![prometheus error](https://static.hsstatic.net/BlogImporterAssetsUI/ex/missing-image.png)

It seems that the used *prometheus.yml* configuration file has syntax errors we need to fix before the [Prometheus](https://prometheus.io/) server can start.

So lets delete the whole setup again. As we installed it with [helm](https://helm.sh/), lets keep using that great tool using the same name used for the install **metrics**:

**helm del –purge metrics**

In the case of Prometheus, there are plenty of issues that can happen because of an invalid config file, therefore you should use the following project to check your config before deploying into the deep kubernetes cluster.

<https://www.robustperception.io/how-to-check-your-prometheus-yml-is-valid>[https://www.robustperception.io/how-to-check-your-prometheus-yml-is-valid](https://www.robustperception.io/how-to-check-your-prometheus-yml-is-valid)

After fixing the Prometheus configuration file and redeploying the whole metrics setup using helm, all went well and the container was running smoothly.

[![Share on twitter](https://4059529.fs1.hubspotusercontent-na1.net/hub/4059529/hubfs/01-marketplace/twitter-color.png?width=35&height=35&name=twitter-color.png)](https://twitter.com/intent/tweet?original_referer=https://codenotary.com/blog/debugging-crashed-kubernetes-container&utm_medium=social&utm_source=twitter&url=https://codenotary.com/blog/debugging-crashed-kubernetes-container&utm_medium=social&utm_source=twitter&source=tweetbutton&text=) [![Share on facebook](https://4059529.fs1.hubspotusercontent-na1.net/hub/4059529/hubfs/01-marketplace/facebook-color.png?width=35&height=35&name=facebook-color.png)](http://www.facebook.com/share.php?u=https://codenotary.com/blog/debugging-crashed-kubernetes-container&utm_medium=social&utm_source=facebook) [![Share on linkedin](https://4059529.fs1.hubspotusercontent-na1.net/hub/4059529/hubfs/01-marketplace/linkedin-color.png?width=35&height=35&name=linkedin-color.png)](http://www.linkedin.com/shareArticle?mini=true&url=https://codenotary.com/blog/debugging-crashed-kubernetes-container&utm_medium=social&utm_source=linkedin) [![Share on pinterest](https://4059529.fs1.hubspotusercontent-na1.net/hub/4059529/hubfs/pinterest.jpg?width=35&height=35&name=pinterest.jpg)](http://pinterest.com/pin/create/button/?url=https://codenotary.com/blog/debugging-crashed-kubernetes-container&utm_medium=social&utm_source=pinterest&media=)

```json
{
  "@context" : "https://schema.org",
  "@type" : "BlogPosting",
  "author" : {
    "@type" : "Person",
    "name" : "Dennis",
    "url" : "https://codenotary.com/blog/author/dennis"
  },
  "dateModified" : "2023-02-14T21:07:01.969Z",
  "datePublished" : "2019-04-04T21:51:12.000Z",
  "headline" : "debugging-crashed-kubernetes-container - Codenotary",
  "image" : [ "https://codenotary.com/hubfs/Imported_Blog_Media/unmet-volumeclaim-1.png" ],
  "mainEntityOfPage" : {
    "@id" : "https://codenotary.com/blog/debugging-crashed-kubernetes-container",
    "@type" : "WebPage"
  },
  "publisher" : {
    "@type" : "Organization",
    "logo" : {
      "@type" : "ImageObject",
      "url" : "https://codenotary.com/hubfs/logo-light.svg"
    },
    "name" : "Codenotary, Inc."
  }
}
```

```json
{
  "@context" : "http://schema.org",
  "@type" : "Article",
  "author" : {
    "@type" : "Person",
    "name" : [ "Dennis" ],
    "url" : "https://codenotary.com/blog/author/dennis"
  },
  "datePublished" : "2019-04-04T21:51:12+0000",
  "description" : "MONITOR & MANAGE THE RISK EXPOSURE OF YOUR APPLICATIONS WITH TRUESBOM®",
  "headline" : "debugging-crashed-kubernetes-container",
  "image" : "https://23873599.fs1.hubspotusercontent-na1.net/hubfs/23873599/Imported_Blog_Media/unmet-volumeclaim-1.png",
  "publisher" : {
    "@type" : "Organization",
    "logo" : {
      "@type" : "ImageObject",
      "url" : "https://cdn2.hubspot.net/hubfs/23873599/logo-light.svg"
    },
    "name" : ""
  },
  "url" : "https://codenotary.com/blog/debugging-crashed-kubernetes-container"
}
```