|
| 1 | +--- |
| 2 | +title: Advanced Docker Container Debugging - A Comprehensive Guide |
| 3 | +authorName: Prajwol Amatya |
| 4 | +authorAvatar: https://1.gravatar.com/avatar/de64e53c0e2cb393dd0d14ffdd53058ee9c607b35e366dd392425bd1b95a034c?size=256 |
| 5 | +authorLink: https://github.com/prajwolamatya |
| 6 | +createdAt: May 07, 2025 |
| 7 | +tags: debugging, docker, docker container |
| 8 | +banner: https://blog.jankaritech.com/src/assets/Debugging/images/Debugging.png |
| 9 | +--- |
| 10 | + |
| 11 | +Docker has revolutionized modern software development by enabling lightweight, portable, and scalable containerized applications. However, as deployments grow in complexity, so do the challenges in debugging and optimizing containers. In this blog, we'll explore powerful debugging techniques using a sample project with Node.js, Nginx, and Redis. You can get the sample project [here](https://github.com/prajwolamatya/debug-docker). |
| 12 | + |
| 13 | +Our setup consists of: |
| 14 | +- A Node.js application (port 3000) |
| 15 | +- Nginx as a reverse proxy (port 80) |
| 16 | +- Redis for caching (port 6379) |
| 17 | + |
| 18 | +## 1. Container Inspection |
| 19 | +### Viewing Running Containers |
| 20 | +First, let's check whether our containers are actually running: |
| 21 | + |
| 22 | +```bash |
| 23 | +docker compose ps |
| 24 | +``` |
| 25 | + |
| 26 | +**Example Output:** |
| 27 | +```console |
| 28 | +Name Command State Ports |
| 29 | +debug-docker_nginx_1 /docker-entrypoint.sh ngin ... Up 0.0.0.0:80->80/tcp |
| 30 | +debug-docker_nodejs-app_1 docker-entrypoint.sh node ... Up 0.0.0.0:3000->3000/tcp |
| 31 | +debug-docker_redis_1 docker-entrypoint.sh redis ... Up 0.0.0.0:6379->6379/tcp |
| 32 | +``` |
| 33 | + |
| 34 | +- **State:** Shows if container is running/stopped |
| 35 | +- **Ports:** Reveals port mappings (host:container) |
| 36 | + |
| 37 | +### Inspecting Container Details |
| 38 | +For deeper inspection of a specific container: |
| 39 | + |
| 40 | +```bash |
| 41 | +docker inspect debug-docker_nodejs-app_1 |
| 42 | +``` |
| 43 | + |
| 44 | +This returns a JSON with all container details including: |
| 45 | +- Network settings |
| 46 | +- Mounts |
| 47 | +- Environment variables |
| 48 | +- IP addresses |
| 49 | + |
| 50 | +**Filter specific information:** |
| 51 | + |
| 52 | +```bash |
| 53 | +docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' debug-docker_nodejs-app_1 |
| 54 | +``` |
| 55 | + |
| 56 | +Let's breakdown the command: |
| 57 | +- `-f` or `--format` activates Go template formatting |
| 58 | +- `{{range .NetworkSettings.Networks}}` iterates through networks |
| 59 | +- `{{.IPAddress}}` extracts the IP for each network |
| 60 | +- `{{end}}` closes the loop |
| 61 | + |
| 62 | +**Example Output:** |
| 63 | + |
| 64 | +(*The container's internal IP in Docker's network*) |
| 65 | +```console |
| 66 | +172.19.0.3 |
| 67 | +``` |
| 68 | + |
| 69 | +Docker uses **Go Templates** for the `--format` filtering in `docker inspect`. You can get more details on how to construct the filter in Go's [text/template](https://pkg.go.dev/text/template) package. |
| 70 | +Also, you can learn more on formatting output in [Format command and log output](https://docs.docker.com/engine/cli/formatting/) section. |
| 71 | + |
| 72 | +## 2. Log Analysis |
| 73 | +### Viewing Container Logs |
| 74 | + |
| 75 | +```bash |
| 76 | +docker compose logs |
| 77 | +``` |
| 78 | + |
| 79 | +This will return the logs of containers that are part of the docker compose setup. You can also get the container specific logs using the following command. |
| 80 | + |
| 81 | +```bash |
| 82 | +docker compose logs nodejs-app |
| 83 | +``` |
| 84 | + |
| 85 | +**Example Output (when accessing the service):** |
| 86 | + |
| 87 | +```console |
| 88 | +debug-docker_nodejs-app_1 | Node.js server running on port 3000 |
| 89 | +``` |
| 90 | + |
| 91 | +This confirms your Node.js application launched successfully. |
| 92 | + |
| 93 | +### Generating Request Logs |
| 94 | +Make a test request to generate logs: |
| 95 | + |
| 96 | +```bash |
| 97 | +curl -v http://localhost:3000 |
| 98 | +``` |
| 99 | + |
| 100 | +After the request, check the logs again to see: |
| 101 | + |
| 102 | +```console |
| 103 | +debug-docker_nodejs-app_1 | Node.js server running on port 3000 |
| 104 | +debug-docker_nodejs-app_1 | GET / 200 7.001 ms - 19 |
| 105 | +``` |
| 106 | + |
| 107 | +To view only GET requests from the last 5 minutes |
| 108 | +```bash |
| 109 | +docker compose logs --since 5m nodejs-app | grep "GET" |
| 110 | +``` |
| 111 | + |
| 112 | +**Example Output:** |
| 113 | + |
| 114 | +```console |
| 115 | +GET / 200 7.001 ms - 19 |
| 116 | +``` |
| 117 | + |
| 118 | +## 3. Network Troubleshooting |
| 119 | +### Checking Container Connectivity |
| 120 | + |
| 121 | +Test if Nginx can reach Node.js: |
| 122 | + |
| 123 | +```bash |
| 124 | +docker exec debug-docker_nginx_1 ping nodejs-app |
| 125 | +``` |
| 126 | + |
| 127 | +- `docker exec`: Executes a command inside a running container (`debug-docker_nginx_1`) |
| 128 | +- `ping nodejs-app`: Calls the Linux `ping` utility to test network reachability to the hostname `nodejs-app` |
| 129 | + |
| 130 | +**Example Output:** |
| 131 | + |
| 132 | +```console |
| 133 | +PING nodejs-app (172.19.0.3): 56 data bytes |
| 134 | +64 bytes from 172.19.0.3: seq=0 ttl=64 time=0.060 ms |
| 135 | +``` |
| 136 | + |
| 137 | +- **Success:** <1ms response confirms network connectivity |
| 138 | +- **Failure:** Would show "unknown host" or timeout |
| 139 | + |
| 140 | +Docker Compose automatically assigns hostnames to containers based on the service names defined in `docker-compose.yml`. |
| 141 | + |
| 142 | +Example: |
| 143 | + |
| 144 | +```yaml |
| 145 | +services: |
| 146 | + nodejs-app: # This becomes the hostname |
| 147 | + image: node:alpine |
| 148 | +``` |
| 149 | +
|
| 150 | +Run `docker compose ps` to see the exact service/container names: |
| 151 | + |
| 152 | +```bash |
| 153 | +docker compose ps --format "table {{.Name}}\t{{.Service}}" |
| 154 | +``` |
| 155 | +Output: |
| 156 | + |
| 157 | +```console |
| 158 | +NAME SERVICE |
| 159 | +debug-docker_nginx_1 nginx |
| 160 | +debug-docker_nodejs-app_1 nodejs-app |
| 161 | +``` |
| 162 | + |
| 163 | +Alternatively, you can use service names directly from `docker-compose.yml` instead of full container name. |
| 164 | + |
| 165 | +```bash |
| 166 | +docker compose exec nginx ping nodejs-app |
| 167 | +``` |
| 168 | + |
| 169 | +**Example Output:** |
| 170 | + |
| 171 | +```console |
| 172 | +PING nodejs-app (172.19.0.3): 56 data bytes |
| 173 | +64 bytes from 172.19.0.3: seq=0 ttl=64 time=0.042 ms |
| 174 | +``` |
| 175 | + |
| 176 | +### Examining Post Accessibility |
| 177 | +Check if Node.js is listening on port 3000 inside its container: |
| 178 | + |
| 179 | +```bash |
| 180 | +docker exec nodejs-app netstat -tuln |
| 181 | +``` |
| 182 | + |
| 183 | +- `docker exec`: Runs a command inside a specific container (`nodejs-app`). |
| 184 | +- `netstat -tuln`: A Linux utility to list all listening network ports with the flags: |
| 185 | + - `-t`: Show TCP ports |
| 186 | + - `-u`: Show UDP ports |
| 187 | + - `-l`: Display only listening ports (services accepting connections) |
| 188 | + - `-n`: Show numeric addresses/ports |
| 189 | + |
| 190 | +**Example Output:** |
| 191 | +```console |
| 192 | +Active Internet connections (only servers) |
| 193 | +Proto Recv-Q Send-Q Local Address Foreign Address State |
| 194 | +tcp 0 0 :::3000 :::* LISTEN |
| 195 | +``` |
| 196 | + |
| 197 | +- Shows Node.js listening on port 3000 |
| 198 | +- No output means the service isn't running properly |
| 199 | + |
| 200 | +## 4. Interactive Debugging |
| 201 | +### Executing into Containers |
| 202 | + |
| 203 | +For Node.js application debugging: |
| 204 | +```bash |
| 205 | +docker exec -it nodejs-app sh |
| 206 | +``` |
| 207 | + |
| 208 | +This gives you full shell access inside the container where you can run any Linux command (as long as the tool exists in the container). Following are few things that you can do. |
| 209 | + |
| 210 | +1. Check running processes: `ps aux` |
| 211 | +```bash |
| 212 | +ps aux |
| 213 | +``` |
| 214 | + |
| 215 | +**Output:** |
| 216 | + |
| 217 | +```console |
| 218 | +PID USER TIME COMMAND |
| 219 | + 1 root 0:00 {MainThread} node app.js |
| 220 | + 28 root 0:00 sh |
| 221 | + 45 root 0:00 ps aux |
| 222 | +``` |
| 223 | +- Shows all running processes in the container |
| 224 | +- `PID1`: Your Node.js application (node app.js) |
| 225 | +- `PID28`: The shell session you just started |
| 226 | +- `PID 45`: The `ps aux` command itself |
| 227 | +- Confirms your application is running as the main process |
| 228 | + |
| 229 | +2. Examine environment variables: `printenv` |
| 230 | +```bash |
| 231 | +printenv |
| 232 | +``` |
| 233 | + |
| 234 | +**Output:** |
| 235 | +```console |
| 236 | +NODE_VERSION=23.11.0 |
| 237 | +HOSTNAME=ada88201c429 |
| 238 | +YARN_VERSION=1.22.22 |
| 239 | +SHLVL=1 |
| 240 | +HOME=/root |
| 241 | +TERM=xterm |
| 242 | +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin |
| 243 | +PWD=/app |
| 244 | +``` |
| 245 | +- Displays all environment variables to your Node.js app |
| 246 | + |
| 247 | +3. Test Redis connectivity: `redis-cli -h redis ping` |
| 248 | +```bash |
| 249 | +redis-cli -h redis ping |
| 250 | +``` |
| 251 | + |
| 252 | +**Output:** |
| 253 | +```console |
| 254 | +PONG |
| 255 | +``` |
| 256 | + |
| 257 | +- Tests connectivity to your Redis container |
| 258 | +- `PONG` response confirms network connectivity |
| 259 | + |
| 260 | +### Debugging Nginx Configuration |
| 261 | +```bash |
| 262 | +docker exec -it debug-docker_nginx_1 nginx -t |
| 263 | +``` |
| 264 | + |
| 265 | +**Example Output:** |
| 266 | +```console |
| 267 | +nginx: the configuration file /etc/nginx/nginx.conf syntax is ok |
| 268 | +nginx: configuration file /etc/nginx/nginx.conf test is successful |
| 269 | +``` |
| 270 | + |
| 271 | +## 5. Health Checks and Readiness Probes |
| 272 | +Docker health checks are automated tests that periodically verify if a container is functioning properly. Health checks transform your containers from static processes into self aware services that can catch issues like application crashes, frozen processes, dependency failures, etc. |
| 273 | +Let's enhance our `docker-compose.yml` with health checks: |
| 274 | + |
| 275 | +```yaml |
| 276 | +services: |
| 277 | + nodejs-app: |
| 278 | + # ... existing config ... |
| 279 | + healthcheck: |
| 280 | + test: ["CMD", "curl", "-f", "http://localhost:3000"] |
| 281 | + interval: 30s |
| 282 | + timeout: 10s |
| 283 | + retries: 3 |
| 284 | +
|
| 285 | + redis: |
| 286 | + # ... existing config ... |
| 287 | + healthcheck: |
| 288 | + test: ["CMD", "redis-cli", "ping"] |
| 289 | + interval: 30s |
| 290 | + timeout: 5s |
| 291 | + retries: 3 |
| 292 | +``` |
| 293 | + |
| 294 | +- Docker runs the `test` command at your specified `interval` (e.g., every 30s) |
| 295 | +- The service is marked `healthy` only if the command succeeds (exit code 0) |
| 296 | +- After `retries` consecutive failures, it's marked `unhealthy` |
| 297 | + |
| 298 | +Now check the container health: |
| 299 | +```bash |
| 300 | +docker ps --format "table {{.Names}}\t{{.Status}}" |
| 301 | +``` |
| 302 | + |
| 303 | +**Example Output:** |
| 304 | +```console |
| 305 | +NAMES STATUS |
| 306 | +debug-docker_nginx_1 Up 5 minutes |
| 307 | +debug-docker_nodejs-app_1 Up 5 minutes (healthy) |
| 308 | +debug-docker_redis_1 Up 5 minutes (healthy) |
| 309 | +``` |
| 310 | + |
| 311 | +## 6. Temporary Debug Containers |
| 312 | +Sometimes you need additional tools that are not part of the containers you are using, for that create a temporary debug container with the needed tools in the same network: |
| 313 | +```bash |
| 314 | +docker run -it --rm --network debug-docker_default alpine sh |
| 315 | +``` |
| 316 | + |
| 317 | +Now from this container you can: |
| 318 | +1. Test DNS resolution: |
| 319 | +```bash |
| 320 | +nslookup nodejs-app |
| 321 | +``` |
| 322 | + |
| 323 | +**Example Output:** |
| 324 | +```console |
| 325 | +Server: 127.0.0.11 |
| 326 | +Address: 127.0.0.11:53 |
| 327 | +
|
| 328 | +Non-authoritative answer: |
| 329 | +Name: nodejs-app |
| 330 | +Address: 172.19.0.3 |
| 331 | +``` |
| 332 | + |
| 333 | +- Confirms Docker's internal DNS is working |
| 334 | +- Shows the service name resolves to the correct container IP (172.19.03) |
| 335 | + |
| 336 | +2. Check connectivity: |
| 337 | +```bash |
| 338 | +wget -qO- http://nodejs-app:3000 |
| 339 | +``` |
| 340 | + |
| 341 | +**Example Output:** |
| 342 | +```console |
| 343 | +Hello from Node.js! |
| 344 | +``` |
| 345 | + |
| 346 | +- Shows successful TCP connection to port 3000 |
| 347 | +- Returns the actual HTTP response from your Node.js app |
| 348 | + |
| 349 | +## 7. Docker System Diagnostics |
| 350 | +When facing resource issues: |
| 351 | +```bash |
| 352 | +docker system df |
| 353 | +``` |
| 354 | + |
| 355 | +**Example Output:** |
| 356 | +```console |
| 357 | +TYPE TOTAL ACTIVE SIZE RECLAIMABLE |
| 358 | +Images 17 3 2.374GB 2.295GB (96%) |
| 359 | +Containers 3 3 2B 0B (0%) |
| 360 | +Local Volumes 2 1 88B 88B (100%) |
| 361 | +Build Cache 108 0 37.82MB 37.82MB |
| 362 | +``` |
| 363 | + |
| 364 | +Check detailed resource usage: |
| 365 | +```bash |
| 366 | +docker stats |
| 367 | +``` |
| 368 | + |
| 369 | +**Example Output:** |
| 370 | +```console |
| 371 | +CONTAINER ID NAME CPU % MEM USAGE/LIMIT MEM % NET I/O BLOCK I/O PIDS |
| 372 | +5111f46d640b debug-docker_nginx_1 0.00% 9.633MiB/31.06GiB 0.03% 41.9kB/126B 0B/4.1kB 13 |
| 373 | +e701e4d02bb0 debug-docker_nodejs-app_1 0.00% 13.31MiB/31.06GiB 0.04% 45.3kB/3.49kB 0B/0B 7 |
| 374 | +3e0399cc7510 debug-docker_redis_1 0.93% 4.691MiB/31.06GiB 0.01% 42.3kB/126B 1.43MB/0B 6 |
| 375 | +``` |
| 376 | + |
| 377 | +## Conclusion |
| 378 | +Effective Docker debugging requires a systematic approach. By mastering these techniques, you'll be able to diagnose and resolve even the most complex Docker issues in production environments. |
0 commit comments