Linux & Command Line ADVANCED Linux โฑ๏ธ 5 min read ๐Ÿ“… Updated: 22 Nov 2026

Advanced: Tracing Interrupted Signals (SIGINT/SIGTERM) in Containers using strace and journald

This guide teaches advanced Linux system administrators how to trace the exact sequence of signals (like SIGINT or SIGTERM) received by processes inside containers, diagnosing why applications appear to shut down abruptly during orchestrated restarts.

Gurveer Singh
Gurveer Singh โœ“ Verified Specialist
Founder & Chief Architect ยท hSECURITIES Labs
โšก FAST-TRACK PROTOCOL

Direct Diagnostic & Remediation Summary

  • Check Systemd Logs: Use journalctl -u service_name.service --since "1 hour ago" | grep -i 'signal' to review recent termination events.
  • Trace Execution: Run the problematic application under strace -f -e trace=signal your_application_command to capture all system calls related to signal handling during a controlled test restart.
๐Ÿ› ๏ธ Tools & Prerequisites: Root or sudo privileges on the host Linux machine, knowledge of Docker/Podman container management, and understanding of Linux signals (SIGINT, SIGTERM).
Advanced: Tracing Interrupted Signals (SIGINT/SIGTERM) in Containers using strace and journald

When deploying microservices orchestrated by Kubernetes or Docker Swarm, containers often receive signals like SIGTERM before being forcefully killed. If an application fails to gracefully catch these signals, it can lead to data corruption or non-clean shutdowns. This guide provides a deep dive into using strace and journalctl on the host system to meticulously trace the signal lifecycle.

๐Ÿ” Root Cause & Diagnostic Overview:

The core issue is often a mismatch between how the container runtime (Docker/CRI) sends termination signals and how the application inside interprets them. By default, container orchestrators send SIGTERM first, allowing for graceful shutdown (e.g., closing database connections). If the application ignores this signal or if the underlying process group handling is flawed, it might terminate immediately or hang, leading to perceived instability.

STEP 1

Preparation: Identifying the Target and Baseline Logging

Before tracing signals, you must confirm which process is failing and ensure host logging is comprehensive. We will use journalctl to establish a timeline.

  • Identify Service Unit: Determine the systemd unit file name associated with your container's management process (e.g., docker.service or the pod controller).
  • Review Initial Logs: Check the logs immediately preceding a known failure time to see if any explicit signal messages are present.
journalctl -u docker.service --since "2 minutes ago" --no-pager

If using Kubernetes, check the pod events: kubectl describe pod -n .

STEP 2

Controlled Test Execution with strace

The most reliable method is to run the application or a wrapper script under strace while simulating a controlled termination. This captures all system calls, including signal reception.

  1. Prepare Test Environment: Ensure your service runs in a non-production, isolated container environment.
  2. Execute strace: Use strace attached to the process ID (PID) or run it directly if possible. We use the -f flag for tracing forks and focus on signal tracing:
# Assume 'my_app' is the entry point script inside the container context
docker run --rm --entrypoint /bin/bash your_image bash -c "sleep 10; trap SIGINT SIG_THREAD_EXIT; exec my_app" | strace -f -e trace=signal

When you manually interrupt this process (Ctrl+C), strace will dump the signal handling calls.

STEP 3

Analysis: Interpreting Signal Traces

After running the trace, examine the output for specific syscalls related to signals. Look for patterns indicating if the application received the signal call versus merely ignoring it.

  • Search for recvmsg(2) or prctl(PR_SET_NAME, ...)`: These calls indicate awareness of signals.
  • Look for Signal Numbers: A successful catch will show the process handling the signal number (e.g., SIGINT=2). If it proceeds without error after receiving the signal call, the handler is likely functional.

If strace shows no explicit signal reception calls during a controlled stop, the issue might be in the container runtime's process group signaling mechanism (PID 1 handling).

TROUBLESHOOTING

Common Pitfalls & Advanced Diagnostics

  • Pitfall: Overlooking Signal Masking: If the application code explicitly masks signals (using sigprocmask), it might never see the termination signal until a higher-level mechanism forces it.
  • Diagnostic: Resource Limits: Check if resource constraints (CPU/Memory limits in K8s) are causing an OOMKill before SIGTERM can be processed. Use docker stats or K8s metrics for this.
  • Pitfall: Signal Chain Interruption: In complex setups, a parent process might terminate before sending the signal to the child container process group. Check the service manager logs (systemd/kubelet).
FINAL STEP

Verification, Results & Long-Term Stability Validation

To validate the fix (usually updating application signal handlers), repeat the controlled termination test. The expected outcome is that upon interruption, the process logs a clean shutdown message and exits with status code 0.

  • Re-run Test: Execute the strace command again during a controlled stop.
  • Verify Exit Code: The application must exit gracefully, ideally logging messages like "Shutting down connections..." before termination.
  • Preventative Maintenance: Always implement signal handlers (e.g., using trap SIGTERM SIGINT TERM_HANDLER in Bash/Shell) within your container's entrypoint script to ensure cleanup logic runs regardless of the originating signal.
Resolution Complete: The ability to trace signals using strace combined with comprehensive logging via journalctl allows administrators to pinpoint whether the failure is due to signal misinterpretation within the application code or an orchestration layer communication gap. If symptoms persist, examine kernel logs (/var/log/kern.log) for lower-level OOM or hardware errors.

Frequently Asked Questions

Q: What is the difference between SIGINT and SIGTERM?

A: SIGINT (Signal Interrupt, value 2) is typically generated by user interruption (like Ctrl+C). SIGTERM (Termination Signal, value 15) is the standard, polite request from an orchestrator to shut down gracefully.

Q: Should I use strace on every container restart?

A: No. Use it only when debugging a suspected signal handling failure during controlled testing. It adds significant overhead and verbosity for normal operations.

Q: How do I force a process to ignore SIGTERM in the test?

A: You can use kill -SIGSTOP followed by monitoring, or modify the application's signal handler within the container entrypoint script to explicitly call signal(SIGTERM, SIG_IGN) (though this is generally discouraged for production).

COMMAND LINE / TERMINAL
journalctl -u docker.service --since "2 minutes ago" --no-pager
docker run --rm --entrypoint /bin/bash your_image bash -c "sleep 10; trap SIGINT SIG_THREAD_EXIT; exec my_app" | strace -f -e trace=signal

Frequently Asked Questions

What is the difference between SIGINT and SIGTERM?

SIGINT (Signal Interrupt, value 2) is typically generated by user interruption (like Ctrl+C). SIGTERM (Termination Signal, value 15) is the standard, polite request from an orchestrator to shut down gracefully.

Should I use strace on every container restart?

No. Use it only when debugging a suspected signal handling failure during controlled testing. It adds significant overhead and verbosity for normal operations.

How do I force a process to ignore SIGTERM in the test?

You can use kill -SIGSTOP <PID> followed by monitoring, or modify the application's signal handler within the container entrypoint script to explicitly call signal(SIGTERM, SIG_IGN) (though this is generally discouraged for production).

Engineering Assistance

Need help implementing this solution?

Our cybersecurity and infrastructure engineers can audit, harden, or configure your systems directly.

Request Technical Assistance โ†’
Gurveer Singh
Gurveer Singh โœ“ Verified Author
Founder & CEO at hSECURITIES | Cybersecurity Educator | Security Researcher | Ethical Hacking Enthusiast | Technology Mentor
Meet the Founder โ†’

Gurveer Singh Saggu is the Founder and CEO of hSECURITIES, a cybersecurity and technology education platform focused on practical learning, digital security awareness, hardware diagnostics, Linux administration, and real-world cybersecurity training.

With a strong passion for technology, ethical hacking, and security education, he has spent years helping students, professionals, and organizations understand modern cyber threats and develop practical technical skills. Through hSECURITIES, he creates hands-on learning resources, technical guides, security awareness content, and community-driven initiatives that make cybersecurity more accessible.

His mission is to bridge the gap between theoretical knowledge and practical implementation by providing step-by-step learning experiences, technical troubleshooting resources, and security-focused educational platforms.

Community Discussion & Confirmations (0)

Technical questions, hardware findings, and verified fix reports.

No reader questions posted yet. Have you tried this solution or experienced an edge case? Post below.

Ask a Question or Share Solution