Linux & Command Line ADVANCED Linux โฑ๏ธ 4 min read ๐Ÿ“… Updated: 15 Sep 2026

Systematic Monitoring for Open File Descriptor Exhaustion (lsof & Ulimit)

This guide teaches advanced Linux system administrators how to proactively monitor, diagnose, and prevent application failures caused by running out of available file descriptors (FDs) using the <code>lsof</code> utility and adjusting resource limits via <code>ulimit</code>.

Gurveer Singh
Gurveer Singh โœ“ Verified Specialist
Founder & Chief Architect ยท hSECURITIES Labs
โšก FAST-TRACK PROTOCOL

Direct Diagnostic & Remediation Summary

  • Check Current Limits: Run ulimit -n to see the current process limit for open files.
  • Monitor Usage: Use lsof | wc -l combined with process ID (PID) filtering to count active descriptors.
  • Increase Limit (Temporary): Run ulimit -n in the current shell session for testing.
๐Ÿ› ๏ธ Tools & Prerequisites: A Linux server or workstation with root/sudo privileges. Basic familiarity with the Bash shell and process management tools.
Systematic Monitoring for Open File Descriptor Exhaustion (lsof & Ulimit)

File descriptors (FDs) are a finite resource managed by the operating system kernel. When an application opens too many files, network sockets, or pipes without closing them properly, it can hit the process or system-wide limit, leading to cryptic failures like "Too many open files." This guide provides the methodology to diagnose and resolve this critical operational bottleneck.

๐Ÿ” Root Cause & Diagnostic Overview:File descriptor exhaustion occurs when a process exceeds its allocated resource ceiling. The primary mechanisms involve two layers: 1) The per-process limit, controlled by ulimit -n (or systemd unit settings), and 2) The kernel's global limits. Poor application design (resource leaks) or high connection volumes are the root causes. Diagnosis requires quantifying current usage against configured ceilings.
STEP 1

Initial Triage & Environment Isolation

The first step is to determine if the issue is systemic (OS level) or localized (specific application/user). Always start by checking resource utilization on a known failing process.

  • Check Current Limits (Shell): Run ulimit -n. This shows the soft limit for the current shell session.
  • Check System-Wide Limits (Root): Examine /etc/security/limits.conf to see if system-wide hard limits are enforced for users or groups.
  • Identify Target PID: Use ps aux | grep to find the Process ID (PID) of the problematic service.
STEP 2

Diagnostic Scans & Hardware Isolation

Use advanced tools to quantify exactly what is consuming the descriptors for that specific PID.

  • Count Open Files per PID: Execute lsof -p | wc -l. This provides a raw count of opened resources associated with the process ID.
  • Analyze Resource Types: Run lsof -p | grep '(/dev/null|\.pipe)' to see if sockets or pipes are contributing heavily, pointing toward networking issues versus file I/O leaks.
  • Check System Usage (Global): Use cat /proc/sys/fs/file-nr. This shows the total number of allocated FDs vs. system capacity, indicating global pressure.
STEP 3

Targeted Remediation & Fixes

Remediation involves two paths: temporary band-aid (increasing limits) or permanent fix (code remediation).

A. Temporary Limit Increase (Testing):

If the system is stable enough for a test, increase the limit for the current shell session:

ulimit -n 65536

For persistent changes across reboots or services, edit /etc/security/limits.conf and add lines like: @users -n hard nofile 1048576 (This example sets a high limit for all users).

B. Permanent Service Fix (Recommended):

If the service runs via systemd, modify its unit file (/etc/systemd/system/.service) and add or modify the LimitNOFILE directive:

[Service]
LimitNOFILE=1048576
[Install]
WantedBy=multi-user.target

After editing, reload systemd: sudo systemctl daemon-reload and restart the service: sudo systemctl restart .

TROUBLESHOOTING

Common Pitfalls & Advanced Diagnostics

  • Pitfall: Running as Wrong User: If you run ulimit -n as a non-root user, the change might only apply to your session. Always confirm service restarts use appropriate privilege escalation (sudo/systemd).
  • Pitfall: Kernel Overwrite: If system limits are set via kernel boot parameters (sysctl), ulimit may be overridden. Check /etc/sysctl.conf for settings like fs.file-max.
  • Advanced Diagnosis: Network Leaks: If the leak is socket-related, investigate application connection pooling or failure to call close() on sockets within the source code (requires developer intervention).
FINAL STEP

Verification, Results & Long-Term Stability Validation

After applying changes (especially systemd unit file modifications), verification is crucial. The service must be restarted to pick up new limits.

  • Re-run Diagnostic Scan: Repeat the lsof -p | wc -l command during peak load simulation. The count should remain significantly below the new limit.
    Example: If the old max was 1024 and the new limit is 1,048,576, the usage must not approach 1024 again.
  • Monitor System Logs: Check /var/log/syslog or journalctl for any related "Resource temporarily unavailable" errors that were previously occurring.
  • Preventative Maintenance: Schedule regular checks (e.g., a cron job running the diagnostic scan every hour) to catch slow memory leaks before they cause outages.
Resolution Complete: File descriptor exhaustion is resolved by raising the system/service limits using ulimit -n or systemd's LimitNOFILE directive, and ensuring applications are coded to properly close all resources upon completion. If symptoms persist, a code audit for resource leaks is mandatory.
COMMAND LINE / TERMINAL
ulimit -n
lsof -p <PID> | wc -l
cat /proc/sys/fs/file-nr
ulimit -n 65536
sudo systemctl daemon-reload
sudo systemctl restart <service_name>

Frequently Asked Questions

What is the difference between ulimit and LimitNOFILE in systemd?

<code>ulimit -n</code> sets the limit for an interactive shell session. The <strong>LimitNOFILE</strong> directive within a systemd unit file configures this resource restriction specifically for the service managed by systemd, making it persistent across reboots.

Should I increase ulimit if my application is leaking files?

No. Increasing the limit only postpones failure. The correct long-term fix is to debug and patch the application code to ensure every opened resource (file handle, socket) is properly closed using <code>try-finally</code> blocks or equivalent constructs.

How do I check if a process is holding too many sockets?

Use <code>lsof -i :<port_number> | wc -l</code> to count active connections on a specific port, combined with the PID from <code>lsof -p <PID> | grep 'TCP'</code> for detailed connection analysis.

Engineering Assistance

Need help implementing this solution?

Our cybersecurity and infrastructure engineers can audit, harden, or configure your systems directly.

Request Technical Assistance โ†’
Gurveer Singh
Gurveer Singh โœ“ Verified Author
Founder & CEO at hSECURITIES | Cybersecurity Educator | Security Researcher | Ethical Hacking Enthusiast | Technology Mentor
Meet the Founder โ†’

Gurveer Singh Saggu is the Founder and CEO of hSECURITIES, a cybersecurity and technology education platform focused on practical learning, digital security awareness, hardware diagnostics, Linux administration, and real-world cybersecurity training.

With a strong passion for technology, ethical hacking, and security education, he has spent years helping students, professionals, and organizations understand modern cyber threats and develop practical technical skills. Through hSECURITIES, he creates hands-on learning resources, technical guides, security awareness content, and community-driven initiatives that make cybersecurity more accessible.

His mission is to bridge the gap between theoretical knowledge and practical implementation by providing step-by-step learning experiences, technical troubleshooting resources, and security-focused educational platforms.

Community Discussion & Confirmations (0)

Technical questions, hardware findings, and verified fix reports.

No reader questions posted yet. Have you tried this solution or experienced an edge case? Post below.

Ask a Question or Share Solution