Automating Log File Rotation and Archiving on Linux with logrotate
This guide teaches administrators how to implement robust, automated log file management on Linux servers using the powerful 'logrotate' utility combined with custom shell scripting for reliable archival.
Direct Diagnostic & Remediation Summary
- Create Config File: Create a new configuration file in /etc/logrotate.d/
detailing rotation parameters. - Test Rotation Manually: Run
sudo logrotate -d /etc/logrotate.confto perform a dry run test.
Log files are essential for auditing and debugging, but if left unmanaged, they will consume all available disk space, leading to critical service failures. logrotate is the standard utility designed to compress, archive, and delete old logs automatically. This guide walks through creating a custom rotation policy.
The primary issue is uncontrolled log growth (log bloat). When applications continuously write to a single file without an expiry mechanism, the disk partition hosting these logs will eventually fill up. This leads to I/O errors, application crashes, and system instability, often presenting as vague 'No space left on device' errors.
Logrotate solves this by managing the lifecycle: it rotates (renames) the active log file, creates a new empty one for the service to use, compresses older copies, and eventually deletes them based on defined rules.
Identifying the Target Log File and Service
Before configuring anything, you must know which service generates the log (e.g., Apache, Nginx, custom application) and the exact path to its log file. Never guess this path.
- Action: Check the application's documentation or configuration files (e.g.,
/etc/nginx/nginx.conffor Nginx) to find the preciseaccess_loganderror_logpaths.
For this guide, we assume a custom service logging to /var/log/my_app/app.log.
Creating the Custom logrotate Configuration File
It is best practice to keep service-specific configurations separate from the main system files. We will create a new file in /etc/logrotate.d/.
- Open or create the configuration file using an editor like Nano or Vim:
sudo nano /etc/logrotate.d/myapp - Paste the following structure, customizing paths and counts as necessary:
/var/log/my_app/app.log {
daily # Rotate daily
missingok # Do not error if the log file is missing
rotate 30 # Keep 30 days of logs (30 rotations)
compress # Compress older logs using gzip
delaycompress # Only compress the second-to-last log, allowing quick access to recent history
create 640 root adm # Create new file with specified permissions/owner
rotate 7 # Keep at least 7 uncompressed copies for immediate viewing
weekly # Optional: Can be added if you prefer weekly rotation over daily
Explanation of Directives:
daily/weekly/monthly: Defines the frequency.rotate N: How many old logs to keep.compress: Reduces disk usage by compressing older files.
Testing the Configuration and Manual Execution
Never let a new configuration run unattended on production hardware first. Always test.
- Dry Run Test (Simulation): This command reads your rules but performs no actual file changes, making it safe.
sudo logrotate -d /etc/logrotate.conf - Force Execution Test: To force the rotation logic to run against all configured files without relying on cron timing:
sudo logrotate -f /etc/logrotate.conf
If the dry run shows warnings or unexpected behavior, review your directives in /etc/logrotate.d/myapp.
Common Pitfalls & Advanced Diagnostics
If rotation fails silently or logs are not being created correctly, check these areas:
- Permission Issues: Ensure the user/group specified in
creatematches the application's running context. Usels -ld /var/log/my_app/to verify directory permissions. - Cron Job Failure: Logrotate usually runs via a system cron job (often at
/etc/cron.daily/logrotate). If this job is disabled, the automation fails. Check system logs (journalctl -u cron.service) for failures related to logrotate execution. - Log File Ownership: If the application runs as a specific user (e.g.,
www-data), ensure thecreatedirective reflects this ownership, not just root/adm.
If compression fails, manually check if gzip is installed on the system via sudo apt install gzip or equivalent.
Verification, Results & Long-Term Stability Validation
After successful manual execution (Step 3), verification is key. Log into the server a day later and check:
- Check File Count: Verify that older logs have been compressed (e.g.,
app.log.1.gz). - Check Disk Usage: Run
df -h /var/logto confirm the disk usage has not increased unexpectedly, and old files are being pruned when the retention count is hit.
For long-term stability, schedule a quarterly review of this configuration file. If you change applications or log paths, update /etc/logrotate.d/myapp immediately.
sudo nano /etc/logrotate.d/myapp
sudo logrotate -d /etc/logrotate.conf
sudo logrotate -f /etc/logrotate.conf
Frequently Asked Questions
What is the difference between 'compress' and 'delaycompress'?
<code>compress</code> immediately compresses the rotated file (e.g., app.log.1 becomes app.log.1.gz). <code>delaycompress</code> leaves the first backup uncompressed for one cycle, only compressing it on the subsequent rotation, which is useful if you need to quickly inspect the immediate previous log without decompression overhead.
Where does logrotate run automatically?
By default, most modern Linux distributions execute logrotate via a cron job scheduled within the daily maintenance scripts (often found in /etc/cron.daily/). This ensures it runs once per day regardless of manual intervention.
What if I need to keep logs for 90 days, but only rotate weekly?
Set the frequency directive to <code>weekly</code> and set the rotation count high enough, such as <code>rotate 13</code> (since 90 days / 7 days per week โ 12-13).
Can logrotate handle logs from multiple services?
Yes, you can create a separate file for each service in /etc/logrotate.d/, or combine them into one large configuration block, ensuring proper separation with comments.
Need help implementing this solution?
Our cybersecurity and infrastructure engineers can audit, harden, or configure your systems directly.
Community Discussion & Confirmations (1)
Technical questions, hardware findings, and verified fix reports.