Programmatic Monitoring of Filesystem Write Errors with Inotifywait and Bash
This guide teaches advanced Linux users how to set up a robust, programmatic watchdog script using inotifywait to detect and alert on specific filesystem write errors or file system changes in real-time.
Direct Diagnostic & Remediation Summary
- Install Dependencies: Ensure the necessary tools are present:
sudo apt update && sudo apt install inotutils bash python3. - Create Watchdog Script: Create a monitoring script, e.g.,
monitor_writes.sh, containing the logic to loop and callinotifywait.
inotify}$ subsystem in Linux provides a near real-time mechanism to monitor filesystem events (create, delete, modify, move). Write errors often manifest as failures during these events or specific metadata changes. We utilize inotifywait in conjunction with scripting logic to capture the event stream and process potential failure indicators or required actions upon detection.
Technical Mechanism: inotify}$ monitors file descriptors for kernel-level notifications. By specifying events like write, attrib, or close_write, we capture the moment data hits the filesystem buffer. Errors are often signaled by subsequent script logic analyzing the event payload or by trapping signals/exit codes if the underlying write operation fails outside of standard file operations (e.g., quota errors).System Preparation and Tool Verification
Before scripting, ensure inotifywait is installed and that you understand the basic event types. We will use a test directory to simulate write activity.
- Install Dependencies: Use your distribution's package manager to install necessary tools (e.g.,
inotutils). - Create Test Directory: Establish a controlled environment for testing the monitor script.
mkdir /tmp/fs_monitor_test
cd /tmp/fs_monitor_test
Developing the Core Monitoring Script (Bash)
The script must loop indefinitely, calling inotifywait with specific event flags and piping its output to a processing function that checks for error indicators or triggers alerts.
- Script Creation: Create the file
monitor_writes.sh. - Populate Script: Add the basic structure, including error handling within the loop. We focus on detecting WRITE and ATTRIB events as primary indicators of modification or metadata change.
#!/bin/bash
WATCH_DIR="/tmp/fs_monitor_test"
echo "Starting filesystem write monitor on $WATCH_DIR..."
while true; do
# Wait for specific events (write, attribute change) and capture the results
EVENT_DATA=$(inotifywait -e close_write,attrib --format '%w %e' "$WATCH_DIR")
if [ $? -ne 0 ]; then
echo "[ERROR] inotifywait failed to read event stream. Check permissions or directory existence." >&2
sleep 5 # Wait longer if the underlying system call fails
continue
fi
# Process the captured data line by line
echo "--- Event Detected ---"
echo "$EVENT_DATA" | while read -r path event; do
echo "[ALERT TRIGGERED] Path: $path, Events: $event"
# Placeholder for actual error checking logic (e.g., check file size delta or run fsck check)
if [[ "$event" == *"WRITE"* || "$event" == *"ATTRIB"* ]]; then
echo "Action Required: File modification detected at $path."
# In a real scenario, you would call an external logging/alerting API here.
fi
done
sleep 1 # Small pause to prevent excessive CPU usage if events are rapid
done
Implementing Simulated Error Triggering (Advanced)
Since inotifywait}$ reports events, not error codes directly, we simulate an error check by monitoring for specific file patterns or by integrating a secondary check (like checking inode status after the event). For true write failure detection (e.g., due to disk full/corruption), manual triggering is needed.
- Simulate Write: Open a separate terminal and write data to the test directory while the monitor runs in another terminal.
- Test Failure Condition (Manual): If you suspect quota issues, attempt writing massive amounts of data until an OS-level error occurs, observing if
inotifywait}$ fails or reports unusual events.
# Run the monitor script in Terminal 1:
bash monitor_writes.sh &
# In Terminal 2 (Simulating write):
echo "Test data" > /tmp/fs_monitor_test/testfile.txt
touch /tmp/fs_monitor_test/newfile.log
Securing and Running the Monitor Service
For continuous operation, the script must be managed by a service manager like systemd to survive reboots.
- Make Executable: Grant execution permissions to the script.
- Create Systemd Unit File: Create
/etc/systemd/system/fsmonitor.service.
[Unit]
Description=Filesystem Write Activity Monitor
After=network.target
[Service]
User=root
ExecStart=/bin/bash /path/to/monitor_writes.sh
Restart=always
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target- Enable and Start: Reload systemd, enable the service, and start it.
sudo systemctl daemon-reload
sudo systemctl enable fsmonitor.service
sudo systemctl start fsmonitor.service
Common Pitfalls & Advanced Diagnostics
If the script hangs or reports no activity, check these areas:
- Permissions Issue: The user running the service (e.g., 'User=root') must have read/write access to
$WATCH_DIR. - Inotify Limits: On heavily loaded systems, the kernel's inotify watch limit might be reached. Check limits using
cat /proc/sys/fs/inotify/max_user_watchesand increase it if necessary:sudo sysctl fs.inotify.max_user_watches=524288. - Script Blocking: If the script blocks indefinitely, ensure your event format (
--format '%w %e') is correctly capturing all necessary path and event information for parsing.
Verification, Results & Long-Term Stability Validation
To verify success, observe the journalctl}$ logs while simulating writes in a separate terminal. The log should show clean event detection and subsequent execution of your alert logic without crashing or stalling.
- Check Logs: Use
journalctl -u fsmonitor.service -fto watch the live output stream for successful event processing. - Stress Test: Run a large batch of file operations (writes, deletes) over several minutes to confirm sustained monitoring capacity and resource usage remain nominal.
- Preventative Maintenance: Periodically review
/etc/systemd/system/fsmonitor.serviceto ensure the$WATCH_DIRpath is still valid and accessible by the service user.
inotifywait}$ and systemd integration. The script now watches for filesystem write events in real time, allowing you to hook into custom alerting or logging routines when data changes occur.mkdir /tmp/fs_monitor_test
chmod +x monitor_writes.sh
sudo systemctl daemon-reload
sudo systemctl enable fsmonitor.service
sudo systemctl start fsmonitor.service
Frequently Asked Questions
What is the difference between 'write' and 'close_write' events?
<code>WRITE</code> fires when data is written to a file descriptor, while <code>CLOSE_WRITE</code> fires when the file containing the write is closed by the process. Using both provides comprehensive coverage for detecting persistence of changes.
Can this monitor detect physical disk failure?
No. This tool detects filesystem events. Detecting physical media failure requires hardware monitoring tools (like SMART checks) or kernel error logs (<code>/var/log/syslog}$), not just file system event watching.
How do I make the script run on boot automatically without systemd?
You can add an entry to the user's crontab using <code>crontab -e</code> and setting it to run at startup, though systemd is vastly superior for service management.
Need help implementing this solution?
Our cybersecurity and infrastructure engineers can audit, harden, or configure your systems directly.
Community Discussion & Confirmations (0)
Technical questions, hardware findings, and verified fix reports.
No reader questions posted yet. Have you tried this solution or experienced an edge case? Post below.