Creating Timestamped Log Files From Crontab Jobs
What is the Core Issue with Cron Job Logging?
Cron jobs run in the background without terminal output, making logging of their activities difficult. However, logging cron job output is essential for monitoring job health, auditing actions, and troubleshooting failures. By default, any output or errors from cron jobs are not captured. To save this output, we need to explicitly redirect it to a file from the cron definition.
The main challenges with logging cron output are:
- Ensuring output is captured and not lost as jobs run unattended
- Including useful context like timestamps for when jobs run
- Preventing logs from growing too large or outpacing available storage
- Centralizing scattered log files for easier analysis
Fortunately, Linux provides flexible tools for addressing all these logging needs – from redirecting job streams to log rotation and aggregation. By leveraging this functionality, we can implement robust cron logging that provides the audit trail and diagnostics needed for stable automation.
Redirecting Output to a Log File
The most basic step for cron logging is redirecting job output streams to a file instead of letting them disappear. This is done directly in the crontab definition using Linux’s standard output redirection operators.
For example, consider this crontab entry running a custom script every 5 minutes:
*/5 * * * * /path/to/my_script.sh
To log its output, we append redirection like so:
*/5 * * * * /path/to/my_script.sh > /var/log/my_cron.log 2>&1
Breaking this down:
- > /var/log/my_cron.log – Redirects standard output (stdout) to the log file
- 2>&1 – Redirects standard error (stderr) to stdout, which then goes to the log
Now all output and errors from my_script.sh will be captured in /var/log/my_cron.log each time the cron job runs. This basic step widely expands visibility into otherwise opaque background jobs.
Timestamping Log File Names
A shortcoming of logging to a single static filename like my_cron.log is that old content gets overwritten as jobs continue running. Without timestamps, determining chronology requires parsing the log text itself.
A cleaner approach is generating unique log filenames based on current dates and times. This way, each invocation of a job appends to a distinct file, avoiding overwrite issues while still providing clear indicators of when each run occurred.
For example, we could enhance my_cron.sh itself to write to timestamped log files like this:
#!/bin/bash LOG=/var/log/my_cron-$(date +\%Y\%m\%d-\%H\%M).log # Rest of my_cron.sh script goes here # Redirect all output to unique log file exec 1>$LOG 2>&1
The key additions:
- Unique $LOG filename with current timestamp from date command
- exec 1>$LOG redirects all following stdout output to dynamic $LOG
This would result in separate logs for each run like:
/var/log/my_cron-20200101-0000.log /var/log/my_cron-20200101-0005.log /var/log/my_cron-20200101-0010.log
Giving us both chronology and absence of overwrites. The concept can be extended to include additional data like process IDs in the log names where useful.
Log Rotation for Timestamped Logs
A risk of granular timestamped logging is accumulating excess logs over time that take up considerable space. This warrants implementing log rotation – automatically deleting or archiving batches of old logs as new ones are created.
Linux hosts a built-in logrotate utility that can handle this cleanly. To set up logrotate for a cron job like my_cron.sh:
- Create a rotation config file – /etc/logrotate.d/my_cron
- Define log location, rotation frequency, compression, etc.
For example:
/var/log/my_cron-*.log {
daily
rotate 30
compress
delaycompress
missingok
}
This rotates logs after 30 days by compressing older ones. logrotate runs daily via cron to enforce these policies. With logrotate managing cleanup, storage usage is contained while retaining recent history needed to observe job trends.
Alternatively, a custom script can handle log rotation by periodically deleting older logs based on crafted fileName patterns rather than relying on an external utility.
Centralized Logging with rsyslog
For larger scale cron job logging across multiple systems and jobs, retaining all logs on their originating hosts incurs admin overhead for aggregation and monitoring. An optimized approach is centralizing logs from dispersed sources into unified repositories.
The rsyslog utility provides robust centralized logging capabilities for Linux environments. After installing the rsyslog daemon on a dedicated logging server, client hosts can be configured to forward select logs to the server – including our cron job trails.
Key features of an rsyslog logging server include:
- Accepting log data via TCP/UDP from any client
- Applying log parsing rules to shape and process streams
- Routing logs to various files, databases, forwarders based on source tags
- Alerting admins when matched error conditions occur
- Scheduled log archiving to compress older logs
With streamlined log forwarding to an rsyslog server, insights into cron job outcomes can be extracted much easier without having to access individual clients. rsyslog scales to aggregate logs from as many clients as needed while retaining a rich querying mechanism for historical analysis.
Troubleshooting Unexpected Cron Output
Once cron job logging is enabled, log data becomes invaluable when jobs begin failing or exhibiting other issues. Let’s walk through some sample log entries from possible problems and how to investigate them.
Script Fails with Non-Zero Exit Code
Many cron problems manifest as exit codes in log output. Scripts communicate completion status via exit codes, with non-zero codes indicating failures. For example:
my_cron.sh: line 10: unexpected error reading database my_cron.sh exited with status code 2
This reveals an error we can search through the script to pinpoint failed operations. We can additionally correlate timing of first failures with other system changes.
Missing Log Entries
/var/log/my_cron: 20200101-0100.log 20200101-0300.log ...
Gaps in consecutive timestamped logs implies job runtime issues. Most likely the job starts but crashes prematurely before redirects can populate logs. Add profiling instrumentation to determine where execution halts.
Permission Errors
my_cron.sh: cannot open /opt/logs/mylog.log: Permission denied
Login context changes or ownership modifications may cause new file access issues. Inspect script privileges and add ACL rules as needed.
Parsing error messages and exit codes reveals starting points to explore culprits behind cron problems. Gradually add debugging handles like variable dumps if needed to zero in on root causes.
