Introduction To Syslog Log Levels/Priorities
A very common question about syslog is how to decide the appropriate log priority (a.k.a. log level) for a specific log message. Deciding on the correct priority depends on a number of different factors.
Syslog allows you define a facility and a log level for each individual log message. The syslog “facility” is used to separate out log messages by application or by function. For example, email logs are normally logged on the LOG_MAIL facility. This groups all email related logs together. Your systems administrator will assign you a log facility to use. You should not use an arbitrary facility.
Log priorities are not as cut and dry. For custom applications, developers and systems administrators need to work together to define what constitutes a certain priority of a message. In the most generic terms possible, syslog levels are defined as:
LOG_EMERG system is unusable LOG_ALERT action must be taken immediately LOG_CRIT critical conditions LOG_ERR error conditions LOG_WARNING warning conditions LOG_NOTICE normal, but significant, condition LOG_INFO informational message LOG_DEBUG debug-level message
Before digging into specifics on the priority definitions, let me address the developers directly. These priorities are defined in syslog.h as follows:
#define LOG_EMERG 0 /* system is unusable */ #define LOG_ALERT 1 /* action must be taken immediately */ #define LOG_CRIT 2 /* critical conditions */ #define LOG_ERR 3 /* error conditions */ #define LOG_WARNING 4 /* warning conditions */ #define LOG_NOTICE 5 /* normal but significant condition */ #define LOG_INFO 6 /* informational */ #define LOG_DEBUG 7 /* debug-level messages */
Normally, your application should call syslog() from some sort of internal logging function/method. This allows you to set an application-wide maximum log level. For example, on a production system, you do not want to waste CPU cycles generating debug error messages. Your application’s logging function should specify a maximum log level and filter the messages internally via some configuration variable.
For instance, in production, you may only care about messages of priority LOG_ERR and higher. So you would specify via some variable that your maximum logging level is LOG_ERR (3) and if a message comes into your logging function with a priority of 4 or greater, it is ignored.
The syslog server has the ability to filter messages of specific priorities as well. Systems administrators may choose to log only LOG_ERR or higher. So if the application is generating LOG_DEBUG error messages and the syslog server is only logging LOG_ERR or higher, this is just wasted processing time for both the application, the syslog server, and possibly even network I/O.
But the main question here is what is the most appropriate log level/priority for a particular message. Again, this is a hard question to answer because it can vary wildly by application, but in general, I would define them as such:
- LOG_EMERG – The application has completely crashed and is no longer functioning. Normally, this will generate a message on the console as well as all root terminals. This is the most serious error possible. This should not normally be used applications outside of the system level (filesystems, kernel, etc). This usually means the entire system has crashed.
- LOG_ALERT – The application is unstable and a crash is imminent. This will generate a message on the console and on root terminals. This should not normally be used applications outside of the system level (filesystems, kernel, etc).
- LOG_CRIT – A serious error occurred during application execution. Someone (systems administrators and/or developers) should be notified and should take action to correct the issue.
- LOG_ERR – An error occurred that should be logged, however it is not critical. The error may be transient by nature, but it should be logged to help debug future problems via error message trending. For example, if a connection to a remote server failed, but it will be retried automatically and is fairly self-healing, it is not critical. But if it fails every night at 2AM, you can look through the logs to find the trend.
- LOG_WARNING – The application encountered a situation that it was not expecting, but it can continue. The application should log the unexpected condition and continue on.
- LOG_NOTICE – The application has detected a situation that it was aware of, it can continue, but the condition is possibly incorrect.
- LOG_INFO – For completely informational purposes, the application is simply logging what it is doing. This is useful when trying to find out where an error message is occurring during code execution.
- LOG_DEBUG – Detailed error messages describing the exact state of internal variables that may be helpful when debugging problems.
So as an application developer, you may be asking yourself why you should not be using LOG_EMERG or LOG_ALERT. This is a valid question and this depends on you working with your systems administrator to determine if these log levels are appropriate. By default, almost every syslog implementation will log all LOG_EMERG and LOG_ALERT messages to the console which can make it difficult to actually work on a system to fix the problem if the log messages are flying by on the screen. Your systems administrator can set up filters on the syslog server to log them to the appropriate place, but before using those two priority levels, you should definitely consult with your systems administrator.
LOG_CRIT should be reserved for error messages that actually need to be visible to systems administrators and/or developers. If the error message you are logging will be ignored by everyone receiving the error, it should not be considered critical. Excuse the tautology, but “Critical errors are critical.” A critical error requires user intervention. If it does not require user intervention, it should be logged as LOG_ERR.
Priorities of LOG_WARNING and lower should be used at the developer’s discretion. It is common practice not to log any message with a priority lower than LOG_ERR on a production system.