Apache logs (part 2): Format of error logs. Module event log

6 April 2023 11 minutes Author: Endpool

Log or magazine

Logs (log files) are files that contain system information about the operation of a server or computer, to which certain user or program actions are recorded. Sometimes the Russian analog of the concept – magazine is also used. Their purpose is to log the operations performed on the machine for further analysis by the administrator. Regular viewing of logs will allow you to identify errors in the operation of the system as a whole, a specific service or site (especially hidden errors that are not displayed when viewing in a browser), diagnose malicious activity, and collect site visit statistics. In general, a log is invaluable information that can be used by any computer user to find out what happened on the site, in the system or in the machine itself, and at what point in time.

To do this, you need to look at the error log. Error logs are useful in many ways. In the case of servers and office networks, error logs track problems encountered by users and help in analyzing the root causes of those problems. A network or system administrator can troubleshoot errors more quickly and easily with information available from error logs. For webmasters, error log analysis provides insight into the issues users are experiencing and can proactively resolve issues without reporting them to anyone. Error logs can also provide insight into hacking attempts, as most hacking attempts on systems and servers result in errors or have a high probability of ending up in error logs as hackers attempt to compromise the system. Therefore, knowing this information and using it in practice, you will be able to track all problems and eliminate them in time.

Apache error logs

The server error log, whose name and location are specified by the ErrorLog directive, is the most important log file. This is where Apache httpd will send diagnostic information and log any errors it encounters when processing requests. It’s the first place to look when there’s a problem with starting up or running the server, as it often details what went wrong and how to fix it. The error log is usually written to a file (usually error_log on Unix systems and error.log on Windows and OS/2). On Unix systems, it is also possible for the server to send errors to the syslog or pass them to the application. The format of the error log is defined by the ErrorLogFormat directive, which allows you to configure which values are written to the log. If you don’t specify one, the default value is used. A typical log message is:

The first item in the log entry is the date and time of the message. Next is the module that generates the message (in the case of authz_core) and the severity level of that message. This is followed by the process ID and, if necessary, the thread ID of the process in which the condition occurred. Next, we have the address of the client that made the request (its IP address and the port number from which the connection was opened). And finally, a detailed error message that the server refused the connection.

Placing the %L token in both the error log and the access log creates a log entry ID that can be used to map an error log entry to an access log entry. If mod_unique_id is loaded, its unique request ID will also be used as the log entry ID. During testing, it is often useful to constantly monitor the error log for problems.

On Unix systems, you can do this using a command like:

Example:

The ErrorLog directive

Sets the location where the server will log errors.

Syntax:

Default value:

Context: server config, virtual hosts. The ErrorLog directive sets the name of the file in which the server will log any errors it encounters.

If the file path is not absolute, it is assumed to be specified relative to ServerRoot:

If the file path starts with the pipe character “|” then this is assumed to be the command to call the error log:

Using syslog instead of filename allows logging via syslogd(8) if the system supports it and mod_syslog is loaded. By default, the syslog local7 facility is used, but you can override this using the syntax syslog:facility, where facility can be one of the names usually documented in syslog(1). The tool is actually global, and if it is changed on individual virtual hosts, this last tool affects the entire server. The same rules apply to the syslog tag, which defaults to the Apache binary name, in most cases httpd. You can also override this using the syslog::tag syntax.

Add-ons can provide their own ErrorLog providers. The syntax is similar to the syslog example above.

Note: When entering a file path on platforms other than Unix, care must be taken to use only forward slashes, even if the platform may allow backslashes. In general, it is recommended to always use a forward slash in configuration files.

The ErrorLogFormat directive

Defining the format for writing the error log.

Syntax:

Context: server config, virtual hosts. ErrorLogFormat allows you to specify what additional information is written to the error log in addition to the actual log message.

Specifying a connection or request as the first parameter allows additional formats to be specified, resulting in additional information being logged when the first message is logged for a particular connection or request, respectively. This additional information is logged only once per connection/request. If a connection or request is processed without notifying the log, no additional information is logged either.

Some format string elements may not be inferred. For example, the Referer header is only present if a log message is associated with the request, and the log message appears when the Referer header has already been read from the client. If no output is produced, the default behavior is to remove everything from the previous whitespace to the next whitespace. This means that the log string is implicitly split into fields at transitions between non-whitespace and whitespace. If the format string element does not output, the entire field is omitted. For example, if the remote address %a in log format [%t] [%l] [%a] %M is not available, the surrounding brackets are also not logged. Space characters can be escaped with a backslash so that they cannot delimit the field. The combination ‘%’ (percentage and space) is a zero-width field delimiter that makes no inference. This behavior can be changed by adding modifiers to the format string element. Modifier – (minus) results in writing a minus if the corresponding element does not make any conclusion. One-per-connection/request formats can also use the + (plus) modifier. If the element with the plus modifier makes no inference, the entire line is omitted. A modifier number can be used to assign a log severity level to a format element. An item will only be logged if the severity of the log message is not greater than the specified log severity level.

The number can vary from 1 (alert) to 4 (warn) and 7 (debug) to 15 (trace8). For example, here’s what happens if you add modifiers to the %{Referer}i token that registers the Referer request header (Screenshot 1). Some date format elements accept additional parameters in curly brackets (Screenshot 2 and Screenshot 3).

Screenshot 1.

Screenshot 2.

Screenshot 3.

The %L log identifier format creates a unique identifier for the connection or request. This can be used to plot which log terms belong to the same connection or to query which query is occurring with which connection. The %L format string is also available in mod_log_config to allow correlating log access entries with log error times. If mod_unique_id is loaded, its unique ID will match the query log’s ID quality.

This may result in error messages such as:

Note that, as discussed above, some fields are omitted entirely because they are undefined:

The LogLevel directive

Controls the verbosity of the ErrorLog.

Syntax:

Default value:

Context: server config, virtual hosts, directories Compatibility: Module-level and directory-level configuration are available in Apache HTTP Server 2.3.6 and later. LogLevel controls the details of the messages that are written to the error logs (see the ErrorLog directive). The following levels are available in descending order of importance (Screenshot 4 and Screenshot 5).

Screenshot 4.

Screenshot 5.

If a level is specified, messages from all other higher priority levels will also be reported. For example, if LogLevel info is specified, messages with notice and warn levels will also be published. Please note that the 404 (file not found) error messages generated by the web server (core) itself (Screenshot 6) have the info status:

Screenshot 6.

This means that by default (LogLevel is set to warn) requests that end with a 404 status will not end up in the error logs!

To fix this, you need to set the level to info:

Note that if a file being processed by another module is not found, that module may set its own level. For example, the php7 module, if no PHP script is found, will set the severity level to error for such a message and such an entry will go to the error log even with the default settings:

A level of at least crit (or lower critical) is recommended.

Example:

Note: When writing to a regular file, messages with a notice level cannot be suppressed and are therefore always logged. However, this is not the case when logging is done using syslog. Specifying no module name will reset the level for all modules up to that level. Specifying level with a module name will set the level of that module only. You can use the name of the module’s source file, the module ID, or the module ID with the trailing _module omitted as a module specification.

This means that the following three specifications are equivalent:

You can also change the level for each directory:

Directory-level configuration for each directory affects only the messages that are logged after the query is parsed and related to it. Log messages related to the server or connection are not affected. However, the latter can be affected by the LogLevelOverride directive.

LogLevelOverride directive

Override ErrorLog verbosity for certain clients.

Syntax:

Context: server config, virtual hosts. Compatibility: Available in Apache HTTP Server 2.5.0 and later. LogLevelOverride configures the LogLevel for requests coming from specific client IP addresses. This allows verbose logging to be enabled for specific test clients only. The IP address is checked very early in connection processing. So LogLevelOverride allows you to change the logging level for things like SSL handshakes that happen before the LogLevel directive is evaluated in the <If> container. LogLevelOverride accepts either a single IP address or a CIDR/subnet_length IP address specification. For queries that match the LogLevelOverride directive, the per-directory LogLevel specifications are ignored.

Examples:

LogLevelOverride only affects log messages related to the request or connection. Log messages related to the server are not affected.

Module event log

The LogLevel directive allows you to specify the threshold level of importance of logging an entry for the skin module. That way, if you’re only troubleshooting one particular module, you can increase the volume of its information in the log without getting additional information about other modules you don’t care about. This is especially useful for modules like mod_proxy or mod_rewrite where you want to know the details of what they are trying to do and what is going on inside them.

Do this by specifying the module name in your LogLevel directive:

This sets the main LogLevel to info, but mod_rewrite will make it trace5. This replaces the per-module logging directives such as RewriteLog that were present in previous versions of the server. Note that information generated by modules always ends up in the error bar, even if it’s not actually an error! Also note that some modules will not show any information unless you set the trace level to be between trace1 and trace8.

Found an error?
If you find an error, take a screenshot and send it to the bot.