Contact Us

Home / Engineering Blog / DevOps

DevOpsEngineering notes

Logging in Angie

Reliable operation of any service rests on its logging system. Logs solve a wide range of problems: troubleshooting, user behavior analytics, security incident investigation, system health monitoring, and more. This article looks at what server logs in Angie can do and at practical techniques for working with them.

Video version

A video version of this article is available on YouTube.

Request log (access_log)

Let's start with the request log. One caveat up front: for now we cover Angie as an HTTP server or a load balancer. Operation in Stream mode (TCP/UDP proxying and balancing) gets its own section.

Logging is the last phase in the processing of any request. This matters especially for long-running requests, which may be in processing for a long time, while their record appears in the log only after they finish.

A default Angie install gives us two request log formats: main and extended. The main format writes the main log (typically /var/log/angie/access_log). Here is part of the standard angie.conf configuration:

http {
   log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    log_format extended '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" rt="$request_time" '
                        '"$http_user_agent" "$http_x_forwarded_for" '
                        'h="$host" sn="$server_name" ru="$request_uri" u="$uri" '
                        'ucs="$upstream_cache_status" ua="$upstream_addr" us="$upstream_status" '
                        'uct="$upstream_connect_time" urt="$upstream_response_time"';

}

This snippet contains two log_format directives, which define the contents of the request log formats. Each format has a name (main, extended) and a set of fields, usually defined through built-in variables. To make parsing a log by fields easier you can use delimiters (for example, quotes around "$http_user_agent") or key=value-style markers (for example, urt="$upstream_response_time"). Angie defines a basic combined log format by default; it is used unless you specify otherwise.

The format you define depends on the task. The general rule: keep as much request data as possible for later analysis. But in the desire to keep this data you inevitably run into storage and transport limits. For example, the standard main format has a small number of fields, enough to record the fact of a request and a few client properties.

192.168.122.1 - - [07/Apr/2026:12:11:10 +0000] "GET / HTTP/1.1" 200 65413 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36" "-"

But this says nothing about how the proxied server is doing, has no timing metrics, and it is unclear which domain name was used in the request. With the extended format we get fuller data. Change the configuration line in angie.conf to:

access_log  /var/log/angie/access.log  extended;

Don't forget to test and reload the configuration:

angie -t
systemctl reload angie

Now the same request is logged in more detail:

192.168.122.1 - - [07/Apr/2026:12:16:20 +0000] "GET / HTTP/1.1" 200 65413 "-" rt="0.039" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36" "-" h="wp.metodlab.ru" sn="wp.metodlab.ru" ru="/" u="/index.php" ucs="-" ua="unix:/run/php/php8.3-fpm.sock" us="200" uct="0.000" urt="0.039"

This message now shows timing metrics, the domain name, and proxied server parameters. If the server runs as a reverse proxy or a load balancer, this format is clearly preferable and can serve as the general server log when several sites are hosted on it. It is enough for standard server control tasks.

If you have a specific task, a custom log format may help. Say we need the request body in the log (the $request_body variable). Define a special log format:

http {

log_format postdata '$remote_addr - $time_local - $request - $request_body';

}

Now enable logging of such requests. You usually do not need to log every request with this format, so we can define the access_log directive at location level.

server {
  listen 80;

  location / {
    root /var/www/html;
    access_log /var/log/angie/post.log postdata;
  }
}

In this configuration we added an access_log directive inside the location, and it applies to requests that hit it. Such requests will not land in the server-wide log — in other words, we have overridden request logging at location level. If you need to keep both journals, add a second access_log directive:

server {
  listen 80;

  location / {
    root /var/www/html;
    access_log /var/log/angie/post.log postdata;
    access_log  /var/log/angie/access.log  extended;
  }
}

For servers with several sites you can take different approaches to combining logs:

  • a separate log per site;
  • one shared server log;
  • a mix of the shared log and per-site logs.

The first approach splits logs across directories and makes it easier to find requests when the site is known. The second suits assessing the state of the server as a whole, especially when looking for anomalies. The third combines the advantages of both, but this comes at the cost of higher logging overhead.

Besides the classic delimiter-separated log line, you can define a log format of JSON messages. The escape parameter of log_format comes in handy:

log_format logger-json escape=json
'{'
'"source": "angie",'
'"message":"angie log captured",'
'"time": $time_iso8601,'
'"resp_body_size": $body_bytes_sent,'
'"host": "$http_host",'
'"address": "$remote_addr",'
'"request_length": $request_length,'
'"method": "$request_method",'
'"uri": "$request_uri",'
'"status": $status,'
'"user_agent": "$http_user_agent",'
'"resp_time": $request_time,'
'"upstream_addr": "$upstream_addr"'
'}';

With this configuration the output is JSON, which is convenient to ship to centralized storage systems.

Historically, request logging was second-precise. At high request rates that precision is not enough to identify request-related events in the system. Starting with version 1.12, Angie lets you set a custom time format with time_format:

http {
  time_format $time_ms "%Y-%m-%dT%H:%M:%S.%L%Z";
  log_format tf '$time_ms $remote_addr "$request" $status';
}

As the example shows, the time_format directive creates a $time_ms variable, which is used in the definition of the tf log format that includes it.

Optimizing request logging

Let's talk about reducing the load from logging. The first technique helps if the server handles static files in separate locations and we are willing to sacrifice data about those requests. For most web applications the number of static requests is several times higher than dynamic ones, so the reduction in log volume is very noticeable. It is enough to turn off logging of static requests:

server {
  listen 80;

  location / {
    proxy_pass http://backend;
  }
  location /static/ {
    access_log off;
    root /var/www/html;
  }
}

In this configuration we turned off logging of requests in location /static/, while the rest of the site inherits the general request log.

Another method of reducing the load from logging: buffering and compression on write. Buffering delays the write to disk until the buffer accumulates, and compression reduces the volume written to disk (at the cost of CPU usage). Enabling compression turns on buffering automatically. An example with both parameters:

access_log  /var/log/angie/access.log.gz extended gzip=3 flush=1s buffer=64k;

This directive writes a compressed log (gzip level 3) with a 64 KB buffer and a forced flush once per second, even if the buffer is not full. The downside of a compressed log is that inspecting the latest entries on the fly is awkward. To do that you first unpack the log and then take the last lines. For example:

zcat access.log.gz | tail

Buffering can also be used without compression, which lets you flush data to disk less often and create fewer block I/O operations.

Finally, you can reduce the load from logging by setting a condition for writing to the journal. For example, you can create a special variable that acts as an indicator of the requests you want and put it in the log write condition:

http {
  map $status $loggable {
    ~^[3]  1;
    default 0;
  }
  access_log /var/log/angie/access-redir.log redir if=$loggable;
}

This example sets a $loggable variable that has the value 1 when the response code starts with the digit 3. The variable is then used for the if parameter of the access_log directive. A value of 0 or an empty string is interpreted as "false". This log will contain only requests with a 3xx response code.

For error logs, version 1.12 added the ability to limit the rate of message writes (the rate parameter):

error_log logs/error.log error rate=100;

The rate parameter limits the rate of message writes per second, which reduces the load on the disk system. The default is 1000 messages per second.

If working with logs locally does not suit you, you can set up shipping logs with the syslog protocol. Both a local service and a remote server will do. An example for Ubuntu 24.04:

access_log syslog:server=unix:/run/systemd/journal/socket,nohostname;

Here the records go to the rsyslogd UNIX socket, and messages land in the file /var/log/syslog by default. On most modern distributions the primary logging system is journald with its own binary format for storing messages. You can write straight into the systemd-journald service:

access_log syslog:server=unix:/dev/log,nohostname extended;

In this case the logs can be viewed with the journalctl utility.

journalctl -u angie

This shows all messages related to the angie.service systemd unit. So far we have talked only about the request log, but error messages matter just as much. Let's go over them in more detail.

Error log (error_log)

Error logging is a critical task for any server. For that reason, the error log configured with the error_log directive differs significantly from the request log setup.

Since the error log has to record error messages no matter what, it cannot be disabled entirely. By default, even if there is no error_log directive in the configuration, the following settings apply:

error_log logs/error.log error;

Note that the server-wide error_log directive is placed in the main context. This is the log to search for problems when the server starts.

All errors are grouped by message level, from the most detailed (debug) to the most critical (emerg). With the error level as in the example above, we see all errors of level error or more important (crit, alert, emerg). For fuller control over server events it is better to set the importance level to info or notice.

error_log  /var/log/angie/error.log notice;

This is useful when working with various modules: Limit Req (request rate limiting), ACME (automatic TLS certificate issuance), and others.

For individual sites you can also define your own error logs; the logic here mirrors request logs.

You can cut the volume of error logging by turning off messages about missing files (the 404 error):

server {
  log_not_found off;
}

In this example, file search error logging is disabled at site level. Keep in mind one non-obvious property of the try_files directive, which is used for conditional file delivery: when a file lookup fails inside try_files, that error does not reach the error log.

The debug log (level debug) provides the maximum message detail and is suitable for solving specific problems, not for constant use. To enable this level you need to replace the executable by pointing the symbolic link at angie:

sudo ln -fs angie-debug /usr/sbin/angie
sudo systemctl restart angie

Keep in mind that the debug executable runs slower than the regular one and the number of messages written to the error log will be very large. To cut the number of debug messages you can limit the log's effect to specific clients with the debug_connection directive:

error_log logs/error.log error;
events {
  debug_connection  127.0.0.1;
}

So the general log keeps the error level, while local connections get the debug log.

Previously the error log format could not be configured, but since version 1.12 you can switch the format to JSON. It is enough to set the format parameter in the error_log directive:

error_log logs/error.log error format=json;

The log line will look like this:

{"time":"2026-07-20T13:27:30.723Z","level":"error","pid":3415,"tid":3415,"connection":27,"message":"open() \"/var/www/html/static/dddd3ddd\" failed","error":{"code":2,"message":"No such file or directory"},"http":{"client":"192.168.122.1","request":{"server":"angie.metodlab.ru","request_line":"GET /static/dddd3ddd HTTP/1.1","host":"192.168.122.141"}},"tags":["http","static_err"]}

Note that the timestamp format includes milliseconds.

Also starting with version 1.12, Angie added the ability to set tags on error messages, which the messages can then be filtered by. Tags support variables, and they are visible in log messages in JSON format. For example, you can set up logging for a chosen location, logging errors only for the HTTP/2 and HTTP/3 protocols into separate files:

server {
    location /static/ {
        error_log /var/log/angie/error-h2.log filter=tag:h2;
        error_log /var/log/angie/error-h3.log filter=tag:h3;
        error_log_user_tag $http2;
        error_log_user_tag $http3;
    }
}

In this configuration, errors from the /static/ location will land in the log /var/log/angie/error‑static.log. When configuring error_log on nested levels, remember the directive inheritance rule: the general error log for the /static/ location is no longer written; if you need it, add one more error_log directive with the required parameters.

Filtering of error messages can work in different modes:

  • logline — by the full text of the message (only for format=default);
  • message — by the message text without the system error;
  • tag — by the tag set with the error_log_user_tag directive;
  • sourcefile — by the source file that produced the message (only in the build with debug logging);
  • any log field — a field of the error message context, for example a JSON field or a plain format field (filter=client:127.0.0.1).

Different filtering modes for error logs make it much easier to find problems in application behavior and to monitor systems.

Logging for TCP/UDP servers

With the Stream (TCP/UDP) module, request logging is not done by default. To be more precise, there are no requests in this context — sessions are logged. So the available logging detail is much lower than with HTTP.

Let's create a basic access log:

stream {
    log_format basic '$remote_addr [$time_local] '
                 '$protocol $status $bytes_sent $bytes_received '
                 '$session_time' "$upstream_addr";

    access_log /var/log/angie/stream-access.log basic buffer=32k;
}

The log messages will look like this:

192.168.122.1 [20/Jan/2026:20:59:16 +0300] TCP 200 148203 134855 814.990192.168.122.142:8443
192.168.122.1 [20/Jan/2026:21:00:00 +0300] TCP 200 240 2066 0.005192.168.122.142:8443

You can see status codes that resemble HTTP (for example, 200), but this is only an analogy to simplify parsing; the possible values are listed in the description of the $status variable.

Parsing logs

Analyzing logs is a routine task for a server administrator. Standard Linux tools work well with text logs.

Usually it is convenient to build a pipeline of several filtering commands to get the result you need. The exact options depend on the log format. A few examples:

awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -20

This command builds a list of the 20 most active client IPs by request count.

For the extended format you can write regular expressions to find the most active clients that produce slow requests for the backend (more than 1 second) in the latest log entries:

tail -n 100000 access.log | fgrep -v 'urt="-"' | grep -P 'urt="[1-9]+' | sort | uniq -c | sort -nr | head -20

Find and count 404 errors (here the response status field is number 9):

awk '($9 ~ /404/)' access.log | awk '{print $7}' | sort | uniq -c | sort -rn

Constructs like these and similar to them extract all sorts of data, and the processing efficiency is quite high thanks to pipelining.

Summary

The Angie server provides extensive options for configuring request and error logging. Proper use of logs is a key server administrator skill, both for routine tasks and for responding to incidents.

← Back to all engineering notes