Exclude certain requests from the Nginx access log

Logs are nice and all that, but sometimes certain entries are there just to fill up the logs or are cluttering them. Here’s a few ways to exclude requests – by URL or visitor IP – from the Nginx access log.

Exclude specific URLs

In my Nginx configs I usually have a location block like this for static resources. It makes sure correct caching headers are sent, but also turns off logging for static resources – both the access log and the error log when a 404 is returned:

location ~* \.(png|jpg|jpeg|gif|ico|woff|otf|ttf|eot|svg|txt|pdf|docx?|xlsx?)$ {
    access_log off;
    log_not_found off;
    expires max;
    add_header Pragma public;
    add_header Cache-Control "public";
    add_header Vary "Accept-Encoding";
}

(I handle JS and CSS files in a separate block. Also note that you can very well use a map instead of the awkwardly long regexp)

Exclude specific user agents

If you’re not interested in logging requests from certain bots/crawlers, or you have a monitoring service, like Pingdom.

map $http_user_agent $log_ua {

    ~Pingdom 0;
    ~Googlebot 0;
    ~Baiduspider 0;

    default 1;
}

server {
       
    […]

    access_log /var/log/nginx/access.log main if=$log_ua;

}

Note that maps must be created outside of the server block.

Exclude specific IP addresses

If you run a cronjob with curl/wget on your localhost polling your website at certain intervals (e.g. you run WordPress but use a real cronjob to poll wp-cron.php), or you have health checks from downstream.

map $remote_addr $log_ip {
    
    "127.0.0.1" 0;
    "10.0.0.2" 0;
    "10.0.0.3" 0;

    default 1;

}

server {
       
    […]

    access_log /var/log/nginx/access.log main if=$log_ip;

}

Combining tests

If you want to match only specific user agents coming for specific IP addresses, it is possible to combine the tests. However, the Nginx config does not support a logical and operator, so we have to resort to a “clever hack”. Here’s a config to check for certain user agents from certain IPs:

map $http_user_agent $log_ua {

    ~Pingdom 0;
    ~Googlebot 0;
    ~Baiduspider 0;

    default 1;
}

map $remote_addr $log_ip {
 
    "127.0.0.1" 0;
    "10.0.0.2" 0;
    "10.0.0.3" 0;

    default 1;

}

server {
 
    […]

    set $logging 1;
    set $logtest '';

    if ( $log_ua = 0 ) {
        set $logtest "${logtest}A";
    }
    if ( $log_ip = 0 ) {
        set $logtest "${logtest}B";
    }
    if ( $logtest = "AB" ) {
        set $logging 0;
    }

    access_log /var/log/nginx/access.log main if=$logging;

}

You can of course add a “test C” checking for the URL too.


By the way: Check out the WordPress conference WordCamp Oslo 2018


There are 3 comments

Your email address will not be published. Required fields are marked *