Logs are nice and all that, but sometimes certain entries are there just to fill up the logs or are cluttering them. Here’s a few ways to exclude requests – by URL or visitor IP – from the Nginx access log.

Exclude specific URLs

In my Nginx configs I usually have a location block like this for static resources. It makes sure correct caching headers are sent, but also turns off logging for static resources – both the access log and the error log when a 404 is returned:

location ~* \.(png|jpg|jpeg|gif|ico|woff|otf|ttf|eot|svg|txt|pdf|docx?|xlsx?)$ {
    access_log off;
    log_not_found off;
    expires max;
    add_header Pragma public;
    add_header Cache-Control "public";
    add_header Vary "Accept-Encoding";
}

(I handle JS and CSS files in a separate block. Also note that you can very well use a map instead of the awkwardly long regexp)

Exclude specific user agents

If you’re not interested in logging requests from certain bots/crawlers, or you have a monitoring service, like Pingdom.

map $http_user_agent $log_ua {

    ~Pingdom 0;
    ~Googlebot 0;
    ~Baiduspider 0;

    default 1;
}

server {
       
    […]

    access_log /var/log/nginx/access.log main if=$log_ua;

}

Note that maps must be created outside of the server block.

Exclude specific IP addresses

If you run a cronjob with curl/wget on your localhost polling your website at certain intervals (e.g. you run WordPress but use a real cronjob to poll wp-cron.php), or you have health checks from downstream.

map $remote_addr $log_ip {
    
    "127.0.0.1" 0;
    "10.0.0.2" 0;
    "10.0.0.3" 0;

    default 1;

}

server {
       
    […]

    access_log /var/log/nginx/access.log main if=$log_ip;

}

Combining tests

If you want to match only specific user agents coming for specific IP addresses, it is possible to combine the tests. However, the Nginx config does not support a logical and operator, so we have to resort to a “clever hack”. Here’s a config to check for certain user agents from certain IPs:

map $http_user_agent $log_ua {

    ~Pingdom 0;
    ~Googlebot 0;
    ~Baiduspider 0;

    default 1;
}

map $remote_addr $log_ip {
 
    "127.0.0.1" 0;
    "10.0.0.2" 0;
    "10.0.0.3" 0;

    default 1;

}

server {
 
    […]

    set $logging 1;
    set $logtest '';

    if ( $log_ua = 0 ) {
        set $logtest "${logtest}A";
    }
    if ( $log_ip = 0 ) {
        set $logtest "${logtest}B";
    }
    if ( $logtest = "AB" ) {
        set $logging 0;
    }

    access_log /var/log/nginx/access.log main if=$logging;

}

You can of course add a “test C” checking for the URL too.