How to Configure a ModSecurity WAF on Nginx (2026 Guide)

If your Nginx-powered site or application is sitting on a dedicated server or VPS with no application-layer filtering, you are exposed. Every SQL injection attempt, cross-site scripting (XSS) payload, and bot-driven scan hits your backend directly.

A self-hosted Web Application Firewall (WAF) closes that gap. The best part? You don't need an expensive managed WAF subscription to get enterprise-grade cybersecurity.

In this guide, you'll build a production-ready ModSecurity WAF on Nginx from source. We will load the OWASP Core Rule Set (CRS) and confirm it is actually blocking malicious requests to protect your web application security.

What You'll Learn

What You'll Learn

• A working ModSecurity v3 installation compiled against your existing Nginx build.

• The OWASP Core Rule Set (CRS) integrated to block OWASP Top 10 vulnerabilities.

• A tested configuration that returns a 403 Forbidden error on SQLi and XSS probes.

• A clear workflow for reading audit logs and fixing false positives.

Why ModSecurity v3 + Nginx?

ModSecurity has a split history worth understanding. Versions before 3.0 were built as an Apache module. Getting that version to work with Nginx meant relying on a wrapper layer that was never fully stable for production reverse proxy setups.

ModSecurity v3 solved this by rewriting the rule engine as a standalone library (libmodsecurity). Nginx now talks to that library through a dedicated connector module.

The practical result is fewer moving parts, better performance under DDoS or high traffic loads, and a setup that's fully supported for modern Nginx deployments.

Prerequisites

Before you start, confirm you have:

• A dedicated server or VPS running Ubuntu 24.04, Debian 12, or AlmaLinux 9.

• Root or sudo access.

• Nginx already installed and serving traffic.

• Comfort with basic Linux command-line operations (compiling from source, editing config files).

Because ModSecurity v3 isn't packaged in the default repositories for these distributions, you'll be compiling it from source. This guarantees you're running the current security patches rather than a stale distro package.

Step 1: Installing Required Dependencies

ModSecurity's build process pulls in several libraries for regex handling, XML parsing, and Lua rule support. Install them upfront so the build doesn't fail halfway through.

On Ubuntu 24.04 / Debian 12:

sudo apt update
sudo apt install -y git build-essential autoconf automake libtool \
  libcurl4-openssl-dev liblua5.3-dev libfuzzy-dev libpcre2-dev \
  libxml2-dev libpcre3-dev libyajl-dev doxygen libgeoip-dev \
  libssl-dev pkgconf zlib1g-dev

On AlmaLinux 9:

sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y git autoconf automake libtool curl-devel \
  lua-devel pcre2-devel libxml2-devel yajl-devel doxygen \
  GeoIP-devel openssl-devel zlib-devel

(Note for Ubuntu 24.04 users: libpcre3-dev still resolves correctly for compatibility, but ModSecurity's build now leans primarily on PCRE2 — hence both packages above.)

Step 2: Compiling and Installing libmodsecurity

With dependencies in place, clone the official repository and build the library itself.

cd /opt
sudo git clone --depth 1 -b v3/master https://github.com/owasp-modsecurity/ModSecurity
cd ModSecurity
sudo git submodule init
sudo git submodule update

Run the bundled build script, which prepares the autotools configuration, then compile:

sudo ./build.sh
sudo ./configure
sudo make -j$(nproc)
sudo make install

The make -j$(nproc) flag parallelizes the build across your available CPU cores. Expect this step to take a few minutes; you're compiling a full C++ rule engine.

Step 3: Compiling the Nginx ModSecurity Connector (The Safe Way)

ModSecurity connects to Nginx through a purpose-built dynamic module. This module must be compiled with the exact same configuration flags as your currently running Nginx, otherwise, it will crash.

First, check your exact Nginx version and current configuration arguments:

nginx -V

Look closely at the output. You will see your Nginx version (e.g., nginx/1.24.0) and a long list of text starting with configure arguments:. Copy everything after configure arguments: and paste it somewhere safe.

Now, download the matching source code for your Nginx version:

cd /opt
wget https://nginx.org/download/nginx-<your-version>.tar.gz
tar -xzvf nginx-<your-version>.tar.gz

Clone the connector module alongside it:

sudo git clone --depth 1 https://github.com/owasp-modsecurity/ModSecurity-nginx

Now build it as a dynamic module. Run ./configure using your copied arguments, and append --add-dynamic-module=../ModSecurity-nginx at the very end.

cd nginx-<your-version>
./configure <PASTE_YOUR_COPIED_ARGUMENTS_HERE> --add-dynamic-module=../ModSecurity-nginx
make modules

Copy the resulting .so file into your Nginx modules directory:

sudo cp objs/ngx_http_modsecurity_module.so /usr/lib/nginx/modules/

(Check nginx -V for your --modules-path if your distro places Nginx modules elsewhere).

Step 4: Configuring Nginx to Use ModSecurity

Tell Nginx to load the module. At the top of /etc/nginx/nginx.conf, before the http {} block, add:

load_module modules/ngx_http_modsecurity_module.so;

Next, set up ModSecurity's base configuration. Copy the recommended config that ships with the source:

sudo mkdir -p /etc/nginx/modsec
sudo cp /opt/ModSecurity/modsecurity.conf-recommended /etc/nginx/modsec/modsecurity.conf
sudo cp /opt/ModSecurity/unicode.mapping /etc/nginx/modsec/

Open /etc/nginx/modsec/modsecurity.conf and find this line:

SecRuleEngine DetectionOnly

Change it to:

SecRuleEngine On

This is the most critical step. DetectionOnly logs suspicious requests but lets them through. On means ModSecurity will actively block requests that match a malicious rule.

Inside your http {} block in nginx.conf, point Nginx at this config:

modsecurity on;
modsecurity_rules_file /etc/nginx/modsec/main.conf;

Step 5: Integrating the OWASP Core Rule Set (CRS)

ModSecurity is just an engine; it needs rules to know what a payload looks like. The industry standard is the OWASP Core Rule Set (CRS).

Clone the current CRS release:

cd /etc/nginx/modsec
sudo git clone --depth 1 https://github.com/coreruleset/coreruleset crs
cd crs
sudo mv crs-setup.conf.example crs-setup.conf
sudo mv rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf.example rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
sudo mv rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf.example rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf

Renaming the exclusion files ensures future CRS updates won't overwrite your custom whitelists. (Pro Tip: The crs-setup.conf file is also where you can tweak the Anomaly Scoring threshold later if you need to adjust strictness).

Build the main.conf that Nginx references:

sudo tee /etc/nginx/modsec/main.conf > /dev/null <<'EOF'
Include /etc/nginx/modsec/modsecurity.conf
Include /etc/nginx/modsec/crs/crs-setup.conf
Include /etc/nginx/modsec/crs/rules/*.conf
EOF

Test the Nginx configuration and reload:

sudo nginx -t
sudo systemctl reload nginx

Step 6: Testing Your New WAF

Let's find out if it works. Two quick curl tests will confirm your setup.

Test 1 — A normal request:

curl -I http://localhost/

Expected output: HTTP/1.1 200 OK.

Test 2 — A simulated SQL injection (SQLi):

curl "http://localhost/?id=1' OR '1'='1"

Expected output: HTTP/1.1 403 Forbidden.

If Test 2 returns 200 OK, double-check that SecRuleEngine is set to On and reload Nginx.

You can also test a basic XSS probe:

curl "http://localhost/?search=<script>alert(1)</script>"

A 403 Forbidden here confirms the CRS is actively blocking XSS payloads.

Step 7: Managing Logs and False Positives

A WAF that blocks legitimate traffic is an outage. You must know how to read the logs and whitelist safe requests.

ModSecurity writes detailed audit logs to /var/log/modsec_audit.log. Tail it in real-time:

sudo tail -f /var/log/modsec_audit.log

Reading the log: Look for Section H. It lists the Rule ID that matched, the message, and the severity. You will use this Rule ID to create exceptions.

Writing a custom exception: If a legitimate admin panel triggers a false positive, add a targeted exclusion to REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf:

SecRule REMOTE_ADDR "@ipMatch 203.0.113.10" \
    "id:1000,phase:1,pass,nolog,ctl:ruleRemoveById=941100"

This bypasses Rule 941100 specifically for the IP 203.0.113.10. Always test and reload Nginx after adding exceptions (sudo nginx -t && sudo systemctl reload nginx).

Conclusion

You now have a self-hosted WAF running in front of Nginx, powered by ModSecurity v3 and the OWASP Core Rule Set. It is actively dropping SQLi and XSS payloads at the edge.

During the first few weeks, monitor modsec_audit.log closely to tune exceptions for your specific web applications.

To take your security posture even further, pair this WAF setup with our guides on Configuring Immutable Backups on a Dedicated Server and Building a Zero-Trust Dedicated Server to ensure full-stack protection.