How to Self-Host a Mail Server with Mailcow on a Dedicated Server

Mailcow Dockerized packages a complete, production-grade email stack SMTP, IMAP, webmail, antivirus, antispam, and an admin panel into a single Docker deployment. Run it on a dedicated server instead of a shared VPS, and you also remove the two things that quietly sink most self-hosted mail projects: a shared IP with someone else's spam history, and unpredictable resource contention during a mail flood.

This guide walks through the full deployment: server requirements, installation, DNS configuration, hardening, backups, and the deliverability checks that determine whether your mail actually lands in the inbox instead of the spam folder.

Table of Contents

Why Run Mailcow on a Dedicated Server Instead of a VPS

Mailcow will install on any VPS that meets its resource minimums. The reason production mail deployments tend to move to dedicated hardware comes down to three things a shared virtual server can't guarantee.

A Clean, Dedicated IP Address

On a shared VPS node, your outbound mail can inherit the reputation of whatever another tenant on the same IP range was doing before you. A dedicated server gives your mail server its own IP with a reputation history that's entirely yours to build and protect.

No Resource Contention

Mailcow runs several containers concurrently Postfix, Dovecot, SOGo, MySQL, Redis, Rspamd, and ClamAV all at once. On an oversold VPS, a spam wave or a large attachment scan can compete with other tenants for CPU cycles. On a dedicated server, the resources you provision are the resources you get.

Full Control Over Reverse DNS (PTR)

Major receiving mail providers check that your server's IP resolves back to a matching hostname (a PTR record) before they'll trust your mail. Dedicated server environments typically give you direct, self-service control over this something budget VPS providers frequently gate behind a support ticket or don't offer at all.

Before you provision anything: confirm with your hosting provider that outbound port 25 is open on your plan. Several major VPS platforms block it by default to curb spam abuse, and it's the one prerequisite that will silently break your entire setup if you skip it.

Prerequisites Checklist

Confirm every item below before you touch the installer. Mailcow's own setup script will validate hardware and Docker version automatically, but the items outside the script — DNS, IP reputation, and port access are on you to check first.

Beyond hardware, confirm:

• A registered domain with DNS management access

• A static IPv4 address that doesn't change on reboot

• An IP with no existing blacklist history — check it at mxtoolbox.com before deployment, not after

• Ports 25, 465, 587, 143, 993, 110, and 995 reachable — covers SMTP, submission, IMAP, and POP3

• Root or sudo SSH access

Step 1: Prepare Your Dedicated Server OS and Docker Environment

Start with a clean OS install and update the system before installing anything else. Mailcow's containers expect a current package base to avoid dependency conflicts during the build.

ssh root@your-server-ip
apt update && apt upgrade -y

Set a proper fully qualified domain name (FQDN) for the server something like mail.yourdomain.com since this becomes the hostname your mail server identifies itself with during SMTP handshakes.

hostnamectl set-hostname mail.yourdomain.com

Then install Docker Engine and the Compose plugin if they aren't already present:

curl -fsSL https://get.docker.com | sh
apt install docker-compose-plugin -y

Step 2: Clone and Configure the Mailcow Dockerized Stack

Clone the official Mailcow repository into a dedicated directory and run the configuration generator. This script asks for your mail hostname and timezone, then performs a memory check on the host.

git clone https://github.com/mailcow/mailcow-dockerized
cd mailcow-dockerized
./generate_config.sh

If the script detects 2.5 GB of RAM or less, it will prompt you to disable ClamAV (SKIP_CLAMD=y) to prevent the container stack from running out of memory. This keeps the install functional on smaller servers, but you lose attachment antivirus scanning not a trade-off worth making on a production mail server if you can provision more RAM instead.

Step 3: Set Up Essential Email DNS Records (MX, SPF, DKIM, DMARC)

This is the step that actually determines whether your email gets delivered. Mail servers with a technically perfect Mailcow install still land in spam constantly because of missing or incorrect DNS records.

Generate your DKIM key from inside the Mailcow admin UI after your first login (covered in the next step), then publish it as a TXT record at your DNS provider. Skipping this single record is the most common reason self-hosted mail gets rejected outright by Gmail and Outlook.

Step 4: Launch Mailcow Containers and Access the Web Admin UI

Pull the container images and bring the stack up in detached mode:

docker compose pull
docker compose up -d

Once the containers are running, access the admin panel at https://mail.yourdomain.com. Log in with the default credentials (admin / moohoo) and change the password immediately this is not optional on a server that's about to be internet-facing.

From the admin panel:

• Add your domain under Configuration → Mail Setup → Domains

• Generate and copy your DKIM key, then publish it to DNS

• Create your first mailbox

Step 5: Test Email Deliverability and IP Reputation

Don't assume the server works because the admin UI loads. Send a real test email to an address at mail-tester.com and review the resulting authentication report it will flag any SPF, DKIM, DMARC, or PTR mismatch directly.

Separately, confirm your server IP isn't already on a blacklist using a checker such as mxtoolbox.com. A pre-blacklisted IP will fail delivery to Gmail and Outlook regardless of how correctly Mailcow itself is configured.

Step 6: Harden Your Mail Server Security and Firewall

A mail server is a permanent, internet-facing target. Treat hardening as part of the deployment, not an optional follow-up.

• Enable Fail2ban — bundled with Mailcow, blocks repeated failed login attempts automatically

• Restrict firewall rules to only the ports mail actually needs (25, 465, 587, 143, 993, 443)

• Enable two-factor authentication in the Mailcow admin UI for every admin account

• Keep containers current using Mailcow's built-in update.sh script on a regular schedule

• Rely on the built-in ACME container for automatic Let's Encrypt certificate renewal rather than managing TLS manually

Step 7: Automate Mailcow Backups for Disaster Recovery

Mailcow ships with its own backup and restore tooling in helper-scripts/backup_and_restore.sh, which captures mail data, databases, and configuration in a single run.

./helper-scripts/backup_and_restore.sh backup all /path/to/backup/location

Follow the standard 3-2-1 backup principle: three copies of your data, on two different storage types, with one copy stored off-server. A backup that only exists on the same disk as the mail server it's protecting isn't a backup it's a single point of failure. Test a full restore at least quarterly to confirm the backups are actually usable when you need them.

Routine Maintenance for Self-Hosted Email Servers

Self-hosted mail is not a "set it up and forget it" system. Build these into a recurring routine:

• Monitor disk usage as mailbox and database size grow set an alert well before 80% capacity

• Watch CPU and RAM under real load, especially during spam waves

• Review the Rspamd quarantine periodically for false positives

• Re-check your IP's blacklist status on a recurring basis, not just at launch

Mailcow vs. Managed Email: An Honest Comparison

Self-hosting isn't the right call for every team. It makes the most sense when you have the technical capacity to maintain it, and when control, privacy, or long-term cost at scale outweigh the convenience of a fully managed inbox.

Run Mailcow on Infrastructure Built for It

A self-hosted mail server is only as reliable as the hardware underneath it. BytesRack dedicated servers give you a clean, dedicated IP, full root access, and the guaranteed resources Mailcow needs to run Postfix, Dovecot, SOGo, and ClamAV without contention.