Server Stories 5 min read

Setting Up a Mail Server Without Losing Your Mind

Postfix, Dovecot, SPF, DKIM, DMARC — the real story of getting emails into inboxes instead of spam folders, from someone who learned it the hard way.

Nobody warns you that email is the hardest part of running your own server. Harder than security. Harder than the database. Harder than getting your app to run in production. I found out after sending a hundred emails that silently disappeared — not bounced, not filtered, just gone from the sender's perspective and never seen by the recipient.

Why Self-Hosted Email

The honest reason is cost. Domain email hosting adds up across multiple domains. Running your own mail server on a VPS you already have costs nothing extra. Privacy is a secondary benefit — you're not pushing business email through a third-party provider. And there's a certain completeness to running the full stack yourself: web server, database, application, and mail.

The hidden cost is setup time. Budget more of it than you think.

The Stack

What I'm running:

  • Postfix — handles outbound email (SMTP). The industry standard for Linux mail sending.
  • Dovecot — handles inbound email storage and IMAP. Lets clients like Outlook or the phone app connect.
  • SnappyMail — lightweight webmail UI. Runs in a browser when you're not using a mail client.
  • Caddy — reverse proxies the webmail on a subdomain with automatic HTTPS.

Installing these isn't the hard part. The hard part is DNS.

SPF, DKIM, DMARC — What They Are and Why You Need All Three

When an email arrives at Gmail or Outlook, the receiving server checks whether it should trust the sender. These three DNS records are the answer:

SPF (Sender Policy Framework) — a DNS TXT record that lists which IP addresses are allowed to send email from your domain. If your server's IP isn't on the list, the email is likely to be marked as suspicious.

DKIM (DomainKeys Identified Mail) — a cryptographic signature attached to every email you send. The receiving server checks your DNS for the public key and verifies the signature. If it matches, the email hasn't been tampered with in transit.

DMARC (Domain-based Message Authentication) — a policy record that tells receiving servers what to do when SPF or DKIM checks fail. Quarantine, reject, or do nothing. It also sends you aggregate reports about who's sending email from your domain.

"Getting all three right took days. One wrong DNS record and everything went to spam. The order of troubleshooting matters: SPF first, then DKIM, then DMARC."

The Things That Still Catch People Out

PTR record (reverse DNS) — your server's IP should resolve back to your mail hostname. This is configured at the hosting provider level, not in your domain's DNS. Easy to miss, significant impact on deliverability.

New IP reputation — a fresh VPS IP has no sending history. Large mail providers are cautious about unknown IPs. Don't send bulk email from a brand new server. Establish a sending history gradually.

Port 25 blocking — some VPS providers block outbound port 25 (SMTP) by default to prevent spam abuse. Check with your provider before assuming it's a configuration issue on your end.

Gmail filters — even with perfect SPF/DKIM/DMARC, Gmail can filter email based on user-level rules. If someone has set a filter on their Gmail that catches your domain, the email disappears silently. It's not a server problem.

Testing Deliverability

Before trusting that your setup works, test it properly. Mail-Tester.com gives you a score and tells you exactly what's failing. Send to a Gmail address and check the full headers — they'll show whether SPF and DKIM passed. Check MXToolbox for your DNS records.

A score of 10/10 on Mail-Tester doesn't guarantee every email arrives in every inbox — but it means your configuration is correct and the rest is outside your control.

Is It Worth It?

For company email on your own domain: yes, once it's working. For transactional email at scale (password resets, notifications for many users): probably use a dedicated service like SendGrid or Mailgun — their deliverability infrastructure is better than any self-hosted server for high volume.

The knowledge you get from running your own mail server — how email actually works, why deliverability is hard, what each DNS record does — is genuinely useful even if you later switch to a managed service. You'll understand why the managed service needs the DNS records it asks for, and you'll know when something's wrong before your users do.

Monitor your mail server with Sentinel

Sentinel checks your mail server's SSL certificate, monitors connection status, and alerts you before your email setup stops working.

Explore Sentinel

SL
Sławomir Luzny
Founder, FixFlex LTD — West London

General builder by day, software builder by night. No CS degree — just AI tools, a live server, and the stubbornness to figure things out. Founder of Sentinel and 24ad.info.

Back to Blog