When you move email-dependent automation from “one shared mailbox” to “inbound email as an API,” the first architectural decision is almost always the same: use a custom domain email setup (usually a dedicated subdomain) so your tests, agents, and QA pipelines get predictable routing, allowlisting compatibility, and clean environment separation.
This guide focuses on the parts that tend to break in real systems:
- Subdomain layout choices that scale across environments and tenants
- MX records (what they really control, and how to verify them)
- Recipient routing patterns (how an email address maps to an inbox or workflow)
- Delivery to code (webhook-first, polling fallback) and what to log for reliability
If you want Mailhook’s canonical integration contract for LLMs and developers, start with llms.txt.
What “custom domain email for automation” actually means
In automation, you typically do not want human-style mailboxes (IMAP, logins, folders). You want:
- A domain you control (or a delegated subdomain)
- Inbound mail to be routable to your automation provider
- A deterministic mapping from recipient to an isolated inbox or run
- A machine-readable message payload (JSON) delivered to your code
That breaks down into three routing layers:
| Layer | What it decides | Common failure mode | What to verify |
|---|---|---|---|
| DNS (MX) | Which mail servers receive mail for your domain/subdomain | Wrong MX target, wrong priority, missing MX |
dig MX, sending a test message |
| Recipient mapping | Which logical inbox/run should receive a given recipient address | Collisions, catch-all noise, alias drift | Address scheme, normalization rules, audit logs |
| Delivery to code | How your app learns a message arrived | Flaky waits, duplicates, webhook spoofing | Webhook verification, idempotency, polling cursor |
A good design makes each layer explicit. A great design makes each layer observable.
Step 1: Pick a subdomain layout that won’t paint you into a corner
For automation, using a subdomain instead of your primary production domain is usually the safest default. It reduces blast radius, keeps policies separate, and makes environment isolation straightforward.
Recommended patterns
Environment subdomains (most common)
mail.dev.example.commail.staging.example.commail.ci.example.com
Delegated test zone (good for larger orgs)
- Create
test.example.comand delegate it to the team that owns automation. - Then use
ci.test.example.com,staging.test.example.com, etc.
Tenant subdomains (useful if you test per-customer routing)
cust-a.mail.example.comcust-b.mail.example.com
Practical rules
- Keep the automation domain distinct from your user-facing email domain.
- Make the domain choice configurable (so you can switch shared domains to custom domains without rewriting tests).
- Plan for parallelism from day one (CI runs, retries, agent sessions). A routing scheme that works for one test often collapses under concurrency.
Step 2: Configure MX records correctly (and understand what they do)
MX records tell the world where to deliver inbound email for a domain. If MX is wrong, nothing downstream matters.
MX basics that matter for automation
- If a domain has no MX, many senders fall back to the A/AAAA record (but you should not rely on this).
- MX “priority” is a preference order (lower number is higher priority).
- Use exactly what your inbound provider instructs you to use.
A minimal verification workflow
After setting MX:
- Query MX:
dig MX mail.ci.example.com - Send a real email to an address on that subdomain.
- Confirm your provider shows receipt (or your webhook receives an event).
If you are debugging “email never arrives,” check MX first, then check whether the email was actually sent to the subdomain you configured.
Do you need SPF/DKIM/DMARC for inbound-only domains?
If the domain is inbound-only (you never send mail “from” it), SPF/DKIM/DMARC are usually not the gating factor for receiving messages. What matters is correct MX routing and deterministic inbox mapping.
If you also send mail from that domain (even occasionally), then SPF/DKIM/DMARC alignment can matter a lot. Keep send-path domains and inbound-test domains separate when possible.
Step 3: Decide how recipients map to inboxes (routing patterns)
Once MX routes mail to your inbound provider, you still need a deterministic rule for mapping:
recipient email address → inbox (or run/session)
There are three patterns you will see repeatedly in automation systems.
Pattern A: Encoded local-part (stateless keys)
You encode a routing key into the local-part:
The provider decodes the key and routes to the right inbox/run.
Pros: Scales well, no database lookup required, great for parallel CI.
Cons: Must design encoding carefully (allowed characters, length, versioning).
Pattern B: Alias tables (stateful mapping)
You create a recipient address and explicitly map it to an inbox:
-
[email protected]→inbox_id=abc123
Pros: Easy to reason about, good when you need friendly addresses.
Cons: Requires storage and cleanup, can drift if not lifecycle-managed.
Pattern C: Catch-all plus correlation
You accept anything at the domain and use correlation rules to pick the right message.
Pros: Fast to start.
Cons: Easy to get wrong under retries and parallel runs, and it increases noise and risk.
Quick comparison table
| Routing pattern | Best for | Avoid when | Reliability notes |
|---|---|---|---|
| Encoded local-part | CI parallelism, high volume, agent sessions | You need human-readable addresses | Prefer deterministic keys, add versioning |
| Alias table | Integrations needing stable addresses | You cannot ensure cleanup | Treat aliases as resources with TTL |
| Catch-all | Low-stakes prototypes | Anything with retries, resends, or parallel CI | Enforce strict correlation and narrow matchers |
Step 4: Deliver inbound mail to code (webhooks first, polling fallback)
Once a message is routed to the correct logical inbox, you need a delivery contract that works in production automation:
- Low latency
- Deterministic waiting (no fixed sleeps)
- Duplicate-tolerant (at-least-once delivery happens in real systems)
- Secure (assume inbound email and inbound webhooks are hostile inputs)
Why webhook-first is the default
Webhooks turn email receipt into an event stream. For automation, that is typically what you want: the system reacts when a message arrives.
But webhooks can fail for transient reasons (deploys, cold starts, network partitions). That is why robust systems add polling as a fallback.
What “polling fallback” should look like
A good poller has:
- A deadline (overall timeout budget)
- A cursor or “seen ids” dedupe strategy
- Backoff (do not hammer the API)
For deterministic automation, a key idea is: create an inbox per attempt (or per run), then wait for messages in that inbox only.
Step 5: Build for duplicates and retries from day one
Email delivery is not exactly-once. Your automation must handle duplicates across multiple layers:
- SMTP retries can cause multiple deliveries
- Providers may retry webhooks
- Your own job system may retry processing
A practical approach is to define idempotency keys at the right layer:
- Delivery id (unique per webhook delivery attempt)
- Message id (stable per email message)
- Artifact hash (stable per extracted OTP/link)
Even if you do not expose these exact fields publicly, the concept matters: you want a stable unit of “already processed” so your agent does not click the same link twice.
Step 6: Treat inbound email as untrusted input (especially for LLM agents)
LLM agents make email workflows powerful, and riskier.
Security basics for agent-driven email automation:
- Do not render HTML in an agent context. Prefer
text/plainor a sanitized extraction. - Validate URLs before fetching (avoid SSRF and open redirects).
- Minimize what the model sees: extract only the needed artifact (OTP or verification URL) instead of passing the whole message.
- Verify webhook authenticity (signatures, timestamp tolerance, replay protection).
These are not “nice to have.” They are how you prevent prompt injection and malicious content from turning into tool actions.
Putting it together with Mailhook (custom domain + JSON + webhooks)
Mailhook is designed around programmable, disposable inboxes: create inboxes via API, receive inbound messages as structured JSON, and integrate via real-time webhooks (with polling available when you need it). It supports shared domains for fast starts and custom domain support when you need allowlisting or governance.
If you are implementing custom domain email for automation with Mailhook, the most reliable architecture is:
- Use a dedicated subdomain for automation
- Point that subdomain’s MX records to your inbound provider (Mailhook)
- Create an isolated inbox per run or per attempt via API
- Deliver inbound messages to your system via webhook, and verify signed payloads
- Keep polling as a fallback for rare webhook disruptions
For the exact API surface and current contract, use Mailhook’s llms.txt.
Troubleshooting checklist (the failures teams hit most often)
Symptom: “Email never arrives”
Most common causes:
- MX is set on
example.combut you are sending tomail.example.com(or the reverse). - DNS changes have not propagated.
- The sending system is using a different recipient than you think (envelope recipient vs
To:header differences can surprise you).
Symptom: “The wrong test consumed the email”
Most common causes:
- Catch-all domains without strict correlation
- Reusing the same recipient address across parallel runs
- A shared inbox model instead of inbox-per-attempt
Symptom: “Webhook delivered twice”
This is normal in at-least-once systems.
Fixes:
- Acknowledge fast, process async
- Make handlers idempotent
- Dedupe using stable identifiers
Frequently Asked Questions
Do I need a custom domain email setup for automation, or can I use a shared domain? You can start with a shared domain for speed, but custom domains help with allowlisting, governance, and isolating reputation and environments.
Should I use a subdomain or my primary domain for inbound automation? A dedicated subdomain is usually safer because it isolates automation from production mail policies and reduces operational risk.
What is the most reliable routing pattern for CI and agent workflows? Encoded local-parts (stateless keys) plus an inbox-per-attempt model is typically the most parallel-safe approach.
Are MX records enough to “prove” the domain works? MX records only prove routing intent. You still need to send a real email end-to-end and verify your system receives and processes it deterministically.
How should LLM agents consume verification emails safely? Prefer structured JSON, extract minimal artifacts (OTP or URL), verify webhook signatures, and validate URLs before any fetch or click action.
Make custom-domain email automation deterministic with Mailhook
If you want inbound email to behave like a reliable API primitive (not a flaky side effect), Mailhook gives you the building blocks: programmable disposable inboxes, emails as structured JSON, real-time webhooks with signed payloads, plus polling when you need a fallback.
Start here:
- Read the canonical contract: Mailhook llms.txt
- Explore the product: Mailhook
