Test deliverability on every PR with GitHub Actions

Email templates get changed by the same people who change your application code: developers, designers, marketers with PR access. They make the same kinds of mistakes too — a broken placeholder, a subject line that triggers SpamAssassin, a Mailgun domain that silently stopped signing DKIM last week. Unlike application code, there is almost never a test that catches this before merge. This article fixes that with a single GitHub Actions workflow.

The workflow in one screenshot

When a developer opens a PR that edits emails/welcome.mjml, the workflow kicks off within a few seconds. About two minutes later a comment appears on the PR: Gmail 10/10 inbox, Outlook 9/10, Yahoo 8/10 — overall 89% inbox, SPF/DKIM/DMARC pass, SpamAssassin 0.8. If the inbox rate drops under the threshold, the check goes red and the merge button is blocked (via branch protection rules).

The complete workflow file

Drop this into .github/workflows/deliverability.yml. Two secrets required: INBOX_CHECK_API_KEY and the default GITHUB_TOKEN.

name: Email deliverability

on:
  pull_request:
    paths:
      - 'emails/**'
      - 'infra/dns/**'
      - '.github/workflows/deliverability.yml'

jobs:
  placement:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
      contents: read
    steps:
      - uses: actions/checkout@v4

      - name: Render template
        id: render
        run: |
          node scripts/render-template.js emails/welcome.mjml > rendered.html
          echo "html<<EOF" >> "$GITHUB_OUTPUT"
          cat rendered.html >> "$GITHUB_OUTPUT"
          echo "EOF" >> "$GITHUB_OUTPUT"

      - name: Start placement test
        id: start
        env:
          IC_KEY: ${{ secrets.INBOX_CHECK_API_KEY }}
        run: |
          ID=$(curl -sS -X POST https://check.live-direct-marketing.online/api/check \
            -H "Authorization: Bearer $IC_KEY" \
            -H "Content-Type: application/json" \
            -d @- <<JSON | jq -r '.id'
          {
            "senderDomain": "news.yourbrand.com",
            "subject":      "Welcome to YourBrand",
            "html":         $(jq -Rs . < rendered.html)
          }
          JSON
          )
          echo "id=$ID" >> "$GITHUB_OUTPUT"

      - name: Poll for result
        id: poll
        env:
          IC_KEY: ${{ secrets.INBOX_CHECK_API_KEY }}
          ID:     ${{ steps.start.outputs.id }}
        run: |
          for i in $(seq 1 40); do
            R=$(curl -sS https://check.live-direct-marketing.online/api/check/$ID \
              -H "Authorization: Bearer $IC_KEY")
            STATUS=$(echo "$R" | jq -r .status)
            if [ "$STATUS" = "complete" ]; then
              echo "$R" > result.json
              exit 0
            fi
            sleep 5
          done
          echo "Timeout" && exit 1

      - name: Compute inbox rate
        id: rate
        run: |
          RATE=$(jq '.summary.inboxCount / .summary.total * 100 | floor' result.json)
          echo "rate=$RATE" >> "$GITHUB_OUTPUT"

      - name: Post PR comment
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const r = JSON.parse(fs.readFileSync('result.json', 'utf8'));
            const rate = ${{ steps.rate.outputs.rate }};
            const body = [
              `### Email deliverability: ${rate}% inbox`,
              `- Inbox: ${r.summary.inboxCount} / ${r.summary.total}`,
              `- Spam:  ${r.summary.spamCount}`,
              `- SPF ${r.auth.spf} · DKIM ${r.auth.dkim} · DMARC ${r.auth.dmarc}`,
              `- SpamAssassin: ${r.spamAssassinScore}`,
            ].join('\n');
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo:  context.repo.repo,
              issue_number: context.issue.number,
              body,
            });

      - name: Fail if under threshold
        if: ${{ steps.rate.outputs.rate < 80 }}
        run: |
          echo "Inbox rate ${{ steps.rate.outputs.rate }}% < 80%"
          exit 1

Triggering on template path changes

The paths filter keeps the workflow cheap. You do not want a placement test on every PR — just the ones that can plausibly change deliverability. For most teams that means:

• emails/** — MJML, Handlebars, React Email templates, whatever your stack uses.
• infra/dns/** — Terraform, Pulumi, or raw BIND zone files for your sending domain.
• config/mail-from.* or similar — changes to the From address, Return-Path or DKIM selector.
• The workflow file itself, so you can iterate without a full-tree trigger.

Using GitHub Secrets for the API key

The API key is a Bearer token that starts with ic_live_. It must never be committed. Add it via Repo → Settings → Secrets and variables → Actions → New repository secret, named INBOX_CHECK_API_KEY. Reference it with ${{ secrets.INBOX_CHECK_API_KEY }}. For monorepos with multiple environments, use an environment secret instead — that way staging and production keys are scoped correctly and deploy protection rules apply.

Polling for test completion

The workflow polls GET /api/check/{id} every 5 seconds for up to ~3 minutes. SSE is nicer for interactive dashboards but unhelpful here: the runner is short-lived and pure-HTTP polling is trivial to retry. If your templates are heavy and take longer (some MJML → HTML + image-hosting pipelines do), extend the loop to 60 attempts.

Posting a PR comment with results

The actions/github-script step uses the default GITHUB_TOKEN with pull-requests: write. No additional secret required. The comment is a fresh one per run — if you prefer a sticky comment that updates in place, use marocchino/sticky-pull-request-comment instead:

      - uses: marocchino/sticky-pull-request-comment@v2
        with:
          header: deliverability
          message: |
            ### Email deliverability: ${{ steps.rate.outputs.rate }}% inbox
            ...

Failing the check if inbox rate is under 80%

The last step exits non-zero when the rate is below 80%. Pair that with a branch protection rule requiring the placement check to pass before merge, and a broken template cannot reach main. Pick a threshold that matches your current baseline — starting at 70% on a sloppy repo, tightening to 85% after a month of cleanup.

Caching

Two opportunities to shave seconds. First, actions/setup-node with a cache: "npm" step if your template render depends on Node modules. Second, a hashFiles-keyed cache of the rendered HTML — if the template hash matches a prior run, reuse the previous placement result and skip the API call entirely. That saves both runner minutes and API quota on no-op PRs (README changes, for instance, that somehow trigger the paths filter).

Reusable workflow pattern for monorepos

If you have multiple apps each owning their own email templates, extract the workflow to a reusable one:

# .github/workflows/deliverability-reusable.yml
on:
  workflow_call:
    inputs:
      template_path: { required: true, type: string }
      sender_domain: { required: true, type: string }
    secrets:
      INBOX_CHECK_API_KEY: { required: true }

jobs:
  placement:
    uses: ./.github/workflows/deliverability.yml
    with:
      template_path: ${{ inputs.template_path }}
      sender_domain: ${{ inputs.sender_domain }}

Each app's workflow then calls the reusable one:

jobs:
  welcome:
    uses: org/infra/.github/workflows/deliverability-reusable.yml@main
    with:
      template_path: apps/billing/emails/welcome.mjml
      sender_domain: billing.yourbrand.com
    secrets:
      INBOX_CHECK_API_KEY: ${{ secrets.INBOX_CHECK_API_KEY }}

A reusable Marketplace action (planned)

We're packaging all of the above as inbox-check/placement-test@v1, planned for September 2026. The action will collapse the four-step workflow into a single step with inputs for sender domain, template path and threshold, and will handle sticky-comment updates and caching out of the box. The raw-YAML version above will keep working — the action is a convenience, not a replacement.

Frequently asked questions