Run an inbox placement test in your CI pipeline

You lint your code, you run your tests, you scan your containers. Why does the actual email template go out untested? A placement test is a 3–5 minute job that belongs between build and deploy, just like any other integration check.

GitHub Actions

Drop this at .github/workflows/deliverability.yml. It runs on changes to emails/** and any pull request against main.

name: deliverability

on:
  pull_request:
    paths:
      - 'emails/**'
      - 'templates/**'

jobs:
  inbox-placement:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4

      - name: Render template
        run: node scripts/render-email.js > /tmp/rendered.html

      - name: Create placement test
        id: create
        env:
          IC_KEY: ${{ secrets.INBOX_CHECK_API_KEY }}
        run: |
          payload=$(jq -Rs --arg subj "CI: ${{ github.sha }}" \
            '{senderDomain:"news.example.com",subject:$subj,html:.}' \
            < /tmp/rendered.html)
          resp=$(curl -sS -X POST \
            https://check.live-direct-marketing.online/api/tests \
            -H "Authorization: Bearer $IC_KEY" \
            -H "Content-Type: application/json" \
            -d "$payload")
          id=$(echo "$resp" | jq -r .id)
          echo "test_id=$id" >> "$GITHUB_OUTPUT"

      - name: Wait for result
        id: wait
        env:
          IC_KEY: ${{ secrets.INBOX_CHECK_API_KEY }}
          TEST_ID: ${{ steps.create.outputs.test_id }}
        run: |
          for i in $(seq 1 40); do
            status=$(curl -sS \
              https://check.live-direct-marketing.online/api/tests/$TEST_ID \
              -H "Authorization: Bearer $IC_KEY")
            state=$(echo "$status" | jq -r .status)
            if [ "$state" = "complete" ]; then
              rate=$(echo "$status" | jq -r .summary.inboxRate)
              echo "inbox_rate=$rate" >> "$GITHUB_OUTPUT"
              exit 0
            fi
            sleep 15
          done
          echo "timeout" && exit 1

      - name: Fail on low inbox rate
        run: |
          rate="${{ steps.wait.outputs.inbox_rate }}"
          awk -v r="$rate" 'BEGIN { exit (r < 0.80) }' \
            || { echo "::error::Inbox rate $rate below 0.80"; exit 1; }

GitLab CI

Same idea, GitLab-native. Uses the built-in rules: section for path filtering.

# .gitlab-ci.yml
stages: [test, deliverability, deploy]

inbox-placement:
  stage: deliverability
  image: alpine:3.20
  timeout: 15m
  rules:
    - changes:
        - emails/**/*
        - templates/**/*
  before_script:
    - apk add --no-cache curl jq nodejs npm bash
  script:
    - node scripts/render-email.js > rendered.html
    - |
      RESP=$(jq -Rs --arg s "CI $CI_COMMIT_SHORT_SHA" \
        '{senderDomain:"news.example.com",subject:$s,html:.}' < rendered.html \
        | curl -sS -X POST https://check.live-direct-marketing.online/api/tests \
            -H "Authorization: Bearer $INBOX_CHECK_API_KEY" \
            -H "Content-Type: application/json" -d @-)
      ID=$(echo "$RESP" | jq -r .id)
      echo "Test id: $ID"
    - |
      for i in $(seq 1 40); do
        S=$(curl -sS https://check.live-direct-marketing.online/api/tests/$ID \
              -H "Authorization: Bearer $INBOX_CHECK_API_KEY")
        ST=$(echo "$S" | jq -r .status)
        [ "$ST" = "complete" ] && break
        sleep 15
      done
      R=$(echo "$S" | jq -r .summary.inboxRate)
      awk -v r="$R" 'BEGIN { exit (r < 0.80) }' || { echo "Inbox $R"; exit 1; }

CircleCI

# .circleci/config.yml
version: 2.1

jobs:
  inbox-placement:
    docker:
      - image: cimg/node:20.11
    steps:
      - checkout
      - run:
          name: Render template
          command: node scripts/render-email.js > /tmp/rendered.html
      - run:
          name: Placement test + gate
          command: |
            set -euo pipefail
            RESP=$(jq -Rs --arg s "CI $CIRCLE_SHA1" \
              '{senderDomain:"news.example.com",subject:$s,html:.}' < /tmp/rendered.html \
              | curl -sS -X POST https://check.live-direct-marketing.online/api/tests \
                  -H "Authorization: Bearer $INBOX_CHECK_API_KEY" \
                  -H "Content-Type: application/json" -d @-)
            ID=$(echo "$RESP" | jq -r .id)
            for i in $(seq 1 40); do
              S=$(curl -sS https://check.live-direct-marketing.online/api/tests/$ID \
                    -H "Authorization: Bearer $INBOX_CHECK_API_KEY")
              [ "$(echo $S | jq -r .status)" = complete ] && break
              sleep 15
            done
            R=$(echo "$S" | jq -r .summary.inboxRate)
            echo "Inbox rate: $R"
            awk -v r="$R" 'BEGIN { exit (r < 0.80) }'

workflows:
  email-ci:
    jobs:
      - inbox-placement:
          filters:
            branches:
              only: /^(main|release\/.+)/

Failing the build: picking a threshold

Do not default to 100%. Real-world noise is 5–10% even for a perfectly-configured sender. Use the threshold that matches the kind of email:

• Transactional (password reset, receipt): fail below 95%.
• Marketing (newsletter, promo): fail below 85%.
• Cold outreach: fail below 70%.

A single outlier test hit can tank a metric — consider running two tests and taking the average before deciding, especially for marketing templates where you're paying per recipient.

Secret management

Never commit the API key. Use the runner's native store:

• GitHub Actions: repo Settings → Secrets → Actions. Name it INBOX_CHECK_API_KEY, reference as ${{ secrets.INBOX_CHECK_API_KEY }}.
• GitLab CI: Settings → CI/CD → Variables. Mark protected + masked.
• CircleCI: project Environment Variables, or an org-level context.

Use a scoped write key (no read access to other tests) so a leaked CI key can't exfiltrate your history.

Output formatting for PR annotations

GitHub Actions supports workflow commands that turn log lines into PR annotations. Emit one on failure:

echo "::error file=emails/weekly.mjml,title=Inbox rate regression::$R < 0.80"

For GitLab, generate a artifacts:reports:codequality JSON file; CircleCI has store_test_results with JUnit XML.

Only run on template changes

A placement test costs 1 credit per run. Path filters keep CI cost bounded:

• GitHub: paths: under on.pull_request.
• GitLab: rules: changes:.
• CircleCI: use dynamic config with the path-filtering orb.

Cost considerations

A team shipping 50 template changes a month runs 50 placement tests in CI — well inside the free-tier API quota. Heavy users (continuous A/B, dozens of brands) should cache: if the template hash hasn't changed, skip the job.

- name: Skip if template unchanged
  id: hash
  run: |
    NEW=$(sha256sum emails/weekly.mjml | cut -d' ' -f1)
    OLD=$(curl -sS -H "Authorization: Bearer $IC_KEY" \
           "https://check.live-direct-marketing.online/api/tests?tag=hash-$NEW&limit=1" \
           | jq -r '.data[0].id // empty')
    if [ -n "$OLD" ]; then echo "skip=true" >> "$GITHUB_OUTPUT"; fi

Frequently asked questions