Who uses what for email DAILY

1. Data source

The dataset is the daily OpenINTEL forward-DNS Tranco snapshot (University of Twente / SURFnet / SIDN Labs). OpenINTEL queries the entire Tranco top-1M domain list daily for MX, TXT, NS, A, AAAA, SOA, CAA, DNSSEC and other records, publishing the results as Apache Parquet. For pre-2022 history we additionally use OpenINTEL's alexa source (the legacy Alexa top-1M list, retired 2023).

Cite: Roland van Rijswijk-Deij et al., "A High-Performance, Scalable Infrastructure for Large-Scale Active DNS Measurements", IEEE JSAC 2016.

2. Sample & cadence

Each report covers a single date — OpenINTEL publishes snapshot D on D+1, so the current UTC date is never treated as an expected snapshot. Freshness means matching the latest date actually present in the OpenINTEL catalogue, typically ~700 k domains with MX records and ~620 k with SPF. The pipeline runs daily at 03:00 UTC, after the usual source-publication window; each daily run produces an HTML report, a JSON summary, an updated time-series, and incremental updates to the domain registry (§ 12). No sub-sampling.

3. Mailbox provider classification

For each domain we read its MX RRset and pick the record with the lowest mx_preference as the primary mailbox host. The hostname of that primary MX is matched against an open regex dictionary (dictionaries/mx_providers.py, maintained as a versioned project asset). Specific patterns (e.g. .mail.protection.outlook.com) are tried first; generic fallbacks (mail.*, mx*.*) only after. Domains whose MX matches no rule are kept as "Unknown / Other" — never dropped — and exported in Unmatched MX targets for dictionary improvement.

4. ESP / SaaS / forwarder / gateway classification (SPF-based)

For each domain's apex SPF record (TXT starting with v=spf1) we extract every include: and redirect= target and resolve them against open classification dictionaries (ESPs, mailbox-as-sender, anti-spam gateways, forwarders, SaaS senders, DMARC vendors, NS providers, verification tokens).

Resolution order: PURE_ESP → MAILBOX_AS_SENDER → GATEWAYS → FORWARDERS → bare-apex substring fallback → SAAS_SENDERS substring iteration. Bare-apex derivation strips leading _spf., _spf-eu., spf., mail. prefixes from dict keys to catch subdomain variants (e.g. _spf.m1.websupport.sk → matches websupport.sk). Malformed includes (no dot, <4 chars) are filtered.

One domain may use several ESPs simultaneously, so ESP shares sum to more than 100% of SPF-publishing domains. Current SPF-include target coverage: 82.6%, recomputed from this snapshot rather than hard-coded.

Limitation: "flattened" SPF (where include chains were replaced with raw IP ranges to fit the 10-lookup limit) is not detectable from DNS alone — those domains appear ESP-less even when an ESP is in fact used.

5. DMARC (deep)

For each domain we query the _dmarc.<domain> TXT record. Records starting with v=DMARC1 are parsed for:

• p= (apex policy): none / quarantine / reject / invalid
• sp= (subdomain policy)
• pct= (rollout percentage)
• rua= aggregate-report destinations → classified into vendor buckets (Postmark DMARC, Valimail, dmarcian, URIports, EasyDMARC, Red Sift, Proofpoint EFD, Agari/Fortra, …) using dictionaries/dmarc_vendors.py

A domain is counted as enforced if p=quarantine or p=reject with pct=100 (or pct absent — defaults to 100).

6. SPF mechanics & health

For every SPF record we additionally extract:

• Final qualifier on the all mechanism: -all (hard fail), ~all (soft fail), ?all (neutral), +all (pass-everything — broken / dangerous), or missing
• DNS lookup count (include:, redirect=, a:, mx:, exists:, ptr:) — each mechanism counts as 1 against the spec limit of 10. Records exceeding the limit return PermErr at recipients
• Mechanism mix: include-only (modern), raw IP4/IP6 ranges only (legacy/flattened), mixed
• Record byte length: percentiles to flag fragmentation risk (>450 B)

7. Email security maturity

Adoption of modern mail-security TXT records, parsed from the same OpenINTEL parquet:

• MTA-STS: TXT at _mta-sts.<domain> with v=STSv1 — domain advertises required-TLS to its MX
• BIMI: TXT at default._bimi.<domain> with v=BIMI1 — brand publishes a verified logo (requires p=reject)
• TLS-RPT (SMTP TLS Reporting): TXT at _smtp._tls.<domain> with v=TLSRPTv1 — domain monitors TLS-failure reports
• DKIM selectors: where the OpenINTEL scan includes *._domainkey.<domain> queries, well-known selectors (google, selector1/2, s1/s2, k1/k2/k3, mailo, mte1, …) are mapped to issuing ESPs

8. DNS provider classification

For each domain's NS RRset, every NS hostname is matched against a suffix dictionary (dictionaries/ns_providers.py) with patterns for Cloudflare, AWS Route 53, Azure DNS, Google Cloud DNS, GoDaddy, Akamai, NS1, UltraDNS, Yandex, DNSPod, Aliyun, OVH, Hetzner, Gandi, registrars, and others. A domain is assigned to its dominant NS provider; ties resolve to whichever pattern was matched first.

9. SaaS-via-verification-tokens

Many SaaS apps a domain is connected to never appear in SPF (because the SaaS doesn't send mail FROM the customer domain). To recover this signal we parse apex TXT records for verification tokens — google-site-verification=…, MS=…, atlassian-domain-verification=…, stripe-verification=…, plus ~70 other patterns in dictionaries/verification_tokens.py. This produces a complementary "SaaS density" metric and surfaces apps that the SPF-only view misses.

10. Email infrastructure modernity

Three additional TLS / IPv6 metrics:

• IPv6-on-MX: MX hostname has at least one AAAA record
• DANE / TLSA on MX: TLSA record at _25._tcp.<mx_host> (where the parquet includes TLSA queries)
• CAA: any 0 issue … / 0 issuewild … record at apex; CAs aggregated into a market-share view (Let's Encrypt, DigiCert, Sectigo, …)

11. Cross-tabulations

• Country × ESP heatmap: top-15 ccTLDs × top-15 ESPs, cell = % of country's SPF-publishing domains using ESP X. Surfaces regional ESP champions (Brevo / FR, Unisender / RU, MailerLite / LT, Akna / BR, Shoptet / CZ).
• Mailbox × ESP combinations: stacked frequency of common stack pairings (Google Workspace + HubSpot, MS365 + Marketo, …).

12. Domain registry & identifiers

Every domain ever observed across snapshots is assigned a stable integer ID in /var/openintel-cache/registry/domains.sqlite3 (currently ~1.2 M domains). Reports reference domains by ID rather than embedding strings; clients resolve names from a single gzipped registry dump (/email-stats/domains.csv.gz, ~13 MB). This keeps per-report payloads compact, enables fast set-operation diffs across snapshots, and gives every domain a first-seen / last-seen timestamp.

13. Daily change-alerts feed

Each daily run computes a domain-level diff vs the previous scan and emits a change-feed at /email-stats/alerts.html (also as JSON + Atom). Detected events: ESP added/removed in SPF, DMARC policy upgrade/downgrade, primary-mailbox-provider change, SPF strict→soft regression, MTA-STS / BIMI first publication. Severity tag is good / bad / info.

14. Per-entity detail pages

The top-100 ESPs, top-100 SaaS senders and top-80 mailbox providers each have a dedicated detail page at /email-stats/detail/<kind>/<slug>.html showing KPI cards, a Chart.js sparkline (from history.json), TLD distribution and a sample of customer domains.

15. Tier breakdown

Each domain is assigned a tier from its Tranco rank: top-1k / top-10k / top-100k / top-1M / unranked.

16. Reproducibility & data hygiene

Every published report includes the exact OpenINTEL date, dictionary file hashes, and counts of unmatched MX hosts and SPF includes — so any reader can verify or reproduce the figures. Raw OpenINTEL parquet is downloaded into a temporary cache and deleted after analysis; only aggregated, non-redistributable counts are kept long-term (per OpenINTEL data agreement). The domain registry stores names but no record-level content.

17. Limitations to be aware of

• Tranco bias. Top-1M skews toward US/EU and global SaaS; ccTLD-only domains with low traffic may be under-represented. The alexa source used for pre-2022 history has a different composition (~960 k domains, US-heavier).
• SPF flattening hides ESP identity (see § 4).
• CNAME chains on MX (e.g. mail.example.com → mail.example.protection.outlook.com) are not unrolled — only the first MX target is matched. This biases a small share of domains toward "Unknown" when their MX is a CNAME to a known provider.
• Vanity MX with white-label provider (Mimecast/Proofpoint customers using their own brand) is not detectable from DNS alone.
• DKIM enumeration is selector-based — only domains whose well-known selectors are scanned by OpenINTEL appear in the DKIM layer.
• Long-tail unmatched: this snapshot contains 79 716 unique unclassified SPF targets. The count is published as a measured coverage KPI rather than a fixed estimate; many are regional hosters, tenant-specific endpoints, malformed records, or single-domain infrastructure.
• Newly-added dictionary entries have no historical sparkline until ≥ 2 daily scans accumulate, since archived raw parquet is purged after parsing per the OpenINTEL data agreement.