Data Quality That Sticks: Tests, Contracts, and Alerts

Quickstart

If you only have an hour, do these in order. The goal is not “perfect data” — it’s preventing the expensive failures (wrong joins, missing days, broken schemas, and quietly duplicated rows).

Overview

Data quality fails in predictable ways: a column changes type, a late upstream job shifts timestamps, a join duplicates rows, a backfill creates gaps, or an upstream system emits new “status” values. The hard part isn’t knowing these exist — it’s building a system that catches them every time.

This post focuses on pragmatic guardrails you can implement with the tools you already have (warehouse, orchestrator, CI/CD, dbt-style transformations, or lightweight validation code). You’ll learn what to test, where to put the checks, how to design data contracts that don’t rot, and how to alert without creating alert fatigue.

Core concepts

Before tools and dashboards, it helps to agree on a few definitions and mental models. Most teams struggle with data quality because they treat it as a one-time cleanup instead of an engineering system.

1) Data quality is a set of guarantees (not a vibe)

“Quality” becomes operational when you express it as guarantees consumers can rely on. Think in terms of: schema, freshness, uniqueness, completeness, and validity (values make sense).

2) Tests are “unit tests for data”

A good data test is cheap, deterministic, and close to the logic that produces the data. That’s why many teams put tests next to transformations (e.g., model definitions), and run them in CI and on schedule.

3) Data contracts are shared agreements

A data contract turns “we think this table looks like…” into “this is what the table guarantees”. Contracts work best when they include: ownership, schema, semantic meaning (what columns represent), and a change policy (how breaking changes are handled).

4) Alerts should be about impact, not curiosity

A “good” alert is one that a human can act on quickly. A “bad” alert is one that teaches humans to ignore alerts. Use severity levels and only page for breaches that affect SLAs or critical consumers.

Step-by-step

This is a practical blueprint you can implement incrementally. You don’t need a giant platform rewrite — you need a consistent approach: contracts for expectations, tests for correctness, and alerts for response.

Step 1 — Inventory critical assets and define SLAs

If everything is critical, nothing is. Pick a small set of tables/models that your business would notice if they went wrong. Then define SLAs that match reality.

Step 2 — Write a lightweight data contract (one page, versioned)

Your contract should be human-readable and machine-checkable. Don’t start with a 40-page spec. Start with schema, ownership, and a few semantic checks that matter.

version: 1
dataset: analytics.orders_daily
owner:
  team: data-platform
  channel: "#data-oncall"
description: "Daily order rollup used by Finance and KPI dashboards."

columns:
  - name: order_id
    type: bigint
    nullable: false
    description: "Stable unique identifier for an order."
  - name: order_date
    type: date
    nullable: false
    description: "Business date in UTC (not event timestamp)."
  - name: revenue
    type: numeric
    nullable: false
    description: "Gross revenue in reporting currency."

checks:
  - name: primary_key_unique
    expr: "count(*) = count(distinct order_id)"
  - name: revenue_non_negative
    expr: "min(revenue) >= 0"

sla:
  freshness_minutes: 120
  expected_partitions: "daily"

change_policy:
  breaking_changes:
    - "rename or remove a column"
    - "change column type"
    - "change meaning of a column"
  process: "open PR + notify consumers + bump contract version"

The contract is your shared source of truth. You can keep it in the same repo as your transformations and review it like code. The key is versioning: when meaning changes, bump the version so consumers can react.

Step 3 — Convert the contract into tests close to the transformations

Contracts define guarantees; tests enforce them. Keep the enforcement near the logic that produces the data (models, transformations, or warehouse views) so failures are easy to trace.

version: 2

models:
  - name: orders_daily
    description: "Daily order rollup used by KPI dashboards."
    columns:
      - name: order_id
        tests:
          - unique
          - not_null
      - name: order_date
        tests:
          - not_null
      - name: status
        tests:
          - accepted_values:
              values: ["paid", "refunded", "chargeback", "pending"]
      - name: revenue
        tests:
          - not_null

    tests:
      - relationships:
          to: ref('dim_customers')
          field: customer_id

sources:
  - name: app
    tables:
      - name: orders
        loaded_at_field: created_at
        freshness:
          warn_after: {count: 90, period: minute}
          error_after: {count: 120, period: minute}

Notes: keep tests small and fast. For large tables, prefer tests on recent partitions or incremental slices (e.g., last 1–3 days) unless you truly need full history scans.

Step 4 — Add data observability checks (freshness + volume + anomaly)

Tests catch “hard” violations. Observability checks catch “soft” drift: sudden drops in row counts, missing partitions, or changes in key distributions. These are often early indicators of upstream issues.

Step 5 — Wire alerts with routing and runbooks

Alerts are the “last mile.” Make them useful. The best data quality programs win not by having the most checks, but by having the most fixes per alert.

Step 6 — Add a tiny “quality gate” in CI and a scheduled monitor

Run tests in two places: (1) CI on code changes (prevents regressions), and (2) scheduled in your orchestrator (catches upstream changes and late data). The code below shows a lightweight pattern for validating a dataset slice and sending one actionable alert.

import os
import datetime as dt
import requests

# Keep thresholds simple and tied to impact.
FRESHNESS_MINUTES_SLA = 120
MIN_ROWS_TODAY = 1000  # example: expected daily volume lower bound

SLACK_WEBHOOK = os.environ.get("SLACK_WEBHOOK_URL", "")

def send_alert(title: str, body: str) -> None:
    if not SLACK_WEBHOOK:
        raise RuntimeError("Missing SLACK_WEBHOOK_URL")
    payload = {"text": f"*{title}*\\n{body}"}
    requests.post(SLACK_WEBHOOK, json=payload, timeout=10)

def minutes_since(ts: dt.datetime) -> int:
    return int((dt.datetime.utcnow() - ts).total_seconds() // 60)

def quality_check(last_loaded_at: dt.datetime, rows_today: int) -> None:
    freshness = minutes_since(last_loaded_at)
    freshness_ok = freshness <= FRESHNESS_MINUTES_SLA
    volume_ok = rows_today >= MIN_ROWS_TODAY

    if freshness_ok and volume_ok:
        return

    problems = []
    if not freshness_ok:
        problems.append(f"- Freshness SLA breached: last load {freshness} min ago (SLA {FRESHNESS_MINUTES_SLA} min)")
    if not volume_ok:
        problems.append(f"- Volume too low: rows_today={rows_today} (min {MIN_ROWS_TODAY})")

    send_alert(
        title="Data quality alert: analytics.orders_daily",
        body="\\n".join(problems) + "\\n\\nOwner: #data-oncall\\nRunbook: (link your internal runbook here)"
    )

# Example usage: in real life, fetch these from your warehouse query.
if __name__ == "__main__":
    simulated_last_loaded = dt.datetime.utcnow() - dt.timedelta(minutes=185)
    simulated_rows_today = 420
    quality_check(simulated_last_loaded, simulated_rows_today)

This pattern is intentionally boring: deterministic thresholds, one message, one owner, one runbook. You can scale it up later — but this alone prevents many “we found out from the CEO” incidents.

Step 7 — Iterate with an incident-driven backlog

The easiest way to prioritize data quality work is to treat each real incident as a backlog item: what check would have caught this earlier? Add that check, then move on.

Common mistakes

Data quality doesn’t fail because teams don’t care. It fails because teams build checks that are too expensive, too vague, or too noisy. Here are the patterns that show up again and again — and the fixes.

FAQ

What’s the difference between data tests and data contracts?

Contracts define the promise (schema + meaning + change policy). Tests enforce the promise by checking data and transformations automatically. A contract without tests becomes documentation; tests without a contract often miss the most important semantic guarantees.

Where should data quality checks live: in the warehouse, the pipeline, or CI?

Use all three, but keep it simple: run fast regression tests in CI (so code changes don’t break outputs), and run scheduled checks in your orchestrator (so upstream delays and backfills are detected). Warehouse-side assertions are great when they’re cheap and close to the dataset being validated.

How many tests should I start with?

Start with 5 baseline tests per critical model (unique, not null, relationships, accepted values, bounds), plus a freshness SLA. Expand only when you have a real incident that suggests a new check.

How do I avoid alert fatigue?

Tie alerts to impact. Page only on SLA breaches or critical correctness failures, and send “FYI” drift signals to a non-paging channel. Also ensure every alert has a clear owner and a runbook link.

What should a “good” data contract include at minimum?

Owner, schema (columns/types/nullability), semantic notes for tricky columns, freshness expectations, and a change policy defining what counts as breaking and how it’s communicated.

Do I need a dedicated data quality tool to do this?

No. You can implement durable guardrails with your existing stack: transformation tests, a few warehouse queries, and an alerting channel. Dedicated tools can help with lineage, anomaly detection, and UI — but the core system is definitions + automation + ownership.

Cheatsheet

A scan-fast checklist for building data quality that sticks. Copy this into your team wiki and iterate as you learn.

Wrap-up

Data quality that sticks isn’t about building the biggest rule set — it’s about building a system people trust: contracts define the promises, tests enforce the promises, and alerts make failures actionable.

From here, the best improvements are incident-driven: every time the business sees “weird numbers,” turn it into a new automated check. After a few cycles, the number of repeat incidents drops — and trust rises.

Related posts

• SQL That Scales: Joins, Windows, and Real-World Patterns
• Explain Plans Demystified: Find the Slow Part Fast
• Star Schema vs Snowflake: Modeling for Analytics
• Postgres Power Moves: Indexes, JSONB, and Partitioning
• Warehouse vs Lake vs Lakehouse: What to Pick in 2026
• dbt in Practice: The Clean Way to Build Transformations