28:null 29:["$","title",null,{"children":"📜 Data Contracts | dduyg | Observable"}] 2e:T40a,# 📜 Data Contracts 2f:T7fe,## What a Data Contract Usually Specifies

Aspect	Description	Example
🗂️ Schema	Structure of the dataset (columns, data types, nullable fields).	`user_id: string`, `signup_date: date`, `is_premium: boolean`
💡 Semantics	Meaning of each field and business logic.	`is_premium = true`→ means the user paid within 30 days.
⏰ SLAs / SLIs	Service-level expectations for data delivery.	Data available by 2 AM daily, ≥99% completeness.
✅ Quality Rules	Accuracy, uniqueness, validity expectations.	No duplicate `user_id`, valid dates only.
📦 Versioning	How schema or logic changes are tracked.	Adding new columns is backward-compatible.
👑 Ownership	Who maintains it.	`marketing-data-team@company.com`

---30:T837,### 📄 Example Data Contract (YAML) ```yaml # File: contracts/user_signups.yaml contract: name: user_signups version: 1.3.0 owner: marketing-data-team@company.com domain: user_acquisition description: > Daily export of new user signups from the application database. Used by analytics and ML teams for retention and growth metrics. schema: fields: - name: user_id type: string description: Unique ID of the user. constraints: required: true unique: true - name: signup_timestamp type: timestamp description: UTC time of signup event. constraints: required: true not_null: true - name: country type: string description: ISO 3166-1 alpha-2 country code. constraints: allowed_values: ["US", "CA", "GB", "IN", "DE", "FR", "OTHER"] - name: referral_source type: string description: How the user found the product. constraints: nullable: true - name: is_premium type: boolean description: Whether the user subscribed to a paid plan. default: false data_quality_rules: - name: no_null_ids condition: "user_id IS NOT NULL" severity: error - name: valid_country_codes condition: "country IN ('US','CA','GB','IN','DE','FR','OTHER')" severity: warning - name: signup_date_recent condition: "signup_timestamp >= CURRENT_DATE - INTERVAL '7 days'" severity: info slas: freshness: "Data must be available by 02:00 UTC daily" completeness: "≥99% of records expected vs previous 7-day rolling average" latency: "Under 30 min for pipeline completion" change_policy: breaking_changes: - removing or renaming columns - changing data types notification: channels: ["#data-contracts", "email:analytics-team@company.com"] notice_period_days: 7 ```31:T452,### ➁ Enforcement via Airflow DAG ```python # airflow_dag_user_signups.py from airflow import DAG from airflow.operators.python import PythonOperator from datetime import datetime, timedelta from validate_user_signups import run_validation default_args = { "owner": "data-team", "start_date": datetime(2025, 10, 1), "retries": 1, "retry_delay": timedelta(minutes=5), } with DAG( "user_signups_pipeline", default_args=default_args, schedule_interval="@daily", catchup=False, ) as dag: extract = PythonOperator( task_id="extract_data", python_callable=lambda: print("Extracting user signups..."), ) validate = PythonOperator( task_id="validate_contract", python_callable=run_validation, op_args=["/data/staging/user_signups.parquet"], ) load_to_warehouse = PythonOperator( task_id="load_to_warehouse", python_callable=lambda: print("Loading data to Snowflake..."), ) extract >> validate >> load_to_warehouse ```32:T4ec,
### ④ Putting It All Together

Layer	Tool	Role
📄 Contract Definition	YAML File	Defines schema, rules, SLAs
✅ Validation	Great Expectations	Checks data vs. contract
🔄 Orchestration	Airflow	Runs validation before load
🔧 Transformation	dbt	Ensures schema compliance
📊 Monitoring	DataHub / Monte Carlo	Tracks lineage & alerts

33:T2bd1,{ const width = 900; const height = 700; const nodes = [ { id: "contract", label: "DATA CONTRACT", x: 450, y: 100, color: "transparent", hoverColor: "#fff", description: "Defines schema, quality rules, and SLAs for data assets", details: [ "Schema definitions", "Data quality rules", "Ownership info", "SLA agreements" ] }, { id: "extraction", label: "EXTRACTION", x: 200, y: 260, color: "#FFF8E6", hoverColor: "#FFDBB5", description: "Extract data from source systems", details: [ "Source connections", "Data retrieval", "Initial filtering", "Change data capture" ] }, { id: "validation", label: "VALIDATION", x: 450, y: 260, color: "#EBE714", hoverColor: "#E5EA36", description: "Validate data against contract rules", details: [ "Schema validation", "Quality checks", "Business rules", "Anomaly detection" ] }, { id: "load", label: "LOAD TO \nWAREHOUSE", x: 450, y: 420, color: "#FFF8E6", hoverColor: "#FFDBB5", description: "Load validated data into warehouse", details: ["Transform data", "Load process", "Indexing", "Partitioning"] }, { id: "warehouse", label: "WAREHOUSE", x: 450, y: 560, color: "#C187D4", hoverColor: "#C187D4", description: "Central data repository for analytics", details: [ "Query interface", "Data storage", "Access control", "Analytics ready" ] } ]; const edges = [ { from: "contract", to: "validation", id: "edge1" }, { from: "extraction", to: "validation", id: "edge2" }, { from: "validation", to: "load", id: "edge3" }, { from: "load", to: "warehouse", id: "edge4" } ]; let selectedNode = null; let isAnimating = false; let animationStep = 0; let currentArrowTransition = null; const container = d3 .create("div") .style("font-family", '"Inria Sans", sans-serif') .style("position", "relative") .style("max-width", "100%") .style("overflow", "hidden"); // --- Control buttons --- const controls = container.append("div").style("margin-bottom", "6px"); const buttonStyle = (btn) => btn .style("padding", "4px") .style("width", "130px") .style("height", "30px") .style("font-size", "11px") .style("margin", "5px") .style("font-family", "monospace") .style("font-weight", "bold") .style("background", "transparent") .style("color", "#6b7280") .style("border", "2px dotted #6b7280") .style("cursor", "pointer"); const animateBtn = buttonStyle(controls.append("button")) .text("⊷ANIMATE FLOW") .on("click", function () { isAnimating = !isAnimating; d3.select(this) .text(isAnimating ? "⊶PAUSE FLOW" : "⊷ANIMATE FLOW") .style("background", isAnimating ? "#D9DBDA" : "transparent"); if (isAnimating) animateFlow(); }); const resetBtn = buttonStyle(controls.append("button")) .text("{⛬ RESET}") .on("click", function () { // Stop animation isAnimating = false; animationStep = 0; selectedNode = null; if (currentArrowTransition) currentArrowTransition.end(); // Reset visuals edgePaths .attr("stroke", "#333") .attr("stroke-width", 2) .attr("marker-end", "url(#arrowhead)") .attr("filter", null); movingArrows.attr("opacity", 0); nodeElements .select("rect") .attr("stroke", "#000") .attr("stroke-width", 2.5) .attr("filter", null) .attr("fill", (d) => d.color) .attr("x", -100) .attr("y", -40) .attr("width", 200) .attr("height", 80); infoPanel.style("display", "none"); animateBtn.text("⊷ANIMATE FLOW").style("background", "transparent"); }); // --- SVG container --- const svgContainer = container .append("div") .style("display", "block") .style("width", "100%") .style("overflow", "hidden") .style("position", "relative"); const svg = svgContainer .append("svg") .attr("viewBox", `0 0 ${width} ${height}`) .attr("preserveAspectRatio", "xMidYMid meet") .style("width", "100%") .style("height", "auto"); // --- Arrow definitions --- const defs = svg.append("defs"); defs .append("marker") .attr("id", "arrowhead") .attr("markerWidth", 10) .attr("markerHeight", 10) .attr("refX", 9) .attr("refY", 3) .attr("orient", "auto") .append("polygon") .attr("points", "0 0, 10 3, 0 6") .attr("fill", "#333"); defs .append("marker") .attr("id", "arrowhead-active") .attr("markerWidth", 10) .attr("markerHeight", 10) .attr("refX", 9) .attr("refY", 3) .attr("orient", "auto") .append("polygon") .attr("points", "0 0, 10 3, 0 6") .attr("fill", "#000"); const filter = defs.append("filter").attr("id", "glow"); filter .append("feGaussianBlur") .attr("stdDeviation", "3") .attr("result", "coloredBlur"); const feMerge = filter.append("feMerge"); feMerge.append("feMergeNode").attr("in", "coloredBlur"); feMerge.append("feMergeNode").attr("in", "SourceGraphic"); // --- Info Panel const infoPanel = svgContainer .append("div") .style("position", "absolute") .style("top", "50%") .style("left", "50%") .style("transform", "translate(-50%, -50%)") .style("width", "60%") .style("background", "white") .style("border", "2px dotted black") .style("padding", "20px") .style("font-family", "monospace") .style("box-shadow", "0 6px 25px rgba(0,0,0,0.3)") .style("border-radius", "10px") .style("z-index", "10") .style("display", "none"); // --- Edges --- const edgeGroup = svg.append("g"); const edgePaths = edgeGroup .selectAll("path") .data(edges) .join("path") .attr("d", (d) => { const from = nodes.find((n) => n.id === d.from); const to = nodes.find((n) => n.id === d.to); if (d.from === "extraction") return `M ${from.x + 100} ${from.y} L ${to.x - 100} ${to.y}`; return `M ${from.x} ${from.y + 40} L ${to.x} ${to.y - 40}`; }) .attr("stroke", "#333") .attr("stroke-width", 2) .attr("fill", "none") .attr("marker-end", "url(#arrowhead)"); const movingArrows = edgeGroup .selectAll("use.movingArrow") .data(edges) .join("use") .attr("href", "#arrowhead-active") .attr("width", 10) .attr("height", 10) .attr("fill", "black") .attr("opacity", 0); // --- Nodes --- const nodeGroup = svg.append("g"); const nodeElements = nodeGroup .selectAll("g") .data(nodes) .join("g") .attr("transform", (d) => `translate(${d.x}, ${d.y})`) .style("cursor", "pointer") .on("mouseenter", function (event, d) { if (!isAnimating) { d3.select(this) .select("rect") .transition() .duration(200) .attr("fill", d.hoverColor) .attr("x", -105) .attr("y", -42) .attr("width", 210) .attr("height", 84); } }) .on("mouseleave", function (event, d) { if (selectedNode !== d) { d3.select(this) .select("rect") .transition() .duration(200) .attr("fill", d.color) .attr("x", -100) .attr("y", -40) .attr("width", 200) .attr("height", 80); } }) .on("click", function (event, d) { event.stopPropagation(); // don’t trigger background click selectedNode = selectedNode === d ? null : d; updateDiagram(); if (selectedNode) { infoPanel.style("display", "block"); infoPanel.html(`

${ selectedNode.label }

${ selectedNode.description }

KEY FEATURES:

${detail}

`); } else { infoPanel.style("display", "none"); } }); // click anywhere closes infoPanel svgContainer.on("click", function () { if (selectedNode) { selectedNode = null; infoPanel.style("display", "none"); updateDiagram(); } }); nodeElements .append("rect") .attr("x", -100) .attr("y", -40) .attr("width", 200) .attr("height", 80) .attr("rx", 15) .attr("fill", (d) => d.color) .attr("stroke", "#000") .attr("stroke-width", 2.5); nodeElements.each(function (d) { const lines = d.label.split("\n"); const g = d3.select(this); lines.forEach((line, i) => { g.append("text") .attr("text-anchor", "middle") .attr( "dy", lines.length === 1 ? "0.35em" : `${(i - (lines.length - 1) / 2) * 1.2 + 0.35}em` ) .attr("font-size", "26px") .attr("font-weight", "600") .style("font-family", "'Figurlce', sans-serif") .text(line); }); }); function updateDiagram() { nodeElements .select("rect") .attr("stroke", (d) => (selectedNode === d ? "#b3746f" : "#000")) .attr("stroke-width", (d) => (selectedNode === d ? 4 : 2.5)) .attr("filter", (d) => (selectedNode === d ? "url(#glow)" : null)); } function animateFlow() { if (!isAnimating) return; const path = edgePaths.filter((d, i) => i === animationStep); const movingArrow = movingArrows.filter((d, i) => i === animationStep); edgePaths .attr("stroke", "#333") .attr("stroke-width", 2) .attr("marker-end", "url(#arrowhead)") .attr("filter", null); movingArrows.attr("opacity", 0); path .attr("stroke", "#000") .attr("stroke-width", 4) .attr("marker-end", "url(#arrowhead-active)") .attr("filter", "url(#glow)"); const totalLength = path.node().getTotalLength(); movingArrow.attr("opacity", 1); currentArrowTransition = movingArrow .transition() .duration(1000) .ease(d3.easeLinear) .attrTween("transform", function () { return function (t) { const p = path.node().getPointAtLength(t * totalLength); return `translate(${p.x},${p.y})`; }; }) .on("end", () => { movingArrow.attr("opacity", 0); animationStep = (animationStep + 1) % edges.length; if (isAnimating) setTimeout(() => animateFlow(), 200); }); } return container.node(); }34:[] 2c:{"id":"91420f16721fd081","slug":"data-contracts","title":"📜 Data Contracts","thumbnail":"25cb39f0324479e78703d080c5585b737e5f1273851ad3733628f7c9337d419a","access_level":"public","create_time":"$D2025-10-21T23:22:55.093Z","update_time":"$D2026-06-10T09:35:26.347Z","view_time":null,"first_public_time":"$D2025-10-21T23:22:55.093Z","has_unpublished_edits":null,"fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"1fa0c77b14fd90d0","login":"dduyg","name":"","avatar_url":"https://avatars-next.observableusercontent.com/8922701a2d973be0158565df96ce0b2fe9ffd55df053350d59973a811bac06af","tier":"starter_2024"},"version":128,"publish_version":null,"role":null,"authored":false,"authors":[{"type":"individual","id":"248f116c85e550e3","login":"dduyg","name":"Duygu Dağdelen","avatar_url":"https://avatars-next.observableusercontent.com/ca098eb30fb37c129368c39d70e98a9a8444cfaa7496c72781905ac410af8cad","tier":null}],"members":[{"type":"team","id":"1fa0c77b14fd90d0","login":"dduyg","name":"","avatar_url":"https://avatars-next.observableusercontent.com/8922701a2d973be0158565df96ce0b2fe9ffd55df053350d59973a811bac06af","tier":"starter_2024","role":"editor"},{"type":"individual","id":"248f116c85e550e3","login":"dduyg","name":"Duygu Dağdelen","avatar_url":"https://avatars-next.observableusercontent.com/ca098eb30fb37c129368c39d70e98a9a8444cfaa7496c72781905ac410af8cad","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":true,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"title":"📜 Data Contracts","cells":[{"id":0,"value":"$2e","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":34,"value":"A data contract is a formal agreement between data producers (the teams or systems that create data) and data consumers (teams, dashboards, models, services that use it).\n\nIt defines what data is shared, how it behaves, and what guarantees are made — just like an API contract but for datasets.\n\nIn simple terms, a data contract is like a \"data API agreement\":\n\n> \"We, the producer, promise to deliver data in this format, at this frequency, with this quality; you, the consumer, can safely rely on it.\"\n\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":37,"value":"$2f","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":52,"value":"## Why Data Contracts Matter\n\n

🛡️Reduce Breakages: Prevents silent pipeline failures when upstream schemas change.
🕸️Enable Decentralization: Supports data mesh / domain-driven design.
💖Build Trust: Consumers rely confidently on stable data.
📜Support Governance: Simplifies lineage, compliance, and auditing.
🤝Foster Collaboration: Clarifies roles between producers and consumers.

\n\n---","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":57,"value":"## How Data Contracts Are Implemented\n\n

Defined declaratively using schema tools (e.g., JSON Schema, Avro, Protocol Buffers).
Version-controlled in Git repositories.
Automatically validated during CI/CD pipeline runs.
Enforced by tools like Airflow, Dagster, or metadata platforms such as DataHub or Monte Carlo.

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":56,"value":"
\n\n### 🏗️ Example Workflow\n\n

Define dataset schema in a YAML contract file.
Commit the file to Git; review changes like PRs.
Run automated tests to verify conformance.
Notify consumers on schema changes; respect notice windows.

","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":61,"value":"$30","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":71,"value":"
\n\n## ඏ Enforcing Data Contracts in Pipelines\n\n
\n\n### ➀ Validation with Great Expectations\n\n```python\n# validate_user_signups.py \nfrom great_expectations.dataset import PandasDataset \nimport pandas as pd \n \nclass UserSignupsDataset(PandasDataset): \n def validate_schema(self): \n self.expect_column_to_exist(\"user_id\") \n self.expect_column_values_to_not_be_null(\"user_id\") \n self.expect_column_to_exist(\"signup_timestamp\") \n self.expect_column_values_to_not_be_null(\"signup_timestamp\") \n self.expect_column_values_to_be_in_set( \n \"country\", [\"US\", \"CA\", \"GB\", \"IN\", \"DE\", \"FR\", \"OTHER\"] \n ) \n \ndef run_validation(file_path): \n df = pd.read_parquet(file_path) \n dataset = UserSignupsDataset(df) \n results = dataset.validate(expectation_suite_name=\"user_signups_suite\") \n if not results[\"success\"]: \n raise ValueError(\"Data contract validation failed!\")\n```\n\n
","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":51,"value":"$31","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":77,"value":"
\n\n### ➂ Integration with dbt\n\n```yaml\n# models/staging/user_signups.yml \nversion: 2 \n \nmodels: \n - name: stg_user_signups \n description: \"Validated user signup data\" \n columns: \n - name: user_id \n tests: \n - not_null \n - unique \n - name: country \n tests: \n - accepted_values: \n values: ['US', 'CA', 'GB', 'IN', 'DE', 'FR', 'OTHER']\n```","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":75,"value":"$32","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":74,"value":"---\n\n
\n\n## Workflow Diagram","pinned":false,"hidden":false,"mode":"md","author":"$undefined","summary":"$undefined","output":"","database":"$undefined"},{"id":111,"value":"$33","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W34","stdlib":"1","titling":"auto"}} 2d:"$2c:body"