Senior Data Engineer
This skill provides production-grade data engineering guidance for building scalable data pipelines, ETL/ELT systems, and data infrastructure. It covers architecture decisions, data modeling, data quality, and performance optimization, with practical tools and reference documentation.
✨ What it does
- Provides an architecture decision framework for batch vs streaming, Lambda vs Kappa, and warehouse vs lakehouse
- Includes quick-start scripts for pipeline orchestration, data quality validation, and ETL performance optimization
- Covers data modeling patterns including dimensional modeling, SCDs, and data vault
- Lists a comprehensive tech stack for orchestration, transformation, streaming, storage, and quality
- Offers reference documentation for pipeline architecture, data modeling, and DataOps best practices
🎯 When to use it
- When designing data pipelines or ETL/ELT processes
- When choosing between batch and streaming or Lambda vs Kappa architectures
- When implementing data modeling patterns like dimensional modeling or SCDs
- When adding data quality checks or data contracts
- When optimizing Spark jobs or Airflow DAGs for performance
🚀 How to use
Trigger the skill by asking questions or making requests related to data engineering, such as 'Design a data pipeline for...' or 'Should I use batch or streaming?'. The skill provides decision frameworks, scripts, and reference docs. Example prompts:
Design a data pipeline for ingesting sales data from Postgres to Snowflake daily.
Should I use Lambda or Kappa architecture for real-time analytics?
📄 Output: Guidance, decision frameworks, and reference documentation; no specific file outputs unless scripts are run.
📦 Add this skill to Claude Code
# 1. Get the skills repo
git clone --depth 1 https://github.com/alirezarezvani/claude-skills /tmp/claude-skills
# 2. Copy this skill into your project (or ~/.claude/skills for all projects)
mkdir -p .claude/skills
cp -r /tmp/claude-skills/engineering-team/skills/senior-data-engineer .claude/skills/senior-data-engineerSkill source: engineering-team/skills/senior-data-engineer/SKILL.md
⚠️ Good to know
The skill assumes familiarity with data engineering concepts and tools; it provides guidance but does not execute actual pipeline builds.
❓ FAQ
What is the difference between Lambda and Kappa architecture?
Lambda uses two codebases (batch and stream) and is suited for ML training on historical data, while Kappa uses a single codebase and is ideal for event-sourced architectures where all processing can be expressed as stream operations.
When should I choose a data warehouse over a data lakehouse?
Choose a warehouse for BI and SQL analytics with mature tooling, and a lakehouse for ML and unstructured data with lower storage costs and schema-on-read flexibility.
What tools are included in the tech stack for data quality?
The tech stack lists Great Expectations, dbt tests, and Monte Carlo for data quality.
🤖 Overview, features, install steps and FAQ were generated from the project's SKILL.md on Sep 4, 2026. Always check the original source before running commands.