Data Pipeline and CI/CD Workshop
About This Workshop
Learn to build production-grade data pipelines and automate software delivery with modern CI/CD practices. This workshop combines hands-on data engineering with DevOps automation — two of the most in-demand skill sets across every technology-driven industry today.
What You'll Build
An end-to-end Data Ingestion and Reporting Pipeline that:
- Fetches live data from a public REST API
- Cleans and transforms it using Python and pandas
- Orchestrates tasks with Apache Airflow
- Automatically tests and deploys using GitHub Actions
- Runs inside Docker containers for consistency
What You'll Learn
| Topic | Details |
|---|---|
| Data Pipeline Concepts | ETL vs ELT, pipeline patterns, data flows |
| Python for Data Engineering | pandas, requests, file I/O, error handling |
| Apache Airflow | DAGs, operators, scheduling, Airflow UI |
| CI/CD Concepts | Continuous Integration, Continuous Delivery |
| GitHub Actions | Workflows, triggers, jobs, matrix builds |
| Docker | Containerization, Dockerfile, docker-compose |
| Pipeline Testing | Unit testing data transformations |
| Monitoring & Logging | Observability basics for pipelines |
Curriculum
Module 1 — Data Pipeline Fundamentals
- What is a data pipeline and why it matters
- ETL (Extract, Transform, Load) vs ELT
- Common pipeline architectures: batch vs streaming
- Overview of the data engineering ecosystem
- Real-world use cases and industry examples
Module 2 — Building Pipelines with Python
- Fetching data from REST APIs with
requests - Data transformation and cleaning with
pandas - Writing reusable, modular pipeline functions
- Error handling, retries, and logging
- Saving output to CSV, JSON, and databases
Module 3 — Apache Airflow
- What is Apache Airflow and why teams use it
- Understanding DAGs (Directed Acyclic Graphs)
- Built-in operators: BashOperator, PythonOperator
- Setting task dependencies and execution order
- Scheduling pipelines with CRON expressions
- Monitoring pipeline runs in the Airflow UI
- Handling failures and alerts
Module 4 — CI/CD Fundamentals
- What is CI/CD and the software delivery lifecycle
- Continuous Integration: automated builds and tests
- Continuous Delivery vs Continuous Deployment
- Introduction to GitHub Actions
- Key concepts: workflows, triggers, runners, jobs, steps
Module 5 — GitHub Actions in Practice
- Writing your first workflow YAML file
- Trigger on push, pull request, and schedule
- Running Python tests automatically
- Matrix builds for multiple Python versions
- Storing secrets and environment variables securely
Module 6 — Docker for Data Pipelines
- Why containerize data pipelines?
- Writing a Dockerfile for a Python script
- Building and running Docker images
- Docker Compose for multi-service setups (Airflow + DB)
- Running Airflow locally with Docker Compose
Module 7 — Integration: Pipeline + CI/CD
- Automated unit tests for data transformation functions
- Running pipeline tests inside GitHub Actions
- Deploying updated pipelines on each merge
- Monitoring and alerting: Airflow alerts + GitHub notifications
Module 8 — Hands-on Project
- Build the complete end-to-end pipeline from scratch
- Integrate Airflow orchestration
- Add CI/CD automation with GitHub Actions
- Code review, best practices, and Q&A
Prerequisites
| Requirement | Details |
|---|---|
| Programming | Basic Python (variables, functions, loops, lists) |
| Tools | GitHub account (free) |
| Concepts | Understanding what an API is (helpful) |
| Prior DevOps experience | Not required |
Workshop Format
| Duration | 1 full day (6–8 hours) |
| Format | Instructor-led, hands-on coding |
| Group Size | Up to 30 students |
| Delivery | On-site at your institute |
Tools & Technologies
- Python 3.12 — pandas, requests, pytest
- Apache Airflow — pipeline orchestration
- Docker & Docker Compose — containerization
- GitHub Actions — CI/CD automation
- VS Code — development environment
Learning Outcomes
By the end of this workshop, students will be able to:
✅ Design and build a data pipeline using Python
✅ Orchestrate pipeline tasks with Apache Airflow DAGs
✅ Write automated tests for data transformation logic
✅ Set up a CI/CD workflow with GitHub Actions
✅ Containerize a Python pipeline with Docker
✅ Deploy and monitor an automated data pipeline