Installation Guide — Data Pipeline & CI/CD Workshop

This guide helps you configure your system before attending the workshop.

System Requirements

Requirement	Minimum	Recommended	Notes
Operating System	Windows 10 / macOS 12+	Windows 11 / macOS 13+	64-bit required
RAM	8 GB	16 GB	Docker needs extra memory
Free Disk Space	10 GB	20 GB	Docker images use ~3–5 GB
Python	3.11	3.12	Must be added to PATH
Docker Desktop	Latest	Latest	Required for Airflow
Git	Latest	Latest	Required for cloning repo

info

If Docker installation fails, a cloud-based fallback (Google Colab) is available for the pipeline exercises.

---

Recommended Setup Order

1. Install Python
2. Install pip packages
3. Install Docker Desktop
4. Install Git
5. Install VS Code + extensions
6. Clone workshop repository
7. Verify all tools

---

Development Operating System

Windows

Windows Setup

Step 1: Install Python 3.12

Download the installer:

https://www.python.org/downloads/

Important

During installation, check "Add Python to PATH" before clicking Install.

Verify:

python --version
pip --version

Step 2: Install Required Python Packages

pip install pandas requests pytest apache-airflow

Verify Airflow:

airflow version

Step 3: Install Docker Desktop

Download Docker Desktop for Windows:

https://www.docker.com/products/docker-desktop/

Enable WSL 2 backend when prompted (recommended).

Restart your machine after installation.

Verify:

docker --version
docker compose version

Step 4: Install Git

Download from:

https://git-scm.com/

Use default settings during installation.

Verify:

git --version

Step 5: Install VS Code + Extensions

Download VS Code:

https://code.visualstudio.com/

Install these extensions from the Extensions panel (Ctrl+Shift+X):

• Python (Microsoft)
• Pylance (Microsoft)
• Docker (Microsoft)
• GitHub Actions (GitHub)

Step 6: Clone Workshop Repository

git clone <workshop-repo-url>
cd data-pipeline-workshop
pip install -r requirements.txt

Step 7: Verify Setup

python -c "import pandas, requests; print('Python packages OK')"
airflow db migrate
docker run --rm hello-world

macOS

macOS Setup

Step 1: Install Homebrew (if not installed)

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Step 2: Install Python 3.12

brew install python@3.12

Verify:

python3 --version
pip3 --version

Step 3: Install Required Python Packages

pip3 install pandas requests pytest apache-airflow

Verify Airflow:

airflow version

Step 4: Install Docker Desktop

Download Docker Desktop for Mac:

https://www.docker.com/products/docker-desktop/

Choose the correct chip version (Apple Silicon or Intel).

Verify:

docker --version
docker compose version

Step 5: Install Git

Git is usually pre-installed on macOS. Verify:

git --version

If not installed:

brew install git

Step 6: Install VS Code + Extensions

Download VS Code:

https://code.visualstudio.com/

Install extensions:

• Python (Microsoft)
• Pylance (Microsoft)
• Docker (Microsoft)
• GitHub Actions (GitHub)

Step 7: Clone Workshop Repository

git clone <workshop-repo-url>
cd data-pipeline-workshop
pip3 install -r requirements.txt

Step 8: Verify Setup

python3 -c "import pandas, requests; print('Python packages OK')"
airflow db migrate
docker run --rm hello-world

---

Emergency Fallback

If local installation fails, workshop exercises can be completed using:

• Google Colab (Python + pandas exercises): https://colab.research.google.com
• GitHub Codespaces (full cloud dev environment with Docker support)

No local installation required for the fallback.

---

Final Verification Checklist

Before the workshop, confirm all of the following work:

• python --version shows 3.11 or higher
• pip install pandas completes without errors
• airflow version prints the Airflow version
• docker --version shows Docker is installed
• docker run hello-world completes successfully
• git --version works
• VS Code opens and Python extension is active

If Web / Colab fallback works, your environment is ready.