# Airflow/YT-DLP Cluster Deployment

This repository contains Ansible playbooks and configuration files for deploying a distributed Airflow cluster with YT-DLP services.

## Prerequisites

1. Install Ansible on your control machine
2. Ensure SSH access to all target nodes
3. Set up your vault password in `.vault_pass` file

## Initial Setup

Generate the inventory and configuration files from your cluster definition:

```bash
./tools/generate-inventory.py cluster.test.yml
```

**Note:** All Ansible commands should be run from the project root directory.

## Full Deployment

### Deploy entire cluster with proxies (recommended for new setups):

```bash
ansible-playbook ansible/playbook-full-with-proxies.yml
```

### Deploy cluster without proxies:

```bash
ansible-playbook ansible/playbook-full.yml
```

## Targeted Deployments

### Deploy only to master node:

```bash
ansible-playbook ansible/playbook-master.yml --limit="af-test"
```

### Deploy only to worker nodes:

```bash
ansible-playbook ansible/playbook-worker.yml
```

## Deploy Specific Steps

To start at a specific task (useful for debugging or partial deployments):

```bash
ansible-playbook ansible/playbook-master.yml --limit="af-test" --start-at-task="Prepare Caddy asset extraction directory"
```

## Debug Deployments

Run with dry-run and verbose output for debugging:

```bash
ansible-playbook ansible/playbook-full.yml --check --diff -vv
```

## DAGs Only Deployment

To update only DAG files and configurations:

```bash
ansible-playbook ansible/playbook-dags.yml
```

## Vault Management

All sensitive data is encrypted with Ansible Vault. The vault password should be stored in `.vault_pass` file in the project root.

To edit vault files:
```bash
ansible-vault edit group_vars/all/vault.yml
```