yt-dlp-dags/README.md

Airflow/YT-DLP Cluster Deployment

This repository contains Ansible playbooks and configuration files for deploying a distributed Airflow cluster with YT-DLP services.

Prerequisites

1. Install Ansible on your control machine
2. Ensure SSH access to all target nodes
3. Set up your vault password in .vault_pass file

Initial Setup

Generate the inventory and configuration files from your cluster definition:

./tools/generate-inventory.py cluster.test.yml

Note: All Ansible commands should be run from the project root directory.

Full Deployment

Deploy entire cluster with proxies (recommended for new setups):

ansible-playbook ansible/playbook-full-with-proxies.yml

Deploy cluster without proxies:

ansible-playbook ansible/playbook-full.yml

Targeted Deployments

Deploy only to master node:

ansible-playbook ansible/playbook-master.yml --limit="af-test"

Deploy only to worker nodes:

ansible-playbook ansible/playbook-worker.yml

Deploy Specific Steps

To start at a specific task (useful for debugging or partial deployments):

ansible-playbook ansible/playbook-master.yml --limit="af-test" --start-at-task="Prepare Caddy asset extraction directory"

Debug Deployments

Run with dry-run and verbose output for debugging:

ansible-playbook ansible/playbook-full.yml --check --diff -vv

DAGs Only Deployment

To update only DAG files and configurations:

ansible-playbook ansible/playbook-dags.yml

Vault Management

All sensitive data is encrypted with Ansible Vault. The vault password should be stored in .vault_pass file in the project root.

To edit vault files:

ansible-vault edit group_vars/all/vault.yml