Ansible for YT-DLP Cluster

This directory contains the Ansible playbooks, roles, and configurations for deploying and managing the YT-DLP Airflow cluster.

Full Deployment

Deploy entire cluster with proxies (recommended for new setups):

ansible-playbook playbook-full-with-proxies.yml

Deploy cluster without proxies:

ansible-playbook playbook-full.yml

Targeted Deployments

Deploy only to master node:

ansible-playbook playbook-master.yml --limit="af-test"

Deploy only to worker nodes:

ansible-playbook playbook-worker.yml

DAGs Only Deployment

To update only DAG files and configurations:

ansible-playbook playbook-dags.yml

Managing Worker State (Pause/Resume)

The system allows for gracefully pausing a worker to prevent it from picking up new tasks. This is useful for maintenance or decommissioning a node. The mechanism uses a lock file (AIRFLOW.PREVENT_URL_PULL.lock) on the worker host.

To Pause a Worker

This command creates the lock file, causing the ytdlp_ops_dispatcher DAG to skip task execution on this host.

# Replace "worker-hostname" with the target host from your inventory
ansible-playbook playbooks/pause_worker.yml --limit "worker-hostname"

To Resume a Worker

This command removes the lock file, allowing the worker to resume picking up tasks.

# Replace "worker-hostname" with the target host from your inventory
ansible-playbook playbooks/resume_worker.yml --limit "worker-hostname"

1.5 KiB Raw Blame History