65 lines
1.7 KiB
Markdown
65 lines
1.7 KiB
Markdown
# Ansible for YT-DLP Cluster
|
|
|
|
This directory contains the Ansible playbooks, roles, and configurations for deploying and managing the YT-DLP Airflow cluster.
|
|
|
|
**Note:** All commands should be run from the project root, not from within this directory.
|
|
Example: `ansible-playbook ansible/playbook-full.yml`
|
|
|
|
## Full Deployment
|
|
|
|
### Deploy entire cluster with proxies (recommended for new setups):
|
|
|
|
```bash
|
|
ansible-playbook ansible/playbook-full-with-proxies.yml
|
|
```
|
|
|
|
### Deploy cluster without proxies:
|
|
|
|
```bash
|
|
ansible-playbook ansible/playbook-full.yml
|
|
```
|
|
|
|
## Targeted Deployments
|
|
|
|
### Deploy only to master node:
|
|
|
|
```bash
|
|
ansible-playbook ansible/playbook-master.yml --limit="af-test"
|
|
```
|
|
|
|
### Deploy only to worker nodes:
|
|
|
|
```bash
|
|
ansible-playbook ansible/playbook-worker.yml
|
|
```
|
|
|
|
## DAGs Only Deployment
|
|
|
|
To update only DAG files and configurations:
|
|
|
|
```bash
|
|
ansible-playbook ansible/playbook-dags.yml
|
|
```
|
|
|
|
## Managing Worker State (Pause/Resume)
|
|
|
|
The system allows for gracefully pausing a worker to prevent it from picking up new tasks. This is useful for maintenance or decommissioning a node. The mechanism uses a lock file (`AIRFLOW.PREVENT_URL_PULL.lock`) on the worker host.
|
|
|
|
### To Pause a Worker
|
|
|
|
This command creates the lock file, causing the `ytdlp_ops_dispatcher` DAG to skip task execution on this host.
|
|
|
|
```bash
|
|
# Replace "worker-hostname" with the target host from your inventory
|
|
ansible-playbook ansible/playbooks/pause_worker.yml --limit "worker-hostname"
|
|
```
|
|
|
|
### To Resume a Worker
|
|
|
|
This command removes the lock file, allowing the worker to resume picking up tasks.
|
|
|
|
```bash
|
|
# Replace "worker-hostname" with the target host from your inventory
|
|
ansible-playbook ansible/playbooks/resume_worker.yml --limit "worker-hostname"
|
|
```
|