80 lines
1.6 KiB
Markdown
80 lines
1.6 KiB
Markdown
# Airflow/YT-DLP Cluster Deployment
|
|
|
|
This repository contains Ansible playbooks and configuration files for deploying a distributed Airflow cluster with YT-DLP services.
|
|
|
|
## Prerequisites
|
|
|
|
1. Install Ansible on your control machine
|
|
2. Ensure SSH access to all target nodes
|
|
3. Set up your vault password in `.vault_pass` file
|
|
|
|
## Initial Setup
|
|
|
|
Generate the inventory and configuration files from your cluster definition:
|
|
|
|
```bash
|
|
./tools/generate-inventory.py cluster.test.yml
|
|
cd ansible
|
|
```
|
|
|
|
## Full Deployment
|
|
|
|
### Deploy entire cluster with proxies (recommended for new setups):
|
|
|
|
```bash
|
|
ansible-playbook playbook-full-with-proxies.yml
|
|
```
|
|
|
|
### Deploy cluster without proxies:
|
|
|
|
```bash
|
|
ansible-playbook playbook-full.yml
|
|
```
|
|
|
|
## Targeted Deployments
|
|
|
|
### Deploy only to master node:
|
|
|
|
```bash
|
|
ansible-playbook playbook-master.yml --limit="af-test"
|
|
```
|
|
|
|
### Deploy only to worker nodes:
|
|
|
|
```bash
|
|
ansible-playbook playbook-worker.yml
|
|
```
|
|
|
|
## Deploy Specific Steps
|
|
|
|
To start at a specific task (useful for debugging or partial deployments):
|
|
|
|
```bash
|
|
ansible-playbook playbook-master.yml --limit="af-test" --start-at-task="Prepare Caddy asset extraction directory"
|
|
```
|
|
|
|
## Debug Deployments
|
|
|
|
Run with dry-run and verbose output for debugging:
|
|
|
|
```bash
|
|
ansible-playbook playbook-full.yml --check --diff -vv
|
|
```
|
|
|
|
## DAGs Only Deployment
|
|
|
|
To update only DAG files and configurations:
|
|
|
|
```bash
|
|
ansible-playbook playbook-dags.yml
|
|
```
|
|
|
|
## Vault Management
|
|
|
|
All sensitive data is encrypted with Ansible Vault. The vault password should be stored in `.vault_pass` file in the project root.
|
|
|
|
To edit vault files:
|
|
```bash
|
|
ansible-vault edit group_vars/all/vault.yml
|
|
```
|