# Airflow Cluster for YT-DLP Operations

This directory contains the configuration and deployment files for an Apache Airflow cluster designed to manage distributed YouTube video downloading tasks using the `ytdlp-ops` service.

## Overview

The cluster consists of:
- **Master Node:** Runs the Airflow webserver, scheduler, and Flower (Celery monitoring). It also hosts shared services like Redis (broker/backend) and MinIO (artifact storage).
- **Worker Nodes:** Run Celery workers that execute download tasks. Each worker node also runs the `ytdlp-ops-service` (Thrift API server), Envoy proxy (load balancer for Thrift traffic), and Camoufox (remote browser instances for token generation).

## Key Components

### Airflow DAGs

- `ytdlp_ops_dispatcher.py`: The "Sensor" part of a Sensor/Worker pattern. It monitors a Redis queue for URLs to process and triggers a `ytdlp_ops_worker_per_url` DAG run for each URL.
- `ytdlp_ops_worker_per_url.py`: The "Worker" DAG. It processes a single URL passed via DAG run configuration. It implements worker affinity (all tasks for a URL run on the same machine) and handles account management (retrying with different accounts, banning failed accounts based on sliding window checks).

### Configuration Files

- `airflow.cfg`: Main Airflow configuration file.
- `config/airflow_local_settings.py`: Contains the `task_instance_mutation_hook` which implements worker affinity by dynamically assigning tasks to queues based on the worker node's hostname.
- `config/custom_task_hooks.py`: Contains the `task_instance_mutation_hook` (duplicated here, but `airflow_local_settings.py` is the active one).
- `config/redis_default_conn.json.j2`: Jinja2 template for the Airflow Redis connection configuration.
- `config/minio_default_conn.json.j2`: Jinja2 template for the Airflow MinIO connection configuration.

### Docker & Compose

- `Dockerfile`: Defines the Airflow image, including necessary dependencies like `yt-dlp`, `ffmpeg`, and Python packages.
- `Dockerfile.caddy`: Defines a Caddy image used as a reverse proxy for serving Airflow static assets.
- `configs/docker-compose-master.yaml.j2`: Jinja2 template for the Docker Compose configuration on the Airflow master node.
- `configs/docker-compose-dl.yaml.j2`: Jinja2 template for the Docker Compose configuration on the Airflow worker nodes.
- `configs/docker-compose-ytdlp-ops.yaml.j2`: Jinja2 template for the Docker Compose configuration for the `ytdlp-ops` services (Thrift API, Envoy, Camoufox) on both master (management role) and worker nodes.
- `configs/docker-compose.camoufox.yaml.j2`: Jinja2 template (auto-generated by `generate_envoy_config.py`) for the Camoufox browser service definitions.
- `configs/docker-compose.config-generate.yaml`: Docker Compose file used to run the `generate_envoy_config.py` script in a container to create the final service configuration files.
- `generate_envoy_config.py`: Script that generates `envoy.yaml`, `docker-compose.camoufox.yaml`, and `camoufox_endpoints.json` based on environment variables.
- `configs/envoy.yaml.j2`: Jinja2 template (used by `generate_envoy_config.py`) for the Envoy proxy configuration.

### Camoufox (Remote Browsers)

- `camoufox/`: Directory containing the Camoufox browser setup.
  - `Dockerfile`: Defines the Camoufox image.
  - `requirements.txt`: Python dependencies for the Camoufox server.
  - `camoufox_server.py`: The core server logic for managing remote browser instances.
  - `start_camoufox.sh`: Wrapper script to start the Camoufox server with Xvfb and VNC.
  - `*.xpi`: Browser extensions used by Camoufox.

## Deployment Process

Deployment is managed by Ansible playbooks located in the `ansible/` directory.

1.  **Inventory Generation:** The `tools/generate-inventory.py` script reads `cluster.yml` and generates `ansible/inventory.ini`, `ansible/host_vars/`, and `ansible/group_vars/all/generated_vars.yml`.
2.  **Full Deployment:** `ansible-playbook playbook-full.yml` is the main command.
    - Installs prerequisites (Docker, pipx, Glances).
    - Ensures the `airflow_proxynet` Docker network exists.
    - Imports and runs `playbook-master.yml` for the master node.
    - Imports and runs `playbook-worker.yml` for worker nodes.
3.  **Master Deployment (`playbook-master.yml`):**
    - Sets system configurations (timezone, NTP, swap, sysctl).
    - Calls `airflow-master` role:
        - Syncs files to `/srv/airflow_master/`.
        - Templates `configs/docker-compose-master.yaml`.
        - Builds the Airflow image.
        - Extracts static assets and builds the Caddy image.
        - Starts services using `docker compose`.
    - Calls `ytdlp-master` role:
        - Syncs `generate_envoy_config.py` and templates.
        - Creates `.env` file.
        - Runs `generate_envoy_config.py` to create service configs.
        - Creates a dummy `docker-compose.camoufox.yaml`.
        - Starts `ytdlp-ops` management services using `docker compose`.
4.  **Worker Deployment (`playbook-worker.yml`):**
    - Sets system configurations (timezone, NTP, swap, sysctl, system limits).
    - Calls `ytdlp-worker` role:
        - Syncs files (including `camoufox/` directory) to `/srv/airflow_dl_worker/`.
        - Creates `.env` file.
        - Runs `generate_envoy_config.py` to create service configs (including `docker-compose.camoufox.yaml`).
        - Builds the Camoufox image.
        - Starts `ytdlp-ops` worker services using `docker compose`.
    - Calls `airflow-worker` role:
        - Syncs files to `/srv/airflow_dl_worker/`.
        - Templates `configs/docker-compose-dl.yaml`.
        - Builds the Airflow image.
        - Starts services using `docker compose`.
    - Verifies Camoufox services are running.

## Service Interaction Flow (Worker Node)

1.  **Airflow Worker:** Pulls tasks from the Redis queue.
2.  **`ytdlp_ops_worker_per_url` DAG:** Executes tasks on the local worker node.
3.  **Thrift Client (in DAG task):** Connects to `localhost:9080` (Envoy's public port).
4.  **Envoy Proxy:** Listens on `:9080`, load balances Thrift requests across internal ports (`9090`, `9091`, `9092` - based on `YTDLP_WORKERS`) of the local `ytdlp-ops-service`.
5.  **`ytdlp-ops-service`:** Receives the Thrift request.
6.  **Token Generation:** If needed, `ytdlp-ops-service` connects to a local Camoufox instance via WebSocket (using `camoufox_endpoints.json` for the address) to generate tokens.
7.  **Camoufox:** Runs a headless Firefox browser, potentially using a SOCKS5 proxy, to interact with YouTube and generate the required tokens.
8.  **Download:** The DAG task uses the token (via `info.json`) and potentially the SOCKS5 proxy to run `yt-dlp` for the actual download.

## Environment Variables

Key environment variables used in `.env` files (generated by Ansible templates) control service behavior:
- `HOSTNAME`: The Ansible inventory hostname.
- `SERVICE_ROLE`: `management` (master) or `worker`.
- `SERVER_IDENTITY`: Unique identifier for the `ytdlp-ops-service` instance.
- `YTDLP_WORKERS`: Number of internal Thrift worker endpoints and Camoufox browser instances.
- `CAMOUFOX_PROXIES`: Comma-separated list of SOCKS5 proxy URLs for Camoufox.
- `MASTER_HOST_IP`: IP address of the Airflow master node (for connecting back to Redis).
- Various passwords and ports.

This setup allows for a scalable and robust system for managing YouTube downloads with account rotation and proxy usage.