YTDLP Client Side Integration

This document describes how to integrate and use the YTDLP client with the token service. The client DAG is ytdlp_client_dag.

Building & Testing on the Remote Worker Host

(OPTIONAL if done) Start server if needed:

cd /srv/airflow_worker/  
docker login pangramia #if not performed auth before
docker compose -f docker-compose-ytdlp-ops.yaml up -d
docker compose -f docker-compose-ytdlp-ops.yaml logs -f

The server is bound to a certain proxy, like "socks5://sslocal-rust-1084:1084".

Rebuild worker with custom base:

cd /srv/airflow_worker/  
docker compose build airflow-worker
docker compose down airflow-worker
docker compose up -d --no-deps airflow-worker

Test the Client Inside a Worker Container:

# Show client help
docker compose exec airflow-worker python /app/ytdlp_ops_client.py --help 

# Get token and info.json 
docker compose exec airflow-worker python /app/ytdlp_ops_client.py --host 85.192.30.55 --port 9090 getToken --url 'https://www.youtube.com/watch?v=vKTVLpmvznI'

# List formats using saved info.json 
docker compose exec airflow-worker yt-dlp --load-info-json "latest.json"  -F

# Simulate download using saved info.json 
docker compose exec airflow-worker yt-dlp --load-info-json "latest.json" --proxy "socks5://sslocal-rust-1084:1084" --simulate --verbose 

# Extract metadata and download URLs using jq 
docker compose exec airflow-worker jq -r '"Title: \(.title)", "Date: \(.upload_date | strptime("%Y%m%d") | strftime("%Y-%m-%d"))", "Author: \(.uploader)", "Length: \(.duration_string)", "", "Download URLs:", (.formats[] | select(.vcodec != "none" or .acodec != "none") | .url)' latest.json

Instructions to run DAGs:

To run the ytdlp_client_dag DAG:

Set up required Airflow variables

docker compose exec airflow-worker airflow variables set DOWNLOAD_OPTIONS '{"formats": ["bestvideo[height<=1080]+bestaudio/best[height<=1080]"]}'                                                  
docker compose exec airflow-worker airflow variables set DOWNLOADS_TEMP '/opt/airflow/downloadfiles'
docker compose exec airflow-worker airflow variables set DOWNLOADS_PATH '/opt/airflow/downloadfiles'     

docker compose exec airflow-worker airflow variables  list
docker compose exec airflow-worker airflow variables set TOKEN_TIMEOUT '300'  

docker compose exec airflow-worker airflow  connections import /opt/airflow/config/docker_hub_repo.json
docker compose exec airflow-worker airflow  connections delete redis_default
docker compose exec airflow-worker airflow  connections import /opt/airflow/config/redis_default_conn.json

Using direct connection:

docker compose exec airflow-worker airflow db reset
docker compose exec airflow-worker airflow dags reserialize

docker compose exec airflow-worker airflow dags list
docker compose exec airflow-worker airflow  dags list-import-errors
docker compose exec airflow-worker airflow tasks test ytdlp_client_dag_v2.1 get_token $(date -u +"%Y-%m-%dT%H:%M:%S+00:00") --task-params '{"url": "https://www.youtube.com/watch?v=sOlTX9uxUtM", "redis_enabled": false, "service_ip": "85.192.30.55", "service_port": 9090}'
docker compose exec airflow-worker  yt-dlp --load-info-json /opt/airflow/downloadfiles/latest.json --proxy "socks5://sslocal-rust-1084:1084" --verbose --simulate

docker compose exec airflow-worker  airflow dags list-runs -d ytdlp_client_dag

or deploy using trigger

docker compose exec airflow-worker airflow dags list
docker compose exec airflow-worker airflow dags unpause ytdlp_client_dag_v2.1

// Check if works from server deploy
docker compose exec airflow-worker airflow dags trigger ytdlp_client_dag_v2.1 -c '{"url": "https://www.youtube.com/watch?v=sOlTX9uxUtM", "account_id": "test_direct", "redis_enabled": false, "service_ip": "85.192.30.55", "service_port": 9090}'

Check Redis for stored data

 docker compose exec  redis redis-cli -a XXXXXX -h 89.253.221.173 -p 52909 HGETALL "token_info:sOlTX9uxUtM" | jq -R -s 'split("\n") | del(.[] | select(. == "")) | [.[range(0;length;2)]]'

4.7 KiB Raw Blame History

YTDLP Client Side Integration

Building & Testing on the Remote Worker Host

4.7 KiB

Raw Blame History