YTDLP Client Side Integration

This document describes how to integrate and use the YTDLP client with the token service.

Build

1. Pull, configure and start server if needed:

   cd /srv/airflow_worker/  
   docker login pangramia # It used to be performed  beforehand otherwise ask pull password
   docker compose -f docker-compose-ytdlp-ops.yaml up -d
   docker compose -f docker-compose-ytdlp-ops.yaml logs -f
   

   The server is bound to a certain proxy, like "socks5://sslocal-rust-1084:1084".

   Also check that redis in bind to 0.0.0.0 in config

2. Build airflow-worker with custom dependencies:

   cd /srv/airflow_worker/  
   docker compose build airflow-worker
   docker compose down airflow-worker
   docker compose up -d --no-deps airflow-worker
   

3. Test the built-in client:

   # Show client help
   docker compose exec airflow-worker python /app/ytdlp_ops_client.py --help 
   
   # Get token and info.json 
   docker compose exec airflow-worker python /app/ytdlp_ops_client.py --host 85.192.30.55 --port 9090 getToken --url 'https://www.youtube.com/watch?v=vKTVLpmvznI'
   
   # List formats using saved info.json 
   docker compose exec airflow-worker yt-dlp --load-info-json "latest.json"  -F
   
   # Simulate download using saved info.json 
   docker compose exec airflow-worker yt-dlp --load-info-json "latest.json" --proxy "socks5://sslocal-rust-1084:1084" --simulate --verbose 
   
   # Extract metadata and download URLs using jq 
   docker compose exec airflow-worker jq -r '"Title: \(.title)", "Date: \(.upload_date | strptime("%Y%m%d") | strftime("%Y-%m-%d"))", "Author: \(.uploader)", "Length: \(.duration_string)", "", "Download URLs:", (.formats[] | select(.vcodec != "none" or .acodec != "none") | .url)' latest.json            
   

4. Test Airflow task:

   To run the ytdlp_client_dag_v2.1 DAG:

   Set up required Airflow variables

   docker compose exec airflow-worker airflow variables set DOWNLOAD_OPTIONS '{"formats": ["bestvideo[height<=1080]+bestaudio/best[height<=1080]"]}'                                                  
   docker compose exec airflow-worker airflow variables set DOWNLOADS_TEMP '/opt/airflow/downloadfiles'
   docker compose exec airflow-worker airflow variables set DOWNLOADS_PATH '/opt/airflow/downloadfiles'     
   
   docker compose exec airflow-worker airflow variables  list
   docker compose exec airflow-worker airflow variables set TOKEN_TIMEOUT '300'  
   
   docker compose exec airflow-worker airflow  connections import /opt/airflow/config/docker_hub_repo.json
   docker compose exec airflow-worker airflow  connections delete redis_default
   docker compose exec airflow-worker airflow  connections import /opt/airflow/config/redis_default_conn.json
   

   Using direct connection with task test:

   docker compose exec airflow-worker airflow db reset
   docker compose exec airflow-worker airflow dags reserialize
   
   docker compose exec airflow-worker airflow dags list
   docker compose exec airflow-worker airflow  dags list-import-errors
   docker compose exec airflow-worker airflow tasks test ytdlp_client_dag_v2.1 get_token $(date -u +"%Y-%m-%dT%H:%M:%S+00:00") --task-params '{"url": "https://www.youtube.com/watch?v=sOlTX9uxUtM", "redis_enabled": false, "service_ip": "85.192.30.55", "service_port": 9090}'
   docker compose exec airflow-worker  yt-dlp --load-info-json /opt/airflow/downloadfiles/latest.json --proxy "socks5://sslocal-rust-1084:1084" --verbose --simulate
   
   docker compose exec airflow-worker  airflow dags list-runs -d ytdlp_client_dag 
   
   
   
   
   
   

   or deploy using trigger

   docker compose exec airflow-worker airflow dags list
   docker compose exec airflow-worker airflow dags unpause ytdlp_client_dag_v2.1
   
   // Try UI or recheck if works from server deploy
   docker compose exec airflow-worker airflow dags trigger ytdlp_client_dag_v2.1 -c '{"url": "https://www.youtube.com/watch?v=sOlTX9uxUtM", "redis_enabled": false, "service_ip": "85.192.30.55", "service_port": 9090}'                                              
   
   

   Check Redis for stored data by videoID

    docker compose exec  redis redis-cli -a XXXXXX -h 89.253.221.173 -p 52909 HGETALL "token_info:sOlTX9uxUtM" | jq -R -s 'split("\n") | del(.[] | select(. == "")) | [.[range(0;length;2)]]'