96 lines
4.7 KiB
Markdown
96 lines
4.7 KiB
Markdown
# YTDLP Client Side Integration
|
|
|
|
This document describes how to integrate and use the YTDLP client with the token service. The client DAG is `ytdlp_client_dag`.
|
|
|
|
## Building & Testing on the Remote Worker Host
|
|
|
|
1. **(OPTIONAL if done) Start server if needed:**
|
|
```bash
|
|
cd /srv/airflow_worker/
|
|
docker login pangramia #if not performed auth before
|
|
docker compose -f docker-compose-ytdlp-ops.yaml up -d
|
|
docker compose -f docker-compose-ytdlp-ops.yaml logs -f
|
|
```
|
|
The server is bound to a certain proxy, like "socks5://sslocal-rust-1084:1084".
|
|
|
|
2. **Rebuild worker with custom base:**
|
|
```bash
|
|
cd /srv/airflow_worker/
|
|
docker compose build airflow-worker
|
|
docker compose down airflow-worker
|
|
docker compose up -d --no-deps airflow-worker
|
|
```
|
|
|
|
3. **Test the Client Inside a Worker Container:**
|
|
```bash
|
|
# Show client help
|
|
docker compose exec airflow-worker python /app/ytdlp_ops_client.py --help
|
|
|
|
# Get token and info.json
|
|
docker compose exec airflow-worker python /app/ytdlp_ops_client.py --host 85.192.30.55 --port 9090 getToken --url 'https://www.youtube.com/watch?v=vKTVLpmvznI'
|
|
|
|
# List formats using saved info.json
|
|
docker compose exec airflow-worker yt-dlp --load-info-json "latest.json" -F
|
|
|
|
# Simulate download using saved info.json
|
|
docker compose exec airflow-worker yt-dlp --load-info-json "latest.json" --proxy "socks5://sslocal-rust-1084:1084" --simulate --verbose
|
|
|
|
# Extract metadata and download URLs using jq
|
|
docker compose exec airflow-worker jq -r '"Title: \(.title)", "Date: \(.upload_date | strptime("%Y%m%d") | strftime("%Y-%m-%d"))", "Author: \(.uploader)", "Length: \(.duration_string)", "", "Download URLs:", (.formats[] | select(.vcodec != "none" or .acodec != "none") | .url)' latest.json
|
|
```
|
|
|
|
4. **Instructions to run DAGs:**
|
|
|
|
To run the `ytdlp_client_dag` DAG:
|
|
|
|
Set up required Airflow variables
|
|
```bash
|
|
docker compose exec airflow-worker airflow variables set DOWNLOAD_OPTIONS '{"formats": ["bestvideo[height<=1080]+bestaudio/best[height<=1080]"]}'
|
|
docker compose exec airflow-worker airflow variables set DOWNLOADS_TEMP '/opt/airflow/downloadfiles'
|
|
docker compose exec airflow-worker airflow variables set DOWNLOADS_PATH '/opt/airflow/downloadfiles'
|
|
|
|
docker compose exec airflow-worker airflow variables list
|
|
docker compose exec airflow-worker airflow variables set TOKEN_TIMEOUT '300'
|
|
|
|
docker compose exec airflow-worker airflow connections import /opt/airflow/config/docker_hub_repo.json
|
|
docker compose exec airflow-worker airflow connections delete redis_default
|
|
docker compose exec airflow-worker airflow connections import /opt/airflow/config/redis_default_conn.json
|
|
```
|
|
|
|
|
|
**Using direct connection:**
|
|
```bash
|
|
docker compose exec airflow-worker airflow db reset
|
|
docker compose exec airflow-worker airflow dags reserialize
|
|
|
|
docker compose exec airflow-worker airflow dags list
|
|
docker compose exec airflow-worker airflow dags list-import-errors
|
|
docker compose exec airflow-worker airflow tasks test ytdlp_client_dag_v2.1 get_token $(date -u +"%Y-%m-%dT%H:%M:%S+00:00") --task-params '{"url": "https://www.youtube.com/watch?v=sOlTX9uxUtM", "redis_enabled": false, "service_ip": "85.192.30.55", "service_port": 9090}'
|
|
docker compose exec airflow-worker yt-dlp --load-info-json /opt/airflow/downloadfiles/latest.json --proxy "socks5://sslocal-rust-1084:1084" --verbose --simulate
|
|
|
|
docker compose exec airflow-worker airflow dags list-runs -d ytdlp_client_dag
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
or deploy using trigger
|
|
```bash
|
|
docker compose exec airflow-worker airflow dags list
|
|
docker compose exec airflow-worker airflow dags unpause ytdlp_client_dag_v2.1
|
|
|
|
// Check if works from server deploy
|
|
docker compose exec airflow-worker airflow dags trigger ytdlp_client_dag_v2.1 -c '{"url": "https://www.youtube.com/watch?v=sOlTX9uxUtM", "account_id": "test_direct", "redis_enabled": false, "service_ip": "85.192.30.55", "service_port": 9090}'
|
|
|
|
```
|
|
|
|
|
|
Check Redis for stored data
|
|
```bash
|
|
docker compose exec redis redis-cli -a XXXXXX -h 89.253.221.173 -p 52909 HGETALL "token_info:sOlTX9uxUtM" | jq -R -s 'split("\n") | del(.[] | select(. == "")) | [.[range(0;length;2)]]'
|
|
```
|
|
|