alex_p/yt-dlp-dags
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update readme with more explicit instructions on pull and build

Browse Source
This commit is contained in:
aperez 2025-04-03 13:33:36 +03:00
parent 8b689fc5c8
commit 8390c8172a
1 changed files with 14 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
README-ytdlp-ops-auth.md
View File

@ -1,19 +1,21 @@
# YTDLP Client Side Integration # YTDLP Client Side Integration
This document describes how to integrate and use the YTDLP client with the token service. The client DAG is `ytdlp_client_dag`. This document describes how to integrate and use the YTDLP client with the token service.
## Building & Testing on the Remote Worker Host ## Build
1. **(OPTIONAL if done) Start server if needed:** 1. **Pull, configure and start server if needed:**
```bash ```bash
cd /srv/airflow_worker/ cd /srv/airflow_worker/
docker login pangramia #if not performed auth before docker login pangramia # It used to be performed beforehand otherwise ask pull password
docker compose -f docker-compose-ytdlp-ops.yaml up -d docker compose -f docker-compose-ytdlp-ops.yaml up -d
docker compose -f docker-compose-ytdlp-ops.yaml logs -f docker compose -f docker-compose-ytdlp-ops.yaml logs -f
``` ```
The server is bound to a certain proxy, like "socks5://sslocal-rust-1084:1084". The server is bound to a certain proxy, like "socks5://sslocal-rust-1084:1084".
2. **Rebuild worker with custom base:** Also check that redis in bind to 0.0.0.0 in config
2. **Build airflow-worker with custom dependencies:**
```bash ```bash
cd /srv/airflow_worker/ cd /srv/airflow_worker/
docker compose build airflow-worker docker compose build airflow-worker
@ -21,7 +23,7 @@ This document describes how to integrate and use the YTDLP client with the token
docker compose up -d --no-deps airflow-worker docker compose up -d --no-deps airflow-worker
``` ```
3. **Test the Client Inside a Worker Container:** 3. **Test the built-in client:**
```bash ```bash
# Show client help # Show client help
docker compose exec airflow-worker python /app/ytdlp_ops_client.py --help docker compose exec airflow-worker python /app/ytdlp_ops_client.py --help
@ -39,9 +41,9 @@ This document describes how to integrate and use the YTDLP client with the token
docker compose exec airflow-worker jq -r '"Title: \(.title)", "Date: \(.upload_date | strptime("%Y%m%d") | strftime("%Y-%m-%d"))", "Author: \(.uploader)", "Length: \(.duration_string)", "", "Download URLs:", (.formats[] | select(.vcodec != "none" or .acodec != "none") | .url)' latest.json docker compose exec airflow-worker jq -r '"Title: \(.title)", "Date: \(.upload_date | strptime("%Y%m%d") | strftime("%Y-%m-%d"))", "Author: \(.uploader)", "Length: \(.duration_string)", "", "Download URLs:", (.formats[] | select(.vcodec != "none" or .acodec != "none") | .url)' latest.json
``` ```
4. **Instructions to run DAGs:** 4. **Test Airflow task:**
To run the `ytdlp_client_dag` DAG: To run the `ytdlp_client_dag_v2.1` DAG:
Set up required Airflow variables Set up required Airflow variables
```bash ```bash
@ -58,7 +60,7 @@ This document describes how to integrate and use the YTDLP client with the token
``` ```
**Using direct connection:** **Using direct connection with task test:**
```bash ```bash
docker compose exec airflow-worker airflow db reset docker compose exec airflow-worker airflow db reset
docker compose exec airflow-worker airflow dags reserialize docker compose exec airflow-worker airflow dags reserialize
@ -82,13 +84,13 @@ This document describes how to integrate and use the YTDLP client with the token
docker compose exec airflow-worker airflow dags list docker compose exec airflow-worker airflow dags list
docker compose exec airflow-worker airflow dags unpause ytdlp_client_dag_v2.1 docker compose exec airflow-worker airflow dags unpause ytdlp_client_dag_v2.1
// Check if works from server deploy // Try UI or recheck if works from server deploy
docker compose exec airflow-worker airflow dags trigger ytdlp_client_dag_v2.1 -c '{"url": "https://www.youtube.com/watch?v=sOlTX9uxUtM", "account_id": "test_direct", "redis_enabled": false, "service_ip": "85.192.30.55", "service_port": 9090}' docker compose exec airflow-worker airflow dags trigger ytdlp_client_dag_v2.1 -c '{"url": "https://www.youtube.com/watch?v=sOlTX9uxUtM", "redis_enabled": false, "service_ip": "85.192.30.55", "service_port": 9090}'
``` ```
Check Redis for stored data Check Redis for stored data by videoID
```bash ```bash
docker compose exec redis redis-cli -a XXXXXX -h 89.253.221.173 -p 52909 HGETALL "token_info:sOlTX9uxUtM" | jq -R -s 'split("\n") | del(.[] | select(. == "")) | [.[range(0;length;2)]]' docker compose exec redis redis-cli -a XXXXXX -h 89.253.221.173 -p 52909 HGETALL "token_info:sOlTX9uxUtM" | jq -R -s 'split("\n") | del(.[] | select(. == "")) | [.[range(0;length;2)]]'
``` ```