255 lines
8.8 KiB
Markdown
255 lines
8.8 KiB
Markdown
# Ansible Playbooks Documentation
|
|
|
|
This document provides an overview of all available playbooks, their purpose, and how to use them operationally.
|
|
|
|
## Table of Contents
|
|
|
|
1. [Deployment Workflow](#deployment-workflow)
|
|
2. [Initial Setup Playbooks](#initial-setup-playbooks)
|
|
3. [Operational Playbooks](#operational-playbooks)
|
|
4. [Monitoring and Inspection](#monitoring-and-inspection)
|
|
5. [Common Operations](#common-operations)
|
|
|
|
## Deployment Workflow
|
|
|
|
The typical deployment workflow follows these steps:
|
|
|
|
1. **Initial Setup**: Configure machines, deploy proxies, install dependencies
|
|
2. **Master Setup**: Configure Redis, MinIO, and other infrastructure services
|
|
3. **Worker Setup**: Configure auth generators and download simulators
|
|
4. **Operational Management**: Start/stop processes, monitor status, cleanup profiles
|
|
|
|
## Initial Setup Playbooks
|
|
|
|
For a complete, automated installation on fresh nodes, you can use the main `full-install` playbook:
|
|
|
|
```bash
|
|
# Run the complete installation process on all nodes
|
|
ansible-playbook ansible/playbook-full-install.yml -i ansible/inventory.green.ini
|
|
```
|
|
|
|
The steps below are for running each part of the installation manually.
|
|
|
|
### Base Installation
|
|
|
|
```bash
|
|
# Deploy base system requirements to all nodes
|
|
ansible-playbook ansible/playbook-base-system.yml -i ansible/inventory.green.ini
|
|
```
|
|
|
|
### Proxy Deployment
|
|
|
|
```bash
|
|
# Deploy shadowsocks proxies to all nodes (as defined in cluster config)
|
|
ansible-playbook ansible/playbook-proxies.yml -i ansible/inventory.green.ini
|
|
```
|
|
|
|
### Code, Environment, and Dependencies
|
|
|
|
```bash
|
|
# Sync code to all nodes
|
|
ansible-playbook ansible/playbook-stress-sync-code.yml -i ansible/inventory.green.ini
|
|
|
|
# Generate .env files for all nodes
|
|
ansible-playbook ansible/playbook-stress-generate-env.yml -i ansible/inventory.green.ini
|
|
|
|
# Install dependencies on all nodes
|
|
ansible-playbook ansible/playbook-stress-install-deps.yml -i ansible/inventory.green.ini
|
|
```
|
|
|
|
## Operational Playbooks
|
|
|
|
### Master Node Services
|
|
|
|
Redis and MinIO are deployed as Docker containers during the initial setup (`playbook-full-install.yml`).
|
|
|
|
To start the policy enforcer and monitoring tmux sessions on the master node:
|
|
```bash
|
|
# Start policy enforcer and monitoring on master
|
|
ansible-playbook ansible/playbook-stress-manage-processes.yml -i ansible/inventory.green.ini -e "start_enforcer=true start_monitor=true"
|
|
```
|
|
|
|
To stop processes on the master node:
|
|
```bash
|
|
# Stop ONLY the policy enforcer
|
|
ansible-playbook ansible/playbook-stress-manage-processes.yml -i ansible/inventory.green.ini -e "stop_enforcer=true"
|
|
|
|
# Stop ONLY the monitor
|
|
ansible-playbook ansible/playbook-stress-manage-processes.yml -i ansible/inventory.green.ini -e "stop_monitor=true"
|
|
|
|
# Stop BOTH the enforcer and monitor
|
|
ansible-playbook ansible/playbook-stress-manage-processes.yml -i ansible/inventory.green.ini -e "stop_sessions=true"
|
|
```
|
|
|
|
### Worker Node Processes
|
|
|
|
```bash
|
|
# Start auth generators and download simulators on all workers
|
|
ansible-playbook ansible/playbook-stress-lifecycle.yml -i ansible/inventory.green.ini -e "action=start"
|
|
|
|
# Start ONLY auth generators on workers
|
|
ansible-playbook ansible/playbook-stress-lifecycle.yml -i ansible/inventory.green.ini -e "action=start-auth"
|
|
|
|
# Start ONLY download simulators on workers
|
|
ansible-playbook ansible/playbook-stress-lifecycle.yml -i ansible/inventory.green.ini -e "action=start-download"
|
|
|
|
# Stop all processes on workers
|
|
ansible-playbook ansible/playbook-stress-lifecycle.yml -i ansible/inventory.green.ini -e "action=stop"
|
|
|
|
# Stop ONLY auth generators on workers
|
|
ansible-playbook ansible/playbook-stress-lifecycle.yml -i ansible/inventory.green.ini -e "action=stop-auth"
|
|
|
|
# Stop ONLY download simulators on workers
|
|
ansible-playbook ansible/playbook-stress-lifecycle.yml -i ansible/inventory.green.ini -e "action=stop-download"
|
|
|
|
# Check status of all worker processes
|
|
ansible-playbook ansible/playbook-stress-lifecycle.yml -i ansible/inventory.green.ini -e "action=status"
|
|
```
|
|
|
|
### Profile Management
|
|
|
|
The `cleanup-profiles` action can be used to remove profiles from Redis. By default, it cleans up "ungrouped" profiles.
|
|
|
|
```bash
|
|
# Perform a dry run of cleaning up ungrouped profiles
|
|
ansible-playbook ansible/playbook-stress-control.yml -i ansible/inventory.green.ini -e "action=cleanup-profiles" -e "dry_run=true"
|
|
|
|
# Clean up ungrouped profiles
|
|
ansible-playbook ansible/playbook-stress-control.yml -i ansible/inventory.green.ini -e "action=cleanup-profiles"
|
|
|
|
# To clean up ALL profiles (destructive), set cleanup_mode=full
|
|
ansible-playbook ansible/playbook-stress-control.yml -i ansible/inventory.green.ini -e "action=cleanup-profiles" -e "cleanup_mode=full"
|
|
|
|
# You can specify a custom setup policy file for cleanup operations
|
|
ansible-playbook ansible/playbook-stress-control.yml -i ansible/inventory.green.ini -e "action=cleanup-profiles" -e "setup_policy=policies/my_custom_setup_policy.yaml"
|
|
```
|
|
|
|
## Monitoring and Inspection
|
|
|
|
### Status Checks
|
|
|
|
```bash
|
|
# Check status of all processes on all nodes
|
|
ansible-playbook ansible/playbook-stress-control.yml -i ansible/inventory.green.ini -e "action=status"
|
|
|
|
# Check enforcer status on master
|
|
ansible-playbook ansible/playbook-stress-control.yml -i ansible/inventory.green.ini -e "action=check-enforcer"
|
|
|
|
# Check profile status
|
|
ansible-playbook ansible/playbook-stress-control.yml -i ansible/inventory.green.ini -e "action=profile-status"
|
|
```
|
|
|
|
### Log Inspection
|
|
|
|
```bash
|
|
# View tmux session output on a specific node
|
|
ssh user@hostname
|
|
tmux attach -t stress-auth-user1,user2 # For auth generator
|
|
tmux attach -t stress-download-user1,user2 # For download simulator
|
|
tmux attach -t stress-enforcer # For policy enforcer on master
|
|
```
|
|
|
|
## Common Operations
|
|
|
|
### Code and Policy Updates
|
|
|
|
First, sync your local changes to the jump host from your development machine:
|
|
```bash
|
|
# Sync project to jump host
|
|
./tools/sync-to-jump.sh
|
|
```
|
|
|
|
Then, from the jump host, you can sync code or policies to the cluster nodes:
|
|
```bash
|
|
# Sync all application code (Python sources, scripts, etc.)
|
|
ansible-playbook ansible/playbook-stress-sync-code.yml -i ansible/inventory.green.ini
|
|
|
|
# Sync only policies and CLI configs
|
|
ansible-playbook ansible/playbook-stress-sync-policies.yml -i ansible/inventory.green.ini
|
|
|
|
# To sync files from a custom source directory on the Ansible controller, use the 'source_base_dir' extra variable:
|
|
ansible-playbook ansible/playbook-stress-sync-policies.yml -i ansible/inventory.green.ini -e "source_base_dir=/path/to/my-custom-source"
|
|
```
|
|
|
|
### Docker Image Updates
|
|
|
|
To update the `yt-dlp` docker image used by download simulators, run the following playbook. This builds the image locally on each worker node.
|
|
|
|
```bash
|
|
# Build the yt-dlp docker image locally on each worker node
|
|
ansible-playbook ansible/playbook-update-yt-dlp-docker.yml -i ansible/inventory.green.ini
|
|
```
|
|
|
|
### Adding a New Worker
|
|
|
|
1. Update `cluster.green.yml` with the new worker definition:
|
|
```yaml
|
|
workers:
|
|
new-worker:
|
|
ip: x.x.x.x
|
|
port: 22
|
|
profile_prefixes:
|
|
- "user4"
|
|
proxies:
|
|
- "sslocal-rust-1090"
|
|
```
|
|
|
|
2. Regenerate inventory:
|
|
```bash
|
|
./tools/generate-inventory.py cluster.green.yml
|
|
```
|
|
|
|
3. Run the full installation playbook, limiting it to the new worker:
|
|
```bash
|
|
ansible-playbook ansible/playbook-full-install.yml -i ansible/inventory.green.ini --limit new-worker
|
|
```
|
|
|
|
4. Start processes on the new worker:
|
|
```bash
|
|
ansible-playbook ansible/playbook-stress-lifecycle.yml -i ansible/inventory.green.ini -e "action=start" --limit new-worker
|
|
```
|
|
|
|
### Removing a Worker
|
|
|
|
1. Stop all processes on the worker:
|
|
```bash
|
|
ansible-playbook ansible/playbook-stress-lifecycle.yml -i ansible/inventory.green.ini -e "action=stop" --limit worker-to-remove
|
|
```
|
|
|
|
2. Remove the worker from `cluster.green.yml`
|
|
|
|
3. Regenerate inventory:
|
|
```bash
|
|
./tools/generate-inventory.py cluster.green.yml
|
|
```
|
|
|
|
### Emergency Stop All
|
|
|
|
```bash
|
|
# Stop all processes on all nodes
|
|
ansible-playbook ansible/playbook-stress-control.yml -i ansible/inventory.green.ini -e "action=stop-all"
|
|
```
|
|
|
|
### Stopping Specific Nodes
|
|
|
|
To stop processes on a specific worker or group of workers, you can use the `stop-nodes` action and limit the playbook run.
|
|
|
|
```bash
|
|
# Stop all processes on a single worker (e.g., dl003)
|
|
ansible-playbook ansible/playbook-stress-control.yml -i ansible/inventory.green.ini -e "action=stop-nodes" --limit dl003
|
|
|
|
# Stop all processes on all workers
|
|
ansible-playbook ansible/playbook-stress-control.yml -i ansible/inventory.green.ini -e "action=stop-nodes" --limit workers
|
|
```
|
|
|
|
### Restart Enforcer and Monitoring
|
|
|
|
```bash
|
|
# Restart monitoring and enforcer on master using default policies
|
|
ansible-playbook ansible/playbook-stress-control.yml -i ansible/inventory.green.ini -e "action=restart-monitoring"
|
|
|
|
# Restart using a custom enforcer policy
|
|
ansible-playbook ansible/playbook-stress-control.yml -i ansible/inventory.green.ini -e "action=restart-monitoring" -e "enforcer_policy=policies/my_other_enforcer.yaml"
|
|
```
|
|
|