Debugging Guide¶

This guide provides systematic approaches to debug issues in the Local AI Cyber Lab environment.

Debugging Methodology¶

1. Information Gathering¶

System State¶

# Check system resources
htop
nvidia-smi
df -h

# Check Docker status
docker ps
docker stats

Logs Collection¶

# Collect service logs
docker-compose logs --tail=100 service-name

# Check system logs
journalctl -u docker
dmesg | grep -i error

2. Problem Isolation¶

Service Dependencies¶

Map service dependencies:
```
docker-compose config --services
```
Check service health:
```
curl http://localhost:port/health
```
Verify network connectivity:
```
docker network inspect bridge
```

Component Testing¶

Test individual components:

# Test AI Guardian
curl -X POST http://localhost:8000/api/v1/validate \
  -H "Authorization: Bearer ${AI_GUARDIAN_API_KEY}" \
  -d '{"prompt": "test"}'

# Test Database
docker-compose exec supabase-db psql -U postgres -c "\l"

Verify configurations:

# Check environment variables
docker-compose config

# Validate config files
yamllint docker-compose.yml

3. Debug Tools¶

Docker Debugging¶

Container inspection:

# Get container details
docker inspect container_id

# Execute commands in container
docker exec -it container_id /bin/bash

Resource monitoring:

# Monitor container resources
docker stats container_id

# Check container logs
docker logs -f container_id

Network Debugging¶

Network connectivity:

# Test inter-container networking
docker-compose exec service1 ping service2

# Check port bindings
netstat -tulpn | grep LISTEN

DNS resolution:

# Test DNS
docker-compose exec service nslookup other-service

4. Performance Debugging¶

CPU Profiling¶

Process monitoring:
```
top -H -p $(pgrep -d',' python)
```
System load analysis:
```
uptime
sar -u 1 10
```

Memory Analysis¶

Memory usage:
```
free -h
vmstat 1 10
```

Container memory:

docker stats --format "table {{.Name}}\t{{.MemUsage}}"

5. Security Debugging¶

Access Control¶

Check permissions:
```
ls -la /path/to/sensitive/files
```

Audit authentication:

# Review auth logs
tail -f /var/log/auth.log

# Check API key usage
grep "Authentication failed" logs/ai-guardian.log

Network Security¶

Port scanning:
```
nmap -p- localhost
```

TLS verification:

openssl s_client -connect localhost:443

6. Debug Logging¶

Enable Debug Logs¶

Update logging configuration:

# In Python services
import logging
logging.basicConfig(level=logging.DEBUG)

Docker logging:

# Update daemon.json
{
  "log-level": "debug"
}

Log Analysis¶

Parse logs:

# Search for errors
grep -r "ERROR" logs/

# Track requests
tail -f access.log | grep "POST /api"

Log aggregation:

# Combine service logs
docker-compose logs --tail=1000 > debug.log

7. Recovery Steps¶

Service Recovery¶

Restart services:
```
docker-compose restart service-name
```

Clean restart:

docker-compose down
docker system prune
docker-compose up -d

Data Recovery¶

Backup data:
```
./scripts/backup.sh
```
Restore from backup:
```
./scripts/restore.sh backup_file
```

Best Practices¶

Systematic Approach:
Start with logs
Isolate components
Test incrementally
Document findings
Monitoring:
Set up alerts
Monitor resource usage
Track error rates
Log key metrics
Prevention:
Regular health checks
Automated testing
Configuration validation
Security audits

Debug Checklist¶

[ ] Collect system information
[ ] Review recent changes
[ ] Check service status
[ ] Verify configurations
[ ] Test component isolation
[ ] Monitor resource usage
[ ] Review security logs
[ ] Document findings
[ ] Implement fixes
[ ] Verify solution
[ ] Update documentation