# BizNode Backup System

Automated daily backup of PostgreSQL database and Qdrant vector collections to Google Drive.

---

## Overview

| Component | What gets backed up |
|-----------|-------------------|
| **PostgreSQL** | Full SQL dump of `biznode` database (all tables, schema + data) |
| **Qdrant** | Snapshots of all vector collections (`biznode_memory`, `biznode_support`, workspace collections) |

**Storage locations:**
- **Local**: `backups/YYYY-MM-DD/` — retained for 7 days
- **Google Drive**: `BizNode_Backups/YYYY-MM-DD/` — retained for 30 days

---

## File Structure

```
backup/
  __init__.py             # Package init
  backup_engine.py        # Core logic: PostgreSQL dump, Qdrant snapshots, orchestration
  gdrive_uploader.py      # Google Drive OAuth2 auth, upload, folder management, cleanup
  run_backup.py           # CLI entry point: manual runs, catch-up checks, scheduler install
  setup_gdrive.py         # One-time Google Drive authorization setup
  fix_schedule.py         # Updates Windows Task Scheduler with catch-up settings
  credentials.json        # Google OAuth client credentials (DO NOT COMMIT)
  token.json              # Saved OAuth token (DO NOT COMMIT)
  last_backup.json        # Tracks last backup time for catch-up logic (DO NOT COMMIT)
  README.md               # This file
```

---

## First-Time Setup

### 1. Google Cloud Credentials

1. Go to [Google Cloud Console](https://console.cloud.google.com/)
2. Create a new project (or select existing)
3. Enable **Google Drive API**: APIs & Services > Library > search "Google Drive API" > Enable
4. Configure **OAuth consent screen**: APIs & Services > OAuth consent screen
   - User type: External
   - Add your Google email as a test user under Audience > Test users
5. Create **OAuth credentials**: APIs & Services > Credentials > Create Credentials > OAuth client ID
   - Application type: Desktop app
   - Download the JSON file
6. Save as `backup/credentials.json`

### 2. Authorize

```bash
python backup/setup_gdrive.py
```

A browser window opens — sign in with your Google account and grant access.
The token is saved to `backup/token.json` for future use.

### 3. Test Run

```bash
python backup/run_backup.py
```

This will:
- Dump the PostgreSQL database using `pg_dump`
- Create and download Qdrant snapshots for all collections
- Upload everything to Google Drive under `BizNode_Backups/YYYY-MM-DD/`

### 4. Schedule Daily Backup

```bash
python backup/run_backup.py --schedule
```

Installs a Windows Task Scheduler task:
- **Name**: `BizNodeDailyBackup`
- **Schedule**: Daily at 5:00 PM
- **Catch-up**: If the computer is off at 5 PM, the backup runs automatically on next login

---

## Usage

### Commands

```bash
# Full backup (PostgreSQL + Qdrant + Google Drive upload)
python backup/run_backup.py

# Local backup only (no Google Drive upload)
python backup/run_backup.py --local

# Check if backup is overdue and run if needed (used by scheduler)
python backup/run_backup.py --check

# Install/reinstall the daily 5 PM scheduled task
python backup/run_backup.py --schedule
```

### Managing the Scheduled Task

```bash
# View task details
schtasks /Query /TN BizNodeDailyBackup /V /FO LIST

# Manually trigger the task
schtasks /Run /TN BizNodeDailyBackup

# Disable the task
schtasks /Change /TN BizNodeDailyBackup /DISABLE

# Re-enable the task
schtasks /Change /TN BizNodeDailyBackup /ENABLE

# Delete the task
schtasks /Delete /TN BizNodeDailyBackup /F
```

---

## How It Works

### Backup Flow

```
1. PostgreSQL Dump
   pg_dump → backups/YYYY-MM-DD/postgresql_biznode_TIMESTAMP.sql

2. Qdrant Snapshots (per collection, with retry)
   POST /collections/{name}/snapshots  → create snapshot
   GET  /collections/{name}/snapshots/{id} → download
   DELETE /collections/{name}/snapshots/{id} → cleanup remote
   → backups/YYYY-MM-DD/qdrant_{collection}_TIMESTAMP.snapshot

3. Google Drive Upload
   Authenticate via OAuth2 (token.json)
   Create folder: BizNode_Backups/YYYY-MM-DD/
   Upload all backup files
   Delete Google Drive backups older than 30 days

4. Local Cleanup
   Delete local backup files older than 7 days
```

### Catch-Up Logic

The `--check` flag (used by the scheduled task) implements catch-up:

1. Reads `last_backup.json` to find the last successful backup time
2. If no backup exists or the last one was more than 20 hours ago, runs a full backup
3. If a recent backup exists, skips with a log message
4. Windows Task Scheduler's `StartWhenAvailable` setting ensures the task runs on next login if the scheduled time was missed

### Retry Logic

Qdrant snapshot creation and download includes automatic retry:
- Up to 3 attempts per collection
- 5-second delay between retries
- Increased timeouts (300s for snapshot creation, 600s for download)
- 64KB chunk size for faster downloads

---

## Configuration

All configuration is read from the project `.env` file:

| Variable | Used For | Default |
|----------|----------|---------|
| `DB_TYPE` | Database engine | `sqlite` |
| `DB_HOST` | PostgreSQL host | `localhost` |
| `DB_PORT` | PostgreSQL port | `5432` |
| `DB_NAME` | Database name | `biznode` |
| `DB_USER` | Database user | `biznode_user` |
| `DB_PASSWORD` | Database password | — |
| `QDRANT_HOST` | Qdrant host | `localhost` |
| `QDRANT_PORT` | Qdrant port | `6333` |

### Internal Paths

| Path | Purpose |
|------|---------|
| `C:\Program Files\PostgreSQL\18\bin\pg_dump.exe` | PostgreSQL dump tool |
| `backups/` | Local backup storage |
| `logs/backup.log` | Backup log file |
| `backup/last_backup.json` | Last backup state tracker |

---

## Retention Policy

| Location | Retention | Cleanup |
|----------|-----------|---------|
| Local (`backups/`) | 7 days | Automatic after each backup |
| Google Drive (`BizNode_Backups/`) | 30 days | Automatic — old date folders deleted |

---

## Logs

Backup logs are written to both the console and `logs/backup.log`:

```
2026-03-07 12:00:00 [biznode.backup] INFO: Dumping PostgreSQL database 'biznode' ...
2026-03-07 12:00:01 [biznode.backup] INFO: PostgreSQL dump complete: postgresql_biznode_20260307.sql (0.07 MB)
2026-03-07 12:00:01 [biznode.backup] INFO: Found 3 Qdrant collections: [...]
2026-03-07 12:01:30 [biznode.backup] INFO: Qdrant snapshot saved: qdrant_biznode_memory_20260307.snapshot (592.31 MB)
2026-03-07 12:03:00 [biznode.backup.gdrive] INFO: Uploaded postgresql_biznode_20260307.sql (0.07 MB)
```

---

## Troubleshooting

### "Google Drive API has not been used in project..."
Enable the Google Drive API at:
`https://console.cloud.google.com/apis/api/drive.googleapis.com/overview`

### "access_denied" during OAuth
Add your Google email as a test user:
Google Cloud Console > APIs & Services > OAuth consent screen > Audience > Test users > Add

### "credentials.json not found"
Download OAuth client credentials from Google Cloud Console and save as `backup/credentials.json`.

### "pg_dump failed"
- Verify PostgreSQL is running: `pg_isready -h localhost -p 5432`
- Check credentials in `.env` (`DB_USER`, `DB_PASSWORD`)
- Verify pg_dump path exists: `"C:\Program Files\PostgreSQL\18\bin\pg_dump.exe" --version`

### Qdrant snapshot timeout
- Large collections may take several minutes to snapshot
- The system retries up to 3 times automatically
- Check Qdrant is running: `curl http://localhost:6333/collections`

### Token expired
Delete `backup/token.json` and run `python backup/setup_gdrive.py` to re-authorize.

---

## Security Notes

The following files contain sensitive data and are excluded from git:

- `backup/credentials.json` — Google OAuth client secret
- `backup/token.json` — Google OAuth access/refresh token
- `backup/last_backup.json` — backup state (not sensitive, but runtime data)

These are listed in `.gitignore`. Never commit them.
