ca-grow-ops-manager/specs/photo-management.md

# Photo Management & Storage Optimization

**Priority**: 🔴 Critical (Required for Phase 2)  
**Team**: 777 Wolfpack  
**Date**: 2025-12-09  
**Status**: Technical Spec

---

## 📋 Overview

Optimize photo storage for daily walkthroughs and plant touch points to minimize storage costs and bandwidth usage while maintaining quality for compliance and documentation.

**Key Constraint**: Photos will be taken frequently (multiple times per day, per user) and need to be stored long-term for compliance.

---

## 🎯 Storage Strategy

### Option 1: Local Storage with Compression (Recommended)

**Pros**:

- No external service costs
- Full control over data
- HIPAA/compliance friendly
- Fast access

**Cons**:

- Need to manage disk space
- Need backup strategy

### Option 2: S3-Compatible Storage (Future)

**Pros**:

- Scalable
- Built-in redundancy
- CDN integration

**Cons**:

- Monthly costs
- External dependency

**Recommendation**: Start with **Option 1** (local storage), migrate to **Option 2** when needed.

---

## 📸 Image Optimization Strategy

### 1. Client-Side Compression (Before Upload)

**Technology**: Browser Canvas API or `browser-image-compression` library

**Settings**:

- **Max Width**: 1920px (sufficient for compliance)
- **Max Height**: 1920px
- **Quality**: 0.8 (80% - good balance)
- **Format**: WebP (30-50% smaller than JPEG)
- **Fallback**: JPEG for older browsers

**Expected Savings**:

- Original: ~3-5 MB per photo
- Compressed: ~200-500 KB per photo
- **Reduction**: 85-90%

### 2. Server-Side Processing

**Technology**: Sharp (Node.js image processing)

**Pipeline**:

1. Receive uploaded image
2. Generate multiple sizes:
   - **Thumbnail**: 200x200px (for lists/previews)
   - **Medium**: 800x800px (for detail views)
   - **Full**: 1920x1920px (for compliance/download)
3. Convert to WebP
4. Store all versions

**Storage Per Photo**:

- Thumbnail: ~10-20 KB
- Medium: ~50-100 KB
- Full: ~200-500 KB
- **Total**: ~300-700 KB (vs 3-5 MB original)

---

## 🗄️ Storage Structure

### File System Organization

```
/srv/storage/ca-grow-ops-manager/
├── photos/
│   ├── walkthroughs/
│   │   ├── 2025/
│   │   │   ├── 12/
│   │   │   │   ├── 09/
│   │   │   │   │   ├── {walkthrough-id}/
│   │   │   │   │   │   ├── reservoir/
│   │   │   │   │   │   │   ├── {photo-id}_thumb.webp
│   │   │   │   │   │   │   ├── {photo-id}_medium.webp
│   │   │   │   │   │   │   └── {photo-id}_full.webp
│   │   │   │   │   │   ├── irrigation/
│   │   │   │   │   │   └── plant-health/
│   ├── touch-points/
│   │   ├── 2025/
│   │   │   ├── 12/
│   │   │   │   └── ...
└── backups/
    └── ...
```

### Database Storage (URLs Only)

```typescript
// Store relative paths, not full URLs
photoUrl: "/photos/walkthroughs/2025/12/09/{walkthrough-id}/reservoir/{photo-id}_full.webp"
thumbnailUrl: "/photos/walkthroughs/2025/12/09/{walkthrough-id}/reservoir/{photo-id}_thumb.webp"
```

---

## 🔧 Implementation

### Backend: Photo Upload Endpoint

```typescript
// backend/src/routes/upload.routes.ts
import { FastifyInstance } from 'fastify';
import multipart from '@fastify/multipart';
import sharp from 'sharp';
import { promises as fs } from 'fs';
import path from 'path';

export async function uploadRoutes(server: FastifyInstance) {
    server.register(multipart, {
        limits: {
            fileSize: 10 * 1024 * 1024, // 10MB max (before compression)
        },
    });

    server.post('/upload/photo', async (request, reply) => {
        const data = await request.file();
        
        if (!data) {
            return reply.code(400).send({ message: 'No file uploaded' });
        }

        const buffer = await data.toBuffer();
        const photoId = generatePhotoId();
        const date = new Date();
        const basePath = `/srv/storage/ca-grow-ops-manager/photos/walkthroughs/${date.getFullYear()}/${date.getMonth() + 1}/${date.getDate()}`;
        
        // Ensure directory exists
        await fs.mkdir(basePath, { recursive: true });

        // Generate 3 sizes
        const sizes = [
            { name: 'thumb', width: 200, height: 200 },
            { name: 'medium', width: 800, height: 800 },
            { name: 'full', width: 1920, height: 1920 },
        ];

        const urls: Record<string, string> = {};

        for (const size of sizes) {
            const filename = `${photoId}_${size.name}.webp`;
            const filepath = path.join(basePath, filename);
            
            await sharp(buffer)
                .resize(size.width, size.height, {
                    fit: 'inside',
                    withoutEnlargement: true,
                })
                .webp({ quality: 80 })
                .toFile(filepath);
            
            urls[size.name] = `/photos/walkthroughs/${date.getFullYear()}/${date.getMonth() + 1}/${date.getDate()}/${filename}`;
        }

        return {
            photoId,
            urls,
        };
    });
}

function generatePhotoId(): string {
    return `photo_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
}
```

### Frontend: Photo Compression

```typescript
// frontend/src/lib/photoCompression.ts
import imageCompression from 'browser-image-compression';

export async function compressPhoto(file: File): Promise<File> {
    const options = {
        maxSizeMB: 1, // Max 1MB
        maxWidthOrHeight: 1920,
        useWebWorker: true,
        fileType: 'image/webp',
    };

    try {
        const compressedFile = await imageCompression(file, options);
        return compressedFile;
    } catch (error) {
        console.error('Photo compression failed:', error);
        return file; // Return original if compression fails
    }
}

// Usage in component
async function handlePhotoCapture(event: React.ChangeEvent<HTMLInputElement>) {
    const file = event.target.files?.[0];
    if (!file) return;

    setIsUploading(true);
    
    // Compress before upload
    const compressedFile = await compressPhoto(file);
    
    // Upload
    const formData = new FormData();
    formData.append('file', compressedFile);
    
    const response = await api.post('/upload/photo', formData, {
        headers: { 'Content-Type': 'multipart/form-data' },
    });
    
    setPhotoUrl(response.data.urls.full);
    setIsUploading(false);
}
```

---

## 📊 Storage Estimates

### Daily Usage (777 Wolfpack)

**Assumptions**:

- 3 staff members
- 1 walkthrough per day per person
- Average 5 photos per walkthrough
- 365 days per year

**Calculations**:

- Photos per day: 3 × 5 = 15 photos
- Storage per photo: ~500 KB (compressed)
- Daily storage: 15 × 500 KB = 7.5 MB/day
- Monthly storage: 7.5 MB × 30 = 225 MB/month
- Yearly storage: 7.5 MB × 365 = 2.7 GB/year

**With Touch Points** (Phase 2):

- Assume 10 touch points per day per person
- 2 photos per touch point
- Additional photos per day: 3 × 10 × 2 = 60 photos
- Additional storage: 60 × 500 KB = 30 MB/day
- **Total daily**: 37.5 MB/day
- **Total yearly**: ~13.7 GB/year

**5-Year Projection**: ~68 GB (very manageable)

---

## 💾 Backup Strategy

### Local Backups

**Frequency**: Daily at 2 AM

**Method**: rsync to backup directory

```bash
rsync -av --delete \
  /srv/storage/ca-grow-ops-manager/photos/ \
  /srv/backups/ca-grow-ops-manager/photos/
```

### Off-Site Backups (Recommended)

**Frequency**: Weekly

**Options**:

1. **Restic** to Backblaze B2 (~$0.005/GB/month)
2. **Rclone** to any S3-compatible storage
3. **Tarsnap** for encrypted backups

**Cost Estimate** (Backblaze B2):

- 70 GB × $0.005 = $0.35/month (5-year data)
- Very affordable

---

## 🔒 Security & Compliance

### Access Control

- Photos only accessible to authenticated users
- Serve via backend proxy (not direct file access)
- Check user permissions before serving

### Retention Policy

- Keep all photos for 7 years (CA compliance)
- Automatic archival after 2 years (move to cold storage)
- Deletion only after retention period

### Privacy

- No EXIF data stored (stripped during processing)
- No GPS coordinates
- Timestamps preserved for audit trail

---

## 📦 Dependencies

### Backend

```json
{
  "dependencies": {
    "@fastify/multipart": "^8.0.0",
    "sharp": "^0.33.0"
  }
}
```

### Frontend

```json
{
  "dependencies": {
    "browser-image-compression": "^2.0.2"
  }
}
```

---

## ✅ Implementation Checklist

### Phase 1: Basic Upload (1-2 hours)

- [ ] Install dependencies (sharp, multipart)
- [ ] Create upload endpoint
- [ ] Create storage directory structure
- [ ] Test upload with Postman

### Phase 2: Compression (2-3 hours)

- [ ] Implement server-side compression (3 sizes)
- [ ] Add client-side compression
- [ ] Test compression quality
- [ ] Measure storage savings

### Phase 3: Integration (2-3 hours)

- [ ] Add photo upload to walkthrough UI
- [ ] Add photo upload to touch points UI
- [ ] Display thumbnails in lists
- [ ] Display full images in detail views

### Phase 4: Optimization (1-2 hours)

- [ ] Add lazy loading for images
- [ ] Implement progressive image loading
- [ ] Add image caching headers
- [ ] Test on slow connections

### Phase 5: Backup (1 hour)

- [ ] Set up daily local backups
- [ ] Set up weekly off-site backups
- [ ] Test restore process

**Total Estimated Time**: 7-11 hours

---

## 🎯 Success Metrics

### Storage Efficiency

- Target: < 500 KB per photo (compressed)
- Target: < 50 MB per day (all photos)
- Target: < 20 GB per year (all photos)

### Performance

- Upload time: < 5 seconds per photo
- Display time: < 1 second (thumbnail)
- Compression time: < 2 seconds client-side

### Reliability

- 99.9% uptime for photo access
- Zero data loss
- Successful daily backups

---

## 💡 Future Enhancements

### Advanced Features

- [ ] AI-powered image tagging
- [ ] Automatic pest detection from photos
- [ ] Image similarity search
- [ ] Timelapse generation from touch point photos
- [ ] OCR for reading labels/tags in photos

### Storage Optimization

- [ ] Migrate old photos to cold storage
- [ ] Implement CDN for faster access
- [ ] Add image deduplication
- [ ] Compress old photos further (lower quality)

---

## 📝 Notes

### WebP Support

- Supported by all modern browsers (95%+ coverage)
- Fallback to JPEG for older browsers
- 30-50% smaller than JPEG at same quality

### Sharp vs Other Libraries

- **Sharp**: Fastest, best quality, recommended
- **ImageMagick**: Slower, more features
- **Jimp**: Pure JS, slower, no native deps

**Recommendation**: Use Sharp for production

---

**Status**: ✅ Spec Complete - Ready for Implementation  
**Priority**: Implement before enabling photo uploads in production  
**Estimated Effort**: 7-11 hours