ca-grow-ops-manager/specs/photo-management.md

Photo Management & Storage Optimization

Priority: 🔴 Critical (Required for Phase 2)
Team: 777 Wolfpack
Date: 2025-12-09
Status: Technical Spec

---

📋 Overview

Optimize photo storage for daily walkthroughs and plant touch points to minimize storage costs and bandwidth usage while maintaining quality for compliance and documentation.

Key Constraint: Photos will be taken frequently (multiple times per day, per user) and need to be stored long-term for compliance.

---

🎯 Storage Strategy

Option 1: Local Storage with Compression (Recommended)

Pros:

• No external service costs
• Full control over data
• HIPAA/compliance friendly
• Fast access

Cons:

• Need to manage disk space
• Need backup strategy

Option 2: S3-Compatible Storage (Future)

Pros:

• Scalable
• Built-in redundancy
• CDN integration

Cons:

• Monthly costs
• External dependency

Recommendation: Start with Option 1 (local storage), migrate to Option 2 when needed.

---

📸 Image Optimization Strategy

1. Client-Side Compression (Before Upload)

Technology: Browser Canvas API or browser-image-compression library

Settings:

• Max Width: 1920px (sufficient for compliance)
• Max Height: 1920px
• Quality: 0.8 (80% - good balance)
• Format: WebP (30-50% smaller than JPEG)
• Fallback: JPEG for older browsers

Expected Savings:

• Original: ~3-5 MB per photo
• Compressed: ~200-500 KB per photo
• Reduction: 85-90%

2. Server-Side Processing

Technology: Sharp (Node.js image processing)

Pipeline:

1. Receive uploaded image
2. Generate multiple sizes:
   • Thumbnail: 200x200px (for lists/previews)
   • Medium: 800x800px (for detail views)
   • Full: 1920x1920px (for compliance/download)
3. Convert to WebP
4. Store all versions

Storage Per Photo:

• Thumbnail: ~10-20 KB
• Medium: ~50-100 KB
• Full: ~200-500 KB
• Total: ~300-700 KB (vs 3-5 MB original)

---

🗄️ Storage Structure

File System Organization

/srv/storage/ca-grow-ops-manager/
├── photos/
│   ├── walkthroughs/
│   │   ├── 2025/
│   │   │   ├── 12/
│   │   │   │   ├── 09/
│   │   │   │   │   ├── {walkthrough-id}/
│   │   │   │   │   │   ├── reservoir/
│   │   │   │   │   │   │   ├── {photo-id}_thumb.webp
│   │   │   │   │   │   │   ├── {photo-id}_medium.webp
│   │   │   │   │   │   │   └── {photo-id}_full.webp
│   │   │   │   │   │   ├── irrigation/
│   │   │   │   │   │   └── plant-health/
│   ├── touch-points/
│   │   ├── 2025/
│   │   │   ├── 12/
│   │   │   │   └── ...
└── backups/
    └── ...

Database Storage (URLs Only)

// Store relative paths, not full URLs
photoUrl: "/photos/walkthroughs/2025/12/09/{walkthrough-id}/reservoir/{photo-id}_full.webp"
thumbnailUrl: "/photos/walkthroughs/2025/12/09/{walkthrough-id}/reservoir/{photo-id}_thumb.webp"

---

🔧 Implementation

Backend: Photo Upload Endpoint

// backend/src/routes/upload.routes.ts
import { FastifyInstance } from 'fastify';
import multipart from '@fastify/multipart';
import sharp from 'sharp';
import { promises as fs } from 'fs';
import path from 'path';

export async function uploadRoutes(server: FastifyInstance) {
    server.register(multipart, {
        limits: {
            fileSize: 10 * 1024 * 1024, // 10MB max (before compression)
        },
    });

    server.post('/upload/photo', async (request, reply) => {
        const data = await request.file();
        
        if (!data) {
            return reply.code(400).send({ message: 'No file uploaded' });
        }

        const buffer = await data.toBuffer();
        const photoId = generatePhotoId();
        const date = new Date();
        const basePath = `/srv/storage/ca-grow-ops-manager/photos/walkthroughs/${date.getFullYear()}/${date.getMonth() + 1}/${date.getDate()}`;
        
        // Ensure directory exists
        await fs.mkdir(basePath, { recursive: true });

        // Generate 3 sizes
        const sizes = [
            { name: 'thumb', width: 200, height: 200 },
            { name: 'medium', width: 800, height: 800 },
            { name: 'full', width: 1920, height: 1920 },
        ];

        const urls: Record<string, string> = {};

        for (const size of sizes) {
            const filename = `${photoId}_${size.name}.webp`;
            const filepath = path.join(basePath, filename);
            
            await sharp(buffer)
                .resize(size.width, size.height, {
                    fit: 'inside',
                    withoutEnlargement: true,
                })
                .webp({ quality: 80 })
                .toFile(filepath);
            
            urls[size.name] = `/photos/walkthroughs/${date.getFullYear()}/${date.getMonth() + 1}/${date.getDate()}/${filename}`;
        }

        return {
            photoId,
            urls,
        };
    });
}

function generatePhotoId(): string {
    return `photo_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
}

Frontend: Photo Compression

// frontend/src/lib/photoCompression.ts
import imageCompression from 'browser-image-compression';

export async function compressPhoto(file: File): Promise<File> {
    const options = {
        maxSizeMB: 1, // Max 1MB
        maxWidthOrHeight: 1920,
        useWebWorker: true,
        fileType: 'image/webp',
    };

    try {
        const compressedFile = await imageCompression(file, options);
        return compressedFile;
    } catch (error) {
        console.error('Photo compression failed:', error);
        return file; // Return original if compression fails
    }
}

// Usage in component
async function handlePhotoCapture(event: React.ChangeEvent<HTMLInputElement>) {
    const file = event.target.files?.[0];
    if (!file) return;

    setIsUploading(true);
    
    // Compress before upload
    const compressedFile = await compressPhoto(file);
    
    // Upload
    const formData = new FormData();
    formData.append('file', compressedFile);
    
    const response = await api.post('/upload/photo', formData, {
        headers: { 'Content-Type': 'multipart/form-data' },
    });
    
    setPhotoUrl(response.data.urls.full);
    setIsUploading(false);
}

---

📊 Storage Estimates

Daily Usage (777 Wolfpack)

Assumptions:

• 3 staff members
• 1 walkthrough per day per person
• Average 5 photos per walkthrough
• 365 days per year

Calculations:

• Photos per day: 3 × 5 = 15 photos
• Storage per photo: ~500 KB (compressed)
• Daily storage: 15 × 500 KB = 7.5 MB/day
• Monthly storage: 7.5 MB × 30 = 225 MB/month
• Yearly storage: 7.5 MB × 365 = 2.7 GB/year

With Touch Points (Phase 2):

• Assume 10 touch points per day per person
• 2 photos per touch point
• Additional photos per day: 3 × 10 × 2 = 60 photos
• Additional storage: 60 × 500 KB = 30 MB/day
• Total daily: 37.5 MB/day
• Total yearly: ~13.7 GB/year

5-Year Projection: ~68 GB (very manageable)

---

💾 Backup Strategy

Local Backups

Frequency: Daily at 2 AM

Method: rsync to backup directory

rsync -av --delete \
  /srv/storage/ca-grow-ops-manager/photos/ \
  /srv/backups/ca-grow-ops-manager/photos/

Off-Site Backups (Recommended)

Frequency: Weekly

Options:

1. Restic to Backblaze B2 (~$0.005/GB/month)
2. Rclone to any S3-compatible storage
3. Tarsnap for encrypted backups

Cost Estimate (Backblaze B2):

• 70 GB × $0.005 = $0.35/month (5-year data)
• Very affordable

---

🔒 Security & Compliance

Access Control

• Photos only accessible to authenticated users
• Serve via backend proxy (not direct file access)
• Check user permissions before serving

Retention Policy

• Keep all photos for 7 years (CA compliance)
• Automatic archival after 2 years (move to cold storage)
• Deletion only after retention period

Privacy

• No EXIF data stored (stripped during processing)
• No GPS coordinates
• Timestamps preserved for audit trail

---

📦 Dependencies

Backend

{
  "dependencies": {
    "@fastify/multipart": "^8.0.0",
    "sharp": "^0.33.0"
  }
}

Frontend

{
  "dependencies": {
    "browser-image-compression": "^2.0.2"
  }
}

---

✅ Implementation Checklist

Phase 1: Basic Upload (1-2 hours)

• Install dependencies (sharp, multipart)
• Create upload endpoint
• Create storage directory structure
• Test upload with Postman

Phase 2: Compression (2-3 hours)

• Implement server-side compression (3 sizes)
• Add client-side compression
• Test compression quality
• Measure storage savings

Phase 3: Integration (2-3 hours)

• Add photo upload to walkthrough UI
• Add photo upload to touch points UI
• Display thumbnails in lists
• Display full images in detail views

Phase 4: Optimization (1-2 hours)

• Add lazy loading for images
• Implement progressive image loading
• Add image caching headers
• Test on slow connections

Phase 5: Backup (1 hour)

• Set up daily local backups
• Set up weekly off-site backups
• Test restore process

Total Estimated Time: 7-11 hours

---

🎯 Success Metrics

Storage Efficiency

• Target: < 500 KB per photo (compressed)
• Target: < 50 MB per day (all photos)
• Target: < 20 GB per year (all photos)

Performance

• Upload time: < 5 seconds per photo
• Display time: < 1 second (thumbnail)
• Compression time: < 2 seconds client-side

Reliability

• 99.9% uptime for photo access
• Zero data loss
• Successful daily backups

---

💡 Future Enhancements

Advanced Features

• AI-powered image tagging
• Automatic pest detection from photos
• Image similarity search
• Timelapse generation from touch point photos
• OCR for reading labels/tags in photos

Storage Optimization

• Migrate old photos to cold storage
• Implement CDN for faster access
• Add image deduplication
• Compress old photos further (lower quality)

---

📝 Notes

WebP Support

• Supported by all modern browsers (95%+ coverage)
• Fallback to JPEG for older browsers
• 30-50% smaller than JPEG at same quality

Sharp vs Other Libraries

• Sharp: Fastest, best quality, recommended
• ImageMagick: Slower, more features
• Jimp: Pure JS, slower, no native deps

Recommendation: Use Sharp for production

---

Status: ✅ Spec Complete - Ready for Implementation
Priority: Implement before enabling photo uploads in production
Estimated Effort: 7-11 hours