ca-grow-ops-manager/specs/paperless-integration.md

# Paperless-ngx Integration Specification

## Overview

Integrate CA Grow Ops Manager with a Paperless-ngx instance for long-term document storage, archival, and retrieval. Paperless-ngx provides OCR, full-text search, tagging, and archival capabilities that complement our application's document management needs.

## Architecture

```
┌─────────────────────────────────────────────────────┐
│                  CA Grow Ops Manager                │
│  ┌───────────────┐    ┌─────────────────────────┐  │
│  │  Documents    │    │  Paperless Service      │  │
│  │  (metadata)   │◄──►│  (API wrapper)          │  │
│  └───────────────┘    └───────────┬─────────────┘  │
└───────────────────────────────────┼─────────────────┘
                                    │ REST API
                                    ▼
                          ┌─────────────────────┐
                          │   Paperless-ngx     │
                          │   (Document Store)  │
                          │   - OCR             │
                          │   - Full-text       │
                          │   - Tagging         │
                          │   - Archival        │
                          └─────────────────────┘
```

## Use Cases

### 1. SOPs and Policies

- Upload SOPs from CA Grow Ops to Paperless for long-term archival
- Retrieve and display archived versions
- Full-text search across all SOPs

### 2. Compliance Documents

- Store state inspection reports
- Archive METRC reports and manifests
- Maintain license documentation

### 3. Batch Documentation

- Archive completed batch records
- Store harvest manifests
- Archive lab test results (COAs)

### 4. Visitor Logs

- Archive signed NDAs
- Store visitor badges and access logs

### 5. Photos (Optional)

- Archive walkthrough photos
- Store plant progress photos with metadata

## Paperless-ngx API Integration

### Authentication

```typescript
// Environment variables
PAPERLESS_URL=https://paperless.runfoo.run
PAPERLESS_TOKEN=your_api_token
```

### API Endpoints Used

| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/api/documents/` | GET | List/search documents |
| `/api/documents/` | POST | Upload new document |
| `/api/documents/{id}/` | GET | Get document metadata |
| `/api/documents/{id}/download/` | GET | Download original file |
| `/api/documents/{id}/preview/` | GET | Get thumbnail/preview |
| `/api/tags/` | GET/POST | Manage tags |
| `/api/correspondents/` | GET/POST | Manage correspondents (document owners) |
| `/api/document_types/` | GET/POST | Manage document types |

## Implementation

### Backend Service (`paperless.service.ts`)

```typescript
interface PaperlessDocument {
    id: number;
    title: string;
    content: string;
    tags: number[];
    document_type: number | null;
    correspondent: number | null;
    created: string;
    added: string;
    archive_serial_number: string | null;
    original_file_name: string;
}

interface UploadDocumentOptions {
    title: string;
    file: Buffer | ReadStream;
    filename: string;
    documentType?: string;
    tags?: string[];
    correspondent?: string;
    archiveSerialNumber?: string;
}

class PaperlessService {
    async uploadDocument(options: UploadDocumentOptions): Promise<PaperlessDocument>;
    async getDocument(id: number): Promise<PaperlessDocument>;
    async searchDocuments(query: string, tags?: string[]): Promise<PaperlessDocument[]>;
    async downloadDocument(id: number): Promise<Buffer>;
    async deleteDocument(id: number): Promise<void>;
    
    // Tag management
    async createTag(name: string, color?: string): Promise<Tag>;
    async getOrCreateTag(name: string): Promise<Tag>;
    
    // Document type management
    async getOrCreateDocumentType(name: string): Promise<DocumentType>;
}
```

### API Routes (`paperless.routes.ts`)

```typescript
// Upload document to Paperless
POST /api/paperless/documents
  Body: multipart/form-data
  - file: File
  - title: string
  - documentType: string (e.g., "SOP", "COMPLIANCE", "BATCH_RECORD")
  - tags: string[] (e.g., ["gorilla-glue", "batch-001"])

// Search Paperless documents
GET /api/paperless/search?q=keyword&tags=tag1,tag2

// Get document preview
GET /api/paperless/documents/:id/preview

// Download original document
GET /api/paperless/documents/:id/download

// Sync local document to Paperless
POST /api/paperless/sync/:localDocumentId

// Get sync status for local document
GET /api/paperless/sync-status/:localDocumentId
```

### Database Schema Addition

```prisma
model Document {
    // ... existing fields ...
    
    // Paperless sync fields
    paperlessId       Int?      @unique
    paperlessSyncedAt DateTime?
    paperlessUrl      String?
}
```

## Tagging Convention

Consistent tagging for easy retrieval:

| Tag Pattern | Example | Purpose |
|-------------|---------|---------|
| `app:grow-ops` | `app:grow-ops` | Source application |
| `type:{category}` | `type:sop`, `type:compliance` | Document category |
| `batch:{name}` | `batch:gg4-b001` | Associated batch |
| `room:{name}` | `room:flower-a` | Associated room |
| `year:{year}` | `year:2025` | Year for archival |
| `user:{email}` | `user:mike` | Document owner |

## Document Types in Paperless

Pre-create these document types:

1. **SOP** - Standard Operating Procedures
2. **Policy** - Company policies
3. **Compliance** - State/regulatory documents
4. **Batch Record** - Completed batch documentation
5. **Lab Result** - COA and test results
6. **Visitor Log** - NDA, access records
7. **Photos** - Facility/plant photos
8. **Invoice** - Purchase records

## Sync Workflow

### Manual Sync

1. User clicks "Archive to Paperless" on a document
2. Backend uploads file to Paperless API
3. Backend stores `paperlessId` and `paperlessUrl` in local DB
4. UI shows sync status indicator

### Automatic Sync (Optional)

1. Cron job runs nightly
2. Finds documents meeting criteria (e.g., approved SOPs, completed batches)
3. Uploads to Paperless if not already synced
4. Updates local records with Paperless IDs

### Retrieval

1. User searches in CA Grow Ops
2. Search includes both local and Paperless documents
3. For Paperless-only documents, proxy the download through our API

## UI Components

### Document Card (Extended)

```tsx
<DocumentCard>
    <DocumentTitle>{doc.title}</DocumentTitle>
    <DocumentMeta>
        <Badge>{doc.type}</Badge>
        {doc.paperlessId && (
            <Badge variant="success">
                <CloudArchive /> Archived
            </Badge>
        )}
    </DocumentMeta>
    <Actions>
        <Button onClick={viewDocument}>View</Button>
        {!doc.paperlessId && (
            <Button onClick={archiveToPaperless}>
                Archive to Paperless
            </Button>
        )}
    </Actions>
</DocumentCard>
```

### Archive Modal

```tsx
<ArchiveModal>
    <Title>Archive to Paperless</Title>
    <TagSelector 
        selected={selectedTags}
        onChange={setSelectedTags}
        suggestions={["batch:current", "type:sop"]}
    />
    <DocumentTypeSelector 
        value={documentType}
        onChange={setDocumentType}
    />
    <Button onClick={handleArchive}>Archive</Button>
</ArchiveModal>
```

## Configuration

### Environment Variables

```bash
# Paperless Connection
PAPERLESS_ENABLED=true
PAPERLESS_BASE_URL=https://paperless.runfoo.run
PAPERLESS_API_TOKEN=pk_xxxxxxxxxxxxx

# Sync Settings
PAPERLESS_AUTO_SYNC=false
PAPERLESS_SYNC_COMPLETED_BATCHES=true
PAPERLESS_SYNC_APPROVED_SOPS=true

# Defaults
PAPERLESS_DEFAULT_CORRESPONDENT=777wolfpack
```

### Settings Page Addition

Add Paperless configuration section to Settings:

- Connection status indicator
- Test connection button
- Enable/disable sync
- Configure auto-sync rules

## Error Handling

| Error | Handling |
|-------|----------|
| Paperless unavailable | Queue for retry, show warning |
| Upload failed | Store locally, mark for retry |
| Authentication error | Alert admin, disable sync |
| Document not found | Remove paperlessId from local record |

## Security Considerations

1. **API Token Security**: Store token in encrypted environment variable
2. **Access Control**: Only users with document permissions can trigger archival
3. **Audit Logging**: Log all Paperless operations to audit log
4. **Network Security**: Require HTTPS for Paperless connection

## Implementation Phases

### Phase 1: Basic Integration

- [ ] Paperless service with upload/download
- [ ] Manual sync button on documents
- [ ] Sync status indicators

### Phase 2: Search Integration

- [ ] Combined search across local + Paperless
- [ ] Tag management UI
- [ ] Preview/thumbnail support

### Phase 3: Automatic Sync

- [ ] Cron job for batch archival
- [ ] Auto-archive completed batches
- [ ] Auto-archive approved SOPs

### Phase 4: Advanced Features

- [ ] Bulk archive UI
- [ ] Archive reports/analytics
- [ ] Retention policies

## Testing

### Unit Tests

- Mock Paperless API responses
- Test upload/download flows
- Test error handling

### Integration Tests

- Set up test Paperless instance
- End-to-end document workflow
- Sync status verification

## Dependencies

```json
{
    "form-data": "^4.0.0",
    "node-fetch": "^3.3.0"
}
```

## References

- [Paperless-ngx Documentation](https://docs.paperless-ngx.com/)
- [Paperless API Reference](https://docs.paperless-ngx.com/api/)
- [Paperless Docker Setup](https://docs.paperless-ngx.com/setup/#docker)