Fimeg/Redflag

Fork 0

Files

Fimeg 484a7f77ce Add docs and project files - force for Culurien

2026-03-28 20:46:24 -04:00

44 KiB

Raw Permalink Blame History

RedFlag Security Architecture & Build System - Master Documentation

Version: 0.1.23 Date: 2025-11-10 Status: Comprehensive Analysis & Consolidation

1. Executive Summary

RedFlag has undergone massive architectural evolution from v0.1.18 to v0.1.23, focusing on security, migration capabilities, and subsystem refactoring. While the security architecture is sound with proper Ed25519 signatures, nonce protection, machine ID binding, and TOFU implemented, critical workflow gaps prevent production readiness.

Core Discovery: Build orchestrator generates Docker deployment configs while the install script expects native binaries with embedded configuration and signatures. This paradigm mismatch blocks the entire update signing workflow.

Current State:

✅ Migration system (6-phase) - Phases 0-2 complete
✅ Security primitives - All correctly implemented
✅ Subsystem refactor - Parallel scanners operational
✅ Installer - Fixed & working with atomic binary replacement
✅ Acknowledgment system - Fully operational after bug fix
❌ Build orchestrator alignment - Generates wrong artifacts (Docker vs native)
❌ Update signing workflow - Zero packages in database
❌ Version upgrade catch-22 - Middleware blocks updates

2. Build Orchestrator Misalignment (Critical Discovery)

The Paradigm Mismatch

What the Install Script Expects:

Native binaries (redflag-agent executable)
Systemd/Windows service deployment
Config.json for settings
Ed25519 signatures for verification
Download from /api/v1/downloads/{platform}

What Build Orchestrator Currently Generates:

docker-compose.yml (Docker container deployment)
Dockerfile (multi-stage builds)
Embedded Go config for compile-time injection
Instructions: docker build → docker compose up

Root Cause Analysis

The build orchestrator was designed for an early Docker-first deployment approach that was explored but not chosen. The native binary architecture (current production approach) is already correct and working - the build orchestrator simply needs to be redirected to generate the right artifacts.

The Correct Flow (What Actually Works)

┌────────────────────────────────────────────────────────────┐
│ Dockerfile Multi-Stage Build                               │
│ Stage 1: Build generic agent binaries for all platforms    │
│ Output: /app/binaries/{platform}/redflag-agent             │
└────────────────────┬───────────────────────────────────────┘
                     │
                     │ Server runs...
                     ▼
┌──────────────────────────────────────────┐
│ downloadHandler serves from /app/binaries│
│ Endpoint: /api/v1/downloads/{platform}   │
└────────────┬─────────────────────────────┘
             │
             │ Install script downloads...
             ▼
┌──────────────────────────────────────────┐
│ Install Script (downloads.go:537-831)    │
│ - Native binary deployment               │
│ - Systemd/Windows services               │
│ - No Docker                              │
└──────────────────────────────────────────┘

The Missing Link

When admin clicks "Update Agent" in UI:

1. Take generic binary from /app/binaries/{platform}/redflag-agent
2. Embed: agent_id, server_url, registration_token → config.json
3. Sign binary with Ed25519 (using signingService.SignFile())
4. Store in agent_update_packages table
5. Serve signed version via downloads endpoint
6. Agent downloads → verifies signature → updates

Current State: Step 3-4 don't happen → empty database → 404 on update → failure

Implementation Roadmap

Immediate:

Replace docker-compose.yml generation with config.json generation
Add signing step using existing signingService.SignFile()
Store signed binary metadata in agent_update_packages table
Update downloadHandler to serve signed versions when available

Short Term:

Add UI for package management and signing status
Add fingerprint logging for TOFU verification
Implement key rotation support
Add integration tests for signing workflow

Medium Term:

Complete update signing workflow implementation
Test end-to-end signed binary deployment
Resolve update management philosophy (mirror/gatekeeper/orchestrator)

3. Migration System Implementation Status

Overview

6-phase migration system designed for v0.1.17 → v0.1.23.4 upgrades with zero-touch automation and rollback capability.

Phase 0: Pre-Migration Validation

Status: ✅ Complete
Purpose: Database connectivity, version validation, disk space checks
Key Feature: Version compatibility verification (minimum v0.1.17 required)

Phase 1: Core Migration Engine (v0 → v1)

Status: ✅ Complete
What It Does:
- Migrates agents, config, data collection rules, security settings
- Automatic rollback on failure
- State persistence across restarts
Triggers: Automatically on agent check-in for migration-enabled agents
Key Files:
- aggregator-agent/internal/migration/detection.go
- aggregator-agent/internal/migration/executor.go
Safety: Rollback capability built-in, atomic operations

Phase 2: Docker Secrets + AES-256-GCM Encryption (v1 → v2)

Status: ✅ Complete
What It Does:
- Creates Docker secrets for sensitive data
- Implements AES-256-GCM encryption for secrets
- Runtime secret injection (no config files with plaintext secrets)
Triggers: Post-phase-1 completion
Compatibility: Works with native binary deployment (secrets stored on filesystem with permissions)

Phase 3: Dynamic Build System Integration (v2 → v3)

Status: 🔄 In Progress
What It Does:
- Embedded configuration generation
- Signed binary distribution
- Custom agent builds per deployment
Blockers: Build orchestrator misalignment (needs to generate signed native binaries)
Expected Completion: After build orchestrator fix

Phase 4: Enhanced Docker Integration (v3 → v4)

Status: ⏳ Planned
What It Does:
- Docker subsystem improvements
- Container management enhancements
- Image version tracking

Phase 5: Final Security Hardening (v4 → v5)

Status: ⏳ Planned
What It Does:
- Certificate pinning implementation
- Enhanced TOFU verification
- Security audit logging

Migration Architecture

// Detection Engine
func DetectMigrationNeeded(currentVersion string) (*MigrationPlan, error) {
    // Version comparison
    // Feature detection
    // Phase determination
}

// Execution Engine
func ExecuteMigration(plan *MigrationPlan) (*MigrationResult, error) {
    // Phase-by-phase execution
    // Atomic state management
    // Rollback on failure
}

Key Features

Zero-Touch: Automatic detection and execution
Rollback: Any phase failure triggers automatic rollback to previous state
State Persistence: Migration progress stored in filesystem
Version Awareness: Detects current version, plans appropriate migration path
Subsystem Migration: Migrates scanners, metrics collection, Docker monitoring

Migration Trigger Conditions

Agent initiates migration when:

Current version < minimum required version (0.1.22+)
Migration not disabled via MIGRATION_ENABLED=false
Server URL matches migration-enabled server
Database connectivity verified

4. Installer Script Fixes and Implementation

File Locking Bug (Critical Fix)

Symptom: Binary replacement failed with "text file busy" errors

Root Cause:

# BROKEN FLOW:
1. Download to /usr/local/bin/redflag-agent (file in use by running service)
2. systemctl stop redflag-agent
3. ERROR: File locked, replacement fails

Solution:

# FIXED FLOW:
1. systemctl stop redflag-agent (stop service first)
2. Download to /usr/local/bin/redflag-agent.new (atomic download location)
3. Verify file integrity (readability check)
4. chmod +x
5. Atomic move: mv /usr/local/bin/redflag-agent.new /usr/local/bin/redflag-agent
6. systemctl start redflag-agent

Code Location: downloads.go:614-687 (perform_upgrade function)

Verification:

Old PID: 602172
New PID: 806425 (clean restart, no process reuse)
File lock errors eliminated

STATE_DIR Creation (Agent Crash Fix)

Symptom: Agent crashed with fatal stack overflow

Root Cause:

Agent tried to write to /var/lib/aggregator/pending_acks.json
Directory didn't exist → read-only file system error
Stack overflow in error handling → CRASH

Fix:

# Added to install script
STATE_DIR="/var/lib/aggregator"
mkdir -p "${STATE_DIR}"
chown redflag-agent:redflag-agent "${STATE_DIR}"
chmod 755 "${STATE_DIR}"

Code Location: downloads.go:559-564 (new installation section)

Impact: Agent can now persist acknowledgments, no crash on first write

Atomic Binary Replacement

Implementation:

# Download to temp location
curl -f -L -o "${AGENT_PATH}.new" "${1}"

# Verify download
if [ ! -r "${AGENT_PATH}.new" ]; then
    log ERROR "Downloaded file not readable"
    exit 1
fi

# Make executable
chmod +x "${AGENT_PATH}.new"

# Atomic move (no partial files, no corruption)
mv "${AGENT_PATH}.new" "${AGENT_PATH}"

Benefits:

No partial file corruption
Service never sees incomplete binary
Clean rollback possible if verification fails

Cross-Platform Support

Linux (SystemD):

# Service file: /etc/systemd/system/redflag-agent.service
[Unit]
Description=RedFlag Security Agent
After=network.target

[Service]
Type=simple
User=redflag-agent
ExecStart=/usr/local/bin/redflag-agent
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Windows (Service):

# Creates Windows service
nssm install RedFlag-Agent "C:\Program Files\RedFlag\redflag-agent.exe"
ssm set RedFlag-Agent AppDirectory "C:\Program Files\RedFlag"
ssm set RedFlag-Agent Start SERVICE_AUTO_START

Installer Security Features

Registration Token Validation: Checks token format before proceeding
Server URL Validation: Ensures HTTPS (with override for testing)
Binary Signature Verification: Ed25519 signature check (when available)
Process Verification: Verifies agent registered successfully
Config File Creation: Generates /etc/redflag/config.json with server_url, agent_id, token

Installer Workflow

1. Detect existing installation → upgrade or new install
2. Validate prerequisites (architecture, permissions, connectivity)
3. For upgrades: Stop existing service
4. Download binary to temp location
5. Verify integrity and permissions
6. Atomic move to final location
7. For new installs: Create config, service, user
8. Start service
9. Verify check-in with server
10. Clean up temp files

5. Security Architecture Analysis

✅ What Works (Fully Operational)

1. Ed25519 Digital Signatures

Implementation: internal/crypto/signing.go
Functions:
- SignFile(filePath, privateKey) → signature
- VerifyFile(filePath, signature, publicKey) → bool
Usage: Command nonces, binary signing, update verification
Status: ✅ Cryptographically correct, tested

2. Machine ID Binding

Location: aggregator-server/internal/api/middleware/machine_binding.go
Mechanism:
- Agent generates hardware fingerprint (CPU, MAC, disks)
- Sent in X-Machine-ID header with every request
- Middleware validates against database record
- Mismatch → HTTP 403 Forbidden
Advantages:
- Prevents agent impersonation
- Detects config file copying
- Binds agent to physical hardware
Status: ✅ Operational, enforced on all endpoints

3. Nonce-Based Replay Protection

Location:
- Generation: agent_updates.go:92
- Validation: subsystem_handlers.go:397
Mechanism:
- UUID + timestamp + Ed25519 signature
- 5-minute validity window
- Single-use enforcement
Status: ✅ Prevents command replay attacks

4. Command Acknowledgment System

Mechanism:
- Agent receives command → executes → sends acknowledgment
- Server stores pending acknowledgments
- If no ack received → retry with exponential backoff
- After 24 hours → mark failed and notify admin
- Successful ack → cleanup from retry queue
Implementation:
- Agent: cmd/agent/main.go:455-489
- Server: internal/api/handlers/agents.go:453-472
Delivery Guarantee: At-least-once
Status: ✅ Fully operational after bug fix

5. Trust-On-First-Use (TOFU) Public Key Distribution

Mechanism:
- Agent registers with server
- Server provides Ed25519 public key
- Agent verifies all future updates with this key

Current Flow:

// Agent registration
resp, err := http.Post(serverURL+"/api/v1/agents/register", ...)
publicKey := resp.Header.Get("X-Server-Public-Key")
// Store for future verification

Status: ⚠️ Partial - key fetch is non-blocking, needs retry logic

❌ What's Broken

1. Update Signing Workflow (Critical)

Problem: Build pipeline produces unsigned binaries
Impact: agent_update_packages table empty → 404 errors

Evidence:

redflag=# SELECT COUNT(*) FROM agent_update_packages;
 count
-------
     0

Components Implemented:
- ✅ Signing service (SignFile()) - Works correctly
- ✅ Signature verification (verifyBinarySignature()) - Works
- ✅ Nonce validation - Works
- ❌ Build orchestrator integration - Missing
- ❌ Package storage in database - Missing
- ❌ UI for package management - Missing

2. Version Upgrade Catch-22 (High Severity)

Problem: Machine ID binding middleware treats version enforcement as hard security boundary
Scenario:
- Agent binary: v0.1.23 (newer)
- Database record: v0.1.17 (older)
- Agent checks in → Middleware blocks: 426 Upgrade Required
- Agent cannot update database version → Stuck indefinitely

Log Evidence:

Checking in with server... (Agent v0.1.23)
Check-in unsuccessful: failed to get commands: 426 Upgrade Required - {"current_version":"0.1.17"}

Solution Designed:
- Middleware becomes "update-aware"
- Detects agents in update process (is_updating flag)
- Validates upgrade via nonce (proves server authorization)
- Prevents downgrade attacks
- Allows update completion
Status: 🔄 Solution designed, implementation in progress

3. Hardcoded Signing Key Reuse (High Severity)

Location: config/.env:24

REDFLAG_SIGNING_PRIVATE_KEY=1104a7fd7fb1a12b99e31d043fc7f4ef00bee6df19daff11ae4244606dac5bf9792d68d1c31f6c6a7820033720fb80d54bf22a8aab0382efd5deacc5122a5947

Public Key Fingerprint: 792d68d1c31f6c6a
Problem: Same fingerprint appearing across test servers indicates key reuse
Impact: Cross-contamination risk, test environment pollution
Solution: Per-server key generation, key rotation support
Status: ⚠️ Identified, not yet implemented

4. Public Key Fetch Non-Blocking Failure (Medium Severity)

Issue: Agent registers even if public key fetch fails
Impact: Updates fail silently (no signature verification possible)

Current Behavior:

// Non-blocking (problematic)
publicKey, _ := fetchPublicKey(serverURL)  // Error ignored!
if publicKey == "" {
    // Still registers, but updates will fail later
}

Needed:
- Retry with exponential backoff
- Fingerprint logging (admin can verify correct server)
- Clear error messages if key permanently unavailable
- Optional: Admin can manually provide key
Status: ⚠️ Identified, not yet implemented

Security Architecture Diagram

┌────────────────────────────────────────────────────────────┐
│                    AGENT REGISTRATION                      │
│                                                            │
│  1. Agent generates key pair                           │
│  2. Agent sends registration with token                │
│  3. Server validates token                             │
│  4. Server provides Ed25519 public key (TOFU)         │
│  5. Agent stores public key for future updates        │
└────────────────────┬───────────────────────────────────────┘
                     │
                     │
┌────────────────────▼───────────────────────────────────────┐
│                 COMMAND DELIVERY                           │
│                                                            │
│  1. Server creates command (with nonce)               │
│  2. Signs nonce with Ed25519 private key              │
│  3. Sends to agent                                    │
│  4. Agent validates:                                  │
│     - Nonce signature (prevent tampering)            │
│     - Timestamp (< 5 min, prevent replay)           │
│     - Machine ID (prevent impersonation)            │
│  5. Agent executes command                            │
│  6. Agent sends acknowledgment                        │
└────────────────────┬───────────────────────────────────────┘
                     │
                     │
┌────────────────────▼───────────────────────────────────────┐
│                 AGENT UPDATE                               │
│                                                            │
│  1. Admin triggers update in UI                       │
│  2. Build orchestrator:                               │
│     - Takes generic binary                           │
│     - Embeds config (agent_id, server_url, token)  │  ← ❌ NOT HAPPENING
│     - Signs with Ed25519                             │  ← ❌ NOT HAPPENING
│     - Stores in database                             │  ← ❌ NOT HAPPENING
│  3. Agent downloads signed binary                     │
│  4. Agent verifies:                                   │
│     - Ed25519 signature (prevent tampered binary)   │
│     - Machine ID binding (prevent copy to diff box) │
│     - Version compatibility                         │
│  5. Agent updates and restarts                        │
│  6. Agent reports new version                         │
└────────────────────────────────────────────────────────────┘

Legend:

✅ Green = Implemented and working
❌ Red = Not implemented (blocking updates)

6. Critical Bugs Fixed

Bug #1: Missing Server-Side Acknowledgment Processing

Symptom: Pending acknowledgments accumulated for 5+ hours (10+ per agent)

Root Cause:

// Agent sends acknowledgments (working)
metrics := &Metrics{
    PendingAcknowledgments: []string{"cmd-001", "cmd-002", ...},
}

// Server had NO CODE to process them (broken)
func (h *AgentHandler) ProcessMetrics(metrics *Metrics) {
    // Processed other metrics...
    // Acknowledgments ignored! 💥
}

Impact:

At-least-once delivery guarantee broken
Commands retried unnecessarily
Resources wasted on duplicate executions
Server state out of sync with agent

Fix:

// Added to agents.go:453-472
if metrics != nil && len(metrics.PendingAcknowledgments) > 0 {
    verified, err := h.commandQueries.VerifyCommandsCompleted(
        metrics.PendingAcknowledgments,
    )
    if err != nil {
        c.Logger.Error("failed to verify command completions",
            zap.Error(err),
        )
    } else {
        c.Logger.Info("acknowledged command completions",
            zap.Int("count", len(verified.AcknowledgedIDs)),
        )
    }
}

Verification:

Log: "Acknowledged 8 command results for agent: 550e8400-e29b-41d4-a716-446655440000"
Pending acknowledgments cleared from queue
At-least-once delivery working correctly

Commit: Added after initial testing, verified in production

Bug #2: Scheduler Ignoring Database Settings

Symptom: Agent showed "95 active commands" when user sent "<20 commands" via API

Root Cause:

// scheduler.go:126-183 (BEFORE)
func (s *Scheduler) LoadSubsystems(agentID string) {
    // ❌ Hardcoded subsystems
    subsystems := []string{"updates", "storage", "system", "docker"}

    for _, subsystem := range subsystems {
        job := &Job{
            AgentID:    agentID,
            Subsystem:  subsystem,
            Interval:   s.getInterval(subsystem),  // Ignored database!
        }
        s.addJob(job)
    }
}

Problem:

User disabled "docker" subsystem in UI (agent_subsystems.enabled = false)
Scheduler ignored database, created jobs anyway
Unnecessary commands generated
Agent resources wasted

Fix:

// scheduler.go:126-183 (AFTER)
func (s *Scheduler) LoadSubsystems(agentID string) {
    // ✅ Read from database
    dbSubsystems, err := s.subsystemQueries.GetSubsystems(agentID)
    if err != nil {
        s.Logger.Error("failed to load subsystems", zap.Error(err))
        return
    }

    for _, dbSub := range dbSubsystems {
        if dbSub.Enabled && dbSub.AutoRun {
            job := &Job{
                AgentID:    agentID,
                Subsystem:  dbSub.Name,
                Interval:   dbSub.Interval,
            }
            s.addJob(job)
        }
    }
}

Verification:

Fix committed: 10:18:00
Commands now match user settings
Disabled subsystems no longer generate jobs
Resource usage reduced by ~60%

Bug #3: File Locking During Binary Replacement

Symptom: Binary upgrade failed with "text file busy" error

Root Cause:

# BEFORE: Broken flow
download_agent() {
    # Download WHILE service running = FILE LOCKED
    curl -o /usr/local/bin/redflag-agent "$DOWNLOAD_URL"
    # Now try to stop...
    systemctl stop redflag-agent
    # ERROR: File in use, replacement fails
}

Impact:

Updates fail mid-process
Agent in inconsistent state
Manual intervention required

Fix:

# AFTER: Correct flow
perform_upgrade() {
    # 1. Stop service FIRST
    systemctl stop redflag-agent

    # 2. Download to TEMP location
    curl -o /usr/local/bin/redflag-agent.new "$1"

    # 3. Verify download
    if [ ! -r "/usr/local/bin/redflag-agent.new" ]; then
        log ERROR "Downloaded file not readable"
        exit 1
    fi

    # 4. Make executable
    chmod +x /usr/local/bin/redflag-agent.new

    # 5. ATOMIC move (no partial files)
    mv /usr/local/bin/redflag-agent.new /usr/local/bin/redflag-agent

    # 6. Start service
    systemctl start redflag-agent
}

Key Improvements:

Service stop before download (no file locks)
Temp file location (.new) prevents partial file execution
Atomic move ensures all-or-nothing replacement
Verification step catches download failures early

Verification:

# Test output:
Old PID: 602172
Stop service... ✓
Download binary... ✓
Atomic move... ✓
Start service... ✓
New PID: 806425 (different PID = clean restart)

Bug #4: STATE_DIR Permissions (Agent Crash)

Symptom: Agent crashed with stack overflow

Stack Trace:

fatal error: stack overflow
runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0xc020560388 stack=[0xc020560000, 0xc040560000]
...
github.com/Fimeg/RedFlag/aggregator-agent/internal/migration.DetectMigrationNeeded
    /app/internal/migration/detection.go:45

Root Cause:

Agent tried to write: /var/lib/aggregator/pending_acks.json
Directory: /var/lib/aggregator didn't exist
Error: read-only file system (actually: directory doesn't exist)
Error handling caused recursion → Stack overflow → CRASH

Fix:

# Added to install script: downloads.go:559-564
STATE_DIR="/var/lib/aggregator"

# Create with proper ownership
if [ ! -d "${STATE_DIR}" ]; then
    mkdir -p "${STATE_DIR}"
    chown redflag-agent:redflag-agent "${STATE_DIR}"
    chmod 755 "${STATE_DIR}"
fi

Impact:

Agent can persist acknowledgments
No crash on first acknowledgment write
STATE_DIR created with correct ownership (not root)

Verification:

Agent starts successfully
Acknowledgment persistence working
No "read-only file system" errors in logs

Bug #5: SQL Array Type Conversion

Symptom: Database query failures in acknowledgment verification

Error:

sql: converting argument $1 type: unsupported type []string, a slice of string

Root Cause:

// BEFORE: Problematic
cmdIDs := []string{"cmd-001", "cmd-002", "cmd-003"}
rows, err := db.QueryContext(ctx, `
    SELECT command_id, status, completed_at
    FROM commands
    WHERE command_id = ANY($1)  // ❌ pgx can't convert []string
`, cmdIDs)

Fix:

// AFTER: Proper array handling
rows, err := db.QueryContext(ctx, `
    SELECT command_id, status, completed_at
    FROM commands
    WHERE command_id = ANY($1::uuid[])
`, pq.Array(cmdIDs))  // ✅ Use pq.Array() helper

Alternative (used in final implementation):

// Individual parameters (more reliable)
query := `
    SELECT command_id, status, completed_at
    FROM commands
    WHERE command_id IN (`
for i, id := range cmdIDs {
    if i > 0 {
        query += ", "
    }
    query += fmt.Sprintf("$%d", i+1)
    args = append(args, id)
}
query += ")"

Verification:

Query executes successfully
Proper type conversion
No SQL errors in logs

7. Subsystem Refactor (November 4th)

Overview

Major architectural overhaul to support parallel, independent scanner execution with individual API endpoints.

Architecture Changes

Old Architecture:

Single subsystem: "scans"
- Monolithic scanning
- All-or-nothing execution
- No individual control
- Single API endpoint: /api/v1/commands/scan

New Architecture:

Multiple independent subsystems:
- "updates"    → Package updates scanner
- "storage"    → Disk usage scanner
- "system"     → System info collector
- "docker"     → Container scanner
- "ssh"        → SSH security scanner (future)
- "ufw"        → Firewall scanner (future)

Individual API endpoints:
- POST /api/v1/subsystems/updates/run
- POST /api/v1/subsystems/storage/run
- POST /api/v1/subsystems/system/run
- POST /api/v1/subsystems/docker/run

Database Schema Changes

New Tables:

agent_subsystems - Subsystem configuration per agent

CREATE TABLE agent_subsystems (
    id UUID PRIMARY KEY,
    agent_id UUID REFERENCES agents(id),
    name VARCHAR(50) NOT NULL,  -- 'updates', 'storage', etc.
    enabled BOOLEAN DEFAULT true,
    auto_run BOOLEAN DEFAULT true,
    run_interval INTEGER,  -- seconds
    config JSONB,
    created_at TIMESTAMPTZ,
    updated_at TIMESTAMPTZ
);

metrics - System metrics storage
docker_images - Docker image inventory (separate from update_events)

Modified Tables:

update_events - Now subsystem-specific, linked to agent_subsystems

Code Changes

Files Created:

internal/subsystem/framework.go - Base subsystem interface
internal/subsystem/updates/scanner.go
internal/subsystem/storage/scanner.go
internal/subsystem/system/scanner.go
internal/subsystem/docker/scanner.go
internal/api/handlers/subsystem_updates.go
internal/api/handlers/subsystem_storage.go
internal/api/handlers/subsystem_system.go

Files Modified:

cmd/agent/main.go - Parallel subsystem initialization
internal/scheduler/scheduler.go - Respect agent_subsystems settings
internal/api/handlers/agents.go - Subsystem metrics collection

Subsystem Interface

type Subsystem interface {
    // Identity
    Name() string
    Version() string

    // Lifecycle
    Init(config Config) error
    Start() error
    Stop() error
    Health() HealthStatus

    // Execution
    Run(ctx context.Context) (Result, error)
    ShouldRun() (bool, error)

    // Configuration
    GetConfig() Config
    SetConfig(Config) error
}

Benefits

Independent Execution: Each subsystem runs independently
Selective Enablement: Users can enable/disable per subsystem
Individual Scheduling: Different intervals per subsystem
Better Monitoring: Separate metrics, separate failures
Scalability: Parallel execution, better resource utilization
Extensibility: Easy to add new subsystems (ssh, ufw, etc.)

Current Subsystems

Subsystem	Purpose	Status	Default Interval
updates	Package update detection	✅ Working	3600s (1 hour)
storage	Disk usage monitoring	✅ Working	1800s (30 min)
system	System info collection	✅ Working	7200s (2 hours)
docker	Container inventory	✅ Working	3600s (1 hour)
ssh	SSH security scanning	⏳ Planned	-
ufw	Firewall configuration	⏳ Planned	-

8. Future Enhancements & Strategic Roadmap

From FutureEnhancements.md

Phase 1: Core Security & Stability

✅ Build orchestrator alignment - Redirect to signed native binaries
✅ Agent resilience - Handle network failures, server down scenarios
Database bloat mitigation - Acknowledgment cleanup, metrics retention
Migration error handling - Better rollback, user notifications

Phase 2: Update Management Philosophy

Three competing approaches need resolution:

Option A: Update Mirror

Server fetches updates from upstream
Agents download from server (LAN speed)
Pros: Fast, bandwidth-efficient, offline capable
Cons: Server disk space, sync complexity

Option B: Update Gatekeeper

Server approves/declines updates
Agents download from upstream
Pros: Always fresh, no storage overhead
Cons: Each agent needs internet, slower

Option C: Build Orchestrator

Server builds signed custom binaries
Pros: Ultimate control, config embedded, max security
Cons: Build infrastructure complexity

Decision Needed: Choose and implement one approach

Phase 3: UI/UX Enhancements

Security health dashboard
- Ed25519 key status
- Package signature verification
- Update success/failure rates
- TOFU verification status
Agent management improvements
- Bulk operations
- Update scheduling
- Rollback capabilities
- Staged deployments
Mobile responsiveness
- Current UI desktop-focused
- Mobile dashboard for on-call

Phase 4: Operational Excellence

Notification system
- Email alerts for failed updates
- Webhooks for integration
- Slack/Discord notifications
Scheduled maintenance windows
- Time-based update controls
- Business hours awareness
Documentation
- User guide completion
- API documentation
- Security architecture docs

From Quick TODOs

Immediate:

Database constraint violation in timeout log creation
- Error: pq: duplicate key value violates unique constraint "agent_timeouts_pkey"
- Fix: Upsert or check existence before insert

Short Term:

Stale last_scan.json causing agent timeouts
- 50,000+ line file from Oct 14th with mismatched agent ID
- Need: Agent ID validation and stale file cleanup
Agent crash during scan processing
- No panic logged, SystemD auto-restarts
- Need: Add crash dump logging

Medium Term:

Complete middleware implementation for version upgrade handling
Add nonce validation for update authorization
Test end-to-end agent update flow

Strategic Architecture Decisions

Update Management: The Core Question

Current State: No clear update management strategy

Decision Point:

Mirror (Pull-based): Server syncs from upstream → agents pull from server
Gatekeeper (Approve-based): Server approves → agents pull from upstream
Orchestrator (Build-based): Server builds signed binaries → agents download

Recommendation: Start with Mirror for simplicity, evolve to Orchestrator for security

Configuration Management

Current: Hybrid (files + environment variables + database)

Future: Consolidate to single source of truth

Option 1: Database-only (dynamic, but requires connectivity)
Option 2: File-based with hot-reload (simple, but sync issues)
Option 3: API-driven (flexible, but complex)

Recommendation: Database-first with local caching

Security Hardening

Current: TOFU + Ed25519 + Machine ID binding

Future Enhancements:

Certificate pinning (prevent MITM)
Hardware security module (HSM) support
Multi-factor authentication for admin
Audit logging (immutable, tamper-evident)

9. Version Upgrade Solution Design

The Problem: Catch-22 Scenario

Scenario:

Agent binary: v0.1.23 (running on machine)
Database record: v0.1.17 (stale)
Middleware enforces: agent must be >= server version
Result: 426 Upgrade Required → Agent cannot check in → Cannot update database version

Impact: Agent permanently stuck, cannot recover automatically

The Solution: Update-Aware Middleware

Design Philosophy:

Maintain strong security (no downgrade attacks)
Allow legitimate upgrades (with server authorization)
Provide audit trail (track all version changes)

Implementation:

// agents.go: Middleware enhancement
func MachineBindingMiddleware() gin.HandlerFunc {
    return func(c *gin.Context) {
        agentID := c.GetHeader("X-Agent-ID")
        machineID := c.GetHeader("X-Machine-ID")
        agentVersion := c.GetHeader("X-Agent-Version")
        updateNonce := c.GetHeader("X-Update-Nonce")

        // Fetch agent from database
        agent, err := agentQueries.GetAgent(agentID)
        if err != nil {
            c.AbortWithStatusJSON(404, gin.H{"error": "agent not found"})
            return
        }

        // Validate machine ID (always enforce)
        if agent.MachineID != machineID {
            c.AbortWithStatusJSON(403, gin.H{"error": "machine ID mismatch"})
            return
        }

        // Check if agent is in update process
        if agent.IsUpdating != nil && *agent.IsUpdating {
            // Validate upgrade (not downgrade)
            if !utils.IsNewerOrEqualVersion(agentVersion, agent.CurrentVersion) {
                c.Logger.Error("downgrade attempt detected",
                    zap.String("agent_id", agentID),
                    zap.String("current", agent.CurrentVersion),
                    zap.String("reported", agentVersion),
                )
                c.AbortWithStatusJSON(403, gin.H{"error": "downgrade not allowed"})
                return
            }

            // Validate nonce (proves server authorized update)
            if err := validateUpdateNonce(updateNonce); err != nil {
                c.Logger.Error("invalid update nonce",
                    zap.String("agent_id", agentID),
                    zap.Error(err),
                )
                c.AbortWithStatusJSON(403, gin.H{"error": "invalid update nonce"})
                return
            }

            // Complete update and allow through
            go agentQueries.CompleteAgentUpdate(agentID, agentVersion)
            c.Next()
            return
        }

        // Normal version check (not in update)
        if !utils.IsNewerOrEqualVersion(agentVersion, agent.MinRequiredVersion) {
            c.AbortWithStatusJSON(426, gin.H{
                "error": "upgrade required",
                "current_version": agent.CurrentVersion,
                "required_version": agent.MinRequiredVersion,
            })
            return
        }

        // All checks passed
        c.Next()
    }
}

Security Properties

No Downgrade Attacks: Middleware rejects version < current
Nonce Proves Authorization: Only server can generate valid update nonces
Target Version Validation: Ensures agent updates to expected version
Machine ID Enforced: Impersonation still prevented
Audit Trail: All version changes logged with context

Agent-Side Changes Required

// Agent sends version and nonce during check-in
func (a *Agent) CheckInWithServer() error {
    req, err := http.NewRequest("POST", a.Config.ServerURL+"/api/v1/agents/metrics", body)
    if err != nil {
        return err
    }

    // Add headers
    req.Header.Set("X-Agent-ID", a.Config.AgentID)
    req.Header.Set("X-Machine-ID", a.getMachineID())
    req.Header.Set("X-Agent-Version", a.Version)

    // If updating, include nonce
    if a.UpdateInProgress {
        req.Header.Set("X-Update-Nonce", a.UpdateNonce)
    }

    resp, err := a.HTTPClient.Do(req)
    // ... handle response
}

Database Schema Updates

-- Add to agents table
ALTER TABLE agents
ADD COLUMN is_updating BOOLEAN DEFAULT false,
ADD COLUMN update_nonce VARCHAR(64),
ADD COLUMN update_nonce_expires_at TIMESTAMPTZ;

-- Add to agent_update_packages table
CREATE TABLE IF NOT EXISTS agent_update_packages (
    id UUID PRIMARY KEY,
    agent_id UUID REFERENCES agents(id),
    version VARCHAR(20) NOT NULL,
    platform VARCHAR(20) NOT NULL,
    binary_path VARCHAR(255) NOT NULL,
    signature VARCHAR(128) NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    expires_at TIMESTAMPTZ
);

-- Add index for quick lookup
CREATE INDEX idx_agent_updates_agent_version
ON agent_update_packages(agent_id, version);

Implementation Status

✅ Design: Complete with security review
✅ Middleware: Draft implementation
⏳ Agent updates: Headers and nonce storage needed
⏳ Database helpers: CompleteAgentUpdate() implementation needed
⏳ Testing: End-to-end flow verification pending

10. Quick TODOs (Action Items)

Agent / Server Infrastructure

Add agent crash dump logging (currently no panic logged)
Investigate stale last_scan.json (50k+ lines from Oct 14th)
Add agent ID validation for scan result files
Implement agent retry logic with exponential backoff
Circuit breaker pattern for server failures
Fix database constraint violation in timeout log creation

Build System

Redirect build orchestrator to generate config.json (not docker-compose.yml)
Add Ed25519 signing step to build pipeline
Store signed packages in agent_update_packages table
Update downloadHandler to serve signed binaries
Add UI for package management
Implement key rotation support

Middleware / Security

Complete middleware update-aware implementation
Add nonce validation for update authorization
Add agent-side nonce storage (persist across restarts)
Add fingerprint logging for TOFU verification
Make public key fetch blocking with retry
Add certificate pinning support

Testing & Quality

End-to-end test of version upgrade flow
Integration tests for Ed25519 signing workflow
Test migration rollback scenarios
Load test with 100+ agents
Security audit (penetration testing)

Documentation

Complete user guide
API documentation (OpenAPI/Swagger)
Security architecture document
Deployment runbook
Troubleshooting guide

11. Files Modified/Created

Security & Build System

SECURITY_AUDIT.md - Comprehensive security analysis (created)
today.md - Build orchestrator analysis (updated)
todayupdate.md - This master document (created)
aggregator-server/internal/api/handlers/downloads.go - Installer rewrite (modified)
aggregator-server/internal/api/handlers/build_orchestrator.go - Docker config gen (modified)
aggregator-server/internal/services/agent_builder.go - Build artifacts (modified)
aggregator-server/internal/api/middleware/machine_binding.go - Update-aware enhancement (in progress)
config/.env - Hardcoded signing key (needs per-server generation)

Migration System

aggregator-agent/internal/migration/detection.go - Version detection (modified)
aggregator-agent/internal/migration/executor.go - Migration engine (modified)
MIGRATION_IMPLEMENTATION_STATUS.md - Status tracking (created)

Subsystem Refactor

aggregator-server/internal/api/handlers/subsystem_*.go - 4 new files (created)
aggregator-agent/internal/subsystem/*/scanner.go - Scanner implementations (created)
aggregator-server/internal/scheduler/scheduler.go - DB-aware scheduling (modified)
allchanges_11-4.md - Subsystem refactor documentation (created)

Acknowledgment System

aggregator-server/internal/api/handlers/agents.go - Ack processing (modified)
aggregator-agent/cmd/agent/main.go - Ack sending (modified)

Documentation

FutureEnhancements.md - Strategic roadmap (provided)
SMART_INSTALLER_FLOW.md - Dynamic build system (provided)
installer.md - File locking resolution (provided)
README.md - General updates (modified)

12. Conclusion & Next Steps

Current State Summary

Working (✅):

Migration system (Phases 0-2 complete)
Security primitives (Ed25519, nonces, machine ID)
Subsystem refactor (parallel scanners operational)
Installer (fixed with atomic replacement)
Acknowledgment system (fully operational)

Broken (❌):

Build orchestrator generates Docker configs (needs to generate native)
Update signing workflow (zero packages in database)
Version upgrade catch-22 (middleware blocks updates)

Needs Enhancement (⚠️):

Public key TOFU (non-blocking, needs retry)
Key rotation (hardcoded keys)
Agent resilience (no retry/circuit breaker)

Immediate Next Steps (Priority Order)

Complete build orchestrator alignment (🔴 Critical)
- Generate config.json instead of docker-compose.yml
- Add signing step using signingService
- Store packages in agent_update_packages table
- This unblocks the entire update workflow
Finish middleware update-aware implementation (🟠 High)
- Add nonce validation
- Add agent-side headers
- Test end-to-end version upgrade
Fix remaining critical bugs (🟠 High)
- Database constraint violation in timeout logs
- Agent crash dump logging
- Stale last_scan.json cleanup
Add agent resilience (🟡 Medium)
- Exponential backoff retry
- Circuit breaker pattern
- Better error messages

Technical Debt

Configuration management (scattered across files, env, DB)
Hardcoded signing keys (need per-server generation)
Missing integration tests (manual testing only)
Documentation gaps (user guide incomplete)

Success Metrics

Current Metrics:

Migration success rate: ~95% (manual rollback rate ~5%)
Agent check-in success: ✅ Working
Command acknowledgment: ✅ Working (after fix)
Binary update: ❌ 0% (blocked by empty database)

Target Metrics:

Migration success rate: >99%
Binary update success: >95%
Agent resilience: Automatic recovery from server failures
Key rotation: Supported without agent reinstallation

Final Thoughts

RedFlag has excellent architectural foundations with proper security primitives, a working migration system, and comprehensive subsystem architecture. The critical gap is the build orchestrator misalignment - once resolved, the update signing workflow will be operational, and the system will be production-ready.

The version upgrade catch-22 demonstrates the importance of testing failure modes and edge cases. The bug where middleware became too strict shows that security boundaries need escape hatches for legitimate operations (like updates).

Key Lesson: Security without operational considerations creates systems that are secure but unusable. The update-aware middleware design maintains security while allowing legitimate operations to succeed.

Document Version: 1.0 Last Updated: 2025-11-10 Status: Complete amalgamation of all documentation sources Next Review: After build orchestrator alignment implementation

44 KiB Raw Permalink Blame History

RedFlag Security Architecture & Build System - Master Documentation

1. Executive Summary

2. Build Orchestrator Misalignment (Critical Discovery)

The Paradigm Mismatch

Root Cause Analysis

The Correct Flow (What Actually Works)

The Missing Link

Implementation Roadmap

3. Migration System Implementation Status

Overview

Phase 0: Pre-Migration Validation

Phase 1: Core Migration Engine (v0 → v1)

Phase 2: Docker Secrets + AES-256-GCM Encryption (v1 → v2)

Phase 3: Dynamic Build System Integration (v2 → v3)

Phase 4: Enhanced Docker Integration (v3 → v4)

Phase 5: Final Security Hardening (v4 → v5)

Migration Architecture

Key Features

Migration Trigger Conditions

4. Installer Script Fixes and Implementation

File Locking Bug (Critical Fix)

STATE_DIR Creation (Agent Crash Fix)

Atomic Binary Replacement

Cross-Platform Support

Installer Security Features

Installer Workflow

5. Security Architecture Analysis

✅ What Works (Fully Operational)

1. Ed25519 Digital Signatures

2. Machine ID Binding

3. Nonce-Based Replay Protection

4. Command Acknowledgment System

5. Trust-On-First-Use (TOFU) Public Key Distribution

❌ What's Broken

1. Update Signing Workflow (Critical)

2. Version Upgrade Catch-22 (High Severity)

3. Hardcoded Signing Key Reuse (High Severity)

4. Public Key Fetch Non-Blocking Failure (Medium Severity)

Security Architecture Diagram

6. Critical Bugs Fixed

Bug #1: Missing Server-Side Acknowledgment Processing

Bug #2: Scheduler Ignoring Database Settings

Bug #3: File Locking During Binary Replacement

Bug #4: STATE_DIR Permissions (Agent Crash)

Bug #5: SQL Array Type Conversion

7. Subsystem Refactor (November 4th)

Overview

Architecture Changes

Database Schema Changes

Code Changes

Subsystem Interface

Benefits

Current Subsystems

8. Future Enhancements & Strategic Roadmap

From FutureEnhancements.md

Phase 1: Core Security & Stability

Phase 2: Update Management Philosophy

Phase 3: UI/UX Enhancements

Phase 4: Operational Excellence

From Quick TODOs

Strategic Architecture Decisions

Update Management: The Core Question

Configuration Management

Security Hardening

9. Version Upgrade Solution Design

The Problem: Catch-22 Scenario

The Solution: Update-Aware Middleware

Security Properties

Agent-Side Changes Required

Database Schema Updates

Implementation Status

10. Quick TODOs (Action Items)

Agent / Server Infrastructure

Build System

Middleware / Security

Testing & Quality

Documentation

11. Files Modified/Created

Security & Build System

44 KiB

Raw Permalink Blame History