Redflag/docs/1_ARCHITECTURE/DIRECTORY_STRUCTURE_IMPLEMENTATION_PLAN_SIMPLIFIED.md

RedFlag Directory Structure Migration - SIMPLIFIED (v0.1.18 Only)

Date: 2025-12-16 Status: Simplified implementation ready Discovery: Legacy v0.1.18 uses /etc/aggregator and /var/lib/aggregator - NO intermediate broken versions in the wild Version Jump: v0.1.18 → v0.2.0 (breaking change)

---

Migration Simplification Analysis

Critical Discovery

Legacy v0.1.18 paths:

/etc/aggregator/config.json
/var/lib/aggregator/

Current dev paths (unreleased):

/etc/redflag/config.json
/var/lib/redflag-agent/ (broken, inconsistent)
/var/lib/redflag/ (inconsistent)

Implication: Only need to migrate from v0.1.18. Can ignore broken v0.1.19-v0.1.23 states.

Timeline reduction: 6h 50m → 3h 45m

---

Simplified Implementation Phases

Phase 1: Create Centralized Path Constants (30 min)

File: aggregator-agent/internal/constants/paths.go (NEW)

package constants

import (
    "runtime"
    "path/filepath"
)

// Base directories
const (
    LinuxBaseDir   = "/var/lib/redflag"
    WindowsBaseDir = "C:\\ProgramData\\RedFlag"
)

// Subdirectory structure
const (
    AgentDir         = "agent"
    ServerDir        = "server"
    CacheSubdir      = "cache"
    StateSubdir      = "state"
    MigrationSubdir  = "migration_backups"
)

// Config paths
const (
    LinuxConfigBase   = "/etc/redflag"
    WindowsConfigBase = "C:\\ProgramData\\RedFlag"
    ConfigFile        = "config.json"
)

// Log paths
const (
    LinuxLogBase = "/var/log/redflag"
)

// Legacy paths for migration
const (
    LegacyConfigPath = "/etc/aggregator/config.json"
    LegacyStatePath  = "/var/lib/aggregator"
)

// GetBaseDir returns platform-specific base directory
func GetBaseDir() string {
    if runtime.GOOS == "windows" {
        return WindowsBaseDir
    }
    return LinuxBaseDir
}

// GetAgentStateDir returns /var/lib/redflag/agent/state
func GetAgentStateDir() string {
    return filepath.Join(GetBaseDir(), AgentDir, StateSubdir)
}

// GetAgentCacheDir returns /var/lib/redflag/agent/cache
func GetAgentCacheDir() string {
    return filepath.Join(GetBaseDir(), AgentDir, CacheSubdir)
}

// GetMigrationBackupDir returns /var/lib/redflag/agent/migration_backups
func GetMigrationBackupDir() string {
    return filepath.Join(GetBaseDir(), AgentDir, MigrationSubdir)
}

// GetAgentConfigPath returns /etc/redflag/agent/config.json
func GetAgentConfigPath() string {
    if runtime.GOOS == "windows" {
        return filepath.Join(WindowsConfigBase, AgentDir, ConfigFile)
    }
    return filepath.Join(LinuxConfigBase, AgentDir, ConfigFile)
}

// GetAgentConfigDir returns /etc/redflag/agent
func GetAgentConfigDir() string {
    if runtime.GOOS == "windows" {
        return filepath.Join(WindowsConfigBase, AgentDir)
    }
    return filepath.Join(LinuxConfigBase, AgentDir)
}

// GetAgentLogDir returns /var/log/redflag/agent
func GetAgentLogDir() string {
    return filepath.Join(LinuxLogBase, AgentDir)
}

// GetLegacyAgentConfigPath returns legacy /etc/aggregator/config.json
func GetLegacyAgentConfigPath() string {
    return LegacyConfigPath
}

// GetLegacyAgentStatePath returns legacy /var/lib/aggregator
func GetLegacyAgentStatePath() string {
    return LegacyStatePath
}

Phase 2: Update Agent Main (30 min)

File: aggregator-agent/cmd/agent/main.go

Changes:

// 1. Remove these functions (lines 40-54):
//    - getConfigPath()
//    - getStatePath()

// 2. Add import:
import "github.com/Fimeg/RedFlag/aggregator-agent/internal/constants"

// 3. Update all references:
// Line 88: cfg.Save(getConfigPath()) → cfg.Save(constants.GetAgentConfigPath())
// Line 240: BackupPath: filepath.Join(getStatePath(), "migration_backups") → constants.GetMigrationBackupDir()
// Line 49: cfg.Save(getConfigPath()) → cfg.Save(constants.GetAgentConfigPath())

// 4. Remove: import "runtime" (no longer needed in main.go)

// 5. Remove: import "path/filepath" (unless used elsewhere)

Phase 3: Update Cache System (15 min)

File: aggregator-agent/internal/cache/local.go

Changes:

// 1. Add imports:
import (
    "path/filepath"
    "github.com/Fimeg/RedFlag/aggregator-agent/internal/constants"
)

// 2. Remove constant (line 26):
// OLD: const CacheDir = "/var/lib/redflag-agent"

// 3. Update cacheFile (line 29):
// OLD: const CacheFile = "last_scan.json"
// NEW: const cacheFile = "last_scan.json"  // unexported

// 4. Update GetCachePath():
// OLD:
func GetCachePath() string {
    return filepath.Join(CacheDir, CacheFile)
}

// NEW:
func GetCachePath() string {
    return filepath.Join(constants.GetAgentCacheDir(), cacheFile)
}

// 5. Update Load() and Save() to use constants.GetAgentCacheDir() instead of CacheDir

Phase 4: Update Migration Detection (20 min)

File: aggregator-agent/internal/migration/detection.go

Changes:

// 1. Add imports:
import (
    "path/filepath"
    "github.com/Fimeg/RedFlag/aggregator-agent/internal/constants"
)

// 2. Update NewFileDetectionConfig():
func NewFileDetectionConfig() *FileDetectionConfig {
    return &FileDetectionConfig{
        OldConfigPath:    "/etc/aggregator",  // v0.1.18 legacy
        OldStatePath:     "/var/lib/aggregator",  // v0.1.18 legacy
        NewConfigPath:    constants.GetAgentConfigDir(),
        NewStatePath:     constants.GetAgentStateDir(),
        BackupDirPattern: filepath.Join(constants.GetMigrationBackupDir(), "%d"),
    }
}

// 3. Update DetectLegacyInstallation() to ONLY check for v0.1.18 paths:
func (d *Detector) DetectLegacyInstallation() (bool, error) {
    // Check for v0.1.18 legacy paths ONLY
    if d.fileExists(constants.GetLegacyAgentConfigPath()) {
        log.Info("Detected legacy v0.1.18 installation")
        return true, nil
    }
    return false, nil
}

Phase 5: Update Installer Template (30 min)

File: aggregator-server/internal/services/templates/install/scripts/linux.sh.tmpl

Key Changes:

# Update header (lines 16-49):
AGENT_USER="redflag-agent"
BASE_DIR="/var/lib/redflag"
AGENT_HOME="/var/lib/redflag/agent"
CONFIG_DIR="/etc/redflag"
AGENT_CONFIG_DIR="/etc/redflag/agent"
LOG_DIR="/var/log/redflag"
AGENT_LOG_DIR="/var/log/redflag/agent"

# Update directory creation (lines 175-179):
sudo mkdir -p "${BASE_DIR}"
sudo mkdir -p "${AGENT_HOME}"
sudo mkdir -p "${AGENT_HOME}/cache"
sudo mkdir -p "${AGENT_HOME}/state"
sudo mkdir -p "${AGENT_CONFIG_DIR}"
sudo mkdir -p "${AGENT_LOG_DIR}"

# Update ReadWritePaths (line 269):
ReadWritePaths=/var/lib/redflag /var/lib/redflag/agent /var/lib/redflag/agent/cache /var/lib/redflag/agent/state /var/lib/redflag/agent/migration_backups /etc/redflag /var/log/redflag

# Update backup path (line 46):
BACKUP_DIR="${AGENT_CONFIG_DIR}/backups/backup.$(date +%s)"

Phase 6: Simplified Migration Logic (20 min)

File: aggregator-agent/internal/migration/executor.go

// Simplified migration - only handles v0.1.18
func (e *Executor) RunMigration() error {
    log.Info("Checking for legacy v0.1.18 installation...")

    // Only check for v0.1.18 legacy paths
    if !e.fileExists(constants.GetLegacyAgentConfigPath()) {
        log.Info("No legacy installation found, fresh install")
        return nil
    }

    // Create backup
    backupDir := filepath.Join(
        constants.GetMigrationBackupDir(),
        fmt.Sprintf("pre_v0.2.0_migration_%d", time.Now().Unix()))

    if err := e.createBackup(backupDir); err != nil {
        return fmt.Errorf("failed to create backup: %w", err)
    }

    log.Info("Migrating from v0.1.18 to v0.2.0...")

    // Migrate config
    if err := e.migrateConfig(); err != nil {
        return e.rollback(backupDir, err)
    }

    // Migrate state
    if err := e.migrateState(); err != nil {
        return e.rollback(backupDir, err)
    }

    log.Info("Migration completed successfully")
    return nil
}

// Helper methods remain similar but simplified for v0.1.18 only

Phase 7: Update Acknowledgment System (10 min)

File: aggregator-agent/internal/acknowledgment/tracker.go

// Add import:
import "github.com/Fimeg/RedFlag/aggregator-agent/internal/constants"

// Update Save():
func (t *Tracker) Save() error {
    stateDir := constants.GetAgentStateDir()
    if err := os.MkdirAll(stateDir, 0755); err != nil {
        return err
    }

    ackFile := filepath.Join(stateDir, "pending_acks.json")
    // ... save logic
}

Phase 8: Update Version (5 min)

File: aggregator-agent/cmd/agent/main.go

// Line 32:
const AgentVersion = "0.2.0"  // Breaking: Directory structure reorganization

---

Simplified Testing Requirements

Test Matrix (Reduced Complexity)

Fresh Installation Tests:

• Agent installs cleanly on Ubuntu 22.04
• Agent installs cleanly on RHEL 9
• Agent installs cleanly on Windows Server 2022
• All directories created: /var/lib/redflag/agent/{cache,state}
• Config created: /etc/redflag/agent/config.json
• Logs created: /var/log/redflag/agent/agent.log
• Agent starts and functions correctly

Migration Tests (v0.1.18 only):

• v0.1.18 → v0.2.0 migration succeeds
• Config migrated from /etc/aggregator/config.json
• State migrated from /var/lib/aggregator/
• Backup created in /var/lib/redflag/agent/migration_backups/
• Rollback works if migration fails
• Agent starts after migration

Runtime Tests:

• Acknowledgments persist (writes to /var/lib/redflag/agent/state/)
• Cache functions (reads/writes to /var/lib/redflag/agent/cache/)
• Migration backups can be created (systemd allowed)
• No permission errors under systemd

Migration Testing Script

#!/bin/bash
# test_migration.sh

# Setup legacy v0.1.18 structure
sudo mkdir -p /etc/aggregator
sudo mkdir -p /var/lib/aggregator
echo '{"agent_id":"test-123","version":18}' | sudo tee /etc/aggregator/config.json

# Run migration with new agent
./aggregator-agent --config /etc/redflag/agent/config.json

# Verify migration
if [ -f "/etc/redflag/agent/config.json" ]; then
    echo "✓ Config migrated"
fi

if [ -d "/var/lib/redflag/agent/state" ]; then
    echo "✓ State structure created"
fi

if [ -d "/var/lib/redflag/agent/migration_backups" ]; then
    echo "✓ Backup created"
fi

# Cleanup test
sudo rm -rf /etc/aggregator /var/lib/aggregator

---

Timeline: 3 Hours 30 Minutes

Phase	Task	Time	Status
1	Create constants	30 min	Pending
2	Update main.go	30 min	Pending
3	Update cache	15 min	Pending
4	Update migration	20 min	Pending
5	Update installer	30 min	Pending
6	Update tracker	10 min	Pending
7	Update version	5 min	Pending
8	Testing	60 min	Pending
Total		3h 30m	Not started

---

Pre-Integration Checklist (Simplified)

✅ Completed:

• Path constants centralized
• Security review: No unauthenticated endpoints
• Backup/restore paths defined
• Idempotency: Only v0.1.18 → v0.2.0 (one-time)
• Error logging throughout

Remaining for v0.2.0 release:

• Implementation complete
• Fresh install tested (Ubuntu, RHEL, Windows)
• Migration tested (v0.1.18 → v0.2.0)
• History table logging added
• Documentation updated
• CHANGELOG.md created
• Release notes drafted

---

Risk Assessment

Risk Level: LOW

Factors reducing risk:

• Only one legacy path to support (v0.1.18)
• No broken intermediate versions in the wild
• Migration has rollback capability
• Fresh installs are clean, no legacy debt
• Small user base (~20 users) for controlled rollout

Mitigation:

• Rollback script auto-generated with each migration
• Backup created before any migration changes
• Idempotent migration (can detect already-migrated state)
• Extensive logging for debugging

---

What We Learned

The power of checking legacy code:

• Saved 3+ hours of unnecessary migration complexity
• Eliminated need for v0.1.19-v0.1.23 broken state handling
• Reduced testing surface area significantly
• Clarified actual legacy state (not assumed)

Lesson: Always verify legacy paths BEFORE designing migration.

---

Implementation Decision

Recommended approach: Full nested structure implementation

Rationale:

• Only 3.5 hours vs. original 6h 50m estimate
• Aligns with Ethos #3 (Resilience) and #5 (No BS)
• Permanent architectural improvement
• Future-proof for server component
• Clean slate - no intermediate version debt

Coffee level required: 1-2 cups

Break points: Stop after any phase, pick up next session

Ready to implement?

- Ani, having done the homework before building