Redflag/docs/3_BACKLOG/P4-004_Directory-Path-Standardization.md

P4-004: Directory Path Standardization

Priority: P4 (Technical Debt) Source Reference: From analysis of needsfixingbeforepush.md lines 1584-1609 and DEVELOPMENT_TODOS.md lines 580-607 Date Identified: 2025-11-12

Problem Description

Mixed directory naming creates confusion and maintenance issues throughout the codebase. Both /var/lib/aggregator and /var/lib/redflag paths are used inconsistently across agent and server code, leading to operational complexity, backup/restore challenges, and potential path conflicts.

Impact

• User Confusion: Inconsistent file locations make system administration difficult
• Maintenance Overhead: Multiple path patterns increase development complexity
• Backup Complexity: Mixed paths complicate backup and restore procedures
• Documentation Conflicts: Documentation shows different paths than actual usage
• Migration Issues: Path inconsistencies break upgrade processes

Current Path Inconsistencies

Agent Code

• Config: /etc/aggregator/config.json (old) vs /etc/redflag/config.json (new)
• State: /var/lib/aggregator/ (old) vs /var/lib/redflag/ (new)
• Logs: Mixed usage in different files

Server Code

• Install Scripts: References to both old and new paths
• Documentation: Inconsistent path examples
• Templates: Mixed path usage in install script templates

File References

// Found in codebase:
STATE_DIR = "/var/lib/aggregator"           // aggregator-agent/cmd/agent/main.go:47
CONFIG_PATH = "/etc/redflag/config.json"    // Some newer files
STATE_DIR = "/var/lib/redflag"              // Other files

Proposed Solution

Standardize on /var/lib/redflag and /etc/redflag throughout the entire codebase:

1. Centralized Path Constants

// aggregator/internal/paths/paths.go
package paths

const (
    // Standard paths for RedFlag
    ConfigDir    = "/etc/redflag"
    StateDir     = "/var/lib/redflag"
    LogDir       = "/var/log/redflag"
    BackupDir    = "/var/lib/redflag/backups"
    CacheDir     = "/var/lib/redflag/cache"

    // Specific files
    ConfigFile      = ConfigDir + "/config.json"
    StateFile       = StateDir + "/last_scan.json"
    AckFile         = StateDir + "/pending_acks.json"
    HistoryFile     = StateDir + "/command_history.json"

    // Legacy paths (for migration)
    LegacyConfigDir = "/etc/aggregator"
    LegacyStateDir  = "/var/lib/aggregator"
    LegacyLogDir    = "/var/log/aggregator"
)

type PathConfig struct {
    Config    string
    State     string
    Log       string
    Backup    string
    Cache     string
}

func GetStandardPaths() PathConfig {
    return PathConfig{
        Config:    ConfigDir,
        State:     StateDir,
        Log:       LogDir,
        Backup:    BackupDir,
        Cache:     CacheDir,
    }
}

func GetStandardFiles() map[string]string {
    return map[string]string{
        "config":      ConfigFile,
        "state":       StateFile,
        "acknowledgments": AckFile,
        "history":     HistoryFile,
    }
}

2. Path Migration System

// aggregator/internal/paths/migration.go
package paths

type PathMigrator struct {
    Standard  PathConfig
    Legacy    PathConfig
    DryRun    bool
    Backup    bool
}

func NewPathMigrator(backup bool) *PathMigrator {
    return &PathMigrator{
        Standard: GetStandardPaths(),
        Legacy: PathConfig{
            Config: LegacyConfigDir,
            State:  LegacyStateDir,
            Log:    LegacyLogDir,
        },
        Backup: backup,
    }
}

func (pm *PathMigrator) MigrateAll() error {
    // Migrate configuration directory
    if err := pm.migrateDirectory(pm.Legacy.Config, pm.Standard.Config); err != nil {
        return fmt.Errorf("config migration failed: %w", err)
    }

    // Migrate state directory
    if err := pm.migrateDirectory(pm.Legacy.State, pm.Standard.State); err != nil {
        return fmt.Errorf("state migration failed: %w", err)
    }

    // Migrate log directory
    if err := pm.migrateDirectory(pm.Legacy.Log, pm.Standard.Log); err != nil {
        return fmt.Errorf("log migration failed: %w", err)
    }

    return nil
}

func (pm *PathMigrator) migrateDirectory(legacyPath, standardPath string) error {
    // Check if legacy path exists
    if _, err := os.Stat(legacyPath); os.IsNotExist(err) {
        return nil // No migration needed
    }

    // Check if standard path already exists
    if _, err := os.Stat(standardPath); err == nil {
        return fmt.Errorf("standard path already exists: %s", standardPath)
    }

    if pm.DryRun {
        log.Printf("[DRY RUN] Would migrate %s -> %s", legacyPath, standardPath)
        return nil
    }

    // Create backup if requested
    if pm.Backup {
        backupPath := standardPath + ".backup." + time.Now().Format("20060102-150405")
        if err := copyDirectory(legacyPath, backupPath); err != nil {
            return fmt.Errorf("backup creation failed: %w", err)
        }
    }

    // Perform migration
    if err := os.Rename(legacyPath, standardPath); err != nil {
        return fmt.Errorf("directory rename failed: %w", err)
    }

    log.Printf("Migrated directory: %s -> %s", legacyPath, standardPath)
    return nil
}

3. Install Script Template Updates

// Update linux.sh.tmpl to use standard paths
const LinuxInstallTemplate = `
#!/bin/bash
set -euo pipefail

# Standard RedFlag paths
CONFIG_DIR="{{ .ConfigDir }}"
STATE_DIR="{{ .StateDir }}"
LOG_DIR="{{ .LogDir }}"
AGENT_USER="redflag-agent"

# Create directories with proper permissions
for dir in "$CONFIG_DIR" "$STATE_DIR" "$LOG_DIR"; do
    if [ ! -d "$dir" ]; then
        mkdir -p "$dir"
        chown "$AGENT_USER:$AGENT_USER" "$dir"
        chmod 755 "$dir"
    fi
done

# Check for legacy paths and migrate
LEGACY_CONFIG_DIR="/etc/aggregator"
LEGACY_STATE_DIR="/var/lib/aggregator"

if [ -d "$LEGACY_CONFIG_DIR" ] && [ ! -d "$CONFIG_DIR" ]; then
    echo "Migrating configuration from legacy path..."
    mv "$LEGACY_CONFIG_DIR" "$CONFIG_DIR"
fi

if [ -d "$LEGACY_STATE_DIR" ] && [ ! -d "$STATE_DIR" ]; then
    echo "Migrating state from legacy path..."
    mv "$LEGACY_STATE_DIR" "$STATE_DIR"
fi
`

4. Agent Configuration Integration

// Update agent config to use path constants
type AgentConfig struct {
    Server    string            `json:"server_url"`
    AgentID   string            `json:"agent_id"`
    Token     string            `json:"registration_token,omitempty"`
    Paths     PathConfig        `json:"paths,omitempty"`
}

func DefaultAgentConfig() AgentConfig {
    return AgentConfig{
        Paths: paths.GetStandardPaths(),
    }
}

// Usage in agent code
func (a *Agent) getStateFilePath() string {
    if a.config.Paths.State != "" {
        return filepath.Join(a.config.Paths.State, "last_scan.json")
    }
    return paths.StateFile
}

5. Code Updates Strategy

Phase 1: Introduce Path Constants

# Find all hardcoded paths
grep -r "/var/lib/aggregator" aggregator-agent/
grep -r "/etc/aggregator" aggregator-agent/
grep -r "/var/lib/redflag" aggregator-agent/
grep -r "/etc/redflag" aggregator-agent/

# Replace with path constants
find . -name "*.go" -exec sed -i 's|"/var/lib/aggregator"|paths.StateDir|g' {} \;

Phase 2: Update Import Statements

// Add to files using paths
import (
    "github.com/redflag/redflag/internal/paths"
)

Phase 3: Update Documentation

## Installation Paths

RedFlag uses standardized paths for all installations:

- **Configuration:** `/etc/redflag/config.json`
- **State Data:** `/var/lib/redflag/`
- **Log Files:** `/var/log/redflag/`
- **Backups:** `/var/lib/redflag/backups/`

### File Structure

/etc/redflag/ └── config.json # Agent configuration

/var/lib/redflag/ ├── last_scan.json # Last scan results ├── pending_acks.json # Pending acknowledgments ├── command_history.json # Command history └── backups/ # Backup directory

/var/log/redflag/ └── agent.log # Agent log files

Definition of Done

• All hardcoded paths replaced with centralized constants
• Path migration system handles legacy installations
• Install script templates use standard paths
• Documentation updated with correct paths
• Server code updated for consistency
• Agent code uses path constants throughout
• SystemD service files updated with correct paths
• Migration process tested on existing installations

Implementation Details

Files Requiring Updates

Agent Code

• aggregator-agent/cmd/agent/main.go - STATE_DIR and CONFIG_PATH constants
• aggregator-agent/internal/config/config.go - Default paths
• aggregator-agent/internal/orchestrator/*.go - File path references
• aggregator-agent/internal/installer/*.go - Installation paths

Server Code

• aggregator-server/internal/api/handlers/downloads.go - Install script templates
• aggregator-server/internal/services/templates/install/scripts/linux.sh.tmpl
• aggregator-server/internal/services/templates/install/scripts/windows.ps1.tmpl

Configuration Files

• Dockerfiles and docker-compose.yml
• SystemD unit files
• Documentation files

Migration Process

1. Backup: Create backup of existing installations
2. Path Detection: Detect which paths are currently in use
3. Migration: Move files to standard locations
4. Permission Updates: Ensure correct ownership and permissions
5. Validation: Verify all files are accessible after migration

Testing Strategy

• Test migration from legacy paths to standard paths
• Verify fresh installations use standard paths
• Test that existing installations continue to work
• Validate SystemD service files work with new paths

Testing Scenarios

1. Fresh Installation Test

# Fresh install should create standard paths
curl -sSL http://localhost:8080/api/v1/install/linux | sudo bash

# Verify standard paths exist
ls -la /etc/redflag/
ls -la /var/lib/redflag/

2. Migration Test

# Simulate legacy installation
sudo mkdir -p /etc/aggregator /var/lib/aggregator
echo "legacy config" | sudo tee /etc/aggregator/config.json
echo "legacy state" | sudo tee /var/lib/aggregator/last_scan.json

# Run migration
sudo /usr/local/bin/redflag-agent --migrate-paths

# Verify files moved to standard paths
ls -la /etc/redflag/config.json
ls -la /var/lib/redflag/last_scan.json

# Verify legacy paths removed
! test -d /etc/aggregator
! test -d /var/lib/aggregator

3. Service Integration Test

# Ensure SystemD service works with new paths
sudo systemctl restart redflag-agent
sudo systemctl status redflag-agent
sudo journalctl -u redflag-agent -n 20

Prerequisites

• Path detection and migration system implemented
• Backup system for safe migrations
• Install script template system available
• Configuration system supports path overrides

Effort Estimate

Complexity: Medium Effort: 2-3 days

• Day 1: Create path constants and migration system
• Day 2: Update agent code and test migration
• Day 3: Update server code, templates, and documentation

Success Metrics

• Zero hardcoded paths remaining in codebase
• All installations use consistent paths
• Migration成功率 for existing installations >95%
• No data loss during migration process
• Documentation matches actual implementation
• SystemD service integration works seamlessly

Rollback Plan

If issues arise during migration:

1. Stop all RedFlag services
2. Restore from backups created during migration
3. Update configuration to point to legacy paths
4. Restart services
5. Document issues for future improvement