Redflag/docs/3_BACKLOG/P4-004_Directory-Path-Standardization.md

# P4-004: Directory Path Standardization

**Priority:** P4 (Technical Debt)
**Source Reference:** From analysis of needsfixingbeforepush.md lines 1584-1609 and DEVELOPMENT_TODOS.md lines 580-607
**Date Identified:** 2025-11-12

## Problem Description

Mixed directory naming creates confusion and maintenance issues throughout the codebase. Both `/var/lib/aggregator` and `/var/lib/redflag` paths are used inconsistently across agent and server code, leading to operational complexity, backup/restore challenges, and potential path conflicts.

## Impact

- **User Confusion:** Inconsistent file locations make system administration difficult
- **Maintenance Overhead:** Multiple path patterns increase development complexity
- **Backup Complexity:** Mixed paths complicate backup and restore procedures
- **Documentation Conflicts:** Documentation shows different paths than actual usage
- **Migration Issues:** Path inconsistencies break upgrade processes

## Current Path Inconsistencies

### Agent Code
- **Config:** `/etc/aggregator/config.json` (old) vs `/etc/redflag/config.json` (new)
- **State:** `/var/lib/aggregator/` (old) vs `/var/lib/redflag/` (new)
- **Logs:** Mixed usage in different files

### Server Code
- **Install Scripts:** References to both old and new paths
- **Documentation:** Inconsistent path examples
- **Templates:** Mixed path usage in install script templates

### File References
```go
// Found in codebase:
STATE_DIR = "/var/lib/aggregator"           // aggregator-agent/cmd/agent/main.go:47
CONFIG_PATH = "/etc/redflag/config.json"    // Some newer files
STATE_DIR = "/var/lib/redflag"              // Other files
```

## Proposed Solution

Standardize on `/var/lib/redflag` and `/etc/redflag` throughout the entire codebase:

### 1. Centralized Path Constants
```go
// aggregator/internal/paths/paths.go
package paths

const (
    // Standard paths for RedFlag
    ConfigDir    = "/etc/redflag"
    StateDir     = "/var/lib/redflag"
    LogDir       = "/var/log/redflag"
    BackupDir    = "/var/lib/redflag/backups"
    CacheDir     = "/var/lib/redflag/cache"

    // Specific files
    ConfigFile      = ConfigDir + "/config.json"
    StateFile       = StateDir + "/last_scan.json"
    AckFile         = StateDir + "/pending_acks.json"
    HistoryFile     = StateDir + "/command_history.json"

    // Legacy paths (for migration)
    LegacyConfigDir = "/etc/aggregator"
    LegacyStateDir  = "/var/lib/aggregator"
    LegacyLogDir    = "/var/log/aggregator"
)

type PathConfig struct {
    Config    string
    State     string
    Log       string
    Backup    string
    Cache     string
}

func GetStandardPaths() PathConfig {
    return PathConfig{
        Config:    ConfigDir,
        State:     StateDir,
        Log:       LogDir,
        Backup:    BackupDir,
        Cache:     CacheDir,
    }
}

func GetStandardFiles() map[string]string {
    return map[string]string{
        "config":      ConfigFile,
        "state":       StateFile,
        "acknowledgments": AckFile,
        "history":     HistoryFile,
    }
}
```

### 2. Path Migration System
```go
// aggregator/internal/paths/migration.go
package paths

type PathMigrator struct {
    Standard  PathConfig
    Legacy    PathConfig
    DryRun    bool
    Backup    bool
}

func NewPathMigrator(backup bool) *PathMigrator {
    return &PathMigrator{
        Standard: GetStandardPaths(),
        Legacy: PathConfig{
            Config: LegacyConfigDir,
            State:  LegacyStateDir,
            Log:    LegacyLogDir,
        },
        Backup: backup,
    }
}

func (pm *PathMigrator) MigrateAll() error {
    // Migrate configuration directory
    if err := pm.migrateDirectory(pm.Legacy.Config, pm.Standard.Config); err != nil {
        return fmt.Errorf("config migration failed: %w", err)
    }

    // Migrate state directory
    if err := pm.migrateDirectory(pm.Legacy.State, pm.Standard.State); err != nil {
        return fmt.Errorf("state migration failed: %w", err)
    }

    // Migrate log directory
    if err := pm.migrateDirectory(pm.Legacy.Log, pm.Standard.Log); err != nil {
        return fmt.Errorf("log migration failed: %w", err)
    }

    return nil
}

func (pm *PathMigrator) migrateDirectory(legacyPath, standardPath string) error {
    // Check if legacy path exists
    if _, err := os.Stat(legacyPath); os.IsNotExist(err) {
        return nil // No migration needed
    }

    // Check if standard path already exists
    if _, err := os.Stat(standardPath); err == nil {
        return fmt.Errorf("standard path already exists: %s", standardPath)
    }

    if pm.DryRun {
        log.Printf("[DRY RUN] Would migrate %s -> %s", legacyPath, standardPath)
        return nil
    }

    // Create backup if requested
    if pm.Backup {
        backupPath := standardPath + ".backup." + time.Now().Format("20060102-150405")
        if err := copyDirectory(legacyPath, backupPath); err != nil {
            return fmt.Errorf("backup creation failed: %w", err)
        }
    }

    // Perform migration
    if err := os.Rename(legacyPath, standardPath); err != nil {
        return fmt.Errorf("directory rename failed: %w", err)
    }

    log.Printf("Migrated directory: %s -> %s", legacyPath, standardPath)
    return nil
}
```

### 3. Install Script Template Updates
```go
// Update linux.sh.tmpl to use standard paths
const LinuxInstallTemplate = `
#!/bin/bash
set -euo pipefail

# Standard RedFlag paths
CONFIG_DIR="{{ .ConfigDir }}"
STATE_DIR="{{ .StateDir }}"
LOG_DIR="{{ .LogDir }}"
AGENT_USER="redflag-agent"

# Create directories with proper permissions
for dir in "$CONFIG_DIR" "$STATE_DIR" "$LOG_DIR"; do
    if [ ! -d "$dir" ]; then
        mkdir -p "$dir"
        chown "$AGENT_USER:$AGENT_USER" "$dir"
        chmod 755 "$dir"
    fi
done

# Check for legacy paths and migrate
LEGACY_CONFIG_DIR="/etc/aggregator"
LEGACY_STATE_DIR="/var/lib/aggregator"

if [ -d "$LEGACY_CONFIG_DIR" ] && [ ! -d "$CONFIG_DIR" ]; then
    echo "Migrating configuration from legacy path..."
    mv "$LEGACY_CONFIG_DIR" "$CONFIG_DIR"
fi

if [ -d "$LEGACY_STATE_DIR" ] && [ ! -d "$STATE_DIR" ]; then
    echo "Migrating state from legacy path..."
    mv "$LEGACY_STATE_DIR" "$STATE_DIR"
fi
`
```

### 4. Agent Configuration Integration
```go
// Update agent config to use path constants
type AgentConfig struct {
    Server    string            `json:"server_url"`
    AgentID   string            `json:"agent_id"`
    Token     string            `json:"registration_token,omitempty"`
    Paths     PathConfig        `json:"paths,omitempty"`
}

func DefaultAgentConfig() AgentConfig {
    return AgentConfig{
        Paths: paths.GetStandardPaths(),
    }
}

// Usage in agent code
func (a *Agent) getStateFilePath() string {
    if a.config.Paths.State != "" {
        return filepath.Join(a.config.Paths.State, "last_scan.json")
    }
    return paths.StateFile
}
```

### 5. Code Updates Strategy

#### Phase 1: Introduce Path Constants
```bash
# Find all hardcoded paths
grep -r "/var/lib/aggregator" aggregator-agent/
grep -r "/etc/aggregator" aggregator-agent/
grep -r "/var/lib/redflag" aggregator-agent/
grep -r "/etc/redflag" aggregator-agent/

# Replace with path constants
find . -name "*.go" -exec sed -i 's|"/var/lib/aggregator"|paths.StateDir|g' {} \;
```

#### Phase 2: Update Import Statements
```go
// Add to files using paths
import (
    "github.com/redflag/redflag/internal/paths"
)
```

#### Phase 3: Update Documentation
```markdown
## Installation Paths

RedFlag uses standardized paths for all installations:

- **Configuration:** `/etc/redflag/config.json`
- **State Data:** `/var/lib/redflag/`
- **Log Files:** `/var/log/redflag/`
- **Backups:** `/var/lib/redflag/backups/`

### File Structure
```
/etc/redflag/
└── config.json           # Agent configuration

/var/lib/redflag/
├── last_scan.json        # Last scan results
├── pending_acks.json     # Pending acknowledgments
├── command_history.json  # Command history
└── backups/              # Backup directory

/var/log/redflag/
└── agent.log            # Agent log files
```
```

## Definition of Done

- [ ] All hardcoded paths replaced with centralized constants
- [ ] Path migration system handles legacy installations
- [ ] Install script templates use standard paths
- [ ] Documentation updated with correct paths
- [ ] Server code updated for consistency
- [ ] Agent code uses path constants throughout
- [ ] SystemD service files updated with correct paths
- [ ] Migration process tested on existing installations

## Implementation Details

### Files Requiring Updates

#### Agent Code
- `aggregator-agent/cmd/agent/main.go` - STATE_DIR and CONFIG_PATH constants
- `aggregator-agent/internal/config/config.go` - Default paths
- `aggregator-agent/internal/orchestrator/*.go` - File path references
- `aggregator-agent/internal/installer/*.go` - Installation paths

#### Server Code
- `aggregator-server/internal/api/handlers/downloads.go` - Install script templates
- `aggregator-server/internal/services/templates/install/scripts/linux.sh.tmpl`
- `aggregator-server/internal/services/templates/install/scripts/windows.ps1.tmpl`

#### Configuration Files
- Dockerfiles and docker-compose.yml
- SystemD unit files
- Documentation files

### Migration Process
1. **Backup:** Create backup of existing installations
2. **Path Detection:** Detect which paths are currently in use
3. **Migration:** Move files to standard locations
4. **Permission Updates:** Ensure correct ownership and permissions
5. **Validation:** Verify all files are accessible after migration

### Testing Strategy
- Test migration from legacy paths to standard paths
- Verify fresh installations use standard paths
- Test that existing installations continue to work
- Validate SystemD service files work with new paths

## Testing Scenarios

### 1. Fresh Installation Test
```bash
# Fresh install should create standard paths
curl -sSL http://localhost:8080/api/v1/install/linux | sudo bash

# Verify standard paths exist
ls -la /etc/redflag/
ls -la /var/lib/redflag/
```

### 2. Migration Test
```bash
# Simulate legacy installation
sudo mkdir -p /etc/aggregator /var/lib/aggregator
echo "legacy config" | sudo tee /etc/aggregator/config.json
echo "legacy state" | sudo tee /var/lib/aggregator/last_scan.json

# Run migration
sudo /usr/local/bin/redflag-agent --migrate-paths

# Verify files moved to standard paths
ls -la /etc/redflag/config.json
ls -la /var/lib/redflag/last_scan.json

# Verify legacy paths removed
! test -d /etc/aggregator
! test -d /var/lib/aggregator
```

### 3. Service Integration Test
```bash
# Ensure SystemD service works with new paths
sudo systemctl restart redflag-agent
sudo systemctl status redflag-agent
sudo journalctl -u redflag-agent -n 20
```

## Prerequisites

- Path detection and migration system implemented
- Backup system for safe migrations
- Install script template system available
- Configuration system supports path overrides

## Effort Estimate

**Complexity:** Medium
**Effort:** 2-3 days
- Day 1: Create path constants and migration system
- Day 2: Update agent code and test migration
- Day 3: Update server code, templates, and documentation

## Success Metrics

- Zero hardcoded paths remaining in codebase
- All installations use consistent paths
- Migration成功率 for existing installations >95%
- No data loss during migration process
- Documentation matches actual implementation
- SystemD service integration works seamlessly

## Rollback Plan

If issues arise during migration:
1. Stop all RedFlag services
2. Restore from backups created during migration
3. Update configuration to point to legacy paths
4. Restart services
5. Document issues for future improvement