Add docs and project files - force for Culurien
This commit is contained in:
Fimeg
@@ -0,0 +1,473 @@
|
||||
# RedFlag Directory Structure Migration - SIMPLIFIED (v0.1.18 Only)
|
||||
|
||||
**Date**: 2025-12-16
|
||||
**Status**: Simplified implementation ready
|
||||
**Discovery**: Legacy v0.1.18 uses `/etc/aggregator` and `/var/lib/aggregator` - NO intermediate broken versions in the wild
|
||||
**Version Jump**: v0.1.18 → v0.2.0 (breaking change)
|
||||
|
||||
---
|
||||
|
||||
## Migration Simplification Analysis
|
||||
|
||||
### **Critical Discovery**
|
||||
|
||||
**Legacy v0.1.18 paths:**
|
||||
```
|
||||
/etc/aggregator/config.json
|
||||
/var/lib/aggregator/
|
||||
```
|
||||
|
||||
**Current dev paths (unreleased):**
|
||||
```
|
||||
/etc/redflag/config.json
|
||||
/var/lib/redflag-agent/ (broken, inconsistent)
|
||||
/var/lib/redflag/ (inconsistent)
|
||||
```
|
||||
|
||||
**Implication:** Only need to migrate from v0.1.18. Can ignore broken v0.1.19-v0.1.23 states.
|
||||
|
||||
**Timeline reduction:** 6h 50m → **3h 45m**
|
||||
|
||||
---
|
||||
|
||||
## Simplified Implementation Phases
|
||||
|
||||
### **Phase 1: Create Centralized Path Constants** (30 min)
|
||||
|
||||
**File:** `aggregator-agent/internal/constants/paths.go` (NEW)
|
||||
|
||||
```go
|
||||
package constants
|
||||
|
||||
import (
|
||||
"runtime"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// Base directories
|
||||
const (
|
||||
LinuxBaseDir = "/var/lib/redflag"
|
||||
WindowsBaseDir = "C:\\ProgramData\\RedFlag"
|
||||
)
|
||||
|
||||
// Subdirectory structure
|
||||
const (
|
||||
AgentDir = "agent"
|
||||
ServerDir = "server"
|
||||
CacheSubdir = "cache"
|
||||
StateSubdir = "state"
|
||||
MigrationSubdir = "migration_backups"
|
||||
)
|
||||
|
||||
// Config paths
|
||||
const (
|
||||
LinuxConfigBase = "/etc/redflag"
|
||||
WindowsConfigBase = "C:\\ProgramData\\RedFlag"
|
||||
ConfigFile = "config.json"
|
||||
)
|
||||
|
||||
// Log paths
|
||||
const (
|
||||
LinuxLogBase = "/var/log/redflag"
|
||||
)
|
||||
|
||||
// Legacy paths for migration
|
||||
const (
|
||||
LegacyConfigPath = "/etc/aggregator/config.json"
|
||||
LegacyStatePath = "/var/lib/aggregator"
|
||||
)
|
||||
|
||||
// GetBaseDir returns platform-specific base directory
|
||||
func GetBaseDir() string {
|
||||
if runtime.GOOS == "windows" {
|
||||
return WindowsBaseDir
|
||||
}
|
||||
return LinuxBaseDir
|
||||
}
|
||||
|
||||
// GetAgentStateDir returns /var/lib/redflag/agent/state
|
||||
func GetAgentStateDir() string {
|
||||
return filepath.Join(GetBaseDir(), AgentDir, StateSubdir)
|
||||
}
|
||||
|
||||
// GetAgentCacheDir returns /var/lib/redflag/agent/cache
|
||||
func GetAgentCacheDir() string {
|
||||
return filepath.Join(GetBaseDir(), AgentDir, CacheSubdir)
|
||||
}
|
||||
|
||||
// GetMigrationBackupDir returns /var/lib/redflag/agent/migration_backups
|
||||
func GetMigrationBackupDir() string {
|
||||
return filepath.Join(GetBaseDir(), AgentDir, MigrationSubdir)
|
||||
}
|
||||
|
||||
// GetAgentConfigPath returns /etc/redflag/agent/config.json
|
||||
func GetAgentConfigPath() string {
|
||||
if runtime.GOOS == "windows" {
|
||||
return filepath.Join(WindowsConfigBase, AgentDir, ConfigFile)
|
||||
}
|
||||
return filepath.Join(LinuxConfigBase, AgentDir, ConfigFile)
|
||||
}
|
||||
|
||||
// GetAgentConfigDir returns /etc/redflag/agent
|
||||
func GetAgentConfigDir() string {
|
||||
if runtime.GOOS == "windows" {
|
||||
return filepath.Join(WindowsConfigBase, AgentDir)
|
||||
}
|
||||
return filepath.Join(LinuxConfigBase, AgentDir)
|
||||
}
|
||||
|
||||
// GetAgentLogDir returns /var/log/redflag/agent
|
||||
func GetAgentLogDir() string {
|
||||
return filepath.Join(LinuxLogBase, AgentDir)
|
||||
}
|
||||
|
||||
// GetLegacyAgentConfigPath returns legacy /etc/aggregator/config.json
|
||||
func GetLegacyAgentConfigPath() string {
|
||||
return LegacyConfigPath
|
||||
}
|
||||
|
||||
// GetLegacyAgentStatePath returns legacy /var/lib/aggregator
|
||||
func GetLegacyAgentStatePath() string {
|
||||
return LegacyStatePath
|
||||
}
|
||||
```
|
||||
|
||||
### **Phase 2: Update Agent Main** (30 min)
|
||||
|
||||
**File:** `aggregator-agent/cmd/agent/main.go`
|
||||
|
||||
**Changes:**
|
||||
```go
|
||||
// 1. Remove these functions (lines 40-54):
|
||||
// - getConfigPath()
|
||||
// - getStatePath()
|
||||
|
||||
// 2. Add import:
|
||||
import "github.com/Fimeg/RedFlag/aggregator-agent/internal/constants"
|
||||
|
||||
// 3. Update all references:
|
||||
// Line 88: cfg.Save(getConfigPath()) → cfg.Save(constants.GetAgentConfigPath())
|
||||
// Line 240: BackupPath: filepath.Join(getStatePath(), "migration_backups") → constants.GetMigrationBackupDir()
|
||||
// Line 49: cfg.Save(getConfigPath()) → cfg.Save(constants.GetAgentConfigPath())
|
||||
|
||||
// 4. Remove: import "runtime" (no longer needed in main.go)
|
||||
|
||||
// 5. Remove: import "path/filepath" (unless used elsewhere)
|
||||
```
|
||||
|
||||
### **Phase 3: Update Cache System** (15 min)
|
||||
|
||||
**File:** `aggregator-agent/internal/cache/local.go`
|
||||
|
||||
**Changes:**
|
||||
```go
|
||||
// 1. Add imports:
|
||||
import (
|
||||
"path/filepath"
|
||||
"github.com/Fimeg/RedFlag/aggregator-agent/internal/constants"
|
||||
)
|
||||
|
||||
// 2. Remove constant (line 26):
|
||||
// OLD: const CacheDir = "/var/lib/redflag-agent"
|
||||
|
||||
// 3. Update cacheFile (line 29):
|
||||
// OLD: const CacheFile = "last_scan.json"
|
||||
// NEW: const cacheFile = "last_scan.json" // unexported
|
||||
|
||||
// 4. Update GetCachePath():
|
||||
// OLD:
|
||||
func GetCachePath() string {
|
||||
return filepath.Join(CacheDir, CacheFile)
|
||||
}
|
||||
|
||||
// NEW:
|
||||
func GetCachePath() string {
|
||||
return filepath.Join(constants.GetAgentCacheDir(), cacheFile)
|
||||
}
|
||||
|
||||
// 5. Update Load() and Save() to use constants.GetAgentCacheDir() instead of CacheDir
|
||||
```
|
||||
|
||||
### **Phase 4: Update Migration Detection** (20 min)
|
||||
|
||||
**File:** `aggregator-agent/internal/migration/detection.go`
|
||||
|
||||
**Changes:**
|
||||
```go
|
||||
// 1. Add imports:
|
||||
import (
|
||||
"path/filepath"
|
||||
"github.com/Fimeg/RedFlag/aggregator-agent/internal/constants"
|
||||
)
|
||||
|
||||
// 2. Update NewFileDetectionConfig():
|
||||
func NewFileDetectionConfig() *FileDetectionConfig {
|
||||
return &FileDetectionConfig{
|
||||
OldConfigPath: "/etc/aggregator", // v0.1.18 legacy
|
||||
OldStatePath: "/var/lib/aggregator", // v0.1.18 legacy
|
||||
NewConfigPath: constants.GetAgentConfigDir(),
|
||||
NewStatePath: constants.GetAgentStateDir(),
|
||||
BackupDirPattern: filepath.Join(constants.GetMigrationBackupDir(), "%d"),
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Update DetectLegacyInstallation() to ONLY check for v0.1.18 paths:
|
||||
func (d *Detector) DetectLegacyInstallation() (bool, error) {
|
||||
// Check for v0.1.18 legacy paths ONLY
|
||||
if d.fileExists(constants.GetLegacyAgentConfigPath()) {
|
||||
log.Info("Detected legacy v0.1.18 installation")
|
||||
return true, nil
|
||||
}
|
||||
return false, nil
|
||||
}
|
||||
```
|
||||
|
||||
### **Phase 5: Update Installer Template** (30 min)
|
||||
|
||||
**File:** `aggregator-server/internal/services/templates/install/scripts/linux.sh.tmpl`
|
||||
|
||||
**Key Changes:**
|
||||
```bash
|
||||
# Update header (lines 16-49):
|
||||
AGENT_USER="redflag-agent"
|
||||
BASE_DIR="/var/lib/redflag"
|
||||
AGENT_HOME="/var/lib/redflag/agent"
|
||||
CONFIG_DIR="/etc/redflag"
|
||||
AGENT_CONFIG_DIR="/etc/redflag/agent"
|
||||
LOG_DIR="/var/log/redflag"
|
||||
AGENT_LOG_DIR="/var/log/redflag/agent"
|
||||
|
||||
# Update directory creation (lines 175-179):
|
||||
sudo mkdir -p "${BASE_DIR}"
|
||||
sudo mkdir -p "${AGENT_HOME}"
|
||||
sudo mkdir -p "${AGENT_HOME}/cache"
|
||||
sudo mkdir -p "${AGENT_HOME}/state"
|
||||
sudo mkdir -p "${AGENT_CONFIG_DIR}"
|
||||
sudo mkdir -p "${AGENT_LOG_DIR}"
|
||||
|
||||
# Update ReadWritePaths (line 269):
|
||||
ReadWritePaths=/var/lib/redflag /var/lib/redflag/agent /var/lib/redflag/agent/cache /var/lib/redflag/agent/state /var/lib/redflag/agent/migration_backups /etc/redflag /var/log/redflag
|
||||
|
||||
# Update backup path (line 46):
|
||||
BACKUP_DIR="${AGENT_CONFIG_DIR}/backups/backup.$(date +%s)"
|
||||
```
|
||||
|
||||
### **Phase 6: Simplified Migration Logic** (20 min)
|
||||
|
||||
**File:** `aggregator-agent/internal/migration/executor.go`
|
||||
|
||||
```go
|
||||
// Simplified migration - only handles v0.1.18
|
||||
func (e *Executor) RunMigration() error {
|
||||
log.Info("Checking for legacy v0.1.18 installation...")
|
||||
|
||||
// Only check for v0.1.18 legacy paths
|
||||
if !e.fileExists(constants.GetLegacyAgentConfigPath()) {
|
||||
log.Info("No legacy installation found, fresh install")
|
||||
return nil
|
||||
}
|
||||
|
||||
// Create backup
|
||||
backupDir := filepath.Join(
|
||||
constants.GetMigrationBackupDir(),
|
||||
fmt.Sprintf("pre_v0.2.0_migration_%d", time.Now().Unix()))
|
||||
|
||||
if err := e.createBackup(backupDir); err != nil {
|
||||
return fmt.Errorf("failed to create backup: %w", err)
|
||||
}
|
||||
|
||||
log.Info("Migrating from v0.1.18 to v0.2.0...")
|
||||
|
||||
// Migrate config
|
||||
if err := e.migrateConfig(); err != nil {
|
||||
return e.rollback(backupDir, err)
|
||||
}
|
||||
|
||||
// Migrate state
|
||||
if err := e.migrateState(); err != nil {
|
||||
return e.rollback(backupDir, err)
|
||||
}
|
||||
|
||||
log.Info("Migration completed successfully")
|
||||
return nil
|
||||
}
|
||||
|
||||
// Helper methods remain similar but simplified for v0.1.18 only
|
||||
```
|
||||
|
||||
### **Phase 7: Update Acknowledgment System** (10 min)
|
||||
|
||||
**File:** `aggregator-agent/internal/acknowledgment/tracker.go`
|
||||
|
||||
```go
|
||||
// Add import:
|
||||
import "github.com/Fimeg/RedFlag/aggregator-agent/internal/constants"
|
||||
|
||||
// Update Save():
|
||||
func (t *Tracker) Save() error {
|
||||
stateDir := constants.GetAgentStateDir()
|
||||
if err := os.MkdirAll(stateDir, 0755); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
ackFile := filepath.Join(stateDir, "pending_acks.json")
|
||||
// ... save logic
|
||||
}
|
||||
```
|
||||
|
||||
### **Phase 8: Update Version** (5 min)
|
||||
|
||||
**File:** `aggregator-agent/cmd/agent/main.go`
|
||||
|
||||
```go
|
||||
// Line 32:
|
||||
const AgentVersion = "0.2.0" // Breaking: Directory structure reorganization
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Simplified Testing Requirements
|
||||
|
||||
### **Test Matrix (Reduced Complexity)**
|
||||
|
||||
**Fresh Installation Tests:**
|
||||
- [ ] Agent installs cleanly on Ubuntu 22.04
|
||||
- [ ] Agent installs cleanly on RHEL 9
|
||||
- [ ] Agent installs cleanly on Windows Server 2022
|
||||
- [ ] All directories created: `/var/lib/redflag/agent/{cache,state}`
|
||||
- [ ] Config created: `/etc/redflag/agent/config.json`
|
||||
- [ ] Logs created: `/var/log/redflag/agent/agent.log`
|
||||
- [ ] Agent starts and functions correctly
|
||||
|
||||
**Migration Tests (v0.1.18 only):**
|
||||
- [ ] v0.1.18 → v0.2.0 migration succeeds
|
||||
- [ ] Config migrated from `/etc/aggregator/config.json`
|
||||
- [ ] State migrated from `/var/lib/aggregator/`
|
||||
- [ ] Backup created in `/var/lib/redflag/agent/migration_backups/`
|
||||
- [ ] Rollback works if migration fails
|
||||
- [ ] Agent starts after migration
|
||||
|
||||
**Runtime Tests:**
|
||||
- [ ] Acknowledgments persist (writes to `/var/lib/redflag/agent/state/`)
|
||||
- [ ] Cache functions (reads/writes to `/var/lib/redflag/agent/cache/`)
|
||||
- [ ] Migration backups can be created (systemd allowed)
|
||||
- [ ] No permission errors under systemd
|
||||
|
||||
### **Migration Testing Script**
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# test_migration.sh
|
||||
|
||||
# Setup legacy v0.1.18 structure
|
||||
sudo mkdir -p /etc/aggregator
|
||||
sudo mkdir -p /var/lib/aggregator
|
||||
echo '{"agent_id":"test-123","version":18}' | sudo tee /etc/aggregator/config.json
|
||||
|
||||
# Run migration with new agent
|
||||
./aggregator-agent --config /etc/redflag/agent/config.json
|
||||
|
||||
# Verify migration
|
||||
if [ -f "/etc/redflag/agent/config.json" ]; then
|
||||
echo "✓ Config migrated"
|
||||
fi
|
||||
|
||||
if [ -d "/var/lib/redflag/agent/state" ]; then
|
||||
echo "✓ State structure created"
|
||||
fi
|
||||
|
||||
if [ -d "/var/lib/redflag/agent/migration_backups" ]; then
|
||||
echo "✓ Backup created"
|
||||
fi
|
||||
|
||||
# Cleanup test
|
||||
sudo rm -rf /etc/aggregator /var/lib/aggregator
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Timeline: 3 Hours 30 Minutes
|
||||
|
||||
| Phase | Task | Time | Status |
|
||||
|-------|------|------|--------|
|
||||
| 1 | Create constants | 30 min | Pending |
|
||||
| 2 | Update main.go | 30 min | Pending |
|
||||
| 3 | Update cache | 15 min | Pending |
|
||||
| 4 | Update migration | 20 min | Pending |
|
||||
| 5 | Update installer | 30 min | Pending |
|
||||
| 6 | Update tracker | 10 min | Pending |
|
||||
| 7 | Update version | 5 min | Pending |
|
||||
| 8 | Testing | 60 min | Pending |
|
||||
| **Total** | | **3h 30m** | **Not started** |
|
||||
|
||||
---
|
||||
|
||||
## Pre-Integration Checklist (Simplified)
|
||||
|
||||
✅ **Completed:**
|
||||
- [x] Path constants centralized
|
||||
- [x] Security review: No unauthenticated endpoints
|
||||
- [x] Backup/restore paths defined
|
||||
- [x] Idempotency: Only v0.1.18 → v0.2.0 (one-time)
|
||||
- [x] Error logging throughout
|
||||
|
||||
**Remaining for v0.2.0 release:**
|
||||
- [ ] Implementation complete
|
||||
- [ ] Fresh install tested (Ubuntu, RHEL, Windows)
|
||||
- [ ] Migration tested (v0.1.18 → v0.2.0)
|
||||
- [ ] History table logging added
|
||||
- [ ] Documentation updated
|
||||
- [ ] CHANGELOG.md created
|
||||
- [ ] Release notes drafted
|
||||
|
||||
---
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
**Risk Level: LOW**
|
||||
|
||||
**Factors reducing risk:**
|
||||
- Only one legacy path to support (v0.1.18)
|
||||
- No broken intermediate versions in the wild
|
||||
- Migration has rollback capability
|
||||
- Fresh installs are clean, no legacy debt
|
||||
- Small user base (~20 users) for controlled rollout
|
||||
|
||||
**Mitigation:**
|
||||
- Rollback script auto-generated with each migration
|
||||
- Backup created before any migration changes
|
||||
- Idempotent migration (can detect already-migrated state)
|
||||
- Extensive logging for debugging
|
||||
|
||||
---
|
||||
|
||||
## What We Learned
|
||||
|
||||
**The power of checking legacy code:**
|
||||
- Saved 3+ hours of unnecessary migration complexity
|
||||
- Eliminated need for v0.1.19-v0.1.23 broken state handling
|
||||
- Reduced testing surface area significantly
|
||||
- Clarified actual legacy state (not assumed)
|
||||
|
||||
**Lesson:** Always verify legacy paths BEFORE designing migration.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Decision
|
||||
|
||||
**Recommended approach:** Full nested structure implementation
|
||||
|
||||
**Rationale:**
|
||||
- Only 3.5 hours vs. original 6h 50m estimate
|
||||
- Aligns with Ethos #3 (Resilience) and #5 (No BS)
|
||||
- Permanent architectural improvement
|
||||
- Future-proof for server component
|
||||
- Clean slate - no intermediate version debt
|
||||
|
||||
**Coffee level required:** 1-2 cups
|
||||
|
||||
**Break points:** Stop after any phase, pick up next session
|
||||
|
||||
**Ready to implement?**
|
||||
|
||||
*- Ani, having done the homework before building*
|
||||
Reference in New Issue
Block a user