BREAKING CHANGE: Storage and system scans no longer create entries in update_logs
**Problem**
- Storage scans were appearing on Updates page (mixed with package updates)
- System scans were appearing on Updates page (mixed with package updates)
- Duplicate "Scan All" entries from collective + individual logging
**Root Cause**
Scan handlers were calling both ReportLog() and dedicated endpoints:
- reportLogWithAck → POST /api/v1/agents/:id/logs → update_logs table
- This caused storage/system metrics to appear alongside package updates
**Fix**
Removed ALL ReportLog() calls from scan handlers:
1. handleScanUpdatesV2 (lines 44-46): Removed collective logging
2. handleScanStorage (lines 103-105): Use only ReportStorageMetrics
3. handleScanSystem (lines 189-191): Use only ReportMetrics
4. handleScanDocker (lines 269-271): Use only ReportDockerImages
**Verification**
- All 4 handlers have working dedicated endpoints (verified via subagent)
- Routes already registered: POST /storage-metrics, POST /metrics, etc.
- Frontend queries correct endpoints (verified)
- No data loss: dedicated endpoints store in proper tables
**Result**
- Storage scans → storage_metrics table → Storage page only ✅
- System scans → system reporting → System page only ✅
- Package updates → update_logs table → Updates page only ✅
- No duplicate "Scan All" entries ✅
**Files Changed**
- aggregator-agent/cmd/agent/subsystem_handlers.go: Removed 20 lines of ReportLog calls
- internal/api/handlers/agents.go: Command recovery enhancements
- internal/api/handlers/updates.go: Subsystem extraction logic
- internal/database/queries/commands.go: GetStuckCommands query
Problem: Version check middleware blocked old agents from checking in to receive
update commands, creating a deadlock where agents couldn't upgrade because they
were blocked from checking in.
Solution: Modified MachineBindingMiddleware to allow old agents checking in for
commands to proceed IF they have a pending update_agent command. This allows
agents to receive the update command even when below minimum version.
Changes:
- Added grace period logic in middleware for command endpoints
- Check if agent has pending update command before blocking
- If update pending, allow check-in and log it
- Added HasPendingUpdateCommand() to AgentQueries for checking pending updates
- Also added same method to CommandQueries for completeness
This prevents the version tracking deadlock while maintaining security for
agents without pending updates.
NOTE: Need to test that old agents can actually receive and execute update
commands when allowed through this path.
Simplified install script approach based on architecture analysis:
- Fresh installs: Create minimal config with registration_token only
- Agent handles registration on first start (leverages existing agent logic)
- Upgrades: Preserve existing config, agent handles migration
- Removed complex credential preservation logic from Windows script
This is more reliable and aligns with the agent's built-in migration system.
Changes:
- Linux: Populate registration_token in config template, keep backup logic
- Windows: Simplified - removed 100+ lines of credential extraction/restoration
- Both: Fresh installs get minimal template, upgrades preserve existing config
NOTE: This commit modified the 'sacred scripts' (install templates) significantly.
Casey found this highly suspect and it may need investigation, but proceeding for now
to test the approach. The changes should be reviewed carefully before v0.1.x release.
- Pass registration token from URL query parameter to install script generation
- Update RenderInstallScriptFromBuild to accept registration token
- Add RegistrationToken field to template data structure
This lays groundwork for fixing agent registration - install scripts will be able
to call the registration API with the provided token.
- Move Update Agent button to Subsystem Configuration header
- Remove duplicate Compact Summary box with misaligned refresh
- Reduce visual separation between sections (same card styling)
- Make security status details visible instead of hidden in tooltips
- Fix enforced status colors (blue instead of red)
- Consolidate enabled/auto-run counts in header
- Reduce spacing between sections for cohesive interface
The enabled/auto-run toggles now properly align with their
subsystems in the table, and critical security information
is immediately visible without hover interactions.
Created InstallTemplateService with clean template-based script generation.
Added linux.sh.tmpl and windows.ps1.tmpl for install scripts.
Removed massive generateLinuxScript and generateWindowsScript functions.
Downloads handler now uses template service (1073 lines → 174 lines).
Templates easily maintainable without modifying Go code.
Critical regression fix - subsystems were hardcoded instead of reading user settings.
Added CreateDefaultSubsystems to queries/subsystems.go.
ConfigService now queries agent_subsystems table for actual user configuration.
AgentLifecycleService creates default subsystems when creating new agents.
Respects user-configured enabled/auto_run settings from UI.
Created centralized ConfigService for configuration management.
Added deprecation comments to ConfigBuilder and AgentBuilder.
Platform-specific defaults centralized in one place.
Removed placeholder ConfigService from agent_lifecycle.go.
Created centralized lifecycle service to handle new, upgrade, and rebuild operations.
Added deprecation notices to old handlers (agent_setup, build_orchestrator, agent_build).
Foundation for consolidating duplicate agent lifecycle logic.
Created aggregator/pkg/common module with shared AgentFile type.
Removed duplicate definitions from migration and services packages.
Both agent and server now use common.AgentFile.
Created version package for semantic version comparison.
Fixed GetLatestVersionByTypeAndArch to use combined platform format.
Replaced inline version comparison with reusable version.Compare().
- Add GET /api/v1/agents/:id/config endpoint for server configuration
- Agent fetches config during check-in and applies updates
- Add version tracking to prevent unnecessary config applications
- Clean separation: config sync independent of commands
- Fix agent UI subsystem settings to actually control agent behavior
- Update Security Health UI with frosted glass styling and tooltips
- Create separate scanner interfaces for storage, system, and docker data
- Add dedicated endpoints for metrics and docker images instead of misclassifying as updates
- Implement proper database tables for storage metrics and docker images
- Fix storage/system metrics appearing incorrectly as package updates
- Add scanner types with proper data structures for each subsystem
- Update agent handlers to use correct endpoints for each data type
- Fix recursive call in reportLogWithAck that caused infinite loop
- Add machine binding and security API endpoints
- Enhance AgentScanners component with security status display
- Update scheduler and timeout service reliability
- Remove deprecated install.sh script
- Add subsystem configuration and logging improvements
added ed25519 key generation to setup endpoint
deployment handler for token CRUD with install commands
wired routes for /api/setup/generate-keys and /admin/deployment
setup generates keypair on demand
deployment endpoints provide one-liner install commands
ready for v0.1.22 testing
Split monolithic scan_updates into individual subsystems (updates/storage/system/docker).
Scanners now run in parallel via goroutines - cuts scan time roughly in half, maybe more.
Agent changes:
- Orchestrator pattern for scanner management
- New scanners: storage (disk metrics), system (cpu/mem/processes)
- New commands: scan_storage, scan_system, scan_docker
- Wrapped existing scanners (APT/DNF/Docker/Windows/Winget) with common interface
- Version bump to 0.1.20
Server changes:
- Migration 015: agent_subsystems table with trigger for auto-init
- Subsystem CRUD: enable/disable, interval (5min-24hr), auto-run toggle
- API routes: /api/v1/agents/:id/subsystems/* (9 endpoints)
- Stats tracking per subsystem
Web UI changes:
- ChatTimeline shows subsystem-specific labels and icons
- AgentScanners got interactive toggles, interval dropdowns, manual trigger buttons
- TypeScript types added for subsystems
Backward compatible with legacy scan_updates - for now. Bugs probably exist somewhere.
Added circuit breakers with configurable timeouts for all subsystems (APT, DNF, Docker, Windows, Winget, Storage). Replaces cron-based scheduler with priority queue that should scale beyond 1000+ agents if your homelab is that big.
Command acknowledgment system ensures results aren't lost on network failures or restarts. Agent tracks pending acknowledgments with persistent state and automatic retry.
- Circuit breakers: 3 failures in 1min opens circuit, 30s cooldown
- Per-subsystem timeouts: 30s-10min depending on scanner
- Priority queue scheduler: O(log n), worker pool, jitter, backpressure
- Acknowledgments: at-least-once delivery, max 10 retries over 24h
- All tests passing (26/26)
- Redesign AgentUpdatesEnhanced with tab-based workflow (pending/approved/installing/installed)
- Add AgentStorage component with disk partition table
- Add AgentScanners component for agent health monitoring
- Fix agent removal not refreshing list (cache invalidation)
- Bump agent version to 0.1.18 (enhanced disk detection)
- Update server default version to 0.1.18
- Add command source tracking (system/manual) migration
- Improve Linux disk detection for all physical mount points
Rate limiter fix:
- Namespace keys by limit type to prevent counter sharing across endpoints
- Previously all KeyByIP endpoints shared same counter causing false rate limits
- Now agent_registration, public_access, etc have separate counters per IP
- Example: "agent_registration:127.0.0.1" vs "public_access:127.0.0.1"
Session loop fix:
- Remove wasInSetupMode from SetupCompletionChecker dependency array
- Use local variable instead of state to prevent interval multiplication
- Prevents rapid refresh loop during server restart after setup
- (turns out useEffect dependency arrays actually matter, who knew)
Tested:
- First agent registration now succeeds without rate limit (was 429)
- Public access requests don't affect agent registration quota
- No UI flashing during server restart
- Rate limit API endpoints functional (Settings UI needs work)
The reboot_reason field was defined as string instead of *string, causing
database scan failures when the column contains NULL values. This broke
agent list loading on existing installations after migration.
- Changed reboot_reason to *string in both Agent and AgentWithLastScan structs
- Added DEFAULT empty string to migration for new installations
- Added README section for full server reinstall procedure
Potential fixes for issues #4 and #6.
Agent version display:
- Set CurrentVersion during registration instead of waiting for first check-in
- Update UI to show "Initial Registration" instead of "Unknown"
Host restart detection:
- Added reboot_required, last_reboot_at, reboot_reason fields to agents table
- Agent now detects pending reboots (Debian/Ubuntu via /var/run/reboot-required, RHEL/Fedora via needs-restarting)
- New reboot command type with 1-minute grace period
- UI shows restart alerts and adds restart button in quick actions
- Restart indicator badge in agent list
The reboot detection runs during system info collection and gets reported back to the server automatically.
Using shutdown command for now until we make the restart mechanism user-adjustable later - need to think on that.
Also need to come up with a Windows derivative outside of reading event log for detecting reboots.
- Add /api/v1/health endpoint for web app compatibility
- Add detailed error logging to setup handler
- Show actual error messages when config save fails
- Helps debug permission and filesystem issues
- Add React setup form matching our design system
- Implement automatic server restart after configuration
- Add WelcomeChecker component for proper routing
- Update API to handle setup endpoints and restart logic
- Improve setup workflow with proper loading states and redirects
Testing complete setup workflow from welcome mode to login.
- Remove aggregator-server/internal/utils/ from gitignore
- Add utils/version.go with version comparison functions
- Resolves 'does not contain package' error when cloning from GitHub
- Update go.mod files to use github.com/Fimeg/RedFlag module path
- Fix all import statements across server and agent code
- Resolves build errors when cloning from GitHub
- Utils package (version comparison) is actually needed and working
Breaking changes for clean alpha releases:
- JWT authentication with user-provided secrets (no more development defaults)
- Registration token system for secure agent enrollment
- Rate limiting with user-adjustable settings
- Enhanced agent configuration with proxy support
- Interactive server setup wizard (--setup flag)
- Heartbeat architecture separation for better UX
- Package status synchronization fixes
- Accurate timestamp tracking for RMM features
Setup process for new installations:
1. docker-compose up -d postgres
2. ./redflag-server --setup
3. ./redflag-server --migrate
4. ./redflag-server
5. Generate tokens via admin UI
6. Deploy agents with registration tokens
- Cross-platform support (Windows/Linux) with Windows Updates and Winget
- Added dependency confirmation workflow and refresh token authentication
- New screenshots: History, Live Operations, Windows Agent Details
- Local CLI features with terminal output and cache system
- Updated known limitations - Proxmox integration is broken
- Organized docs to docs/ folder and updated .gitignore
- Probably introduced a dozen bugs with Windows agents - stay tuned
- Fixed gitignore to allow Screenshots/*.png files
- Added all screenshots for README documentation
- Fixed gitignore to be less restrictive with image files
- Includes dashboard, agent, updates, and docker screenshots
