Add docs and project files - force for Culurien

This commit is contained in:
Fimeg
2026-03-28 20:46:24 -04:00
parent dc61797423
commit 484a7f77ce
343 changed files with 119530 additions and 0 deletions

View File

@@ -0,0 +1,347 @@
# 🚩 RedFlag Development - Session 5 Complete!
## Project State (Session 5 Complete - October 15, 2025)
**STATUS**: Session 5 successfully completed - JWT authentication fixed, Docker API endpoints implemented, agent architecture decided!
### ✅ What We Accomplished This Session
#### **JWT Authentication Completely Fixed** ✅
- **CRITICAL ISSUE RESOLVED**: JWT secret mismatch between config default and .env file
- **Root Cause Fixed**: Authentication middleware using different secret than token generation
- **Debug Implementation**: Added comprehensive logging for JWT validation troubleshooting
- **Result**: Consistent authentication across web interface and API endpoints
- **Security Note**: Development authentication now production-ready for dev environment
#### **Docker API Endpoints Fully Implemented** ✅
- **NEW Docker Handler**: Complete implementation at internal/api/handlers/docker.go (356 lines)
- **Full CRUD Operations**: Container listing, statistics, update approval/rejection/installation
- **Proper Authentication**: All Docker endpoints protected with JWT middleware
- **Cross-Agent Support**: Statistics and container management across all agents
- **Response Formats**: Paginated container lists with total counts and metadata
- **Route Registration**: All endpoints properly registered in main.go
#### **Docker Model Architecture Complete** ✅
- **DockerContainer Struct**: Container representation with update metadata and agent relationships
- **DockerStats Struct**: Cross-agent statistics and metrics for dashboard overview
- **Response Models**: Paginated container lists with proper JSON tags and pagination
- **Status Tracking**: Update availability, current/available versions, agent associations
- **Compilation Success**: All Docker models compile without errors
#### **Agent Architecture Decision Made** ✅
- **Universal Strategy Confirmed**: Single Linux agent + Windows agent (not platform-specific)
- **Rationale**: More maintainable, Docker runs on all platforms, plugin-based detection
- **Architecture Plan**:
- Linux agent handles APT/YUM/DNF/Docker
- Windows agent handles Winget/Windows Updates
- **Benefits**: Easier deployment, unified codebase, cross-platform Docker support
- **Future Enhancement**: Plugin system for platform-specific optimizations
#### **Compilation and Technical Issues Resolved** ✅
- **JSONB Handling**: Fixed metadata access from interface type to map operations
- **Model Field References**: Corrected VersionTo → AvailableVersion field references
- **Type Safety**: Proper UUID parsing and error handling throughout
- **Authentication Debug**: JWT validation logging for easier troubleshooting
- **Result**: All endpoints compile and run without technical errors
### 🚧 Current System Status
#### **Working Components** ✅
- **Backend Server**: Running on port 8080, full REST API with authentication
- **Web Dashboard**: Running on port 3000, authenticated, CORS enabled
- **Agent Registration**: Enhanced system information collection working
- **Docker API**: Complete container management endpoints functional
- **JWT Authentication**: Consistent token validation across all endpoints
- **Database**: Event sourcing architecture handling thousands of updates
#### **Web Dashboard State** ✅
- **Authentication**: Working with JWT tokens and proper session management
- **Agents Page**: Can view agents with system specs, trigger scans
- **Agent Details**: Complete system information display with resource indicators
- **Updates Page**: Handling thousands of updates with pagination
- **Docker Integration**: Ready for Docker container management features
- **CORS**: Fixed - no more cross-origin errors
#### **Agent Capabilities** ✅
- **System Information**: Enhanced collection with CPU, memory, disk, processes, uptime
- **Registration**: Working with detailed metadata and proper JWT authentication
- **Command Processing**: Ready to handle scan commands and update operations
- **Package Detection**: APT scanner operational, DNF/RPM support ready for implementation
- **Docker Scanning**: Production-ready with Registry API v2 integration
### 🎯 What's Next (Session 6 Priorities)
### **HIGH Priority 1: System Domain Reorganization** ⚠️⚠️⚠️
**Status**: Updates page needs categorization by System Domain
**Files**: aggregator-web/src/pages/Updates.tsx, internal/models/
**Estimated Effort**: 2-3 hours
**Why Critical**: Makes the update management system usable and organized
**Implementation Needed**:
- Categorize updates into: OS & System, Applications & Services, Container Images, Development Tools
- Update UI to show categorized sections
- Add filtering by System Domain
- Prepare for AI subcomponent integration (Phase 3)
### **HIGH Priority 2: Agent Status Display Fixes** ⚠️⚠️
**Status**: Last check-in times not updating properly in UI
**Files**: aggregator-web/src/pages/Agents.tsx, internal/api/handlers/agents.go
**Estimated Effort**: 1-2 hours
**Why Critical**: User feedback indicates agent status display issues
**Implementation**:
- Fix last check-in time updates in agent status
- Ensure real-time status determination works correctly
- Update agent heartbeat mechanism
### **MEDIUM Priority 3: UI/UX Cleanup** 🔜
**Status**: Duplicate fields and layout improvements needed
**Files**: aggregator-web/src/pages/
**Estimated Effort**: 1-2 hours
**Why Soon**: Improves user experience and reduces confusion
**Features Needed**:
- Remove duplicate text fields in agent UI
- Improve layout and visual hierarchy
- Fix any remaining display inconsistencies
### **LOW Priority 4: DNF/RPM Package Scanner** 🔜
**Status**: Fedora agents can't scan packages (only APT available)
**Files**: aggregator-agent/internal/scanners/
**Estimated Effort**: 3-4 hours
**Why Later**: Critical but system organization and UI fixes take priority
### **MEDIUM Priority 5: Rate Limiting & Security** ⚠️
**Status**: API endpoints lack rate limiting (security gap vs PatchMon)
**Files**: aggregator-server/internal/api/middleware/
**Estimated Effort**: 2-3 hours
**Why Important**: Critical security feature before production use
## 🔧 Current Technical Status
### **Build Environment - FULLY FUNCTIONAL** ✅
```bash
# Backend Server
cd aggregator-server
./redflag-server
# ✅ Running on :8080, PostgreSQL connected, JWT auth working, Docker API ready
# Enhanced Agent
cd aggregator-agent
sudo ./aggregator-agent -register -server http://localhost:8080
# ✅ Enhanced system info, JWT authentication, Docker scanning
# Web Dashboard
cd aggregator-web
yarn dev
# ✅ Running on :3000, authenticated, CORS enabled, thousands of updates displayed
```
### **Database Status** ✅
- PostgreSQL running in Docker container
- Event sourcing architecture implemented (update_events, current_package_state tables)
- All migrations executed successfully
- Enhanced agent metadata stored and retrievable
- Scaling to thousands of updates efficiently
### **API Endpoints - Working** ✅
- `POST /api/v1/auth/login` - ✅ Authentication
- `GET /api/v1/auth/verify` - ✅ Token verification
- `GET /api/v1/stats/summary` - ✅ Dashboard statistics
- `GET /api/v1/agents` - ✅ Agent listing
- `GET /api/v1/agents/:id` - ✅ Enhanced agent details
- `POST /api/v1/agents/:id/scan` - ✅ Scan triggering
- `DELETE /api/v1/agents/:id` - ✅ Agent unregistration
- `GET /api/v1/updates` - ✅ Update listing (with pagination)
- `GET /api/v1/docker/containers` - ✅ Docker container listing ✅ NEW
- `GET /api/v1/docker/stats` - ✅ Docker statistics ✅ NEW
- `GET /api/v1/docker/agents/:id/containers` - ✅ Agent-specific containers ✅ NEW
### **Docker API Functionality** ✅
```go
// Key endpoints implemented:
GET /api/v1/docker/containers // List all containers across agents
GET /api/v1/docker/stats // Docker statistics across all agents
GET /api/v1/docker/agents/:id/containers // Containers for specific agent
POST /api/v1/docker/containers/:id/images/:id/approve // Approve update
POST /api/v1/docker/containers/:id/images/:id/reject // Reject update
POST /api/v1/docker/containers/:id/images/:id/install // Install immediately
```
## 📋 Session 6 Recommended Workflow
### **Phase 1: System Domain Reorganization (2-3 hours)**
1. **Categorization Logic**
- Define System Domain categories (OS & System, Applications & Services, Container Images, Development Tools)
- Implement categorization logic in backend or frontend
- Update database schema if needed for domain tagging
2. **UI Implementation**
- Reorganize Updates page with categorized sections
- Add filtering by System Domain
- Improve visual hierarchy and navigation
### **Phase 2: Agent Status Fixes (1-2 hours)**
1. **Agent Health Monitoring**
- Fix last check-in time updates
- Improve real-time status determination
- Update agent heartbeat mechanism
2. **UI Status Display**
- Ensure agent status updates correctly in web interface
- Fix any status display inconsistencies
### **Phase 3: UI/UX Cleanup (1-2 hours)**
1. **Layout Improvements**
- Remove duplicate fields and text
- Improve visual hierarchy
- Fix remaining display issues
2. **User Experience**
- Streamline navigation
- Improve responsive design
- Enhance visual feedback
### **Phase 4: Security & Rate Limiting (2-3 hours)**
1. **Rate Limiting Implementation**
- Add rate limiting middleware
- Implement different limits for different endpoint types
- Add proper error handling for rate-limited requests
2. **Security Hardening**
- Audit all API endpoints for security issues
- Add input validation
- Implement proper error handling
## 🎯 Success Criteria for Session 6
### **Minimum Viable Success**
✅ System Domain reorganization implemented
✅ Agent status display issues fixed
✅ UI cleanup completed (duplicate fields removed)
✅ Rate limiting implemented for security
### **Good Success**
✅ All minimum criteria PLUS
✅ Enhanced filtering and search capabilities
✅ Improved responsive design
✅ Better error handling and user feedback
### **Exceptional Success**
✅ All good criteria PLUS
✅ Real-time WebSocket updates for agent status
✅ Advanced bulk operations for updates
✅ Export functionality for reports
✅ Mobile-responsive design improvements
## 🚀 Current Technical Debt Status
### **Resolved Issues** ✅
- **JWT Authentication** - Fixed completely with debug logging
- **Docker API** - Full implementation with proper authentication
- **Compilation Errors** - All JSONB and model reference issues resolved
- **Agent Architecture** - Universal strategy decided and documented
- **Database Scalability** - Event sourcing handles thousands of updates
### **Outstanding Items**
- **System Domain Organization** - Updates need categorization (HIGH)
- **Agent Status Display** - Last check-in times not updating (HIGH)
- **UI/UX Cleanup** - Duplicate fields and layout issues (MEDIUM)
- **Rate Limiting** - Security gap vs PatchMon (HIGH)
- **DNF/RPM Support** - Fedora agents need package scanning (MEDIUM)
## 📚 Documentation Status
### **Technical Implementation** ✅
- JWT authentication system with comprehensive debug logging
- Docker API endpoints with full CRUD operations
- Event sourcing database architecture for scalability
- Universal agent architecture decision documented
- System domain categorization strategy
### **Architecture Improvements** ✅
- Cross-platform agent strategy defined
- Docker-first update management approach
- Event sourcing for scalable update handling
- Proper authentication and security practices
- RESTful API design with proper error handling
### **Testing Status** ✅
- JWT authentication flow verified
- Docker API endpoints tested and functional
- Agent registration with enhanced data verified
- Database event sourcing tested with thousands of updates
- Web dashboard pagination working correctly
## 🔍 Key Questions for Session 6
1. **System Domain Implementation**: Should categorization be handled in backend (database) or frontend (logic)?
2. **AI Integration Preparation**: How should we structure the System Domain categories to prepare for Phase 3 AI analysis?
3. **Rate Limiting Strategy**: What are appropriate rate limits for different endpoint types?
4. **Real-time Updates**: Should we implement WebSocket updates for agent status, or stick with polling?
5. **User Feedback**: Which UI improvements should take priority based on user experience?
## 🛠️ Development Environment Setup
### **Current Working Environment** ✅
```bash
# 1. Database (PostgreSQL in Docker)
docker run -d --name redflag-postgres -e POSTGRES_DB=aggregator -e POSTGRES_USER=aggregator -e POSTGRES_PASSWORD=aggregator -p 5432:5432 postgres:15
# 2. Backend Server (Terminal 1)
cd aggregator-server
./redflag-server
# ✅ Running on :8080, JWT auth working, Docker API ready
# 3. Web Dashboard (Terminal 2)
cd aggregator-web
yarn dev
# ✅ Running on :3000, authenticated, thousands of updates displayed
# 4. Enhanced Agent Registration (Terminal 3)
cd aggregator-agent
sudo ./aggregator-agent -register -server http://localhost:8080
# ✅ Enhanced system info, JWT authentication, Docker scanning
```
### **Current Agents Status** ✅
- **Agent 1**: `78d1e052-ff6d-41be-b064-fdd955214c4b` (Fedora, enhanced system specs)
- **Agent 2**: `49f9a1e8-66db-4d21-b3f4-f416e0523ed1` (Fedora, enhanced system specs)
- **Docker Images**: Multiple containers discovered and tracked
- **Updates**: Thousands of updates managed with event sourcing
### **Authentication Tokens** ✅
- **Web Dashboard Token**: Valid JWT with 24-hour expiry
- **Agent Authentication**: Proper JWT token validation
- **Debug Logging**: Comprehensive authentication debugging available
## 🎉 Session 5 Complete!
**RedFlag Session 5 is highly successful** with major foundational improvements:
- ✅ **JWT Authentication** - Completely fixed with comprehensive debugging
- ✅ **Docker API Implementation** - Full CRUD operations for container management
- ✅ **Agent Architecture Decision** - Universal strategy defined and documented
- ✅ **Technical Debt Resolution** - All compilation and authentication issues resolved
- ✅ **Scalability Foundation** - Event sourcing database handling thousands of updates
- ✅ **Security Enhancements** - Proper authentication with debug capabilities
**Current System Status**: ~75% functional for core use case
- Backend server: ✅ Fully operational with Docker API and JWT authentication
- Web dashboard: ✅ Built and accessible with authentication, handling thousands of updates
- Agent registration: ✅ Enhanced with comprehensive system information
- Docker management: ✅ Complete API foundation for container update orchestration
- Update discovery: ✅ Scaling to thousands of updates with event sourcing
- Update organization: 🔧 System domain categorization needed
**Session 6 Goal**: Implement System Domain reorganization to make the update management system truly usable and organized, preparing for advanced AI-powered update intelligence in Phase 3.
**Let's make RedFlag truly organized and intelligent!** 🚩
---
*Last Updated: 2025-10-15 (Session 5 Complete - JWT Auth + Docker API Complete)*
*Next Session: Session 6 - System Domain Organization + UI/UX Improvements*

View File

@@ -0,0 +1,368 @@
# 🚩 RedFlag (Aggregator)
**"From each according to their updates, to each according to their needs"**
> 🚧 **IN ACTIVE DEVELOPMENT - NOT PRODUCTION READY**
> Alpha software - use at your own risk. Breaking changes expected.
A self-hosted, cross-platform update management platform that provides centralized visibility and control over system updates across your entire infrastructure.
## What is RedFlag?
RedFlag is an open-source update management dashboard that gives you a **single pane of glass** for:
- **Windows Updates** (coming soon)
- **Linux packages** (apt, yum/dnf - MVP has apt)
- **Winget applications** (coming soon)
- **Docker containers** ✅
Think of it as your own self-hosted RMM (Remote Monitoring & Management) for updates, but:
- ✅ **Open source** (AGPLv3)
- ✅ **Self-hosted** (your data, your infrastructure)
- ✅ **Beautiful** (modern React dashboard)
- ✅ **Cross-platform** (Go agents + web interface)
## Current Status: Session 4 Complete (October 13, 2025)
⚠️ **ALPHA SOFTWARE - Development in Progress**
🎉 **✅ What's Working Now:**
- ✅ **Server backend** (Go + Gin + PostgreSQL) - Production ready
- ✅ **Linux agent** with APT scanner + local CLI features
- ✅ **Docker scanner** with real Registry API v2 integration
- ✅ **Web dashboard** (React + TypeScript + TailwindCSS) - Full UI
- ✅ **Agent registration** and check-in loop
- ✅ **Update discovery** and reporting
- ✅ **Update approval** workflow (web UI + API)
- ✅ **REST API** for all operations
- ✅ **Local CLI tools** (--scan, --status, --list-updates, --export)
🚧 **Current Limitations:**
- ❌ No actual update installation yet (just discovery and approval)
- ❌ No CVE data enrichment from security advisories
- ❌ No Windows agent (planned)
- ❌ No rate limiting on API endpoints (security concern)
- ❌ Docker deployment not ready (needs networking config)
- ❌ No real-time WebSocket updates (polling only)
🔜 **Next Development Session:**
- Real-time updates with WebSocket or polling
- Update installation execution (APT packages first)
- Rate limiting and security hardening
- Docker Compose deployment with proper networking
- Windows agent foundation
## Architecture
```
┌─────────────────┐
│ Web Dashboard │ ✅ React + TypeScript + TailwindCSS
└────────┬────────┘
│ HTTPS
┌────────▼────────┐
│ Server (Go) │ ✅ Production Ready
│ + PostgreSQL │
└────────┬────────┘
│ Pull-based (agents check in every 5 min)
┌────┴────┬────────┐
│ │ │
┌───▼──┐ ┌──▼──┐ ┌──▼───┐
│Linux │ │Linux│ │Linux │
│Agent │ │Agent│ │Agent │
└──────┘ └─────┘ └──────┘
```
## Quick Start
⚠️ **BEFORE YOU BEGIN**: Read [SECURITY.md](SECURITY.md) and change your JWT secret!
### Prerequisites
- Go 1.25+
- Docker & Docker Compose
- PostgreSQL 16+ (provided via Docker Compose)
- Linux system (for agent testing)
### 1. Start the Database
```bash
make db-up
```
This starts PostgreSQL in Docker.
### 2. Start the Server
```bash
cd aggregator-server
cp .env.example .env
# Edit .env if needed (defaults are fine for local development)
go run cmd/server/main.go
```
The server will:
- Connect to PostgreSQL
- Run database migrations automatically
- Start listening on `:8080`
You should see:
```
✓ Executed migration: 001_initial_schema.up.sql
🚩 RedFlag Aggregator Server starting on :8080
```
### 3. Register an Agent
On the machine you want to monitor:
```bash
cd aggregator-agent
go build -o aggregator-agent cmd/agent/main.go
# Register with server
sudo ./aggregator-agent -register -server http://YOUR_SERVER:8080
```
You should see:
```
✓ Agent registered successfully!
Agent ID: 550e8400-e29b-41d4-a716-446655440000
```
### 4. Run the Agent
```bash
sudo ./aggregator-agent
```
The agent will:
- Check in with the server every 5 minutes
- Scan for APT updates
- Scan for Docker image updates
- Report findings to the server
### 5. Access the Web Dashboard
```bash
cd aggregator-web
yarn install
yarn dev
```
Visit http://localhost:3000 and login with your JWT token.
## API Usage
### List All Agents
```bash
curl http://localhost:8080/api/v1/agents
```
### Trigger Update Scan
```bash
curl -X POST http://localhost:8080/api/v1/agents/{agent-id}/scan
```
### List All Updates
```bash
# All updates
curl http://localhost:8080/api/v1/updates
# Filter by severity
curl http://localhost:8080/api/v1/updates?severity=critical
# Filter by status
curl http://localhost:8080/api/v1/updates?status=pending
# Filter by package type
curl http://localhost:8080/api/v1/updates?package_type=apt
```
### Approve an Update
```bash
curl -X POST http://localhost:8080/api/v1/updates/{update-id}/approve
```
## Project Structure
```
RedFlag/
├── aggregator-server/ # Go server (Gin + PostgreSQL)
│ ├── cmd/server/ # Main entry point
│ ├── internal/
│ │ ├── api/ # HTTP handlers & middleware
│ │ ├── database/ # Database layer & migrations
│ │ ├── models/ # Data models
│ │ └── config/ # Configuration
│ └── go.mod
├── aggregator-agent/ # Go agent
│ ├── cmd/agent/ # Main entry point
│ ├── internal/
│ │ ├── client/ # API client
│ │ ├── scanner/ # Update scanners (APT, Docker)
│ │ └── config/ # Configuration
│ └── go.mod
├── aggregator-web/ # React dashboard ✅
├── docker-compose.yml # PostgreSQL for local dev
├── Makefile # Common tasks
└── README.md # This file
```
## Database Schema
**Key Tables:**
- `agents` - Registered agents
- `update_packages` - Discovered updates
- `agent_commands` - Command queue for agents
- `update_logs` - Execution logs
- `agent_tags` - Agent tagging/grouping
See `aggregator-server/internal/database/migrations/001_initial_schema.up.sql` for full schema.
## Configuration
### Server (.env)
```bash
SERVER_PORT=8080
DATABASE_URL=postgres://aggregator:aggregator@localhost:5432/aggregator?sslmode=disable
JWT_SECRET=change-me-in-production
CHECK_IN_INTERVAL=300 # seconds
OFFLINE_THRESHOLD=600 # seconds
```
### Agent (/etc/aggregator/config.json)
Auto-generated on registration:
```json
{
"server_url": "http://localhost:8080",
"agent_id": "uuid",
"token": "jwt-token",
"check_in_interval": 300
}
```
## Development
### Makefile Commands
```bash
make help # Show all commands
make db-up # Start PostgreSQL
make db-down # Stop PostgreSQL
make server # Run server (with auto-reload)
make agent # Run agent
make build-server # Build server binary
make build-agent # Build agent binary
make test # Run tests
make clean # Clean build artifacts
```
### Running Tests
```bash
cd aggregator-server && go test ./...
cd aggregator-agent && go test ./...
```
## Security
- **Agent Authentication**: JWT tokens with 24h expiry
- **Pull-based Model**: Agents poll server (firewall-friendly)
- **Command Validation**: Whitelisted commands only
- **TLS Required**: Production deployments must use HTTPS
## Roadmap
### Phase 1: MVP (✅ Current)
- [x] Server backend with PostgreSQL
- [x] Agent registration & check-in
- [x] Linux APT scanner
- [x] Docker scanner
- [x] Update approval workflow
### Phase 2: Feature Complete (Next)
- [x] Web dashboard ✅ (React + TypeScript + TailwindCSS)
- [ ] Windows agent (Windows Update + Winget)
- [ ] Update installation execution
- [ ] Maintenance windows
- [ ] YUM/DNF scanner
- [ ] Rollback capability
- [ ] Real-time updates (WebSocket or polling)
- [ ] Docker deployment with proper networking
### Phase 3: AI Integration
- [ ] Natural language queries
- [ ] Intelligent scheduling
- [ ] Failure analysis
- [ ] AI chat sidebar in UI
### Phase 4: Enterprise Features
- [ ] Multi-tenancy
- [ ] RBAC
- [ ] SSO integration
- [ ] Compliance reporting
- [ ] Prometheus metrics
## Contributing
We welcome contributions! Areas that need help:
- **Windows agent** - Windows Update API integration
- **Package managers** - snap, flatpak, chocolatey, brew
- **Web dashboard** - React frontend
- **Documentation** - Installation guides, troubleshooting
- **Testing** - Unit tests, integration tests
## License
**AGPLv3** - This ensures:
- Modifications must stay open source
- No proprietary SaaS forks without contribution
- Commercial use allowed with attribution
- Forces cloud providers to contribute back
For commercial licensing options (if AGPL doesn't work for you), contact the project maintainers.
## Why "RedFlag"?
The project embraces a tongue-in-cheek communist theming:
- **Updates are the "means of production"** (they produce secure systems)
- **Commercial RMMs are "capitalist tools"** (expensive, SaaS-only)
- **RedFlag "seizes" control** back to the user (self-hosted, free)
But ultimately, it's a serious tool with a playful brand. The core mission is providing enterprise-grade update management to everyone, not just those who can afford expensive RMMs.
## Documentation
- 🏠 **Website**: Open `docs/index.html` in your browser for a fun intro!
- 📖 **Getting Started**: `docs/getting-started.html` - Complete setup guide
- 🔐 **Security Guide**: `SECURITY.md` - READ THIS BEFORE DEPLOYING
- 💬 **Discussions**: GitHub Discussions
- 🐛 **Bug Reports**: GitHub Issues
- 🚀 **Feature Requests**: GitHub Issues
## Acknowledgments
Built with:
- **Go** - Server & agent
- **Gin** - HTTP framework
- **PostgreSQL** - Database
- **Docker** - For development & deployment
- **React** (completed) - Web dashboard
Inspired by: ConnectWise Automate, Grafana, Wazuh, and the self-hosting community.
---
**Built with ❤️ for the self-hosting community**
🚩 **Seize the means of production!**