Redflag/docs/3_BACKLOG/P5-002_Development-Workflow-Documentation.md

# P5-002: Development Workflow Documentation

**Priority:** P5 (Process/Documentation)
**Source Reference:** From analysis of DEVELOPMENT_TODOS.md and codebase development patterns
**Date Identified:** 2025-11-12

## Problem Description

RedFlag lacks comprehensive development workflow documentation. Current development processes are undocumented, leading to inconsistent practices, onboarding difficulties, and potential quality issues. New developers lack guidance for contributing effectively.

## Impact

- **Onboarding Difficulty:** New contributors lack development guidance
- **Inconsistent Processes:** Different developers use different approaches
- **Quality Variations:** No standardized code review or testing procedures
- **Knowledge Loss:** Development practices exist only in team members' heads
- **Collaboration Issues:** No shared understanding of development workflows

## Current State Analysis

### Existing Documentation Gaps
- No step-by-step development setup guide
- No code contribution guidelines
- No pull request process documentation
- No testing requirements documentation
- No release process guidelines
- No debugging and troubleshooting guides

### Informal Practices Observed
- Docker-based development environment
- Multi-component architecture (server, agent, web)
- Go backend with React frontend
- PostgreSQL database with migrations
- Cross-platform agent builds

## Proposed Solution

Create comprehensive development workflow documentation:

### 1. Development Setup Guide
```markdown
# RedFlag Development Setup

## Prerequisites
- Docker and Docker Compose
- Go 1.21+ (for local development)
- Node.js 18+ (for frontend development)
- PostgreSQL client tools (optional)

## Quick Start (Docker Environment)
```bash
# Clone repository
git clone https://github.com/redflag/redflag.git
cd redflag

# Start development environment
docker-compose up -d

# Initialize database
docker-compose exec server ./redflag-server migrate

# Access services
# Web UI: http://localhost:3000
# API: http://localhost:8080
# Database: localhost:5432
```

## Local Development Setup
```bash
# Install dependencies
make install-deps

# Setup database
make setup-db

# Build components
make build

# Run tests
make test

# Start development servers
make dev
```

## Development Workflow
1. **Create feature branch**: `git checkout -b feature/your-feature`
2. **Make changes**: Edit code, add tests
3. **Run tests**: `make test-all`
4. **Lint code**: `make lint`
5. **Commit changes**: Follow commit message format
6. **Push and create PR**: Submit for code review
```

### 2. Code Contribution Guidelines
```markdown
# Code Contribution Guidelines

## Coding Standards

### Go Code Style
- Follow standard Go formatting (`gofmt`)
- Use meaningful variable and function names
- Add comments for public functions and complex logic
- Handle errors explicitly
- Use `golint` and `go vet` for static analysis

### TypeScript/React Code Style
- Use Prettier for formatting
- Follow TypeScript best practices
- Use functional components with hooks
- Add JSDoc comments for complex logic
- Use ESLint for static analysis

### File Organization
```
RedFlag/
├── aggregator-server/          # Go server
│   ├── cmd/                    # Main applications
│   ├── internal/               # Internal packages
│   │   ├── api/               # API handlers
│   │   ├── database/          # Database operations
│   │   ├── models/            # Data models
│   │   └── services/          # Business logic
│   └── migrations/             # Database migrations
├── aggregator-agent/           # Go agent
│   ├── cmd/                   # Agent commands
│   ├── internal/              # Internal packages
│   │   ├── scanner/           # Update scanners
│   │   ├── installer/         # Package installers
│   │   └── orchestrator/      # Command orchestration
│   └── pkg/                   # Public packages
├── aggregator-web/             # React frontend
│   ├── src/
│   │   ├── components/        # Reusable components
│   │   ├── pages/             # Page components
│   │   ├── lib/               # Utilities
│   │   └── types/             # TypeScript types
│   └── public/                # Static assets
└── docs/                      # Documentation
```

## Testing Requirements

### Unit Tests
- All new code must have unit tests
- Test coverage should not decrease
- Use table-driven tests for multiple scenarios
- Mock external dependencies

### Integration Tests
- API endpoints must have integration tests
- Database operations must be tested
- Agent-server communication should be tested
- Use test database for integration tests

### Test Organization
```go
// Example test structure
func TestFunctionName(t *testing.T) {
    tests := []struct {
        name     string
        input    InputType
        expected OutputType
        wantErr  bool
    }{
        {
            name:     "Valid input",
            input:    validInput,
            expected: expectedOutput,
            wantErr:  false,
        },
        {
            name:     "Invalid input",
            input:    invalidInput,
            expected: OutputType{},
            wantErr:  true,
        },
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            result, err := FunctionName(tt.input)
            if tt.wantErr {
                assert.Error(t, err)
                return
            }
            assert.NoError(t, err)
            assert.Equal(t, tt.expected, result)
        })
    }
}
```

## Code Review Process

### Before Submitting PR
1. **Self-review**: Review your own code changes
2. **Testing**: Ensure all tests pass
3. **Documentation**: Update relevant documentation
4. **Style**: Run linting and formatting tools

### PR Requirements
- Clear description of changes
- Link to related issues
- Tests for new functionality
- Documentation updates
- Screenshots for UI changes

### Review Guidelines
- Review code logic and design
- Check for potential security issues
- Verify test coverage
- Ensure documentation is accurate
- Check for performance implications
```

### 3. Pull Request Process
```markdown
# Pull Request Process

## PR Template
```markdown
## Description
Brief description of changes made

## Type of Change
- [ ] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update

## Testing
- [ ] Unit tests pass
- [ ] Integration tests pass
- [ ] Manual testing completed
- [ ] Cross-platform testing (if applicable)

## Checklist
- [ ] Code follows style guidelines
- [ ] Self-review completed
- [ ] Documentation updated
- [ ] Tests added/updated
- [ ] Database migrations included (if needed)
- [ ] Security considerations addressed

## Related Issues
Fixes #123
Related to #456
```

## Review Process
1. **Automated Checks**
   - CI/CD pipeline runs tests
   - Code quality checks
   - Security scans

2. **Peer Review**
   - At least one developer approval required
   - Reviewer checks code quality and logic
   - Security review for sensitive changes

3. **Merge Process**
   - Address all reviewer comments
   - Ensure CI/CD checks pass
   - Merge with squash or rebase

## Release Process
1. **Prepare Release**
   - Update version numbers
   - Update CHANGELOG.md
   - Tag release commit

2. **Build and Test**
   - Build all components
   - Run full test suite
   - Perform manual testing

3. **Deploy**
   - Deploy to staging environment
   - Perform smoke tests
   - Deploy to production
```

### 4. Debugging and Troubleshooting Guide
```markdown
# Debugging and Troubleshooting Guide

## Common Development Issues

### Database Connection Issues
```bash
# Check database connectivity
docker-compose exec server ping postgres

# Reset database
docker-compose down -v
docker-compose up -d
docker-compose exec server ./redflag-server migrate
```

### Agent Connection Problems
```bash
# Check agent logs
sudo journalctl -u redflag-agent -f

# Test agent connectivity
./redflag-agent --server http://localhost:8080 --check

# Verify agent registration
curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:8080/api/v1/agents
```

### Build Issues
```bash
# Clean build
make clean
make build

# Check Go version
go version

# Check dependencies
go mod tidy
go mod verify
```

## Debugging Tools

### Server Debugging
```bash
# Enable debug logging
export LOG_LEVEL=debug

# Run server with debugger
dlv debug ./cmd/server

# Profile server performance
go tool pprof http://localhost:8080/debug/pprof/profile
```

### Agent Debugging
```bash
# Run agent in debug mode
./redflag-agent --debug --server http://localhost:8080

# Test specific scanner
./redflag-agent --scan-only dnf --debug

# Check agent configuration
./redflag-agent --config-check
```

### Frontend Debugging
```bash
# Start development server
cd aggregator-web
npm run dev

# Run tests with coverage
npm run test:coverage

# Check for linting issues
npm run lint
```

## Performance Debugging

### Database Performance
```sql
-- Check slow queries
SELECT query, mean_time, calls
FROM pg_stat_statements
ORDER BY mean_time DESC
LIMIT 10;

-- Check database size
SELECT pg_size_pretty(pg_database_size('aggregator'));

-- Check table sizes
SELECT schemaname, tablename,
       pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;
```

### Agent Performance
```bash
# Monitor agent resource usage
top -p $(pgrep redflag-agent)

# Check agent memory usage
ps aux | grep redflag-agent

# Profile agent performance
go tool pprof http://localhost:8081/debug/pprof/profile
```

## Log Analysis

### Server Logs
```bash
# View server logs
docker-compose logs -f server

# Filter logs by level
docker-compose logs server | grep ERROR

# Analyze log patterns
docker-compose logs server | grep "rate limit"
```

### Agent Logs
```bash
# View agent logs
sudo journalctl -u redflag-agent -f

# Filter by log level
sudo journalctl -u redflag-agent | grep ERROR

# Check specific time period
sudo journalctl -u redflag-agent --since "2025-01-01" --until "2025-01-02"
```

## Environment-Specific Issues

### Development Environment
```bash
# Reset development environment
make dev-reset

# Check service status
docker-compose ps

# Access development database
docker-compose exec postgres psql -U aggregator -d aggregator
```

### Production Environment
```bash
# Check service health
curl -f http://localhost:8080/health || echo "Health check failed"

# Monitor system resources
htop
iostat -x 1

# Check disk space
df -h
```
```

### 5. Release and Deployment Guide
```markdown
# Release and Deployment Guide

## Version Management

### Semantic Versioning
- Major version: Breaking changes
- Minor version: New features (backward compatible)
- Patch version: Bug fixes (backward compatible)

### Version Number Format
```
vX.Y.Z
X = Major version
Y = Minor version
Z = Patch version
```

### Version Bump Checklist
1. **Update version numbers**
   - `aggregator-server/internal/version/version.go`
   - `aggregator-agent/cmd/agent/version.go`
   - `aggregator-web/package.json`

2. **Update CHANGELOG.md**
   - Add new version section
   - Document all changes
   - Credit contributors

3. **Tag release**
   ```bash
   git tag -a v0.2.0 -m "Release v0.2.0"
   git push origin v0.2.0
   ```

## Build Process

### Automated Builds
```bash
# Build all components
make build-all

# Build specific component
make build-server
make build-agent
make build-web

# Build for all platforms
make build-cross-platform
```

### Release Builds
```bash
# Create release artifacts
make release

# Verify builds
make verify-release
```

## Deployment Process

### Staging Deployment
1. **Prepare staging environment**
   ```bash
   # Deploy to staging
   make deploy-staging
   ```

2. **Run smoke tests**
   ```bash
   make test-staging
   ```

3. **Manual verification**
   - Check web UI functionality
   - Verify API endpoints
   - Test agent registration

### Production Deployment
1. **Pre-deployment checklist**
   - [ ] All tests passing
   - [ ] Documentation updated
   - [ ] Security review completed
   - [ ] Performance tests passed
   - [ ] Backup created

2. **Deploy to production**
   ```bash
   # Deploy to production
   make deploy-production
   ```

3. **Post-deployment verification**
   ```bash
   # Health checks
   make verify-production
   ```

## Rollback Procedures

### Quick Rollback
```bash
# Rollback to previous version
make rollback-to v0.1.23
```

### Full Rollback
1. **Stop current deployment**
2. **Restore from backup**
3. **Deploy previous version**
4. **Verify functionality**
5. **Communicate rollback**

## Monitoring After Deployment

### Health Checks
```bash
# Check service health
curl -f http://localhost:8080/health

# Check database connectivity
docker-compose exec server ./redflag-server health-check

# Monitor agent check-ins
curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:8080/api/v1/agents | jq '. | length'
```

### Performance Monitoring
```bash
# Monitor response times
curl -w "@curl-format.txt" http://localhost:8080/api/v1/agents

# Check error rates
docker-compose logs server | grep ERROR | wc -l
```

## Communication

### Release Announcement Template
```markdown
## Release v0.2.0

### New Features
- Feature 1 description
- Feature 2 description

### Bug Fixes
- Bug fix 1 description
- Bug fix 2 description

### Breaking Changes
- Breaking change 1 description

### Upgrade Instructions
1. Backup your installation
2. Follow upgrade guide
3. Verify functionality

### Known Issues
- Any known issues or limitations
```
```

## Definition of Done

- [ ] Development setup guide created and tested
- [ ] Code contribution guidelines documented
- [ ] Pull request process defined
- [ ] Debugging guide created
- [ ] Release and deployment guide documented
- [ ] Developer onboarding checklist created
- [ ] Code review checklist developed
- [ ] Makefile targets for all documented processes

## Implementation Details

### Documentation Structure
```
docs/
├── development/
│   ├── setup.md
│   ├── contributing.md
│   ├── pull-request-process.md
│   ├── debugging.md
│   ├── release-process.md
│   └── onboarding.md
├── templates/
│   ├── pull-request-template.md
│   ├── release-announcement.md
│   └── bug-report.md
└── scripts/
    ├── setup-dev.sh
    ├── test-all.sh
    └── release.sh
```

### Makefile Targets
```makefile
.PHONY: install-deps setup-db build test lint dev clean release

install-deps:
	# Install development dependencies

setup-db:
	# Setup development database

build:
	# Build all components

test:
	# Run all tests

lint:
	# Run code quality checks

dev:
	# Start development environment

clean:
	# Clean build artifacts

release:
	# Create release artifacts
```

## Prerequisites

- Development environment standards established
- CI/CD pipeline in place
- Code review process defined
- Documentation templates created
- Team agreement on processes

## Effort Estimate

**Complexity:** Medium
**Effort:** 1-2 weeks
- Week 1: Create core development documentation
- Week 2: Review, test, and refine processes

## Success Metrics

- New developer onboarding time reduced
- Consistent code quality across contributions
- Faster PR review and merge process
- Fewer deployment issues
- Better team collaboration
- Improved development productivity

## Monitoring

Track these metrics after implementation:
- Developer onboarding time
- Code review turnaround time
- PR merge time
- Deployment success rate
- Developer satisfaction surveys
- Documentation usage analytics