Add docs and project files - force for Culurien
This commit is contained in:
Fimeg
715
docs/3_BACKLOG/P5-002_Development-Workflow-Documentation.md
Normal file
715
docs/3_BACKLOG/P5-002_Development-Workflow-Documentation.md
Normal file
@@ -0,0 +1,715 @@
|
||||
# P5-002: Development Workflow Documentation
|
||||
|
||||
**Priority:** P5 (Process/Documentation)
|
||||
**Source Reference:** From analysis of DEVELOPMENT_TODOS.md and codebase development patterns
|
||||
**Date Identified:** 2025-11-12
|
||||
|
||||
## Problem Description
|
||||
|
||||
RedFlag lacks comprehensive development workflow documentation. Current development processes are undocumented, leading to inconsistent practices, onboarding difficulties, and potential quality issues. New developers lack guidance for contributing effectively.
|
||||
|
||||
## Impact
|
||||
|
||||
- **Onboarding Difficulty:** New contributors lack development guidance
|
||||
- **Inconsistent Processes:** Different developers use different approaches
|
||||
- **Quality Variations:** No standardized code review or testing procedures
|
||||
- **Knowledge Loss:** Development practices exist only in team members' heads
|
||||
- **Collaboration Issues:** No shared understanding of development workflows
|
||||
|
||||
## Current State Analysis
|
||||
|
||||
### Existing Documentation Gaps
|
||||
- No step-by-step development setup guide
|
||||
- No code contribution guidelines
|
||||
- No pull request process documentation
|
||||
- No testing requirements documentation
|
||||
- No release process guidelines
|
||||
- No debugging and troubleshooting guides
|
||||
|
||||
### Informal Practices Observed
|
||||
- Docker-based development environment
|
||||
- Multi-component architecture (server, agent, web)
|
||||
- Go backend with React frontend
|
||||
- PostgreSQL database with migrations
|
||||
- Cross-platform agent builds
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
Create comprehensive development workflow documentation:
|
||||
|
||||
### 1. Development Setup Guide
|
||||
```markdown
|
||||
# RedFlag Development Setup
|
||||
|
||||
## Prerequisites
|
||||
- Docker and Docker Compose
|
||||
- Go 1.21+ (for local development)
|
||||
- Node.js 18+ (for frontend development)
|
||||
- PostgreSQL client tools (optional)
|
||||
|
||||
## Quick Start (Docker Environment)
|
||||
```bash
|
||||
# Clone repository
|
||||
git clone https://github.com/redflag/redflag.git
|
||||
cd redflag
|
||||
|
||||
# Start development environment
|
||||
docker-compose up -d
|
||||
|
||||
# Initialize database
|
||||
docker-compose exec server ./redflag-server migrate
|
||||
|
||||
# Access services
|
||||
# Web UI: http://localhost:3000
|
||||
# API: http://localhost:8080
|
||||
# Database: localhost:5432
|
||||
```
|
||||
|
||||
## Local Development Setup
|
||||
```bash
|
||||
# Install dependencies
|
||||
make install-deps
|
||||
|
||||
# Setup database
|
||||
make setup-db
|
||||
|
||||
# Build components
|
||||
make build
|
||||
|
||||
# Run tests
|
||||
make test
|
||||
|
||||
# Start development servers
|
||||
make dev
|
||||
```
|
||||
|
||||
## Development Workflow
|
||||
1. **Create feature branch**: `git checkout -b feature/your-feature`
|
||||
2. **Make changes**: Edit code, add tests
|
||||
3. **Run tests**: `make test-all`
|
||||
4. **Lint code**: `make lint`
|
||||
5. **Commit changes**: Follow commit message format
|
||||
6. **Push and create PR**: Submit for code review
|
||||
```
|
||||
|
||||
### 2. Code Contribution Guidelines
|
||||
```markdown
|
||||
# Code Contribution Guidelines
|
||||
|
||||
## Coding Standards
|
||||
|
||||
### Go Code Style
|
||||
- Follow standard Go formatting (`gofmt`)
|
||||
- Use meaningful variable and function names
|
||||
- Add comments for public functions and complex logic
|
||||
- Handle errors explicitly
|
||||
- Use `golint` and `go vet` for static analysis
|
||||
|
||||
### TypeScript/React Code Style
|
||||
- Use Prettier for formatting
|
||||
- Follow TypeScript best practices
|
||||
- Use functional components with hooks
|
||||
- Add JSDoc comments for complex logic
|
||||
- Use ESLint for static analysis
|
||||
|
||||
### File Organization
|
||||
```
|
||||
RedFlag/
|
||||
├── aggregator-server/ # Go server
|
||||
│ ├── cmd/ # Main applications
|
||||
│ ├── internal/ # Internal packages
|
||||
│ │ ├── api/ # API handlers
|
||||
│ │ ├── database/ # Database operations
|
||||
│ │ ├── models/ # Data models
|
||||
│ │ └── services/ # Business logic
|
||||
│ └── migrations/ # Database migrations
|
||||
├── aggregator-agent/ # Go agent
|
||||
│ ├── cmd/ # Agent commands
|
||||
│ ├── internal/ # Internal packages
|
||||
│ │ ├── scanner/ # Update scanners
|
||||
│ │ ├── installer/ # Package installers
|
||||
│ │ └── orchestrator/ # Command orchestration
|
||||
│ └── pkg/ # Public packages
|
||||
├── aggregator-web/ # React frontend
|
||||
│ ├── src/
|
||||
│ │ ├── components/ # Reusable components
|
||||
│ │ ├── pages/ # Page components
|
||||
│ │ ├── lib/ # Utilities
|
||||
│ │ └── types/ # TypeScript types
|
||||
│ └── public/ # Static assets
|
||||
└── docs/ # Documentation
|
||||
```
|
||||
|
||||
## Testing Requirements
|
||||
|
||||
### Unit Tests
|
||||
- All new code must have unit tests
|
||||
- Test coverage should not decrease
|
||||
- Use table-driven tests for multiple scenarios
|
||||
- Mock external dependencies
|
||||
|
||||
### Integration Tests
|
||||
- API endpoints must have integration tests
|
||||
- Database operations must be tested
|
||||
- Agent-server communication should be tested
|
||||
- Use test database for integration tests
|
||||
|
||||
### Test Organization
|
||||
```go
|
||||
// Example test structure
|
||||
func TestFunctionName(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input InputType
|
||||
expected OutputType
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "Valid input",
|
||||
input: validInput,
|
||||
expected: expectedOutput,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "Invalid input",
|
||||
input: invalidInput,
|
||||
expected: OutputType{},
|
||||
wantErr: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
result, err := FunctionName(tt.input)
|
||||
if tt.wantErr {
|
||||
assert.Error(t, err)
|
||||
return
|
||||
}
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, tt.expected, result)
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Code Review Process
|
||||
|
||||
### Before Submitting PR
|
||||
1. **Self-review**: Review your own code changes
|
||||
2. **Testing**: Ensure all tests pass
|
||||
3. **Documentation**: Update relevant documentation
|
||||
4. **Style**: Run linting and formatting tools
|
||||
|
||||
### PR Requirements
|
||||
- Clear description of changes
|
||||
- Link to related issues
|
||||
- Tests for new functionality
|
||||
- Documentation updates
|
||||
- Screenshots for UI changes
|
||||
|
||||
### Review Guidelines
|
||||
- Review code logic and design
|
||||
- Check for potential security issues
|
||||
- Verify test coverage
|
||||
- Ensure documentation is accurate
|
||||
- Check for performance implications
|
||||
```
|
||||
|
||||
### 3. Pull Request Process
|
||||
```markdown
|
||||
# Pull Request Process
|
||||
|
||||
## PR Template
|
||||
```markdown
|
||||
## Description
|
||||
Brief description of changes made
|
||||
|
||||
## Type of Change
|
||||
- [ ] Bug fix
|
||||
- [ ] New feature
|
||||
- [ ] Breaking change
|
||||
- [ ] Documentation update
|
||||
|
||||
## Testing
|
||||
- [ ] Unit tests pass
|
||||
- [ ] Integration tests pass
|
||||
- [ ] Manual testing completed
|
||||
- [ ] Cross-platform testing (if applicable)
|
||||
|
||||
## Checklist
|
||||
- [ ] Code follows style guidelines
|
||||
- [ ] Self-review completed
|
||||
- [ ] Documentation updated
|
||||
- [ ] Tests added/updated
|
||||
- [ ] Database migrations included (if needed)
|
||||
- [ ] Security considerations addressed
|
||||
|
||||
## Related Issues
|
||||
Fixes #123
|
||||
Related to #456
|
||||
```
|
||||
|
||||
## Review Process
|
||||
1. **Automated Checks**
|
||||
- CI/CD pipeline runs tests
|
||||
- Code quality checks
|
||||
- Security scans
|
||||
|
||||
2. **Peer Review**
|
||||
- At least one developer approval required
|
||||
- Reviewer checks code quality and logic
|
||||
- Security review for sensitive changes
|
||||
|
||||
3. **Merge Process**
|
||||
- Address all reviewer comments
|
||||
- Ensure CI/CD checks pass
|
||||
- Merge with squash or rebase
|
||||
|
||||
## Release Process
|
||||
1. **Prepare Release**
|
||||
- Update version numbers
|
||||
- Update CHANGELOG.md
|
||||
- Tag release commit
|
||||
|
||||
2. **Build and Test**
|
||||
- Build all components
|
||||
- Run full test suite
|
||||
- Perform manual testing
|
||||
|
||||
3. **Deploy**
|
||||
- Deploy to staging environment
|
||||
- Perform smoke tests
|
||||
- Deploy to production
|
||||
```
|
||||
|
||||
### 4. Debugging and Troubleshooting Guide
|
||||
```markdown
|
||||
# Debugging and Troubleshooting Guide
|
||||
|
||||
## Common Development Issues
|
||||
|
||||
### Database Connection Issues
|
||||
```bash
|
||||
# Check database connectivity
|
||||
docker-compose exec server ping postgres
|
||||
|
||||
# Reset database
|
||||
docker-compose down -v
|
||||
docker-compose up -d
|
||||
docker-compose exec server ./redflag-server migrate
|
||||
```
|
||||
|
||||
### Agent Connection Problems
|
||||
```bash
|
||||
# Check agent logs
|
||||
sudo journalctl -u redflag-agent -f
|
||||
|
||||
# Test agent connectivity
|
||||
./redflag-agent --server http://localhost:8080 --check
|
||||
|
||||
# Verify agent registration
|
||||
curl -H "Authorization: Bearer $TOKEN" \
|
||||
http://localhost:8080/api/v1/agents
|
||||
```
|
||||
|
||||
### Build Issues
|
||||
```bash
|
||||
# Clean build
|
||||
make clean
|
||||
make build
|
||||
|
||||
# Check Go version
|
||||
go version
|
||||
|
||||
# Check dependencies
|
||||
go mod tidy
|
||||
go mod verify
|
||||
```
|
||||
|
||||
## Debugging Tools
|
||||
|
||||
### Server Debugging
|
||||
```bash
|
||||
# Enable debug logging
|
||||
export LOG_LEVEL=debug
|
||||
|
||||
# Run server with debugger
|
||||
dlv debug ./cmd/server
|
||||
|
||||
# Profile server performance
|
||||
go tool pprof http://localhost:8080/debug/pprof/profile
|
||||
```
|
||||
|
||||
### Agent Debugging
|
||||
```bash
|
||||
# Run agent in debug mode
|
||||
./redflag-agent --debug --server http://localhost:8080
|
||||
|
||||
# Test specific scanner
|
||||
./redflag-agent --scan-only dnf --debug
|
||||
|
||||
# Check agent configuration
|
||||
./redflag-agent --config-check
|
||||
```
|
||||
|
||||
### Frontend Debugging
|
||||
```bash
|
||||
# Start development server
|
||||
cd aggregator-web
|
||||
npm run dev
|
||||
|
||||
# Run tests with coverage
|
||||
npm run test:coverage
|
||||
|
||||
# Check for linting issues
|
||||
npm run lint
|
||||
```
|
||||
|
||||
## Performance Debugging
|
||||
|
||||
### Database Performance
|
||||
```sql
|
||||
-- Check slow queries
|
||||
SELECT query, mean_time, calls
|
||||
FROM pg_stat_statements
|
||||
ORDER BY mean_time DESC
|
||||
LIMIT 10;
|
||||
|
||||
-- Check database size
|
||||
SELECT pg_size_pretty(pg_database_size('aggregator'));
|
||||
|
||||
-- Check table sizes
|
||||
SELECT schemaname, tablename,
|
||||
pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as size
|
||||
FROM pg_tables
|
||||
WHERE schemaname = 'public'
|
||||
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;
|
||||
```
|
||||
|
||||
### Agent Performance
|
||||
```bash
|
||||
# Monitor agent resource usage
|
||||
top -p $(pgrep redflag-agent)
|
||||
|
||||
# Check agent memory usage
|
||||
ps aux | grep redflag-agent
|
||||
|
||||
# Profile agent performance
|
||||
go tool pprof http://localhost:8081/debug/pprof/profile
|
||||
```
|
||||
|
||||
## Log Analysis
|
||||
|
||||
### Server Logs
|
||||
```bash
|
||||
# View server logs
|
||||
docker-compose logs -f server
|
||||
|
||||
# Filter logs by level
|
||||
docker-compose logs server | grep ERROR
|
||||
|
||||
# Analyze log patterns
|
||||
docker-compose logs server | grep "rate limit"
|
||||
```
|
||||
|
||||
### Agent Logs
|
||||
```bash
|
||||
# View agent logs
|
||||
sudo journalctl -u redflag-agent -f
|
||||
|
||||
# Filter by log level
|
||||
sudo journalctl -u redflag-agent | grep ERROR
|
||||
|
||||
# Check specific time period
|
||||
sudo journalctl -u redflag-agent --since "2025-01-01" --until "2025-01-02"
|
||||
```
|
||||
|
||||
## Environment-Specific Issues
|
||||
|
||||
### Development Environment
|
||||
```bash
|
||||
# Reset development environment
|
||||
make dev-reset
|
||||
|
||||
# Check service status
|
||||
docker-compose ps
|
||||
|
||||
# Access development database
|
||||
docker-compose exec postgres psql -U aggregator -d aggregator
|
||||
```
|
||||
|
||||
### Production Environment
|
||||
```bash
|
||||
# Check service health
|
||||
curl -f http://localhost:8080/health || echo "Health check failed"
|
||||
|
||||
# Monitor system resources
|
||||
htop
|
||||
iostat -x 1
|
||||
|
||||
# Check disk space
|
||||
df -h
|
||||
```
|
||||
```
|
||||
|
||||
### 5. Release and Deployment Guide
|
||||
```markdown
|
||||
# Release and Deployment Guide
|
||||
|
||||
## Version Management
|
||||
|
||||
### Semantic Versioning
|
||||
- Major version: Breaking changes
|
||||
- Minor version: New features (backward compatible)
|
||||
- Patch version: Bug fixes (backward compatible)
|
||||
|
||||
### Version Number Format
|
||||
```
|
||||
vX.Y.Z
|
||||
X = Major version
|
||||
Y = Minor version
|
||||
Z = Patch version
|
||||
```
|
||||
|
||||
### Version Bump Checklist
|
||||
1. **Update version numbers**
|
||||
- `aggregator-server/internal/version/version.go`
|
||||
- `aggregator-agent/cmd/agent/version.go`
|
||||
- `aggregator-web/package.json`
|
||||
|
||||
2. **Update CHANGELOG.md**
|
||||
- Add new version section
|
||||
- Document all changes
|
||||
- Credit contributors
|
||||
|
||||
3. **Tag release**
|
||||
```bash
|
||||
git tag -a v0.2.0 -m "Release v0.2.0"
|
||||
git push origin v0.2.0
|
||||
```
|
||||
|
||||
## Build Process
|
||||
|
||||
### Automated Builds
|
||||
```bash
|
||||
# Build all components
|
||||
make build-all
|
||||
|
||||
# Build specific component
|
||||
make build-server
|
||||
make build-agent
|
||||
make build-web
|
||||
|
||||
# Build for all platforms
|
||||
make build-cross-platform
|
||||
```
|
||||
|
||||
### Release Builds
|
||||
```bash
|
||||
# Create release artifacts
|
||||
make release
|
||||
|
||||
# Verify builds
|
||||
make verify-release
|
||||
```
|
||||
|
||||
## Deployment Process
|
||||
|
||||
### Staging Deployment
|
||||
1. **Prepare staging environment**
|
||||
```bash
|
||||
# Deploy to staging
|
||||
make deploy-staging
|
||||
```
|
||||
|
||||
2. **Run smoke tests**
|
||||
```bash
|
||||
make test-staging
|
||||
```
|
||||
|
||||
3. **Manual verification**
|
||||
- Check web UI functionality
|
||||
- Verify API endpoints
|
||||
- Test agent registration
|
||||
|
||||
### Production Deployment
|
||||
1. **Pre-deployment checklist**
|
||||
- [ ] All tests passing
|
||||
- [ ] Documentation updated
|
||||
- [ ] Security review completed
|
||||
- [ ] Performance tests passed
|
||||
- [ ] Backup created
|
||||
|
||||
2. **Deploy to production**
|
||||
```bash
|
||||
# Deploy to production
|
||||
make deploy-production
|
||||
```
|
||||
|
||||
3. **Post-deployment verification**
|
||||
```bash
|
||||
# Health checks
|
||||
make verify-production
|
||||
```
|
||||
|
||||
## Rollback Procedures
|
||||
|
||||
### Quick Rollback
|
||||
```bash
|
||||
# Rollback to previous version
|
||||
make rollback-to v0.1.23
|
||||
```
|
||||
|
||||
### Full Rollback
|
||||
1. **Stop current deployment**
|
||||
2. **Restore from backup**
|
||||
3. **Deploy previous version**
|
||||
4. **Verify functionality**
|
||||
5. **Communicate rollback**
|
||||
|
||||
## Monitoring After Deployment
|
||||
|
||||
### Health Checks
|
||||
```bash
|
||||
# Check service health
|
||||
curl -f http://localhost:8080/health
|
||||
|
||||
# Check database connectivity
|
||||
docker-compose exec server ./redflag-server health-check
|
||||
|
||||
# Monitor agent check-ins
|
||||
curl -H "Authorization: Bearer $TOKEN" \
|
||||
http://localhost:8080/api/v1/agents | jq '. | length'
|
||||
```
|
||||
|
||||
### Performance Monitoring
|
||||
```bash
|
||||
# Monitor response times
|
||||
curl -w "@curl-format.txt" http://localhost:8080/api/v1/agents
|
||||
|
||||
# Check error rates
|
||||
docker-compose logs server | grep ERROR | wc -l
|
||||
```
|
||||
|
||||
## Communication
|
||||
|
||||
### Release Announcement Template
|
||||
```markdown
|
||||
## Release v0.2.0
|
||||
|
||||
### New Features
|
||||
- Feature 1 description
|
||||
- Feature 2 description
|
||||
|
||||
### Bug Fixes
|
||||
- Bug fix 1 description
|
||||
- Bug fix 2 description
|
||||
|
||||
### Breaking Changes
|
||||
- Breaking change 1 description
|
||||
|
||||
### Upgrade Instructions
|
||||
1. Backup your installation
|
||||
2. Follow upgrade guide
|
||||
3. Verify functionality
|
||||
|
||||
### Known Issues
|
||||
- Any known issues or limitations
|
||||
```
|
||||
```
|
||||
|
||||
## Definition of Done
|
||||
|
||||
- [ ] Development setup guide created and tested
|
||||
- [ ] Code contribution guidelines documented
|
||||
- [ ] Pull request process defined
|
||||
- [ ] Debugging guide created
|
||||
- [ ] Release and deployment guide documented
|
||||
- [ ] Developer onboarding checklist created
|
||||
- [ ] Code review checklist developed
|
||||
- [ ] Makefile targets for all documented processes
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Documentation Structure
|
||||
```
|
||||
docs/
|
||||
├── development/
|
||||
│ ├── setup.md
|
||||
│ ├── contributing.md
|
||||
│ ├── pull-request-process.md
|
||||
│ ├── debugging.md
|
||||
│ ├── release-process.md
|
||||
│ └── onboarding.md
|
||||
├── templates/
|
||||
│ ├── pull-request-template.md
|
||||
│ ├── release-announcement.md
|
||||
│ └── bug-report.md
|
||||
└── scripts/
|
||||
├── setup-dev.sh
|
||||
├── test-all.sh
|
||||
└── release.sh
|
||||
```
|
||||
|
||||
### Makefile Targets
|
||||
```makefile
|
||||
.PHONY: install-deps setup-db build test lint dev clean release
|
||||
|
||||
install-deps:
|
||||
# Install development dependencies
|
||||
|
||||
setup-db:
|
||||
# Setup development database
|
||||
|
||||
build:
|
||||
# Build all components
|
||||
|
||||
test:
|
||||
# Run all tests
|
||||
|
||||
lint:
|
||||
# Run code quality checks
|
||||
|
||||
dev:
|
||||
# Start development environment
|
||||
|
||||
clean:
|
||||
# Clean build artifacts
|
||||
|
||||
release:
|
||||
# Create release artifacts
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Development environment standards established
|
||||
- CI/CD pipeline in place
|
||||
- Code review process defined
|
||||
- Documentation templates created
|
||||
- Team agreement on processes
|
||||
|
||||
## Effort Estimate
|
||||
|
||||
**Complexity:** Medium
|
||||
**Effort:** 1-2 weeks
|
||||
- Week 1: Create core development documentation
|
||||
- Week 2: Review, test, and refine processes
|
||||
|
||||
## Success Metrics
|
||||
|
||||
- New developer onboarding time reduced
|
||||
- Consistent code quality across contributions
|
||||
- Faster PR review and merge process
|
||||
- Fewer deployment issues
|
||||
- Better team collaboration
|
||||
- Improved development productivity
|
||||
|
||||
## Monitoring
|
||||
|
||||
Track these metrics after implementation:
|
||||
- Developer onboarding time
|
||||
- Code review turnaround time
|
||||
- PR merge time
|
||||
- Deployment success rate
|
||||
- Developer satisfaction surveys
|
||||
- Documentation usage analytics
|
||||
Reference in New Issue
Block a user