community-ade/docs/redis-schema.md

# Community ADE Approval System - Redis Schema

## Overview

This document defines all Redis key patterns used by the Approval System. All keys use the `ade:` prefix for namespacing.

---

## Key Naming Convention

```
ade:{category}:{subcategory}:{identifier}:{attribute}
```

| Segment | Description | Examples |
|---------|-------------|----------|
| `ade` | Global namespace prefix | - |
| `category` | High-level component | `lock`, `approval`, `task`, `session` |
| `subcategory` | Specific entity type | `task`, `resource`, `agent`, `user` |
| `identifier` | Unique entity ID | UUID or slug |
| `attribute` | Property/attribute | `data`, `queue`, `index` |

---

## Lock Keys

### Primary Lock Storage

#### Task Lock (Exclusive)
```
Key: ade:lock:task:{task_id}
Type: Hash
TTL: 30 seconds (renewable)

Fields:
  holder_agent_id    (string)    UUID of agent holding the lock
  acquired_at        (string)    ISO 8601 timestamp
  expires_at         (string)    ISO 8601 timestamp
  purpose            (string)    Human-readable purpose
  heartbeat_count    (integer)   Number of heartbeats received
  queue_length       (integer)   Number of waiters in queue

Example:
  HSET ade:lock:task:550e8400-e29b-41d4-a716-446655440000 \
    holder_agent_id agent-123 \
    acquired_at "2026-03-18T15:30:00Z" \
    expires_at "2026-03-18T15:30:30Z" \
    purpose "Applying database migration"
```

#### Resource Lock (Shared/Exclusive)
```
Key: ade:lock:resource:{resource_type}:{resource_id}
Type: Hash
TTL: 30 seconds (renewable)

Fields:
  mode               (string)    "exclusive" or "shared"
  holders            (JSON)      Array of {agent_id, acquired_at}
  exclusive_holder   (string)    Agent ID (if exclusive mode)
  acquired_at        (string)    ISO 8601 timestamp
  expires_at         (string)    ISO 8601 timestamp

Example:
  HSET ade:lock:resource:database:prod-db-01 \
    mode "exclusive" \
    exclusive_holder agent-456 \
    acquired_at "2026-03-18T15:30:00Z" \
    expires_at "2026-03-18T15:30:30Z"
```

#### Agent Capacity Lock
```
Key: ade:lock:agent:{agent_id}:capacity
Type: Hash
TTL: None (persistent, cleaned up on agent deregistration)

Fields:
  max_tasks          (integer)   Maximum concurrent tasks
  active_tasks       (integer)   Currently executing tasks
  queued_tasks       (integer)   Tasks waiting for capacity
  last_heartbeat     (string)    ISO 8601 timestamp
  status             (string)    "active", "draining", "offline"

Example:
  HSET ade:lock:agent:agent-123:capacity \
    max_tasks 10 \
    active_tasks 3 \
    queued_tasks 1 \
    last_heartbeat "2026-03-18T15:30:00Z" \
    status "active"
```

### Lock Queue Keys

#### Lock Wait Queue (Ordered list of waiting agents)
```
Key: ade:lock:task:{task_id}:queue
Type: Sorted Set (ZSET)
TTL: 5 minutes (cleaned up when lock released)

Score: Unix timestamp (millisecond precision for FIFO ordering)
Value: JSON object

Value Format:
  {
    "agent_id": "agent-uuid",
    "mode": "exclusive",
    "priority": 100,
    "requested_at": "2026-03-18T15:30:00Z",
    "max_wait_seconds": 60
  }

Example:
  ZADD ade:lock:task:550e8400-e29b-41d4-a716-446655440000:queue \
    1710775800000 '{"agent_id":"agent-789","mode":"exclusive","priority":100,...}'
```

#### Lock Notification Channel
```
Key: ade:lock:task:{task_id}:channel
Type: Pub/Sub Channel

Events:
  "acquired:{agent_id}"      - Lock acquired
  "released:{agent_id}"      - Lock released
  "expired"                  - Lock expired
  "queued:{agent_id}"        - Agent added to queue
  "promoted:{agent_id}"      - Agent promoted from queue
```

### Lock Index Keys

#### Active Locks by Agent (Reverse index)
```
Key: ade:lock:index:agent:{agent_id}
Type: Set
TTL: Matches individual lock TTLs

Members: Lock key references
  ade:lock:task:{task_id}
  ade:lock:resource:{type}:{id}

Purpose: Quick lookup of all locks held by an agent
```

#### Active Locks by Resource Type
```
Key: ade:lock:index:resource:{resource_type}
Type: Set
TTL: Matches individual lock TTLs

Members: Resource lock keys
  ade:lock:resource:database:prod-db-01
  ade:lock:resource:service:api-gateway
```

#### Global Lock Registry
```
Key: ade:lock:registry
Type: Sorted Set
TTL: None

Score: Expiration timestamp
Value: Lock key

Purpose: Background cleanup of expired locks
Example:
  ZADD ade:lock:registry 1710775830 "ade:lock:task:550e8400-..."
```

### Deadlock Detection Keys

#### Wait-For Graph Edge
```
Key: ade:lock:waitfor:{agent_id}
Type: Set
TTL: 5 minutes

Members: Lock keys the agent is waiting for
  ade:lock:task:{task_id}
  ade:lock:resource:{type}:{id}

Purpose: Build wait-for graph for deadlock detection
```

#### Deadlock Detection Timestamp
```
Key: ade:lock:deadlock:check:{agent_id}
Type: String
TTL: 30 seconds

Value: ISO 8601 timestamp of last deadlock check

Purpose: Rate limit deadlock detection attempts
```

---

## Approval Keys

### Approval Request Keys

#### Approval Request Data
```
Key: ade:approval:request:{approval_id}
Type: Hash
TTL: 30 days (archived after completion)

Fields:
  task_id            (string)    UUID of associated task
  reviewer_id        (string)    User ID of assigned reviewer
  reviewer_name      (string)    Display name
  status             (string)    "PENDING", "APPROVED", "REJECTED", "DELEGATED"
  priority           (string)    "LOW", "NORMAL", "HIGH", "URGENT"
  delegated_to       (string)    User ID (if delegated)
  delegation_chain   (JSON)      Array of user IDs in delegation chain
  created_at         (string)    ISO 8601 timestamp
  due_at             (string)    ISO 8601 timestamp
  responded_at       (string)    ISO 8601 timestamp
  response_action    (string)    "approve", "reject", "request_changes"
  response_reason    (string)    Free text explanation
  reviewed_by        (string)    Final responding user ID

Example:
  HSET ade:approval:request:app-123 \
    task_id "task-456" \
    reviewer_id "user-789" \
    status "PENDING" \
    priority "HIGH" \
    created_at "2026-03-18T15:30:00Z" \
    due_at "2026-03-20T15:30:00Z"
```

### Approval Queue Keys

#### User Approval Queue (Pending approvals for a user)
```
Key: ade:approval:queue:user:{user_id}
Type: Sorted Set
TTL: None (entries expire based on approval TTL)

Score: Priority score (higher = more urgent)
  Calculated as: (risk_score * 10) + priority_bonus
  priority_bonus: URGENT=1000, HIGH=500, NORMAL=100, LOW=0

Value: approval_id

Example:
  ZADD ade:approval:queue:user:user-789 850 "app-123"
  ZADD ade:approval:queue:user:user-789 450 "app-124"
```

#### Task Approval Index (All approvals for a task)
```
Key: ade:approval:index:task:{task_id}
Type: Set
TTL: Matches approval data TTL

Members: approval_ids
  app-123
  app-124
  app-125
```

#### Global Approval Queue (All pending approvals)
```
Key: ade:approval:queue:global
Type: Sorted Set
TTL: None

Score: Due timestamp (Unix seconds)
Value: approval_id

Purpose: Background worker for escalation/timeout handling
```

### Approval Statistics Keys

#### User Approval Stats
```
Key: ade:approval:stats:user:{user_id}
Type: Hash
TTL: None (rolling window)

Fields:
  pending_count      (integer)   Current pending approvals
  approved_today     (integer)   Approvals given today
  rejected_today     (integer)   Rejections given today
  avg_response_time  (float)     Average response time in seconds
  last_action_at     (string)    ISO 8601 timestamp

Note: Daily counters reset at midnight UTC via background job
```

#### Task Approval Stats
```
Key: ade:approval:stats:task:{task_id}
Type: Hash
TTL: 30 days

Fields:
  required_count     (integer)   Required approvals
  approved_count     (integer)   Current approvals
  rejected_count     (integer)   Current rejections
  pending_count      (integer)   Awaiting response
  quorum_reached     (boolean)   Whether minimum approvals met
```

### Delegation Keys

#### User Delegation Policy
```
Key: ade:approval:delegation:{user_id}:{policy_id}
Type: Hash
TTL: Based on policy expiration

Fields:
  owner_id           (string)    Policy owner
  delegate_to        (string)    Delegated reviewer
  conditions         (JSON)      Matching conditions
  cascade            (boolean)   Allow further delegation
  active             (boolean)   Policy enabled/disabled
  created_at         (string)    ISO 8601 timestamp
  expires_at         (string)    ISO 8601 timestamp

Example:
  HSET ade:approval:delegation:user-123:policy-456 \
    owner_id "user-123" \
    delegate_to "user-789" \
    conditions '{"task_types":["infrastructure"],"risk_above":50}' \
    cascade "true" \
    active "true"
```

#### Delegation Policy Index
```
Key: ade:approval:delegation:index:{user_id}
Type: Set
TTL: None

Members: policy_ids for the user
  policy-456
  policy-789
```

---

## Task Keys

### Task Data Keys

#### Task State
```
Key: ade:task:{task_id}:state
Type: String
TTL: 90 days

Value: Current state
  DRAFT, SUBMITTED, REVIEWING, APPROVED, APPLYING, COMPLETED, REJECTED, CANCELLED

Example:
  SET ade:task:task-123:state "REVIEWING"
```

#### Task Data (Full object)
```
Key: ade:task:{task_id}:data
Type: JSON (RedisJSON module) or String (serialized JSON)
TTL: 90 days

Value: Complete task object including config, metadata, execution results

Note: For Redis versions without JSON module, store as serialized string
```

#### Task Configuration (Immutable)
```
Key: ade:task:{task_id}:config
Type: Hash
TTL: 90 days

Fields:
  type               (string)    Task type
  version            (string)    Config version
  description        (string)    Human-readable description
  parameters         (JSON)      Task parameters
  resources          (JSON)      Array of resource references
  rollback_strategy  (string)    "automatic", "manual", "none"
  timeout_seconds    (integer)   Execution timeout
  priority           (integer)   0-100 priority score
```

#### Task Metadata
```
Key: ade:task:{task_id}:metadata
Type: Hash
TTL: 90 days

Fields:
  author_id          (string)    Creating user
  author_name        (string)    Display name
  team               (string)    Team/organization
  ticket_ref         (string)    External ticket reference
  tags               (JSON)      Array of string tags
  created_at         (string)    ISO 8601 timestamp
  updated_at         (string)    ISO 8601 timestamp
  submitted_at       (string)    ISO 8601 timestamp
  approved_at        (string)    ISO 8601 timestamp
  applying_at        (string)    ISO 8601 timestamp
  completed_at       (string)    ISO 8601 timestamp
```

### Task State Index Keys

#### Tasks by State
```
Key: ade:task:index:state:{state}
Type: Sorted Set
TTL: None (members removed on state change)

Score: created_at timestamp (Unix seconds)
Value: task_id

Example Keys:
  ade:task:index:state:DRAFT
  ade:task:index:state:REVIEWING
  ade:task:index:state:APPROVED
```

#### Tasks by Author
```
Key: ade:task:index:author:{user_id}
Type: Sorted Set
TTL: 90 days

Score: created_at timestamp
Value: task_id
```

#### Tasks by Resource
```
Key: ade:task:index:resource:{resource_type}:{resource_id}
Type: Sorted Set
TTL: 90 days

Score: created_at timestamp
Value: task_id

Example:
  ade:task:index:resource:database:prod-db-01
```

#### Tasks by Tag
```
Key: ade:task:index:tag:{tag_name}
Type: Sorted Set
TTL: 90 days

Score: created_at timestamp
Value: task_id
```

### Task Execution Keys

#### Task Execution Status
```
Key: ade:task:{task_id}:execution
Type: Hash
TTL: 90 days

Fields:
  started_at         (string)    ISO 8601 timestamp
  completed_at       (string)    ISO 8601 timestamp
  agent_id           (string)    Executing agent
  result             (string)    "success", "failure", "timeout", "cancelled"
  output             (string)    Execution output (truncated)
  output_key         (string)    Key to full output in S3/blob storage
  error              (string)    Error message (if failed)
  error_details      (JSON)      Structured error information
  retry_count        (integer)   Number of retry attempts
```

#### Task Preview Results
```
Key: ade:task:{task_id}:preview
Type: JSON/String
TTL: 7 days

Value: Preview result object with changes, warnings, errors
```

#### Task Risk Assessment
```
Key: ade:task:{task_id}:risk
Type: Hash
TTL: 90 days

Fields:
  score              (integer)   0-100 risk score
  level              (string)    "LOW", "MEDIUM", "HIGH", "CRITICAL"
  factors            (JSON)      Array of risk factors
  auto_approvable    (boolean)   Can skip human review
  assessed_at        (string)    ISO 8601 timestamp
  assessed_by        (string)    Algorithm version
```

---

## Session Keys

#### User Session
```
Key: ade:session:{session_id}
Type: Hash
TTL: 24 hours

Fields:
  user_id            (string)    Authenticated user
  user_name          (string)    Display name
  roles              (JSON)      Array of role strings
  permissions        (JSON)      Array of permission strings
  created_at         (string)    ISO 8601 timestamp
  last_active        (string)    ISO 8601 timestamp
  ip_address         (string)    Client IP
  user_agent         (string)    Client user agent
```

#### User Active Sessions
```
Key: ade:session:index:user:{user_id}
Type: Set
TTL: 24 hours

Members: session_ids
```

---

## Rate Limiting Keys

#### API Rate Limit
```
Key: ade:ratelimit:{endpoint}:{user_id}
Type: String (counter) or Redis Cell (if available)
TTL: 1 minute (sliding window)

Value: Request count

Example:
  ade:ratelimit:tasks:create:user-123
  ade:ratelimit:approvals:respond:user-456
```

#### Lock Acquisition Rate Limit (per agent)
```
Key: ade:ratelimit:lock:acquire:{agent_id}
Type: String (counter)
TTL: 1 minute

Value: Lock acquisition attempts

Purpose: Prevent lock starvation attacks
```

---

## Event Keys

#### Event Stream (Redis Streams)
```
Key: ade:events:{event_type}
Type: Stream
TTL: 7 days (MAXLEN ~10000)

Event Types:
  ade:events:task
  ade:events:approval
  ade:events:lock

Entry Fields:
  event              (string)    Event name
  timestamp          (string)    ISO 8601 timestamp
  payload            (JSON)      Event data
  source             (string)    Service/agent that generated event

Example:
  XADD ade:events:task * \
    event "task:state_changed" \
    timestamp "2026-03-18T15:30:00Z" \
    payload '{"task_id":"...","from":"DRAFT","to":"SUBMITTED"}' \
    source "api-server-01"
```

#### Event Consumer Groups
```
Key: ade:events:{event_type}:consumers
Type: Stream Consumer Group

Groups:
  notification-service
  audit-logger
  webhook-dispatcher
  analytics-pipeline
```

---

## Background Job Keys

#### Job Queue
```
Key: ade:job:queue:{queue_name}
Type: List or Sorted Set
TTL: None

Queues:
  ade:job:queue:lock_cleanup      - Expired lock cleanup
  ade:job:queue:approval_timeout  - Approval escalation
  ade:job:queue:task_timeout      - Task execution timeout
  ade:job:queue:deadlock_detect   - Deadlock detection
  ade:job:queue:archive           - Old data archival
```

#### Scheduled Jobs
```
Key: ade:job:scheduled
Type: Sorted Set
TTL: None

Score: Execution timestamp (Unix seconds)
Value: JSON job description

Example:
  ZADD ade:job:scheduled 1710776400 \
    '{"type":"lock_cleanup","target":"ade:lock:task:123"}'
```

#### Job Locks (prevent duplicate job execution)
```
Key: ade:job:lock:{job_id}
Type: String
TTL: Job execution timeout

Value: Worker instance ID
```

---

## Configuration Keys

#### System Configuration
```
Key: ade:config:{config_name}
Type: String or Hash
TTL: None

Configs:
  ade:config:lock:default_ttl          (integer, seconds)
  ade:config:lock:max_ttl              (integer, seconds)
  ade:config:lock:heartbeat_interval   (integer, seconds)
  ade:config:approval:default_timeout  (integer, seconds)
  ade:config:approval:max_timeout      (integer, seconds)
  ade:config:task:default_timeout      (integer, seconds)
  ade:config:risk:thresholds           (JSON)
```

#### Feature Flags
```
Key: ade:feature:{flag_name}
Type: String
TTL: None

Value: "enabled" or "disabled"

Examples:
  ade:feature:auto_approve_low_risk
  ade:feature:deadlock_detection
  ade:feature:batch_approvals
```

---

## Key Lifecycle Summary

| Key Pattern | Type | Default TTL | Cleanup Strategy |
|-------------|------|-------------|------------------|
| `ade:lock:*` (active) | Hash | 30s | Heartbeat extends, expires auto-release |
| `ade:lock:*:queue` | ZSET | 5m | Cleared on lock release |
| `ade:lock:registry` | ZSET | None | Background job cleans expired |
| `ade:approval:request:*` | Hash | 30d | Archived, then deleted |
| `ade:approval:queue:*` | ZSET | None | Entries removed on status change |
| `ade:task:*:state` | String | 90d | Archived to cold storage |
| `ade:task:*:data` | JSON | 90d | Archived to cold storage |
| `ade:task:index:*` | ZSET | 90d | Cleared on task deletion |
| `ade:session:*` | Hash | 24h | Auto-expire |
| `ade:events:*` | Stream | 7d | MAXLEN eviction |
| `ade:ratelimit:*` | String | 1m | Auto-expire |

---

## Redis Commands Reference

### Lock Operations

```bash
# Acquire lock (with NX - only if not exists)
HSET ade:lock:task:123 \
  holder_agent_id agent-001 \
  acquired_at "2026-03-18T15:30:00Z" \
  expires_at "2026-03-18T15:30:30Z" \
  NX

# Extend lock TTL
HEXPIRE ade:lock:task:123 30

# Check lock
HGETALL ade:lock:task:123

# Release lock (use Lua for atomic check-and-delete)
# Lua script:
# if redis.call('hget', KEYS[1], 'holder_agent_id') == ARGV[1] then
#   return redis.call('del', KEYS[1])
# end
# return 0

# Add to queue
ZADD ade:lock:task:123:queue 1710775800000 '{"agent_id":"agent-002",...}'

# Get next waiter
ZPOPMIN ade:lock:task:123:queue 1
```

### Approval Operations

```bash
# Create approval request
HSET ade:approval:request:app-123 \
  task_id task-456 \
  reviewer_id user-789 \
  status PENDING

# Add to user queue
ZADD ade:approval:queue:user:user-789 850 app-123

# Record response
HSET ade:approval:request:app-123 \
  status APPROVED \
  responded_at "2026-03-18T16:00:00Z" \
  response_action approve

# Remove from queue
ZREM ade:approval:queue:user:user-789 app-123
```

### Task Operations

```bash
# Create task
SET ade:task:task-123:state DRAFT
HSET ade:task:task-123:metadata \
  author_id user-001 \
  created_at "2026-03-18T15:00:00Z"

# Update state (atomic)
SET ade:task:task-123:state REVIEWING
ZREM ade:task:index:state:DRAFT task-123
ZADD ade:task:index:state:REVIEWING 1710774000 task-123

# Get task with all data
HMGET ade:task:task-123:metadata author_id created_at
GET ade:task:task-123:state
```

---

## Cluster Mode Considerations

When using Redis Cluster, ensure related keys are on the same hash slot using hash tags:

```
ade:{task:123}:state        → hash slot for "task:123"
ade:{task:123}:data         → same slot
ade:{task:123}:execution    → same slot
ade:lock:task:{task:123}    → hash slot for "task:123"
ade:approval:index:task:{task:123} → hash slot for "task:123"
```

This enables multi-key operations (transactions, Lua scripts) on related data.

---

## Migration Notes

### From v1 to v2
- Renamed `lock:*` to `ade:lock:*` for namespacing
- Changed approval status from integers to strings
- Added JSON support for complex fields (requires RedisJSON or serialization)

### Backup Strategy
```bash
# Daily RDB snapshot
# Real-time AOF for point-in-time recovery
# Cross-region replication for disaster recovery
```