From 9cbb8fa42acc8656330c07e788f35ad1dd1fe291 Mon Sep 17 00:00:00 2001 From: Sarah Wooders Date: Wed, 3 Dec 2025 13:02:48 -0800 Subject: [PATCH] feat: add `.skills/db-migrations-schema-changes` (#6476) --- .skills/db-migrations-schema-changes/SKILL.md | 111 ++++++++++++++++++ .../references/migration-commands.md | 72 ++++++++++++ .../references/sqlite-vs-postgres-gotchas.md | 41 +++++++ 3 files changed, 224 insertions(+) create mode 100644 .skills/db-migrations-schema-changes/SKILL.md create mode 100644 .skills/db-migrations-schema-changes/references/migration-commands.md create mode 100644 .skills/db-migrations-schema-changes/references/sqlite-vs-postgres-gotchas.md diff --git a/.skills/db-migrations-schema-changes/SKILL.md b/.skills/db-migrations-schema-changes/SKILL.md new file mode 100644 index 00000000..7241ae60 --- /dev/null +++ b/.skills/db-migrations-schema-changes/SKILL.md @@ -0,0 +1,111 @@ +--- +name: DB migrations and schema changes +description: >- + Workflows and commands for managing Alembic database migrations and schema changes + in the letta-cloud core app, including using uv, just, LETTA_PG_URI, and + switching between SQLite and Postgres. +--- + +# DB migrations and schema changes (letta-cloud core) + +Use this skill whenever you need to change the database schema or debug Alembic +migrations in `apps/core` of the letta-cloud repo. + +This skill assumes: +- Working directory: `apps/core` +- Migrations: Alembic in `apps/core/alembic` +- Python runner: `uv` +- Helper: `just ready` for environment + DB setup + +## Quick start + +1. Ensure environment is ready: + - `just ready` +2. For Postgres migrations, set: + - `export LETTA_PG_URI=postgresql+pg8000://postgres:postgres@localhost:5432/letta-core` +3. Make your ORM/schema change. +4. Autogenerate migration: + - `uv run alembic revision --autogenerate -m ""` +5. Apply migration: + - `uv run alembic upgrade head` + +See `references/migration-commands.md` for exact commands and variants. + +## Standard workflows + +### 1. Add or modify a column (ORM-first) + +1. Identify the ORM model and table. +2. Update the SQLAlchemy model in `letta/orm/...`: + - Prefer using mixins (e.g. `ProjectMixin`) when available instead of + duplicating columns. +3. Run `just ready` if dependencies or environment may have changed. +4. Ensure `LETTA_PG_URI` is set if you want the migration to target Postgres. +5. Autogenerate Alembic revision with `uv`. +6. Inspect the generated file under `alembic/versions/`: + - Confirm `op.add_column` / `op.alter_column` match expectations. +7. Apply migrations with `uv run alembic upgrade head`. + +Use this pattern for changes like adding `project_id` columns via `ProjectMixin`. + +### 2. Data backfill / one-off data migration + +1. Make sure the schema change (if any) is already represented in ORM + Alembic. +2. Create a new Alembic revision **without** autogenerate (or edit an + autogen file) and add Python logic in `upgrade()` that: + - Uses `op.get_bind()` and SQLAlchemy Core/SQL to backfill data. +3. Keep `downgrade()` simple and safe (ideally reversible). +4. Run against Postgres with `LETTA_PG_URI` set, using `uv run alembic upgrade head`. + +### 3. Fixing a bad migration + +Typical cases: +- Migration fails only on SQLite (ALTER constraint limitations). +- Migration was generated while pointing at SQLite instead of Postgres. + +Workflow: +1. Identify the failing revision in `alembic/versions/`. +2. If failure is SQLite-specific, prefer running migrations against Postgres by + exporting `LETTA_PG_URI` and re-running upgrade. +3. If logic is wrong, create a **new** migration that fixes the problem rather + than editing an applied revision (especially in shared environments). +4. For purely local/dev history, you can delete and regenerate migrations but + only if no one else depends on them. + +See `references/sqlite-vs-postgres-gotchas.md` for SQLite-specific issues. + +### 4. Switching between SQLite and Postgres + +Alembic picks the engine based on `letta.settings.DatabaseChoice` and +environment variables. + +General rules: +- For local dev stateful runs, `just ready` handles baseline migrations. +- For schema design and production-like migrations, prefer Postgres and set + `LETTA_PG_URI`. + +Workflow for Postgres-targeted migration: +1. `export LETTA_PG_URI=postgresql+pg8000://postgres:postgres@localhost:5432/letta-core` +2. From `apps/core`: + - `uv run alembic upgrade head` + - `uv run alembic revision --autogenerate -m "..."` + +## Troubleshooting + +- **"Target database is not up to date" when autogenerating** + - First run `uv run alembic upgrade head` (with appropriate engine/URI). +- **SQLite NotImplementedError about ALTER CONSTRAINT** + - Switch to Postgres by setting `LETTA_PG_URI` and rerun. +- **Autogenerated migration missing expected changes** + - Ensure ORM imports and metadata (`Base.metadata`) are correct and that the + changed model is imported in Alembic env context. +- **Autogenerated migration has unexpected drops/renames** + - Review model changes; consider explicit operations instead of relying on + autogenerate. + +## References + +- `references/migration-commands.md` — canonical commands for `uv`, Alembic, + and `just`. +- `references/sqlite-vs-postgres-gotchas.md` — engine-specific pitfalls and + how to avoid them. diff --git a/.skills/db-migrations-schema-changes/references/migration-commands.md b/.skills/db-migrations-schema-changes/references/migration-commands.md new file mode 100644 index 00000000..24204a62 --- /dev/null +++ b/.skills/db-migrations-schema-changes/references/migration-commands.md @@ -0,0 +1,72 @@ +# Migration commands (letta-cloud core) + +Working directory for all commands: `apps/core`. + +## Environment setup + +- One-shot environment + DB setup: + - `just ready` + +## Postgres connection + +Set the Postgres URI (adjust as needed for your env): + +```bash +export LETTA_PG_URI=postgresql+pg8000://postgres:postgres@localhost:5432/letta-core +``` + +Alembic will log the effective URL (e.g. `Using database: postgresql+pg8000://...`). + +## Alembic basics (with uv) + +- Upgrade to latest: + +```bash +uv run alembic upgrade head +``` + +- Downgrade one step: + +```bash +uv run alembic downgrade -1 +``` + +- Downgrade to a specific revision: + +```bash +uv run alembic downgrade +``` + +- Generate new revision (autogenerate): + +```bash +uv run alembic revision --autogenerate -m "short_message" +``` + +- Generate empty revision (manual operations): + +```bash +uv run alembic revision -m "manual_migration" +``` + +## Typical workflow snippets + +### Add/modify column + +```bash +cd apps/core +just ready # optional but recommended +export LETTA_PG_URI=postgresql+pg8000://postgres:postgres@localhost:5432/letta-core +uv run alembic upgrade head # ensure DB is up to date +uv run alembic revision --autogenerate -m "add__to_" +uv run alembic upgrade head +``` + +### Re-run last migration after edit (local only) + +```bash +cd apps/core +export LETTA_PG_URI=postgresql+pg8000://postgres:postgres@localhost:5432/letta-core +uv run alembic downgrade -1 +uv run alembic upgrade head +``` diff --git a/.skills/db-migrations-schema-changes/references/sqlite-vs-postgres-gotchas.md b/.skills/db-migrations-schema-changes/references/sqlite-vs-postgres-gotchas.md new file mode 100644 index 00000000..ea81cdd8 --- /dev/null +++ b/.skills/db-migrations-schema-changes/references/sqlite-vs-postgres-gotchas.md @@ -0,0 +1,41 @@ +# SQLite vs Postgres gotchas (letta-cloud core) + +## SQLite limitations + +SQLite has limited support for `ALTER TABLE`, especially around constraints and +foreign keys. In Alembic this often shows up as: + +- `NotImplementedError: No support for ALTER of constraints in SQLite dialect...` + +In `apps/core`, you may hit this when running migrations against SQLite that +drop or change foreign keys or constraints. + +### How to handle + +- Prefer running schema-changing migrations against Postgres by setting + `LETTA_PG_URI` and using: + + ```bash + uv run alembic upgrade head + ``` + +- If you must support SQLite, use Alembic batch mode patterns, but for this + project most complex migrations should target Postgres. + +## Autogenerate differences + +Running `alembic revision --autogenerate` against SQLite vs Postgres can +produce different diffs (especially around indexes and constraints). + +Recommendations: + +- For production/real migrations, always autogenerate against Postgres. +- If you see lots of unexpected index drops/adds, confirm which engine the + migration is inspecting and rerun with Postgres. + +## Engine selection reminder + +- Engine is controlled by `letta.settings.DatabaseChoice` and environment + (notably `LETTA_PG_URI`). +- `just ready` will also run migrations; ensure your desired engine is set + before relying on its database steps.