Migrations & Planning

Moose’s migration system works like version control for your infrastructure. It automatically detects changes in your code and applies them to your data infrastructure with confidence.

How It Works

Moose collects all objects defined in your main file (index.tsmain.py) and automatically generates infrastructure operations to match your code:

interface UserSchema {
  id: string;
  name: string;
  email: string;
}
 
export const usersTable = new OlapTable<UserSchema>("Users");
export const userEvents = new Stream<UserSchema>("UserEvents");

from pydantic import BaseModel
from moose_lib import OlapTable, Stream
 
class UserSchema(BaseModel):
    id: str
    name: str
    email: str
 
users_table = OlapTable[UserSchema]("Users")
user_events = Stream[UserSchema]("UserEvents")

When you add these objects, Moose automatically creates:

• A ClickHouse table named Users with the UserSchema
• A Redpanda topic named UserEvents with the UserSchema

Development Workflow

When running your code in development mode, Moose will automatically hot-reload migrations to your local infrastructure as you save code changes.

Quick Start

Start your development environment:

moose dev

This automatically:

1. Recursively watches your /app directory for code changes
2. Parses objects defined in your main file
3. Compares the new objects with the current infrastructure state Moose stores internally
4. Generates and applies migrations in real-time based on the differences
5. Provides immediate feedback on any errors or warnings
6. Updates the internal state of your infrastructure to reflect the new state

Example: Adding a New Table

// Before
export const usersTable = new OlapTable<UserSchema>("Users");
 
// After (add analytics table)
export const usersTable = new OlapTable<UserSchema>("Users");
export const analyticsTable = new OlapTable<AnalyticsSchema>("Analytics");

# Before
users_table = OlapTable[UserSchema]("Users")
 
# After (add analytics table)
users_table = OlapTable[UserSchema]("Users")
analytics_table = OlapTable[AnalyticsSchema]("Analytics")

What happens:

• Moose detects the new analyticsTable object
• Compares: “No Analytics table exists”
• Generates migration: “Create Analytics table”
• Applies migration automatically
• Updates internal state

Example: Schema Changes

// After (add age field)
interface UserSchema {
  id: string;
  name: string;
  email: string;
  age: number;  // New field
}

# After (add age field)
class UserSchema(BaseModel):
    id: str
    name: str
    email: str
    age: int  # New field

What happens:

• Moose detects the new age field
• Generates migration: “Add age column to Users table”
• Applies migration
• Existing rows get NULL/default values

Production Workflow

Remote Planning

For production deployments, preview changes before applying:

moose plan --url https://your-production-instance

[authentication]
admin_api_key = "your-hashed-token"

Understanding Plan Output

Moose shows exactly what will change:

Infrastructure Changes:
✨ New Components
  + analytics_table (OLAP Table)
  + metrics_topic (Streaming Topic)
  + api/v2/metrics (API Endpoint)
 
🔄 Modified Components
  ~ users_table (Schema Update)
  ~ events_topic (Config Change)
 
❌ Removed Components
  - deprecated_process (Background Process)

Production Deployment Flow

1. Develop locally with moose dev
2. Test changes in local environment
3. Plan against production with moose plan
4. Review changes carefully
5. Deploy using your deployment process

Migration Types

Viewing Infrastructure State

Via CLI

# Check current infrastructure objects
moose ls
 
# View migration logs
moose logs

Via Direct Connection

Connect to your local infrastructure using details from moose.config.toml:

[features]
olap = true              # ClickHouse for analytics
streaming_engine = true  # Redpanda for streaming
workflows = false        # Temporal for workflows
 
[clickhouse_config]
host = "localhost"
host_port = 18123
native_port = 9000
db_name = "local"
user = "panda"
password = "pandapass"
 
[redpanda_config]
broker = "localhost:19092"
message_timeout_ms = 1000
retention_ms = 30000
replication_factor = 1

Best Practices

Development

• Use moose dev for all local development
• Monitor plan outputs for warnings
• Test schema changes with sample data

Production

• Always use remote planning before deployments
• Review changes carefully in production plans
• Maintain proper authentication
• Test migrations in staging first

Troubleshooting

Authentication Errors

• Verify your authentication token
• Generate a new token: moose generate hash-token
• Check server configuration in moose.config.toml

Migration Issues

• Check moose logs for detailed error messages
• Verify object definitions in your main file
• Ensure all required fields are properly typed