Migration Examples & Advanced Development

Migration Types

Viewing:

This guide provides detailed examples of different migration types. For the complete workflow overview, see Migrations & Planning.

Adding New Infrastructure Components

Keep in mind that only the modules that you have enabled in your moose.config.toml will be included in your migrations.

[features]
olap = true
streaming_engine = true
workflows = true

New OLAP Table or Materialized View

interface AnalyticsSchema {
  id: string;
  event_type: string;
  timestamp: Date;
  user_id: string;
  value: number;
}
 
export const analyticsTable = new OlapTable<AnalyticsSchema>("Analytics");

from pydantic import BaseModel
from datetime import datetime
from moose_lib import OlapTable
 
class AnalyticsSchema(BaseModel):
    id: str
    event_type: str
    timestamp: datetime
    user_id: str
    value: float
 
analytics_table = OlapTable[AnalyticsSchema]("Analytics")

Migration Result: Creates ClickHouse table Analytics with all fields from AnalyticsSchema

Ensure OLAP is enabled

If you have not enabled the olap feature flag, you will not be able to create new OLAP tables.

[features]
olap = true

New Stream

export const userEvents = new Stream<UserSchema>("UserEvents");
export const systemEvents = new Stream<SystemEventSchema>("SystemEvents");

Migration Result: Creates Redpanda topics UserEvents and SystemEvents

Ensure Streaming Engine is enabled

If you have not enabled the streaming_engine feature flag, you will not be able to create new streaming topics.

[features]
streaming_engine = true

Schema Modifications

Adding Fields

// Before
interface UserSchema {
  id: string;
  name: string;
  email: string;
}
 
// After
interface UserSchema {
  id: string;
  name: string;
  email: string;
  age: number;
  created_at: Date;
  is_active: boolean;
}

# Before
class UserSchema(BaseModel):
    id: str
    name: str
    email: str
 
# After
class UserSchema(BaseModel):
    id: str
    name: str
    email: str
    age: int
    created_at: datetime
    is_active: bool

Migration Result: Adds age, created_at, and is_active columns to existing table

Removing Fields

// Before
interface UserSchema {
  id: string;
  name: string;
  email: string;
  age: number;
  deprecated_field: string;  // Will be removed
}
 
// After
interface UserSchema {
  id: string;
  name: string;
  email: string;
  age: number;
}

# Before
class UserSchema(BaseModel):
    id: str
    name: str
    email: str
    age: int
    deprecated_field: str  # Will be removed
 
# After
class UserSchema(BaseModel):
    id: str
    name: str
    email: str
    age: int

Migration Result: Drops deprecated_field column (data permanently lost)

Type Changes

// Before
interface UserSchema {
  id: string;
  name: string;
  email: string;
  score: number;  // Will change to string
}
 
// After
interface UserSchema {
  id: string;
  name: string;
  email: string;
  score: string;  // Changed from number
}

# Before
class UserSchema(BaseModel):
    id: str
    name: str
    email: str
    score: float  # Will change to str
 
# After
class UserSchema(BaseModel):
    id: str
    name: str
    email: str
    score: str  # Changed from float

Migration Result: Alters score column type (data converted if compatible)

Removing Infrastructure

// Before
export const usersTable = new OlapTable<UserSchema>("Users");
export const analyticsTable = new OlapTable<AnalyticsSchema>("Analytics");
export const deprecatedTable = new OlapTable<DeprecatedSchema>("Deprecated");
 
// After (remove deprecated table)
export const usersTable = new OlapTable<UserSchema>("Users");
export const analyticsTable = new OlapTable<AnalyticsSchema>("Analytics");

# Before
users_table = OlapTable[UserSchema]("Users")
analytics_table = OlapTable[AnalyticsSchema]("Analytics")
deprecated_table = OlapTable[DeprecatedSchema]("Deprecated")
 
# After (remove deprecated table)
users_table = OlapTable[UserSchema]("Users")
analytics_table = OlapTable[AnalyticsSchema]("Analytics")

Migration Result: Drops Deprecated table (all data lost)

Working with Local Infrastructure

There are two main ways to inspect your local infrastructure to see how your migrations are applied:

Using the CLI

Run moose ls to see the current state of your infrastructure:

# Verify object definitions
moose ls

Connecting to your local infrastructure

You can also connect directly to your local infrastructure to see the state of your infrastructure.

Credentials Located in Project Config

All credentials for your local infrastructure are located in your project config file (moose.config.toml).

Connecting to ClickHouse

# Using clickhouse-client
clickhouse-client --host localhost --port 18123 --user panda --password pandapass --database local
 
# Using connection string
clickhouse-client "clickhouse://panda:pandapass@localhost:18123/local"

Connecting to Redpanda

# Using kafka-console-consumer
kafka-console-consumer --bootstrap-server localhost:19092 --topic UserEvents --from-beginning
 
# Using kafka-console-producer
kafka-console-producer --bootstrap-server localhost:19092 --topic UserEvents

Viewing Temporal Workflows

Navigate to http://localhost:8080 to view the Temporal UI and see registered workflows.

Gotchas:

Your dev server must be running to connect to your local infrastructure.

moose dev

Only the modules that you have enabled in your moose.config.toml will be included in your migrations:

[features]
olap = true # Required for OLAP Tables and Materialized Views
streaming_engine = true # Required for Streams
workflows = true # Required for Workflows and Tasks

Moose Migrate

Lifecycle Management