# Moose / Migrate Documentation – Python

## Included Files
1. moose/migrate/lifecycle.mdx
2. moose/migrate/migration-types.mdx

## LifeCycle Management

Source: moose/migrate/lifecycle.mdx

Control how Moose manages database and streaming resources when your code changes

# LifeCycle Management

## Overview

The `LifeCycle` enum controls how Moose manages the lifecycle of database/streaming resources when your code changes.
This feature gives you fine-grained control over whether Moose automatically updates your database schema or
leaves it under external/manual control.

## LifeCycle Modes

### `FULLY_MANAGED` (Default)

This is the default behavior where Moose has complete control over your database resources. When you change your data models, Moose will automatically:

- Add new columns or tables
- Remove columns or tables that no longer exist in your code
- Modify existing column types and constraints

This mode can perform destructive operations. Data may be lost if you remove fields from your data models or if you perform operations that require a destroy and recreate to be effective, like changing the `order_by_fields` field .  

```py filename="FullyManagedExample.py" copy
from moose_lib import OlapTable, OlapConfig, LifeCycle
from pydantic import BaseModel

class UserData(BaseModel):
    id: str
    name: str
    email: str

# Default behavior - fully managed
user_table = OlapTable[UserData]("users")

# Explicit fully managed configuration
explicit_table = OlapTable[UserData]("users", OlapConfig(
    order_by_fields=["id"],
    life_cycle=LifeCycle.FULLY_MANAGED
))
```

### `DELETION_PROTECTED`

This mode allows Moose to automatically add new database structures but prevents it from removing existing ones.
Perfect for production environments where you want to evolve your schema safely without risking data loss.

**What Moose will do:**
- Add new columns, tables
- Modify column types (if compatible)
- Update non-destructive configurations

**What Moose won't do:**
- Drop columns or tables
- Perform destructive schema changes

```py filename="DeletionProtectedExample.py" copy
from moose_lib import IngestPipeline, IngestPipelineConfig, OlapConfig, StreamConfig, LifeCycle
from pydantic import BaseModel
from datetime import datetime

class ProductEvent(BaseModel):
    id: str
    product_id: str
    timestamp: datetime
    action: str

product_analytics = IngestPipeline[ProductEvent]("product_analytics", IngestPipelineConfig(
    table=OlapConfig(
        order_by_fields=["timestamp", "product_id"],
        engine=ClickHouseEngines.ReplacingMergeTree,
    ),
    stream=StreamConfig(
        parallelism=4,
    ),
    ingest_api=True,
  # automatically applied to the table and stream
  life_cycle=LifeCycle.DELETION_PROTECTED
))
```

### `EXTERNALLY_MANAGED`

This mode tells Moose to completely hands-off your resources.
You become responsible for creating and managing the database schema. This is useful when:

- You have existing database tables managed by another team
- You're integrating with another system (e.g. PeerDB)
- You have strict database change management processes

With externally managed resources, you must ensure your database schema matches your data models exactly, or you may encounter runtime errors.

```py filename="ExternallyManagedExample.py" copy
from moose_lib import Stream, OlapTable, OlapConfig, StreamConfig, LifeCycle, Key
from pydantic import BaseModel
from datetime import datetime

class ExternalUserData(BaseModel):
    user_id: Key[str]
    full_name: str
    email_address: str
    created_at: datetime

# Connect to existing database table
legacy_user_table = OlapTable[ExternalUserData]("legacy_users", OlapConfig(
    life_cycle=LifeCycle.EXTERNALLY_MANAGED
))

# Connect to existing Kafka topic
legacy_stream = Stream[ExternalUserData]("legacy_user_stream", StreamConfig(
    life_cycle=LifeCycle.EXTERNALLY_MANAGED,
    destination=legacy_user_table
))
```

---

## Migration Examples & Advanced Development

Source: moose/migrate/migration-types.mdx

Detailed migration examples and advanced development topics

# Migration Types

This guide provides detailed examples of different migration types. For the complete workflow overview, see [Migrations & Planning](/moose/migrate/planning).

## Adding New Infrastructure Components

Keep in mind that only the modules that you have enabled in your `moose.config.toml` will be included in your migrations.

```toml file="moose.config.toml"
[features]
olap = true
streaming_engine = true
workflows = true
```

### New OLAP Table or Materialized View

```python file="app/main.py"
from pydantic import BaseModel
from datetime import datetime
from moose_lib import OlapTable

class AnalyticsSchema(BaseModel):
    id: str
    event_type: str
    timestamp: datetime
    user_id: str
    value: float

analytics_table = OlapTable[AnalyticsSchema]("Analytics")
```

**Migration Result:** Creates ClickHouse table `Analytics` with all fields from `AnalyticsSchema`

If you have not enabled the `olap` feature flag, you will not be able to create new OLAP tables.

```toml file="moose.config.toml"
[features]
olap = true
```

Check out the OLAP migrations guide to learn more about the different migration modes.

### New Stream

```python file="app/main.py"
user_events = Stream[UserSchema]("UserEvents")
system_events = Stream[SystemEventSchema]("SystemEvents")
```

**Migration Result:** Creates Redpanda topics `UserEvents` and `SystemEvents`

If you have not enabled the `streaming_engine` feature flag, you will not be able to create new streaming topics.

```toml file="moose.config.toml"
[features]
streaming_engine = true
```

## Schema Modifications

### Adding Fields

```python file="app/main.py"
# Before
class UserSchema(BaseModel):
    id: str
    name: str
    email: str

# After
class UserSchema(BaseModel):
    id: str
    name: str
    email: str
    age: int
    created_at: datetime
    is_active: bool
```

**Migration Result:** Adds `age`, `created_at`, and `is_active` columns to existing table

### Removing Fields

```python file="app/main.py"
# Before
class UserSchema(BaseModel):
    id: str
    name: str
    email: str
    age: int
    deprecated_field: str  # Will be removed

# After
class UserSchema(BaseModel):
    id: str
    name: str
    email: str
    age: int
```

**Migration Result:** Drops `deprecated_field` column (data permanently lost)

### Type Changes

```python file="app/main.py"
# Before
class UserSchema(BaseModel):
    id: str
    name: str
    email: str
    score: float  # Will change to str

# After
class UserSchema(BaseModel):
    id: str
    name: str
    email: str
    score: str  # Changed from float
```

**Migration Result:** Alters `score` column type (data converted if compatible)

## Removing Infrastructure

```python file="app/main.py"
# Before
users_table = OlapTable[UserSchema]("Users")
analytics_table = OlapTable[AnalyticsSchema]("Analytics")
deprecated_table = OlapTable[DeprecatedSchema]("Deprecated")

# After (remove deprecated table)
users_table = OlapTable[UserSchema]("Users")
analytics_table = OlapTable[AnalyticsSchema]("Analytics")
```

**Migration Result:** Drops `Deprecated` table (all data lost)

## Working with Local Infrastructure

There are two main ways to inspect your local infrastructure to see how your migrations are applied:

### Using the CLI

Run `moose ls` to see the current state of your infrastructure:

```bash
# Verify object definitions
moose ls
```

### Connecting to your local infrastructure
You can also connect directly to your local infrastructure to see the state of your infrastructure.

All credentials for your local infrastructure are located in your project config file (`moose.config.toml`).

#### Connecting to ClickHouse

```bash
# Using clickhouse-client
clickhouse-client --host localhost --port 18123 --user panda --password pandapass --database local

# Using connection string
clickhouse-client "clickhouse://panda:pandapass@localhost:18123/local"
```

#### Connecting to Redpanda

```bash
# Using kafka-console-consumer
kafka-console-consumer --bootstrap-server localhost:19092 --topic UserEvents --from-beginning

# Using kafka-console-producer
kafka-console-producer --bootstrap-server localhost:19092 --topic UserEvents
```

#### Viewing Temporal Workflows

Navigate to `http://localhost:8080` to view the Temporal UI and see registered workflows.

## Gotchas:

Your dev server must be running to connect to your local infrastructure.

```bash
moose dev
```

Only the modules that you have enabled in your `moose.config.toml` will be included in your migrations:

```toml file="moose.config.toml"
[features]
olap = true # Required for OLAP Tables and Materialized Views
streaming_engine = true # Required for Streams
workflows = true # Required for Workflows and Tasks
```