# Moose / Reference / Ts Moose Lib Documentation – Python

## Included Files
1. moose/reference/ts-moose-lib/ts-moose-lib.mdx

## TypeScript Moose Lib Reference

Source: moose/reference/ts-moose-lib/ts-moose-lib.mdx

TypeScript Moose Lib Reference

# API Reference

This is a comprehensive reference for   `moose_lib` , detailing all exported components, types, and utilities.

## Core Types

### `Key`
A type for marking fields as primary keys in data models.
```ts
// Example
interface MyModel {
  id: Key<string>; // Marks 'id' as a primary key of type string
}
```

### `JWT`
A type for working with JSON Web Tokens.
```ts
// Example
type UserJWT = JWT<{ userId: string, role: string }>;
```

### `ApiUtil`
Interface providing utilities for analytics APIs.
```ts
interface ApiUtil {
  client: MooseClient;  // Client for interacting with the database
  sql: typeof sql;      // SQL template tag function
  jwt: JWTPayload | undefined;  // Current JWT if available
}
```

## Date and Time Types

Type aliases that map JavaScript temporal data to ClickHouse DateTime types.
Choose between Date objects or strings based on the precision you need to persist in ClickHouse.

### `DateTime`
A type alias for JavaScript's native `Date` object that maps to ClickHouse `DateTime` (second precision).

```ts
interface Event {
  // Stored as ClickHouse DateTime (second precision)
  // Runtime: JavaScript Date object
  createdAt: DateTime;
}
```

**Important:** While JavaScript `Date` objects can hold millisecond precision, only the second portion will be persisted in ClickHouse since the database type is `DateTime`.

**Use when:**
- Second precision is sufficient for your use case
- You need standard JavaScript Date objects in your application
- Working with user-visible timestamps where sub-second precision isn't critical

### `DateTime64`
A type for datetime values with configurable precision (0-9 decimal places) that maps to ClickHouse `DateTime64(P)`.
At runtime, JavaScript `Date` objects are used.

```ts
interface Event {
  // ✅ Safe: ClickHouse DateTime64(3), JavaScript Date holds milliseconds
  timestamp: DateTime64<3>;

  // ⚠️ Precision loss: ClickHouse stores microseconds, but Date truncates to milliseconds
  highPrecisionTimestamp: DateTime64<6>;
}
```

**Precision Limits:**
- **P ≤ 3** (milliseconds): JavaScript `Date` can represent this precision - **use this safely**
- **P > 3** (microseconds and beyond): ClickHouse stores the full precision, but `Date` objects truncate to milliseconds during JSON parsing - **precision loss occurs**

**Use when:**
- You need millisecond precision (P ≤ 3) with Date objects
- You're okay with Date API in your application code

**Don't use when:**
- You need microsecond or nanosecond precision - use `DateTime64String` instead

### `DateTimeString`
A string type that maps to ClickHouse `DateTime` (second precision) but stays as a string at runtime instead of being parsed into a `Date` object.

```ts
interface Event {
  // Stored as ClickHouse DateTime (second precision)
  // Runtime: ISO 8601 string (e.g., "2024-01-15T10:30:00Z")
  createdAt: DateTimeString;
}
```

**Use when:**
- You prefer string-based datetime handling in your application code
- You don't need Date object methods

### `DateTime64String`
A string type that maps to ClickHouse `DateTime64(P)` and preserves full precision by keeping datetimes as strings at runtime instead of parsing into `Date` objects.

```ts
interface Event {
  // ✅ Full precision preserved: ClickHouse stores microseconds, runtime keeps string
  highPrecisionTimestamp: DateTime64String<6>;  // e.g., "2024-01-15T10:30:00.123456Z"

  // ✅ Full precision preserved: ClickHouse stores nanoseconds, runtime keeps string
  veryHighPrecisionTimestamp: DateTime64String<9>;  // e.g., "2024-01-15T10:30:00.123456789Z"
}
```

**Use when:**
- **You need microsecond (P≥4) or higher precision** - this is the only type that preserves it
- You're ingesting high-precision timestamps from external systems
- You're okay working with ISO 8601 strings instead of Date objects

**Why strings?** JavaScript `Date` objects truncate to milliseconds. By keeping timestamps as strings, you preserve the full precision that ClickHouse stores.

### Comparison Table

| Type | ClickHouse | Runtime Type | Precision | Use Case |
|------|-----------|--------------|-----------|----------|
| `DateTime` | `DateTime` | `Date` | 1 second | Standard timestamps |
| `DateTime64<3>` | `DateTime64(3)` | `Date` | 1 millisecond | JavaScript-compatible high precision |
| `DateTime64<6>` | `DateTime64(6)` | `Date` | ⚠️ ~1 millisecond* | Not recommended - use DateTime64String |
| `DateTimeString` | `DateTime` | `string` | 1 second | String-based datetime handling |
| `DateTime64String<6>` | `DateTime64(6)` | `string` | 1 microsecond | High-precision requirements |
| `DateTime64String<9>` | `DateTime64(9)` | `string` | 1 nanosecond | Ultra-high-precision requirements |

*Loses microseconds during JSON parsing

### Example: Choosing the Right Type
```ts
interface Telemetry {
  // User-visible timestamp - Date object is fine
  createdAt: DateTime;

  // High-frequency sensor data - need microseconds
  sensorTimestamp: DateTime64String<6>;

  // API response timestamp - millisecond precision
  processedAt: DateTime64<3>;

  // Compliance log - preserve exact string format
  auditTimestamp: DateTimeString<9>;
}
```

## Infrastructure Components

### `OlapTable`
Creates a ClickHouse table with the schema of type T.
```ts
// Basic usage with MergeTree (default)
);

// With sorting configuration (expression)
);

// Disable sorting entirely
);

// For deduplication, explicitly set the ReplacingMergeTree engine
);
```

### `BaseOlapConfig`
Base configuration interface for `OlapTable` with common table configuration options.
```ts
interface BaseOlapConfig {
  // Optional database name (defaults to moose.config.toml clickhouse_config.db_name)
  database?: string;

  // Optional array of field names to order by
  orderByFields?: (keyof T & string)[];

  // Optional SQL expression for ORDER BY clause (alternative to orderByFields)
  orderByExpression?: string;

  // Optional table engine (defaults to MergeTree)
  engine?: ClickHouseEngines;

  // Optional settings for table configuration
  settings?: { [key: string]: string };

  // Optional lifecycle mode (defaults to MOOSE_MANAGED)
  lifeCycle?: LifeCycle;

  // Additional engine-specific fields (ver, isDeleted, keeperPath, etc.)
  // depend on the engine type
}
```

Example with database override:
```ts
// Table in custom database
);

// Default database (from moose.config.toml)
);
```

### `Stream`
Creates a Redpanda topic with the schema of type T.
```ts
// Basic usage
);

// Adding transformations
myConfiguredStream.addTransform(
  destinationStream,
  (record) => transformFunction(record)
);
```

### `IngestApi`
Creates an HTTP endpoint for ingesting data of type T.
```ts
// Basic usage with destination stream
);
```

### `Api<T, R>`
Creates an HTTP endpoint for querying data with request type T and response type R.
```ts
// Basic usage
) => {
    const result = await client.query.execute(
      sql`SELECT * FROM user_profiles WHERE age > ${params.minAge} LIMIT 10`
    );
    return result;
  }
);
```

### `IngestPipeline`
Combines ingest API, stream, and table creation in a single component.
```ts
// Basic usage
);

// With advanced configuration
);
```

### `MaterializedView`
Creates a materialized view in ClickHouse.
```ts
// Basic usage
);
```

## SQL Utilities

### `sql` Template Tag
Template tag for creating type-safe SQL queries with parameters.
```ts
// Basic usage
const query = sql`SELECT * FROM users WHERE id = ${userId}`;

// With multiple parameters
const query = sql`
  SELECT * FROM users 
  WHERE age > ${minAge} 
  AND country = ${country}
  LIMIT ${limit}
`;
```

### `MooseClient`
Client for interacting with ClickHouse and Temporal.
```ts
class MooseClient {
  query: QueryClient;      // For database queries
  workflow: WorkflowClient; // For workflow operations
}
```

## ClickHouse Utilities

### Table Engine Configurations

#### `ClickHouseEngines` Enum
Available table engines:
```ts
enum ClickHouseEngines {
  MergeTree = "MergeTree",
  ReplacingMergeTree = "ReplacingMergeTree",
  AggregatingMergeTree = "AggregatingMergeTree",
  SummingMergeTree = "SummingMergeTree",
  ReplicatedMergeTree = "ReplicatedMergeTree",
  ReplicatedReplacingMergeTree = "ReplicatedReplacingMergeTree",
  ReplicatedAggregatingMergeTree = "ReplicatedAggregatingMergeTree",
  ReplicatedSummingMergeTree = "ReplicatedSummingMergeTree",
  S3Queue = "S3Queue"
}
```

#### `ReplacingMergeTreeConfig`
Configuration for ReplacingMergeTree tables:
```ts
type ReplacingMergeTreeConfig = {
  engine: ClickHouseEngines.ReplacingMergeTree;
  orderByFields?: (keyof T & string)[];
  ver?: keyof T & string;  // Optional: version column for keeping latest
  isDeleted?: keyof T & string;  // Optional: soft delete marker (requires ver)
  settings?: { [key: string]: string };
}
```

#### Replicated Engine Configurations
Configuration for replicated table engines:
```ts
// ReplicatedMergeTree
type ReplicatedMergeTreeConfig = {
  engine: ClickHouseEngines.ReplicatedMergeTree;
  keeperPath?: string;  // Optional: ZooKeeper/Keeper path (omit for Cloud)
  replicaName?: string; // Optional: replica name (omit for Cloud)
  orderByFields?: (keyof T & string)[];
  settings?: { [key: string]: string };
}

// ReplicatedReplacingMergeTree
type ReplicatedReplacingMergeTreeConfig = {
  engine: ClickHouseEngines.ReplicatedReplacingMergeTree;
  keeperPath?: string;  // Optional: ZooKeeper/Keeper path (omit for Cloud)
  replicaName?: string; // Optional: replica name (omit for Cloud)
  ver?: keyof T & string;  // Optional: version column
  isDeleted?: keyof T & string;  // Optional: soft delete marker
  orderByFields?: (keyof T & string)[];
  settings?: { [key: string]: string };
}

// ReplicatedAggregatingMergeTree
type ReplicatedAggregatingMergeTreeConfig = {
  engine: ClickHouseEngines.ReplicatedAggregatingMergeTree;
  keeperPath?: string;  // Optional: ZooKeeper/Keeper path (omit for Cloud)
  replicaName?: string; // Optional: replica name (omit for Cloud)
  orderByFields?: (keyof T & string)[];
  settings?: { [key: string]: string };
}

// ReplicatedSummingMergeTree
type ReplicatedSummingMergeTreeConfig = {
  engine: ClickHouseEngines.ReplicatedSummingMergeTree;
  keeperPath?: string;  // Optional: ZooKeeper/Keeper path (omit for Cloud)
  replicaName?: string; // Optional: replica name (omit for Cloud)
  columns?: string[];   // Optional: columns to sum
  orderByFields?: (keyof T & string)[];
  settings?: { [key: string]: string };
}
```

**Note**: The `keeperPath` and `replicaName` parameters are optional. When omitted, Moose uses smart defaults that work in both ClickHouse Cloud and self-managed environments (default path: `/clickhouse/tables/{uuid}/{shard}` with replica `{replica}`). You can still provide both parameters explicitly if you need custom replication paths.

### `S3QueueTableSettings`
Type-safe interface for S3Queue-specific table settings (ClickHouse 24.7+).
```ts
interface S3QueueTableSettings {
  mode?: "ordered" | "unordered";  // Processing mode
  after_processing?: "keep" | "delete";  // File handling after processing
  keeper_path?: string;  // ZooKeeper path for coordination
  loading_retries?: string;  // Number of retry attempts
  processing_threads_num?: string;  // Parallel processing threads
  // ... and many more settings
}
```

### S3Queue Configuration
Configure S3Queue tables for streaming data from S3 buckets (ORDER BY is not supported):
```ts
);
```

### S3 Configuration
Configure S3 tables for direct read/write access to S3 storage:
```ts
);

// Public bucket (no authentication) - omit credentials for NOSIGN
);
```

### Buffer Configuration
Configure Buffer tables for high-throughput buffered writes (ORDER BY is not supported):
```ts
// First create destination table
);

// Then create buffer table
);
```

### Distributed Configuration
Configure Distributed tables for cluster-wide distributed queries (ORDER BY is not supported):
```ts
);
```

## Task Management

### `Task<T, R>`
A class that represents a single task within a workflow system.
```ts
// No input, no output
);

// With input and output
);
```

### `TaskContext`
A context object that includes input & state passed between the task's run/cancel functions.
```ts
export type TaskContext = T extends null ? { state: any; input?: null } : { state: any; input: T };
```

### `TaskConfig<T, R>`
Configuration options for tasks.
```ts
interface TaskConfig<T, R> {
  // The main function that executes the task logic
  run: (context: TaskContext) => Promise;

  // Optional array of tasks to execute after this task completes
  onComplete?: (Task | Task)[];

  // Optional function that is called when the task is cancelled.
  onCancel?: (context: TaskContext) => Promise<void>;

  // Optional timeout duration (e.g., "30s", "5m", "never")
  timeout?: string;

  // Optional number of retry attempts
  retries?: number;
}
```

### `Workflow`
A class that represents a complete workflow composed of interconnected tasks.
```ts
const myWorkflow = new Workflow("getData", {
  startingTask: callAPI,
  schedule: "@every 5s", // Run every 5 seconds
  timeout: "1h",
  retries: 3
});

```

### `WorkflowConfig`
Configuration options for defining a workflow.
```ts
interface WorkflowConfig {
  // The initial task that begins the workflow execution
  startingTask: Task<null, any> | Task<null, void> | Task<any, any> | Task<any, void>;

  // Optional number of retry attempts
  retries?: number;

  // Optional timeout duration (e.g., "10m", "1h", "never")
  timeout?: string;

  // Optional cron-style schedule string
  schedule?: string;
}
```

---

**Important:** The following components must be exported from your `app/index.ts` file for Moose to detect them:

- `OlapTable` instances
- `Stream` instances  
- `IngestApi` instances
- `Api` instances
- `IngestPipeline` instances
- `MaterializedView` instances
- `Task` instances
- `Workflow` instances

**Configuration objects and utilities** (like `DeadLetterQueue`, `Key`, `sql`) do not need to be exported as they are used as dependencies of the main components.

## Registry Functions

The registry functions allow you to programmatically access all registered Moose resources at runtime. This is useful for inspecting resources, building dynamic tools, meta-programming, and testing.

### `getTables()`
Get all registered OLAP tables.
```ts

const tables = getTables();  // Returns Map<string, OlapTable<any>>
for (const [name, table] of tables) {
  console.log(`Table: ${name}`);
}
```

### `getTable(name)`
Get a specific OLAP table by name.
```ts

const userTable = getTable("Users");
if (userTable) {
  console.log(`Found table: ${userTable.name}`);
}
```

### `getStreams()`
Get all registered streams.
```ts

const streams = getStreams();  // Returns Map<string, Stream<any>>
```

### `getStream(name)`
Get a specific stream by name.
```ts

const eventStream = getStream("UserEvents");
```

### `getIngestApis()`
Get all registered ingestion APIs.
```ts

const ingestApis = getIngestApis();  // Returns Map<string, IngestApi<any>>
```

### `getIngestApi(name)`
Get a specific ingestion API by name.
```ts

const userApi = getIngestApi("IngestUsers");
```

### `getApis()`
Get all registered consumption/egress APIs.
```ts

const apis = getApis();  // Returns Map<string, Api<any>>
```

### `getApi(nameOrPath)`
Get a registered API by name, version, or custom path.

Supports multiple lookup strategies:
- Direct lookup by full key (`name` or `name:version`)
- Alias lookup by base name when only one versioned API exists
- Lookup by custom path (if configured)

```ts

// Direct lookup
const api = getApi("QueryUsers");

// Version lookup
const apiV1 = getApi("QueryUsers:1.0");

// Path lookup
const apiByPath = getApi("/custom/query");

// Automatic aliasing: if only one version exists, unversioned lookup works
const api = getApi("QueryUsers");  // Returns QueryUsers:1.0 if it's the only version
```

### `getSqlResources()`
Get all registered SQL resources (views, etc.).
```ts

const resources = getSqlResources();  // Returns Map<string, SqlResource>
```

### `getSqlResource(name)`
Get a specific SQL resource by name.
```ts

const view = getSqlResource("UserSummaryView");
```

### `getWorkflows()`
Get all registered workflows.
```ts

const workflows = getWorkflows();  // Returns Map<string, Workflow>
```

### `getWorkflow(name)`
Get a specific workflow by name.
```ts

const etlWorkflow = getWorkflow("DailyETL");
```

### `getWebApps()`
Get all registered web apps.
```ts

const webApps = getWebApps();  // Returns Map<string, WebApp>
```

### `getWebApp(name)`
Get a specific web app by name.
```ts

const adminApp = getWebApp("AdminPanel");
if (adminApp) {
  console.log(`Mounted at: ${adminApp.config.mountPath}`);
}
```