As always, to get the latest:

bash -i <(curl -fsSL https://fiveonefour.com/install.sh) moose,aurora

Friday, 30th May 2025

Aurora

Aurora connect ClickHouse

aurora connect clickhouse --connection-string "https://explorer:@play.clickhouse.com:443/?database=default" --mcp cursor-project

See connect docs. One line configuration to connect your Client of choice to your ClickHouse database with Aurora’s MCP tools. Choose your MCP client of choice with the --mcp flag.

https://explorer:@play.clickhouse.com:443/?database=default

Friday, 23rd May 2025

Moose

Dead Letter Queue Support

No more pipeline failures from bad data.

You had to choose between strict data validation that could break your entire pipeline or loose validation that let bad data through. With Moose Dead Letter Queues, you get both reliability and data quality. When your stream processing hits malformed data, network failures, or validation errors, Moose automatically routes failed messages to a configured Dead Letter Queue. Your pipeline keeps running while you capture every failure for analysis and recovery.

TypeScript Setup:

import { DeadLetterQueue, IngestPipeline } from "@514labs/moose-lib";
 
interface UserEvent {
  userId: string;
  action: string;
  timestamp: number;
}
 
// Create dead letter queue for failed transformations
const eventDLQ = new DeadLetterQueue<UserEvent>("EventDLQ");
 
// Set up your data pipelines
const rawEvents = new IngestPipeline<UserEvent>("raw_events", {
  ingest: true,
  stream: true,
  table: false
});
 
const processedEvents = new IngestPipeline<ProcessedEvent>("processed_events", {
  ingest: false,
  stream: true,
  table: true
});
 
// Configure transform with error handling
rawEvents.stream!.addTransform(
  processedEvents.stream!,
  (event: UserEvent) => {
    // Your transformation logic that might fail
    if (!event.userId) {
      throw new Error("Invalid userId");
    }
    return { ...event, processedAt: new Date() };
  },
  { deadLetterQueue: eventDLQ }  // Failed messages go here
);
 
// Monitor and recover from failures
eventDLQ.addConsumer((deadLetter) => {
  const originalEvent: UserEvent = deadLetter.asTyped();
  console.log(`Error: ${deadLetter.errorMessage}`);
  // Implement recovery logic
});

Python Setup:

from moose_lib import DeadLetterQueue, IngestPipeline, TransformConfig
from pydantic import BaseModel
from datetime import datetime
 
class UserEvent(BaseModel):
    user_id: str
    action: str
    timestamp: float
 
# Create dead letter queue
event_dlq = DeadLetterQueue[UserEvent](name="EventDLQ")
 
# Set up your data pipelines
raw_events = IngestPipeline[UserEvent]("raw_events", {
    "ingest": True,
    "stream": True,
    "table": False
})
 
processed_events = IngestPipeline[ProcessedEvent]("processed_events", {
    "ingest": False,
    "stream": True,
    "table": True
})
 
def process_event(event: UserEvent) -> ProcessedEvent:
    # Your transformation logic that might fail
    if not event.user_id:
        raise ValueError("Invalid user_id")
    return ProcessedEvent(
        user_id=event.user_id,
        action=event.action,
        processed_at=datetime.now()
    )
 
# Configure transform with error handling
raw_events.get_stream().add_transform(
    destination=processed_events.get_stream(),
    transformation=process_event,
    config=TransformConfig(dead_letter_queue=event_dlq)
)
 
# Monitor and recover from failures
def monitor_failures(dead_letter):
    original_event = dead_letter.as_typed()
    print(f"Error: {dead_letter.error_message}")
    # Implement recovery logic
 
event_dlq.add_consumer(monitor_failures)

This means you can eliminate downtime from bad data, debug failures faster with full error context, and build self-healing systems with automated recovery patterns. You get comprehensive error metadata including the original record, error message, error type, timestamp, and failure source. Access your original typed data with the asTyped() method to build recovery workflows.

Coming next:

• IngestApi integration - HTTP endpoints will route validation failures to DLQs with client notifications
• OlapTable direct insert - New batch insert APIs with DLQ support for failed operations
• Built-in metrics - DLQ volume, error types, and recovery rates
• Retention policies - Configurable cleanup for dead letter messages

See Dead Letter Queues documentation for complete setup and recovery patterns.

Monday, 19th May 2025

Moose + Aurora

Descriptions in your Moose primitives

Store context about why you are building what you are building, for you and your agents.

Moose primitives can now include descriptions.

const acPipeline = new IngestPipeline<AircraftTrackingProcessed>(
    "AircraftTrackingProcessed",
    {
      table: true,
      stream: true,
      ingest: false,
      metadata: {
          description: "Pipeline for ingesting raw aircraft data" } // new description field!
      }
    },
);

Where is this used? Aurora tools that create Moose primitives will now write the intention of the Moose primitive into the description field to give tools/agents that work with that primitive in the future more context. Practically, every description is now served to the tools as context too when the infra map is loaded up as context.

Future of the feature: Two main extensions of this feature are planned:

• Embedding the descriptions into the underlying infrastructure (e.g. as table level comments in ClickHouse)
• Extending the metadata:
  • To be on a field level as well as a primitive level
  • To include any arbitrary key value pair, not just a description (use this for managing labels like PII!)

Friday, 16th May 2025

Moose + Aurora

Quality of life and bug fixes:

• Flexible JSON ingest. Our agents were confused by needing to specify whether ingested data was a JSON object or an array of JSON objects. Now by default both work.

Aurora MCP

[External] DuckDB MCP

You can now bring DuckDB’s MCP tool into a Moose Project and manage that MCP through the Aurora CLI! Try aurora config tools and enable external-duck-db-mcp. Docs. You’ll need to configure the MCP with your duckdb path—either after selectign the tool-set through aurora config tools, or by editing the MCP.json file directly.

String DuckDB tool calls with Aurora tool calls to manipulate local files for ingestion. Try our Aurora tools for creating a DuckDB database from your local flat files.

Future of the feature:

• Remote Data: We’re working on bringing remote data into Moose management with this MCP toolset

Data Collection

Aurora is in research preview, to help us improve our product, we’ve added data collection to our MCP tools. Ready-to-go templates default to comprehensive data collection (we figure that these templates are least likely to have sensitive information), and standard templates default to standard data collection. You can change your data collection preferences whenever you want (just by editing your project’s aurora.config.toml file. Docs.

Aurora CLI

Quality of life and bug fixes:

• LLM Docs. Want docs for your LLMs that work with Moose projects? Here.
• Read only tools. We’ve had requests for a read-only toolset, especially for people using chat based MCP clients. We’ve split standard tools into: read-only and egress-tools. Docs.
• Sorting Projects. Projects in the CLI are now sorted (hopefully y’all have enough Aurora projects that this is useful)! Affects aurora config tools and structure of ~/.aurora/config.toml. Docs. Deleted projects will now be cleaned from ~/.aurora/config.toml and from whenever the CLI prints lists of projects.
• 🐛Read tools now only read. ClickHouse read tool was doing a little more than read, especially when used by Claude. Tool has been tightened up, and if you want extra certainty, create a read-only user (see snippet below).

CREATE USER username IDENTIFIED BY 'password' SETTINGS PROFILE 'readonly'
GRANT SHOW TABLES, SELECT ON db+name.* TO username

In Progress

Let us know if these features are interesting to you, we are always looking for beta users, design partners, and general feedback!

• Google Docs MCP integration. Get data from Google Docs / Sheets for use as context, or as a source!
• First Party Chat. Chat with your data from our hosting platform, Boreal.

Sneak Peek: First Party Chat

Actively seeking feedback on this one!

A user can start a chat anywhere in Boreal—in a project, on a table view, wherever they might be. Where they start the chat informs:

1. the context that is available to the chat
2. the actions that the chat window suggests that the user takes.

Here’s an example of autocomplete—the chat suggests relevant tables and files to use as context. You can accept the suggestion, select specific context to add with “tab”. You can add other context with ”@”.

Here’s an example of chat with tool-calls.

The user can create persistent artifacts in chat (not static, they receive data from APIs created by the MCP tools called in the chat)

The user can then deploy and share those artifacts.