Highlights

• Moose and Aurora users can now embed metadata into their Moose primitives, for use by users, their agents, and their metadata tools.
• Aurora users can read and write from Databricks (more coming here soon).

Aurora MCP

[Experimental] Wrangler Agent—Databricks tools.

We’ve had a bunch of our users (especially in the enterprise) request deeper integration with Databricks. We’ve created MCP tooling to allow you to read from Databricks, and create new derivative Databricks managed tables.

Turn it on with aurora config tools, and adding experimental-databricks-tools. Docs. To connect with your Databricks instance, you’ll need to modify the relevant MCP.json file to add:

"DATABRICKS_HOST": "[-].databricks.com",
"DATABRICKS_PATH": "/sql/1.0/warehouses/[-]",
"DATABRICKS_TOKEN": "[-]",

This allows you to:

• Read from Databricks tables
• Create queries and run them against Databricks tables
• Create new Databricks managed tables

Future of the feature:

• Workflows V2: We’re working on bringing schemas into our workflow creation tools, our Databricks integration will be able to leverage these in interacting with Databricks.
• DatabricksTable: We’re working on a new primitive that will allow you to create a Databricks table from a Moose primitive.

Moose + Aurora

Descriptions in your Moose primitives

Store context about why you are building what you are building, for you and your agents.

Moose primitives can now include descriptions.

const acPipeline = new IngestPipeline<AircraftTrackingProcessed>(
    "AircraftTrackingProcessed",
    {
      table: true,
      stream: true,
      ingest: false,
    },
    { description: "Pipeline for ingesting raw aircraft data" } // new description field!
);

Where is this used? Aurora tools that create Moose primitives will now write the intention of the Moose primitive into the description field to give tools/agents that work with that primitive in the future more context. Practically, every description is now served to the tools as context too when the infra map is loaded up as context.

Future of the feature: Two main extensions of this feature are planned:

• Embedding the descriptions into the underlying infrastructure (e.g. as table level comments in ClickHouse)
• Extending the metadata:
  • To be on a field level as well as a primitive level
  • To include any arbitrary key value pair, not just a description (use this for managing labels like PII!)