Moose Changelog

Bug fixes and reliability improvements.

• S3Queue Engine Configuration — Added complete support for all ClickHouse S3Queue engine parameters including authentication (aws_access_key_id, aws_secret_access_key), compression, custom headers, and NOSIGN for public buckets. PR #2674
• Generic Settings Framework — Introduced a flexible settings system that allows any engine to use configuration settings, laying groundwork for future engine implementations.
• Enhanced Documentation — Added comprehensive documentation for OlapTable S3Queue configuration in both TypeScript and Python SDKs.

• Improved Architecture — Moved ClickHouse-specific types from core infrastructure to ClickHouse module for better separation of concerns.
• Settings Location — Engine-specific settings are now properly encapsulated within their respective engine configurations (e.g., s3QueueEngineConfig.settings for S3Queue).
• API Consistency — Unified configuration APIs across TypeScript and Python SDKs for S3Queue engine.

• Compilation Issues — Fixed struct patterns to handle new S3Queue parameter structure correctly.
• Diff Strategy — Enhanced diff strategy to properly handle S3Queue parameter changes.

• IngestPipeline Parameter Renamed — The ingest parameter in IngestPipeline configurations has been renamed for clarity:

  • TypeScript: ingest: true → ingestApi: true
  • Python: ingest=True → ingest_api=True

  The old parameter names continue to work with deprecation warnings to ensure backwards compatibility. [Current PR]

• TypeScript workflow Task types now properly support optional input/output with null types, enabling more flexible task definitions like Task<Foo, null> and Task<null, Bar>.
  #2442 by DatGuyJonathan

• OlapTable direct insert API with insert() method supporting arrays and Node.js streams. Features comprehensive typia-based validation, three error handling strategies (fail-fast, discard, isolate), configurable error thresholds, memoized ClickHouse connections, and detailed insertion results with failed record tracking.
  #2437 by callicles
• Enhanced typia validation integration for OlapTable and IngestPipeline with validateRecord(), isValidRecord(), and assertValidRecord() methods providing compile-time type safety and runtime validation.
  #2437 by callicles
• Python Workflows V2 with Task[InputType, OutputType] and Workflow classes for dynamic workflow orchestration. Replaces the legacy @task decorator approach with explicit task definitions, enabling flexible task composition, type-safe chaining via on_complete, retries, timeouts, and scheduling with cron expressions.
  #2439 by

• TypeScript Workflows V2 with Task<InputType, OutputType> and Workflow classes for dynamic workflow orchestration. Replaces the legacy file-based routing approach with explicit task definitions, enabling flexible task composition, type-safe chaining via onComplete, configurable retries and timeouts, and flexible scheduling with cron expressions.
  #2421 by DatGuyJonathan

• TypeScript DeadLetterQueue<T> class with type guards and transform methods for handling failed streaming function messages with full type safety.
  #2356 by phiSgr

• Renamed DeadLetterModel.as_t() to DeadLetterModel.as_typed() in Python for better API clarity and consistency.
  #2356 by phiSgr

• Updated CLI 'peek' command to use a unified 'name' parameter and new flags (--table, --stream) to specify resource type. Default is table. Documentation updated to match.
  #2361 by callicles

• State reconciliation for infrastructure planning — Moose now checks and updates its in-memory infra map to match the real database state before planning changes. Makes infra planning robust to manual DB changes and failed runs.
  #2341 by callicles

• Handling of nested data structures in Moose models for correct support of complex objects and arrays.
  #2357 by georgevanderson

• IPv4 and IPv6 data types for ClickHouse integration, supporting native IP address storage and operations.
  #2373 by phiSgr
• Enhanced type parser to handle IP address types across the Moose ecosystem.
  #2374 by phiSgr

• ClickHouse Date column support for native date types in Moose schemas and ingestion.
  #2352, #2351 by phiSgr

• Metadata map propagation to infra map for consistent tracking and availability in both Python and TypeScript.
  #2326 by georgevanderson

• AggregateFunction in Python now accepts list[str] for more expressive and type-safe aggregations.
  #2321 by phiSgr

• Python dead letter queues for handling and retrying failed messages in Python streaming functions.
  #2324 by phiSgr

• TypeScript JSON columns to have consistent casing, avoiding confusion and errors in your code.
  #2320 by phiSgr

• TypeScript ClickHouse JSON columns to use Record<string, any> for type-safe JSON fields. #2317 by phiSgr
• Pydantic mixin for parsing integer enums by name for more robust Python data models.
  #2316 by phiSgr
• Better Python enum handling in data models for easier enum usage.
  #2315 by phiSgr

• refresh command to reload data and schema with a single command.
  #2309 by phiSgr
• Python support for LowCardinality(T) to improve performance for categorical columns.
  #2313 by phiSgr

• Order changes for materialized views based on dependency to ensure correct execution order for dependent changes.
  #2294 by callicles

• Row parsing in QueryClient with type mapping for Python.
  #2299 by phiSgr

• datetime64 parsing and row parsing in QueryClient for more reliable data handling in Python.
  #2299 by phiSgr

• uint type support in TypeScript for unsigned integers in Moose models.
  #2295 by phiSgr

• Explicit dependency tracking for materialized views to make migrations and data lineage more robust and easier to understand.
  #2282 by callicles

• JSON_ARRAY to allow both array and single element ingestion for more flexible data handling.
  #2285 by phiSgr

• Python rich ClickHouse type support with:
  • Decimal: clickhouse_decimal(precision, scale)
  • datetime with precision: clickhouse_datetime64(precision)
  • date: date
  • int with size annotations: Annotated[int, 'int8'], Annotated[int, 'int32'], etc.
  • UUID: UUID for more expressive data modeling.
    #2284 by phiSgr