1. MooseStack
  3. Engines
  5. Buffer

In-Memory Buffer (Buffer)

The Buffer engine provides an in-memory buffer that flushes data to a destination table based on time, row count, or size thresholds:

BufferTable.ts
import { OlapTable, ClickHouseEngines } from '@514labs/moose-lib'; // First create the destination tableexport const destinationTable = new OlapTable<Record>("destination", {  engine: ClickHouseEngines.MergeTree,  orderByFields: ["id", "timestamp"]}); // Then create buffer that writes to itexport const bufferTable = new OlapTable<Record>("buffer", {  engine: ClickHouseEngines.Buffer,  targetDatabase: "local",  targetTable: "destination",  numLayers: 16,  minTime: 10,        // Min 10 seconds before flush  maxTime: 100,       // Max 100 seconds before flush  minRows: 10000,     // Min 10k rows before flush  maxRows: 1000000,   // Max 1M rows before flush  minBytes: 10485760, // Min 10MB before flush  maxBytes: 104857600 // Max 100MB before flush});
Buffer Engine Considerations
  • Data in buffer is lost if server crashes before flush
  • Not suitable for critical data that must be durable
  • Best for high-throughput scenarios where minor data loss is acceptable
  • Buffer and destination table must have identical schemas
  • Cannot use orderByFields, partitionBy, or sampleByExpression on buffer tables

For more details, see the ClickHouse Buffer documentation.

FiveonefourFiveonefour
Fiveonefour Docs
MooseStackTemplatesGuides
Release Notes
Source523
  • Overview
Build a New App
  • 5 Minute Quickstart
  • Browse Templates
  • Existing ClickHouse
Add to Existing App
  • Next.js
  • Fastify
Fundamentals
  • Moose Runtime
  • MooseDev MCP
  • Data Modeling
Moose Modules
  • Moose OLAP
  • Moose Streaming
  • Moose Workflows
  • Moose APIs & Web Apps
Deployment & Lifecycle
  • Moose Migrate
  • Moose Deploy
Reference
  • API Reference
  • Data Types
  • Table Engines
  • CLI
  • Configuration
  • Observability Metrics
  • Help
  • Release Notes
Contribution
  • Documentation
  • Framework
BufferTable.ts
import { OlapTable, ClickHouseEngines } from '@514labs/moose-lib'; // First create the destination tableexport const destinationTable = new OlapTable<Record>("destination", {  engine: ClickHouseEngines.MergeTree,  orderByFields: ["id", "timestamp"]}); // Then create buffer that writes to itexport const bufferTable = new OlapTable<Record>("buffer", {  engine: ClickHouseEngines.Buffer,  targetDatabase: "local",  targetTable: "destination",  numLayers: 16,  minTime: 10,        // Min 10 seconds before flush  maxTime: 100,       // Max 100 seconds before flush  minRows: 10000,     // Min 10k rows before flush  maxRows: 1000000,   // Max 1M rows before flush  minBytes: 10485760, // Min 10MB before flush  maxBytes: 104857600 // Max 100MB before flush});