The JSON type stores arbitrary JSON data, supporting both unstructured and typed configurations for performance and type safety.
For completely dynamic JSON data without a fixed schema:
from typing import Any, Dict class Event(BaseModel): metadata: Dict[str, Any] # Basic JSON - accepts any structure config: Any # Basic JSON - fully dynamicFor better performance and validation, define typed fields within your JSON using ClickHouseJson. This creates a ClickHouse JSON column with explicit type hints for specific paths while still allowing additional dynamic fields.
from typing import Annotated, Optionalfrom pydantic import BaseModel, ConfigDictfrom moose_lib.data_models import ClickHouseJsonfrom datetime import datetime # Define the structure for your JSON payloadclass PayloadStructure(BaseModel): model_config = ConfigDict(extra='allow') # Required for JSON types name: str count: int timestamp: Optional[datetime] = None class Event(BaseModel): id: str # JSON with typed paths - better performance, allows extra fields payload: Annotated[PayloadStructure, ClickHouseJson()]Configure ClickHouseJson with options to control resource usage:
from typing import Annotated, Optionalfrom pydantic import BaseModel, ConfigDictfrom moose_lib import Key, ClickHouseJsonfrom datetime import datetime class ProductProperties(BaseModel): model_config = ConfigDict(extra='allow') category: str price: float in_stock: bool class ProductEvent(BaseModel): event_id: Key[str] timestamp: datetime properties: Annotated[ProductProperties, ClickHouseJson( max_dynamic_paths=128, # Limit tracked paths max_dynamic_types=8, # Limit type variations skip_paths=("_internal",), # Exclude specific paths skip_regexes=(r"^debug_",) # Exclude paths matching regex )]| Option | Type | Description |
|---|---|---|
max_dynamic_paths | number | Maximum unique JSON paths to track. Controls memory for variable structures. |
max_dynamic_types | number | Maximum type variations per path. Useful when paths may contain different types. |
skip_paths | string[] | Exact JSON paths to ignore (e.g., ["temp", "debug.info"]). |
skip_regexps | string[] | Regex patterns for paths to exclude (e.g., ["^tmp\\.", ".*_internal$"]). |
With typed JSON, you get the best of both worlds—type safety for known fields and flexibility for dynamic data:
# This data is valid:{ "event_id": "evt_123", "timestamp": "2025-10-22T12:00:00Z", "properties": { "category": "electronics", # Typed field ✓ "price": 99.99, # Typed field ✓ "in_stock": True, # Typed field ✓ "custom_tag": "holiday-sale", # Extra field - accepted ✓ "brand_id": 42, # Extra field - accepted ✓ "_internal": "ignored" # Skipped by skip_paths ✓ }}any, Dict[str, Any]): Use when JSON structure is completely unknown or rarely queriedClickHouseJson): Use when you have known fields that need indexing/querying, but want to allow additional dynamic fields