# Moose / Deploying Documentation – Python

## Included Files
1. moose/deploying/configuring-moose-for-cloud.mdx
2. moose/deploying/deploying-on-an-offline-server.mdx
3. moose/deploying/deploying-on-ecs.mdx
4. moose/deploying/deploying-on-kubernetes.mdx
5. moose/deploying/deploying-with-docker-compose.mdx
6. moose/deploying/monitoring.mdx
7. moose/deploying/packaging-moose-for-deployment.mdx
8. moose/deploying/preparing-clickhouse-redpanda.mdx

## Configuring Moose for cloud environments

Source: moose/deploying/configuring-moose-for-cloud.mdx

Configuring Moose for cloud environments

# Configuring Moose for cloud environments

In the [Packaging Moose for deployment](packaging-moose-for-deployment.mdx) page, we looked at how to package your moose 
application into Docker containers (using the `moose build —-docker` command), and you've pushed them to your container repository.

We can connect and configure your container image with remote ClickHouse and Redis-hosted services. You can also optionally 
use Redpanda for event streaming and Temporal for workflow orchestration.

The methods used to accomplish this are generally similar, but the specific details depend on your target cloud infrastructure.
So, we'll look at the overarching concepts and provide some common examples.

## Specifying your repository container

Earlier, we created two local containers and pushed them to a docker repository.

```txt filename="Terminal" copy
>docker images
REPOSITORY                                              TAG               IMAGE ID       CREATED              SIZE
moose-df-deployment-aarch64-unknown-linux-gnu           0.3.175           c50674c7a68a   About a minute ago   155MB
moose-df-deployment-x86_64-unknown-linux-gnu            0.3.175           e5b449d3dea3   About a minute ago   163MB
```

We pushed the containers to the `514labs` Docker Hub account. So, we have these two containers available for use:

```
514labs/moose-df-deployment-aarch64-unknown-linux-gnu:0.3.175
514labs/moose-df-deployment-x86_64-unknown-linux-gnu:0.3.175
```

In later examples, we'll use an AMD64 (x86_64) based machine, so we'll stick to using the following container image: `514labs/moose-df-deployment-x86_64-unknown-linux-gnu:0.3.175`

We'll also examine how the container image name can be used in various cloud providers and scenarios.

## General overview

The general approach is to use a cloud provider that supports specifying a container image to launch your application. Examples include the Google Kubernetes Engine (GKE), Amazon's Elastic Kubernetes Service (EKS), and Elastic Container Service (ECS). Each provider also offers a way of configuring container environment variables that your container application will have access to.

## Essential Environment Variables

Based on our production deployments, here are the essential environment variables you'll need to configure for your Moose application in cloud environments:

### Logging and Telemetry

```
# Logger configuration
MOOSE_LOGGER__LEVEL=Info
MOOSE_LOGGER__STDOUT=true
MOOSE_LOGGER__FORMAT=Json

# Telemetry configuration
MOOSE_TELEMETRY__ENABLED=false
MOOSE_TELEMETRY__EXPORT_METRICS=true

# For debugging
RUST_BACKTRACE=1
```

### HTTP Server Configuration

```
# HTTP server settings
MOOSE_HTTP_SERVER_CONFIG__HOST=0.0.0.0
MOOSE_HTTP_SERVER_CONFIG__PORT=4000
```

### External Service Connections

For detailed configuration of the external services, refer to the [Preparing ClickHouse and Redpanda](preparing-clickhouse-redpanda.mdx) page.

#### ClickHouse

```
MOOSE_CLICKHOUSE_CONFIG__DB_NAME=<your_database_name>
MOOSE_CLICKHOUSE_CONFIG__USER=<your_clickhouse_user>
MOOSE_CLICKHOUSE_CONFIG__PASSWORD=<your_clickhouse_password>
MOOSE_CLICKHOUSE_CONFIG__HOST=<your_clickhouse_host>
MOOSE_CLICKHOUSE_CONFIG__HOST_PORT=8443
MOOSE_CLICKHOUSE_CONFIG__USE_SSL=1
MOOSE_CLICKHOUSE_CONFIG__NATIVE_PORT=9440
```

#### Redis

Moose requires Redis for caching and message passing:

```
MOOSE_REDIS_CONFIG__URL=<your_redis_url>
MOOSE_REDIS_CONFIG__KEY_PREFIX=<your_key_prefix>
```

#### Redpanda (Optional)

If you choose to use Redpanda for event streaming:

```
MOOSE_REDPANDA_CONFIG__BROKER=<your_redpanda_broker>
MOOSE_REDPANDA_CONFIG__NAMESPACE=<your_namespace>
MOOSE_REDPANDA_CONFIG__MESSAGE_TIMEOUT_MS=10043
MOOSE_REDPANDA_CONFIG__SASL_USERNAME=<your_redpanda_username>
MOOSE_REDPANDA_CONFIG__SASL_PASSWORD=<your_redpanda_password>
MOOSE_REDPANDA_CONFIG__SASL_MECHANISM=SCRAM-SHA-256
MOOSE_REDPANDA_CONFIG__SECURITY_PROTOCOL=SASL_SSL
MOOSE_REDPANDA_CONFIG__REPLICATION_FACTOR=3
```

#### Temporal (Optional)

If you choose to use Temporal for workflow orchestration:

```
MOOSE_TEMPORAL_CONFIG__CA_CERT=/etc/ssl/certs/ca-certificates.crt
MOOSE_TEMPORAL_CONFIG__API_KEY=<your_temporal_api_key>
MOOSE_TEMPORAL_CONFIG__TEMPORAL_HOST=<your_temporal_namespace>.tmprl.cloud
```

## Securing Sensitive Information

When deploying to cloud environments, it's important to handle sensitive information like passwords and API keys securely. Each cloud provider offers mechanisms for this:

- **Kubernetes**: Use Secrets to store sensitive data. See our [Kubernetes deployment guide](deploying-on-kubernetes.mdx) for examples.
- **Amazon ECS**: Use AWS Secrets Manager or Parameter Store to securely inject environment variables.
- **Other platforms**: Use the platform's recommended secrets management approach.

Never hardcode sensitive values directly in your deployment configuration files.

Please share your feedback about Moose monitoring capabilities through [our GitHub repository](https://github.com/514-labs/moose/issues/new?title=Feedback%20for%20%E2%80%9CMonitoring%E2%80%9D&labels=feedback).

---

## Deploying on an offline server

Source: moose/deploying/deploying-on-an-offline-server.mdx

Deploying on an offline server

# Building and Deploying Moose Applications

This guide will walk you through the process of building a Moose application and deploying it to a server that does not have internet access. 
We'll cover both the build environment setup and the deployment environment requirements.

## Build Environment Setup

### Prerequisites

Before you can build a Moose application, you need to set up your build environment with the following dependencies:

OS:
- Debian 10+
- Ubuntu 18.10+
- Fedora 29+
- CentOS/RHEL 8+
- Amazon Linux 2023+
- Mac OS 13+

Common CLI utilities:
- zip
- curl (optional, for installing the Moose CLI)

Python build environment requirements:
1. Python 3.12 or later (we recommend using pyenv for Python version management)
2. pip 

### Setting up the Build Environment

First, install the required system dependencies:

```bash
sudo apt update
sudo apt install build-essential libssl-dev zlib1g-dev libbz2-dev \
  libreadline-dev libsqlite3-dev curl git libncursesw5-dev xz-utils \
  tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
```

Install pyenv and configure your shell:

```bash
curl -fsSL https://pyenv.run | bash
```

Add the following to your `~/.bashrc` or `~/.zshrc`:

```bash
export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
```

Install and set Python 3.12:

```bash
pyenv install 3.12
pyenv global 3.12
```

Verify the installation:
```bash
python --version
```

### Installing Moose CLI (Optional)

You can install the Moose CLI using the official installer:

```bash
curl -SfsL https://fiveonefour.com/install.sh | bash -s -- moose
source ~/.bashrc  # Or restart your terminal
```
or

```bash
pip install moose-cli
```

## Building Your Application

### 1. Initialize a New Project (Optional)
This step is optional if you already have a Moose project.
Create a new Moose project:

```bash
moose init your-project-name py
cd your-project-name
```

### 2. Build the Application

  Make sure you have the `zip` utility installed (`sudo apt install zip`) before building your application.

if you installed the moose cli to be available globally, you can build the application with the following command:
```bash
moose build
```
Or if you installed the moose cli to be available locally, you can build the application with the following command:

The build process will create a deployable package:

```bash
moose build
```

This will create a zip file in your project directory with a timestamp, for example: `your-project-name-YYYY-MM-DD.zip`

## Deployment Environment Setup

### Prerequisites

The deployment server requires:

1. Python 3.12 or later
3. Unzip utility

### Setting up the Deployment Environment

1. Install the runtime environment:

Follow the Python installation steps from the build environment setup section.

2. Install the unzip utility:
```bash
sudo apt install unzip
```

## Deploying Your Application

1. Copy your built application package to the deployment server

2. Extract the application:
```bash
unzip your-project-name-YYYY-MM-DD.zip -d ./app
cd ./app/packager
```

3. Start your application:

```bash
moose prod
```

  Ensure all required environment variables and configurations are properly set before starting your application.

## Troubleshooting

- Verify that Python is properly installed using `python --version`
- Check that your application's dependencies are properly listed in `requirements.txt`
- If you encounter Python import errors, ensure your `PYTHONPATH` is properly set

---

## Deploying on Amazon ECS

Source: moose/deploying/deploying-on-ecs.mdx

Deploying on Amazon ECS

# Deploying on Amazon ECS

Moose can be deployed to Amazon's Elastic Container Service (ECS). ECS offers a managed container orchestrator at a fraction of the complexity of managing a Kubernetes cluster.

If you're relatively new to ECS we recommend the following resources:

- [Amazon Elastic Container Service (ECS) with a Load Balancer | AWS Tutorial with New ECS Experience](https://www.youtube.com/watch?v=rUgZNXKbsrY)
- [Tutorial: Deploy NGINX Containers On ECS Fargate with Load Balancer](https://bhaveshmuleva.hashnode.dev/tutorial-deploy-nginx-containers-on-ecs-fargate-with-load-balancer)
- [How to configure target groups ports with listeners and tasks](https://stackoverflow.com/questions/66275574/how-to-configure-target-groups-ports-with-listeners-and-tasks)

The first step is deciding whether you'll host your Moose container on Docker Hub or Amazon's Elastic Container Registry (ECR).
Amazon ECR is straightforward and is designed to work out of the box with ECS. Using Docker Hub works if your moose container is publicly available; however,
 if your container is private, you'll need to do a bit more work to provide ECS with your Docker credentials.

> See: [Authenticating with Docker Hub for AWS Container Services](https://aws.amazon.com/blogs/containers/authenticating-with-docker-hub-for-aws-container-services/)

Here is an overview of the steps required:

1. You'll first need to create or use an existing ECS cluster.
2. Then, you'll need to create an ECS `Task definition.` This is where you'll specify whether you want to use AWS Fargate or AWS EC2 instances. 
   You'll also have options for selecting your OS and Architecture. Specify `Linux/X86-64` or `Linux/ARM-64`. This is important as you'll also need to 
   specify a matching moose container image, such as `moose-df-deployment-x86_64-unknown-linux-gnu:0.3.175` or `moose-df-deployment-aarch64-unknown-linux-gnu:0.3.175`
3. As with all AWS services, if you're using secrets to store credentials, you will need to specify an IAM role with an `AmazonECSTaskExecutionRolePolicy` and `SecretsManagerReadWrite`
   policy.
4. Under the Container section, specify the name of your moose deployment and provide the container image name you're using.
5. Next, specify the Container Port as 4000.

## Configuring container environment variables

While still in the Amazon ECS Task definition section, you'll need to provide the environment variables on which your Moose application depends.
Scroll down to the Environment variables section and fill in each of the following variables.

  ClickHouse and Redis are required components for Moose. Redpanda and Temporal are optional - configure them only if you're using these components in your application.

> Note: if you prefer, you can provide the environment variables below via an env file hosted on S3 or using AWS Secrets Manager for sensitive values.

### Core Configuration

| Key | Description | Example Value |
|-----|-------------|---------------|
| MOOSE_LOGGER__LEVEL | Log level | Info |
| MOOSE_LOGGER__STDOUT | Enable stdout logging | true |
| MOOSE_LOGGER__FORMAT | Log format | Json |
| RUST_BACKTRACE | Enable backtraces for debugging | 1 |

### HTTP Server Configuration

| Key | Description | Example Value |
|-----|-------------|---------------|
| MOOSE_HTTP_SERVER_CONFIG__HOST | Your moose network binding address | 0.0.0.0 |
| MOOSE_HTTP_SERVER_CONFIG__PORT | The network port your moose server is using | 4000 |

### ClickHouse Configuration (Required)

| Key | Description | Example Value |
|-----|-------------|---------------|
| MOOSE_CLICKHOUSE_CONFIG__DB_NAME | The name of your ClickHouse database | moose_production |
| MOOSE_CLICKHOUSE_CONFIG__USER | The database user name | clickhouse_user |
| MOOSE_CLICKHOUSE_CONFIG__PASSWORD | The password to your ClickHouse database | (use AWS Secrets Manager) |
| MOOSE_CLICKHOUSE_CONFIG__HOST | The hostname for your ClickHouse database | your-clickhouse.cloud.example.com |
| MOOSE_CLICKHOUSE_CONFIG__HOST_PORT | The HTTPS port for your ClickHouse database | 8443 |
| MOOSE_CLICKHOUSE_CONFIG__USE_SSL | Whether your database connection requires SSL | 1 |
| MOOSE_CLICKHOUSE_CONFIG__NATIVE_PORT | The native port for your ClickHouse database | 9440 |

### Redis Configuration (Required)

| Key | Description | Example Value |
|-----|-------------|---------------|
| MOOSE_REDIS_CONFIG__URL | Redis connection URL | redis://user:password@redis.example.com:6379 |
| MOOSE_REDIS_CONFIG__KEY_PREFIX | Prefix for Redis keys to isolate namespaces | moose_production |

### Redpanda Configuration (Optional)

| Key | Description | Example Value |
|-----|-------------|---------------|
| MOOSE_REDPANDA_CONFIG__BROKER | The hostname for your Redpanda instance | seed-5fbcae97.example.redpanda.com:9092 |
| MOOSE_REDPANDA_CONFIG__NAMESPACE | Namespace for isolation | moose_production |
| MOOSE_REDPANDA_CONFIG__MESSAGE_TIMEOUT_MS | The message timeout delay in milliseconds | 10043 |
| MOOSE_REDPANDA_CONFIG__SASL_USERNAME | Your Redpanda user name | redpanda_user |
| MOOSE_REDPANDA_CONFIG__SASL_PASSWORD | Your Redpanda password | (use AWS Secrets Manager) |
| MOOSE_REDPANDA_CONFIG__SASL_MECHANISM | SASL mechanism | SCRAM-SHA-256 |
| MOOSE_REDPANDA_CONFIG__SECURITY_PROTOCOL | The Redpanda security protocol | SASL_SSL |
| MOOSE_REDPANDA_CONFIG__REPLICATION_FACTOR | Topic replication factor | 3 |

### Temporal Configuration (Optional)

| Key | Description | Example Value |
|-----|-------------|---------------|
| MOOSE_TEMPORAL_CONFIG__CA_CERT | Path to CA certificate | /etc/ssl/certs/ca-certificates.crt |
| MOOSE_TEMPORAL_CONFIG__API_KEY | Temporal Cloud API key | (use AWS Secrets Manager) |
| MOOSE_TEMPORAL_CONFIG__TEMPORAL_HOST | Temporal Cloud namespace host | your-namespace.tmprl.cloud |

Consider using a value of greater than 1000ms (1 second) for the Redpanda message timeout delay if you're using a hosted Redpanda cloud service.

Review other options on the Task Creation page and press the `Create` button when ready.

## Using AWS Secrets Manager

For sensitive information like passwords and API keys, we recommend using AWS Secrets Manager. To configure a secret:

1. Go to AWS Secrets Manager and create a new secret
2. Choose "Other type of secret" and add key-value pairs for your secrets
3. Name your secret appropriately (e.g., `moose/production/credentials`)
4. In your ECS task definition, reference the secret:
   - For environment variables, select "ValueFrom" and enter the ARN of your secret with the key name
   - Example: `arn:aws:secretsmanager:region:account:secret:moose/production/credentials:MOOSE_CLICKHOUSE_CONFIG__PASSWORD::`

## Building an ECS Service

Once you've completed creating an ECS Task, you're ready to create an ECS Service. An ECS Service is a definition that allows you to specify how your cluster will be managed.
Navigate to your cluster's Service page and press the `Create` button to create your new Moose service.
The section we're interested in is the `Deployment configuration` section. There, you'll specify the Task Definition you created earlier. You can also specify the name 
of your service—perhaps something creative like `moose-service`—and the number of tasks to launch.

  Note at this time, we recommend that you only launch a single instance of
  Moose in your cluster. We're currently developing for multi-instance
  concurrent usage.

The remaining sections on the create service page allow you to specify networking considerations and whether you'll use a load balancer.

You can press the `Create` button to launch an instance of your new ECS Moose service.

## Setting up health checks

Your generated Moose containers include a health check endpoint at `/health` that should be configured in your ECS service. We recommend configuring the following health check settings:

### Container-level Health Check

In your task definition's container configuration:

```
healthCheck:
  command: ["CMD-SHELL", "curl -f http://localhost:4000/health || exit 1"]
  interval: 30
  timeout: 5
  retries: 3
  startPeriod: 60
```

### Load Balancer Health Check

If you're using an Application Load Balancer:

1. Create a target group for your service
2. Set the health check path to `/health`
3. Configure appropriate health check settings:
   - Health check protocol: HTTP
   - Health check port: 4000
   - Health check path: /health
   - Healthy threshold: 2
   - Unhealthy threshold: 2
   - Timeout: 5 seconds
   - Interval: 15 seconds
   - Success codes: 200

These health check configurations ensure that your Moose service is properly monitored and that traffic is only routed to healthy containers.

---

## Deploying on Kubernetes

Source: moose/deploying/deploying-on-kubernetes.mdx

Deploying on Kubernetes

# Deploying on Kubernetes

Moose applications can be deployed to Kubernetes clusters, whether it's your own on-prem
cluster or through a cloud service like Google's Kubernetes Engine (GKE) or Amazon's
Elastic Kubernetes Service (EKS).

  Note at this time, we recommend that you only launch a single instance of
  moose in one cluster. We're currently developing for multi-instance concurrent
  usage.

    Essentially you'll need to create a moose-deployment YAML file. Here is an example:

```yaml filename="moose-deployment.yaml-fragment" copy
apiVersion: apps/v1
kind: Deployment
metadata:
  name: moosedeployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: moosedeploy
  template:
    metadata:
      labels:
        app: moosedeploy
    spec:
      containers:
        - name: moosedeploy
          image: 514labs/moose-df-deployment-x86_64-unknown-linux-gnu:latest
          ports:
            - containerPort: 4000
```

> Make sure to update the image key above with the location of your repository and image tag.

You may also need to configure a load balancer to route external traffic to your moose ingest points.

```yaml filename="moose-lb-service.yaml" copy
apiVersion: v1
kind: Service
metadata:
  name: moose-service
spec:
  selector:
    app: moosedeploy
  ports:
    - protocol: TCP
      port: 4000
      targetPort: 4000
  type: LoadBalancer
```

Another approach would be to use a service type of `ClusterIP`:

```yaml filename="moose-service.yaml" copy
apiVersion: v1
kind: Service
metadata:
  name: moose-service
spec:
  selector:
    app: moosedeploy
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 4000
      targetPort: 4000
```

The approach you decide on will depend on your specific Kubernetes networking requirements.

## Setting up health checks and probes

Your generated Moose docker containers feature a health check endpoint at `/health` that can be used by Kubernetes to monitor the health of your application. Based on our production deployment, we recommend configuring the following probes:

```yaml
# Startup probe - gives Moose time to initialize before accepting traffic
startupProbe:
  httpGet:
    path: /health
    port: 4000
  initialDelaySeconds: 60
  timeoutSeconds: 3
  periodSeconds: 5
  failureThreshold: 30
  successThreshold: 3

# Readiness probe - determines when the pod is ready to receive traffic
readinessProbe:
  httpGet:
    path: /health
    port: 4000
  initialDelaySeconds: 5
  timeoutSeconds: 3
  periodSeconds: 3
  failureThreshold: 2
  successThreshold: 5

# Liveness probe - restarts the pod if it becomes unresponsive
livenessProbe:
  httpGet:
    path: /health
    port: 4000
  initialDelaySeconds: 5
  timeoutSeconds: 3
  periodSeconds: 5
  failureThreshold: 5
  successThreshold: 1
```

## Zero-downtime deployments with lifecycle hooks

For production deployments, we recommend configuring a preStop lifecycle hook to ensure graceful pod termination during updates:

```yaml
lifecycle:
  preStop:
    exec:
      command: ["/bin/sleep", "60"]
```

This gives the pod time to finish processing in-flight requests before termination. You should also set an appropriate
 `terminationGracePeriodSeconds` value (we recommend 70 seconds) to work with this hook.

## Resource requirements

Based on our production deployments, we recommend the following resource allocation for a standard Moose deployment:

```yaml
resources:
  requests:
    cpu: "1000m"
    memory: "8Gi"
```

You can adjust these values based on your application's specific needs and workload.

## Configuring container environment variables

Inside your `moose-deployment.yaml` file, you will need to add an `env` section for environment variables. 

  The example below includes actual sample values for clarity. In production deployments, you should use Kubernetes secrets for sensitive information as shown in the second example.

  Note that both Redpanda and Temporal are optional. If you're not using these components, you can omit their respective configuration sections.

### Example with hardcoded values (for development/testing only):

  The example below includes actual sample values for clarity. In production deployments, you should use Kubernetes secrets for sensitive information as shown in the second example.

  Note that both Redpanda and Temporal are optional. If you're not using these components, you can omit their respective configuration sections.

```yaml filename="moose-deployment-dev.yaml" copy
apiVersion: apps/v1
kind: Deployment
metadata:
  name: moosedeployment
spec:
  # For zero-downtime deployments
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  replicas: 1
  selector:
    matchLabels:
      app: moosedeploy
  template:
    metadata:
      labels:
        app: moosedeploy
    spec:
      # For graceful shutdowns
      terminationGracePeriodSeconds: 70
      containers:
        - name: moosedeploy
          image: 514labs/moose-df-deployment-x86_64-unknown-linux-gnu:latest
          ports:
            - containerPort: 4000
          # Lifecycle hook to delay pod shutdown
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sleep", "60"]
          # Startup probe
          startupProbe:
            httpGet:
              path: /health
              port: 4000
            initialDelaySeconds: 60
            timeoutSeconds: 3
            periodSeconds: 5
            failureThreshold: 30
            successThreshold: 3
          # Readiness probe
          readinessProbe:
            httpGet:
              path: /health
              port: 4000
            initialDelaySeconds: 5
            timeoutSeconds: 3
            periodSeconds: 3
            failureThreshold: 2
            successThreshold: 5
          # Liveness probe
          livenessProbe:
            httpGet:
              path: /health
              port: 4000
            initialDelaySeconds: 5
            timeoutSeconds: 3
            periodSeconds: 5
            failureThreshold: 5
            successThreshold: 1
          # Resource requirements
          resources:
            requests:
              cpu: "1000m"
              memory: "8Gi"
          env:
            # Logger configuration
            - name: MOOSE_LOGGER__LEVEL
              value: "Info"
            - name: MOOSE_LOGGER__STDOUT
              value: "true"
            - name: MOOSE_LOGGER__FORMAT
              value: "Json"
            # Telemetry configuration
            - name: MOOSE_TELEMETRY__ENABLED
              value: "true"
            - name: MOOSE_TELEMETRY__EXPORT_METRICS
              value: "true"
            # Debugging
            - name: RUST_BACKTRACE
              value: "1"
            # HTTP server configuration
            - name: MOOSE_HTTP_SERVER_CONFIG__HOST
              value: "0.0.0.0"
            - name: MOOSE_HTTP_SERVER_CONFIG__PORT
              value: "4000"
            # ClickHouse configuration
            - name: MOOSE_CLICKHOUSE_CONFIG__DB_NAME
              value: "moose_production"
            - name: MOOSE_CLICKHOUSE_CONFIG__USER
              value: "clickhouse_user"
            - name: MOOSE_CLICKHOUSE_CONFIG__PASSWORD
              value: "clickhouse_password_example"
            - name: MOOSE_CLICKHOUSE_CONFIG__HOST
              value: "your-clickhouse.cloud.example.com"
            - name: MOOSE_CLICKHOUSE_CONFIG__HOST_PORT
              value: "8443"
            - name: MOOSE_CLICKHOUSE_CONFIG__USE_SSL
              value: "1"
            - name: MOOSE_CLICKHOUSE_CONFIG__NATIVE_PORT
              value: "9440"
            # Redis configuration
            - name: MOOSE_REDIS_CONFIG__URL
              value: "redis://redis_user:redis_password_example@redis.example.com:6379"
            - name: MOOSE_REDIS_CONFIG__KEY_PREFIX
              value: "moose_production"
            # Redpanda configuration (Optional)
            - name: MOOSE_REDPANDA_CONFIG__BROKER
              value: "seed-5fbcae97.example.redpanda.com:9092"
            - name: MOOSE_REDPANDA_CONFIG__NAMESPACE
              value: "moose_production"
            - name: MOOSE_REDPANDA_CONFIG__MESSAGE_TIMEOUT_MS
              value: "10043"
            - name: MOOSE_REDPANDA_CONFIG__SASL_USERNAME
              value: "redpanda_user"
            - name: MOOSE_REDPANDA_CONFIG__SASL_PASSWORD
              value: "redpanda_password_example"
            - name: MOOSE_REDPANDA_CONFIG__SASL_MECHANISM
              value: "SCRAM-SHA-256"
            - name: MOOSE_REDPANDA_CONFIG__SECURITY_PROTOCOL
              value: "SASL_SSL"
            - name: MOOSE_REDPANDA_CONFIG__REPLICATION_FACTOR
              value: "3"
            # Temporal configuration (Optional)
            - name: MOOSE_TEMPORAL_CONFIG__CA_CERT
              value: "/etc/ssl/certs/ca-certificates.crt"
            - name: MOOSE_TEMPORAL_CONFIG__API_KEY
              value: "temporal_api_key_example"
            - name: MOOSE_TEMPORAL_CONFIG__TEMPORAL_HOST
              value: "your-namespace.tmprl.cloud"
      imagePullSecrets:
        - name: moose-docker-repo-credentials
```

---

## deploying-with-docker-compose

Source: moose/deploying/deploying-with-docker-compose.mdx

# Deploying with Docker Compose

Deploying a Moose application with all its dependencies can be challenging and time-consuming. You need to properly configure multiple services,
 ensure they communicate with each other, and manage their lifecycle. 

 Docker Compose solves this problem by allowing you to deploy your entire stack with a single command.

This guide shows you how to set up a production-ready Moose environment on a single server using Docker Compose, with proper security, 
monitoring, and maintenance practices.

  This guide describes a single-server deployment. For high availability (HA) deployments, you'll need to:
  - Deploy services across multiple servers
  - Configure service replication and redundancy
  - Set up load balancing
  - Implement proper failover mechanisms

We are also offering an HA managed deployment option for Moose called [Boreal](https://fiveonefour.com/boreal).

## Prerequisites

Before you begin, you'll need:

- Ubuntu 24 or above (for this guide)
- Docker and Docker Compose (minimum version 2.23.1)
- Access to a server with at least 8GB RAM and 4 CPU cores

The Moose stack consists of:
- Your Moose Application
- [ClickHouse](https://clickhouse.com) (required)
- [Redis](https://redis.io) (required)
- [Redpanda](https://redpanda.com) (optional for event streaming)
- [Temporal](https://temporal.io) (optional for workflow orchestration)

## Setting Up a Production Server

### Installing Required Software

First, install Docker on your Ubuntu server:

```bash
# Update the apt package index
sudo apt-get update

# Install packages to allow apt to use a repository over HTTPS
sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Set up the stable repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Update apt package index again
sudo apt-get update

# Install Docker Engine
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
```

Next, install Node.js or Python depending on your Moose application:

```bash
# For Node.js applications
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt-get install -y nodejs

# OR for Python applications
sudo apt-get install -y python3.12 python3-pip
```

### Configure Docker Log Size Limits

To prevent Docker logs from filling up your disk space, configure log rotation:

```bash
sudo mkdir -p /etc/docker
sudo vim /etc/docker/daemon.json
```

Add the following configuration:

```json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "3"
  }
}
```

Restart Docker to apply the changes:

```bash
sudo systemctl restart docker
```

### Enable Docker Non-Root Access

To run Docker commands without sudo:

```bash
# Add your user to the docker group
sudo usermod -aG docker $USER

# Apply the changes (log out and back in, or run this)
newgrp docker
```

### Setting Up GitHub Actions Runner (Optional)

If you want to set up CI/CD automation, you can install a GitHub Actions runner:

1. Navigate to your GitHub repository
2. Go to Settings > Actions > Runners
3. Click "New self-hosted runner"
4. Select Linux and follow the instructions shown

To configure the runner as a service (to run automatically):

```bash
cd actions-runner
sudo ./svc.sh install
sudo ./svc.sh start
```

## Setting up a Foo Bar Moose Application (Optional)
If you already have a Moose application, you can skip this section.
You should copy the moose project to your server and then build the application with the `--docker` flag and get the built image 
on the server.

### Install Moose CLI 
```bash
bash -i <(curl -fsSL https://fiveonefour.com/install.sh) moose
```

### Create a new Moose Application
Please follow the initialization instructions for your language.

```bash
moose init test-ts typescript
cd test-ts
npm install
```

or 

```bash
moose init test-py python
cd test-py
pip install -r requirements.txt
```

### Build the application on AMD64
```bash
moose build --docker --amd64
```

### Build the application on ARM64
```bash
moose build --docker --arm64
```

### Confirm the image was built
```bash
docker images
```

For more information on packaging Moose for deployment, see the full packaging guide.

## Preparing for Deployment

### Create Environment Configuration

First, create a file called `.env` in your project directory to specify component versions:

```bash
# Create and open the .env file
vim .env
```

Add the following content to the `.env` file:

```
# Version configuration for components
POSTGRESQL_VERSION=14.0
TEMPORAL_VERSION=1.22.0
TEMPORAL_UI_VERSION=2.20.0
REDIS_VERSION=7
CLICKHOUSE_VERSION=25.4
REDPANDA_VERSION=v24.3.13
REDPANDA_CONSOLE_VERSION=v3.1.0
```

Additionally, create a `.env.prod` file for your Moose application-specific secrets and configuration:

```bash
# Create and open the .env.prod file
vim .env.prod
```

Add your application-specific environment variables:

```
# Application-specific environment variables
APP_SECRET=your_app_secret
# Add other application variables here
```

## Deploying with Docker Compose

Create a file called `docker-compose.yml` in the same directory:

```bash
# Create and open the docker-compose.yml file
vim docker-compose.yml
```

Add the following content to the file:

```yaml file=./docker-compose.yml
name: moose-stack
volumes:
  # Required volumes
  clickhouse-0-data: null
  clickhouse-0-logs: null
  redis-0: null
  # Optional volumes
  redpanda-0: null
  postgresql-data: null

configs:
  temporal-config:
    # Using the "content" property to inline the config
    content: |
      limit.maxIDLength:
        - value: 255
          constraints: {}
      system.forceSearchAttributesCacheRefreshOnRead:
        - value: true # Dev setup only. Please don't turn this on in production.
          constraints: {}

services:
  # REQUIRED SERVICES

  # ClickHouse - Required analytics database
  clickhouse-0:
    container_name: clickhouse-0
    restart: always
    image: clickhouse/clickhouse-server:${CLICKHOUSE_VERSION}
    volumes:
      - clickhouse-0-data:/var/lib/clickhouse/
      - clickhouse-0-logs:/var/log/clickhouse-server/
    environment:
      # Enable SQL-driven access control and user management
      CLICKHOUSE_ALLOW_INTROSPECTION_FUNCTIONS: 1
      # Default admin credentials
      CLICKHOUSE_USER: admin
      CLICKHOUSE_PASSWORD: adminpassword
      # Disable default user
      CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1
      # Database setup
      CLICKHOUSE_DB: moose
    # Uncomment this if you want to access clickhouse from outside the docker network
    # ports:
    #   - 8123:8123
    #   - 9000:9000
    healthcheck:
      test: wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 30s
    ulimits:
      nofile:
        soft: 262144
        hard: 262144
    networks:
      - moose-network

  # Redis - Required for caching and pub/sub
  redis-0:
    restart: always
    image: redis:${REDIS_VERSION}
    volumes:
      - redis-0:/data
    command: redis-server --save 20 1 --loglevel warning
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - moose-network

  # OPTIONAL SERVICES

  # --- BEGIN REDPANDA SERVICES (OPTIONAL) ---
  # Remove this section if you don't need event streaming
  redpanda-0:
    restart: always
    command:
      - redpanda
      - start
      - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
      # Address the broker advertises to clients that connect to the Kafka API.
      # Use the internal addresses to connect to the Redpanda brokers'
      # from inside the same Docker network.
      # Use the external addresses to connect to the Redpanda brokers'
      # from outside the Docker network.
      - --advertise-kafka-addr internal://redpanda-0:9092,external://localhost:19092
      - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082
      # Address the broker advertises to clients that connect to the HTTP Proxy.
      - --advertise-pandaproxy-addr internal://redpanda-0:8082,external://localhost:18082
      - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081
      # Redpanda brokers use the RPC API to communicate with each other internally.
      - --rpc-addr redpanda-0:33145
      - --advertise-rpc-addr redpanda-0:33145
      # Mode dev-container uses well-known configuration properties for development in containers.
      - --mode dev-container
      # Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system.
      - --smp 1
      - --default-log-level=info
    image: docker.redpanda.com/redpandadata/redpanda:${REDPANDA_VERSION}
    container_name: redpanda-0
    volumes:
      - redpanda-0:/var/lib/redpanda/data
    networks:
      - moose-network
    healthcheck:
      test: ["CMD-SHELL", "rpk cluster health | grep -q 'Healthy:.*true'"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s

  # Optional Redpanda Console for visualizing the cluster
  redpanda-console:
    restart: always
    container_name: redpanda-console
    image: docker.redpanda.com/redpandadata/console:${REDPANDA_CONSOLE_VERSION}
    entrypoint: /bin/sh
    command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console'
    environment:
      CONFIG_FILEPATH: /tmp/config.yml
      CONSOLE_CONFIG_FILE: |
        kafka:
          brokers: ["redpanda-0:9092"]
        # Schema registry config moved outside of kafka section
        schemaRegistry:
          enabled: true
          urls: ["http://redpanda-0:8081"]
        redpanda:
          adminApi:
            enabled: true
            urls: ["http://redpanda-0:9644"]
    ports:
      - 8080:8080
    depends_on:
      - redpanda-0
    healthcheck:
      test: ["CMD", "wget", "--spider", "--quiet", "http://localhost:8080/admin/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s
    networks:
      - moose-network
  # --- END REDPANDA SERVICES ---

  # --- BEGIN TEMPORAL SERVICES (OPTIONAL) ---
  # Remove this section if you don't need workflow orchestration

  # Temporal PostgreSQL database
  postgresql:
    container_name: temporal-postgresql
    environment:
      POSTGRES_PASSWORD: temporal
      POSTGRES_USER: temporal
    image: postgres:${POSTGRESQL_VERSION}
    restart: always
    volumes:
      - postgresql-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U temporal"]
      interval: 10s
      timeout: 5s
      retries: 3
    networks:
      - moose-network

  # Temporal server
  # For initial setup, use temporalio/auto-setup image
  # For production, switch to temporalio/server after first run
  temporal:
    container_name: temporal
    depends_on:
      postgresql:
        condition: service_healthy
    environment:
      # Database configuration
      - DB=postgres12
      - DB_PORT=5432
      - POSTGRES_USER=temporal
      - POSTGRES_PWD=temporal
      - POSTGRES_SEEDS=postgresql
      # Namespace configuration
      - DEFAULT_NAMESPACE=moose-workflows
      - DEFAULT_NAMESPACE_RETENTION=72h
      # Auto-setup options - set to false after initial setup
      - AUTO_SETUP=true
      - SKIP_SCHEMA_SETUP=false
      # Service configuration - all services by default
      # For high-scale deployments, run these as separate containers
      # - SERVICES=history,matching,frontend,worker
      # Logging and metrics
      - LOG_LEVEL=info
      # Addresses
      - TEMPORAL_ADDRESS=temporal:7233
      - DYNAMIC_CONFIG_FILE_PATH=/etc/temporal/config/dynamicconfig/development-sql.yaml
    # For initial deployment, use the auto-setup image
    image: temporalio/auto-setup:${TEMPORAL_VERSION}
    # For production, after initial setup, switch to server image:
    # image: temporalio/server:${TEMPORAL_VERSION}
    restart: always
    ports:
      - 7233:7233
    # Volume for dynamic configuration - essential for production
    configs:
      - source: temporal-config
        target: /etc/temporal/config/dynamicconfig/development-sql.yaml
        mode: 0444
    networks:
      - moose-network
    healthcheck:
      test: ["CMD", "tctl", "--ad", "temporal:7233", "cluster", "health", "|", "grep", "-q", "SERVING"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 30s

  # Temporal Admin Tools - useful for maintenance and debugging
  temporal-admin-tools:
    container_name: temporal-admin-tools
    depends_on:
      - temporal
    environment:
      - TEMPORAL_ADDRESS=temporal:7233
      - TEMPORAL_CLI_ADDRESS=temporal:7233
    image: temporalio/admin-tools:${TEMPORAL_VERSION}
    restart: "no"
    networks:
      - moose-network
    stdin_open: true
    tty: true

  # Temporal Web UI
  temporal-ui:
    container_name: temporal-ui
    depends_on:
      - temporal
    environment:
      - TEMPORAL_ADDRESS=temporal:7233
      - TEMPORAL_CORS_ORIGINS=http://localhost:3000
    image: temporalio/ui:${TEMPORAL_UI_VERSION}
    restart: always
    ports:
      - 8081:8080
    networks:
      - moose-network
    healthcheck:
      test: ["CMD", "wget", "--spider", "--quiet", "http://localhost:8080/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s
  # --- END TEMPORAL SERVICES ---

  # Your Moose application
  moose:
    image: moose-df-deployment-x86_64-unknown-linux-gnu:latest  # Update with your image name
    depends_on:
      # Required dependencies
      - clickhouse-0
      - redis-0
      # Optional dependencies - remove if not using
      - redpanda-0
      - temporal
    restart: always
    environment:
      # Logging and debugging
      RUST_BACKTRACE: "1"
      MOOSE_LOGGER__LEVEL: "Info"
      MOOSE_LOGGER__STDOUT: "true"

      # Required services configuration
      # ClickHouse configuration
      MOOSE_CLICKHOUSE_CONFIG__DB_NAME: "moose"
      MOOSE_CLICKHOUSE_CONFIG__USER: "moose"
      MOOSE_CLICKHOUSE_CONFIG__PASSWORD: "your_moose_password"
      MOOSE_CLICKHOUSE_CONFIG__HOST: "clickhouse-0"
      MOOSE_CLICKHOUSE_CONFIG__HOST_PORT: "8123"

      # Redis configuration
      MOOSE_REDIS_CONFIG__URL: "redis://redis-0:6379"
      MOOSE_REDIS_CONFIG__KEY_PREFIX: "moose"

      # Optional services configuration
      # Redpanda configuration (remove if not using Redpanda)
      MOOSE_REDPANDA_CONFIG__BROKER: "redpanda-0:9092"
      MOOSE_REDPANDA_CONFIG__MESSAGE_TIMEOUT_MS: "1000"
      MOOSE_REDPANDA_CONFIG__RETENTION_MS: "30000"
      MOOSE_REDPANDA_CONFIG__NAMESPACE: "moose"

      # Temporal configuration (remove if not using Temporal)
      MOOSE_TEMPORAL_CONFIG__TEMPORAL_HOST: "temporal:7233"
      MOOSE_TEMPORAL_CONFIG__NAMESPACE: "moose-workflows"

      # HTTP Server configuration
      MOOSE_HTTP_SERVER_CONFIG__HOST: 0.0.0.0
    ports:
      - 4000:4000
    env_file:
      - path: ./.env.prod
        required: true
    networks:
      - moose-network
    healthcheck:
      test: ["CMD-SHELL", "curl -s http://localhost:4000/health | grep -q '\"unhealthy\": \\[\\]' && echo 'Healthy'"]
      interval: 30s
      timeout: 5s
      retries: 10
      start_period: 60s

# Define the network for all services
networks:
  moose-network:
    driver: bridge
```

At this point, don't start the services yet. First, we need to configure the individual services for production use as described in the following sections.

## Configuring Services for Production

### Configuring ClickHouse Securely (Required)

For production ClickHouse deployment, we'll use environment variables to configure users and access control 
(as recommended in the [official Docker image documentation](https://hub.docker.com/r/clickhouse/clickhouse-server)):

1. First, start the ClickHouse container:

```bash
# Start just the ClickHouse container
docker compose up -d clickhouse-0
```

2. After ClickHouse has started, connect to create additional users:

```bash
# Connect to ClickHouse with the admin user
docker exec -it clickhouse-0 clickhouse-client --user admin --password adminpassword

# Create moose application user
CREATE USER moose IDENTIFIED BY 'your_moose_password';
GRANT ALL ON moose.* TO moose;

# Create read-only user for BI tools (optional)
CREATE USER power_bi IDENTIFIED BY 'your_powerbi_password' SETTINGS PROFILE 'readonly';
GRANT SHOW TABLES, SELECT ON moose.* TO power_bi;
```

3. To exit the ClickHouse client, type `\q` and press Enter.

4. Update your Moose environment variables to use the new moose user:

```bash
vim docker-compose.yml
```

```yaml
MOOSE_CLICKHOUSE_CONFIG__USER: "moose"
MOOSE_CLICKHOUSE_CONFIG__PASSWORD: "your_moose_password"
```

5. Remove the following environement variables from the clickhouse service in the docker-compose.yml file:

```yaml
MOOSE_CLICKHOUSE_CONFIG__USER: "admin"
MOOSE_CLICKHOUSE_CONFIG__PASSWORD: "adminpassword"
```

6. For additional security in production, consider using Docker secrets for passwords.

7. Restart the ClickHouse container to apply the changes:

```bash
docker compose restart clickhouse-0
```

8. Verify that the new configuration works by connecting with the newly created user:

```bash
# Connect with the new moose user
docker exec -it moose-stack-clickhouse-0-1 clickhouse-client --user moose --password your_moose_password

# Test access by listing tables
SHOW TABLES FROM moose;

# Exit the clickhouse client
\q
```

If you can connect successfully and run commands with the new user, your ClickHouse configuration is working properly.

### Securing Redpanda (Optional)

For production, it's recommended to restrict external access to Redpanda:

1. Modify your Docker Compose file to remove external access:
   - Use only internal network access for production
   - If needed, use a reverse proxy with authentication for external access

2. For this simple deployment, we'll keep Redpanda closed to the external world with no authentication required, 
   as it's only accessible from within the Docker network.

### Configuring Temporal (Optional)

If your Moose application uses Temporal for workflow orchestration, the configuration above includes all necessary services based on the
[official Temporal Docker Compose examples](https://github.com/temporalio/docker-compose).

If you're not using Temporal, simply remove the Temporal-related services (postgresql, temporal, temporal-ui) 
and environment variables from the docker-compose.yml file.

#### Temporal Deployment Process: From Setup to Production

Deploying Temporal involves a two-phase process: initial setup followed by production operation. Here are step-by-step instructions for each phase:

##### Phase 1: Initial Setup

1. **Start the PostgreSQL database**:

```bash
docker compose up -d postgresql
```

2. **Wait for PostgreSQL to be healthy** (check the status):

```bash
docker compose ps postgresql
```

Look for `healthy` in the output before proceeding.

3. **Start Temporal with auto-setup**:

```bash
docker compose up -d temporal
```

During this phase, Temporal's auto-setup will:
- Create the necessary PostgreSQL databases
- Initialize the schema tables
- Register the default namespace (moose-workflows)

4. **Verify Temporal server is running**:

```bash
docker compose ps temporal
```

5. **Start the Admin Tools and UI**:

```bash
docker compose up -d temporal-admin-tools temporal-ui
```

6. **Create the namespace manually**:

```bash
# Register the moose-workflows namespace with a 3-day retention period
docker compose exec temporal-admin-tools tctl namespace register --retention 72h moose-workflows
```

Verify that the namespace was created:

```bash
# List all namespaces
docker compose exec temporal-admin-tools tctl namespace list

# Describe your namespace
docker compose exec temporal-admin-tools tctl namespace describe moose-workflows
```

You should see details about the namespace including its retention policy.

##### Phase 2: Transition to Production

After successful initialization, modify your configuration for production use:

1. **Stop Temporal services**:

```bash
docker compose stop temporal temporal-ui temporal-admin-tools
```

2. **Edit your docker-compose.yml file** to:
   - Change image from `temporalio/auto-setup` to `temporalio/server`
   - Set `SKIP_SCHEMA_SETUP=true`

Example change:
```yaml
# From:
image: temporalio/auto-setup:${TEMPORAL_VERSION}

# To:
image: temporalio/server:${TEMPORAL_VERSION}

# And change:
- AUTO_SETUP=true
- SKIP_SCHEMA_SETUP=false

# To:
- AUTO_SETUP=false
- SKIP_SCHEMA_SETUP=true
```

3. **Restart services with production settings**:

```bash
docker compose up -d temporal temporal-ui temporal-admin-tools
```

4. **Verify services are running with new configuration**:

```bash
docker compose ps
```

## Starting and Managing the Service

### Starting the Services

Start all services with Docker Compose:

```bash
docker compose up -d
```

### Setting Up Systemd Service for Docker Compose

For production, create a systemd service to ensure Docker Compose starts automatically on system boot:

1. Create a systemd service file:

```bash
sudo vim /etc/systemd/system/moose-stack.service
```

2. Add the following configuration (adjust paths as needed):

```
[Unit]
Description=Moose Stack
Requires=docker.service
After=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/path/to/your/compose/directory
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target
```

3. Enable and start the service:

```bash
sudo systemctl enable moose-stack.service
sudo systemctl start moose-stack.service
```

## Deployment Workflow

You get a smooth deployment process with these options:

### Automated Deployment with CI/CD

1. Set up a CI/CD pipeline using GitHub Actions (if runner is configured)
2. When code is pushed to your repository:
   - The GitHub Actions runner builds your Moose application
   - Updates the Docker image
   - Deploys using Docker Compose

### Manual Deployment

Alternatively, for manual deployment:

1. Copy the latest version of the code to the machine
2. Run `moose build`
3. Update the Docker image tag in your docker-compose.yml
4. Restart the stack with `docker compose up -d`

## Monitoring and Maintenance

No more worrying about unexpected outages or performance issues. Set up proper monitoring:

- Set up log monitoring with a tool like [Loki](https://grafana.com/oss/loki/)
- Regularly backup your volumes (especially ClickHouse data)
- Monitor disk space usage
- Set up alerting for service health

---

## Monitoring your Moose App

Source: moose/deploying/monitoring.mdx

This content has moved to the unified Observability page

> This page has moved. See the unified [/moose/metrics](/moose/metrics) page for observability across development and production.

---

## Packaging Moose for deployment

Source: moose/deploying/packaging-moose-for-deployment.mdx

Packaging Moose for deployment

# Packaging Moose for Deployment

Once you've developed your Moose application locally, you can package it for deployment to your on-prem or cloud infrastructure.

The first step is to navigate (`cd`) to your moose project in your terminal.

```txt filename="Terminal" copy
cd my-moose-project
```

The Moose CLI you've used to build your Moose project also has a handy flag that will automate the packaging and building of your project into docker images.

```txt filename="Terminal" copy
moose build --docker
```

After the above command completes you can view your newly created docker files by running the `docker images` command:

```txt filename="Terminal" copy
>docker images
REPOSITORY                                              TAG               IMAGE ID       CREATED              SIZE
moose-df-deployment-aarch64-unknown-linux-gnu           latest            c50674c7a68a   About a minute ago   155MB
moose-df-deployment-x86_64-unknown-linux-gnu            latest            e5b449d3dea3   About a minute ago   163MB
```

> Notice that you get two `moose-df-deployment` containers, one for the `aarch64` (ARM64) architecture and another for the `x86_64` architecture. This is necessary to allow you to choose the version that matches your cloud or on-prem machine architecture.

You can then use standard docker commands to push your new project images to your container repository of choice.

First tag your local images:

```txt filename="Terminal" copy
docker tag moose-df-deployment-aarch64-unknown-linux-gnu:latest {your-repo-user-name}/moose-df-deployment-aarch64-unknown-linux-gnu:latest
docker tag moose-df-deployment-x86_64-unknown-linux-gnu:latest {your-repo-user-name}/moose-df-deployment-x86_64-unknown-linux-gnu:latest
```

Then `push` your files to your container repository.

```txt filename="Terminal" copy
docker push {your-repo-user-name}/moose-df-deployment-aarch64-unknown-linux-gnu:latest
docker push {your-repo-user-name}/moose-df-deployment-x86_64-unknown-linux-gnu:latest
```

You can also use the following handy shell script to automate the steps above.

```bash filename="push.sh" copy
#!/bin/bash

version=$2

if [ -z "$1" ]
then
      echo "You must specify the dockerhub repository as an argument. Example: ./push.sh container-repo-name"
      echo "Note: you can also provide a second argument to supply a specific version tag - otherwise this script will use the same version as the latest moose-cli on Github."
      exit 1
fi

if [ -z "$2" ]
then
      output=$(npx @514labs/moose-cli -V)
      version=$(echo "$output" | sed -n '2p' | awk '{print $2}')
fi

echo "Using version: $version"
arch="moose-df-deployment-aarch64-unknown-linux-gnu"
docker tag $arch:$version $1/$arch:$version
docker push $1/$arch:$version

arch="moose-df-deployment-x86_64-unknown-linux-gnu"
docker tag $arch:$version $1/$arch:$version
docker push $1/$arch:$version
```

---

## Preparing access to ClickHouse, Redis, Temporal and Redpanda

Source: moose/deploying/preparing-clickhouse-redpanda.mdx

Preparing access to ClickHouse, Redis, Temporal and Redpanda

# Preparing access to ClickHouse, Redis, Temporal and Redpanda 

Your hosted Moose application requires access to hosted ClickHouse and Redis service instances. You can also optionally use Redpanda for event streaming. 
You can stand up open source versions of these applications within your environments or opt to use cloud-hosted versions available at:

- [ClickHouse Cloud](https://clickhouse.com)
- [Redis Cloud](https://redis.com)
- [Redpanda Cloud](https://redpanda.com) 
- [Temporal Cloud](https://temporal.io)

## ClickHouse Configuration

If you're using `state_config.storage = "clickhouse"` in your config (serverless mode without Redis), your ClickHouse instance must support the **KeeperMap** table engine. This is used for migration state storage and distributed locking.

✅ **ClickHouse Cloud**: Supported by default
✅ **`moose dev` / `moose prod`**: Already configured in our Docker setup
⚠️ **Self-hosted ClickHouse**: See [ClickHouse KeeperMap documentation](https://clickhouse.com/docs/en/engines/table-engines/special/keeper-map) for setup requirements

If you're using Redis for state storage (`state_config.storage = "redis"`), you don't need KeeperMap.

For ClickHouse, you'll need the following information:

| Parameter | Description | Default Value |
|-----------|-------------|---------------|
| DB_NAME | Database name to use | Your branch or application ID |
| USER | Username for authentication | - |
| PASSWORD | Password for authentication | - |
| HOST | Hostname or IP address | - |
| HOST_PORT | HTTPS port | 8443 |
| USE_SSL | Whether to use SSL (1 for true, 0 for false) | 1 |
| NATIVE_PORT | Native protocol port | 9440 |

These values are used to configure the Moose application's connection to ClickHouse through environment variables following this pattern:

```
MOOSE_CLICKHOUSE_CONFIG__=<value>
```

For example:
```
MOOSE_CLICKHOUSE_CONFIG__DB_NAME=myappdb
MOOSE_CLICKHOUSE_CONFIG__HOST=myclickhouse.example.com
MOOSE_CLICKHOUSE_CONFIG__USE_SSL=1
MOOSE_CLICKHOUSE_CONFIG__HOST_PORT=8443
MOOSE_CLICKHOUSE_CONFIG__NATIVE_PORT=9440
```

## Redis Configuration

Moose requires Redis for caching and as a message broker. You'll need the following configuration:

| Parameter | Description |
|-----------|-------------|
| URL | Redis connection URL |
| KEY_PREFIX | Prefix for Redis keys to isolate namespaces |

These values are configured through:
```
MOOSE_REDIS_CONFIG__URL=redis://username:password@redis.example.com:6379
MOOSE_REDIS_CONFIG__KEY_PREFIX=myapp
```

## Temporal Configuration (Optional)

Temporal is an optional workflow orchestration platform that can be used with Moose. If you choose to use Temporal, you'll need the following configuration:

| Parameter | Description | Default Value |
|-----------|-------------|---------------|
| CA_CERT | Path to CA certificate | /etc/ssl/certs/ca-certificates.crt |
| API_KEY | Temporal Cloud API key | - |
| TEMPORAL_HOST | Temporal Cloud namespace host | Your namespace + .tmprl.cloud |

These values are configured through:
```
MOOSE_TEMPORAL_CONFIG__CA_CERT=/etc/ssl/certs/ca-certificates.crt
MOOSE_TEMPORAL_CONFIG__API_KEY=your-temporal-api-key
MOOSE_TEMPORAL_CONFIG__TEMPORAL_HOST=your-namespace.tmprl.cloud
```

## Redpanda Configuration (Optional)

Redpanda is an optional component that can be used for event streaming. If you choose to use Redpanda, you'll need the following information:

| Parameter | Description | Default Value |
|-----------|-------------|---------------|
| BROKER | Bootstrap server address | - |
| NAMESPACE | Namespace for isolation (often same as branch or app ID) | - |
| MESSAGE_TIMEOUT_MS | Message timeout in milliseconds | 10043 |
| SASL_USERNAME | SASL username for authentication | - |
| SASL_PASSWORD | SASL password for authentication | - |
| SASL_MECHANISM | SASL mechanism | SCRAM-SHA-256 |
| SECURITY_PROTOCOL | Security protocol | SASL_SSL |
| REPLICATION_FACTOR | Topic replication factor | 3 |

These values are used to configure the Moose application's connection to Redpanda through environment variables following this pattern:

```
MOOSE_REDPANDA_CONFIG__=<value>
```

For example:
```
MOOSE_REDPANDA_CONFIG__BROKER=seed-5fbcae97.example.redpanda.com:9092
MOOSE_REDPANDA_CONFIG__NAMESPACE=myapp
MOOSE_REDPANDA_CONFIG__SECURITY_PROTOCOL=SASL_SSL
MOOSE_REDPANDA_CONFIG__SASL_MECHANISM=SCRAM-SHA-256
MOOSE_REDPANDA_CONFIG__REPLICATION_FACTOR=3
```

## Using Environment Variables in Deployment

When deploying your Moose application, you'll need to pass these configurations as environment variables. 
Refer to the deployment guides for your specific platform (Kubernetes, ECS, etc.) for details on how to securely 
provide these values to your application.