Error Detection and Handling

Moose provides multiple layers of error protection, both at the workflow and task level:

Workflow-Level Retries and Timeouts

Moose automatically catches any runtime errors during workflow execution. Errors are logged for debugging, and the orchestrator will retry failed tasks according to the retries option.

In your Workflow, you can configure the following options to control workflow behavior, including timeouts and retries:

import { Task, Workflow } from "@514labs/moose-lib";
 
export const myworkflow = new Workflow("myworkflow", {
  startingTask: task1,
  retries: 1,
  timeout: "10m",
});

from moose_lib import Task, TaskConfig, Workflow, WorkflowConfig
 
myworkflow = Workflow(
  name="myworkflow",
  config=WorkflowConfig(starting_task=task1, retries=1, timeout="10m")
)

Task-Level Errors and Retries

For more granular control over task-level errors and retries, you can configure your individual tasks to have their own retry behavior:

import { Task, Workflow } from "@514labs/moose-lib";
 
export const task1 = new Task<Foo, void>("task1", {
  run: async (input: Foo) => {},
  retries: 1,
  timeout: "5m"
});
 
export const myworkflow = new Workflow("myworkflow", {
  startingTask: task1,
  retries: 2,
  timeout: "10m",
});

from moose_lib import Task, TaskConfig, Workflow, WorkflowConfig
 
def run_task1(input: Foo) -> None:
    pass
 
task1 = Task[Foo, None](
    name="task1",
    config=TaskConfig(run=run_task1, retries=1, timeout="5m")
)
 
myworkflow = Workflow(
  name="myworkflow",
  config=WorkflowConfig(starting_task=task1, retries=2, timeout="10m")
)

Example: Workflow and Task Retry Interplay

When configuring retries, it’s important to understand how workflow-level and task-level retries interact. Consider the following scenario:

import { Task, Workflow } from "@514labs/moose-lib";
 
export const task1 = new Task<Foo, void>("task1", {
  run: async (input: Foo) => {},
  retries: 3,
});
 
export const myworkflow = new Workflow("myworkflow", {
  startingTask: task1,
  retries: 2,
});

from moose_lib import Task, TaskConfig, Workflow, WorkflowConfig
 
def run_task1(input: Foo) -> None:
    pass
 
task1 = Task[Foo, None](
    name="task1",
    config=TaskConfig(run=run_task1, retries=2)
)
 
myworkflow = Workflow(
  name="myworkflow",
  config=WorkflowConfig(starting_task=task1, retries=3)
)

If the execution of the workflow encounters an error, the retry sequence would proceed as follows:

1. Workflow Attempt 1

   • Task Attempt 1: Task fails
   • Task Attempt 2: Task fails
   • Task Attempt 3: Task fails
   • Workflow attempt fails after exhausting task retries

2. Workflow Attempt 2

   • Task Attempt 1: Task fails
   • Task Attempt 2: Task fails
   • Task Attempt 3: Task fails
   • Workflow attempt fails after exhausting task retries

In this example, the workflow will make a total of 2 attempts, and each task within those attempts will retry up to 3 times before the workflow itself retries.