Ingest
Ingest data periodically from an API
This demo will walk you through using Aurora tools to prompt your way to ingesting a whole bunch of aircraft telemetry data from adsb.lol.
Skip to prompt if you started with the ads-b template.
Prerequisites
- Aurora and Moose CLIs: Install them here
- OS: macOS or Linux (WSL supported for Windows)
- Docker Desktop/Engine: 24.0.0+
- Node: version 20+ (LTS recommended)
- Anthropic API Key: Get one here
- Client: Cursor
Create and initialize a new typescript project
aurora init ingest-demo typescript-empty
cd ingest-demo
npm install
Open the project in Cursor
Run the Moose Dev Servers
moose dev
Initialize the Aurora MCP
Navigate to Cursor > Settings > Cursor Settings > Tools and Integrations
then toggle on the Aurora MCP
tool.
For best results, set the LLM to claude-4-sonnet
Gemini 2.5 and o3 are also reasonably good, but claude-4-sonnet has the most consistent results.
Prompt the LLM to create an ingest
I want to ingest data from the following aircraft transponder data api every 5 seconds for the purpose of creating a set of visualizations.
API: @https://api.adsb.lol/v2/mil
Docs: @https://api.adsb.lol/docs#/v2/v2_mil_v2_mil_get (note, I think the schema here is inaccurate, check out the data before you trust the docs).
Can you execute on this?
The LLM might do a planning step, in which case, you can ask it to execute on the plan.
Go for it!
Action
The LLM should now use aurora tools to:
- get sample data from the API using a temporary script
- use that sample to create a schema that’s used to create an ingest pipeline (ingest API, Redpanda stream, ClickHouse table)
- create a Moose managed temporal workflow to periodically ingest the data
You’ll know it is succeeding if you see dozens of events per second hit your Moose dev console.