Templates

Github Trending Topics

Github Trending Topics

This template demonstrates how to build a real-time data pipeline and dashboard for tracking GitHub trending topics. It showcases a complete full-stack architecture that combines a Next.js frontend with a Moose backend for data ingestion, processing, and API generation.

Architecture Overview

The template implements a modern full-stack architecture with the following components:

  1. Frontend Layer (Next.js)

    • Real-time dashboard for trending topics
    • Interactive data visualization
    • Type-safe API integration with Moose Consumption APIs
    • Built with TypeScript and Tailwind CSS

  2. Backend Layer (Moose)

    • GitHub Events data ingestion
    • Real-time data processing pipeline
    • Type-safe APIs with Moose Consumption APIs
    • ClickHouse for analytics storage

  3. Infrastructure Layer

    • ClickHouse for high-performance analytics
    • Redpanda for event streaming
    • Temporal for workflow orchestration
    • Type-safe HTTP API endpoints

Getting Started

Clone the repository:

To get started with this template, you can run the following command:

terminal
moose init moose-github-dev-trends github-dev-trends

Install dependencies:

terminal
cd moose-github-dev-trends/moose-backend && npm install

Project Structure

The template is organized into two main components:

moose-github-dev-trends/
├── dashboard/                    # Frontend dashboard
│   ├── app/                      # Next.js pages and routes
│   ├── components/               # React components
│   ├── generated-client/         # Auto-generated API client
│   └── lib/                      # Utility functions
│
├── moose-backend/               # Backend services
│   ├── app/
│   │   ├── apis/                # Consumption API definitions
│   │   ├── ingest/              # Data ingestion logic
│   │   ├── blocks/              # Reusable processing blocks
│   │   └── scripts/             # Workflow scripts
│   └── moose.config.toml        # Moose configuration

Run the application

terminal
moose dev

Start the Workflow to Poll GitHub for Trending Topics

This workflow will run a script every 60 seconds to poll GitHub for trending topics:

terminal
moose workflow run getGithubEvents

Scheduled Workflows

The workflow is scheduled to run every 60 seconds. Learn how to schedule workflows.

(Optional) Configure GitHub API Token

terminal
touch .env
.env
GITHUB_TOKEN=<your-github-api-key>

API Rate Limits

Without authentication, you’re limited to 60 requests/hour. With a token, this increases to 5,000 requests/hour.

Development Workflow

Modern Stack

The visualization combines several modern technologies:

  • Moose for type-safe API generation
  • Tanstack Query for data management
  • Recharts for visualization
  • shadcn/ui for UI components
  • Tailwind CSS for styling

The template demonstrates a complete integration between Moose and Next.js:

Moose Backend

The backend is responsible for data ingestion and API generation:

terminal
cd moose-backend && moose dev

Key components:

  • GitHub Event Polling: Workflow in app/scripts/ fetches public events and posts them to the Moose Ingest Pipeline.
  • Ingest Pipeline: Data Model and infrastructure for ingesting GitHub Events in app/ingest/WatchEvent.ts
  • Consumption APIs: Consumption APIs in app/apis/ expose the data for the frontend

Start the Moose Backend

Spin up the Moose dev server:

terminal
moose dev

This will automatically start the workflow that polls GitHub for trending topics and ingests the data into ClickHouse.

Frontend Integration

The frontend automatically integrates with the Moose backend through generated APIs:

API Client Generation

Moose generates an OpenAPI spec from your Consumption APIs that we use with OpenAPI Generator:

  • Generated client located in live-dev-trends-dashboard/generated-client
  • Based on OpenAPI schema from moose-backend/.moose/openapi.yaml

Data Flow & Visualization

The dashboard implements a modern data flow pattern using Tanstack Query and Shadcn/UI Charts:

Start the Frontend

terminal
cd dashboard && npm install && npm run dev

Visit http://localhost:3000 to see real-time trending topics.

Next Steps

Once you have the data flowing, you can:

  1. Add custom metrics and visualizations
  2. Implement additional GitHub data sources
  3. Create new API endpoints for specific analyses
  4. Build alerting for trending topic detection

Feel free to modify the data models, processing functions, or create new APIs to suit your specific needs!

Live Heart Rate Monitoring