Github Trending Topics

This template demonstrates how to build a real-time data pipeline and dashboard for tracking GitHub trending topics. It showcases a complete full-stack architecture that combines a Next.js frontend with a Moose backend for data ingestion, processing, and API generation.

Architecture Overview

The template implements a modern full-stack architecture with the following components:

1. Frontend Layer (Next.js)

   • Real-time dashboard for trending topics
   • Interactive data visualization
   • Type-safe API integration with Moose Consumption APIs
   • Built with TypeScript and Tailwind CSS

2. Backend Layer (Moose)

   • GitHub Events data ingestion
   • Real-time data processing pipeline
   • Type-safe APIs with Moose Consumption APIs
   • ClickHouse for analytics storage

3. Infrastructure Layer

   • ClickHouse for high-performance analytics
   • Redpanda for event streaming
   • Temporal for workflow orchestration
   • Type-safe HTTP API endpoints

Getting Started

moose init moose-github-dev-trends github-dev-trends

cd moose-github-dev-trends/moose-backend && npm install

moose-github-dev-trends/
├── dashboard/                    # Frontend dashboard
│   ├── app/                      # Next.js pages and routes
│   ├── components/               # React components
│   ├── generated-client/         # Auto-generated API client
│   └── lib/                      # Utility functions
│
├── moose-backend/               # Backend services
│   ├── app/
│   │   ├── apis/                # Consumption API definitions
│   │   ├── ingest/              # Data ingestion logic
│   │   ├── blocks/              # Reusable processing blocks
│   │   └── scripts/             # Workflow scripts
│   └── moose.config.toml        # Moose configuration

moose dev

touch .env

GITHUB_TOKEN=<your-github-api-key>

Development Workflow

The template demonstrates a complete integration between Moose and Next.js:

Moose Backend

The backend is responsible for data ingestion and API generation:

cd moose-backend && moose dev

Key components:

• GitHub Event Polling: Workflow in app/scripts/ fetches public events and posts them to the Moose Ingest Pipeline.
• Ingest Pipeline: Data Model and infrastructure for ingesting GitHub Events in app/ingest/WatchEvent.ts
• Consumption APIs: Consumption APIs in app/apis/ expose the data for the frontend

Start the Moose Backend

Spin up the Moose dev server:

moose dev

This will automatically start the workflow that polls GitHub for trending topics and ingests the data into ClickHouse.

Frontend Integration

The frontend automatically integrates with the Moose backend through generated APIs:

cd dashboard && npm install && npm run dev

Next Steps

Once you have the data flowing, you can:

1. Add custom metrics and visualizations
2. Implement additional GitHub data sources
3. Create new API endpoints for specific analyses
4. Build alerting for trending topic detection

Feel free to modify the data models, processing functions, or create new APIs to suit your specific needs!

Deployment:

Deploying this project involves deploying the Moose backend service and the frontend dashboard separately.

Prerequisites:

• A GitHub account and your project code pushed to a GitHub repository.
• A Boreal account for the backend.
• A Vercel account (or similar platform) for the frontend.

1. Push to GitHub

Push your code to a GitHub repository. Configure a remote repository for your Moose project.

git remote add origin <your-github-repo-url>
git push -u origin main

2. Deploying the Moose Backend (Boreal)

• Create Boreal Project:
  • Log in to your Boreal account and create a new project.
  • Connect Boreal to your GitHub account and select the repository containing your project.
  • Configure the project settings, setting the Project Root to the moose-backend directory.
• Configure Environment Variables:
  • In the Boreal project settings, add the GITHUB_TOKEN environment variable with your GitHub Personal Access Token as the value.
• Deploy: Boreal should automatically build and deploy your Moose service based on your repository configuration. It will also typically start any polling sources (like the GitHub event poller) defined in your moose.config.toml.
• Note API URL: Once deployed, Boreal will provide a public URL for your Moose backend API. You will need this for the frontend deployment.

3. Deploying the Frontend Dashboard (Vercel)

• Create Vercel Project:
  • Log in to your Vercel account and create a new project.
  • Connect Vercel to your GitHub account and select the repository containing your project.
  • Set the Root Directory in Vercel to dashboard (or wherever your frontend code resides within the repository).
• Configure Environment Variables:
  • This is crucial: The frontend needs to know where the deployed backend API is located.
  • Add an environment variable in Vercel to point to your Boreal API URL. The variable name depends on how the frontend code expects it (e.g., NEXT_PUBLIC_API_URL). Check the frontend code (dashboard/) for the exact variable name.

    # Example Vercel Environment Variable
    NEXT_PUBLIC_API_URL=https://your-boreal-project-url.boreal.cloud
• Deploy: Vercel will build and deploy your Next.js frontend.

Once both backend and frontend are deployed and configured correctly, your live GitHub Trends Dashboard should be accessible via the Vercel deployment URL.