Samuel Enocsson
33a962e6b8
Extract 3410-line server.js into 12 focused modules:
- src/db.js: database init and migrations
- src/models/{player,course}.js: DB helper functions
- src/scrapers/{browser,player-http,player-puppeteer,course-puppeteer}.js
- src/services/{player-service,rating-calculator}.js
- src/routes/{players,courses,pages}.js
Remove dead code: duplicate saveRatingHistoryToDB, legacy
getPlayerCompetitionRatings/getPredictedRating/getAllRatingsWithScraping,
unused getCourseFromDB/getLatestOfficialRoundDate/testPDGARateLimit,
legacy cache Map, and POST /api/predicted-rating route.
Consolidate 5 duplicated Puppeteer launch blocks into launchBrowser().
server.js is now 28 lines: imports, middleware, mount routers, bootstrap.
PDGA Ratings Scraper
A comprehensive web application that scrapes PDGA player ratings, calculates rating predictions, and displays them with interactive charts and real-time updates.
Setup
Local Development
- Install dependencies:
npm install
-
Add PDGA numbers to
pdga-numbers.txt(one per line) -
Start the server:
npm start
- Open http://localhost:3000 in your browser
Docker (Recommended)
- Build and run with Docker:
docker build -t pdga-ratings .
docker run -p 3000:3000 --name pdga-ratings-container pdga-ratings
- Open http://localhost:3000 in your browser
Features
Core Functionality
- Database-first architecture: SQLite database serves as single source of truth
- Automatic startup population: Reads
pdga-numbers.txtand populates missing players at startup - User-controlled refresh: Data only updates when user explicitly clicks refresh icons
- Optimized PDGA scraping: Uses
/detailspage as baseline + only scrapes NEW tournaments for predictions
Rating System
- Official PDGA ratings: Current player ratings from PDGA website
- Rating predictions: Multi-day tournament support with weighted calculations
- Rating history charts: Interactive SVG charts with hover tooltips
- Rating changes: Shows recent rating changes with color-coded indicators
User Interface
- Mobile-responsive design: Optimized layouts for desktop and mobile
- Expandable player rows: Click to view detailed rating history charts
- Individual refresh controls: Separate refresh icons for ratings, predictions, and charts
- Progress tracking: Real-time progress bars for bulk operations
- Error handling: Comprehensive error messages with retry suggestions
Performance & Reliability
- Respectful scraping: 2-second delays between PDGA requests
- Smart caching: Database persistence with user-controlled freshness
- Docker optimization: Alpine Linux with Chromium for stable Puppeteer execution
- Retry logic: Automatic retries with fallback strategies for network issues
Usage
Basic Operations
- View ratings: Page loads instantly from database
- Refresh player data: Click refresh icon next to player rating
- Refresh predictions: Click refresh icon next to predicted rating
- View rating history: Click on any player row to expand chart
- Refresh charts: Click refresh icon in chart title
Player Management
- Edit
pdga-numbers.txtto add/remove players - Restart server to automatically populate new players
- Use "Load All" button to force refresh all player data
API Endpoints
GET /api/ratings- Get all player ratings from databasePOST /api/refresh-player/:pdgaNumber- Refresh specific playerPOST /api/refresh-round-history/:pdgaNumber- Refresh player predictionsGET /api/rating-history/:pdgaNumber- Get player rating historyGET /api/database-status- Check database population status
Technical Details
Architecture
- Backend: Node.js + Express + SQLite
- Frontend: Vanilla JavaScript with Server-Sent Events
- Scraping: Puppeteer with Alpine Linux + Chromium
- Charts: Custom SVG implementation with interactive tooltips
PDGA Integration
- Uses official PDGA
/detailspage for rating calculations - Only scrapes tournaments played AFTER last official round
- Respects PDGA servers with proper delays and retry logic
- Handles both current ratings and historical data
Docker Configuration
- Alpine Linux base for minimal footprint
- Pre-installed Chromium for Puppeteer
- Non-root user for security
- Proper volume mounting for data persistence