shcizo
858143d149
Features added: - Course directory scraping with pagination for Swedish courses - Layout scraping from course detail pages (AJAX tabs) - Event results scraping to calculate layout ratings - Mean rating calculation based on players who shot par - Last played date tracking for each layout (extracted from event pages) - Multi-event aggregation for accurate ratings across tournaments Database: - Added courses table (name, link, city, last_updated) - Added layouts table (name, par, mean_rating, rating_count, last_played) - Added database migrations for new columns - Foreign key relationship between courses and layouts API endpoints: - POST /api/scrape-courses - scrape course directory - POST /api/scrape-layouts/:courseId - scrape layouts and events (combined) - POST /api/scrape-all-layouts - bulk scrape all courses - POST /api/scrape-event-results/:courseId - process event results - GET /api/courses - fetch all courses - GET /api/layouts/:courseId - fetch layouts for course UI: - New courses.html page for course/layout management - Expandable course rows showing layouts - Display layout par, mean rating, and last played date - Layouts sorted by most recently played (newest first) - Individual and bulk scraping controls Technical details: - Date extraction using regex pattern matching from event pages - Proper detection of division results in details/table.results structure - Round score and rating extraction from td.round/td.round-rating pairs - Course location from td.views-field-field-course-location 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
PDGA Ratings Scraper
A comprehensive web application that scrapes PDGA player ratings, calculates rating predictions, and displays them with interactive charts and real-time updates.
Setup
Local Development
- Install dependencies:
npm install
-
Add PDGA numbers to
pdga-numbers.txt(one per line) -
Start the server:
npm start
- Open http://localhost:3000 in your browser
Docker (Recommended)
- Build and run with Docker:
docker build -t pdga-ratings .
docker run -p 3000:3000 --name pdga-ratings-container pdga-ratings
- Open http://localhost:3000 in your browser
Features
Core Functionality
- Database-first architecture: SQLite database serves as single source of truth
- Automatic startup population: Reads
pdga-numbers.txtand populates missing players at startup - User-controlled refresh: Data only updates when user explicitly clicks refresh icons
- Optimized PDGA scraping: Uses
/detailspage as baseline + only scrapes NEW tournaments for predictions
Rating System
- Official PDGA ratings: Current player ratings from PDGA website
- Rating predictions: Multi-day tournament support with weighted calculations
- Rating history charts: Interactive SVG charts with hover tooltips
- Rating changes: Shows recent rating changes with color-coded indicators
User Interface
- Mobile-responsive design: Optimized layouts for desktop and mobile
- Expandable player rows: Click to view detailed rating history charts
- Individual refresh controls: Separate refresh icons for ratings, predictions, and charts
- Progress tracking: Real-time progress bars for bulk operations
- Error handling: Comprehensive error messages with retry suggestions
Performance & Reliability
- Respectful scraping: 2-second delays between PDGA requests
- Smart caching: Database persistence with user-controlled freshness
- Docker optimization: Alpine Linux with Chromium for stable Puppeteer execution
- Retry logic: Automatic retries with fallback strategies for network issues
Usage
Basic Operations
- View ratings: Page loads instantly from database
- Refresh player data: Click refresh icon next to player rating
- Refresh predictions: Click refresh icon next to predicted rating
- View rating history: Click on any player row to expand chart
- Refresh charts: Click refresh icon in chart title
Player Management
- Edit
pdga-numbers.txtto add/remove players - Restart server to automatically populate new players
- Use "Load All" button to force refresh all player data
API Endpoints
GET /api/ratings- Get all player ratings from databasePOST /api/refresh-player/:pdgaNumber- Refresh specific playerPOST /api/refresh-round-history/:pdgaNumber- Refresh player predictionsGET /api/rating-history/:pdgaNumber- Get player rating historyGET /api/database-status- Check database population status
Technical Details
Architecture
- Backend: Node.js + Express + SQLite
- Frontend: Vanilla JavaScript with Server-Sent Events
- Scraping: Puppeteer with Alpine Linux + Chromium
- Charts: Custom SVG implementation with interactive tooltips
PDGA Integration
- Uses official PDGA
/detailspage for rating calculations - Only scrapes tournaments played AFTER last official round
- Respects PDGA servers with proper delays and retry logic
- Handles both current ratings and historical data
Docker Configuration
- Alpine Linux base for minimal footprint
- Pre-installed Chromium for Puppeteer
- Non-root user for security
- Proper volume mounting for data persistence