Compare commits
9 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 66e892893f | |||
| 5b9138da25 | |||
| b1e8d63a63 | |||
| c6ac174921 | |||
| fba1bea247 | |||
| a63da6f3ca | |||
| 3cfdc305ec | |||
| d336156bbb | |||
| e1b9e97484 |
@@ -0,0 +1,42 @@
|
||||
name: Build and deploy
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- "v*"
|
||||
|
||||
jobs:
|
||||
build-and-push:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Login to Gitea registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: gitea.shcizo.se
|
||||
username: ${{ gitea.actor }}
|
||||
password: ${{ secrets.PACKAGES_TOKEN }}
|
||||
|
||||
- name: Build and push
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: .
|
||||
push: true
|
||||
tags: |
|
||||
gitea.shcizo.se/shcizo/pdga-rating:${{ gitea.ref_name }}
|
||||
gitea.shcizo.se/shcizo/pdga-rating:latest
|
||||
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
needs: [build-and-push]
|
||||
steps:
|
||||
- uses: shcizo/package-updater-action@v1
|
||||
with:
|
||||
endpoint: https://updater.shcizo.se/update
|
||||
image: gitea.shcizo.se/shcizo/pdga-rating
|
||||
tag: ${{ gitea.ref_name }}
|
||||
token: ${{ secrets.UPDATER_TOKEN }}
|
||||
@@ -1,37 +0,0 @@
|
||||
name: Docker Build
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- 'v*'
|
||||
|
||||
jobs:
|
||||
docker:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: docker/login-action@v3
|
||||
with:
|
||||
registry: gitea.shcizo.se
|
||||
username: ${{ gitea.actor }}
|
||||
password: ${{ secrets.PACKAGES_TOKEN }}
|
||||
|
||||
- uses: docker/metadata-action@v5
|
||||
id: meta
|
||||
with:
|
||||
images: gitea.shcizo.se/shcizo/pdga-rating
|
||||
tags: |
|
||||
type=semver,pattern={{version}}
|
||||
type=semver,pattern={{major}}.{{minor}}
|
||||
type=raw,value=latest
|
||||
|
||||
- uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
@@ -4,13 +4,14 @@ PDGA rating scraper and display app. Scrapes player ratings and course data from
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Runtime:** Node.js 18 (Alpine in Docker)
|
||||
- **Runtime:** Node.js 22 (slim/Debian-based in Docker)
|
||||
- **Hosting:** Gitea (`gitea.shcizo.se/shcizo/pdga-rating`) — use `tea` CLI for issues/PRs, not `gh`
|
||||
- **Server:** Express with EJS templates
|
||||
- **Database:** SQLite3 (file: `ratings.db`, Docker: `/app/data/ratings.db`)
|
||||
- **Frontend:** HTMX + vanilla JS (in `public/js/`)
|
||||
- **Scraping:** Puppeteer (with stealth plugin) + direct HTTP
|
||||
- **Logging:** Pino (JSON in production, pino-pretty in dev)
|
||||
- **CI/CD:** Release Please + Docker build/push to GHCR
|
||||
- **CI/CD:** Gitea Actions (tag-triggered docker build/push to `gitea.shcizo.se/shcizo/pdga-rating`)
|
||||
|
||||
## Project Structure
|
||||
|
||||
@@ -38,15 +39,25 @@ public/
|
||||
- `LOG_LEVEL=debug npm start` — Enable debug logging
|
||||
- `docker compose up` — Run via Docker
|
||||
|
||||
**No test framework or lint setup** — `package.json` has only `start` and `dev` scripts. If adding either, document it here.
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Logging:** Use `require('./logger')` (or relative path). Never use `console.log/error` in backend code. Use appropriate Pino levels: `debug` for verbose/diagnostic data, `info` for operational status, `warn` for retries/degraded state, `error` for failures, `fatal` for startup crashes.
|
||||
- **Frontend JS:** `console.error` is fine in `public/js/` — runs in browser, no Pino.
|
||||
- **Commits:** Conventional commits (`feat:`, `fix:`, `refactor:`, `chore:`) — drives release-please.
|
||||
- **Commits:** Conventional commits (`feat:`, `fix:`, `refactor:`, `chore:`, `ci:`).
|
||||
- **Releases:** Manual version bump — edit `version` in `package.json` + `package-lock.json`, commit as `<version>`, tag `v<version>`, push commit + tag (`git push origin main v<version>`). Triggers `.gitea/workflows/docker-build.yml` which builds and pushes the image. Auth uses repo secret `PACKAGES_TOKEN` (PAT with `write:package`) — the auto-injected `GITEA_TOKEN` does not have effective registry access.
|
||||
- **Scraping:** Two strategies per entity: direct HTTP (fast, preferred) with Puppeteer fallback (stealth plugin for anti-bot). Rate limiting must be respected.
|
||||
- **Database:** Migrations run automatically on startup in `db.js`. Schema changes go there.
|
||||
- **Templates:** EJS with shared layout in `views/partials/`. Pages use HTMX for dynamic content loading.
|
||||
|
||||
## PDGA Domain Notes
|
||||
|
||||
- **Rating publication cycle:** PDGA officially recalculates ratings on the **second Tuesday of each month**. `getNextPDGAUpdateDate()` in `src/services/rating-calculator.js` computes this — round filtering uses it as cutoff.
|
||||
- **Predicted rating algorithm:** `calculatePredictedRating(roundRatings)` replicates PDGA's formula — 12-mo window (expands to 24 if <8 rounds), outlier removal at ≥7 rounds (2.5σ + 100pt threshold), double-weighting of recent 25% at ≥9 rounds. Returns `{rating, stdDev, debugLog}`.
|
||||
- **Rate limits:** `POST /api/refresh-round-history/:pdgaNumber` enforces a 24h cooldown per player (`src/routes/players.js`). Don't bypass — PDGA's site rate-limits aggressively.
|
||||
- **Round history refresh** uses Puppeteer (stealth plugin), other scraping prefers direct HTTP. Predicted rating is recomputed and stored on each refresh.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "pdga-ratings",
|
||||
"version": "1.2.3",
|
||||
"version": "1.2.4",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "pdga-ratings",
|
||||
"version": "1.2.3",
|
||||
"version": "1.2.4",
|
||||
"dependencies": {
|
||||
"ejs": "^4.0.1",
|
||||
"express": "^4.18.2",
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "pdga-ratings",
|
||||
"version": "1.2.3",
|
||||
"version": "1.2.4",
|
||||
"description": "PDGA rating scraper and display",
|
||||
"main": "server.js",
|
||||
"scripts": {
|
||||
|
||||
+17
-10
@@ -26,23 +26,30 @@ function applyDeltaPill(pillEl, value) {
|
||||
pillEl.appendChild(numSpan);
|
||||
}
|
||||
|
||||
function setupTooltipsAfterSwap() {
|
||||
document.body.addEventListener('htmx:afterSwap', function(event) {
|
||||
if (event.detail.target.id === 'ratings-table') {
|
||||
initRatingsTooltips();
|
||||
}
|
||||
// After player history partial loads, render the chart
|
||||
const target = event.detail.target;
|
||||
if (target.id && target.id.startsWith('history-content-')) {
|
||||
const container = target.querySelector('.player-chart, .chart-container');
|
||||
if (container && container.dataset.history) {
|
||||
function initChartsIn(rootEl) {
|
||||
rootEl.querySelectorAll('.player-chart').forEach(function(container) {
|
||||
if (container.dataset.charted === 'true') return;
|
||||
if (!container.dataset.history) return;
|
||||
try {
|
||||
const history = JSON.parse(container.dataset.history);
|
||||
createRatingChart(container, history);
|
||||
container.dataset.charted = 'true';
|
||||
} catch (e) {
|
||||
console.error('Error rendering chart:', e);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
function setupAfterTableSwap() {
|
||||
document.body.addEventListener('htmx:afterSwap', function(event) {
|
||||
const target = event.detail.target;
|
||||
if (target.id === 'ratings-table') {
|
||||
initRatingsTooltips();
|
||||
initChartsIn(target);
|
||||
return;
|
||||
}
|
||||
if (target.id && target.id.startsWith('history-content-')) {
|
||||
initChartsIn(target);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
+28
-1
@@ -253,6 +253,32 @@ function getAllMonthlyHistoriesFromDB(months = 12) {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetches the full rating history for ALL players in one query.
|
||||
* Returns Map<pdgaNumber, {rating, date}[]> ordered chronologically (oldest → newest).
|
||||
* Mirrors getAllMonthlyHistoriesFromDB but returns every point, not monthly snapshots.
|
||||
*/
|
||||
function getAllRatingHistoriesFromDB() {
|
||||
return new Promise((resolve, reject) => {
|
||||
db.all(
|
||||
`SELECT p.pdga_number, rh.date, rh.rating
|
||||
FROM rating_history rh
|
||||
JOIN players p ON rh.player_id = p.id
|
||||
ORDER BY p.pdga_number, rh.date ASC`,
|
||||
[],
|
||||
(err, rows) => {
|
||||
if (err) return reject(err);
|
||||
const map = new Map();
|
||||
for (const row of rows) {
|
||||
if (!map.has(row.pdga_number)) map.set(row.pdga_number, []);
|
||||
map.get(row.pdga_number).push({ date: row.date, rating: row.rating });
|
||||
}
|
||||
resolve(map);
|
||||
}
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
function getLastRefresh() {
|
||||
return new Promise((resolve, reject) => {
|
||||
db.get(
|
||||
@@ -278,5 +304,6 @@ module.exports = {
|
||||
savePredictedRatingToDB,
|
||||
getLastRefresh,
|
||||
getMonthlyHistory,
|
||||
getAllMonthlyHistoriesFromDB
|
||||
getAllMonthlyHistoriesFromDB,
|
||||
getAllRatingHistoriesFromDB
|
||||
};
|
||||
|
||||
@@ -5,7 +5,7 @@ const { getPlayerFromDB, savePlayerToDB, getRatingHistoryFromDB, saveRatingHisto
|
||||
const { fetchPlayerDataHTTP, parsePlayerData, fetchRatingHistory, parseRatingHistory } = require('../scrapers/player-http');
|
||||
const { getOfficialRatingHistory, getOptimizedPlayerRounds } = require('../scrapers/player-puppeteer');
|
||||
const { launchBrowser } = require('../scrapers/browser');
|
||||
const { getPlayerDataFromDB, scrapePDGARating, getAllRatingsFromDB, refreshAllPlayersInDB, getPredictedRatingFromDB } = require('../services/player-service');
|
||||
const { getPlayerDataFromDB, scrapePDGARating, getAllRatingsFromDB, refreshAllPlayersInDB, getPredictedRatingFromDB, formatDisplayDate } = require('../services/player-service');
|
||||
const { getTopbarLocals } = require('../services/topbar-service');
|
||||
const { calculatePredictedRating } = require('../services/rating-calculator');
|
||||
const logger = require('../logger');
|
||||
@@ -43,6 +43,8 @@ router.get('/partials/ratings-table', async (req, res) => {
|
||||
}
|
||||
});
|
||||
|
||||
// Used only by the per-player "refresh rating history" button. The initial table render
|
||||
// pre-attaches history via getAllRatingsFromDB to avoid the load-then-fetch race.
|
||||
router.get('/partials/player-history/:pdgaNumber', async (req, res) => {
|
||||
try {
|
||||
const { pdgaNumber } = req.params;
|
||||
@@ -61,7 +63,7 @@ router.get('/partials/player-history/:pdgaNumber', async (req, res) => {
|
||||
const formattedHistory = (history || []).map(row => ({
|
||||
date: row.date,
|
||||
rating: row.rating,
|
||||
displayDate: new Date(row.date).toLocaleDateString('en-US', { day: '2-digit', month: 'short', year: 'numeric' })
|
||||
displayDate: formatDisplayDate(row.date)
|
||||
}));
|
||||
|
||||
const player = await getPlayerDataFromDB(pdgaNumber);
|
||||
@@ -117,11 +119,7 @@ router.get('/api/rating-history/:pdgaNumber', async (req, res) => {
|
||||
const formattedHistory = cachedHistory.map(row => ({
|
||||
date: row.date,
|
||||
rating: row.rating,
|
||||
displayDate: new Date(row.date).toLocaleDateString('en-US', {
|
||||
day: '2-digit',
|
||||
month: 'short',
|
||||
year: 'numeric'
|
||||
})
|
||||
displayDate: formatDisplayDate(row.date)
|
||||
}));
|
||||
|
||||
res.json({
|
||||
|
||||
@@ -1,9 +1,15 @@
|
||||
const { db } = require('../db');
|
||||
const { getPlayerFromDB, getRoundHistoryFromDB, savePredictedRatingToDB, savePlayerToDB, getMonthlyHistory, getAllMonthlyHistoriesFromDB } = require('../models/player');
|
||||
const { getPlayerFromDB, getRoundHistoryFromDB, savePredictedRatingToDB, savePlayerToDB, getMonthlyHistory, getAllMonthlyHistoriesFromDB, getAllRatingHistoriesFromDB } = require('../models/player');
|
||||
const { fetchPlayerDataHTTP, parsePlayerData } = require('../scrapers/player-http');
|
||||
const { calculatePredictedRating } = require('./rating-calculator');
|
||||
const logger = require('../logger');
|
||||
|
||||
function formatDisplayDate(dateStr) {
|
||||
return new Date(dateStr).toLocaleDateString('en-US', {
|
||||
day: '2-digit', month: 'short', year: 'numeric'
|
||||
});
|
||||
}
|
||||
|
||||
// Derives previous-month rating and the delta to it. Prefers PDGA's reported
|
||||
// rating_change (canonical), falls back to our own monthly snapshots when
|
||||
// rating_change is missing — common for players whose latest scrape failed.
|
||||
@@ -167,6 +173,7 @@ async function getAllRatingsFromDB(progressCallback = null) {
|
||||
|
||||
// Fetch all monthly histories in one query so the per-player loop doesn't add N extra queries
|
||||
const monthlyHistoryMap = await getAllMonthlyHistoriesFromDB(12);
|
||||
const ratingHistoryMap = await getAllRatingHistoriesFromDB();
|
||||
|
||||
const ratings = [];
|
||||
const total = allPlayers.length;
|
||||
@@ -189,6 +196,12 @@ async function getAllRatingsFromDB(progressCallback = null) {
|
||||
|
||||
if (playerData) {
|
||||
playerData.monthlyHistory = monthlyHistoryMap.get(pdgaNumber) ?? [];
|
||||
const rawHistory = ratingHistoryMap.get(pdgaNumber) ?? [];
|
||||
playerData.ratingHistory = rawHistory.map(row => ({
|
||||
date: row.date,
|
||||
rating: row.rating,
|
||||
displayDate: formatDisplayDate(row.date)
|
||||
}));
|
||||
// Re-derive now that history is attached — bulk path skipped includeMonthlyHistory
|
||||
const derived = deriveMonthlyDeltas(playerData.rating, player.rating_change, playerData.monthlyHistory);
|
||||
playerData.lastMonthRating = derived.lastMonthRating;
|
||||
@@ -218,7 +231,8 @@ async function getAllRatingsFromDB(progressCallback = null) {
|
||||
stdDev: null,
|
||||
lastMonthRating: (errorRating != null && errorRatingChange != null) ? errorRating - errorRatingChange : null,
|
||||
deltaPredicted: null,
|
||||
monthlyHistory: []
|
||||
monthlyHistory: [],
|
||||
ratingHistory: []
|
||||
};
|
||||
ratings.push(errorData);
|
||||
|
||||
@@ -344,5 +358,6 @@ module.exports = {
|
||||
getPredictedRatingFromDB,
|
||||
getAllRatingsFromDB,
|
||||
refreshAllPlayersInDB,
|
||||
computeKpis
|
||||
computeKpis,
|
||||
formatDisplayDate
|
||||
};
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
const { getLastRefresh } = require('../models/player');
|
||||
const { getNextPDGAUpdateDate } = require('./rating-calculator');
|
||||
const logger = require('../logger');
|
||||
|
||||
const DAY_NAMES = ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat'];
|
||||
const MONTH_NAMES = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'];
|
||||
|
||||
function formatRelative(isoString) {
|
||||
if (!isoString) return 'Never';
|
||||
const then = new Date(isoString.replace(' ', 'T') + (isoString.endsWith('Z') ? '' : 'Z'));
|
||||
@@ -18,25 +22,18 @@ function formatRelative(isoString) {
|
||||
return then.toISOString().slice(0, 10);
|
||||
}
|
||||
|
||||
// First Tuesday of next month — approximation of PDGA's monthly cycle
|
||||
function computeNextUpdate(now = new Date()) {
|
||||
const year = now.getUTCFullYear();
|
||||
const month = now.getUTCMonth() + 1; // next month, may roll over
|
||||
const candidate = new Date(Date.UTC(month === 12 ? year + 1 : year, month === 12 ? 0 : month, 1));
|
||||
// 0=Sun, 1=Mon, 2=Tue
|
||||
const offset = (2 - candidate.getUTCDay() + 7) % 7;
|
||||
candidate.setUTCDate(1 + offset);
|
||||
const months = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'];
|
||||
return `${['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat'][candidate.getUTCDay()]} ${candidate.getUTCDate()} ${months[candidate.getUTCMonth()]}`;
|
||||
function formatNextUpdate(date) {
|
||||
return `${DAY_NAMES[date.getDay()]} ${date.getDate()} ${MONTH_NAMES[date.getMonth()]}`;
|
||||
}
|
||||
|
||||
async function getTopbarLocals() {
|
||||
const nextUpdate = formatNextUpdate(getNextPDGAUpdateDate());
|
||||
try {
|
||||
const lastIso = await getLastRefresh();
|
||||
return { lastRefresh: formatRelative(lastIso), nextUpdate: computeNextUpdate() };
|
||||
return { lastRefresh: formatRelative(lastIso), nextUpdate };
|
||||
} catch (err) {
|
||||
logger.warn({ err }, 'topbar locals fallback');
|
||||
return { lastRefresh: 'Unknown', nextUpdate: computeNextUpdate() };
|
||||
return { lastRefresh: 'Unknown', nextUpdate };
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -109,7 +109,7 @@
|
||||
activePage: 'players',
|
||||
cssFiles: ['players.css'],
|
||||
jsFiles: ['tooltips.js', 'chart.js', 'players.js'],
|
||||
initScript: 'setupTooltipsAfterSwap();',
|
||||
initScript: 'setupAfterTableSwap();',
|
||||
body: body,
|
||||
modals: modals
|
||||
}) %>
|
||||
|
||||
@@ -89,8 +89,12 @@ function renderSparkline(values) {
|
||||
</tr>
|
||||
<tr id="history-<%= player.pdgaNumber %>" class="expanded-content">
|
||||
<td colspan="5" class="expanded-cell">
|
||||
<div id="history-content-<%= player.pdgaNumber %>">
|
||||
<div class="loading-chart">Click to load rating history...</div>
|
||||
<div id="history-content-<%= player.pdgaNumber %>" data-loaded="true">
|
||||
<%- include('player-history', {
|
||||
pdgaNumber: player.pdgaNumber,
|
||||
history: player.ratingHistory || [],
|
||||
player: player
|
||||
}) %>
|
||||
</div>
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
Reference in New Issue
Block a user