Compare commits
13 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 47e1734917 | |||
| 307dffd3a7 | |||
| 46f78b42dc | |||
| d0c278ea1b | |||
| d0040901ab | |||
| c0f9dd5f33 | |||
| 66e892893f | |||
| 5b9138da25 | |||
| b1e8d63a63 | |||
| c6ac174921 | |||
| fba1bea247 | |||
| a63da6f3ca | |||
| 3cfdc305ec |
@@ -0,0 +1,42 @@
|
|||||||
|
name: Build and deploy
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
tags:
|
||||||
|
- "v*"
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
build-and-push:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
packages: write
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Login to Gitea registry
|
||||||
|
uses: docker/login-action@v3
|
||||||
|
with:
|
||||||
|
registry: gitea.shcizo.se
|
||||||
|
username: ${{ gitea.actor }}
|
||||||
|
password: ${{ secrets.PACKAGES_TOKEN }}
|
||||||
|
|
||||||
|
- name: Build and push
|
||||||
|
uses: docker/build-push-action@v5
|
||||||
|
with:
|
||||||
|
context: .
|
||||||
|
push: true
|
||||||
|
tags: |
|
||||||
|
gitea.shcizo.se/shcizo/pdga-rating:${{ gitea.ref_name }}
|
||||||
|
gitea.shcizo.se/shcizo/pdga-rating:latest
|
||||||
|
|
||||||
|
deploy:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
needs: [build-and-push]
|
||||||
|
steps:
|
||||||
|
- uses: https://gitea.shcizo.se/shcizo/package-updater-action@v1
|
||||||
|
with:
|
||||||
|
endpoint: https://updater.shcizo.se/update
|
||||||
|
image: gitea.shcizo.se/shcizo/pdga-rating
|
||||||
|
tag: ${{ gitea.ref_name }}
|
||||||
|
token: ${{ secrets.UPDATER_API_KEY }}
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
name: Docker Build
|
|
||||||
|
|
||||||
on:
|
|
||||||
push:
|
|
||||||
tags:
|
|
||||||
- 'v*'
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
docker:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
packages: write
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@v4
|
|
||||||
|
|
||||||
- uses: docker/login-action@v3
|
|
||||||
with:
|
|
||||||
registry: gitea.shcizo.se
|
|
||||||
username: ${{ gitea.actor }}
|
|
||||||
password: ${{ secrets.PACKAGES_TOKEN }}
|
|
||||||
|
|
||||||
- uses: docker/metadata-action@v5
|
|
||||||
id: meta
|
|
||||||
with:
|
|
||||||
images: gitea.shcizo.se/shcizo/pdga-rating
|
|
||||||
tags: |
|
|
||||||
type=semver,pattern={{version}}
|
|
||||||
type=semver,pattern={{major}}.{{minor}}
|
|
||||||
type=raw,value=latest
|
|
||||||
|
|
||||||
- uses: docker/build-push-action@v6
|
|
||||||
with:
|
|
||||||
context: .
|
|
||||||
push: true
|
|
||||||
tags: ${{ steps.meta.outputs.tags }}
|
|
||||||
labels: ${{ steps.meta.outputs.labels }}
|
|
||||||
@@ -11,7 +11,7 @@ PDGA rating scraper and display app. Scrapes player ratings and course data from
|
|||||||
- **Frontend:** HTMX + vanilla JS (in `public/js/`)
|
- **Frontend:** HTMX + vanilla JS (in `public/js/`)
|
||||||
- **Scraping:** Puppeteer (with stealth plugin) + direct HTTP
|
- **Scraping:** Puppeteer (with stealth plugin) + direct HTTP
|
||||||
- **Logging:** Pino (JSON in production, pino-pretty in dev)
|
- **Logging:** Pino (JSON in production, pino-pretty in dev)
|
||||||
- **CI/CD:** Gitea Actions (tag-triggered docker build/push to `gitea.shcizo.se/shcizo/pdga-rating`)
|
- **CI/CD:** Gitea Actions (tag-triggered build + push + deploy via `.gitea/workflows/deploy.yml`)
|
||||||
|
|
||||||
## Project Structure
|
## Project Structure
|
||||||
|
|
||||||
@@ -46,7 +46,7 @@ public/
|
|||||||
- **Logging:** Use `require('./logger')` (or relative path). Never use `console.log/error` in backend code. Use appropriate Pino levels: `debug` for verbose/diagnostic data, `info` for operational status, `warn` for retries/degraded state, `error` for failures, `fatal` for startup crashes.
|
- **Logging:** Use `require('./logger')` (or relative path). Never use `console.log/error` in backend code. Use appropriate Pino levels: `debug` for verbose/diagnostic data, `info` for operational status, `warn` for retries/degraded state, `error` for failures, `fatal` for startup crashes.
|
||||||
- **Frontend JS:** `console.error` is fine in `public/js/` — runs in browser, no Pino.
|
- **Frontend JS:** `console.error` is fine in `public/js/` — runs in browser, no Pino.
|
||||||
- **Commits:** Conventional commits (`feat:`, `fix:`, `refactor:`, `chore:`, `ci:`).
|
- **Commits:** Conventional commits (`feat:`, `fix:`, `refactor:`, `chore:`, `ci:`).
|
||||||
- **Releases:** Manual version bump — edit `version` in `package.json` + `package-lock.json`, commit as `<version>`, tag `v<version>`, push commit + tag (`git push origin main v<version>`). Triggers `.gitea/workflows/docker-build.yml` which builds and pushes the image. Auth uses repo secret `PACKAGES_TOKEN` (PAT with `write:package`) — the auto-injected `GITEA_TOKEN` does not have effective registry access.
|
- **Releases:** Manual version bump — edit `version` in `package.json` + `package-lock.json`, commit as `<version>`, tag `v<version>`, push commit + tag (`git push origin main v<version>`). Triggers `.gitea/workflows/deploy.yml` which (1) builds and pushes the image to `gitea.shcizo.se/shcizo/pdga-rating:<tag>` + `:latest`, then (2) calls `package-updater-action` against `updater.shcizo.se/update` to roll out the new image. Required secrets: `PACKAGES_TOKEN` (PAT with `write:package`, for registry auth — the auto-injected `GITEA_TOKEN` does not have effective registry access) and `UPDATER_API_KEY` (for the updater endpoint). The action repo `shcizo/package-updater-action` is referenced via full Gitea URL (`https://gitea.shcizo.se/...`) since `uses:` defaults to GitHub.
|
||||||
- **Scraping:** Two strategies per entity: direct HTTP (fast, preferred) with Puppeteer fallback (stealth plugin for anti-bot). Rate limiting must be respected.
|
- **Scraping:** Two strategies per entity: direct HTTP (fast, preferred) with Puppeteer fallback (stealth plugin for anti-bot). Rate limiting must be respected.
|
||||||
- **Database:** Migrations run automatically on startup in `db.js`. Schema changes go there.
|
- **Database:** Migrations run automatically on startup in `db.js`. Schema changes go there.
|
||||||
- **Templates:** EJS with shared layout in `views/partials/`. Pages use HTMX for dynamic content loading.
|
- **Templates:** EJS with shared layout in `views/partials/`. Pages use HTMX for dynamic content loading.
|
||||||
|
|||||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
|||||||
{
|
{
|
||||||
"name": "pdga-ratings",
|
"name": "pdga-ratings",
|
||||||
"version": "1.2.3",
|
"version": "1.2.7",
|
||||||
"lockfileVersion": 3,
|
"lockfileVersion": 3,
|
||||||
"requires": true,
|
"requires": true,
|
||||||
"packages": {
|
"packages": {
|
||||||
"": {
|
"": {
|
||||||
"name": "pdga-ratings",
|
"name": "pdga-ratings",
|
||||||
"version": "1.2.3",
|
"version": "1.2.7",
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"ejs": "^4.0.1",
|
"ejs": "^4.0.1",
|
||||||
"express": "^4.18.2",
|
"express": "^4.18.2",
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "pdga-ratings",
|
"name": "pdga-ratings",
|
||||||
"version": "1.2.3",
|
"version": "1.2.7",
|
||||||
"description": "PDGA rating scraper and display",
|
"description": "PDGA rating scraper and display",
|
||||||
"main": "server.js",
|
"main": "server.js",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
|
|||||||
+21
-14
@@ -26,23 +26,30 @@ function applyDeltaPill(pillEl, value) {
|
|||||||
pillEl.appendChild(numSpan);
|
pillEl.appendChild(numSpan);
|
||||||
}
|
}
|
||||||
|
|
||||||
function setupTooltipsAfterSwap() {
|
function initChartsIn(rootEl) {
|
||||||
document.body.addEventListener('htmx:afterSwap', function(event) {
|
rootEl.querySelectorAll('.player-chart').forEach(function(container) {
|
||||||
if (event.detail.target.id === 'ratings-table') {
|
if (container.dataset.charted === 'true') return;
|
||||||
initRatingsTooltips();
|
if (!container.dataset.history) return;
|
||||||
|
try {
|
||||||
|
const history = JSON.parse(container.dataset.history);
|
||||||
|
createRatingChart(container, history);
|
||||||
|
container.dataset.charted = 'true';
|
||||||
|
} catch (e) {
|
||||||
|
console.error('Error rendering chart:', e);
|
||||||
}
|
}
|
||||||
// After player history partial loads, render the chart
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function setupAfterTableSwap() {
|
||||||
|
document.body.addEventListener('htmx:afterSwap', function(event) {
|
||||||
const target = event.detail.target;
|
const target = event.detail.target;
|
||||||
|
if (target.id === 'ratings-table') {
|
||||||
|
initRatingsTooltips();
|
||||||
|
initChartsIn(target);
|
||||||
|
return;
|
||||||
|
}
|
||||||
if (target.id && target.id.startsWith('history-content-')) {
|
if (target.id && target.id.startsWith('history-content-')) {
|
||||||
const container = target.querySelector('.player-chart, .chart-container');
|
initChartsIn(target);
|
||||||
if (container && container.dataset.history) {
|
|
||||||
try {
|
|
||||||
const history = JSON.parse(container.dataset.history);
|
|
||||||
createRatingChart(container, history);
|
|
||||||
} catch (e) {
|
|
||||||
console.error('Error rendering chart:', e);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|||||||
+28
-1
@@ -253,6 +253,32 @@ function getAllMonthlyHistoriesFromDB(months = 12) {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fetches the full rating history for ALL players in one query.
|
||||||
|
* Returns Map<pdgaNumber, {rating, date}[]> ordered chronologically (oldest → newest).
|
||||||
|
* Mirrors getAllMonthlyHistoriesFromDB but returns every point, not monthly snapshots.
|
||||||
|
*/
|
||||||
|
function getAllRatingHistoriesFromDB() {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
db.all(
|
||||||
|
`SELECT p.pdga_number, rh.date, rh.rating
|
||||||
|
FROM rating_history rh
|
||||||
|
JOIN players p ON rh.player_id = p.id
|
||||||
|
ORDER BY p.pdga_number, rh.date ASC`,
|
||||||
|
[],
|
||||||
|
(err, rows) => {
|
||||||
|
if (err) return reject(err);
|
||||||
|
const map = new Map();
|
||||||
|
for (const row of rows) {
|
||||||
|
if (!map.has(row.pdga_number)) map.set(row.pdga_number, []);
|
||||||
|
map.get(row.pdga_number).push({ date: row.date, rating: row.rating });
|
||||||
|
}
|
||||||
|
resolve(map);
|
||||||
|
}
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
function getLastRefresh() {
|
function getLastRefresh() {
|
||||||
return new Promise((resolve, reject) => {
|
return new Promise((resolve, reject) => {
|
||||||
db.get(
|
db.get(
|
||||||
@@ -278,5 +304,6 @@ module.exports = {
|
|||||||
savePredictedRatingToDB,
|
savePredictedRatingToDB,
|
||||||
getLastRefresh,
|
getLastRefresh,
|
||||||
getMonthlyHistory,
|
getMonthlyHistory,
|
||||||
getAllMonthlyHistoriesFromDB
|
getAllMonthlyHistoriesFromDB,
|
||||||
|
getAllRatingHistoriesFromDB
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ const { getPlayerFromDB, savePlayerToDB, getRatingHistoryFromDB, saveRatingHisto
|
|||||||
const { fetchPlayerDataHTTP, parsePlayerData, fetchRatingHistory, parseRatingHistory } = require('../scrapers/player-http');
|
const { fetchPlayerDataHTTP, parsePlayerData, fetchRatingHistory, parseRatingHistory } = require('../scrapers/player-http');
|
||||||
const { getOfficialRatingHistory, getOptimizedPlayerRounds } = require('../scrapers/player-puppeteer');
|
const { getOfficialRatingHistory, getOptimizedPlayerRounds } = require('../scrapers/player-puppeteer');
|
||||||
const { launchBrowser } = require('../scrapers/browser');
|
const { launchBrowser } = require('../scrapers/browser');
|
||||||
const { getPlayerDataFromDB, scrapePDGARating, getAllRatingsFromDB, refreshAllPlayersInDB, getPredictedRatingFromDB } = require('../services/player-service');
|
const { getPlayerDataFromDB, scrapePDGARating, getAllRatingsFromDB, refreshAllPlayersInDB, getPredictedRatingFromDB, formatDisplayDate } = require('../services/player-service');
|
||||||
const { getTopbarLocals } = require('../services/topbar-service');
|
const { getTopbarLocals } = require('../services/topbar-service');
|
||||||
const { calculatePredictedRating } = require('../services/rating-calculator');
|
const { calculatePredictedRating } = require('../services/rating-calculator');
|
||||||
const logger = require('../logger');
|
const logger = require('../logger');
|
||||||
@@ -43,6 +43,8 @@ router.get('/partials/ratings-table', async (req, res) => {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Used only by the per-player "refresh rating history" button. The initial table render
|
||||||
|
// pre-attaches history via getAllRatingsFromDB to avoid the load-then-fetch race.
|
||||||
router.get('/partials/player-history/:pdgaNumber', async (req, res) => {
|
router.get('/partials/player-history/:pdgaNumber', async (req, res) => {
|
||||||
try {
|
try {
|
||||||
const { pdgaNumber } = req.params;
|
const { pdgaNumber } = req.params;
|
||||||
@@ -61,7 +63,7 @@ router.get('/partials/player-history/:pdgaNumber', async (req, res) => {
|
|||||||
const formattedHistory = (history || []).map(row => ({
|
const formattedHistory = (history || []).map(row => ({
|
||||||
date: row.date,
|
date: row.date,
|
||||||
rating: row.rating,
|
rating: row.rating,
|
||||||
displayDate: new Date(row.date).toLocaleDateString('en-US', { day: '2-digit', month: 'short', year: 'numeric' })
|
displayDate: formatDisplayDate(row.date)
|
||||||
}));
|
}));
|
||||||
|
|
||||||
const player = await getPlayerDataFromDB(pdgaNumber);
|
const player = await getPlayerDataFromDB(pdgaNumber);
|
||||||
@@ -117,11 +119,7 @@ router.get('/api/rating-history/:pdgaNumber', async (req, res) => {
|
|||||||
const formattedHistory = cachedHistory.map(row => ({
|
const formattedHistory = cachedHistory.map(row => ({
|
||||||
date: row.date,
|
date: row.date,
|
||||||
rating: row.rating,
|
rating: row.rating,
|
||||||
displayDate: new Date(row.date).toLocaleDateString('en-US', {
|
displayDate: formatDisplayDate(row.date)
|
||||||
day: '2-digit',
|
|
||||||
month: 'short',
|
|
||||||
year: 'numeric'
|
|
||||||
})
|
|
||||||
}));
|
}));
|
||||||
|
|
||||||
res.json({
|
res.json({
|
||||||
|
|||||||
@@ -1,9 +1,15 @@
|
|||||||
const { db } = require('../db');
|
const { db } = require('../db');
|
||||||
const { getPlayerFromDB, getRoundHistoryFromDB, savePredictedRatingToDB, savePlayerToDB, getMonthlyHistory, getAllMonthlyHistoriesFromDB } = require('../models/player');
|
const { getPlayerFromDB, getRoundHistoryFromDB, savePredictedRatingToDB, savePlayerToDB, getMonthlyHistory, getAllMonthlyHistoriesFromDB, getAllRatingHistoriesFromDB } = require('../models/player');
|
||||||
const { fetchPlayerDataHTTP, parsePlayerData } = require('../scrapers/player-http');
|
const { fetchPlayerDataHTTP, parsePlayerData } = require('../scrapers/player-http');
|
||||||
const { calculatePredictedRating } = require('./rating-calculator');
|
const { calculatePredictedRating } = require('./rating-calculator');
|
||||||
const logger = require('../logger');
|
const logger = require('../logger');
|
||||||
|
|
||||||
|
function formatDisplayDate(dateStr) {
|
||||||
|
return new Date(dateStr).toLocaleDateString('en-US', {
|
||||||
|
day: '2-digit', month: 'short', year: 'numeric'
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
// Derives previous-month rating and the delta to it. Prefers PDGA's reported
|
// Derives previous-month rating and the delta to it. Prefers PDGA's reported
|
||||||
// rating_change (canonical), falls back to our own monthly snapshots when
|
// rating_change (canonical), falls back to our own monthly snapshots when
|
||||||
// rating_change is missing — common for players whose latest scrape failed.
|
// rating_change is missing — common for players whose latest scrape failed.
|
||||||
@@ -167,6 +173,7 @@ async function getAllRatingsFromDB(progressCallback = null) {
|
|||||||
|
|
||||||
// Fetch all monthly histories in one query so the per-player loop doesn't add N extra queries
|
// Fetch all monthly histories in one query so the per-player loop doesn't add N extra queries
|
||||||
const monthlyHistoryMap = await getAllMonthlyHistoriesFromDB(12);
|
const monthlyHistoryMap = await getAllMonthlyHistoriesFromDB(12);
|
||||||
|
const ratingHistoryMap = await getAllRatingHistoriesFromDB();
|
||||||
|
|
||||||
const ratings = [];
|
const ratings = [];
|
||||||
const total = allPlayers.length;
|
const total = allPlayers.length;
|
||||||
@@ -189,6 +196,12 @@ async function getAllRatingsFromDB(progressCallback = null) {
|
|||||||
|
|
||||||
if (playerData) {
|
if (playerData) {
|
||||||
playerData.monthlyHistory = monthlyHistoryMap.get(pdgaNumber) ?? [];
|
playerData.monthlyHistory = monthlyHistoryMap.get(pdgaNumber) ?? [];
|
||||||
|
const rawHistory = ratingHistoryMap.get(pdgaNumber) ?? [];
|
||||||
|
playerData.ratingHistory = rawHistory.map(row => ({
|
||||||
|
date: row.date,
|
||||||
|
rating: row.rating,
|
||||||
|
displayDate: formatDisplayDate(row.date)
|
||||||
|
}));
|
||||||
// Re-derive now that history is attached — bulk path skipped includeMonthlyHistory
|
// Re-derive now that history is attached — bulk path skipped includeMonthlyHistory
|
||||||
const derived = deriveMonthlyDeltas(playerData.rating, player.rating_change, playerData.monthlyHistory);
|
const derived = deriveMonthlyDeltas(playerData.rating, player.rating_change, playerData.monthlyHistory);
|
||||||
playerData.lastMonthRating = derived.lastMonthRating;
|
playerData.lastMonthRating = derived.lastMonthRating;
|
||||||
@@ -218,7 +231,8 @@ async function getAllRatingsFromDB(progressCallback = null) {
|
|||||||
stdDev: null,
|
stdDev: null,
|
||||||
lastMonthRating: (errorRating != null && errorRatingChange != null) ? errorRating - errorRatingChange : null,
|
lastMonthRating: (errorRating != null && errorRatingChange != null) ? errorRating - errorRatingChange : null,
|
||||||
deltaPredicted: null,
|
deltaPredicted: null,
|
||||||
monthlyHistory: []
|
monthlyHistory: [],
|
||||||
|
ratingHistory: []
|
||||||
};
|
};
|
||||||
ratings.push(errorData);
|
ratings.push(errorData);
|
||||||
|
|
||||||
@@ -344,5 +358,6 @@ module.exports = {
|
|||||||
getPredictedRatingFromDB,
|
getPredictedRatingFromDB,
|
||||||
getAllRatingsFromDB,
|
getAllRatingsFromDB,
|
||||||
refreshAllPlayersInDB,
|
refreshAllPlayersInDB,
|
||||||
computeKpis
|
computeKpis,
|
||||||
|
formatDisplayDate
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -109,7 +109,7 @@
|
|||||||
activePage: 'players',
|
activePage: 'players',
|
||||||
cssFiles: ['players.css'],
|
cssFiles: ['players.css'],
|
||||||
jsFiles: ['tooltips.js', 'chart.js', 'players.js'],
|
jsFiles: ['tooltips.js', 'chart.js', 'players.js'],
|
||||||
initScript: 'setupTooltipsAfterSwap();',
|
initScript: 'setupAfterTableSwap();',
|
||||||
body: body,
|
body: body,
|
||||||
modals: modals
|
modals: modals
|
||||||
}) %>
|
}) %>
|
||||||
|
|||||||
@@ -89,8 +89,12 @@ function renderSparkline(values) {
|
|||||||
</tr>
|
</tr>
|
||||||
<tr id="history-<%= player.pdgaNumber %>" class="expanded-content">
|
<tr id="history-<%= player.pdgaNumber %>" class="expanded-content">
|
||||||
<td colspan="5" class="expanded-cell">
|
<td colspan="5" class="expanded-cell">
|
||||||
<div id="history-content-<%= player.pdgaNumber %>">
|
<div id="history-content-<%= player.pdgaNumber %>" data-loaded="true">
|
||||||
<div class="loading-chart">Click to load rating history...</div>
|
<%- include('player-history', {
|
||||||
|
pdgaNumber: player.pdgaNumber,
|
||||||
|
history: player.ratingHistory || [],
|
||||||
|
player: player
|
||||||
|
}) %>
|
||||||
</div>
|
</div>
|
||||||
</td>
|
</td>
|
||||||
</tr>
|
</tr>
|
||||||
|
|||||||
Reference in New Issue
Block a user