Skip to content

Commit 311e528

Browse files
patrickrbclaude
patrickrb
and
claude
authored
fix(sync): correctness overhaul of QRZ + LoTW sync flows (#185)
* fix(sync): correctness overhaul of QRZ + LoTW sync flows Both sync paths had bugs causing silent data loss in production: QRZ download - OPTION param: send a single value with `;` separators, not two separate fields (FormData duplicates lose `TYPE:ADIF`) - MODSINCE (not MODIFIEDSINCE) — the date filter was being silently ignored, every "incremental" download pulled the full logbook - add STATUS:CONFIRMED to limit payload, extract APP_QRZLOG_STATUS - new matchQRZConfirmation: call+band+mode+station_callsign exact + ±15min tolerance (was call+date+time only with ±5min) LoTW download - URL params: qso_qslsince / qso_qslbefore (no underscores) — date filter was a no-op - add qso_qsldetail/qso_mydetail so LoTW returns enriched location fields (state, county, CQZ, ITUZ, DXCC, grid) - match: require call+band+mode+station_callsign exact, ±15min, proper UTC parsing, satellite-mode rule (sat_name must agree) - on confirmation: enrich state/cnty/cqz/ituz/dxcc/country/grid; persist stations.lotw_last_qsl_rcvd_date for incremental fetches LoTW upload (.tq8 signing) - replace the OpenSSL fallback stub (which logged a warning then returned UNSIGNED ADIF — every Vercel cron upload was rejected by LoTW) with a pure-Node node-forge implementation that produces wavelog-compatible .tq8 files (gzip + per-QSO RSA-SHA1 signature in <SIGN_LOTW_V2.0>, with the canonical sign-string echoed in <SIGNDATA>); runtime self-verify before each upload - multipart `upfile` POST + check for `<!-- .UPL. accepted -->` - filter unsupported prop_modes (INTERNET, RPT) — flagged 'I' - normalize callsigns (W1AW_P → W1AW/P) Cross-service sync - when LoTW confirms a QRZ-uploaded QSO, set qrz_qsl_sent='M' so the next QRZ sync re-uploads with OPTION=REPLACE; mirrored for the QRZ→LoTW direction Other - validateLoTWCredentials now actually parses the response body (the old response.url check always returned true) - new checkLotwCertCrl helper queries lotw.arrl.org/lotw/crl?serial= - vercel.json: maxDuration 300s on cron + LoTW routes Migration: migrations/sync_qrz_lotw_fixes.sql adds p12_password, cert_serial, crl_status to lotw_credentials; lotw_last_qsl_rcvd_date to stations; prop_mode/sat_name/band_rx/freq_rx/iota to contacts; CHECK constraints on the QSL-sent enums. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(migration): add QRZ column setup + migration runner script Discovered while applying to local + prod that: 1. Some envs initialized contacts without qrz_qsl_sent / qrz_qsl_rcvd (those were added by the in-app /install route, not the base schema). Add them with IF NOT EXISTS to make the migration self-sufficient. 2. The PG 18+ syntax `CREATE TRIGGER IF NOT EXISTS` in the existing postgres-lotw-migration.sql breaks on PG 15/17. Added a one-line transform in scripts/run-migration.mjs to rewrite it as DROP+CREATE so the legacy file works without editing it. scripts/run-migration.mjs is a small Node runner that takes DATABASE_URL via env and a SQL file via argv; it wraps everything in a transaction and rolls back on failure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(lotw): capture P12 password during certificate upload The signer needs the P12 password to decrypt the private key. Until now the certificate-upload route stored only the .p12 file blob, so buildSignedTq8 had to assume an empty password — fine for unprotected TQSL exports, broken for password-protected ones. API: /api/lotw/certificate now accepts an optional p12_password form field. Before insert, it parses the P12 with node-forge using that password (via readCertMetadata). Bad password / corrupt file fails fast with a clear error rather than landing an unusable cert in the DB. We also persist cert_serial and cert_expires_at extracted from the parse, populating columns the migration just added. UI: both upload forms got a password input with show/hide toggle. Helper text explains the password is required to sign uploads, stored encrypted, and never returned to the browser. Empty is accepted for TQSL exports without a password. Side fix: the LoTW dashboard form was already broken — it didn't send cert_name (required by the API). Added a Certificate Name field there too, and restructured the 3-col grid into a 2x2 layout so all four fields fit cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent f4ea0a5 commit 311e528

16 files changed

Lines changed: 1410 additions & 409 deletions

File tree

‎migrations/sync_qrz_lotw_fixes.sql‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
-- Migration: QRZ + LoTW sync correctness fixes
2+
-- Adds fields required to:
3+
-- 1. Sign LoTW .tq8 files in pure Node (lotw_credentials.p12_password)
4+
-- 2. Track CRL status of stored certificates
5+
-- 3. Drive incremental QRZ/LoTW downloads via last-confirmed timestamps
6+
-- 4. Carry the cross-service 'M' (modified) and 'I' (ignore) flags
7+
--
8+
-- Idempotent — safe to re-run.
9+
--
10+
-- Prerequisite: postgres-lotw-migration.sql must have been run first to create
11+
-- the lotw_credentials / lotw_upload_logs / lotw_download_logs tables.
12+
13+
-- 0. contacts: ensure QRZ tracking columns exist (some envs were initialized
14+
-- before the in-app /install route added these). sync_qrz_lotw_fixes is the
15+
-- canonical setup for the QRZ flow now.
16+
ALTER TABLE contacts ADD COLUMN IF NOT EXISTS qrz_qsl_sent VARCHAR(10);
17+
ALTER TABLE contacts ADD COLUMN IF NOT EXISTS qrz_qsl_rcvd VARCHAR(10);
18+
ALTER TABLE contacts ADD COLUMN IF NOT EXISTS qrz_qsl_sent_date DATE;
19+
ALTER TABLE contacts ADD COLUMN IF NOT EXISTS qrz_qsl_rcvd_date DATE;
20+
CREATE INDEX IF NOT EXISTS idx_contacts_qrz_qsl_sent ON contacts(qrz_qsl_sent);
21+
CREATE INDEX IF NOT EXISTS idx_contacts_qrz_qsl_rcvd ON contacts(qrz_qsl_rcvd);
22+
23+
-- 1. lotw_credentials: store the encrypted P12 password and CRL state.
24+
ALTER TABLE lotw_credentials ADD COLUMN IF NOT EXISTS p12_password TEXT;
25+
ALTER TABLE lotw_credentials ADD COLUMN IF NOT EXISTS cert_serial TEXT;
26+
ALTER TABLE lotw_credentials ADD COLUMN IF NOT EXISTS crl_status VARCHAR(16);
27+
ALTER TABLE lotw_credentials ADD COLUMN IF NOT EXISTS crl_checked_at TIMESTAMP;
28+
CREATE INDEX IF NOT EXISTS idx_lotw_credentials_cert_serial ON lotw_credentials(cert_serial);
29+
30+
-- 2. stations: incremental download bookmarks.
31+
ALTER TABLE stations ADD COLUMN IF NOT EXISTS lotw_last_qsl_rcvd_date DATE;
32+
ALTER TABLE stations ADD COLUMN IF NOT EXISTS qrz_last_qsl_rcvd_date DATE;
33+
34+
-- 3. contacts: prop_mode/sat_name/band_rx/freq_rx columns required to build a
35+
-- valid LoTW upload + match satellite confirmations correctly. The mode +
36+
-- band columns already exist; these are the missing TQSL inputs.
37+
ALTER TABLE contacts ADD COLUMN IF NOT EXISTS prop_mode VARCHAR(16);
38+
ALTER TABLE contacts ADD COLUMN IF NOT EXISTS sat_name VARCHAR(32);
39+
ALTER TABLE contacts ADD COLUMN IF NOT EXISTS band_rx VARCHAR(20);
40+
ALTER TABLE contacts ADD COLUMN IF NOT EXISTS freq_rx DECIMAL(10, 6);
41+
ALTER TABLE contacts ADD COLUMN IF NOT EXISTS iota VARCHAR(16);
42+
ALTER TABLE contacts ADD COLUMN IF NOT EXISTS lotw_qslrdate DATE;
43+
CREATE INDEX IF NOT EXISTS idx_contacts_prop_mode ON contacts(prop_mode);
44+
CREATE INDEX IF NOT EXISTS idx_contacts_sat_name ON contacts(sat_name);
45+
46+
-- 4. contacts: enforce QRZ/LoTW status enum values (Y/N/R/M/I/Q + NULL).
47+
-- 'M' = modified, queued for re-upload after a cross-service confirmation.
48+
-- 'I' = ignored, prop_mode is unsupported by LoTW (INTERNET, RPT).
49+
DO $$
50+
BEGIN
51+
IF NOT EXISTS (
52+
SELECT 1 FROM information_schema.table_constraints
53+
WHERE constraint_name = 'contacts_qrz_qsl_sent_check'
54+
) THEN
55+
ALTER TABLE contacts
56+
ADD CONSTRAINT contacts_qrz_qsl_sent_check
57+
CHECK (qrz_qsl_sent IS NULL OR qrz_qsl_sent IN ('Y','N','R','M','I','Q'));
58+
END IF;
59+
60+
IF NOT EXISTS (
61+
SELECT 1 FROM information_schema.table_constraints
62+
WHERE constraint_name = 'contacts_lotw_qsl_sent_check'
63+
) THEN
64+
ALTER TABLE contacts
65+
ADD CONSTRAINT contacts_lotw_qsl_sent_check
66+
CHECK (lotw_qsl_sent IS NULL OR lotw_qsl_sent IN ('Y','N','R','M','I','Q'));
67+
END IF;
68+
END$$;
69+
70+
-- 5. stations: ensure DXCC entity / location fields exist (used by .tq8 builder).
71+
ALTER TABLE stations ADD COLUMN IF NOT EXISTS dxcc_entity_code INTEGER;
72+
ALTER TABLE stations ADD COLUMN IF NOT EXISTS state_province VARCHAR(64);
73+
ALTER TABLE stations ADD COLUMN IF NOT EXISTS county VARCHAR(64);
74+
ALTER TABLE stations ADD COLUMN IF NOT EXISTS itu_zone INTEGER;
75+
ALTER TABLE stations ADD COLUMN IF NOT EXISTS cq_zone INTEGER;
76+
77+
SELECT 'sync_qrz_lotw_fixes migration completed' AS message;

‎package-lock.json‎

Lines changed: 21 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎package.json‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
"lucide-react": "^0.539.0",
4141
"next": "^16.2.6",
4242
"next-auth": "^4.24.11",
43+
"node-forge": "^1.4.0",
4344
"node-html-parser": "^7.0.1",
4445
"pg": "^8.11.3",
4546
"react": "^19.1.1",
@@ -56,6 +57,7 @@
5657
"@types/bcryptjs": "^2.4.6",
5758
"@types/jsonwebtoken": "^9.0.10",
5859
"@types/node": "^20",
60+
"@types/node-forge": "^1.3.14",
5961
"@types/pg": "^8.11.2",
6062
"@types/react": "^19",
6163
"@types/react-dom": "^19",

‎scripts/run-migration.mjs‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
// One-shot migration runner.
2+
// Usage: DATABASE_URL=postgres://... node scripts/run-migration.mjs <sql-file>
3+
//
4+
// Wraps the file contents in a transaction so a partial-fail leaves the DB
5+
// untouched. Idempotent migrations with IF NOT EXISTS / DO $$ BEGIN ... END$$
6+
// guards are safe to re-run.
7+
8+
import { readFileSync } from 'node:fs';
9+
import { resolve } from 'node:path';
10+
import pg from 'pg';
11+
12+
const sqlPath = process.argv[2];
13+
const dbUrl = process.env.DATABASE_URL;
14+
15+
if (!sqlPath) {
16+
console.error('Usage: DATABASE_URL=... node scripts/run-migration.mjs <sql-file>');
17+
process.exit(2);
18+
}
19+
if (!dbUrl) {
20+
console.error('DATABASE_URL is required');
21+
process.exit(2);
22+
}
23+
24+
let sql = readFileSync(resolve(sqlPath), 'utf8');
25+
26+
// Postgres 15/17 don't support `CREATE TRIGGER IF NOT EXISTS` (PG 18+ feature).
27+
// Rewrite each occurrence as a DROP IF EXISTS + CREATE pair so the SQL works
28+
// across versions without editing the source migration files.
29+
sql = sql.replace(
30+
/CREATE TRIGGER IF NOT EXISTS\s+(\w+)\s+([\s\S]*?);/gi,
31+
(_match, triggerName, body) => {
32+
// Extract the table name from the trigger body (e.g., "BEFORE UPDATE ON foo").
33+
const tableMatch = body.match(/\bON\s+(\w+)/i);
34+
const table = tableMatch ? tableMatch[1] : '';
35+
return `DROP TRIGGER IF EXISTS ${triggerName} ON ${table};\nCREATE TRIGGER ${triggerName} ${body};`;
36+
}
37+
);
38+
39+
// Azure Postgres requires SSL; localhost typically doesn't. Detect from URL.
40+
const ssl = /azure\.com|sslmode=require/i.test(dbUrl) ? { rejectUnauthorized: false } : false;
41+
const client = new pg.Client({ connectionString: dbUrl, ssl });
42+
43+
const target = dbUrl.replace(/:[^@:/]*@/, ':****@');
44+
console.log(`Running ${sqlPath} against ${target}`);
45+
46+
try {
47+
await client.connect();
48+
await client.query('BEGIN');
49+
await client.query(sql);
50+
await client.query('COMMIT');
51+
console.log('Migration applied successfully.');
52+
} catch (err) {
53+
try { await client.query('ROLLBACK'); } catch {}
54+
console.error('Migration failed; rolled back.');
55+
console.error(err.message);
56+
process.exit(1);
57+
} finally {
58+
await client.end();
59+
}

‎src/app/api/contacts/qrz-download/route.ts‎

Lines changed: 38 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@ import jwt from 'jsonwebtoken';
33
import { User } from '@/models/User';
44
import { Contact } from '@/models/Contact';
55
import { Station } from '@/models/Station';
6-
import { downloadQSOsFromQRZ } from '@/lib/qrz';
6+
import { downloadQSOsFromQRZ, matchQRZConfirmation } from '@/lib/qrz';
7+
import { query } from '@/lib/db';
78

89
export async function POST(request: NextRequest) {
910
try {
@@ -67,21 +68,44 @@ export async function POST(request: NextRequest) {
6768
console.log(`Found ${stationContacts.length} unconfirmed contacts for station ${station.callsign}`);
6869

6970
let confirmationsFound = 0;
70-
71-
// Match QRZ QSOs with our contacts to find confirmations
72-
for (const contact of stationContacts) {
71+
72+
// Annotate each contact with the station callsign so the matcher can
73+
// cross-check against QRZ's STATION_CALLSIGN field. ContactData has
74+
// station_id but not station_callsign — fill it in from the iterated station.
75+
const stationCall = station.callsign;
76+
const annotated = stationContacts.map(c => ({ ...c, station_callsign: stationCall }));
77+
78+
// Match QRZ QSOs with our contacts. Tighter rules: callsign + band +
79+
// mode + station_callsign + ±15min, so two QSOs on different bands at
80+
// the same minute don't false-match.
81+
for (const contact of annotated) {
7382
for (const qrzQSO of downloadResult.qsos) {
74-
if (Contact.matchQSO(contact, qrzQSO)) {
75-
console.log(`Found confirmation match for ${contact.callsign} on ${contact.datetime}`);
76-
77-
// Check if QRZ shows this as confirmed
78-
if (qrzQSO.qsl_rcvd === 'Y' || qrzQSO.qsl_sent === 'Y') {
79-
console.log(`Marking ${contact.callsign} as QRZ confirmed`);
80-
await Contact.updateQrzQsl(contact.id, undefined, 'Y'); // Mark received
81-
confirmationsFound++;
82-
}
83-
break; // Found match, no need to check other QRZ QSOs for this contact
83+
if (!matchQRZConfirmation(contact, qrzQSO)) continue;
84+
85+
// QRZ marks confirmed records with app_qrzlog_status='C'. The legacy
86+
// qsl_rcvd / qsl_sent fields aren't always populated, so prefer the
87+
// app field when present.
88+
const isConfirmed =
89+
qrzQSO.app_qrzlog_status?.toUpperCase() === 'C' ||
90+
qrzQSO.qsl_rcvd === 'Y' ||
91+
qrzQSO.qsl_sent === 'Y';
92+
if (!isConfirmed) {
93+
break;
94+
}
95+
96+
console.log(`Marking ${contact.callsign} as QRZ confirmed (status=${qrzQSO.app_qrzlog_status ?? '<missing>'})`);
97+
await Contact.updateQrzQsl(contact.id, undefined, 'Y');
98+
confirmationsFound++;
99+
100+
// Cross-sync: if LoTW already shipped this QSO, flag for re-upload
101+
// so the new qrz_qsl_rcvd value propagates back into LoTW (wavelog 'M').
102+
if (contact.lotw_qsl_sent === 'Y') {
103+
await query(
104+
`UPDATE contacts SET lotw_qsl_sent = 'M', updated_at = NOW() WHERE id = $1`,
105+
[contact.id]
106+
);
84107
}
108+
break;
85109
}
86110
}
87111

‎src/app/api/lotw/certificate/route.ts‎

Lines changed: 41 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
import { NextRequest, NextResponse } from 'next/server';
44
import { verifyToken } from '@/lib/auth';
55
import { query } from '@/lib/db';
6+
import { encryptString, readCertMetadata } from '@/lib/lotw';
67
import { LotwCertificateResponse } from '@/types/lotw';
78

89
export async function POST(request: NextRequest) {
@@ -17,6 +18,8 @@ export async function POST(request: NextRequest) {
1718
const stationId = formData.get('station_id') as string;
1819
const callsign = formData.get('callsign') as string;
1920
const certName = formData.get('cert_name') as string;
21+
// Optional — TQSL exports without a password are common; node-forge accepts ''.
22+
const p12Password = (formData.get('p12_password') as string | null) ?? '';
2023

2124
if (!file || !stationId || !callsign || !certName) {
2225
return NextResponse.json({
@@ -68,6 +71,22 @@ export async function POST(request: NextRequest) {
6871
}, { status: 400 });
6972
}
7073

74+
// Validate the P12 by parsing it with the supplied password. This catches
75+
// wrong-password / corrupt-file uploads before they sit unusable in the DB.
76+
let certMeta: { serial: string; notAfter?: Date; dxcc?: number };
77+
try {
78+
certMeta = readCertMetadata(fileBuffer, p12Password);
79+
} catch (parseError) {
80+
const msg = parseError instanceof Error ? parseError.message : 'Unknown error';
81+
// node-forge throws "PKCS#12 MAC could not be verified" / similar on bad password
82+
const isPasswordError = /mac|password|invalid|decrypt/i.test(msg);
83+
return NextResponse.json({
84+
error: isPasswordError
85+
? 'Could not parse certificate with the supplied password. Re-export from TQSL and re-enter the password.'
86+
: `Certificate parse failed: ${msg}`
87+
}, { status: 400 });
88+
}
89+
7190
// Check if certificate already exists for this station
7291
const existingCertResult = await query(
7392
'SELECT id FROM lotw_credentials WHERE station_id = $1 AND is_active = true',
@@ -82,24 +101,37 @@ export async function POST(request: NextRequest) {
82101
);
83102
}
84103

85-
// Store new certificate in lotw_credentials table
104+
// Encrypt the P12 password at rest. Empty string is encrypted as well so
105+
// the upload route can simply decrypt-or-default; storing NULL would
106+
// require a branch in every read path.
107+
const encryptedPassword = encryptString(p12Password);
108+
109+
// Store new certificate + metadata extracted from the P12.
86110
const insertResult = await query(
87111
`INSERT INTO lotw_credentials
88-
(station_id, name, callsign, p12_cert, cert_created_at, is_active)
89-
VALUES ($1, $2, $3, $4, NOW(), true)
90-
RETURNING id, cert_created_at`,
91-
[parseInt(stationId), certName.trim(), callsign.toUpperCase(), fileBuffer]
112+
(station_id, name, callsign, p12_cert, p12_password,
113+
cert_serial, cert_created_at, cert_expires_at, is_active)
114+
VALUES ($1, $2, $3, $4, $5, $6, NOW(), $7, true)
115+
RETURNING id, cert_created_at, cert_expires_at`,
116+
[
117+
parseInt(stationId),
118+
certName.trim(),
119+
callsign.toUpperCase(),
120+
fileBuffer,
121+
encryptedPassword,
122+
certMeta.serial,
123+
certMeta.notAfter ?? null,
124+
]
92125
);
93126

94127
const newCredential = insertResult.rows[0];
95128

96-
// TODO: Extract certificate expiration date from P12 file
97-
// This would require additional crypto libraries to parse the certificate
98-
99129
const response: LotwCertificateResponse = {
100130
success: true,
101131
credential_id: newCredential.id,
102-
// cert_expires_at: expirationDate?.toISOString()
132+
cert_expires_at: newCredential.cert_expires_at
133+
? new Date(newCredential.cert_expires_at).toISOString()
134+
: undefined,
103135
};
104136

105137
return NextResponse.json(response);

‎src/app/api/lotw/download-contact/route.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,11 @@ export async function POST(request: NextRequest) {
123123
const dateToStr = dateTo.toISOString().split('T')[0];
124124

125125
// Build LoTW download URL with date range
126-
const downloadUrl = buildLoTWDownloadUrl(lotwUsername, lotwPassword, dateFromStr, dateToStr);
126+
const downloadUrl = buildLoTWDownloadUrl(lotwUsername, lotwPassword, {
127+
dateFrom: dateFromStr,
128+
dateTo: dateToStr,
129+
ownCallsign: contact.station_callsign,
130+
});
127131

128132
// Download confirmations from LoTW
129133
let adifContent: string;

0 commit comments

Comments
 (0)