Skip to content

Speed up main page load and show explicit loading/error states#661

Merged
caufieldjh merged 1 commit into
mainfrom
issue-660-main-page-loading
Jul 15, 2026
Merged

Speed up main page load and show explicit loading/error states#661
caufieldjh merged 1 commit into
mainfrom
issue-660-main-page-loading

Conversation

@caufieldjh

Copy link
Copy Markdown
Collaborator

Fixes #660.

The main page fetched the full ~30 MB registry/kgs.jsonld on every visit and parsed it before rendering anything, while the count badges were hardcoded to 0 resources / 0 KGs. For several seconds the registry looked empty, with no indication that data was still loading. A failed load reset the badges to 0 too, so an error looked almost identical to both "still loading" and "registry is empty".

This addresses all three items in #660.

1. Improve data loading speed

The table only reads ~13 fields per resource (id, name, description, category, domains, collection, taxon, activity_status, evaluation_page, homepage_url, license, publications) — the rest of the 30 MB is contact/product/publication detail the main page never renders.

New util/make_summary_json.py generates registry/kgs-summary.json with just those fields, wired into the Makefile next to the existing kgs.jsonld target. The main page fetches that instead:

Raw Gzipped (what Pages serves) Fetch + parse (localhost)
kgs.jsonld (before) 28.5 MB 1.75 MB ~880 ms
kgs-summary.json (after) 0.67 MB 0.16 MB ~30 ms

~11x less to download, and the slow synchronous JSON.parse of a 30 MB string is gone. The full kgs.jsonld is untouched and still offered for download / API use; custom.js falls back to it automatically (with a console warning) if the summary is missing, e.g. on a deployment built before this change.

2. Loading indicator

The badges and table area now ship a Loading registry data… spinner in the static markup, so it's visible from first paint rather than only after the fetch resolves (the old Loading resources... only appeared after the fetch already succeeded, so it never covered the actual slow part). Counts appear only once data has loaded, and No resources found now shows only when a real filter returns nothing.

3. Make errors obvious vs. still-loading

A failed load now shows a distinct error alert that names the cause, states explicitly that this is a load failure rather than an empty registry, and offers a Retry button. Badges go to with bg-danger instead of 0, so failure can't be confused with loading or emptiness.

A failed taxon_mapping.yaml fetch (previously console-only, invisible) now shows a dismissible warning that taxon filtering is degraded to exact matches; the table itself is unaffected.

Also fixed (found while investigating — the "subtle error" from #660)

  • The <noscript> fallback was broken: _includes/resource_table.html iterated {% raw %}{% for ont in site.ontologies %}{% endraw %}, but the data key is site.resources, so no-JS users got an empty table plus OBO-style links (purl.obolibrary.org, Ontobee) irrelevant to this registry. Replaced with a short notice + data download links. Deliberately kept small: <noscript> content is downloaded by every browser even with JS enabled, so rendering 1000+ rows there would add page weight for everyone and undercut the point of this PR.
  • jQuery was loaded twice with different versions — 3.6.1 in index.html, then 3.6.0 again in _includes/scripts.html. custom.js is the only jQuery consumer in the repo and index.html loads its own, so the duplicate is dropped.

Verification

I drove the real custom.js against the real registry data over HTTP in jsdom (real jQuery 3.6.1, real DOM, real sorttable.js). 31/31 checks pass, covering:

  • loading spinner visible while data is in flight; badges say "Loading", never 0
  • successful render (1012 rows), correct counts (1012 resources / 153 KGs)
  • resource-type, taxon (hierarchy-expanded), and search filters all still work on the slim data
  • empty search result shows the info "no resources found", not an error
  • error state: distinct alert, badges with bg-danger, Retry button present
  • Retry recovers and renders the full table with normal badges
  • summary-missing → transparent fallback to kgs.jsonld
  • taxon-mapping failure → visible warning, table still renders

I also diffed the summary against kgs.yml field-by-field across all 1012 resources: identical values, and zero rendering differences for the publication button (the 111 resources whose publications count differs all have empty lists, which render the same disabled button either way).

Not covered: no local Jekyll install, so the full site build isn't exercised here — but resources is confirmed as a top-level key in _config.yml, so the {% raw %}{{ site.resources | size }}{% endraw %} in the new noscript block resolves.

🤖 Generated with Claude Code

The main page fetched the full ~30 MB registry/kgs.jsonld on every visit
and parsed it before rendering anything, while the count badges were
hardcoded to "0 resources / 0 KGs". For several seconds the registry
looked empty, with no indication that data was still loading. A failed
load reset the badges to 0 as well, so an error was nearly
indistinguishable from both "still loading" and "registry is empty".

Load speed: the table only reads ~13 fields per resource, so add
util/make_summary_json.py to generate registry/kgs-summary.json with just
those fields, and fetch that instead. 28.5 MB -> 0.67 MB raw, 1.75 MB ->
0.16 MB gzipped (what GitHub Pages actually serves). Fetch + parse drops
from ~880 ms to ~30 ms on localhost. The full kgs.jsonld is untouched and
still offered for download; custom.js falls back to it if the summary is
missing, e.g. on a deployment built before this change.

Loading state: the badges and table area now ship a "Loading registry
data..." spinner in the static markup, so it is visible from first paint
rather than only after the fetch resolves. Counts are only shown once
data has actually loaded, and "No resources found" now appears only after
a real filter returns nothing.

Error state: a failed load shows a distinct error alert naming the cause,
explaining that this is a load failure rather than an empty registry, and
offering a Retry button. Badges go to "-" with bg-danger instead of "0",
so failure can never be mistaken for loading or emptiness. A failed
taxon_mapping.yaml fetch now shows a dismissible warning that taxon
filtering is degraded, instead of only logging to the console.

Also fixes two problems found while investigating:
- The <noscript> fallback iterated site.ontologies, which does not exist
  (the key is site.resources), so it rendered an empty table with
  OBO-style links irrelevant to this registry. It now shows a short
  notice with data download links. Kept deliberately small: <noscript>
  content is downloaded by every browser, so rendering 1000+ rows there
  would add page weight for everyone.
- jQuery was loaded twice with different versions (3.6.1 in index.html,
  then 3.6.0 again in _includes/scripts.html). custom.js is the only
  jQuery consumer and index.html loads its own, so drop the duplicate.

Verified by driving the real custom.js against the real registry data
over HTTP in jsdom: 31 checks covering the loading state, successful
render (1012 rows), domain/type/taxon/search filters, the empty-result
path, the error state, Retry recovery, the kgs.jsonld fallback, and the
degraded-taxon warning.

Refs #660

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@caufieldjh
caufieldjh merged commit 336984b into main Jul 15, 2026
2 checks passed
@caufieldjh
caufieldjh deleted the issue-660-main-page-loading branch July 15, 2026 22:20
caufieldjh added a commit that referenced this pull request Jul 15, 2026
After #661 removed the 30 MB download, the main page still took ~2.3s from
navigation to a rendered table. Profiling showed that was not our JS (a CPU
profile over the whole load found 1151 ms idle, 799 ms "(program)", and only
98 ms of JS) but two structural problems.

1. The data request was blocked behind the CDN stylesheets.

A parser-inserted <script> must wait for all pending stylesheets before it
executes. custom.js sits at the end of <body> and runs inside
jQuery(document).ready(), so the fetch could not start until the CDN CSS and
jQuery had both resolved. Measured: registry-data.js finished downloading at
107 ms, but the fetch did not start until 3917 ms, behind bootstrap.min.css.

The fetch needs neither jQuery nor CSS, so it now lives in a dependency-free
assets/js/registry-data.js loaded from the TOP of <head>, above the stylesheet
links, where nothing blocks it. The request now goes out at ~76-105 ms, in
parallel with the CDN assets. custom.js awaits the in-flight promise. Retry
still issues a fresh request rather than replaying the settled one.

2. Building 1012 rows of DOM cost ~600 ms, and was repaid on every filter.

Isolating the phases: building the row HTML string for 1012 rows takes 16 ms,
but inserting it (~1977 KB of HTML, 8000+ cells) takes 597 ms; the first 50
rows cost 47 ms. So the cost is DOM construction, and the fix is to not build
nodes until they are needed. (content-visibility: auto was tested first and
only gave 1.2x, since it skips layout but not construction.)

The table now renders 60 rows and appends more as an IntersectionObserver
sentinel scrolls into view. Because only part of the table is in the DOM,
sorting can no longer reorder rows the way sorttable.js did -- it now sorts the
dataset and re-renders from the top, preserving the aria-sort and chevron
behavior. sorttable.js has no remaining references and is removed; nothing else
used it (no CSS, no other tables).

Also removed as part of this: the 50-row setTimeout chunking (it split a 16 ms
string concat across ~21 macrotasks and inserted the DOM only at the end, so it
bought nothing), the MutationObserver that re-ran initSortableTables on every
mutation, and the dead $tableMain/#table-main no-ops. renderTable no longer
shows a spinner on re-render, which fixes the flicker on every keystroke noted
in #662.

Measured on a local Jekyll build, navigation -> first row (median of 3):

                       localhost      5 Mbps/40ms
  before #661           2863 ms         49416 ms
  after #661            2340 ms          3303 ms
  after this change      556 ms          2150 ms

A filter change from 1012 -> 153 rows drops from ~263 ms to ~125 ms.

Verified in Chromium against the built site: 17 checks covering the fetch
ordering, batch rendering, full-count badges, scrolling to all 1012 rows,
whole-dataset sorting in both directions (descending correctly yields "zwd",
which DOM-only sorting could not), license sorting through the nested object,
aria-sort state, filters, search, the empty-result path, and absence of spinner
flicker.

Known tradeoff: in-page Ctrl-F only finds rows that have been scrolled into
existence. The search box searches the whole dataset and is unaffected.

Refs #662

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Main page is slow to load and shows '0 resources' while data is still loading

1 participant