diff --git a/README.md b/README.md index 18424d5d..6032a40d 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,9 @@ docker exec -it bash -c "eslint src --ext .jsx,.js" docker exec -it oclmap-web-1 bash -c "npm test" ``` +### i18n key audit +`npm run i18n:audit` lists the English keys in `src/i18n/locales/en/translations.json` that the source doesn't use, and compares each locale with English. See [scripts/i18n-audit/README.md](scripts/i18n-audit/README.md). + #### Major/minor version increase In order to increase major/minor version you need to set the new version in [package.json](package.json). diff --git a/package.json b/package.json index 2407f34d..0a88b712 100644 --- a/package.json +++ b/package.json @@ -38,7 +38,8 @@ "eslint": "./node_modules/.bin/eslint ./src --ext .jsx,.js", "test": "node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON --test 'src/**/__tests__/**/*.test.js'", "test:integration": "node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON --import ./test-integration-setup.js --test 'src/**/__integration__/**/*.test.js'", - "test:all": "npm test && npm run test:integration" + "test:all": "npm test && npm run test:integration", + "i18n:audit": "node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON scripts/i18n-audit/audit.mjs" }, "devDependencies": { "@babel/core": "^7.26.10", diff --git a/scripts/i18n-audit/README.md b/scripts/i18n-audit/README.md new file mode 100644 index 00000000..b1665a7f --- /dev/null +++ b/scripts/i18n-audit/README.md @@ -0,0 +1,132 @@ +# i18n key audit + +`npm run i18n:audit` reports which English keys in +`src/i18n/locales/en/translations.json` the Mapper's source uses, and how each +other locale compares with English (OpenConceptLab/ocl_issues#2880). + +- [`report.md`](report.md) is its output for this tree. +- [`findings.md`](findings.md) is the review done with it in October 2026: + the manual check of the unreferenced keys, and the claims it was asked to + assess. +- [`families.json`](families.json) declares the keys some dynamic call sites + can produce (see [Namespace-wide families](#namespace-wide-families)). +- The logic is in `src/i18n/audit.js`, tested by `src/i18n/__tests__/audit.test.js` + and `auditCli.test.js`; [`audit.mjs`](audit.mjs) reads the files and prints + the report. + +## Run it + +```bash +npm run i18n:audit # print the report +npm run i18n:audit -- --out scripts/i18n-audit/report.md # refresh report.md +npm run i18n:audit -- --root # audit another tree, e.g. an older commit +``` + +`--root` reads `src/` from the other checkout but keeps this checkout's +`families.json`; the report flags any declaration that doesn't fit that tree. +To audit an older commit: `git worktree add `, then +`npm run i18n:audit -- --root ` from here. + +The same tree always gives the same report: it has no dates or absolute paths, +and no list in it depends on the order files are read in or on the locale. To check that `report.md` is current, refresh +it and run `git diff --exit-code scripts/i18n-audit/report.md`. + +## What counts as a reference + +The audit parses every `.js` and `.jsx` file under `src/`, except those in +`__tests__/` and `__integration__/`, with Babel. It reads the key argument of +every call to `t` or `.t` (`t()`, `i18n.t()`, `this.props.t()`, +optional ones like `i18n?.t()` too), and every `i18nKey` attribute. For each +it works out the keys it can produce: + +| Key expression | Counts as | +|---|---| +| `'common.save'`, or a template literal without a placeholder | that key | +| `cond ? 'a.x' : 'a.y'`, `custom \|\| 'a.x'` | each branch | +| `['a.x', 'a.y']` (i18next fallback keys) | each element | +| a variable assigned once (`const key = …`) | its value, worked out the same way | +| `` `map_project.preview_limit_title_${kind}` ``, `'map_project.view_' + state` | a dynamic key family: every key the pattern (`map_project.view_*`) matches | +| a parameter, a property, a function call | a call site the audit can't read | + +A string literal anywhere in the scanned source that equals a key counts as +well. That covers keys stored in config and passed to `t()` later, such as +the column definitions in `ResultConstants.js` (`labelKey: 'common.id'`) that +`TableResults.jsx` passes to `t(headCell.labelKey)`. + +Each English key goes in the first of these groups that fits: referenced by a +literal key, matched only by a dynamic key family, mentioned only as a string +literal, or unreferenced. + +### Namespace-wide families + +A family whose whole key after the namespace is dynamic, such as +`'map_project.' + id`, matches every key in that namespace, so on its own it +would hide every unused `map_project` key. `families.json` declares the keys +such a call site can produce, worked out by reading the code, with the evidence: + +```json +{ + "file": "src/components/map-projects/ScoreBucketButton.jsx", + "expression": "'map_project.' + id", + "keys": ["map_project.recommended", "map_project.available", "map_project.low_ranked"], + "evidence": "ScoreBucketButton renders BucketButton with id recommended, available and low_ranked." +} +``` + +`expression` is the key argument's source text with its whitespace collapsed, +as the report shows it; the declaration covers every call in `file` with that +text. A narrower family can be declared the same way to tighten it. + +The report's *Needs attention* section lists: + +- a namespace-wide family with no declaration (every key it matches then + counts as used); +- a declared key the expression can't produce, or the catalog lacks; +- a declaration whose call site has gone; +- a literal key the English catalog lacks (i18next shows the raw key, unless + the call passes a default). + +## Locales + +Every directory under `src/i18n/locales/` is compared with English, key by key: + +| | Means | +|---|---| +| Missing | Not in the locale; i18next falls back to English. | +| Blank | Empty or only whitespace; i18next shows the blank, so the text disappears. | +| Same as English | Untranslated, or right as it is (`ID`, `JSON`, `OK`); needs a translator to tell. | +| Not in English | Only this locale has it: a leftover, or a key English lacks. | + +## Limits of static detection + +The audit can count a key as used when it isn't, and the opposite in the cases +below. Check a key by hand before removing it. + +- A translation function under another name (`const { t: translate } = …`), + or reached through a computed property (`i18n['t']`), isn't read. Today every + call is `t()` or `this.props.t()`. +- A key built outside the call (`const key = 'map_project.' + id; t(key)`) + is read only when it's a variable in scope that's assigned once; a key built + in another function and passed in is a call site the audit can't read. +- A pattern matches by text, so `map_project.decision_*` covers + `map_project.decision_tab_*` too. An undeclared family can only over-count, + so a key it matches may still be unused. +- A declaration is only as good as its evidence. The audit notices when a + call site's text changes, not when the code that feeds it values does. +- A string literal that equals a key counts even if it's never translated, or + sits in config the Mapper never shows: `ResultConstants.js` has columns for + users, orgs and repos, but the Mapper only lists concepts. +- Code that never runs still counts, such as `ConceptHeader.jsx`'s + `false && …` block, which calls `t('common.actions')`. +- Plural and context suffixes (`key_one`, `key_other`, `key_male`) aren't + derived from `t(key, {count})`, and i18next nesting (`$t(other.key)` in a + value) isn't followed. The catalogs use neither today. +- A `` without `i18nKey` uses its children as the key; the audit lists + it as a call site it can't read. There are none today. + +## Removing keys + +The unreferenced list is a list of candidates. Check each key by hand: search +the source for its last segment, and look at the history of the code that last +used it (`git log -S'' -- src`). Then remove keys from every locale at once, +in a separate, reviewed change. diff --git a/scripts/i18n-audit/audit.mjs b/scripts/i18n-audit/audit.mjs new file mode 100644 index 00000000..9f9e1ad8 --- /dev/null +++ b/scripts/i18n-audit/audit.mjs @@ -0,0 +1,43 @@ +#!/usr/bin/env node +// Prints the i18n key audit (OpenConceptLab/ocl_issues#2880) for this +// checkout, or for another one with --root. See README.md. +// +// npm run i18n:audit # print the report +// npm run i18n:audit -- --out scripts/i18n-audit/report.md # write it +// npm run i18n:audit -- --root # audit another checkout's src/ +import fs from 'node:fs' +import path from 'node:path' +import { fileURLToPath } from 'node:url' +import { parseArgs } from 'node:util' + +import { auditKeys, renderReport } from '../../src/i18n/audit.js' + +const { values: options } = parseArgs({ options: { root: { type: 'string' }, out: { type: 'string' } } }) +const here = path.dirname(fileURLToPath(import.meta.url)) +const root = path.resolve(options.root || path.join(here, '..', '..')) + +const SKIP = new Set(['__tests__', '__integration__', 'node_modules']) +const listSources = dir => fs.readdirSync(dir, { withFileTypes: true }).flatMap(entry => { + const full = path.join(dir, entry.name) + if(entry.isDirectory()) + return SKIP.has(entry.name) ? [] : listSources(full) + return /\.jsx?$/.test(entry.name) ? [full] : [] +}) + +const relative = file => path.relative(root, file).split(path.sep).join('/') +const sources = listSources(path.join(root, 'src')).map(file => ({ file: relative(file), code: fs.readFileSync(file, 'utf8') })) + +const localesDir = path.join(root, 'src', 'i18n', 'locales') +const catalogs = Object.fromEntries(fs.readdirSync(localesDir).map(locale => [ + locale, JSON.parse(fs.readFileSync(path.join(localesDir, locale, 'translations.json'), 'utf8')) +])) + +// This checkout's declarations, also for --root: an older tree has none, and +// the report flags any that no longer fit it. +const families = JSON.parse(fs.readFileSync(path.join(here, 'families.json'), 'utf8')) + +const report = renderReport(auditKeys({ sources, catalogs, families })) +if(options.out) + fs.writeFileSync(options.out, report) +else + process.stdout.write(report) diff --git a/scripts/i18n-audit/families.json b/scripts/i18n-audit/families.json new file mode 100644 index 00000000..1ddf199e --- /dev/null +++ b/scripts/i18n-audit/families.json @@ -0,0 +1,32 @@ +[ + { + "file": "src/components/concepts/ConceptProperties.jsx", + "expression": "`concept.${prop}`", + "keys": ["concept.concept_class", "concept.datatype"], + "evidence": "prop is a property code. The first call translates only concept_class and datatype; the second runs when the concept has no properties, and sortedProperties is then those same two." + }, + { + "file": "src/components/map-projects/MapProject.jsx", + "expression": "'map_project.' + qualityBucket", + "keys": ["map_project.recommended", "map_project.available", "map_project.low_ranked"], + "evidence": "getScoreDetails() in viewBuilders.js sets qualityBucket to recommended, available or low_ranked, or leaves it unset, and the call then uses map_project.unranked." + }, + { + "file": "src/components/map-projects/MapProject.jsx", + "expression": "'map_project.' + quality.bucket", + "keys": ["map_project.recommended", "map_project.available", "map_project.low_ranked"], + "evidence": "getRowQuality() in rowQuality.js: getBucket() returns recommended, available or low_ranked, and a proposed match without a score is low_ranked." + }, + { + "file": "src/components/map-projects/ScoreBucketButton.jsx", + "expression": "'map_project.' + id", + "keys": ["map_project.recommended", "map_project.available", "map_project.low_ranked"], + "evidence": "ScoreBucketButton renders BucketButton with id recommended, available and low_ranked." + }, + { + "file": "src/components/search/SearchResults.jsx", + "expression": "`search.${resource}`", + "keys": ["search.concepts"], + "evidence": "map-projects/Search.jsx and Candidates.jsx render SearchResults, both with resource='concepts'." + } +] diff --git a/scripts/i18n-audit/findings.md b/scripts/i18n-audit/findings.md new file mode 100644 index 00000000..be52d957 --- /dev/null +++ b/scripts/i18n-audit/findings.md @@ -0,0 +1,147 @@ +# i18n audit findings, October 2026 + +The review OpenConceptLab/ocl_issues#2880 asked for, done with `npm run i18n:audit` +on `main` at 0f6b0be. [`report.md`](report.md) has the full lists, and the +[README](README.md) the method and its limits. No catalog key is removed here: +that's a separate, reviewed change. + +## Counts + +Of the 778 English keys: + +| | Keys | +|---|---:| +| Referenced by a literal key | 487 | +| Matched only by a dynamic key family | 62 | +| Mentioned only as a string literal | 21 | +| Unreferenced | 208 | + +Every dynamic key family is accounted for. Of the 20, 13 have a literal part +narrow enough to count every key their pattern matches. The other 7 span a +whole namespace: two calls of `` `concept.${prop}` ``, one of +`` `search.${resource}` `` and four of `'map_project.' + `. For those, +`families.json` declares the keys, citing the code that produces them. + +Of the 4 call sites the audit can't read, three get their keys from string +literals, and the fourth gets none today: + +- `AnnouncementBanner.jsx` `t(labelKey)`: `ANNOUNCEMENT_LINKS`, in the same file. +- `MapperQuotaChip.jsx` `t(labelKey)`: the three `meterLine('map_project.preview_quota_…')` calls below it. +- `TableResults.jsx` `t(headCell.labelKey)`: column configs in `ResultConstants.js`, `Candidates.jsx` and `Search.jsx`. +- `MultiAlgoSelector.jsx` `t(algo.name?.props?.i18nKey)`: the `i18nKey` of an algorithm whose name is a + `` element. No algorithm defines its name that way today. + +Apart from `src/i18n/audit.js` itself, every file the audit scans is imported, +directly or through others, from `src/index.jsx`, so no reference comes from a +file the app never loads. + +## The unreferenced keys, checked by hand + +For each of the 208 keys, the source (tests excluded) was searched for the +key's last segment inside any string or template literal. 58 keys had hits, +and none of the hits builds a key: + +- CSS values and class names (`clear`, `display`, `visibility`), URL segments + (`/compare`, `versions/`), API fields (`first_name`, `include_retired`, + `map_types`) and storage keys (`session_expired`). +- Log action types: `ProjectLogs.jsx` picks an icon by action (`created`, + `auto_matched`); it doesn't translate it. +- Display modes: `DisplayMenu.jsx` labels `card` and `table` with + `search.list_view` and `search.table_view`, not `search.card` and `search.table`. +- Sort fields: `SortMenu.jsx` labels them with `startCase()`, not `t()`, so + `search.score` is unused. + +All 208 stand. + +14 of the 21 keys mentioned only as a string literal are probably unused too. +They appear only in `ALL_COLUMNS` in `ResultConstants.js`, in the column sets +for mappings, repos, orgs, users and the URL registry. `TableResults.jsx` is the +only reader, through `SearchResults`, and both places that render +`SearchResults` pass `resource='concepts'`. These are `common.created_on`, the +three `mapping.*`, `repo.repo_id`, `repo.repo_type`, the two `url_registry.*` +and the six `user.*` keys in the report's string-literal table. Removing them +means trimming those column sets as well. + +## Keys the code uses that English lacks + +The report's *Needs attention* section lists five: + +- `common.none` (`MatchDetailsPanel.jsx`), `common.saved` (`MultiAlgoSelector.jsx`) + and `search.save_default_filters` (`SearchFilters.jsx`) render as the raw key + in every language. +- `map_project.candidates_legacy_shape` (`MapProject.jsx`) and + `map_project.target_repo_version_required` (`ConfigurationForm.jsx`, + `MapProject.jsx`) pass an English default, so they show English in every + language. + +## Locales + +| Locale | Keys | Missing | Blank | Same as English | Not in English | +|---|---:|---:|---:|---:|---:| +| es | 780 | 0 | 0 | 9 | 2 | +| pt-BR | 778 | 0 | 0 | 12 | 0 | +| zh | 778 | 0 | 0 | 5 | 0 | + +- No locale is missing a key or has a blank value. The 8 keys es and zh each + lacked, as the ticket records, were added by a0de2db + (OpenConceptLab/oclmap#93) on 2026-10-08. +- Most values that are the same as English are names or acronyms that read the + same (`ID`, `JSON`, `OK`, `AutoMatch`, `Token`, `Avatar`). For a translator to + confirm: pt-BR `concept.namespace` (`Namespace`), `repo.source` and + `map_project.source` (`Source`), and zh `common.with` (`with`, which is + unreferenced anyway). +- es has two keys English doesn't: `common.or_go_to_your` and `concept.history`. + +## The claims in the review of OpenConceptLab/oclmap#90 + +### "228 of 752 English keys are unused" + +752 English keys is the state of the pt-BR branch from b05133c to 25e2f8a, +2026-10-02 to 2026-10-05. ca4024e then removed three keys, and the merge +(06d8841) left `main` with 771. At 25e2f8a, the audit counts 211 of the 752 as +unreferenced (460 literal, 62 dynamic, 19 string literal only). To repeat it: +`git worktree add 25e2f8a`, then `npm run i18n:audit -- --root `. + +228 can't be reproduced exactly, since the claim doesn't state its rules, and +the count depends on them. The table applies simpler rules to the same tree: +the first two rows are plain regexes over the source text, the next four use +the audit's scan with fewer rules, and only the last row is the audit's own +output. At 25e2f8a: + +| Rules | Unused of 752 | +|---|---:| +| A plain regex for `t('key')` | 307 | +| …and for `i18nKey='key'` | 300 | +| Parsed: literal keys, conditionals, fallback arrays, constants | 292 | +| …and template-literal families | 212 | +| …and concatenation families (each counting all keys it matches) | 171 | +| …and string literals | 155 | +| The audit: as above, with namespace-wide families declared | 211 | + +Of the first 35 keys in the claim's list, `common.a` to `common.read_more`, +the audit at 25e2f8a has 33 unreferenced. The other two are in use: column +configs pass `common.action` (`Candidates.jsx`, `Search.jsx`) and +`common.created_on` (`ResultConstants.js`) to `t()` as `labelKey`. Rules that miss keys passed that way would add the 19 +string-literal-only keys to the audit's 211, giving 230, close to 228. + +The ticket's "301 unmatched by a basic static regex" fits the plain regex for +`t('key')` and `i18nKey='key'` (300). + +### `user.recent_activity`, `user.own_recent_activity`, `user.recent_activity_of` + +The claim that all three are unused is right: + +- `user.recent_activity` came in with the import from oclweb3 (5224e29, + 2025-02-06). Its only call site, in `src/components/common/Events.jsx`, went + when c619b5e deleted that file on 2026-06-30, so it was unused from then on. +- `user.own_recent_activity` and `user.recent_activity_of` were added by + b05133c and removed by ca4024e, both on the pt-BR branch, so neither reached + `main`. No source file outside the catalogs ever named them + (`git log -S'own_recent_activity' -- src` finds only those two commits). +- At 25e2f8a the audit lists all three as unreferenced; they are the only + difference between its list there and its list on `main` today. ca4024e + removed all three, so none exists on `main`. The later note that the last two + were "never added" holds for `main` only. + +`user.your`, the other key `Events.jsx` used, is still in every catalog and is +unreferenced. diff --git a/scripts/i18n-audit/report.md b/scripts/i18n-audit/report.md new file mode 100644 index 00000000..f75b48bd --- /dev/null +++ b/scripts/i18n-audit/report.md @@ -0,0 +1,336 @@ +# i18n key audit + +Generated by `npm run i18n:audit` from the source under `src/` (tests excluded) and the +catalogs in `src/i18n/locales/`. Rerun it rather than edit this file. The method and its +limits are in `scripts/i18n-audit/README.md`. + +## Summary + +| English keys | Count | +|---|---:| +| English catalog | 778 | +| Referenced by a literal key | 487 | +| Matched only by a dynamic key family | 62 | +| Mentioned only as a string literal | 21 | +| **Unreferenced** | **208** | + +Scanned 171 source files: 613 call sites with a literal key, 20 dynamic key families, 4 call sites whose key the audit can't read. + +## Needs attention + +- `common.none` is referenced (src/components/map-projects/MatchDetailsPanel.jsx:135) but isn't in the English catalog. +- `common.saved` is referenced (src/components/map-projects/MultiAlgoSelector.jsx:403) but isn't in the English catalog. +- `map_project.candidates_legacy_shape` is referenced (src/components/map-projects/MapProject.jsx:920) but isn't in the English catalog. +- `map_project.target_repo_version_required` is referenced (src/components/map-projects/ConfigurationForm.jsx:247, src/components/map-projects/MapProject.jsx:1580) but isn't in the English catalog. +- `search.save_default_filters` is referenced (src/components/search/SearchFilters.jsx:307) but isn't in the English catalog. + +## Locales compared with English + +| Locale | Keys | Missing | Blank | Same as English | Not in English | +|---|---:|---:|---:|---:|---:| +| es | 780 | 0 | 0 | 9 | 2 | +| pt-BR | 778 | 0 | 0 | 12 | 0 | +| zh | 778 | 0 | 0 | 5 | 0 | + +### es + +- Missing: none +- Blank: none +- Same as English: `common.id` (`ID`), `common.json` (`JSON`), `common.ok` (`OK`), `map_project.auto_match` (`AutoMatch`), `map_project.bridge_canonical_short` (`Bridge:`), `map_project.lookup_configuration_token` (`Token`), `repo.experimental` (`Experimental`), `url_registry.global` (`Global`), `user.avatar` (`Avatar`) +- Not in English: `common.or_go_to_your`, `concept.history` + +### pt-BR + +- Missing: none +- Blank: none +- Same as English: `common.id` (`ID`), `common.json` (`JSON`), `common.ok` (`OK`), `concept.namespace` (`Namespace`), `map_project.auto_match` (`AutoMatch`), `map_project.bridge_canonical_short` (`Bridge:`), `map_project.lookup_configuration_token` (`Token`), `map_project.source` (`Source`), `repo.experimental` (`Experimental`), `repo.source` (`Source`), `url_registry.global` (`Global`), `user.avatar` (`Avatar`) +- Not in English: none + +### zh + +- Missing: none +- Blank: none +- Same as English: `common.id` (`ID`), `common.json` (`JSON`), `common.with` (`with`), `map_project.api_url` (`API URL`), `map_project.auto_match` (`AutoMatch`) +- Not in English: none + +## Dynamic key families + +A key built at run time, as a pattern. A declared family counts only the keys +`scripts/i18n-audit/families.json` lists for it; any other counts every key its pattern matches. + +| Call site | Key expression | Pattern | Keys | +|---|---|---|---| +| src/components/common/QuotaDialog.jsx:31 | ``[`quota.consequence.${meter}.${surface}`, `quota.consequence.${meter}.default`, 'quota.consequence.generic']`` | `quota.consequence.*.*` | `quota.consequence.ai_assistant_calls.auto_match`, `quota.consequence.ai_assistant_calls.default`, `quota.consequence.ai_assistant_change_comments.default`, `quota.consequence.ai_assistant_change_comments.tbv3_change_comment`, `quota.consequence.match_operations.auto_match`, `quota.consequence.match_operations.default`, `quota.consequence.match_operations.tbv3_match`, `quota.consequence.projects.default`, `quota.consequence.projects.new_project`, `quota.consequence.rows.default`, `quota.consequence.rows.import` | +| src/components/common/QuotaDialog.jsx:31 | ``[`quota.consequence.${meter}.${surface}`, `quota.consequence.${meter}.default`, 'quota.consequence.generic']`` | `quota.consequence.*.default` | `quota.consequence.ai_assistant_calls.default`, `quota.consequence.ai_assistant_change_comments.default`, `quota.consequence.match_operations.default`, `quota.consequence.projects.default`, `quota.consequence.rows.default` | +| src/components/common/QuotaDialog.jsx:44 | `` `quota.meter.${meter}` `` | `quota.meter.*` | `quota.meter.ai_assistant_calls`, `quota.meter.ai_assistant_change_comments`, `quota.meter.generic`, `quota.meter.match_operations`, `quota.meter.projects`, `quota.meter.rows` | +| src/components/concepts/ConceptProperties.jsx:29 | `` `concept.${prop}` `` | `concept.*` | declared: `concept.concept_class`, `concept.datatype` | +| src/components/concepts/ConceptProperties.jsx:34 | `` `concept.${prop}` `` | `concept.*` | declared: `concept.concept_class`, `concept.datatype` | +| src/components/concepts/LocaleForm.jsx:30 | `` `concept.form.${localeType}` `` | `concept.form.*` | `concept.form.concept_class`, `concept.form.concept_details.header`, `concept.form.datatype`, `concept.form.description`, `concept.form.descriptions.header`, `concept.form.external_id`, `concept.form.id`, `concept.form.locale`, `concept.form.locale_preferred`, `concept.form.name`, `concept.form.names.header`, `concept.form.type` | +| src/components/map-projects/ConfigurationForm.jsx:525 | `` `map_project.preview_config_unavailable_${reason}` `` | `map_project.preview_config_unavailable_*` | `map_project.preview_config_unavailable_custom_algorithms`, `map_project.preview_config_unavailable_org_projects`, `map_project.preview_config_unavailable_title` | +| src/components/map-projects/DataGridControls.jsx:151 | `` `map_project.decision_${_decision}` `` | `map_project.decision_*` | `map_project.decision_changed_to_none`, `map_project.decision_exclude`, `map_project.decision_map`, `map_project.decision_none`, `map_project.decision_propose`, `map_project.decision_tab_candidates`, `map_project.decision_tab_discuss`, `map_project.decision_tab_propose`, `map_project.decision_tab_search` | +| src/components/map-projects/MapProject.jsx:748 | `` `map_project.preview_limit_title_${newProjectBlockReason}` `` | `map_project.preview_limit_title_*` | `map_project.preview_limit_title_access`, `map_project.preview_limit_title_ai_assistant`, `map_project.preview_limit_title_custom_algorithms`, `map_project.preview_limit_title_generic`, `map_project.preview_limit_title_match_operations`, `map_project.preview_limit_title_match_operations_not_entitled`, `map_project.preview_limit_title_org_projects`, `map_project.preview_limit_title_projects`, `map_project.preview_limit_title_projects_not_entitled`, `map_project.preview_limit_title_rows`, `map_project.preview_limit_title_rows_not_entitled`, `map_project.preview_limit_title_scispacy` | +| src/components/map-projects/MapProject.jsx:1008 | `'map_project.' + qualityBucket` | `map_project.*` | declared: `map_project.recommended`, `map_project.available`, `map_project.low_ranked` | +| src/components/map-projects/MapProject.jsx:1161 | `'map_project.' + quality.bucket` | `map_project.*` | declared: `map_project.recommended`, `map_project.available`, `map_project.low_ranked` | +| src/components/map-projects/MapProject.jsx:1172 | `'map_project.' + quality.bucket` | `map_project.*` | declared: `map_project.recommended`, `map_project.available`, `map_project.low_ranked` | +| src/components/map-projects/MapProject.jsx:6484 | `'map_project.view_' + getStateFromIndex(rowIndex).toLowerCase()` | `map_project.view_*` | `map_project.view_all`, `map_project.view_approved`, `map_project.view_existing_analysis`, `map_project.view_proposed`, `map_project.view_raw_json`, `map_project.view_readyforreview`, `map_project.view_reviewed`, `map_project.view_unmapped` | +| src/components/map-projects/MapProject.jsx:6540 | `'map_project.decision_tab_' + _tab` | `map_project.decision_tab_*` | `map_project.decision_tab_candidates`, `map_project.decision_tab_discuss`, `map_project.decision_tab_propose`, `map_project.decision_tab_search` | +| src/components/map-projects/MatchSummaryCard.jsx:48 | `'map_project.view_' + label.toLowerCase()` | `map_project.view_*` | `map_project.view_all`, `map_project.view_approved`, `map_project.view_existing_analysis`, `map_project.view_proposed`, `map_project.view_raw_json`, `map_project.view_readyforreview`, `map_project.view_reviewed`, `map_project.view_unmapped` | +| src/components/map-projects/PreviewLimitDialog.jsx:30 | `` `map_project.preview_limit_title_${kind}` `` | `map_project.preview_limit_title_*` | `map_project.preview_limit_title_access`, `map_project.preview_limit_title_ai_assistant`, `map_project.preview_limit_title_custom_algorithms`, `map_project.preview_limit_title_generic`, `map_project.preview_limit_title_match_operations`, `map_project.preview_limit_title_match_operations_not_entitled`, `map_project.preview_limit_title_org_projects`, `map_project.preview_limit_title_projects`, `map_project.preview_limit_title_projects_not_entitled`, `map_project.preview_limit_title_rows`, `map_project.preview_limit_title_rows_not_entitled`, `map_project.preview_limit_title_scispacy` | +| src/components/map-projects/PreviewLimitDialog.jsx:37 | `` `map_project.preview_limit_body_${kind}` `` | `map_project.preview_limit_body_*` | 19 keys | +| src/components/map-projects/PreviewLimitDialog.jsx:38 | `` `map_project.preview_limit_body_${kind}_no_count` `` | `map_project.preview_limit_body_*_no_count` | `map_project.preview_limit_body_access_no_count`, `map_project.preview_limit_body_ai_assistant_no_count`, `map_project.preview_limit_body_custom_algorithms_no_count`, `map_project.preview_limit_body_generic_no_count`, `map_project.preview_limit_body_match_operations_no_count`, `map_project.preview_limit_body_match_operations_not_entitled_no_count`, `map_project.preview_limit_body_org_projects_no_count`, `map_project.preview_limit_body_projects_no_count`, `map_project.preview_limit_body_projects_not_entitled_no_count`, `map_project.preview_limit_body_rows_no_count`, `map_project.preview_limit_body_rows_not_entitled_no_count`, `map_project.preview_limit_body_scispacy_no_count` | +| src/components/map-projects/ScoreBucketButton.jsx:25 | `'map_project.' + id` | `map_project.*` | declared: `map_project.recommended`, `map_project.available`, `map_project.low_ranked` | +| src/components/search/SearchResults.jsx:178 | `` `search.${resource}` `` | `search.*` | declared: `search.concepts` | + +## Call sites whose key the audit can't read + +The key comes from a parameter or a property. Such keys are usually string literals +elsewhere, which the next section counts. + +| Call site | Key expression | +|---|---| +| src/components/app/AnnouncementBanner.jsx:109 | `labelKey` | +| src/components/map-projects/MapperQuotaChip.jsx:25 | `labelKey` | +| src/components/map-projects/MultiAlgoSelector.jsx:304 | `algo.name?.props?.i18nKey` | +| src/components/search/TableResults.jsx:48 | `headCell.labelKey` | + +## Mentioned only as a string literal + +| Key | Where | +|---|---| +| `announcement.link_label` | src/components/app/AnnouncementBanner.jsx:20 | +| `announcement.pricing_link_label` | src/components/app/AnnouncementBanner.jsx:21 | +| `common.action` | src/components/map-projects/Candidates.jsx:208, src/components/map-projects/Search.jsx:67 | +| `common.created_on` | src/components/search/ResultConstants.js:43 | +| `map_project.preview_quota_ai_calls` | src/components/map-projects/MapperQuotaChip.jsx:41 | +| `map_project.preview_quota_match_operations` | src/components/map-projects/MapperQuotaChip.jsx:40 | +| `map_project.preview_quota_projects` | src/components/map-projects/MapperQuotaChip.jsx:36 | +| `mapping.fromAndTargetSource` | src/components/search/ResultConstants.js:29 | +| `mapping.fromConcept` | src/components/search/ResultConstants.js:30 | +| `mapping.map_type` | src/components/search/ResultConstants.js:31 | +| `mapping.same_as_mappings` | src/components/map-projects/Candidates.jsx:233, src/components/map-projects/Search.jsx:87 | +| `repo.repo_id` | src/components/search/ResultConstants.js:57 | +| `repo.repo_type` | src/components/search/ResultConstants.js:37, src/components/search/ResultConstants.js:58 | +| `url_registry.canonical_url` | src/components/search/ResultConstants.js:55 | +| `url_registry.target_owner` | src/components/search/ResultConstants.js:56 | +| `user.company` | src/components/search/ResultConstants.js:49 | +| `user.email` | src/components/search/ResultConstants.js:48 | +| `user.joined_on` | src/components/search/ResultConstants.js:52 | +| `user.location` | src/components/search/ResultConstants.js:50 | +| `user.username` | src/components/search/ResultConstants.js:46 | +| `user.website` | src/components/search/ResultConstants.js:51 | + +## Unreferenced keys (208) + +No call site, dynamic family or string literal in the source can produce these. Check +each by hand before removing it (see the README). + +| Key | English | +|---|---| +| `auth.session_expired` | `Your session has expired. Please log in to continue.` | +| `auth.update_password` | `Update password` | +| `bookmarks.name` | `Bookmarks` | +| `bookmarks.pinned` | `Pinned` | +| `bookmarks.pinned_repos` | `Pinned repositories` | +| `common.a` | `a` | +| `common.about` | `About` | +| `common.access_level` | `Access Level` | +| `common.an` | `an` | +| `common.attribute` | `Attribute` | +| `common.clear` | `Clear` | +| `common.content_summary` | `Content summary` | +| `common.create` | `Create` | +| `common.created` | `Created` | +| `common.custom_attributes` | `Custom Attributes` | +| `common.draft` | `Draft` | +| `common.for` | `for` | +| `common.from` | `From` | +| `common.full_name` | `Full Name` | +| `common.group` | `Group` | +| `common.highlights` | `Highlights` | +| `common.history` | `History and comments` | +| `common.import` | `Import` | +| `common.json` | `JSON` | +| `common.languages` | `Languages` | +| `common.last_updated_on` | `Last updated on` | +| `common.less` | `Less` | +| `common.manage` | `Manage` | +| `common.metadata` | `Metadata` | +| `common.navigate` | `Navigate` | +| `common.ok` | `OK` | +| `common.open` | `Open` | +| `common.overview` | `Overview` | +| `common.preferred_language` | `Preferred Language` | +| `common.private` | `Private` | +| `common.public` | `Public` | +| `common.read_less` | `Read less` | +| `common.read_more` | `Read more` | +| `common.resolve` | `Resolve` | +| `common.result` | `Result` | +| `common.select` | `Select` | +| `common.short_code` | `Short Code` | +| `common.sort` | `Sort` | +| `common.statistics` | `Statistics` | +| `common.subscribed` | `Subscribed` | +| `common.summary` | `Summary` | +| `common.target` | `Target` | +| `common.updated_on` | `Last Updated` | +| `common.upload` | `Upload` | +| `common.value` | `Value` | +| `common.versions` | `Versions` | +| `common.view_all_attributes` | `View all attributes` | +| `common.with` | `with` | +| `concept.concept_classes` | `Concept Classes` | +| `concept.concepts` | `Concepts` | +| `concept.datatypes` | `Datatypes` | +| `concept.name_type` | `Name Type` | +| `concept.name_types` | `Name Types` | +| `concept.namespace_associations` | `Namespace Associations` | +| `concept.retired_concepts` | `Retired Concepts` | +| `concept.tab_bridge_concept_details` | `Bridge Concept Details` | +| `custom_attributes.add` | `Add a Custom Attribute` | +| `dashboard.anonymous_heading` | `What's new?` | +| `dashboard.community_blog` | `Community Blog` | +| `dashboard.create_repository` | `Create Repository` | +| `dashboard.events_scope_sign_in` | `You need to be signed in to see content from publishers you follow` | +| `dashboard.hello` | `Hello` | +| `dashboard.no_events` | `There are no updates to show yet.` | +| `dashboard.take_a_tour` | `View release notes and take a tour` | +| `dashboard.welcome_line` | `Welcome to OCL.` | +| `map_project.ai_assistant_limit_reached` | `You've used all of your preview AI Assistant calls. Matching will continue without AI analysis for the remaining rows.` | +| `map_project.algorithm_custom_description` | `Custom $match API` | +| `map_project.algorithm_custom_label` | `Custom API` | +| `map_project.algorithm_es_label` | `Generic Elastic Search Matching` | +| `map_project.algorithm_llm_label` | `Semantic Search (all-MiniLM-L6-v2)` | +| `map_project.auto_match_note_no_counts` | `Note: This will not affect approved input rows but will override proposed input rows` | +| `map_project.auto_matched` | `AutoMatched` | +| `map_project.bridge_terminology_search` | `Bridge terminology search<0>Premium` | +| `map_project.bridge_terminology_search_description` | `Include mappings in the <0>CIEL Interface Terminology to identify additional high quality candidates. Only available for compatible target repositories and matching algorithms.` | +| `map_project.bulk_clear_selection` | `Clear selection` | +| `map_project.ciel_bridge_terminology_candidates` | `CIEL Bridge Terminology Candidates` | +| `map_project.custom_match_api` | `Custom Match API` | +| `map_project.custom_match_api_description` | `Configure External Match API URL and Token` | +| `map_project.delete_this_project` | `Delete mapping project` | +| `map_project.enable_reranker` | `Enable Reranker` | +| `map_project.include_retired` | `Include Retired` | +| `map_project.lookup_required` | `Lookup Required` | +| `map_project.map_target` | `Map Target` | +| `map_project.match_api_token` | `Match API Token` | +| `map_project.match_api_token_placeholder` | `e.g. XXXXXXXXXXX` | +| `map_project.match_api_url` | `Match API URL` | +| `map_project.match_api_url_placeholder` | `e.g. https://api.openconceptlab.org/concepts/$match/?includeSearchMeta=true` | +| `map_project.ocl_ciel_bridge_algorithm` | `OCL CIEL Bridge Terminology` | +| `map_project.ocl_custom_algorithm` | `OCL Custom Algorithm` | +| `map_project.ocl_scispacy_loinc_algorithm` | `OCL ScispaCy LOINC Algorithm` | +| `map_project.rows_per_match_api` | `Rows per $match API` | +| `map_project.save_this_project_to_collection` | `Save target concepts to a collection` | +| `map_project.scispacy_loinc_search` | `ScispaCy LOINC search<0>Premium` | +| `map_project.select_an_algorithm` | `Select an algorithm` | +| `map_project.show_more` | `Show more` | +| `map_project.target_repo_required_on_load` | `This project is missing a target repository. Configure the target repository to see saved candidates.` | +| `mapping.from_concept_sources` | `From Concept Sources` | +| `mapping.map_types` | `Map Types` | +| `mapping.retired_mappings` | `Retired Mappings` | +| `mapping.sort_weight` | `Sort Weight` | +| `mapping.to_concept_sources` | `Target Concept Sources` | +| `mapping.type` | `Type` | +| `org.about_the_org` | `About the Organization` | +| `org.browse_members` | `Browse members` | +| `org.member` | `Member` | +| `org.member_of` | `Member of` | +| `org.org` | `Organization` | +| `org.org_have_not_created_public_repos_suffix` | `have not created any public repositories` | +| `repo.add_concept` | `Add Concept` | +| `repo.autoid_concept_description_external_id` | `Concept Description External ID Auto Assignment` | +| `repo.autoid_concept_external_id` | `Concept External ID Auto Assignment` | +| `repo.autoid_concept_mnemonic` | `Concept ID Auto Assignment` | +| `repo.autoid_concept_name_external_id` | `Concept Name External ID Auto Assignment` | +| `repo.autoid_mapping_external_id` | `Mapping External ID Auto Assignment` | +| `repo.autoid_mapping_mnemonic` | `Mapping ID Auto Assignment` | +| `repo.case_sensitive` | `Case Sensitive` | +| `repo.collection_type` | `Collection Type` | +| `repo.compare` | `Compare` | +| `repo.compare_destination_version_disabled_tooltip` | `You must select a newer version to compare` | +| `repo.compare_origin_version_disabled_tooltip` | `You must select an older version to compare` | +| `repo.compare_versions` | `Compare versions` | +| `repo.compositional` | `Compositional` | +| `repo.contact` | `Contact` | +| `repo.content_type` | `Content Type` | +| `repo.copyright` | `Copyright` | +| `repo.custom_validation_schema` | `Custom Validation Schema` | +| `repo.experimental` | `Experimental` | +| `repo.hierarchy_meaning` | `Hierarchy Meaning` | +| `repo.identifier` | `Identifier` | +| `repo.jurisdiction` | `Jurisdiction` | +| `repo.locale` | `Locale` | +| `repo.locales` | `Locales` | +| `repo.manage` | `Manage Repository` | +| `repo.no_overview` | `The owner has not created an Overview page for this repository` | +| `repo.property` | `Property` | +| `repo.publisher` | `Publisher` | +| `repo.purpose` | `Purpose` | +| `repo.refresh_summary` | `Refresh summary` | +| `repo.release_status` | `Release status` | +| `repo.repo_summary_is_calculating` | `Repo summary is calculating, please check back after some time.` | +| `repo.repo_version_summary` | `Repository Version Summary` | +| `repo.revision_date` | `Revision Date` | +| `repo.source_type` | `Source Type` | +| `repo.summary_active_concepts_from_total` | `{{active}} active concepts ({{retired}} retired)` | +| `repo.summary_active_mappings_from_total` | `{{active}} active mappings ({{retired}} retired)` | +| `repo.summary_concepts_references` | `{{count}} concept references` | +| `repo.summary_mappings_references` | `{{count}} mapping references` | +| `repo.summary_released_versions_from_total` | `{{released}} released versions ({{unreleased}} unreleased)` | +| `repo.version_needed` | `Version Needed` | +| `repo.visibility` | `Visibility` | +| `search.card` | `Card` | +| `search.confidence` | `Confidence` | +| `search.display` | `Display` | +| `search.input_placeholder` | `Search OCL...` | +| `search.input_placeholder_drawer` | `Enter a search term or tap / for options` | +| `search.loading` | `Refining your search results...` | +| `search.mappings` | `Mappings` | +| `search.orgs` | `Organizations` | +| `search.repos` | `Repositories` | +| `search.score` | `Score` | +| `search.search_all_concepts` | `Search all concepts` | +| `search.search_all_site` | `Search all of OCL for ` | +| `search.search_this_repository` | `Search this repository` | +| `search.search_this_url_registry` | `Search this URL registry for ` | +| `search.table` | `Table` | +| `search.url_registry` | `entries` | +| `search.users` | `Users` | +| `url_registry.add_entry` | `Add Canonical URL` | +| `url_registry.global` | `Global` | +| `url_registry.registry_owner` | `Registry Owner` | +| `url_registry.resolve_failed_with_entry` | `{{url}} matched an entry in the <0/> Registry but did not resolve to a repository` | +| `url_registry.resolve_failed_without_entry` | `{{url}} did not match an entry in the <0/> Registry or the Global Registry and did not resolve to a repository` | +| `url_registry.resolve_failed_without_entry_global` | `{{url}} did not match an entry in the Global Registry` | +| `url_registry.resolve_success_with_entry` | `{{url}} resolved to <0/> in <1/> using an entry in the <2/> Registry` | +| `url_registry.resolve_success_without_entry` | `{{url}} resolved to <0/> in <1/> by matching a repository in the <2/>` | +| `url_registry.test_canonical_url` | `Test a Canonical URL` | +| `url_registry.test_canonical_url_description` | `Enter a canonical URL to test how OCL will resolve it in a specific namespace, such as your user workspace or an organization that you are a member of. Use this to simulate how OCL will evaluate references and mappings in your repositories. <0>Learn more at OCL Docs` | +| `url_registry.url_registry` | `Canonical URL registry` | +| `url_registry.view_canonical_url_registry` | `View Canonical URL registry` | +| `user.about_me` | `About me` | +| `user.about_me_description` | `Create an introduction for visiting your profile (200 characters).` | +| `user.avatar` | `Avatar` | +| `user.bio` | `Short bio` | +| `user.company_name` | `Company Name` | +| `user.copy_api_token` | `Copy API token` | +| `user.copy_token_success` | `Copied token to your clipboard` | +| `user.edit_my_profile` | `Edit my profile` | +| `user.email_address` | `Email Address` | +| `user.first_name` | `First Name` | +| `user.last_login` | `Last Login` | +| `user.last_name` | `Last Name` | +| `user.login_and_security` | `Login and security` | +| `user.my_bookmarks` | `My Bookmarks` | +| `user.my_organizations` | `My Organizations` | +| `user.name_and_description` | `Name and description` | +| `user.password` | `Password` | +| `user.profile_update_failure` | `An error occurred while updating profile.` | +| `user.profile_update_success` | `Profile updated successfully` | +| `user.update_avatar` | `Update your avatar` | +| `user.user` | `User` | +| `user.user_has_not_created_public_repos_suffix` | `has not created any public repositories` | +| `user.users` | `Users` | +| `user.your` | `Your` | diff --git a/src/i18n/__tests__/audit.test.js b/src/i18n/__tests__/audit.test.js new file mode 100644 index 00000000..dd9ccdae --- /dev/null +++ b/src/i18n/__tests__/audit.test.js @@ -0,0 +1,224 @@ +import test from 'node:test' +import assert from 'node:assert/strict' + +import { + auditKeys, + compareLocale, + flattenCatalog, + renderReport, + scanSource +} from '../audit.js' + +const CATALOG = { + common: { save: 'Save', cancel: 'Cancel', id: 'ID', unused: 'Unused' }, + map_project: { + view_all: 'All', + view_mapped: 'Mapped', + recommended: 'Recommended', + available: 'Available', + low_ranked: 'Low Ranked', + never_shown: 'Never shown' + }, + quota: { meter: { rows: 'Rows', projects: 'Projects' } }, + search: { concepts: 'concepts', mappings: 'mappings' } +} + +const keysOf = scan => scan.sites.flatMap(site => site.keys).sort() +const patternsOf = scan => scan.sites.flatMap(site => site.patterns).sort() + +test('flattenCatalog joins nested keys with dots', () => { + assert.deepEqual(flattenCatalog({ a: { b: 'x', c: { d: 'y' } }, e: 'z' }), { 'a.b': 'x', 'a.c.d': 'y', e: 'z' }) +}) + +test('scanSource reads literal keys from t(), i18n.t(), this.props.t() and i18nKey', () => { + const scan = scanSource(` + const A = () => { + const { t } = useTranslation() + return
+ {t('common.save')}{t(\`common.id\`)}{i18n.t('a.b')}{this.props.t('c.d')} + ]} /> +
+ }`, 'A.jsx') + assert.deepEqual(keysOf(scan), ['a.b', 'c.d', 'common.cancel', 'common.id', 'common.save', 'e.f', 'g.h']) + assert.deepEqual(patternsOf(scan), []) +}) + +test('scanSource reads optional calls: i18n?.t(), t?.() and this.props?.t()', () => { + const scan = scanSource("i18n?.t('a.b'); t?.('c.d'); this.props?.t('e.f')", 'H.js') + assert.deepEqual(keysOf(scan), ['a.b', 'c.d', 'e.f']) +}) + +test('scanSource takes every branch of a conditional, a logical fallback and a fallback-key array', () => { + const scan = scanSource(` + t(isCap ? 'quota.cap' : 'quota.quota') + t(custom || 'common.save') + t(['quota.consequence.' + meter, 'quota.consequence.generic'])`, 'B.js') + assert.deepEqual(keysOf(scan), ['common.save', 'quota.cap', 'quota.consequence.generic', 'quota.quota']) + assert.deepEqual(patternsOf(scan), ['quota.consequence.*']) + assert.equal(scan.sites[1].opaque, true, 'custom could be any key') +}) + +test('scanSource turns template literals and concatenation into key patterns', () => { + const scan = scanSource(` + t(\`map_project.preview_limit_body_\${kind}_no_count\`) + t('map_project.view_' + getState(row).toLowerCase()) + t(\`quota.consequence.\${meter}.\${surface}\`)`, 'C.js') + assert.deepEqual(patternsOf(scan), [ + 'map_project.preview_limit_body_*_no_count', + 'map_project.view_*', + 'quota.consequence.*.*' + ]) +}) + +test('scanSource lists a pattern once when several branches produce it', () => { + const scan = scanSource(` + const reason = ready ? getReason() : null + t(\`map_project.preview_limit_title_\${reason}\`)`, 'G.js') + assert.deepEqual(scan.sites[0].patterns, ['map_project.preview_limit_title_*']) +}) + +test('scanSource follows a const to its value, but not a parameter or a destructured name', () => { + const scan = scanSource(` + const key = active ? 'common.save' : 'common.cancel' + t(key) + const Label = ({ labelKey }) => t(labelKey) + const { other } = props + t(other)`, 'D.jsx') + assert.deepEqual(keysOf(scan), ['common.cancel', 'common.save']) + assert.deepEqual(scan.sites.filter(site => site.opaque).map(site => site.expression), ['labelKey', 'other']) +}) + +test('scanSource records a Trans without i18nKey as a site it cannot read', () => { + const scan = scanSource('const A = () => Hello world', 'E.jsx') + assert.equal(scan.sites.length, 1) + assert.equal(scan.sites[0].opaque, true) + assert.equal(scan.sites[0].expression, ' without i18nKey') +}) + +test('scanSource collects string literals so keys stored in config objects count', () => { + const scan = scanSource(` + const COLUMNS = [{ id: 'id', labelKey: 'common.id' }, { labelKey: \`common.save\` }] + const prefixed = \`common.\${x}\``, 'F.js') + const values = scan.literals.map(literal => literal.value) + assert.ok(values.includes('common.id')) + assert.ok(values.includes('common.save')) + assert.ok(!values.includes('common.'), 'a template with a placeholder is not a literal') +}) + +test('auditKeys sorts keys into literal, dynamic, string-literal and unreferenced', () => { + const result = auditKeys({ + sources: [ + { file: 'src/A.jsx', code: "t('common.save'); t('map_project.view_' + state); t(`quota.meter.${meter}`)" }, + { file: 'src/B.js', code: "export const COLUMNS = [{ labelKey: 'common.id' }]; t(col.labelKey)" } + ], + catalogs: { en: CATALOG } + }) + assert.deepEqual(result.keys.static, ['common.save']) + assert.deepEqual(result.keys.dynamic, ['map_project.view_all', 'map_project.view_mapped', 'quota.meter.projects', 'quota.meter.rows']) + assert.deepEqual(result.keys.literal, ['common.id']) + assert.deepEqual(result.keys.unreferenced, [ + 'common.cancel', 'common.unused', + 'map_project.available', 'map_project.low_ranked', 'map_project.never_shown', 'map_project.recommended', + 'search.concepts', 'search.mappings' + ]) + assert.deepEqual(result.opaqueSites.map(site => `${site.file}:${site.line} ${site.expression}`), ['src/B.js:1 col.labelKey']) +}) + +test('auditKeys counts an undeclared namespace-wide family against every key it matches, and flags it', () => { + const result = auditKeys({ + sources: [{ file: 'src/Bucket.jsx', code: "t('map_project.' + id)" }], + catalogs: { en: CATALOG } + }) + const family = result.families[0] + assert.equal(family.broad, true) + assert.equal(family.declared, false) + assert.equal(family.keys.length, 6) + assert.ok(!result.keys.unreferenced.some(key => key.startsWith('map_project.'))) + assert.deepEqual(result.problems.undeclaredBroad.map(f => f.expression), ["'map_project.' + id"]) +}) + +test('auditKeys uses the declared keys of a namespace-wide family', () => { + const result = auditKeys({ + sources: [ + { file: 'src/Bucket.jsx', code: "t('map_project.' + id)\nt('map_project.' + id)" }, + { file: 'src/Search.jsx', code: 't(`search.${resource}`)' } + ], + catalogs: { en: CATALOG }, + families: [ + { file: 'src/Bucket.jsx', expression: "'map_project.' + id", keys: ['map_project.recommended', 'map_project.available', 'map_project.low_ranked'], evidence: 'BucketButton ids' }, + { file: 'src/Search.jsx', expression: '`search.${resource}`', keys: ['search.concepts'], evidence: 'resource is always concepts' } + ] + }) + assert.equal(result.families.length, 3) + assert.ok(result.families.every(family => family.declared)) + assert.deepEqual(result.keys.dynamic, ['map_project.available', 'map_project.low_ranked', 'map_project.recommended', 'search.concepts']) + assert.ok(result.keys.unreferenced.includes('map_project.never_shown')) + assert.ok(result.keys.unreferenced.includes('search.mappings')) + assert.deepEqual(result.problems.undeclaredBroad, []) +}) + +test('auditKeys flags declarations that no longer fit the code or the catalog', () => { + const result = auditKeys({ + sources: [{ file: 'src/Bucket.jsx', code: "t('map_project.' + id)" }], + catalogs: { en: CATALOG }, + families: [ + { file: 'src/Bucket.jsx', expression: "'map_project.' + id", keys: ['map_project.recommended', 'map_project.very_high', 'common.save'] }, + { file: 'src/Gone.jsx', expression: "'search.' + resource", keys: ['search.concepts'] } + ] + }) + assert.deepEqual(result.problems.declaredMissingFromCatalog, ['map_project.very_high']) + assert.deepEqual(result.problems.declaredOutsidePattern, ['common.save']) + assert.deepEqual(result.problems.staleDeclarations.map(family => family.file), ['src/Gone.jsx']) +}) + +test('auditKeys reports literal keys the English catalog lacks', () => { + const result = auditKeys({ + sources: [{ file: 'src/A.jsx', code: "t('common.save'); t('common.not_there')" }], + catalogs: { en: CATALOG } + }) + assert.deepEqual(result.problems.missingFromCatalog, [{ key: 'common.not_there', sites: ['src/A.jsx:1'] }]) +}) + +test('compareLocale separates missing keys, blank values and values identical to English', () => { + const english = { 'a.one': 'One', 'a.two': 'Two', 'a.id': 'ID', 'a.blank': 'Blank' } + const other = { 'a.one': 'Um', 'a.id': 'ID', 'a.blank': ' ', 'a.extra': 'Extra' } + assert.deepEqual(compareLocale(english, other), { + total: 4, + missing: ['a.two'], + blank: ['a.blank'], + sameAsEnglish: ['a.id'], + extra: ['a.extra'] + }) +}) + +test('auditKeys compares every other locale with English', () => { + const result = auditKeys({ + sources: [], + catalogs: { en: { a: { one: 'One', two: 'Two' } }, es: { a: { one: 'Uno' } }, 'pt-BR': { a: { one: 'Um', two: 'Two' } } } + }) + assert.deepEqual(result.locales.map(locale => [locale.locale, locale.missing, locale.sameAsEnglish]), [ + ['es', ['a.two'], []], + ['pt-BR', [], ['a.two']] + ]) +}) + +test('auditKeys orders call sites by code unit, whatever the locale', () => { + const result = auditKeys({ + sources: [{ file: 'src/a.js', code: 't(x)' }, { file: 'src/B.js', code: 't(y)' }], + catalogs: { en: CATALOG } + }) + assert.deepEqual(result.opaqueSites.map(site => site.file), ['src/B.js', 'src/a.js']) +}) + +test('auditKeys and renderReport do not depend on the order files are read in', () => { + const sources = [ + { file: 'src/A.jsx', code: "t('common.save'); t(labelKey)" }, + { file: 'src/B.jsx', code: "t('map_project.view_' + x); t(isCap ? 'common.cancel' : 'common.id')" } + ] + const catalogs = { en: CATALOG, es: { common: { save: 'Guardar' } } } + const forward = renderReport(auditKeys({ sources, catalogs })) + const backward = renderReport(auditKeys({ sources: [...sources].reverse(), catalogs })) + assert.equal(forward, backward) + assert.match(forward, /\| English catalog \| 14 \|/) + assert.match(forward, /`common\.unused`/) +}) diff --git a/src/i18n/__tests__/auditCli.test.js b/src/i18n/__tests__/auditCli.test.js new file mode 100644 index 00000000..d1e729ce --- /dev/null +++ b/src/i18n/__tests__/auditCli.test.js @@ -0,0 +1,30 @@ +/* eslint-env node */ +import test from 'node:test' +import assert from 'node:assert/strict' +import { execFileSync } from 'node:child_process' +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' +import { fileURLToPath } from 'node:url' + +const CLI = fileURLToPath(new URL('../../../scripts/i18n-audit/audit.mjs', import.meta.url)) + +const writeFile = (root, file, content) => { + fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true }) + fs.writeFileSync(path.join(root, file), content) +} + +test('npm run i18n:audit -- --root audits another tree with this checkout\'s families.json', () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'i18n-audit-')) + try { + writeFile(root, 'src/components/map-projects/ScoreBucketButton.jsx', "export const Label = ({ id }) => t('map_project.' + id)\n") + writeFile(root, 'src/i18n/locales/en/translations.json', JSON.stringify({ + map_project: { recommended: 'Recommended', available: 'Available', low_ranked: 'Low Ranked', unused: 'Unused' } + })) + const report = execFileSync(process.execPath, ['--disable-warning=MODULE_TYPELESS_PACKAGE_JSON', CLI, '--root', root], { encoding: 'utf8' }) + assert.match(report, /\| \*\*Unreferenced\*\* \| \*\*1\*\* \|/) + assert.match(report, /\| `map_project\.unused` \| `Unused` \|/) + } finally { + fs.rmSync(root, { recursive: true, force: true }) + } +}) diff --git a/src/i18n/audit.js b/src/i18n/audit.js new file mode 100644 index 00000000..41ec30fb --- /dev/null +++ b/src/i18n/audit.js @@ -0,0 +1,322 @@ +/** + * Static audit of the Mapper's i18n keys (OpenConceptLab/ocl_issues#2880): + * which English keys the source references, and how each locale compares + * with English. scripts/i18n-audit/audit.mjs reads the files and prints + * renderReport(auditKeys(...)); the app never imports this module. + * scripts/i18n-audit/README.md describes the method and its limits. + */ +import babel from '@babel/core' + +// The part of a key the code computes at run time. +const ANY = Symbol('any') + +// A family whose pattern is a whole namespace (`map_project.*`) would count +// every key in it as used, so its keys have to be declared. +const NAMESPACE_WIDE = /^[^.*]+\.\*$/ + +export const flattenCatalog = (catalog, prefix = '') => { + const flat = {} + Object.entries(catalog).forEach(([name, value]) => { + const key = prefix ? `${prefix}.${name}` : name + if(value && typeof value === 'object' && !Array.isArray(value)) + Object.assign(flat, flattenCatalog(value, key)) + else + flat[key] = value + }) + return flat +} + +// Joins neighbouring text and neighbouring ANYs: ['a', '.', ANY, ANY] -> ['a.', ANY] +const tidy = parts => parts.reduce((out, part) => { + const last = out[out.length - 1] + if(part === '' || (part === ANY && last === ANY)) + return out + if(typeof part === 'string' && typeof last === 'string') + return [...out.slice(0, -1), last + part] + return [...out, part] +}, []) + +const product = (lefts, rights) => lefts.flatMap(left => rights.map(right => tidy([...left, ...right]))) + +// Every key an expression can evaluate to, as patterns of text and ANY: +// 'a.b' is ['a.b'], `a.${b}` is ['a.', ANY], cond ? 'x' : 'y' is two +// patterns, and a const is read through to its value. Anything else, such as +// a parameter or a property, is [ANY]. +const patternsOf = (path, resolving = new Set()) => { + if(path.isStringLiteral()) + return [[path.node.value]] + if(path.isTemplateLiteral()) { + const quasis = path.node.quasis + return path.get('expressions').reduce( + (patterns, expression, i) => product(product(patterns, patternsOf(expression, resolving)), [[quasis[i + 1].value.cooked]]), + [tidy([quasis[0].value.cooked])] + ) + } + if(path.isBinaryExpression({ operator: '+' })) + return product(patternsOf(path.get('left'), resolving), patternsOf(path.get('right'), resolving)) + if(path.isConditionalExpression()) + return [...patternsOf(path.get('consequent'), resolving), ...patternsOf(path.get('alternate'), resolving)] + if(path.isLogicalExpression()) + return [...patternsOf(path.get('left'), resolving), ...patternsOf(path.get('right'), resolving)] + if(path.isArrayExpression()) + return path.get('elements').flatMap(element => patternsOf(element, resolving)) + if(path.isIdentifier()) { + const binding = path.scope.getBinding(path.node.name) + const declarator = binding?.path + if(binding?.constant && !resolving.has(binding) && declarator.isVariableDeclarator() && + declarator.get('id').isIdentifier() && declarator.node.init) + return patternsOf(declarator.get('init'), new Set([...resolving, binding])) + } + return [[ANY]] +} + +const isTranslate = callee => + (callee.type === 'Identifier' && callee.name === 't') || + (['MemberExpression', 'OptionalMemberExpression'].includes(callee.type) && !callee.computed && callee.property.name === 't') + +/** + * The keys one file passes to t() (t, i18n.t, this.props.t) or a Trans + * i18nKey, and every string literal in it. + * + * @returns {{sites: object[], literals: {value: string, line: number}[]}} + * one site per call: its literal keys, its patterns ('a.*' for `a.${b}`), + * and opaque when some value of it can't be read (a parameter, a property, + * a Trans without i18nKey). + */ +export const scanSource = (code, file) => { + const ast = babel.parseSync(code, { + filename: file, babelrc: false, configFile: false, sourceType: 'module', parserOpts: { plugins: ['jsx'] } + }) + const sites = [] + const literals = [] + const addSite = path => { + const patterns = patternsOf(path) + const unique = values => [...new Set(values)] + sites.push({ + file, + line: path.node.loc.start.line, + expression: code.slice(path.node.start, path.node.end).replace(/\s+/g, ' '), + keys: unique(patterns.filter(pattern => !pattern.includes(ANY)).map(pattern => pattern.join(''))), + patterns: unique(patterns + .filter(pattern => pattern.includes(ANY) && pattern.some(part => part !== ANY)) + .map(pattern => pattern.map(part => part === ANY ? '*' : part).join(''))), + opaque: patterns.some(pattern => pattern.length === 1 && pattern[0] === ANY) + }) + } + babel.traverse(ast, { + // i18n?.t('key') and t?.('key') are OptionalCallExpressions + 'CallExpression|OptionalCallExpression'(path) { + if(isTranslate(path.node.callee) && path.node.arguments.length) + addSite(path.get('arguments.0')) + }, + JSXAttribute(path) { + if(path.node.name.name !== 'i18nKey' || !path.node.value) + return + const value = path.get('value') + addSite(value.isJSXExpressionContainer() ? value.get('expression') : value) + }, + JSXOpeningElement(path) { + const { name, attributes } = path.node + if(name.name === 'Trans' && !attributes.some(attribute => attribute.name?.name === 'i18nKey')) + sites.push({ file, line: path.node.loc.start.line, expression: ' without i18nKey', keys: [], patterns: [], opaque: true }) + }, + StringLiteral(path) { + literals.push({ value: path.node.value, line: path.node.loc.start.line }) + }, + TemplateLiteral(path) { + if(!path.node.expressions.length) + literals.push({ value: path.node.quasis[0].value.cooked, line: path.node.loc.start.line }) + } + }) + return { sites, literals } +} + +/** + * How a locale's catalog differs from English, both flattened. Missing keys + * fall back to English; blank values show nothing; values identical to + * English may be untranslated or may be right (ID, JSON). + */ +export const compareLocale = (english, other) => { + const isBlank = key => typeof other[key] === 'string' && !other[key].trim() + const keys = Object.keys(english).sort() + return { + total: Object.keys(other).length, + missing: keys.filter(key => !(key in other)), + blank: keys.filter(key => key in other && isBlank(key)), + sameAsEnglish: keys.filter(key => key in other && !isBlank(key) && other[key] === english[key]), + extra: Object.keys(other).filter(key => !(key in english)).sort() + } +} + +const toRegExp = pattern => new RegExp(`^${pattern.split('*').map(text => text.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')).join('.*')}$`) + +/** + * @param {object} options + * @param {{file: string, code: string}[]} options.sources the files to scan + * @param {object} options.catalogs locale -> catalog, including en + * @param {{file: string, expression: string, keys: string[], evidence: string}[]} [options.families] + * the keys a dynamic call site can produce, checked by hand: scripts/i18n-audit/families.json + * @returns {object} each English key's evidence of use, the dynamic families + * and opaque sites behind it, problems to fix, and each locale against English + */ +export const auditKeys = ({ sources, catalogs, families: declarations = [] }) => { + const english = flattenCatalog(catalogs.en) + const keys = Object.keys(english).sort() + // Files in code-unit order, as Array#sort uses, so the report doesn't change + // with the locale; each file's sites and literals are in source order. + const scanned = [...sources] + .sort((a, b) => (a.file < b.file ? -1 : a.file > b.file ? 1 : 0)) + .map(({ file, code }) => ({ file, ...scanSource(code, file) })) + const sites = scanned.flatMap(scan => scan.sites) + + const staticSites = {} + sites.forEach(site => site.keys.forEach(key => (staticSites[key] = staticSites[key] || []).push(`${site.file}:${site.line}`))) + + const used = new Set() + const declaredOutsidePattern = new Set() + const declaredMissingFromCatalog = new Set() + const families = sites.filter(site => site.patterns.length).flatMap(site => { + const declaration = declarations.find(family => family.file === site.file && family.expression === site.expression) + const regExps = site.patterns.map(toRegExp) + declaration?.keys.forEach(key => { + if(!regExps.some(regExp => regExp.test(key))) + declaredOutsidePattern.add(key) + else if(!(key in english)) + declaredMissingFromCatalog.add(key) + }) + return site.patterns.map((pattern, i) => { + const familyKeys = (declaration ? declaration.keys.filter(key => key in english) : keys).filter(key => regExps[i].test(key)) + familyKeys.forEach(key => used.add(key)) + return { + file: site.file, line: site.line, expression: site.expression, pattern, keys: familyKeys, + broad: NAMESPACE_WIDE.test(pattern), declared: Boolean(declaration) + } + }) + }) + + const literalSites = {} + scanned.forEach(scan => scan.literals.forEach(({ value, line }) => { + if(value in english && !(value in staticSites) && !used.has(value)) + (literalSites[value] = literalSites[value] || []).push(`${scan.file}:${line}`) + })) + + return { + files: sources.length, + english, + keys: { + static: keys.filter(key => key in staticSites), + dynamic: keys.filter(key => !(key in staticSites) && used.has(key)), + literal: keys.filter(key => key in literalSites), + unreferenced: keys.filter(key => !(key in staticSites) && !used.has(key) && !(key in literalSites)) + }, + staticSiteCount: sites.filter(site => site.keys.length).length, + families, + opaqueSites: sites.filter(site => site.opaque), + literalSites, + problems: { + missingFromCatalog: Object.keys(staticSites).filter(key => !(key in english)).sort() + .map(key => ({ key, sites: staticSites[key] })), + undeclaredBroad: families.filter(family => family.broad && !family.declared), + declaredOutsidePattern: [...declaredOutsidePattern].sort(), + declaredMissingFromCatalog: [...declaredMissingFromCatalog].sort(), + staleDeclarations: declarations.filter(family => !sites.some(site => site.patterns.length && site.file === family.file && site.expression === family.expression)) + }, + locales: Object.keys(catalogs).filter(locale => locale !== 'en').sort() + .map(locale => ({ locale, ...compareLocale(english, flattenCatalog(catalogs[locale])) })) + } +} + +// A Markdown code span that survives backticks and table pipes in the text. +const code = text => { + const flat = String(text).replace(/\s*\n\s*/g, ' ') + const fence = '`'.repeat(Math.max(0, ...(flat.match(/`+/g) || []).map(run => run.length)) + 1) + const pad = flat.startsWith('`') || flat.endsWith('`') ? ' ' : '' + return `${fence}${pad}${flat}${pad}${fence}`.replace(/\|/g, '\\|') +} + +const keyList = keys => keys.length ? keys.map(code).join(', ') : 'none' + +export const renderReport = result => { + const { keys, english, families, opaqueSites, literalSites, problems, locales } = result + const problemLines = [ + ...problems.missingFromCatalog.map(({ key, sites }) => `- ${code(key)} is referenced (${sites.join(', ')}) but isn't in the English catalog.`), + ...problems.undeclaredBroad.map(family => `- ${family.file}:${family.line} ${code(family.expression)} spans the whole namespace; declare its keys in \`scripts/i18n-audit/families.json\`. Until then all ${family.keys.length} keys it matches count as used.`), + ...problems.declaredOutsidePattern.map(key => `- ${code(key)} is declared in \`families.json\` but its call site's expression can't produce it.`), + ...problems.declaredMissingFromCatalog.map(key => `- ${code(key)} is declared in \`families.json\` but isn't in the English catalog.`), + ...problems.staleDeclarations.map(family => `- \`families.json\` declares ${family.file} ${code(family.expression)}, which no longer exists.`) + ] + return [ + '# i18n key audit', + '', + 'Generated by `npm run i18n:audit` from the source under `src/` (tests excluded) and the', + 'catalogs in `src/i18n/locales/`. Rerun it rather than edit this file. The method and its', + 'limits are in `scripts/i18n-audit/README.md`.', + '', + '## Summary', + '', + '| English keys | Count |', + '|---|---:|', + `| English catalog | ${Object.keys(english).length} |`, + `| Referenced by a literal key | ${keys.static.length} |`, + `| Matched only by a dynamic key family | ${keys.dynamic.length} |`, + `| Mentioned only as a string literal | ${keys.literal.length} |`, + `| **Unreferenced** | **${keys.unreferenced.length}** |`, + '', + `Scanned ${result.files} source files: ${result.staticSiteCount} call sites with a literal key, ` + + `${families.length} dynamic key families, ${opaqueSites.length} call sites whose key the audit can't read.`, + '', + '## Needs attention', + '', + ...(problemLines.length ? problemLines : ['Nothing.']), + '', + '## Locales compared with English', + '', + '| Locale | Keys | Missing | Blank | Same as English | Not in English |', + '|---|---:|---:|---:|---:|---:|', + ...locales.map(l => `| ${l.locale} | ${l.total} | ${l.missing.length} | ${l.blank.length} | ${l.sameAsEnglish.length} | ${l.extra.length} |`), + '', + ...locales.flatMap(l => [ + `### ${l.locale}`, + '', + `- Missing: ${keyList(l.missing)}`, + `- Blank: ${keyList(l.blank)}`, + `- Same as English: ${l.sameAsEnglish.length ? l.sameAsEnglish.map(key => `${code(key)} (${code(english[key])})`).join(', ') : 'none'}`, + `- Not in English: ${keyList(l.extra)}`, + '' + ]), + '## Dynamic key families', + '', + 'A key built at run time, as a pattern. A declared family counts only the keys', + '`scripts/i18n-audit/families.json` lists for it; any other counts every key its pattern matches.', + '', + '| Call site | Key expression | Pattern | Keys |', + '|---|---|---|---|', + ...families.map(family => `| ${family.file}:${family.line} | ${code(family.expression)} | ${code(family.pattern)} | ` + + `${family.declared ? 'declared: ' : ''}${family.keys.length > 12 ? `${family.keys.length} keys` : keyList(family.keys)} |`), + '', + '## Call sites whose key the audit can\'t read', + '', + 'The key comes from a parameter or a property. Such keys are usually string literals', + 'elsewhere, which the next section counts.', + '', + '| Call site | Key expression |', + '|---|---|', + ...opaqueSites.map(site => `| ${site.file}:${site.line} | ${code(site.expression)} |`), + '', + '## Mentioned only as a string literal', + '', + '| Key | Where |', + '|---|---|', + ...keys.literal.map(key => `| ${code(key)} | ${literalSites[key].join(', ')} |`), + '', + `## Unreferenced keys (${keys.unreferenced.length})`, + '', + 'No call site, dynamic family or string literal in the source can produce these. Check', + 'each by hand before removing it (see the README).', + '', + '| Key | English |', + '|---|---|', + ...keys.unreferenced.map(key => `| ${code(key)} | ${code(english[key])} |`), + '' + ].join('\n') +}