feat: import Duala and Bassa vocabulary via NTeALan API - #26
Merged
Conversation
Adds a dedicated NTeALanClient/NTeALanImportService pair (cahier des charges §5.1) that fetches, normalizes, and imports dictionary articles from the NTeALan collaborative API into content-service's word repository. Key design points: - Manual JsonNode parsing instead of strict POJO binding: the API's repeatable fields (translations.equivalent, examples.example) inconsistently serialize as either an array or a bare object depending on cardinality, an XML-to-JSON conversion artifact confirmed against real API responses for both dictionaries. - Idempotent: re-running an import skips words already present (case-insensitive match), safe to re-trigger. - Per-word failure isolation: one bad entry doesn't abort the batch. - Separate @qualifier'd RestTemplate beans (loadBalancedRestTemplate vs externalRestTemplate) so calls to the real NTeALan domain don't get routed through Eureka's service resolution, which only applies to this project's own registered services. - New POST /languages/{languageId}/import-ntealan admin endpoint, triggered manually once per language. Tests use real captured API responses as fixtures (both the array and bare-object equivalent cases) rather than synthetic data. Verified manually end to end: Duala (dl_fr_2018) imported 192/192 words, Bassa (bs_fr_2019) imported 727/728 words (one entry filtered for missing translation, by design), zero failures, zero duplicates on re-run.
📝 WalkthroughWalkthroughThis PR adds NTeALan dictionary import functionality to the content-service. It introduces ChangesNTeALan Import Feature
Estimated code review effort: 3 (Moderate) | ~30 minutes Sequence Diagram(s)sequenceDiagram
participant LanguageController
participant NTeALanImportService
participant LanguageService
participant NTeALanClient
participant WordService
LanguageController->>NTeALanImportService: importDictionary(languageId, dictionaryId)
NTeALanImportService->>LanguageService: getEntityById(languageId)
loop pages until empty or MAX_PAGES
NTeALanImportService->>NTeALanClient: fetchArticles(dictionaryId, page, limit)
NTeALanClient-->>NTeALanImportService: List<NTeALanWordEntry>
NTeALanImportService->>WordService: wordExists(languageId, word)
alt word not existing
NTeALanImportService->>WordService: create(CreateWordRequest)
else word exists
NTeALanImportService->>NTeALanImportService: increment skipped
end
end
NTeALanImportService-->>LanguageController: ImportResult(imported, skipped, failed)
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
services/content-service/src/test/java/cm/afrilingua/content/service/NTeALanImportServiceTest.java (1)
127-138: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a test for the MAX_PAGES safety net.
No test verifies that the loop stops at
MAX_PAGES(50) when the API never returns an empty page — an important safety behavior that is currently untested.✅ Suggested test
`@Test` void stopsAtMaxPagesWhenApiNeverReturnsEmptyPage() { NTeALanWordEntry entry = new NTeALanWordEntry("aba", "partager", "verb"); when(nteALanClient.fetchArticles(eq("dl_fr_2018"), anyInt(), anyInt())) .thenReturn(List.of(entry)); when(wordService.wordExists(eq(languageId), anyString())).thenReturn(false); when(wordService.create(eq(languageId), any(CreateWordRequest.class))) .thenReturn(mock(cm.afrilingua.content.dto.Word.class)); NTeALanImportService.ImportResult result = importService.importDictionary(languageId, "dl_fr_2018"); assertThat(result.imported()).isEqualTo(50); verify(nteALanClient, times(50)).fetchArticles(eq("dl_fr_2018"), anyInt(), anyInt()); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@services/content-service/src/test/java/cm/afrilingua/content/service/NTeALanImportServiceTest.java` around lines 127 - 138, Add a test in NTeALanImportServiceTest to cover the MAX_PAGES safety net in NTeALanImportService.importDictionary: mock nteALanClient.fetchArticles so it always returns a non-empty page, then assert the import stops after 50 pages and the ImportResult imported count matches that limit. Use the existing importService, nteALanClient, and wordService setup, and verify fetchArticles is invoked exactly 50 times to prove the loop terminates at MAX_PAGES.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@services/content-service/src/main/java/cm/afrilingua/content/client/NTeALanClient.java`:
- Around line 39-70: The fetchArticles method in NTeALanClient is swallowing
request/parse failures by returning List.of(), which makes NTeALanImportService
treat transient errors as end-of-pagination. Update fetchArticles to let
exceptions propagate (or return a failure-aware result) instead of converting
them to an empty list, and keep the empty-list behavior only for the genuine “no
articles” case after a successful response.
In
`@services/content-service/src/main/java/cm/afrilingua/content/config/RestTemplateConfig.java`:
- Around line 22-24: The externalRestTemplate() bean currently creates a plain
RestTemplate with no network safeguards, which can let NTeALan calls hang
indefinitely; update RestTemplateConfig.externalRestTemplate to configure
connect and read timeouts on the underlying request factory before returning the
bean. Keep the fix localized to externalRestTemplate and ensure the timeouts are
appropriate for calls to external services.
In
`@services/content-service/src/main/java/cm/afrilingua/content/service/NTeALanImportService.java`:
- Around line 42-46: The import loop in NTeALanImportService#importDictionary
silently exits when the MAX_PAGES safety limit is hit, which can make a partial
import look complete. Update the logic in importDictionary to detect when the
loop ends because page reached MAX_PAGES rather than entries.isEmpty(), and emit
a warning through the existing logger in that case. Keep the current completion
log, but make sure it clearly distinguishes a full import from a truncated one
so large dictionaries are not misreported as fully imported.
- Around line 48-66: Add a database-level uniqueness guard for words because
WordService’s existence check can race with concurrent writes. Update the words
schema in the migration to enforce uniqueness per language using a
case-insensitive key on word, then adjust WordService.create and the
NTeALanImportService import flow to catch that duplicate-key path and count it
as skipped instead of failed, using the existing WordService and
NTeALanImportService symbols to locate the change.
---
Nitpick comments:
In
`@services/content-service/src/test/java/cm/afrilingua/content/service/NTeALanImportServiceTest.java`:
- Around line 127-138: Add a test in NTeALanImportServiceTest to cover the
MAX_PAGES safety net in NTeALanImportService.importDictionary: mock
nteALanClient.fetchArticles so it always returns a non-empty page, then assert
the import stops after 50 pages and the ImportResult imported count matches that
limit. Use the existing importService, nteALanClient, and wordService setup, and
verify fetchArticles is invoked exactly 50 times to prove the loop terminates at
MAX_PAGES.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: ff41194f-1d8b-4f59-b808-0cecaff09f43
📒 Files selected for processing (15)
services/content-service/src/main/java/cm/afrilingua/content/client/NTeALanClient.javaservices/content-service/src/main/java/cm/afrilingua/content/client/NTeALanWordEntry.javaservices/content-service/src/main/java/cm/afrilingua/content/client/RecommendationClient.javaservices/content-service/src/main/java/cm/afrilingua/content/config/RestTemplateConfig.javaservices/content-service/src/main/java/cm/afrilingua/content/controller/LanguageController.javaservices/content-service/src/main/java/cm/afrilingua/content/repository/WordRepository.javaservices/content-service/src/main/java/cm/afrilingua/content/service/NTeALanImportService.javaservices/content-service/src/main/java/cm/afrilingua/content/service/WordService.javaservices/content-service/src/main/resources/openapi/content-service.yamlservices/content-service/src/test/java/cm/afrilingua/content/client/NTeALanClientTest.javaservices/content-service/src/test/java/cm/afrilingua/content/service/NTeALanImportServiceTest.javaservices/content-service/src/test/resources/ntealan/bassa_single_equivalent.jsonservices/content-service/src/test/resources/ntealan/duala_array_equivalent.jsonservices/content-service/src/test/resources/ntealan/empty_page.jsonservices/content-service/src/test/resources/ntealan/missing_translation.json
Comment on lines
+39
to
+70
| public List<NTeALanWordEntry> fetchArticles(String dictionaryId, int page, int limit) { | ||
| String url = String.format("%s/%s?limit=%d&page=%d&sort=ASC", BASE_URL, dictionaryId, limit, page); | ||
|
|
||
| String rawResponse; | ||
| try { | ||
| rawResponse = restTemplate.getForObject(url, String.class); | ||
| } catch (Exception ex) { | ||
| log.error("Failed to fetch NTeALan dictionary {} page {}", dictionaryId, page, ex); | ||
| return List.of(); | ||
| } | ||
|
|
||
| if (rawResponse == null || rawResponse.isBlank()) { | ||
| return List.of(); | ||
| } | ||
|
|
||
| try { | ||
| JsonNode root = objectMapper.readTree(rawResponse); | ||
| JsonNode articles = root.path("articles"); | ||
|
|
||
| List<NTeALanWordEntry> entries = new ArrayList<>(); | ||
| for (JsonNode articleWrapper : articles) { | ||
| NTeALanWordEntry entry = parseEntry(articleWrapper); | ||
| if (entry != null) { | ||
| entries.add(entry); | ||
| } | ||
| } | ||
| return entries; | ||
| } catch (Exception ex) { | ||
| log.error("Failed to parse NTeALan response for dictionary {} page {}", dictionaryId, page, ex); | ||
| return List.of(); | ||
| } | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify how NTeALanImportService uses the empty list returned by fetchArticles —
# specifically whether it treats empty as "stop pagination" with no retry on error.
rg -n --type=java -C5 'fetchArticles' services/content-service/src/main/java/cm/afrilingua/content/service/NTeALanImportService.java 2>/dev/null || echo "Import service not found at expected path"
fd NTeALanImportService.java services/content-service/src/main/javaRepository: ASSONDJI/afrilingua
Length of output: 650
Propagate fetch failures instead of returning an empty page. NTeALanImportService breaks pagination on entries.isEmpty(), so swallowing fetch exceptions here turns a transient network error into a silent, truncated import. Let the exception bubble up or return a result that distinguishes “no more data” from “request failed.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@services/content-service/src/main/java/cm/afrilingua/content/client/NTeALanClient.java`
around lines 39 - 70, The fetchArticles method in NTeALanClient is swallowing
request/parse failures by returning List.of(), which makes NTeALanImportService
treat transient errors as end-of-pagination. Update fetchArticles to let
exceptions propagate (or return a failure-aware result) instead of converting
them to an empty list, and keep the empty-list behavior only for the genuine “no
articles” case after a successful response.
Comment on lines
+22
to
+24
| public RestTemplate externalRestTemplate() { | ||
| return new RestTemplate(); | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Add connect and read timeouts to externalRestTemplate.
A plain new RestTemplate() uses SimpleClientHttpRequestFactory with no connect or read timeout, meaning calls to the NTeALan API can block indefinitely if the external service is slow or unresponsive. Since this bean is specifically for external domains outside your control, this is a real availability risk for the import endpoint.
⏱️ Proposed fix
`@Bean`
public RestTemplate externalRestTemplate() {
- return new RestTemplate();
+ var factory = new SimpleClientHttpRequestFactory();
+ factory.setConnectTimeout(5_000);
+ factory.setReadTimeout(15_000);
+ return new RestTemplate(factory);
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| public RestTemplate externalRestTemplate() { | |
| return new RestTemplate(); | |
| } | |
| public RestTemplate externalRestTemplate() { | |
| var factory = new SimpleClientHttpRequestFactory(); | |
| factory.setConnectTimeout(5_000); | |
| factory.setReadTimeout(15_000); | |
| return new RestTemplate(factory); | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@services/content-service/src/main/java/cm/afrilingua/content/config/RestTemplateConfig.java`
around lines 22 - 24, The externalRestTemplate() bean currently creates a plain
RestTemplate with no network safeguards, which can let NTeALan calls hang
indefinitely; update RestTemplateConfig.externalRestTemplate to configure
connect and read timeouts on the underlying request factory before returning the
bean. Keep the fix localized to externalRestTemplate and ensure the timeouts are
appropriate for calls to external services.
Comment on lines
+42
to
+46
| for (int page = 1; page <= MAX_PAGES; page++) { | ||
| List<NTeALanWordEntry> entries = nteALanClient.fetchArticles(dictionaryId, page, PAGE_SIZE); | ||
| if (entries.isEmpty()) { | ||
| break; | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Log a warning when the MAX_PAGES safety limit is reached.
If a dictionary has more than 5000 entries (50 × 100), the loop silently stops. The completion log at line 69 would show what looks like a successful full import, masking unimported entries.
🛡️ Proposed fix
for (int page = 1; page <= MAX_PAGES; page++) {
List<NTeALanWordEntry> entries = nteALanClient.fetchArticles(dictionaryId, page, PAGE_SIZE);
if (entries.isEmpty()) {
break;
}
+ if (page == MAX_PAGES) {
+ log.warn("Reached MAX_PAGES ({}) for dictionary {} -- there may be unimported entries remaining", MAX_PAGES, dictionaryId);
+ }
+
for (NTeALanWordEntry entry : entries) {📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| for (int page = 1; page <= MAX_PAGES; page++) { | |
| List<NTeALanWordEntry> entries = nteALanClient.fetchArticles(dictionaryId, page, PAGE_SIZE); | |
| if (entries.isEmpty()) { | |
| break; | |
| } | |
| for (int page = 1; page <= MAX_PAGES; page++) { | |
| List<NTeALanWordEntry> entries = nteALanClient.fetchArticles(dictionaryId, page, PAGE_SIZE); | |
| if (entries.isEmpty()) { | |
| break; | |
| } | |
| if (page == MAX_PAGES) { | |
| log.warn("Reached MAX_PAGES ({}) for dictionary {} -- there may be unimported entries remaining", MAX_PAGES, dictionaryId); | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@services/content-service/src/main/java/cm/afrilingua/content/service/NTeALanImportService.java`
around lines 42 - 46, The import loop in NTeALanImportService#importDictionary
silently exits when the MAX_PAGES safety limit is hit, which can make a partial
import look complete. Update the logic in importDictionary to detect when the
loop ends because page reached MAX_PAGES rather than entries.isEmpty(), and emit
a warning through the existing logger in that case. Keep the current completion
log, but make sure it clearly distinguishes a full import from a truncated one
so large dictionaries are not misreported as fully imported.
Comment on lines
+48
to
+66
| for (NTeALanWordEntry entry : entries) { | ||
| try { | ||
| if (wordService.wordExists(languageId, entry.word())) { | ||
| skipped++; | ||
| continue; | ||
| } | ||
|
|
||
| CreateWordRequest request = new CreateWordRequest() | ||
| .word(entry.word()) | ||
| .translation(entry.translation()) | ||
| .grammaticalCategory(entry.grammaticalCategory()); | ||
|
|
||
| wordService.create(languageId, request); | ||
| imported++; | ||
| } catch (Exception ex) { | ||
| log.warn("Failed to import NTeALan word '{}' for dictionary {}", entry.word(), dictionaryId, ex); | ||
| failed++; | ||
| } | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Search for unique constraints on the Word entity
rg -n -C3 'uniqueConstraints|UniqueConstraint|`@Table`' --type java \
-g '**/Word.java' \
services/content-service/src/main/java/cm/afrilingua/content/entity/
# Also check for Flyway/Liquibase migration files referencing the word table
fd -e sql . services/content-service/src/main/resources/ | head -20
rg -n -i 'unique.*word|word.*unique' services/content-service/src/main/resources/ --type sqlRepository: ASSONDJI/afrilingua
Length of output: 1003
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== Word entity ==\n'
cat -n services/content-service/src/main/java/cm/afrilingua/content/entity/Word.java
printf '\n== V1 migration ==\n'
cat -n services/content-service/src/main/resources/db/migration/V1__create_languages_and_words_tables.sql
printf '\n== V2 migration ==\n'
cat -n services/content-service/src/main/resources/db/migration/V2__add_tone_fields_to_words.sql
printf '\n== Search for any word-table unique indexes/constraints ==\n'
rg -n -i 'unique|index|constraint' services/content-service/src/main/resources/db/migration/ services/content-service/src/main/java/cm/afrilingua/content/entity/Word.javaRepository: ASSONDJI/afrilingua
Length of output: 3971
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== WordService ==\n'
cat -n services/content-service/src/main/java/cm/afrilingua/content/service/WordService.java
printf '\n== Word repository matches ==\n'
rg -n -C3 'wordExists|IgnoreCase|findBy.*Word|existsBy.*Word|lower\\(|LOWER\\(' services/content-service/src/main/java/cm/afrilingua/content/
printf '\n== Related repository files ==\n'
fd -e java . services/content-service/src/main/java/cm/afrilingua/content/ | rg 'Repository|Service|Word'Repository: ASSONDJI/afrilingua
Length of output: 5675
Add a database unique constraint for words
services/content-service/src/main/java/cm/afrilingua/content/service/WordService.java checks existence case-insensitively, but words has no unique constraint in services/content-service/src/main/resources/db/migration/V1__create_languages_and_words_tables.sql. A concurrent import or create can still slip in a duplicate between the check and save(). Add a unique constraint/index on (language_id, lower(word)) and treat that violation as a skip.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@services/content-service/src/main/java/cm/afrilingua/content/service/NTeALanImportService.java`
around lines 48 - 66, Add a database-level uniqueness guard for words because
WordService’s existence check can race with concurrent writes. Update the words
schema in the migration to enforce uniqueness per language using a
case-insensitive key on word, then adjust WordService.create and the
NTeALanImportService import flow to catch that duplicate-key path and count it
as skipped instead of failed, using the existing WordService and
NTeALanImportService symbols to locate the change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a dedicated NTeALanClient/NTeALanImportService pair (cahier des charges §5.1) that fetches, normalizes, and imports dictionary articles from the NTeALan collaborative API into content-service's word repository.
Key design points:
Tests use real captured API responses as fixtures (both the array and bare-object equivalent cases) rather than synthetic data.
Verified manually end to end: Duala (dl_fr_2018) imported 192/192 words, Bassa (bs_fr_2019) imported 727/728 words (one entry filtered for missing translation, by design), zero failures, zero duplicates on re-run.
Summary by CodeRabbit
New Features
Bug Fixes