Skip to content

Bug: Get article version file list using article version files endpoint #136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
HafeezOJ merged 13 commits into from
Jun 23, 2026
Merged

Bug: Get article version file list using article version files endpoint #136

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
dd99953
Get article version files with version files endpoint
HafeezOJ Jun 16, 2026
e7807d9
Remove dead code
HafeezOJ Jun 17, 2026
5edd736
Handle potential error in the exception
HafeezOJ Jun 17, 2026
70a8e95
Move page initialization out of loop
HafeezOJ Jun 17, 2026
cb80a63
Move page size out of loop
HafeezOJ Jun 17, 2026
48a0265
Retrieve full file list or nothing
HafeezOJ Jun 18, 2026
d9b2f75
Update counts
HafeezOJ Jun 18, 2026
3671368
Remove dead code
HafeezOJ Jun 18, 2026
9fc55df
Update comment and lint
HafeezOJ Jun 19, 2026
d3e6fcf
Update skipped versions dict
HafeezOJ Jun 22, 2026
7249560
Move counting failed items to end of processing
HafeezOJ Jun 22, 2026
fd97753
Match counting variable names with usage
HafeezOJ Jun 22, 2026
19793af
Update logs summary
HafeezOJ Jun 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions ReBACH_Logs_Summary_Description.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,17 @@

| Log Message | Description |
|-------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Total matched unique articles | Total number of articles whose article_ids are found both in ReDATA and curation storage. This excludes already preserved articles. <br/>**Note**: A preserved article can be in either of the archival storages. |
| Total matched unique articles | Total number of articles whose article_ids are found both in ReDATA and curation storage during preprocessing. This excludes already preserved articles. <br/>**Note**: A preserved article can be in either of the archival storages. |
| Total unmatched unique articles | Total number of articles whose article_ids are found in ReDATA but **not** in curation storage. This excludes already preserved articles. |
| Total matched article versions | Total number of article versions whose article_ids are found in ReDATA and curation storage contains a folder of the version. This excludes already preserved versions. |
| Total unmatched article versions | Total number of article versions whose article_ids are found in ReDATA but curation storage **does not** contain a folder of the version. This excludes already preserved versions. This number should always be zero. If non-zero, check the logs. |
| Total skipped unique articles | Total number of articles that are already preserved and are not processed. These articles have at least one version in archival staging storage or archival storage . |
| Total skipped article versions | Total number of article versions that are already preserved and are not processed. |
| Total skipped unique articles | Total number of articles that are found already preserved during preprocessing and are not processed. These articles have at least one version in archival staging storage or archival storage . |
| Total skipped article versions | Total number of article versions that are found already preserved during preprocessing and are not processed. |
| Total articles | Total number of articles in ReDATA, both published and unpublished. |
| Total published articles/article versions | Total number of published articles against total number of published versions in ReDATA. |
| Total count of already preserved (skipped) articles / article versions | Total count of already preserved articles against total count of already preserved article versions. <br/>**Note:** These figures include all articles and article versions already preserved in either of the remote archival storages and are skipped during processing. |
| Total count of articles with fetch error / articles | Total number of articles with error while fetching items either from ReDATA or curation storage against total number of published articles. <br/>**Note**: Articles with fetch error are skipped during processing. |
| Total count of article versions with fetch error / article versions | Total number of article versions with error while fetching items either from ReDATA or curation storage against total number of published article versions. <br/>**Note**: Article versions with fetch error are skipped during processing. |
| Total count of articles with fetch error / articles | Total number of articles with error while fetching items either from ReDATA (either API errors or file download errors) or curation storage against total number of published articles. <br/>**Note**: Articles with fetch error are skipped during processing. |
| Total count of article versions with fetch error / article versions | Total number of article versions with error while fetching items either from ReDATA (either API errors or file download errors) or curation storage against total number of published article versions. <br/>**Note**: Article versions with fetch error are skipped during processing. |
| Total articles versions matched/published (unskipped) | Total number of article versions with folders in curation storage against total number of published article versions in ReDATA. <br/>**Note**: The two numbers should be equal if nothing went wrong. If the first number is less than the second, it means there was an issue matching at least one of the published article versions that are published in ReDATA. The logs should be examined |
| Total articles versions processed/matched | Total number of article versions successfully processed against total number of article versions in ReDATA with a folder in curation storage. <br/>**Note**: The first number will always be less than or equal to the second. If the numbers are not equal, the log should be checked. |
| Total count of already preserved article versions in archival storage | Total number of already preserved article versions in archival storage. <br/>**Note:** These article versions are skipped during processing. |
Expand Down
35 changes: 18 additions & 17 deletions app.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,14 +154,14 @@ def main():
log.write_log_in_file('info', " ", True)
log.write_log_in_file('info', "------- Fetching articles -------", True)
article_obj = Article(config, log, args.ids)
article_data, already_preserved_counts_dict = article_obj.get_articles()
article_data, skipped_items_counts_dict = article_obj.get_articles()

already_preserved_articles_count = len(already_preserved_counts_dict['already_preserved_article_ids'])
already_preserved_versions_count = already_preserved_counts_dict['already_preserved_versions']
locally_preserved_article_version_count = already_preserved_counts_dict['articles_locally_preserved']
already_preserved_articles_count = len(skipped_items_counts_dict['already_preserved_article_ids'])
already_preserved_versions_count = skipped_items_counts_dict['already_preserved_versions']
locally_preserved_article_version_count = skipped_items_counts_dict['articles_locally_preserved']

articles_with_error_count = len(already_preserved_counts_dict['articles_with_error'])
article_versions_with_error_count = already_preserved_counts_dict['article_versions_with_error']
articles_with_fetch_error_count = len(skipped_items_counts_dict['articles_with_fetch_error'])
article_versions_with_fetch_error_count = skipped_items_counts_dict['article_versions_with_fetch_error']
published_articles_count = 0
published_articles_versions_count = 0
published_unpublished_count = 0
Expand All @@ -174,9 +174,9 @@ def main():

log.write_log_in_file('info', "Fetched: "
+ f"Total articles: {published_unpublished_count}, "
+ f"Published articles: {published_articles_count + already_preserved_articles_count + articles_with_error_count}, "
+ f"Published articles: {published_articles_count + already_preserved_articles_count + articles_with_fetch_error_count}, "
+ "Published article versions: "
+ f"{published_articles_versions_count + already_preserved_versions_count + article_versions_with_error_count}",
+ f"{published_articles_versions_count + already_preserved_versions_count + article_versions_with_fetch_error_count}",
True)
print(" ")

Expand All @@ -195,8 +195,8 @@ def main():
print(" ")

# Start articles processing after completing fetching data from API
processed_articles_versions_count, ap_trust_preserved_article_version_count, wasabi_preserved_versions \
= article_obj.process_articles(article_data)
processed_articles_versions_count, ap_trust_preserved_article_version_count, wasabi_preserved_versions, articles_with_processing_error, \
articles_versions_with_processing_error = article_obj.process_articles(article_data)

# Start collections processing after completing fetching data from API and articles processing.
processed_collections_versions_count, already_preserved_collections_counts = collection_obj.process_collections(collection_data)
Expand All @@ -214,25 +214,26 @@ def main():

log.write_log_in_file('info',
"Total published articles/article versions: \t\t\t\t\t"
+ f'{published_articles_count + already_preserved_articles_count + articles_with_error_count} / '
+ f'{published_articles_versions_count + already_preserved_versions_count + article_versions_with_error_count}',
+ f'{published_articles_count + already_preserved_articles_count + articles_with_fetch_error_count} / '
+ f'{published_articles_versions_count + already_preserved_versions_count + article_versions_with_fetch_error_count}',
True)

log.write_log_in_file('info',
"Total count of already preserved (skipped) articles / article versions: \t\t"
+ f'{already_preserved_articles_count} / {already_preserved_versions_count}',
+ f'{already_preserved_articles_count} / '
+ f'{published_articles_versions_count + already_preserved_versions_count + article_versions_with_fetch_error_count}',
True)

log.write_log_in_file('info',
"Total count of articles with fetch error / articles: \t\t\t\t"
+ f'{articles_with_error_count} / '
+ f'{published_articles_count + already_preserved_articles_count + articles_with_error_count}',
+ f'{articles_with_fetch_error_count + articles_with_processing_error} / '
+ f'{published_articles_count + already_preserved_articles_count + articles_with_fetch_error_count}',
True)

log.write_log_in_file('info',
"Total count of article versions with fetch error / article versions: \t\t"
+ f'{article_versions_with_error_count} / '
+ f'{published_articles_versions_count + already_preserved_versions_count + article_versions_with_error_count}',
+ f'{article_versions_with_fetch_error_count + articles_versions_with_processing_error} / '
+ f'{published_articles_versions_count + already_preserved_versions_count + article_versions_with_fetch_error_count}',
True)

if article_obj.processor.duplicate_bag_in_preservation_storage_count > 0:
Expand Down
Loading
Loading