Add functionality to review web pages by forsyth2 · Pull Request #4 · E3SM-Project/e3sm-comms

forsyth2 · 2026-05-06T17:50:13Z

Add functionality to review web pages for inclusion/exclusion on the public-facing website.

Copilot

Pull request overview

Adds new tooling and reporting to support reviewing WordPress-exported content
and Confluence pages for sensitive terms and publish/archive decisions for the
public-facing e3sm.org site.

Changes:

Add new e3sm-comms-exported-xml-reviewer CLI that parses WordPress export XML
and generates multiple review reports (sensitive terms, navigation issues, and
internal link validity).
Expand e3sm-comms-e3sm-org-reviewer to use WordPress XML exports plus
Confluence-derived inputs to generate Markdown path/terms/action-item reports.
Improve website reviewer output by recording Confluence page created dates in
the sensitive-terms output and making Confluence traversal more fault-tolerant.

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
README.md	Updates command documentation and inputs/outputs for the new workflows.
pyproject.toml	Adds a new console script entry point for the exported XML reviewer.
examples/review_xml.bash	Adds a runnable example workflow for exported XML review.
examples/review_terms.bash	Adds a runnable example workflow for terms/path review.
e3sm_comms/website_reviewer/main.py	Adjusts default Confluence top-level input selection.
e3sm_comms/page_reviewer/utils_website_reviewer.py	Adds created-date context to sensitive-term output (but currently contains a syntax error).
e3sm_comms/page_reviewer/utils_base.py	Adds `created_date` on pages and introduces `get_e3sm_url_status`.
e3sm_comms/page_reviewer/confluence_page_reviewer.py	Improves resilience and logging during Confluence traversal and newsletter processing.
e3sm_comms/exported_xml_reviewer/main.py	Implements the new WordPress-exported XML reviewer and report generation.
e3sm_comms/exported_xml_reviewer/init.py	Declares the new package module.
e3sm_comms/e3sm_org_reviewer/main.py	Replaces the older reviewer with an XML-export-driven analyzer producing multiple Markdown reports.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 26 out of 27 changed files in this pull request and generated 8 comments.

forsyth2 · 2026-08-11T00:12:16Z

The latest commit (8512e8c) ports over remaining functionality in e3sm_org_reviewer to exported_xml_reviewer. This will allow us to remove the former.

Remaining action items:

Remove all e3sm_org_reviewer-specific code, reducing the size of the diff
Re-run exported_xml_reviewer to be sure it behaves as expected.
Optional, or in later PR: do any refactoring that could reduce the code complexity, or large number of output reports.
Visually inspect remaining pull request
Merge

Copilot

🟡 Changes recommended

It contains at least one correctness issue in report categorization/counting and a security hardening gap in XML parsing that should be addressed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

e3sm_comms/utils.py:2

Using the stdlib xml.etree.ElementTree parser leaves this tool vulnerable to XML entity-expansion / related denial-of-service issues if someone accidentally (or maliciously) supplies a non-WordPress XML file. Since this code already anticipates defusedxml, prefer it when available (with a safe fallback) so the default behavior is hardened.

import re
import xml.etree.ElementTree as ET

Files reviewed: 17/18 changed files
Comments generated: 2
Review effort level: Lite

+            if redirect_target and normalize_url(redirect_target) in actual_urls:
+                redirected_links.add(linked_norm)
+                valid_links.add(linked_norm)
+            elif timed_out:


+    except requests.exceptions.HTTPError as e:
+        status_code = e.response.status_code if e.response is not None else None
+        if status_code == 503 and (
+            e.response.url if e.response is not None else ""
+        ).startswith("https://e3sm.org"):
+            return "link not whitelisted"
+        return "link raises RequestException"
+    except requests.exceptions.RequestException:
+        return "link raises RequestException"


forsyth2

I've done a very high-level visual inspection to go along with Copilot's review.

The functionality here was mostly ad-hoc for the website review, so it's not overly important that we have "productized" code (i.e., if we need a different report later, that's fine).

forsyth2 · 2026-09-04T23:53:58Z

+    print(f"Wrote report to {OUTPUT_TERMS_REPORT}")
+    print(f"Wrote hierarchical outline to {OUTPUT_HIERARCHICAL_OUTLINE}")
+    print(f"Wrote navigation issues report to {OUTPUT_NAVIGATION_ISSUES_REPORT}")
+    print(
+        f"Wrote invalid internal links report to {OUTPUT_INVALID_INTERNAL_LINKS_REPORT}"
+    )
+    print(f"Wrote published pages link report to {OUTPUT_PUBLISHED_PAGES_LINK_REPORT}")
+    print(f"Wrote external links report to {OUTPUT_EXTERNAL_LINKS_REPORT}")


Consider reordering the reports to be alphabetical or in any case a more coherent order (e.g., internal links should be listed paired with external links)

forsyth2 · 2026-09-04T23:58:12Z

+echo "1. ${IO_DIR}/output/exported_xml_reviewer/wordpress_sensitive_terms_report.md"
+echo "2. ${IO_DIR}/output/exported_xml_reviewer/wordpress_hierarchical_outline.txt"
+echo "3. ${IO_DIR}/output/exported_xml_reviewer/wordpress_navigation_issues_report.md"
+echo "4. ${IO_DIR}/output/exported_xml_reviewer/wordpress_invalid_internal_links_report.md"
+echo "5. ${IO_DIR}/output/exported_xml_reviewer/wordpress_published_pages_link_report.md"
+echo "6. ${IO_DIR}/output/exported_xml_reviewer/wordpress_invalid_external_links_report.md"
+echo "Optional (pass --check-non-published-access to also get):"


Reorder these as well.

forsyth2 · 2026-09-04T23:58:30Z

+  - From WordPress under Tools > Export: xml file of WordPress pages, xml file of WordPress posts. NOTE: Only ever use the XML files downloaded directly from WordPress; the source must be trusted.
+  - From output of `e3sm-comms-website-reviewer`: txt file of hierarchical outline of Confluence pages
+  - Other: txt file of sensitive terms, txt file of whitelisted e3sm.org pages, txt file of links known to be inaccessible for automated review
+- output: 6 files under `output/exported_xml_reviewer/`: (1) `wordpress_sensitive_terms_report.md`, (2) `wordpress_hierarchical_outline.txt`, (3) `wordpress_navigation_issues_report.md`, (4) `wordpress_invalid_internal_links_report.md`, (5) `wordpress_published_pages_link_report.md`, (6) `wordpress_invalid_external_links_report.md`


Reorder these as well

forsyth2 · 2026-09-04T23:59:17Z

+        f.write(f"## 5. Requested links ({len(sorted_requested_link_records)})\n\n")
+        if sorted_requested_link_records:
+            f.write(
+                "| e3sm.org link | Included later on this page? | Current status | Currently whitelisted? | Requesting URLs |\n"


Is "Included later on this page?" a holdover from a previous ordering? This is the last section on the report currently.

forsyth2 self-assigned this May 6, 2026

forsyth2 added the Website restructure label May 6, 2026

forsyth2 changed the title ~~Add functionality to review web pages for inclusion/exclusion~~ Add functionality to review web pages May 6, 2026

forsyth2 marked this pull request as ready for review June 1, 2026 19:45

forsyth2 requested a review from Copilot June 1, 2026 19:45

Copilot started reviewing on behalf of forsyth2 June 1, 2026 19:45 View session

Copilot AI reviewed Jun 1, 2026

View reviewed changes

forsyth2 force-pushed the enable-disable-pages branch from 29b6b25 to 42d6888 Compare June 1, 2026 20:17

forsyth2 requested a review from Copilot July 2, 2026 16:38

Copilot started reviewing on behalf of forsyth2 July 2, 2026 16:38 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Add xml reviewer

07bcdb7

forsyth2 force-pushed the enable-disable-pages branch from 8512e8c to 07bcdb7 Compare September 4, 2026 20:55

forsyth2 added 2 commits September 4, 2026 16:42

Remove e3sm_org_reviewer

0caa85a

Fixes from Claude

44f4a7c

forsyth2 requested a lite review from Copilot September 4, 2026 23:33

Copilot started reviewing on behalf of forsyth2 September 4, 2026 23:33 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

forsyth2 commented Sep 5, 2026

View reviewed changes

Uh oh!

Conversation

forsyth2 commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

forsyth2 commented Aug 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

forsyth2 left a comment

Choose a reason for hiding this comment

Uh oh!

forsyth2 Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

forsyth2 Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

forsyth2 Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

forsyth2 Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

forsyth2 commented May 6, 2026 •

edited

Loading