Add Comprehensive Appium E2E Testing Framework for Sefaria Mobile App by HershelT · Pull Request #175 · Sefaria/Sefaria-Mobile

HershelT · 2025-07-14T16:08:57Z

This PR introduces a robust, modular end-to-end (E2E) testing framework for the Sefaria mobile app, built with WebdriverIO and Appium. The framework enables automated testing on Android devices both locally and via BrowserStack, supporting continuous integration with GitHub Actions.

Key Features:

Modular test structure for easy maintenance and expansion
Automated logging of test runs and UI errors (with screenshots)
Support for local and cloud (BrowserStack) device testing
Clear documentation for setup, running, and contributing new tests
Utilities for interacting with app UI, navigation, and Sefaria APIs
Example tests covering navigation, language toggling, learning schedules, and more
Why this matters:
Automated E2E testing will help ensure the quality and reliability of the Sefaria app across updates, reduce manual QA effort, and make it easier for contributors to add new tests as features evolve.

How to get started:
See the included README.md for setup instructions, prerequisites, and contribution guidelines. All contributors are welcome to add new tests or improve existing ones!

… Removed .only from testing so all tests can run

HershelT · 2025-07-14T16:09:33Z

@akiva10b @yitzhakc @schreibersefaria Pull Request created. Woohoo Automatic tests!

…t, and improved swipe functionality

…ions, and creating texts tab check (almost done)

…gewt the topics page and specific topics, enhanced ui checker for specific screenshot checking, and continued to use newly implemented text_constants

schreibersefaria

It would be helpful if you fix up the readme before I continue the PR.
Alternatively, could you add to the PR description more details about each file and the general structure.

…ull review

…files

Seperated the README into 4 seperate components for better instructions and clarity. (SETUP, TEST_GUIDE, FILE_OVERVIEW, README). This creates greater understanding of what everything does, how to create tests, and easier way to run them. Also, added headers to each file, explaining what each file does, its purpose, and its usage.

…f concerns. FILE_OVERVIEW reflects this change

schreibersefaria

I'm still (almost) only commenting on the MD files but enough has accumulated to push the review.
I think it would make sense to huddle to discuss the PR before you put too much work into it and before I continue to the rest of the content

…nts.

…ject Model (POM)

…xed browserstack error where it would not launch app.

…ainability - Introduced centralized constants for gestures, colors, timeouts, and selectors to replace hardcoded values across the testing framework. - Updated gesture utility functions to utilize new gesture configuration constants for swipe distances and durations. - Refactored UI interaction utilities to use centralized selectors for offline popups and text finding, enhancing consistency. - Enhanced timeout management by consolidating timeout values into a dedicated constants file for better readability and maintenance. - Improved documentation across various utility files to clarify usage and structure.

- Consolidated static and dynamic error messages into structured objects in error_constants.ts for better organization and maintainability. - Updated all references to error messages throughout the codebase to use the new structure. - Removed unused gesture validation thresholds from gestures.ts. - Cleaned up selectors.ts by adding a new selector for content description. - Removed unnecessary timeout constants from timeouts.ts. - Enhanced logging in e2e tests to provide clearer information on test execution status. - Updated helper functions and UI checker utilities to utilize the new error handling structure. - Improved debug logging for better traceability during test execution.

… and maintainability across multiple components

…agement in tests

… consistency. Removed repeated code.

schreibersefaria

Hope to do some more tomorrow

schreibersefaria · 2025-08-12T20:34:18Z

+  Central import point for all components, allowing easy access to all page objects.
+
+- **\*.ts**  
+  Add new component helpers as needed.


ahh I think it's understood implicitly

schreibersefaria · 2025-08-12T20:36:06Z

+   - Upload your APK/AAB (Android) or IPA (iOS) file to BrowserStack using their [App Upload page](https://app-automate.browserstack.com/dashboard/v2/app-upload).
+   - Copy the App ID (e.g., `bs://<app-id>`) and set it in your `.env` as `ANDROID_BROWSERSTACK_APP_ID` or `IOS_BROWSERSTACK_APP_ID`:
+   ```env
+   - **Note:** You must upload your app every time you build a new APK/AAB/IPA. The App ID changes with each upload, so always update `*_BROWSERSTACK_APP_ID` in your `.env`.


mmm in that case personally I would prefer not to have it at all.
I'd probable just run it locally anyway. I'm not sure I even have permissions to edit the secrets

schreibersefaria · 2025-08-12T21:01:10Z

+    if (parasha) {
+      await TEXT_FINDER.findTextElement(client, parasha.displayValue.en);
+    } else {
+      throw new Error(DYNAMIC_ERRORS.apiResultMismatch("Parashat Hashavua", parasha!.displayValue.en));    


but this if is on SEFARIA_API.getCurrentParashatHashavua not TEXT_FINDER

schreibersefaria · 2025-08-12T21:17:06Z

Not sure where the conversation about this is so opening it again.
I think it makes more sense to have the tests in logical files.
You would then define what tests are regression somewhere else.

Hmmm, logically that makes sense. I will have to go through and possible edit my parallel running script if I do that.

schreibersefaria

I've added some comments that generalise to most the code.
The big things I noticed are error handling and return values.
In general, consistency is key (but very hard to do).

schreibersefaria · 2025-08-13T13:34:42Z

+ * @param client - WebdriverIO browser instance.
+ * @param isEnglish - Current language state; true if English, false if Hebrew.
+ */
+export async function toggleLanguageButton(client: Browser, isEnglish: boolean = true): Promise<void> {


I don't think you need isEnglish, you can just check the current language.
It could be more elegant to have a param of targetLanguage, then you check if we are in the target language and only change if needed.

…o enhance clarity and maintainability

…ents

…larity. Turn Uppercase into PascalCase, and working on continuing to make imports better and more readable

…dication and improve error handling with helper function HelperFunctions.ensureElementDisplayed().

Copilot

Pull Request Overview

This PR introduces a comprehensive end-to-end testing framework for the Sefaria mobile app using WebdriverIO and Appium, supporting both Android and iOS platforms with local and cloud testing capabilities.

Modular, cross-platform architecture with centralized constants and platform-specific selectors
Automated UI validation including color pixel testing and visual regression capabilities
Integration with Sefaria APIs for dynamic content validation and BrowserStack for cloud testing

Reviewed Changes

Copilot reviewed 39 out of 40 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
utils/ui_checker.ts	Pixel color validation utilities with cross-platform coordinate system handling
utils/text_finder.ts	Text and element location helpers for robust UI interaction
utils/sefaria_api.ts	API utilities for fetching calendar data and learning schedules
utils/offline_popup.ts	Popup handling utilities for app initialization flows
utils/load_credentials.ts	Platform-agnostic credential loader for local and BrowserStack configurations
utils/helper_functions.ts	Core helper functions for text matching, color conversion, and test lifecycle
utils/gesture.ts	Gesture and scrolling utilities with platform-aware element visibility
utils/browserstack_report.ts	BrowserStack session reporting and status management
constants/*.ts	Centralized configuration for timeouts, colors, gestures, and error messages
components/*.ts	Page object models for navbar, reader, search, topics, and display settings
selectors/*.ts	Platform-specific UI selectors for Android and iOS
package.json	Dependencies and test execution scripts
Other files	Configuration, documentation, and test execution infrastructure

Comments suppressed due to low confidence (1)

automatic-e2e-tests/constants/errors.ts:1

Using non-null assertion parasha!.displayValue.en when parasha was already checked to be falsy in the condition above. This will throw a runtime error. The error message should use a default value or handle the undefined case properly.

/**

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

…ance error logging in ensureElementDisplayed function

…ces in documentation and code

…Added ability to close pop ups (closing it whenever it pops up in random intervals is still work in progress)

…elper functions with screenshot capture on failure. Working on speed and closing popups.

…one session instead of multiple versions.

…screenshot functionality. (Improving testing time)

… the use case of the file structure.

…top checking for specific types of popUps that appear.

…y and clarity

… This is for setup and reporting to browserstack to make it mkore efficient.

…ne. Increases speed.

schreibersefaria

A few big comments and some smaller ones.
Overall it looks to be going in the right direction but there are some big changes still ahead of us.
As you may notice I haven't yet gone over every line so after you get to all the comments, please request another review and ping me.
And you don't have to wait for the next review for questions. I'm generally available :-)

schreibersefaria · 2025-09-16T07:28:35Z

+  Centralized logs and screenshots for both platforms.
+
+- **scripts/**  
+  Utility scripts (e.g., cleanup.js). `run-parallel-tests.js` for running tests in parallel on multiple devices.


You give two examples. But one is in brackets and one isn't.
I'd would unify the format here

schreibersefaria · 2025-09-16T07:33:59Z

+
+---
+
+## components/


Confusing styling.
Some dirs are h2 and some are bold.
Also, I'd order the dirs by importance

schreibersefaria · 2025-09-16T07:36:49Z

I'm not sure why this is needed.
I understood the logic around selectors but not sure it duplicates to other dirs.

schreibersefaria · 2025-09-16T07:43:48Z

+  Selectors = require('../selectors/android/selectors');
+}
+
+export { Selectors };


I find it confusing that you combined the index of constants and selectors.
Was there a reason for that

schreibersefaria · 2025-09-16T07:47:28Z

Here too, if there isn't a unified ios/android import, I don't know if there is a need for the index file.
If people need to import a lot of constance then a long import line / many imports are appropriate

schreibersefaria · 2025-09-16T10:11:19Z

+export const APP_PACKAGE = 'org.sefaria.sefaria';
+
+// Base UiSelector patterns
+export const BASE_SELECTORS = {


Another naming comment.
Not sure why selectors would be all caps. They are functions.
This could be a best practice I'm not familiar with

schreibersefaria · 2025-09-16T10:13:20Z

+
+// iOS Accessibility patterns
+export const ACCESSIBILITY_PATTERNS = {
+  englishTextPrefix: '⁦',


If you are using a hidden unicode char you should definitely comment it

schreibersefaria · 2025-09-16T10:13:53Z

Generally in both selector files, some more commenting would be very helpful for anyone who needs to edit it

schreibersefaria · 2025-09-16T10:16:30Z

+ * @throws Will throw an error if the text is not found
+ * @returns boolean indicating success
+ */
+export async function verifyTitleContains(client: Browser, expectedText: string): Promise<boolean> {


A lot of duplication from the previous function.
Could you combine them?

schreibersefaria · 2025-09-16T10:17:34Z

You have one document for all topic related things.
Maybe split each page within topics (the actual topic page and the navigation pages) into different files

…for improved test tracking

- Add CLAUDE.md: architecture, env vars, session-per-suite lifecycle, conventions, and platform gotchas - Rename ReadMe.md -> README.md so cross-doc backlinks resolve; rewrite with an accurate <suite>:<platform>:<env> command reference (documents regression:* and the test:* vs regression/sanity distinction) - SETUP.md: fix malformed/unclosed env code fence; tabulate env config, troubleshooting, and CI secrets - TEST_GUIDE.md: correct to session-per-suite model and ../log_init import; fix handleSetup/handleTeardown casing and duplicate PopUps import - FILE_OVERVIEW.md: fix wrong filenames (gesture_constants/popups/sefaria_api) and add missing files (log_init, .mocharc*, tsconfig, scripts) - example.env: fix swapped iOS/Android example app-path comments

- Delete the unused `isModularization` env-flag (read but never used) - Remove the leftover commented-out beforeEach/afterEach block from the old session-per-test model - Drop the now-obsolete MODULARIZATION row from CLAUDE.md

HershelT added 2 commits July 14, 2025 10:58

Add Appium E2E tests in automatic-e2e-tests folder

5b92ab8

fix: comments for browserstacks-test-android github actions workflow.…

138ecfb

… Removed .only from testing so all tests can run

HershelT assigned akiva10b and yitzhakc Jul 16, 2025

HershelT requested review from akiva10b, schreibersefaria and yitzhakc July 16, 2025 12:18

HershelT assigned schreibersefaria Jul 16, 2025

HershelT modified the milestone: Mobile Testing Jul 16, 2025

HershelT added 3 commits July 16, 2025 11:52

feat: enhance E2E testing framework with new constants, cleanup scrip…

24435cd

…t, and improved swipe functionality

feat: add cleanup instructions to ReadMe, refactor text utility funct…

60b0f5c

…ions, and creating texts tab check (almost done)

feat (Top[ics page test): Created topics page component to help navia…

99731e2

…gewt the topics page and specific topics, enhanced ui checker for specific screenshot checking, and continued to use newly implemented text_constants

schreibersefaria reviewed Jul 17, 2025

View reviewed changes

Comment thread automatic-e2e-tests/android/testing-framework/components/topics_page.ts Outdated

Comment thread automatic-e2e-tests/ReadMe.md Outdated

HershelT added 4 commits July 17, 2025 10:17

feat: cleaning up files, getting file names and .md files ready for p…

f549e9f

…ull review

Working on description for each utils file and creating seperate .md …

514860f

…files

feat: seperated logging declarations to test_init.ts for seperation o…

cd3e685

…f concerns. FILE_OVERVIEW reflects this change

HershelT requested a review from schreibersefaria July 17, 2025 20:43

schreibersefaria reviewed Jul 20, 2025

View reviewed changes

HershelT added 2 commits July 20, 2025 15:36

Edit documentation (.md files) to address and satisfy PR review comme…

1a16345

…nts.

docs(.md): clarify that components/ structure aligns with the Page Ob…

b689cd4

…ject Model (POM)

HershelT requested a review from schreibersefaria July 20, 2025 20:43

HershelT added 6 commits July 21, 2025 14:56

chore: looking closer at .md files to ensure better understanding. Fi…

04e8a77

…xed browserstack error where it would not launch app.

refactor: consolidate imports from constants for improved readability…

481e121

… and maintainability across multiple components

refactor: introduce NAVBAR_SELECTORS for improved navigation item man…

49f3ee3

…agement in tests

refactor: rename UI color checking functions for improved clarity and…

30b5a54

… consistency. Removed repeated code.

schreibersefaria reviewed Aug 12, 2025

View reviewed changes

schreibersefaria reviewed Aug 13, 2025

View reviewed changes

HershelT added 4 commits August 13, 2025 14:53

refactor: Update file roles and descriptions for various components t…

a85f3b3

…o enhance clarity and maintainability

refractor: working on enhancing erros and return values of POM Compon…

0d866fe

…ents

Refactor utility imports and constants for improved consistency and c…

e961bc4

…larity. Turn Uppercase into PascalCase, and working on continuing to make imports better and more readable

refactor: Update component functions to return boolean for success in…

7c00fef

…dication and improve error handling with helper function HelperFunctions.ensureElementDisplayed().

HershelT requested a review from Copilot August 31, 2025 13:57

Copilot AI reviewed Aug 31, 2025

View reviewed changes

HershelT added 11 commits September 1, 2025 11:25

refactor: Adjust swipe configuration for TEXT_SCROLL_DISTANCE and enh…

04fa999

…ance error logging in ensureElementDisplayed function

refactor: Rename e2e.spec.ts to regression.spec.ts and update referen…

c79a82b

…ces in documentation and code

feat: Starting work on creating sanity checks

8cc364f

feat: Finished up the sanity checks (maybe refine it a little more). …

952591e

…Added ability to close pop ups (closing it whenever it pops up in random intervals is still work in progress)

refactor: fixing up all the sanity tests. Enhance error handling in h…

f05178d

…elper functions with screenshot capture on failure. Working on speed and closing popups.

feat: working on increasing test speed and redliability. Made it all …

0d0fa2b

…one session instead of multiple versions.

feat: enhance popup handling with continuous monitoring and improved …

1bd2c0c

…screenshot functionality. (Improving testing time)

chore: fixing up some text and missing colons.

9005898

chore: cleaning up lack of colons and unsued imports

b23d3f2

chore: moved around screenshot capture to ui_checker.ts to align with…

c8313e1

… the use case of the file structure.

chore: some tweaks and notes from pre Review from Daniel

880f968

HershelT requested a review from schreibersefaria September 11, 2025 08:01

HershelT added 5 commits September 11, 2025 12:40

refractor: Fixed up global popUp Monitor to increase efficiency and s…

e9d5613

…top checking for specific types of popUps that appear.

refactor: Update popup handling and test setup for improved efficienc…

528237a

…y and clarity

feat: Moving around hard coded steps and attaching them to functions.…

f05be63

… This is for setup and reporting to browserstack to make it mkore efficient.

feat: Refining backgroun popup monitor to close pop up quickly.

333bd51

chore: fixing up closePopUp to consolidate it from two functions to o…

990cb47

…ne. Increases speed.

schreibersefaria reviewed Sep 16, 2025

View reviewed changes

yitzhakc removed their request for review October 16, 2025 11:39

HershelT added 3 commits June 1, 2026 09:40

feat: Add regression test scripts and enhance BrowserStack reporting …

dac6013

…for improved test tracking


		---

		## components/

Uh oh!

Conversation

HershelT commented Jul 14, 2025

Uh oh!

HershelT commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

schreibersefaria left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

schreibersefaria left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

schreibersefaria left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HershelT Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

schreibersefaria left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

schreibersefaria left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

HershelT commented Jul 14, 2025 •

edited

Loading

HershelT Aug 13, 2025 •

edited

Loading