Skip to content

WIP: Playwright tests with shared repo #713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mickmister wants to merge 4 commits into from
Closed

WIP: Playwright tests with shared repo #713

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
dbfda81
works with submodule
mickmister Nov 17, 2023
8247a28
update playwright workflow
mickmister Nov 17, 2023
3acaf1d
empty commit to trigger playwright workflow
mickmister Nov 17, 2023
2b2c6a0
update go version in workflow
mickmister Nov 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 54 additions & 27 deletions .github/workflows/playwright.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
name: Playwright Tests
on:
schedule:
- cron: "0 0 * * *"
workflow_run:
workflows: ["ci"]
types:
- completed
push:
branches:
- master
tags:
- "v*"
pull_request:
workflow_dispatch:

env:
TERM: xterm
GO_VERSION: 1.19.6
NODE_VERSION: 16.15.0
GO_VERSION: 1.21

jobs:
playwright-run:
Expand Down Expand Up @@ -98,6 +98,8 @@ jobs:
steps:
- name: ci/checkout-repo
uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # v3.3.0
with:
submodules: 'true'

- name: ci/setup-go
uses: actions/setup-go@6edd4406fa81c3da01a34fa6f6343087c207a568 # v3.5.0
Expand All @@ -109,8 +111,8 @@ jobs:
uses: actions/setup-node@64ed1c7eab4cce3362f8c340dee64e5eaeef8f7c # v3.6.0
with:
node-version-file: ".nvmrc"
cache: "npm"
cache-dependency-path: webapp/package-lock.json
# cache: "npm"
# cache-dependency-path: webapp/package-lock.json
Comment on lines +114 to +115

@hanzei hanzei Nov 17, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did this get removed?


- name: ci/create-admin-user
run: |
Expand All @@ -119,7 +121,8 @@ jobs:

- name: ci/install-go-dependencies
run: go mod tidy
- name: ci/install-npm-dependencies

- name: ci/install-plugin-webapp-deps
run: cd webapp && npm ci

- name: ci/build-and-install-plugin
Expand All @@ -128,45 +131,69 @@ jobs:
PLUGIN_E2E_MOCK_OAUTH_SERVER_URL: http://172.17.0.1:8080
E2E_TESTING: true

- name: ci/setup-node
# - name: ci/checkout-mattermost-monorepo
# uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # v3.3.0
# with:
# path: ../mattermost
# repository: mattermost/mattermost

- name: ci/checkout-mattermost-monorepo
run: |
git clone https://github.com/mattermost/mattermost.git ../mattermost
Comment on lines +140 to +142

@hanzei hanzei Nov 17, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if the version should be opt-in here. Otherwise changes on mattermost could break the E2E tests in this repo. On the other hand, it might be good if we notice a breaking change early.

@mickmister mickmister Nov 20, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hanzei This is intentional to have it always use master, though I think we could have two workflows if we also want a pinned version, like we did with the Apps plugin.

could break the E2E tests in this repo

The main thing that was breaking things in the plugin e2e tests related to the monorepo where the Playwright helpers that the e2e tests were using, like getLastPost. The function implementations didn't change (because the general DOM structure of the webapp didn't change), but where the function is located can change at any time. These are also not necessarily meant to be consumed by other projects.

If the plugin e2e test fails because a behavior change in the webapp's functionality, I think we would want to see the test failure to see if it's something we need to address. If this does cause an issue, we'll handle it on a case-by-case basis.

To solve the issue of the helpers changing paths etc., the plan is to have these helpers intended to be used by plugin e2e tests defined in a separate repo, currently housed at mattermost-community/mattermost-plugin-e2e-test-utils#1

@hanzei hanzei Nov 27, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional to have it always use master, though I think we could have two workflows if we also want a pinned version, like we did with the Apps plugin.

👍 for using the same pattern as the Apps plugin (Release on PR, master on master)


- name: ci/setup-node-for-mattermost-monorepo
uses: actions/setup-node@64ed1c7eab4cce3362f8c340dee64e5eaeef8f7c # v3.6.0
with:
node-version: "16"
cache: "npm"
cache-dependency-path: e2e/playwright/package-lock.json
node-version-file: "../mattermost/.nvmrc"

- name: ci/install-mattermost-monorepo-webapp-deps
run: |
cd ../mattermost/webapp
npm i

- name: ci/install-tests-deps
- name: ci/install-mattermost-monorepo-playwright-deps
run: |
cd e2e/playwright
./setup-environment
cd ../mattermost/e2e-tests/playwright
npm i
npx playwright install --with-deps

- name: ci/setup-node-for-mattermost-plugin
uses: actions/setup-node@64ed1c7eab4cce3362f8c340dee64e5eaeef8f7c # v3.6.0
with:
node-version-file: ".nvmrc"

- name: ci/install-plugin-playwright-deps
run: |
cd e2e/playwright/mattermost-plugin-e2e-test-utils
npm i

- name: ci/tsc
run: |
cd e2e/playwright
cd e2e/playwright/mattermost-plugin-e2e-test-utils
npm run tsc

- name: ci/lint
run: |
cd e2e/playwright
npm run lint
# - name: ci/lint
# run: |
# cd e2e/playwright
# npm run lint

- name: ci/run-playwright-tests
run: |
cd e2e/playwright
cd e2e/playwright/mattermost-plugin-e2e-test-utils
npm run test-ci
env:
PLUGIN_E2E_MOCK_OAUTH_TOKEN: ${{ secrets.MM_GITHUB_PLUGIN_E2E_TOKEN }}
PLUGIN_E2E_MOCK_OAUTH_TOKEN: faketoken
PW_BASE_URL: ${{ env.MM_SERVICESETTINGS_SITEURL }}

- name: ci/move-artifacts
if: success() || failure()
run: |
cd e2e/playwright
cd e2e/playwright/mattermost-plugin-e2e-test-utils
mkdir results
mv playwright-report results

- uses: actions/upload-artifact@v3
if: success() || failure()
with:
name: test-results
path: e2e/playwright/results
path: e2e/playwright/mattermost-plugin-e2e-test-utils/results
3 changes: 3 additions & 0 deletions .gitmodules
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "e2e/playwright/mattermost-plugin-e2e-test-utils"]
path = e2e/playwright/mattermost-plugin-e2e-test-utils
url = https://github.com/mattermosttest/mattermost-plugin-e2e-test-utils.git
2 changes: 1 addition & 1 deletion .nvmrc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
14.21.1
16.13.1
1 change: 1 addition & 0 deletions e2e/playwright/mattermost-plugin-e2e-test-utils
Open in desktop
Submodule mattermost-plugin-e2e-test-utils added at 15f719
5 changes: 3 additions & 2 deletions e2e/playwright/support/utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,15 @@ export const submitDialog = async (page: Page) => {
};

export const postMessage = async (message: string, c: ChannelsPage, page: Page) => {
await c.postMessage(message);
await c.centerView.postCreate.input.waitFor();
await c.centerView.postCreate.postMessage(message);
await page.getByTestId('SendMessageButton').click();
};

export const clickPostAction = async (name: string, c: ChannelsPage) => {
// We need to wait for the next post to come up, since this opening a new tab and OAuth redirect can take an undeterminate
// https://mattermost.atlassian.net/browse/MM-51906
const postElement = await c.getLastPost();
const postElement = await c.centerView.getLastPost();
await postElement.container.getByText(name).last().click();
};

Expand Down
17 changes: 10 additions & 7 deletions e2e/playwright/tests/command/autocomplete.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import {test, expect} from '@e2e-support/test_fixture';

import {SlashCommandSuggestions} from '../../support/components/slash_commands';
import {getGithubBotDMPageURL} from '../../support/utils';
import {fillMessage, postMessage} from '../../mattermost-plugin-e2e-test-utils/support/utils';

const completeCommands = [
{position: 1, cmd: 'connect'},
Expand All @@ -33,11 +34,11 @@ export default {
const dmURL = await getGithubBotDMPageURL(adminClient, '', adminUser.id);
await page.goto(dmURL, {waitUntil: 'load'});

const c = new pages.ChannelsPage(page);
// const c = new pages.ChannelsPage(page);
const slash = new SlashCommandSuggestions(page.locator('#suggestionList'));

// # Run incomplete command to trigger help
await c.postMessage('/github');
await postMessage('/github', page);

// * Assert suggestions are visible
await expect(slash.container).toBeVisible();
Expand All @@ -62,7 +63,7 @@ export default {
const slash = new SlashCommandSuggestions(page.locator('#suggestionList'));

// # Run incomplete command+space to trigger autocomplete
await c.postMessage('/github ');
await postMessage('/github ', page);

// * Assert suggestions are visible
await expect(slash.container).toBeVisible();
Expand Down Expand Up @@ -90,7 +91,7 @@ export default {
const slash = new SlashCommandSuggestions(page.locator('#suggestionList'));

// # Run incomplete command to trigger help
await c.postMessage('/github');
await postMessage('/github', page);

// * Assert suggestions are visible
await expect(slash.container).toBeVisible();
Expand All @@ -116,7 +117,7 @@ export default {
const slash = new SlashCommandSuggestions(page.locator('#suggestionList'));

// # Run incomplete command+space to trigger autocomplete
await c.postMessage('/github ');
await postMessage('/github ', page);

// * Assert suggestions are visible
await expect(slash.container).toBeVisible();
Expand All @@ -143,7 +144,8 @@ export default {
const slash = new SlashCommandSuggestions(page.locator('#suggestionList'));

// # Run incomplete command to trigger help
await c.postMessage('/github');
await fillMessage('/github', page);
// await postMessage('/github', page);

// * Assert suggestions are visible
await expect(slash.container).toBeVisible();
Expand All @@ -166,7 +168,8 @@ export default {
const slash = new SlashCommandSuggestions(page.locator('#suggestionList'));

// # Run incomplete command+space to trigger autocomplete
await c.postMessage('/github ');
await fillMessage('/github ', page);
// await postMessage('/github ', page);

// * Assert suggestions are visible
await expect(slash.container).toBeVisible();
Expand Down
19 changes: 10 additions & 9 deletions e2e/playwright/tests/command/me.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import {
getBotTagFromPost,
getPostAuthor,
} from '../../support/components/post';
import {postMessage} from '../../mattermost-plugin-e2e-test-utils/support/utils';

const mmGithubHandle = 'MM-Github-Plugin';

Expand All @@ -32,14 +33,14 @@ export default {
const c = new pages.ChannelsPage(page);

// # Run comand
await c.postMessage('/github me');
await c.sendMessage();
await postMessage('/github me', page);
// await c.sendMessage();

@hanzei hanzei Nov 17, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove?


// # Wait for new messages to ensure the last post is the one we want
await waitForNewMessages(page);

// # Get last post
const post = await c.getLastPost();
const post = await c.centerView.getLastPost();
const postId = await post.getId();

// * Verify that message is sent by the github bot
Expand Down Expand Up @@ -77,14 +78,14 @@ export default {
const c = new pages.ChannelsPage(page);

// # Run comand
await c.postMessage('/github me');
await c.sendMessage();
await postMessage('/github me', page);
// await c.sendMessage();

// # Wait for new messages to ensure the last post is the one we want
await waitForNewMessages(page);

// # Get last post
const post = await c.getLastPost();
const post = await c.centerView.getLastPost();
const postId = await post.getId();

// * Verify that message is sent by the github bot
Expand Down Expand Up @@ -116,14 +117,14 @@ export default {
const c = new pages.ChannelsPage(page);

// # Run comand
await c.postMessage('/github me');
await c.sendMessage();
await postMessage('/github me', page);
// await c.sendMessage();

// # Wait for new messages to ensure the last post is the one we want
await waitForNewMessages(page);

// # Get last post
const post = await c.getLastPost();
const post = await c.centerView.getLastPost();
const postId = await post.getId();

// * Verify that message is sent by the github bot
Expand Down
19 changes: 10 additions & 9 deletions e2e/playwright/tests/command/todo.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import TodoMessage, {GithubRHSCategory} from '../../support/components/todo_mess
import {messages} from '../../support/constants';
import {getGithubBotDMPageURL, waitForNewMessages} from '../../support/utils';
import {getBotTagFromPost, getPostAuthor} from '../../support/components/post';
import {postMessage} from '../../mattermost-plugin-e2e-test-utils/support/utils';

const repoRegex = /https:\/\/github.com\/[\w-]+\/[\w-]+/;
const prRegex = /https:\/\/github.com\/[\w-]+\/[\w-]+\/pull\/\d+/;
Expand All @@ -31,14 +32,14 @@ export default {
const c = new pages.ChannelsPage(page);

// # Run todo command
await c.postMessage('/github todo');
await c.sendMessage();
await postMessage('/github todo', page);
// await c.sendMessage();

// # Wait for new messages to ensure the last post is the one we want
await waitForNewMessages(page);

// # Get last post
const post = await c.getLastPost();
const post = await c.centerView.getLastPost();
const postId = await post.getId();

const todo = new TodoMessage(post.container);
Expand Down Expand Up @@ -108,14 +109,14 @@ export default {
const c = new pages.ChannelsPage(page);

// # Run todo command
await c.postMessage('/github todo');
await c.sendMessage();
await postMessage('/github todo', page);
// await c.sendMessage();

// # Wait for new messages to ensure the last post is the one we want
await waitForNewMessages(page);

// # Get last post
const post = await c.getLastPost();
const post = await c.centerView.getLastPost();
const postId = await post.getId();

// * Verify that message is sent by the github bot
Expand Down Expand Up @@ -147,14 +148,14 @@ export default {
const c = new pages.ChannelsPage(page);

// # Run todo command
await c.postMessage('/github todo');
await c.sendMessage();
await postMessage('/github todo', page);
// await c.sendMessage();

// # Wait for new messages to ensure the last post is the one we want
await waitForNewMessages(page);

// # Get last post
const post = await c.getLastPost();
const post = await c.centerView.getLastPost();
const postId = await post.getId();

// * Verify that message is sent by the github bot
Expand Down
Loading