PoC: Visual regression testing with Playground #62729
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Size Change: 0 B Total Size: 1.76 MB ℹ️ View Unchanged
|
WunderBart
force-pushed
the
try/add-vis-reg-test
branch
from
449a2d5
to
3c2989a
Compare
WunderBart
force-pushed
the
try/add-vis-reg-test
branch
from
3c2989a
to
6fd5216
Compare
|
I tried to test this, and I'm not sure if I'm testing incorrectly or if there is something else going on. If I manually try the PR preview page with PR 59242, I receive an error... |
You're testing correctly. The problem is the PR you entered (#59242), where the zipped artifact for the build from that PR is no longer available. This is because those artifacts have a maximum retention period of 90 days. I don't know the current setting for the retention days in Gutenberg, but that PR is already over 130 days old. @smithjw1, would you consider it a blocking limitation for the visual regression comparison if we couldn't test PRs older than 90 days? As a side note, the code that checks if the artifact is available is not working as expected, which is why the correct error is not being thrown: edit: The issue has been reported in WordPress/wordpress-playground#1568 |
No, not blocking. Automated testing shouldn't be an issue since the goal here is automated testing. I picked that PR because I know it has changes that the test should pick up. Thanks for logging that other issue! I'll look through more recent PRs to locate the change the test will reveal. |
Cool! Let me know if you need help designing a test for a specific regression. It would be nice to see how Playwright performs here. 🙌 |
WunderBart
force-pushed
the
try/add-vis-reg-test
branch
from
b8f5a90
to
faad6a5
Compare
WunderBart
force-pushed
the
try/add-vis-reg-test
branch
from
faad6a5
to
a3eefdf
Compare
|
Flaky tests detected in a3eefdf. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/9856148260
|
|
This is lovely @WunderBart, thank you so much for sharing! It's already great for running tests. I wonder if we could go a step further and record and replay that directly in the browser. I know Chrome devtools have this recorder: https://developer.chrome.com/docs/devtools/recorder I wonder if there are any ways of controlling it from a website or a browser extension. I know there are browser extensions that record to Playwright, I wonder if there's anything to run those recorded tests – even if they were restricted to a subset of commands, not the full power of Playwright. If that was possible, we could add fantastic debugging, error reporting, and E2E testing features to Playground. |
|
Warning: Type of PR label mismatch To merge this PR, it requires exactly 1 label indicating the type of PR. Other labels are optional and not being checked here.
Read more about Type labels in Gutenberg. Don't worry if you don't have the required permissions to add labels; the PR reviewer should be able to help with the task. |
bgrgicak
pushed a commit
to WordPress/wordpress-playground
that referenced
this pull request
Jul 16, 2024
## Motivation for the change, related issues This is related to supporting visual regression testing of Gutenberg PRs with Playground. WordPress/gutenberg#62729 Importing content into a Playground makes testing easier. This will allow the supported `import-site` parameter to be used on this page as well. ## Implementation details This change adds a check for a the `import-site` query parameter and adds a step to the blueprint when it is found. The link has minimal validation, but I don't believe this creates a significant security concern. In theory someone could corrupt the blueprint by directing people to this URL with a malformed query parameter. Validating that the parameter is a URL, and not passing the query string or hash seemed to mitigate the most obvious offenses. ## Testing Instructions (or ideally a Blueprint) 1. Run locally 2. Visit `gutenberg.html` with a query parameter `import-site` pointing to a valid Playground ZIP import. 3. Confirm that Playground has imported content from ZIP file.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What?
This is a simple PoC for testing visual regressions using Playwright + Playground's Gutenberg PR Preview tool.
The example test compares the editor's default view with the one from a given PR number.
At this point, it does not use any tools from the Gutenberg E2E framework. It can, though, but to be functional, it doesn't really need anything from this repository apart from the Playwright dependency. It was just easier to put it here than create a new repo.
Warning
The Gutenberg PR Preview tool relies on the availability of build artifacts uploaded to GitHub. These artifacts have a maximum retention period of 90 days. Consequently, this PoC cannot test PRs older than 90 days.
How?
For example:
Headlessly:
Headfully:
Update snapshots:
cc: @adamziel @dmsnell @smithjw1 @griffbrad