Move test files to respective plugins

[thelovekesh]

Summary

Previously #1065
See #1065 (comment)

Relevant technical choices

• Move test files from tests/plugins to respective plugins.
• Move helper files to the tests/data directories.

[thelovekesh]

Autoloading test helpers with Composer has been removed so PSR4 file naming is no longer, required as it is not ideal to autoload test helpers for multiple plugins from a single location. In most cases, PHPUnit's file loader will handle this effectively.

[westonruter]

Where were the autoloading test helpers removed?

[thelovekesh]

Moved them to their plugin tests/data directories.

[thelovekesh]

This change is necessary because when PHPUnit's file loader requires this file, the base FS file is not loaded.

[westonruter]

Would this make more sense in the set_up_mock_filesystem() method?

[thelovekesh]

We don't need to require it manually because filesystem_method_file already loads it if necessary. However, there is an issue with how PHPUnit includes files defined in a test suite. When PHPUnit tries to include this file, WP_Filesystem_Base isn't loaded by WordPress, resulting in a fatal error:

[error]

PHP Fatal error:  Uncaught Error: Class "WP_Filesystem_Base" not found in /app/public/content/plugins/performance/plugins/performance-lab/tests/data/class-wp-filesystem-mockfilesystem.php:20
Stack trace:
#0 phar:///usr/local/bin/phpunit/phpunit/Util/FileLoader.php(57): include_once()
#1 phar:///usr/local/bin/phpunit/phpunit/Util/FileLoader.php(43): PHPUnit\Util\FileLoader::load('/app/public/con...')
#2 phar:///usr/local/bin/phpunit/phpunit/Framework/TestSuite.php(309): PHPUnit\Util\FileLoader::checkAndLoad('/app/public/con...')
#3 phar:///usr/local/bin/phpunit/phpunit/Framework/TestSuite.php(397): PHPUnit\Framework\TestSuite->addTestFile('/app/public/con...')
#4 phar:///usr/local/bin/phpunit/phpunit/TextUI/TestSuiteMapper.php(56): PHPUnit\Framework\TestSuite->addTestFiles(Array)
#5 phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php(307): PHPUnit\TextUI\TestSuiteMapper->map(Object(PHPUnit\TextUI\XmlConfiguration\TestSuiteCollection), 'performance-lab')
#6 phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php(106): PHPUnit\TextUI\Command->handleArguments(Array)
#7 phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php(96): PHPUnit\TextUI\Command->run(Array, true)
#8 /usr/local/bin/phpunit(2520): PHPUnit\TextUI\Command::main()
#9 {main}

Next PHPUnit\TextUI\RuntimeException: Class "WP_Filesystem_Base" not found in phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:98
Stack trace:
#0 /usr/local/bin/phpunit(2520): PHPUnit\TextUI\Command::main()
#1 {main}
  thrown in phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php on line 98

Fatal error: Uncaught Error: Class "WP_Filesystem_Base" not found in /app/public/content/plugins/performance/plugins/performance-lab/tests/data/class-wp-filesystem-mockfilesystem.php:20
Stack trace:
#0 phar:///usr/local/bin/phpunit/phpunit/Util/FileLoader.php(57): include_once()
#1 phar:///usr/local/bin/phpunit/phpunit/Util/FileLoader.php(43): PHPUnit\Util\FileLoader::load('/app/public/con...')
#2 phar:///usr/local/bin/phpunit/phpunit/Framework/TestSuite.php(309): PHPUnit\Util\FileLoader::checkAndLoad('/app/public/con...')
#3 phar:///usr/local/bin/phpunit/phpunit/Framework/TestSuite.php(397): PHPUnit\Framework\TestSuite->addTestFile('/app/public/con...')
#4 phar:///usr/local/bin/phpunit/phpunit/TextUI/TestSuiteMapper.php(56): PHPUnit\Framework\TestSuite->addTestFiles(Array)
#5 phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php(307): PHPUnit\TextUI\TestSuiteMapper->map(Object(PHPUnit\TextUI\XmlConfiguration\TestSuiteCollection), 'performance-lab')
#6 phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php(106): PHPUnit\TextUI\Command->handleArguments(Array)
#7 phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php(96): PHPUnit\TextUI\Command->run(Array, true)
#8 /usr/local/bin/phpunit(2520): PHPUnit\TextUI\Command::main()
#9 {main}

Next PHPUnit\TextUI\RuntimeException: Class "WP_Filesystem_Base" not found in phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:98
Stack trace:
#0 /usr/local/bin/phpunit(2520): PHPUnit\TextUI\Command::main()
#1 {main}
  thrown in phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php on line 98

[westonruter]

And this is needed because autoloading of test helpers was removed?

[thelovekesh]

Yes.

[westonruter]

I guess not. It's just not clear to me how this WP_Filesystem_Base class was being autoloaded before but now it isn't.

[thelovekesh]

performance/composer.json

Line 38 in 9c410e5

"PerformanceLab\\Tests\\": "tests/utils"

And on trunk it's inside the utils - https://github.com/WordPress/performance/tree/trunk/tests/utils/Filesystem

[westonruter]

So will the autoload-dev then be removed from composer.json?

[westonruter]

What if instead of psr-4 it was replaced with a classmap instead? Then we could keep the autoloading, although I guess the issue then is that we have a classmap for all of the plugins together?

[thelovekesh]

Yes that's the concern. It seems like I haven't removed the autoloading from composer yet, but that's need to be removed.

[thelovekesh]

😎

[westonruter]

Assuming this PR is ready for review? Approved! cc @joemcgill

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: thelovekesh <thelovekesh@git.wordpress.org>
Co-authored-by: westonruter <westonruter@git.wordpress.org>
Co-authored-by: joemcgill <joemcgill@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[joemcgill]

Without repeating all of my previous comments from #1012 (comment) and #1012 (comment) I still think we would be better served to organize and run our PHPUnit tests from a single root directory.

However, I acknowledge that ever since we began the move from modules to standalone plugins for performance features, we are already functionally treating each set of tests for each plugin as separate suites and this PR does nothing to change that, but instead makes the current organization of the files make more sense now that we've moved the main Performance Lab plugin into the /plugins/ folder, so I'm ok for us to merge this.

That said, I would strongly suggest that we first agree on the solution in #1012 or a new issue just to define our PHPUnit structure before moving forward with anything else that was originally implemented in #1065, i.e.:

• Move tests to their plugin directories and move any helper files to the tests/data directory.
• Add separate phpunit.xml.dist to plugin directories. The current setup also allows to override of PHPUnit config per plugin e.g. put a phpunit.xml file in the plugin directory and PHPUnit will override phpunit.xml.dist.
• Move test commands from composer to npm scripts. NPM allows to run scripts from any directory and its config feature helps provide required args from CLI to npm commands out of the box. See: https://docs.npmjs.com/cli/v10/using-npm/config#command-line-flags

[westonruter]

As @joemcgill and I discussed over DM, even though all of the standalone plugins are being tested in isolation today, we should look for opportunities to test them together to ensure that they all work together while all are active. To this end, could still put tests in the root tests directory specifically for testing plugins together (beyond just running all of the tests as part of a single test suite in one process). So we could have a /tests/auto-sizes-and-webp-uploads directory for tests that have both of those plugins active, for example. Otherwise, if a test is specifically and only for the standalone plugin, it can be put in plugins/*/tests which will ensure the bootstrap logic will only need to load that one plugin to do the tests. Relatedly, in the Image Prioritizer plugin PR I've updated the bootstrap logic to support plugin dependencies: https://github.com/WordPress/performance/pull/1235/files#diff-96a82592013e5f4b91e682332583b7a4a6ca1ff0431d267c61179b1a21b167da

We could also consider adding an all testsuite which runs all of the plugins' unit tests suites in the same process. This will be essentially bringing back what we had before with modules and we'd be able to check for fatal errors when multiple plugins are active.

In this way, we'll have both angles covered: plugins should be able to work 100% while running in complete isolation as a standalone product, but the plugins should also be able to work 100% when all other plugins are active as part of the complete suite of Performance Lab plugins. We just need to make sure that in this process that we minimize as much as possible any fragmentation or duplication of the infrastructure code being used by the standalone plugins.