Add devcontainer + Codespaces support

[helen]

Finally opening up a Trac ticket after posting on ​Make/Core :)

I'd like to start adding "official" support for remote dev environments to contribute to core (including testing) - I believe this makes contributions more accessible to a wider audience, by making fully-featured dev environments available on more devices and getting people started much faster (i.e. you click a button on a repo instead of braving conference wifi to download Docker images or passing around USB sticks).

This ticket is more specifically for devcontainer + Codespaces support in core so you can start a Codespace from GitHub web - devcontainers are a set of config files that can be used with local Docker as well via VSCode (and otherwise). Codespaces is a GitHub-hosted solution to then run those devcontainers in the cloud (Azure) with the default ability to open VSCode for the Web connected to said container for a completely in-browser experience, meaning you can have a fully-featured code editing, building, and app running experience on a tablet.

Here's what I think is needed but need opinions on where we should have boundaries between published images, devcontainers, and setup scripts as I am not myself particularly knowledgeable about containers

• A baseline image that can be used for any number of dev needs - core, Gutenberg, other plugins and themes, testing, etc. This should probably include Composer, the appropriate Node+npm versions, and wp-cli. I have never yet gotten Xdebug to work in Codespaces. My current PR uses the Docker-published wordpress image, not the ones we use for wp-env - seems to me it would be nice to have something that spins up faster but I don't know enough here.
• A devcontainer.json that includes some VSCode extensions (or explicitly decide not to include any).
• Documentation, esp in the Make/Core Handbook. :)

Full disclosure: I do currently work at GitHub; not in this area, but I do have access to helpful resources as needed. There are a number of services out there for remote dev with varying tiers of free usage, over time it would be great to support multiple options.

Pulling over a PR I was originally noodling on here: ​https://github.com/helen/wordpress-develop/pull/53

This works - you can spin up a new Codespace on this branch and after waiting patiently for the setup script to run (it's too quiet, you have to watch setup.log to check on progress), you should be able to go to the preview URL and log in and everything. A 2-core works fine, 4-core is quite speedy but of course uses double the core hours.

However, I don't think this is ready to go into core :) I don't think it needs to be perfect but it feels a little weird to use the Docker-published wordpress image, and there are some details in here that don't feel right either.

Here are the lingering todo items I currently have:

• [ ] Do we actually need to be using the www-data user?
• [ ] Can we use the same containers that wp env uses?
• [ ] Allow some output to the terminal so it's clear what's happening
• [ ] Sometimes navigating to /wp-admin kicks me to https://localhost/wp-admin - no idea why, going to wp-login.php works fine, maybe something in the container somewhere

Trac ticket: https://core.trac.wordpress.org/ticket/57187#ticket

Hi 👋

I am Samruddhi, a Software Engineer for GitHub, working with the ​GitHub Codespaces team, and a maintainer for the ​dev containers org 👩‍💻

Thank you @helen for doing all the ground work for setting up a dev container (ref: ​https://github.com/WordPress/wordpress-develop/pull/3670). This PR is greatly inspired from the contents of #3670 with few tweaks.

Thanks,
Samruddhi

https://core.trac.wordpress.org/ticket/57187

Closing in favor of #4031

[helen]

I'm pulling this into the 6.2 milestone to indicate my intention to commit this, though I'm going to consider it "beta support" for Codespaces and we can kick it to a future milestone for further work as makes sense. It has no impact on the core build as everything is outside of src so I have no qualms about bringing it in during beta, though I will wait a day or so for any discussions folks might want to have about this.

I got some help from an engineer on Codespaces (samruddhikhandale) who's made this more robust as a first run and it would be great to have this in trunk so that folks can try it out at the upcoming WordCamp Asia contributor day on February 17. If you haven't yet used Codespaces, this allows people to essentially "hit play" and a cloud container spins up with trunk running from src installed at a portforwarded URL (private by default but can even be opened to the public), and VSCode connects to it from the browser or the desktop app. This does mean requiring connectivity to work, but contributors won't need to download images to their machines over venue wifi, and can even contribute from mobile devices and tablets with the in-browser VSCode option. 120 "core hours" are free for individual users, which really equates to 60 hours of usage at 2 cores (decent) or 30 hours at 4 cores (zippy). The underpinning of this is a devcontainer, which is an open spec that means you can also run the same environment via local Docker or any other tool that supports devcontainers. I just happen to be able to solicit direct help from my coworkers who are specifically working to enable OSS adoption of the tool and we have the GH mirror already :)

I would very much value feedback from a contributor day type of event so that we as a project can make a decision about whether this is worth getting further help on solidifying a container strategy, as the current iteration relies on the wordpress image that is actually published by Docker, not us. That's the primary reason I would consider this "beta support". I've also not tested multisite or running any tests in the container.

[SergeyBiryukov]

Replying to helen:

it would be great to have this in trunk so that folks can try it out at the upcoming WordCamp Asia contributor day

Indeed, this sounds like it would make it easier to get started on a contributor day :)

Once a contributor closes their VS Code browser window, how long does the Codespace remain active for?

By default, codespaces time out after 30 minutes of inactivity. When a codespace times out it is stopped and will no longer incur charges for compute usage.

Doc reference - ​here

@peterwilsoncc Let me know if this info would be helpful in the doc, happy to add that.

@samruddhikhandale Given the default spend limit is set to zero, I think it's fine to leave it out for now. It's easy enough to add something later if it becomes an issue.

[peterwilsoncc]

I've both tested and approved the PR; this looks good for commit in the coming days.

Testing process:

• built a codespace from the code dropdown on the PR
• checked npm and node versions in terminal: they match the expected versions from the nvm config file
• successfully logged in to WP Admin and viewed the front end

I broke the final step initially due to impatience, a note to be patient has since been added to the readme file.

Maybe some people who are unfamiliar with codespaces would be interested to know that I have been using a codespace to develop a plugin for the plugin directory, and the youth of the product absolutely shows. I have been hit by two major outages in little over a month, the latest one being ongoing for more than 48 hours and preventing me from releasing a new version to the plugin directory. The outage was ​reported as resolved, but either you are not testing closely enough, or deliberately misreporting to hide the high duration of downtime. The support ticket I submitted for the previous outage took 7 days to be answered, so I'm expecting 7 full days of downtime and 7 days delay to a plugin release that should already be completed by now. With service like this, there is no chance I will upgrade to the paid tier anytime soon. Not voting down this patch at all, but I wouldn't be encouraging the events team to jump into codespaces with both feet just yet, or it could result in an attendee's whole contributor day being wasted.

I actually think ​WordPress Playground would be a better long-term solution for this purpose, but it too obviously has a long way to go.

Hi @carlomanf - totally understood that you've experienced frustrations. We use Codespaces for our own dev environments at GitHub so any outages you experience we do as well. I cannot speak to this product area as far as work goes and don't think this is the greatest venue to go back and forth about it, in any case.

As said, this is a way to get a sense of how this particular tool and cloud dev environments as a general concept might benefit the core contribution case. devcontainers as an open spec should mean that this can lay the groundwork for generally being usable across different options based on what works best for the contributor, which should always be our goal - to make it easier for folks to contribute given their differing needs and constraints. A Wasm option would also be great! Let's just not let ourselves get stuck in attempting to be perfect from the start, that's already happened to this effort a couple times.

[helen]

In 55303:

Build/Test Tools: Add a devcontainer for Codespaces support.

This adds a devcontainer configuration tested for use with GitHub Codespaces. It is currently built on top of the Docker wordpress image for initial trial purposes. We should eventually be using our own containers to fully own the environment.

This should be considered a beta run to get a sense of how well cloud dev containers work for core contributors, in particular at contributor day events with limited bandwidth and mobile devices.

Props samruddhikhandale, dinhtungdu, helen, craiglpeters, sam1el, wirecat.
See #57187.

Committed in https://core.trac.wordpress.org/changeset/55303

[helen]

Notes:

• I think we need to commit the correct permissions on .devcontainer/install-tools.sh and .devcontainer/setup.sh rather than changing them at runtime, because otherwise they show as modified which interferes with doing version control tasks.
• It took 11 minutes to get from clicking "create a new codespace" to the environment being fully ready with WordPress installed - this is manageable with set expectations as a trial run but we need to explore our options for bringing this down. It was 2.5 minutes to start the container and open VSCode, so I believe it's npm install that is taking the bulk of the time here. Under 3 minutes total seems ideal to me although it's just an arbitrary number. I'll ask for help in understanding our options - because contributors should be running the Codespace from a fork in order to create a branch and open a PR, I don't think prebuilds work for this case.
• The setup script output is getting lost - at some point in iterations we were writing this to setup.log which is why it's in the .gitignore but it's gotten dropped - should decide what to do about that (write it to a file, output it to the terminal, something else?). Output to the terminal is helpful in reassuring users things are still moving along.

[johnbillion]

I can't get the devcontainer to start up in VS Code locally, I'm getting the following error.

Shall we use this ticket as a thread for now or shall we create separate tickets for issues now that it's been merged?

ERROR: Service 'wordpress' failed to build : Build failed
[14996 ms] Error: Command failed: docker-compose --project-name wp_devcontainer -f /Users/john/sites/wp/.devcontainer/docker-compose.yml -f /Users/john/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data/docker-compose/docker-compose.devcontainer.build-1676035102524.yml build
[14996 ms]     at gF (/Users/john/.vscode/extensions/ms-vscode-remote.remote-containers-0.275.1/dist/spec-node/devContainersSpecCLI.js:1872:412)
[14996 ms]     at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
[14996 ms]     at async rie (/Users/john/.vscode/extensions/ms-vscode-remote.remote-containers-0.275.1/dist/spec-node/devContainersSpecCLI.js:1872:2381)
[14996 ms]     at async tie (/Users/john/.vscode/extensions/ms-vscode-remote.remote-containers-0.275.1/dist/spec-node/devContainersSpecCLI.js:1854:2396)
[14996 ms]     at async Fie (/Users/john/.vscode/extensions/ms-vscode-remote.remote-containers-0.275.1/dist/spec-node/devContainersSpecCLI.js:1921:2266)
[14996 ms]     at async Vf (/Users/john/.vscode/extensions/ms-vscode-remote.remote-containers-0.275.1/dist/spec-node/devContainersSpecCLI.js:1921:3241)
[14996 ms]     at async eoe (/Users/john/.vscode/extensions/ms-vscode-remote.remote-containers-0.275.1/dist/spec-node/devContainersSpecCLI.js:2045:17324)
[14996 ms]     at async Qse (/Users/john/.vscode/extensions/ms-vscode-remote.remote-containers-0.275.1/dist/spec-node/devContainersSpecCLI.js:2045:17065)
[14998 ms] Exit code 1
[14999 ms] Command failed: /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin) --ms-enable-electron-run-as-node /Users/john/.vscode/extensions/ms-vscode-remote.remote-containers-0.275.1/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/john/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --container-session-data-folder /tmp/devcontainers-14b29ec9-458f-4953-9888-1ba9c74df4941676035088525 --workspace-folder /Users/john/sites/wp --workspace-mount-consistency cached --id-label devcontainer.local_folder=/Users/john/sites/wp --id-label devcontainer.config_file=/Users/john/sites/wp/.devcontainer/devcontainer.json --log-level debug --log-format json --config /Users/john/sites/wp/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[14999 ms] Exit code 1

Environment

• VS Code 1.75.1
• macOS 12.3
• Docker Desktop 4.7.1

[helen]

@johnbillion I'm fine with continuing on this ticket for the moment, I do think making sure it at least works consistently locally is important to keeping this in core. Are you getting any other output? It is coming up for me although it looks like we need to do some conditional setting of the workspace folder, that's where it fails and doesn't fully finish setting up WordPress for me right now.

[johnbillion]

Here's the full terminal log: ​https://gist.github.com/johnbillion/1a51bad2eb523e491cc08582513bd749

There's an error on line 142 in addition to the one I reported above:

 => ERROR [dev_containers_target_stage 3/6] RUN cd /tmp/build-features/co  0.4s
------                                                                          
 > [dev_containers_target_stage 3/6] RUN cd /tmp/build-features/common-utils_1 && chmod +x ./devcontainer-features-install.sh && ./devcontainer-features-install.sh:                                                                            
#11 0.129 ===========================================================================                                                                           
#11 0.129 Feature       : Common Utilities
#11 0.129 Description   : Installs a set of common command line utilities, Oh My Zsh!, and sets up a non-root user.
#11 0.129 Id            : ghcr.io/devcontainers/features/common-utils
#11 0.129 Version       : 2.0.8
#11 0.129 Documentation : https://github.com/devcontainers/features/tree/main/src/common-utils
#11 0.129 Options       :
#11 0.129     INSTALLZSH="true"
#11 0.129     CONFIGUREZSHASDEFAULTSHELL="false"
#11 0.129     INSTALLOHMYZSH="true"
#11 0.129     UPGRADEPACKAGES="true"
#11 0.129     USERNAME="wordpress"
#11 0.129     USERUID="automatic"
#11 0.129     USERGID="automatic"
#11 0.129     NONFREEPACKAGES="false"
#11 0.129 ===========================================================================
#11 0.173 Packages to verify are installed: apt-utils         openssh-client         gnupg2         dirmngr         iproute2         procps         lsof         htop         net-tools         psmisc         curl         tree         wget         rsync         ca-certificates         unzip         bzip2         zip         nano         vim-tiny         less         jq         lsb-release         apt-transport-https         dialog         libc6         libgcc1         libkrb5-3         libgssapi-krb5-2         libicu[0-9][0-9]         liblttng-ust[0-9]         libstdc++6         zlib1g         locales         sudo         ncdu         man-db         strace         manpages         manpages-dev         init-system-helpers libssl1.1 git
#11 0.244 Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
#11 0.305 Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
#11 0.315 Err:1 http://deb.debian.org/debian bullseye InRelease
#11 0.315   At least one invalid signature was encountered.
#11 0.334 Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
#11 0.338 Err:2 http://deb.debian.org/debian-security bullseye-security InRelease
#11 0.338   At least one invalid signature was encountered.
#11 0.360 Err:3 http://deb.debian.org/debian bullseye-updates InRelease
#11 0.360   At least one invalid signature was encountered.
#11 0.361 Reading package lists...
#11 0.366 W: GPG error: http://deb.debian.org/debian bullseye InRelease: At least one invalid signature was encountered.
#11 0.366 E: The repository 'http://deb.debian.org/debian bullseye InRelease' is not signed.
#11 0.366 W: GPG error: http://deb.debian.org/debian-security bullseye-security InRelease: At least one invalid signature was encountered.
#11 0.366 E: The repository 'http://deb.debian.org/debian-security bullseye-security InRelease' is not signed.
#11 0.366 W: GPG error: http://deb.debian.org/debian bullseye-updates InRelease: At least one invalid signature was encountered.
#11 0.366 E: The repository 'http://deb.debian.org/debian bullseye-updates InRelease' is not signed.
#11 0.367 ERROR: Feature "Common Utilities" (ghcr.io/devcontainers/features/common-utils) failed to install! Look at the documentation at https://github.com/devcontainers/features/tree/main/src/common-utils for help troubleshooting this error.

[helen]

@johnbillion Seems like this might be a cache issue (what isn't...), could you try running docker system prune and rebuild the dev container to see if that fixes things? Not ideal but if that's the issue then we can explore a better experience from there.

We've got some other fixes incoming for running on local Docker - don't know if I would consider it a great experience but at least it'll get a contributor up and running if they bravely want to try this instead of wp-env right now.

[helen]

In 55353:

Build/Test Tools: Improve devcontainer setup for non-Codespaces usage

This now runs in local Docker as well. Sets the executable propset to explicitly identify install-tool.sh and setup.sh as executable. Unclear if this will sync via the Git mirror; ideally it would as otherwise the two files show as modified in a Git clone even in a fresh container.

Props samruddhikhandale.
See #57187.

[helen]

On the items I noted in 15:

• The correct permissions made it from the svn propset to Git and then to GitHub correctly! Hooray!
• @craiglpeters confirmed that we cannot use prebuilds effectively because of forking, this is probably worthy of chasing down via another ticket if we keep the devcontainer (i.e. actually close this ticket).
• Turns out this goes into the creation log that Codespaces helpfully prompts you to check out if you want more information, I think that's fine then. Just worth noting in docs.

Going to leave this open over the weekend for WordCamp Asia feedback, and if nothing blocking comes out of there I'll close this as fixed and we can move into details to get it to something that is "fully supported" rather than this beta-esque state.

[johnbillion]

Did this get tested much at WordCamp Asia contributor day?

The devcontainer is working fine for me now, I had to clear out some Docker networking config that was also affecting containers in other projects.

Brain dump items that we can either address here or in follow-up tickets:

1. README.md needs to document the three environment choices now available to devs (Codespaces, local devcontainer, local environment).
2. The devcontainer doesn't use the same environment for WordPress that you get when you follow the "Local development" instructions in the readme. We're going to end up with two confusingly separate but similar environments. Could and should the same environment be used to run WordPress within the devcontainer?
3. setup.sh assumes the workspace directory name is /workspaces/wordpress-develop but the local directory name may differ. The ${localWorkspaceFolderBasename} variable needs to be passed in to that script from devcontainer.json.

[costdev]

This ticket was discussed during the bug scrub. @helen @johnbillion Is there any further work likely to be done on this before Thursday 9th March (6.2 RC1)? Or is this ready to close and the items in comment:23 addressed in follow-up tickets?

Additional props: @mukesh27

[costdev]

This ticket was discussed during the bug scrub. As we're about to enter RC, closing this ticket out for 6.2 and iteration can happen in follow-up tickets. 🙂

[SergeyBiryukov]

Follow-up: #57896