Skip to main content
The 2024 Developer Survey results are live! See the results

Should you test all your project every time a push is made?

Created
Active
Viewed 370 times
11 replies
11

Disclaimer : I know almost nothing about DevOps and have been tasked with implementing CI/CD for this project. My question may be unclear and lacking in detail, so any guidance is appreciated. Please, gently correct me and tell me what I can do to improve my question :)

Question

While configuring the YAML file to automate the tests, a question came to my mind:

Should the test_job run all the tests each time the pipeline is launched? Is there any best practice or convention regarding this?

Context

I am using PHPUnit 10.5 to test my project, which is built with PHP 8.1 and hosted on GitLab. The project is quite large, with a size of about 6.1 GB, using about 200 tables from two separate databases, and is deployed across multiple servers, including a test server and several production servers.

Nothing about continuous integration or continuous deployment has been implemented yet. I am starting from scratch.

I thought the best way to start is by creating some test files, configuring the YAML file, and let some pipelines run so I could learn the basics and have something to start with.

Then I realized how big the project was, and how long it would take to run all the tests. It is simply not an option to run all of them each time a dev push into the repository.

For now, I only have a few tests done so the pipeline only lasts a few minuts. But because of the size of this project, my task is also to configure the pipeline so it only run the necessary tests.

Research Directions

Separating tests into test suites might resolve this problem; however, how can I know which test suite to launch?

I thought about two options:

  1. Mapping-Based Approach:

    • Check the files that have been changed by the commits and use a mapping to know which test suite covers these specific files.

    • The problem is this solution is not dynamic at all, and the pipeline would not be able to launch freshly added files or tests if they're not in the mapping. Additionally, it feels like bad practice to hard-code a mapping into the configuration.

  2. Coverage-Based Approach:

    • Check the files that have been changed and use the coverage information of each test suite to determine which test suite covers the files from the commit.

    • The challenge here is that I have no experience with code coverage tools and am unsure if the effort put into learning them would resolve the problem.

There should be more options and potentially more issues with my current approaches. I would be pleased if you could enlighten me about them.

I believe this problem is common for large projects, but I haven't been able to find resources or guidance to help me. How would you do it ?

  • 39
  • 7

11 replies

Sorted by:
78687365
1
  • 121
  • 1
  • 8

I always run all the pipeline jobs necessary to ensure that the new code works with the existing code when merged. Otherwise, there is no point in developing tests in my eyes.

It may be possible to parallelise the tests, for example with https://github.com/paratestphp/paratest.

But maybe it is also an idea to use micro services to avoid this problem? These can then be tested in individual repositories. monolithic problem
78697724
0
  • 39
  • 7
Author

I did not know about Paratest; it seems very helpful, thanks!

You may be right; it seems pointless to test only the files that have been updated.

However, that is not an option. My team and I do many pushes per hour, and we cannot spend our time waiting for the pipeline to finish.

Microservices could have been a great way to code our project. It would have indeed made the whole testing process way simpler. But it's too late now.

This Paratest extension seems like a great way to accelerate my pipeline. If it speeds it up enough, it may be the solution to my problem.
78716826
0
  • 4.5k
  • 12
  • 40
  • 55

My team and I do many pushes per hour, and we cannot spend our time waiting for the pipeline to finish.

(said by @Kolobo here)

The more complex the infrastructure/project becomes, the more you have to deal with it, imho. In my opinion, this is a common problem that can only be optimized a little and cannot be avoided.
78688333
1
  • 103
  • 6

Here twio things is there, PUSH or MERGE.

It depends on the nature of the code in the project. It's always better to test your code before merge.

You can use github pre-commit and pre-merge-commit hook to invoke the build before push/merge happens. But it will only be able to help you in testing the unit test cases. You will not be able to ensure that your pushed code will work fine in the run time environment.

May be you can use the trank based development appoach for this during the merge.
Once coding is done you can just deploy from the feature branch to test your code and then put it for merge.

We should adhere to the best-practices always.
78697633
0
  • 39
  • 7
Author

My project is hosted on GitLab, not GitHub! :)

It's a good idea; I could test specific bits of code on PUSH and launch all tests on MERGE. Is this what you are advising?

I am already using trunk-based development with my team, and it's indeed a good idea to combine it with this approach of PUSH and MERGE tests.
78697933
0
  • 103
  • 6

You can configure and use Webhook for gitlab for Testcases, refer to this documentation. It will atleast help to check the test cases before merge.

https://docs.gitlab.com/ee/ci/test_cases/index.html
78716148
0
  • 196k
  • 55
  • 444
  • 846

start with the gitlab ci help that ships with your gitlab instance. and there start with the simple ci build, the one with the three stages. of those you take the first two (build and test). in build you checkout from source and then install all dependencies, e.g. if you use composer, you do the composer install, then make the vendor folder (and other files created during the build) the build artifact/s.

every later stage has those artifact/s then.

in the test stage, setup the test environment and execute the tests. start with little or no setup, make use of phpunit bootstrap.php so development environment and test stage can share most - if not all - of the configuration with little overhead and same proceedings.

if you have different test "jobs" with different setup, it is often a good idea to separate them on their own jobs in gitlab ci as well. for starters you assign them all on the test stage. all jobs in the same stage are run in parallel by default. start with the fastest build job, defer the others.

if you later have more complicated setups, you can depend on earlier jobs (acg), this is easy to add later if you start with the very simple, stage based recommendation as outlined within the documentation.

and yes, a pipeline is always / by default for each revision (push) or revision production (merge). if you want to have less information for a revision, you can control what should run when and for whom in Gitlab. but then again, start with the three stage build, add deployment on tagging after test, or a staging deployment to run a test-application managed by gitlab ci (this is also a good training ground for deployments).

finally deploy after tagging.
78770701
1
  • 1.2k
  • 1
  • 10
  • 21

I'm not a CI/CD expert, but I've been playing with it a little in the last years, so I'll add my two cents here.

You'd better have more than one test suite (e.g. unit tests and integration tests) so you can validate the code both individually and acting in the whole system. Unit tests should run very quickly; integration tests can use mock entities (eg. databases, APIs) to reduce the requirements, but in the end you'll also want to test the whole thing altogether.

I've always read that the recommendation is to run unit tests frequently and integration tests occasionally (since they demand more resources), so you can set up your CI/CD pipelines in that way, running unit tests after every push and a thorough integration test after a nightly build. You'll probably want the nightly test to produce a report that is sent to all the responsibles, so if there's any test fail they'll give attention to it in the first work hours. If your CI environment allows for it, the unit test should send a report to the dev that made the push in case of any test failed, so the owner of the push may be notified that the changes he/she just pushed caused an error and can fix it ASAP (although an even better practice would be each dev run the unit test suite locally¹ before pushing, so he/she can avoid pushing code with errors).

¹ In his/her own computer / dev environment.
78777246
0
  • 39
  • 7
Author

Thank you for your comment, I find it very usefull !

However, how do you configure a GitLab pipeline to be launched on a daily basis ? Like, in your example, run the integration tests each night ?

Is it possible, or do we need to use another tool ?
78781361
1
  • 1.2k
  • 1
  • 10
  • 21

Unfortunately, I don't know the specifics of GitLab. Here where I work, we use Azure DevOps, so I can't say to you how you can configure GitLab to work like I suggested.

There's another reply here that suggests you to start with the GitLab CI Help, so you can see if you find anything like I recommended in the tutorials / help, or you can search specifically for that in the net, you'll probably find something to follow.
78771106
1
  • 81
  • 5

Crowdstrike should be taking notes