CI/CD options such as bitbucket pipelines and github actions make use of virtual machines called runners. Changes in source code trigger the runners, which in turn trigger a set of commands. The commands may include instructions to run a test suite. The test suite checks whether source code changes break existing functionality.
Contributors often develop tests on their local machines. Tests that succeed on one platform do not necessarily succeed on another platform. This means that tests running on a remote machine can (and often do) behave unpredictably as compared with how they behave under local machine conditions. For example, a test that relies on Windows-style line returns may fail if it runs under a Linux operating system.
There are a few ways to circumvent this problem:
- Make test assertions platform-independent. In the above example, this might mean replacing every newline character with an empty string.
- Replicate the runner conditions on the local machine using containerization.
- Accept failing tests in the local machine
Only the second option seems viable, however I have seen very little to no support for this in the documentation of most CI/CD tools. This leads me to believe that I might be missing something. For the professional software developers/testers out there, has anyone run into this problem? What is the correct way to address it?