So a couple of my tests were failing the daily tests for several days in a row. The tests run fine in my vagrant environment and CI but not in beta. The first step with fixing this was to skip the failing tests with ‘it.skip’ for each test that was failing, as shown here.
The next step was to target the beta cluster (because this is where it was failing) when running the tests in order to see what exactly was failing. To do this I had to set MW_SERVER=https://en.wikipedia.beta.wmflabs.org , as well as change the values for MEDIAWIKI_USER and MEDIAWIKI_PASSWORD to my test account credentials in beta. The whole command looked like this when I was targeting the failing ‘welcome’ test:
MW_SERVER=https://en.wikipedia.beta.wmflabs.org MEDIAWIKI_USER=testaccount MEDIAWIKI_PASSWORD=testpassword npm run selenium-test -- --spec tests/selenium/specs/echo.js --mochaOpts.grep welcome
When I ran both failing tests I noticed that captcha was required when logging in so it was at this point that the tests were failing since both required a new user to log in and stopped at the captcha point.
I created a new patch that tagged the working tests with @daily so that only the working tests would run daily on beta cluster. I removed the ‘skip’ from the failing tests so that they would resume running in CI but not run in daily (since they were not tagged @daily). The task has been resolved this way for now and I learned more about the slight differences between environments and how to make the tests work given the differences.
