627.md

# Flaky tests

> 原文：[https://docs.gitlab.com/ee/development/testing_guide/flaky_tests.html](https://docs.gitlab.com/ee/development/testing_guide/flaky_tests.html)

*   [What’s a flaky test?](#whats-a-flaky-test)
*   [Quarantined tests](#quarantined-tests)
    *   [Quarantine tests on the CI](#quarantine-tests-on-the-ci)
*   [Automatic retries and flaky tests detection](#automatic-retries-and-flaky-tests-detection)
*   [Problems we had in the past at GitLab](#problems-we-had-in-the-past-at-gitlab)
    *   [Time-sensitive flaky tests](#time-sensitive-flaky-tests)
    *   [Array order expectation](#array-order-expectation)
    *   [Feature tests](#feature-tests)
        *   [Capybara viewport size related issues](#capybara-viewport-size-related-issues)
        *   [Capybara JS driver related issues](#capybara-js-driver-related-issues)
        *   [PhantomJS / WebKit related issues](#phantomjs--webkit-related-issues)
        *   [Capybara expectation times out](#capybara-expectation-times-out)
*   [Resources](#resources)

# Flaky tests[](#flaky-tests "Permalink")

## What’s a flaky test?[](#whats-a-flaky-test "Permalink")

这个测试有时会失败，但是如果您重试了多次，它最终会通过.

## Quarantined tests[](#quarantined-tests "Permalink")

当`master`的测试频繁失败时，应创建[〜" master：broken"问题](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) . 如果无法及时修复测试，则会影响所有开发人员的工作效率，因此应通过分配`:quarantine`元数据将其放入隔离`:quarantine` .

这意味着除非使用`--tag quarantine`运行，否则它将被跳过：

```
bin/rspec --tag quarantine 
```

**在对隔离进行测试之前，应确保存在〜" master：broken"问题，这样它才不会永远留在隔离区.**

Once a test is in quarantine, there are 3 choices:

*   测试应该固定（即摆脱其脆弱性）吗？
*   是否应将测试移至较低级别的测试？
*   是否应该完全删除该测试（例如，因为已经有一个较低级别的测试，或者它正在复制另一个相同级别的测试，或者它测试的太多等等）？

### Quarantine tests on the CI[](#quarantine-tests-on-the-ci "Permalink")

隔离测试在允许失败的专用作业中在 CI 上运行：

*   `rspec-pg-quarantine` （CE 和 EE）
*   `rspec-pg-quarantine-ee` （仅适用于 EE）

## Automatic retries and flaky tests detection[](#automatic-retries-and-flaky-tests-detection "Permalink")

在 CI 上，我们使用[RSpec :: Retry](https://github.com/NoRedInk/rspec-retry)自动重试失败的示例几次（有关准确的重试次数，请参阅[`spec/spec_helper.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/spec_helper.rb) ）.

我们还使用自制的`RspecFlaky::Listener`侦听`RspecFlaky::Listener` ，该`RspecFlaky::Listener`将`RspecFlaky::Listener`示例记录在`master`上的 JSON 报告文件中（ `retrieve-tests-metadata`和`update-tests-metadata`作业）.

这最初是在以下[网址中](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021)实现的： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021) : [//gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021) .

如果要在本地启用重试，则可以使用`RETRIES`环境变量. 例如`RETRIES=1 bin/rspec ...`将重试失败的示例一次.

## Problems we had in the past at GitLab[](#problems-we-had-in-the-past-at-gitlab "Permalink")

*   [`rspec-retry` is biting us when some API specs fail](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29242): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9825](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9825)
*   [Sporadic RSpec failures due to `PG::UniqueViolation`](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28307#note_24958837): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9846](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9846)
    *   Follow-up: [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10688](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10688)
    *   [Capybara.reset_session! should be called before requests are blocked](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33779): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12224](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12224)
*   FFaker 生成测试尚未准备就绪的时髦数据（测试应该是可预测的，所以很糟糕！）：
    *   [使`spec/mailers/notify_spec.rb`更加健壮](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/20121) ： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10015) : [`spec/mailers/notify_spec.rb`](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/20121)
    *   [`spec/requests/api/commits_spec.rb`暂时失败](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/27988#note_25342521) ： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9944) : [`spec/requests/api/commits_spec.rb`](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/27988#note_25342521)
    *   [用序列替换 FFaker 工厂数据](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29643) ： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10184) : [//gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10184](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29643)
    *   [spec / finders / issues_finder_spec.rb 中的暂时失败](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30211#note_26707685) ： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10404) : [//gitlab.com/gitlab-org/gitlab-foss/-//merge_requests/10404](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30211#note_26707685)

### Time-sensitive flaky tests[](#time-sensitive-flaky-tests "Permalink")

*   [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10046](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10046)
*   [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10306](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10306)

### Array order expectation[](#array-order-expectation "Permalink")

*   [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10148](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10148)

### Feature tests[](#feature-tests "Permalink")

*   [Be sure to create all the data the test need before starting exercise](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/32622#note_31128195): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12059](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12059)
*   [Bis](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34609#note_34048715): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12604](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12604)
*   [Bis](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34698#note_34276286): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12664](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12664)
*   [Assert against the underlying database state instead of against a page’s content](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/31437): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10934](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10934)
*   在 JS 测试中，当元素在 Capybara 发送点击的确切时间移动时，移动元素会导致 Capybara 误点击
    *   [由于窗口大小和滚动位置，下拉菜单向上或向下渲染](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17660)
    *   [延迟加载的图像可能导致水豚误点击](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18713)
*   [Triggering JS events before the event handlers are set up](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18742)
*   [Wait for the image to be lazy-loaded when asserting on a Markdown image’s `src` attribute](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25408)

#### Capybara viewport size related issues[](#capybara-viewport-size-related-issues "Permalink")

*   [Transient failure of spec/features/issues/filtered_search/filter_issues_spec.rb](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29241#note_26743936): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10411](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10411)

#### Capybara JS driver related issues[](#capybara-js-driver-related-issues "Permalink")

*   [Don’t wait for AJAX when no AJAX request is fired](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30461): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10454](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10454)
*   [Bis](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34647): [https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12626](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12626)

#### PhantomJS / WebKit related issues[](#phantomjs--webkit-related-issues "Permalink")

*   记忆在屋顶！ （TL; DR：加载图像但阻止图像请求！）： [https](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003) : [//gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003)

#### Capybara expectation times out[](#capybara-expectation-times-out "Permalink")

*   [Test imports a project (via Sidekiq) that is growing over time, leading to timeouts when the import takes longer than 60 seconds](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22599)

## Resources[](#resources "Permalink")

*   [Flaky Tests: Are You Sure You Want to Rerun Them?](https://semaphoreci.com/blog/2017/04/20/flaky-tests.html)
*   [How to Deal With and Eliminate Flaky Tests](https://semaphoreci.com/community/tutorials/how-to-deal-with-and-eliminate-flaky-tests)
*   [Tips on Treating Flakiness in your Rails Test Suite](https://semaphoreci.com/blog/2017/08/03/tips-on-treating-flakiness-in-your-test-suite.html)
*   [‘Flaky’ tests: a short story](https://www.ombulabs.com/blog/rspec/continuous-integration/how-to-track-down-a-flaky-test.html)
*   [Using Insights to Discover Flaky, Slow, and Failed Tests](https://s0circleci0com.icopy.site/blog/using-insights-to-discover-flaky-slow-and-failed-tests/)

* * *

[Return to Testing documentation](index.html)