E2E tests, Integrated tests vs Collaboration and Contract tests

测试金字塔

参考以下文章，减少 E2E 测试的比例，Google 的建议：70% unit tests, 20% integration tests, and 10% end-to-end tests.

这样看来， Pact.io 类似的功能相当于 Integration tests 层，可能是目前大多数团队所忽略的部分？ 然而看到下面这篇和系列文章，又说 Integrated Tests are a Scam(坑/骗局)

下面这篇又说 Integration Test 不同于 Integrated Test，所以 20% integration tests 应该是合适的，那么区别在哪儿？

区别可能在评论区 Comments 中提到:

  1. Integration Tests would be a abstract idea. The tests that give you confidence that your systems will work together in production.
  2. Integrated Test is the current default implementation of Integration Test and a scam.
  3. Collaboration Test is an implementation of Integration Test and the approach you recommend.
Intergration <|-- Intergrated       :scam
Intergration <|-- Collaboration     :recommend

这么说 Collaboration Test 是 Integration Tests 的一种实现，我猜 Contract Test 也是吧。

文中有个比喻，正常人刷墙，用滚子刷大块儿的面积，用小刷子处理墙角、窗户边、电源插座等细节部位，用贴纸保护不该刷到的部位，这就是正确的方法，大面积有大面积的处理方式，细节有细节的处理方式。然而错误的方式，就好比你有个朋友刷墙是站在三米开外，提着桶往上泼，无论他泼的技术有多好（世界冠军），也没办法把墙很均匀地刷好，同时还保证不该刷的地方不被泼到。

By putting all the components together and running them in a single test, you’re throwing tests at the system, hoping to cover the whole thing.

作者借这个比喻来说明，把所有组件放在一起的集成测试 Intergrated tests，就好像 就象往墙上泼一样，而使用 Collaboration + Contract tests 互相印证，可以达到用正确工具刷墙一样的效果，再加上充分的 Unit tests 可能才是正确的软件测试思路。

I believe that this results from how utterly distracting integrated tests become over time: riddled with duplicated, mostly-irrelevant details.

我相信，假以时日，那些令人彻底分心的集成测试的结果会变得满是重复和无关紧要的细节。

These details assault our senses and beat our conscious minds into submission. Our eyes glaze over. Before long, we stop paying attention. Over time it becomes easier, not harder, to make a mistake.

这些细节将袭击我们的感官，战胜我们清醒的思维。 我们双眼呆滞。 用不了多久，我们就无法专注了。 一段时间后，就变得更容易（而不是更难）犯错了。

Contract = Semantics + Syntax(Compile check will do this)

I, programmer, write contract tests in order to avoid end-to-end tests. It would satisfy me to replace my end-to-end tests with collaboration and contract tests over time.

My System 作者的思路：

Rather than throw paint at the wall, let me describe what I do, which corresponds to painting the corners of the wall with a brush.

First, I describe the key properties of collaboration and contract tests:

  • A stub in a collaboration test corresponds to the expected result of an assertion in a contract test.
  • An expectation in a collaboration test corresponds to the action of a contract test.
  • These correspondences apply equally well in both directions.
stub ~= assert equals expected result
expectation ~= action

From this, we can extract some helpful rules:

  • When I write a stub in a collaboration test, then I remind myself to also write a contract test where the stubbed return value becomes the expected result of the contract test.
  • When I write a contract test with assert_equals() in it, then I can safely stub that method to return that value in a collaboration test. Moreover, I should probably try to imagine a collaboration test for the client that stubs that method to return that value so as to document what happens in that case. Maybe I need it; maybe I don’t.
  • When a collaboration test expects a method (with certain parameters), then I remind myself to also write a contract test that describes what happens when we invoke that method with those parameters. Any module that implements that method must behave accordingly.
  • When I write a contract test with the action foo(a, b, c) in it, then I can safely write a collaboration test that expects (“mocks”) foo(a, b, c).

Simulating Failure in Collaboration Tests

Limiting beliefs and unstated assumptions interfere with our performance.

有限的信念和未声明的假设会妨碍我们的表现。

Crash Test Dummy pattern: implementing the interface with methods that intentionally raise errors. 崩溃测试模拟模式：实现那个故意产生错误的方法的接口

Knowing where to find experienced people who can answer your questions doesn’t only help you get answers to your questions, but more importantly, it frees you to focus on the details and allow yourself to struggle, but of which are essential for effective learning.

知道去哪儿找有经验的人回答你的问题，不光有助于你得到答案，而且更重要的，它解放了你，让你专注于细节，让你自己去奋斗，而这对于高效学习是必不可少的。

You don’t have to worry both about how to do the thing and whether you’re doing “the right” thing. You can get something working, then take a breath, then ask for advice. We all need help like this from time to time, so it helps to have a safe place to ask these kinds of questions where someone with experience can answer them.

你不必担心怎么做这件事或者你是不是做对了。 你可以先让某些东西工作起来，然后喘口气，问下一步的建议。 我们都时时刻刻需要类似这样的帮助，于是有个安全的地方去问这类（有经验的人能回答的）问题将会是非常有帮助的。

Since integrated tests are still a scam, you might prefer to use Collaboration and Contract Tests to check the integration between the Controller and the Repository.

因为集成测试始终是个陷阱，你可能想去使用 Collaboration 协同测试 和 Contract Tests 契约测试去检查 Controller 控制层和 Repository 仓库层的集成。

由此可见， Collaboration tests + Contract tests => Integration tests 即是整合测试的具体实现。 暂且翻译 Integrated tests 为 集成测试，而 Integration tests 为 整合测试。

文中提到 Test Doubles，这又是什么呢？

Test Doubles 测试替身

闹了半天就是 Mock，不过可能专业名词就是： test doubles

Typically, when you think you want to mock an entity, first try mocking the repository from whence it came.

Services, you can mock with impunity. In fact, the only time not to mock a service is when you’re testing the service.

Put another way, I lean on interaction-based testing to verify how my object talks to its collaborators; but I lean on state-based testing to verify how well that object listens.

Never mock values, sometimes mock entities, but mock services freely.

如果是检查我的对象如何跟其他说话，我偏向基于交互的测试，即 mock； 如果是检查我的对象听得如何，我偏向于基于状态的测试，即 fake/stub。

When I want to rescue legacy code, I reach for Mockito. When I want to design for new features, I reach for JMock.

Verification Tests

  • chadojs is a mocking library for nodejs which reduces the need for integration tests. Instead of integration tests it supports writing verification tests.

The problem with mocks is, that the tests still pass, although the real objects might not work together any more.

There are two ways how you can deal with that.

  • Avoid mocks and use the real object instead. ==> not always possible or desired.
  • Assure that mocks are in sync with the real object. ==> Whenever you mock a unit, you verify that the real object can behave like the mock.

chadojs uses an assume-verify-approach

Microflow in TDD

  1. Write a new failing test.
  2. Notice a refactoring that would make the next step easier.
  3. Ignore/delete the new failing test.
  4. Refactor.
  5. Rewrite the intended failing test—or occasionally a different one!
  6. Continue.