Principal

The idea is to have documentation generated from the code so that it reflects the current behavior of the application.

We can then validate, after proofreading, that what is written corresponds to the expectations of the product.

These documents become the reference on which we will rely to detect regressions. Any change will be considered a failed test.

Another way to test

With this approach, we have to change the way we test software. We need to think more about documentation than about tests. We just have to make visible how the software behave whatever this behavior. Secondly, we will decide if this behavior is the one we expect.

What is a test for

A test is done to ensure the behavior of the application. To do that, we just have to check that a sequence of actions gives the expected result.

In a generated document from the code, if all the information describing this behavior and the result appears, we are able to validate or not the application.

As long as the application produce the same validated document, that means that the behavior is preserved. If we need a human to certify the correctness of the first document version, the next checks can be done automatically by comparing with the original version.

What are the advantages

First of all, we have created automated tests. They are different from those we are used to, but they do the same job. If we just want this automation, there is almost no overhead to write them comparing to classical ones.

Once this done, we have a full documentation of cases we have tested. It could be just a list of title with some output at the beginning but too often no one knows what automated tests are being performed.

With a little work, we can make the document produced more explicit. It can then be read by anyone and be always synchronized with the code. What is written is the truth of the application.

There are no constraints on the produced document. We can therefore organize it as we wish so that it is as understandable as possible. If it can help, we can generate or include diagram, pictures, other documents or anything that can improve understanding.

As the documents are generated, it is possible to produce several versions for different uses. It’s very easy to reuse some parts without needing to maintain each of them. For example, we can have a document with the standard use cases of the main functionalities and another one with all details and special cases.

Compare with other approches

  • Living documentation: It’s a living documentation with in addition the assurance that there is no change without control.

  • BDD: It’s a way to do BDD. Often we automated features created using Gherkin syntax. This is very restrictive because it’s instructions to the machine. Here, we are free to produce the document we need. This gives more possibilities to write a readable documentation.

  • Golden Master (or Approval): We use the golden master approach but reference files are not just data. We can understand each case and validate them more easily when functional change.

  • TDD: It’s not really made for TDD because it can slow down the rytme if we want to produce very readable documentation. But, at the end, our test are not only a technical code lost in the sources but they provides a real documentation.

Frequently Asked Questions

Are the tests more complicated to write ?

If you don’t need an advanced presentation, the test is not more complicated. It can be even simpler in some cases. What can be difficult is to define which information we need to provide. It’s another way of thinking tests.

Of course, a complex rendering requires an extra effort, but are we comparing the same thing ? If we do that, it’s because the result brings more value than the version with classical tests.

But finally, a person need to validate the document ?

Yes, as you need to have someone to choose input data and expected values in your tests. Once the document is approved there is no need to validate it again until it change.

Why asciidoc ?

In fact, we can use any text format (and even other). Markup language is very close to the text displayed so, it’s very easy to compare two versions of the document. Asciidoc is just one of them that provide enough capabilities to create a full documentation (tables, links, includes, code, attributes, …​).

See: asciidoc-vs-markdown

What are the impacts on performance ?

Details

It’s right that there is an impact on performance but the main reason is the use of parsing to extract information from code mmore than the file writing.

The other part that takes time is the HTML generation from adoc files. It’s needed to publish the doc that can be done on a continuous integration and not in the development phase.

You can find in demos a performance evaluation that show a significant degradation on fast tests.