Nebojša Stričević wrote this on October 10, 2012

Cleaning up after before(:all) blocks

Recently we added a project that is active since 2007 to Semaphore. Quickly after that, some RSpec scenarios started failing randomly.

After quick inspection we saw a number of before(:all) blocks that were not cleaning the database after itself.

The problem

In a typical scenario, test examples are run in a transaction. The problem lies in the fact that data created in a before(:all) block is NOT rolled back after tests are done (RSpec/Transactions).

Every before(:all) block should have an after(:all) block that will clean a database. But, sometimes you don’t really know all the records that are created in a before(:all) block.

The project was using FactoryGirl to create records in tests. And with FactoryGirl’s support for creating associated object, it’s easy to create few more records that you are not aware of. Although that is a sign of a smell in your tests, sometimes you need a quick fix to continue your work.

Quick fix

The quick fix that we tried was to substitute all before(:all) blocks with before(:each) blocks that are run in a transaction. The fix worked and there were no more randomly failing tests, but time to run the test suite increased significantly.

The real solution

If you use Database Cleaner you already have the tool that can save your day. A usefull feature of Database Cleaner allows you to clean a database with a strategy that you choose:

after(:all) do
  DatabaseCleaner.clean_with(:truncation)
end

This will clean the database with truncation strategy. You should experiment with different strategies to find the one that works best for you.

In this case it was easy to blame Semaphore for failing tests, but the thing is that developers rarely run the whole test suite since that took around 30 minutes to complete. So after you add a project to a CI service you should expect that your test suite will be executed a number of times each day, instead once or twice a week. In this kind of scenario, you should expect that all deficiencies of the test suite will quickly be revealed, and it’s worth taking some time to optimize.