Rails 5.0 changed the default order of its test suites from
:sorted to :random. This change is meant to encourage best practices
and ensure your tests don’t have order dependencies, which could hide bugs. You
can see part of this behavior in the test suite output:
~$ bin/rails test
Run options: --seed 44656
# Running:
.....
Finished in 0.027476s, 36.3952 runs/s, 72.9237 assertions/s.
5 runs, 12 assertions, 0 failures, 0 errors, 0 skips
When running tests, Rails generates a seed and uses it to randomly order the tests. If you want to re-run the tests in the same order, you can pass a seed value to the test command and Rails will use that as the seed:
~$ bin/rails test TESTOPTS=--seed 44646
Tests are always run in the same order for a given seed. But how does that work? Let’s dig through the call stack to find where the test ordering happens.
Under the hood, Rails uses the Minitest gem as the test framework. The overall call stack looks like this:
~$ bin/rails test
Rails::TestUnit::Runner.rake_run
Rails::TestUnit::Runner.run
Minitest.autorun
Minitest.run(args)
Within the Minitest::run definition, the command line arguments from the rake
task invocation get processed:
desc = "Sets random seed. Also via env. Eg: SEED=n rake"
opts.on "-s", "--seed SEED", Integer, desc do |m|
options[:seed] = m.to_i
end
A bit later, options[:seed] is passed to a function I’ve never seen
before, srand:
srand options[:seed]
Here’s what the documentation says about that method:
Seeds the system pseudo-random number generator,
Random::DEFAULT, withnumber. The previous seed value is returned.If number is omitted, seeds the generator using a source of entropy provided by the operating system, if available (
/dev/urandomon Unix systems or the RSA cryptographic provider on Windows), which is then combined with the time, the process id, and a sequence number.
srandmay be used to ensure repeatable sequences of pseudo-random numbers between different runs of the program. By setting the seed to a known value, programs can be made deterministic during testing.
That last paragraph is key: We can make random things deterministic by setting
the seed to an arbitrary value. Let’s dig just a little further and take a look
at how Minitest is randomizing test orders. That appears to happen
here, where we call shuffle on the array of runnable tests. Let’s
try that method in an IRB session along with some srand calls to test the
behavior:
>> array = %w(Lorem ipsum dolor sit amet consectetur)
=> ["Lorem", "ipsum", "dolor", "sit", "amet", "consectetur"]
>> array.shuffle
=> ["consectetur", "Lorem", "sit", "amet", "dolor", "ipsum"]
>> array.shuffle
=> ["Lorem", "ipsum", "dolor", "consectetur", "sit", "amet"]
>> srand 42
=> 42
>> array.shuffle
=> ["Lorem", "ipsum", "consectetur", "dolor", "amet", "sit"]
>> array.shuffle
=> ["sit", "Lorem", "ipsum", "dolor", "consectetur", "amet"]
>> srand 42
=> 42
>> array.shuffle
=> ["Lorem", "ipsum", "consectetur", "dolor", "amet", "sit"]
>> array.shuffle
=> ["sit", "Lorem", "ipsum", "dolor", "consectetur", "amet"]
As expected, Array#shuffle randomizes the order. If we call srand with a
specific value, we can get repeatable orders out of a function meant to
randomize the array items.
If my previous post about the PHP Internals inspired you to dig a bit deeper, you can check out the C implementation of the Array shuffle methods here: