A few months ago, Planet Argon kicked off a Rails 2.1 to Rails 3.0 / Ruby 1.8.7 to Ruby 1.9.3 upgrade for one of our clients. The monolithic Rails app contained over 1 million lines of code and accumulated over 9,000 RSpec tests throughout its decade of development. Once the test suite was running on the new version, 8,000 of these specs failed.
This blog post is a collection of my lessons learned while tackling this upgrade in hopes you may be better prepared to fearlessly face 1000 failing specs. We’ll cover what to do when facing 1000 failing specs, and then move on to learning what to do to prevent 1000 failing specs in the future.
When the 1000 failing specs are already there
Ok, so you’ve upgraded your Gemfile, finished your first successfully failing RSpec run and see 1,000 or so failed specs. Now what?
1. Strategize where you start
Breaking the work into manageable chunks will keep your process calm, organized, and productive. I took different approaches based on the kinds of failures left. I preferred to focus first on fixing errors with similar messages, then fixing all errors related to a functional part of the app.
Strategy A: Fix Errors with Similar Messages
Once the test suite runs all the way through, you’ll likely see a lot of similar messages or error types. ActionController::RoutingError, ActiveRecord::RecordNotSaved, undefined local variable or method.
Read the failure output and begin to familiarize yourself with common error messages in your test run. Outputting the test run into a file using RSpec's built-in --output method is quite helpful here.
# outputs the run of "example_spec" to a file called rspec.txt on your computer
$ rspec example_spec.rb --format documentation --out rspec.txt
Since there are so many failures, it’s not important to pick the one occurring at the highest frequency. Just pick one you see a lot. Or one that you feel confident you could fix. Focusing on fixing that one kind of error in as many places as you can will start whittling the errors down.
When you feel confident it's resolved, conduct another full suite run. Search for any references to that error message. When there's zero results, move onto the next one.
This strategy will become less useful as the total errors diminish. At that point, I recommend focusing on fixing specs that share Objects.
Strategy B: Fix all specs related to the same Object
Let's say you have a User model. Locate all the spec files that relate to Users (ex. User models, User requests, User controllers, User features, User reports, etc.). Fix those spec files. Perhaps even clean up those files as you go by removing old TODOs or updating organization to reflect current best practices. Laying to rest a particular part of the app clears precious mental space to narrow down where the stickiest problems remain.
2. Specs aren’t resolved 1:1
Fixing that AbstractController::ActionNotFound error doesn’t always mean that the number of passing specs increased.
Resolving some specs may take a few extra rounds. Your hours of hard work allowed RSpec to evaluate the next line in the stack, that could have an issue with it too. Especially early on, keep the mantra of “get to the next error message”, close to your heart. Eventually, you’ll weed through the stack and start seeing green, but finding success in the meantime will be paramount to maintaining morale.
3. Separate WIPs into a child-branch and compare the pass rate with a parent-branch
Just because your code change made a miraculous difference in one spec file, doesn’t mean that 10 errors weren't reopened in another part of your app.
To prevent this tail-chasing conundrum, work on your manageable chunks in child-branches that get merged into a parent-branch. Using the parent-branch test run as the standard, you can see if child-branch changes break specs that were already fixed.
4. Be flexible in your focus (or take mini-project breaks)
In the upgrade I mentioned at the beginning, I ran into a big problem -- Locales completely changed between the versions of Rails. At first, I commented out the code, wrote a TODO, and moved onto a different error. Eventually, I acknowledged this patch-method was too big of a blocker to continue making meaningful progress in other parts of the app.
You could think of it like a vacation. Immerse yourself in totally upgrading this entire part of the app. Understand how it changed and why. Test it in the browser. Minimize your attention on the specs. The other failures will be waiting for you once you’re finished (though some may happily resolve themselves in the process.)
5. Document what works
That feeling of, “I think I saw something like this last week…” can be satiated with good notes.
Keep a running log of successful syntax changes. Help your future self find the answers. You can also use this log to quell feelings of doubt during the upgrade journey by reviewing all you've learned along the way.
Bonus: Share notes with the team post-upgrade to help them get up to speed on new syntax and patterns required by the version change.
6. Keep an open mind of what might be causing the failure you face
Failed Specs arise from a number of sources:
- Ruby version changes
- Rails version changes
- Gem version changes (especially RSpec)
Each version change is usually accompanied by a well-documented explanation of the shake-ups between old and new. They also usually have APIs with detailed explanations of which methods exist in different versions of the framework. Especially if you take the bump-everything-up-at-once route, the error could come from any of these sources.
If you're feeling stuck, ask yourself:
- Does the method this spec relies upon still exist in the new version?
- Does the method do something drastically different than it used to?
- Is there a different process for performing this, or a similar, action in the new versions?
Have the Changelog, Upgrade Guide, API, and READMEs close at hand to evaluate whether this is something that should be expected based on the code, or something that broke because of the tests evaluating the code.
Prevent seeing 1000 failing specs in the first place
Preparing your Rails app for an upgrade will reduce the number of failing specs you have to deal with. You may prevent seeing 1000 failing specs in the first place by following these tips.
1. Upgrade dependencies as close to the version you'll use for the Rails upgrade as you can
If a version change is required to keep your gems compatible with the upgraded version of Rails, you may run into spec failures due to syntax changes between the versions. Even if you can't upgrade to the same version you'll use once you've upgraded, pushing the version higher in your current not-yet-upgraded app could introduce helpful deprecation warnings to adjust code before it breaks. It’s also not always possible, and that’s okay too.
Test out one change at a time. Change the gem version, run the test suite, fix what fails, and move on to the next gem.
The main gems I’d recommend updating before upgrading would be anything related to testing, such as RSpec, FactoryBot, Capybara, etc. You’ll save yourself the hassle of wondering if a failure is due to something that changed in Rails or something that changed in RSpec.
2. Fix deprecation warnings
We hate them. We ignore them. We become blind to them. Now is not the time – pay attention before you make that version switch to stave off unnecessary errors that would’ve been resolved by taking the advice of the maintainers.
3. Make sure your test coverage is robust
If you’re reading this post, your app probably has 1,000+ specs. However, I wanted to share this:
Test coverage is a great way to evaluate whether your application is functioning as expected during and after an upgrade. However, having a lot of specs doesn’t necessarily mean they’re evaluating the most important stuff in your app.
Use a test coverage evaluation tool, such as SimpleCov, to determine if the app you’re about to upgrade has a high percentage of coverage in key areas. If the coverage isn’t there, you may want to write some new specs before starting the upgrade.
4. Follow the Changelog and Upgrade Guides
Those failures before you get to the failures; the RSpec errors preventing the suite from even running? They're probably related to fixing your config files to accommodate the new changes in Rails. Follow the official Rails upgrade guides to move on to the good stuff.
Maybe there isn’t time for you to spend on fixing all of these specific items. Often, upgrades are done under a time crunch. That’s totally fine. These are hindsight tips I wished I had done as I was working through the last big upgrade I worked on.
Our process involved bumping up the whole Gemfile, and once we got a successful bundle, diving into the failed specs. We were able to get a ~10,000 spec project back to green without the mindful adjustment of dependencies ahead of time. We even simultaneously upgraded Ruby and Rails. It still worked out. I don't know how morale or efficiency could have transformed had we applied these ideas.
Conclusion
Dealing with 1,000+ failing specs on a Rails app is no easy task, and it's just one part of your larger upgrade goal.
An upgrade is like a marathon. You’re not going to finish by beating yourself up or burning yourself out. You need to take care of yourself to achieve your goal.
Like any endurance activity, your quick successes will slow down. Where one change might fix 100 specs, as the upgrade goes on, you may need to spend a few days fixing a single spec. Don’t get down on yourself. That’s just how this works.
It took 27 seconds for the test suite to start running on the upgrade project that sparked this article. During those seconds, I started closing my eyes, taking three deep breaths and then watching the tests run. Find moments to recharge while you work.
Sticking to small goals and healthy habits will keep you sane. Bargaining with yourself to solve one more spec before taking a break probably won’t.
Green suites will come, don't lose hope. Persevere! Take care of yourself. Good luck!
.png)