10 Strategies for Upgrading Ruby or Rails Applications with Low Test Coverage
Are you considering an upgrade for your Rails or Ruby application, but you’re concerned about low test coverage? Don’t worry! In this post, we’ll explore effective strategies to address the risks associated with low test coverage. By implementing these strategies, you’ll be able to upgrade your application with confidence. Let’s dive in!
TLDR;
Upgrade your Ruby or Rails application confidently, even with low test coverage, by implementing these strategies:
- Define a Critical Path/Smoke Test
- Utilize Bug Tracking Software
- Utilize a Staging Environment for PRs
- Gradually Increase Test Coverage
- Embrace Small, Specific PRs
- Squash PRs into a Single Commit
- Merge Frequently and Strategically
- Only merge PRs that pass CI/CD
- Revert quickly
- Use Feature Flag Effectively
Intro
In an ideal world, our applications would boast high-quality tests, providing us with comprehensive coverage of at least 80% of our codebase. However, the reality is that successful applications are often shipped with low test coverage. While this is not ideal, it doesn’t mean we can’t take steps to mitigate the risks associated with it.
Low test coverage undeniably amplifies the risks of encountering bugs and experiencing downtime during a Ruby or Rails upgrade. Additionally, the absence of tests significantly prolongs the time required to identify and update incompatible components of our application.
By acknowledging these challenges, we can proactively tackle them and minimize the impact on our upgrade process.
We wrote in the past about the steps you could take to upgrade a Rails application without a test suite . In this post we would like to provide you with additional precautions you could take to ensure a smooth upgrade process.
Strategies
Define a Critical Path
To mitigate the risks associated with low test coverage, the first step is to define a critical path for our application. This critical path encompasses the most common parts of the app that an average user would interact with.
The specific areas covered by the critical path will vary from project to project. When defining the critical path, there are several important considerations to keep in mind:
- Simplicity: Keep the critical path as simple as possible since it needs to be performed for every pull request (PR) before it can be merged in.
- Inclusion of Essential Steps: Ensure that the critical path include all essential steps such as user signup and payment processing. We must avoid merging any changes that would prevent users from making payments, as this directly impacts the business.
- Documentation and Understanding: Clearly document and communicate the critical path to anyone responsible for performing the tests. It should be well understood by all involved parties.
- Tracking Files and Directories: Keep track of the files and directories that affect the critical path. Files on the critical path are important and should be flagged in PRs that modify them.
We higly recommend utilizing a tool like Capybara to conduct critical path tests for your application. By gradually converting manual tests into automated Capybara tests, we can streamline our testing process. This minimal time investment enables us to steadily reduce the burden placed on the manual QA process.
By defining and adhering to a well-documented critical path, we can effectively test and validate the crucial functionalities of our application, reducing the risks associated with low test coverage.
Utilize Bug Tracking Software
To ensure a successful Ruby or Rails upgrade with low test coverage, it is crucial to have reliable bug tracking software in place. While our critical path and QA process cover significant portions of our application, they cannot catch every possible bug.
While we strive to avoid pushing bugs to production that would negatively impact our users, it is inevitable that some bugs may slip through. Bug tracking software plays a vital role in capturing exceptions that occur in the production environment. It provides valuable information, such as stack traces, which help us understand the conditions that led to the exception.
With the insights provided by the bug tracking software, we can replicate the bug in a development environment. Once we successfully replicate the issue, we can implement the necessary changes to fix it. During this stage, we may consider adding additional tests or modifying our QA process to detect similar bugs in the future.
Bug tracking software, such as Sentry , Honeybadger , Bugsnag or Rollbar , serves as a powerful tool for identifying and resolving issues that arise in production. It enables us to gather crucial information, debug effectively, and enhance the overall stability and reliability of our application.
Utilize a Staging Environment for PRs
One of the key reasons for defining a critical path is to reduce the time and effort required to review a PR before it is merged into production. A vital step in any robust review process is Quality Assurance (QA). It is essential to consider our QA resources when defining the critical path. A well-defined critical path instills confidence in proceeding with a PR and deploying it to production.
To effectively conduct the QA steps outlined by our critical path, it is crucial to have a staging environment that closely resembles our production environment. This environment should be readily available for engineers or QA personnel to quickly spin up, enabling them to test the critical path with minimal effort and time.
The value of having a critical path is undermined if we do not perform QA on every PR that is merged. By diligently executing QA processes, we are rewarded each time we identify and rectify bugs before they reach the production environment. This practice ensures a higher level of reliability and stability for our application.
Gradually Increase Test Coverage
To address the challenge of low test coverage, we can gradually increase our testing efforts. The focus should be on adding minimal tests that raise exceptions when critical parts of the application break. While we don’t need to create a test for every single file we modify, it is crucial to prioritize testing the components identified as part of the critical path.
When adding tests, it is advisable to minimize the use of mocks. Mocking can lead to passing tests that do not accurately reflect the behavior of our code in a production environment.
The following example demonstrates how a simple test can be added to catch raised exceptions . The goal is to include minimal tests to capture important functionality without letting the scope of the upgrade project become overwhelming.
# Example test to catch exceptions
describe "Critical Path Testing" do
it "raises an exception when critical functionality breaks" do
expect { your_code_here }.not_to raise_error
end
end
By gradually increasing test coverage, focusing on critical areas, and avoiding excessive mocking, we can improve the reliability and stability of our application during the upgrade process without overwhelming the project scope.
Embrace Small, Specific PRs
Each PR represents a snapshot of our application at a specific point in time. A well-crafted PR title and description enable us to quickly understand the state of the app at that particular moment.
As with any software, bugs can surface in production. When a bug arises, our priority is to restore the application to a functional state. Often, the quickest way to achieve this is by rolling back to a previous commit or reverting the offending commits.
Having smaller PRs allows us to pinpoint and roll back the application to a very specific state. It becomes easier to identify where a bug was introduced when the PRs are smaller in scope.
The benefits of small PRs are numerous. They are easier and faster to review, and the likelihood of encountering merge conflicts is reduced. Fixing merge conflicts in small PRs is undoubtedly simpler compared to larger ones.
By adopting a practice of submitting small, specific PRs, we enhance the efficiency of code reviews, minimize merge conflicts, and enable better bug tracking and resolution. This approach leads to a smoother and more manageable upgrade process.
Squash PRs into a Single Commit
Small, frequent commits offer numerous advantages, including the ability to reset our work to a previously stable state quickly. However, when we make these small commits, it is common for our PRs to end up with multiple commits that spread the intended change across various commits.
To facilitate easier reverting of changes introduced by a PR, it is beneficial to squash the PR into a single commit. This consolidation simplifies the process of rolling back the changes, particularly when dealing with low test coverage.
Merge Frequently and Strategically
Merging changes frequently offers several advantages for the team and the overall development process. When we merge often, it becomes easier for the rest of the team to work on top of the most recent changes. Additionally, this approach ensures that any bugs that slip past the QA phase are discovered promptly, rather than weeks or months later when the specifics of a particular PR might have been forgotten.
Moreover, the effectiveness of bug tracking software is greatly enhanced when more people use our application. By deploying changes to production sooner, we can fully utilize and benefit from our bug tracking software, enabling efficient bug tracking and resolution.
During an upgrade process, various parts of the application are improved, dependencies are upgraded, and test coverage often increases. By merging frequently, we enable the entire team and user base to benefit from these incremental steps.
Only merge PRs that pass CI/CD
Even with low test coverage, it is crucial to leverage the benefits of a continuous integration (CI) pipeline as part of our development workflow.
By enforcing the requirement for PRs to pass the CI/CD process before merging, we ensure that only code that adheres to our coding standards and does not introduce changes that cause test failures is merged into our codebase.
Even though our test coverage may be limited, leveraging CI/CD ensures that the code we merge is validated to the best extent possible within our existing testing framework. This practice minimizes the chances of introducing errors and helps maintain the overall integrity of our application.
Revert quickly
Not all bugs in production are created equal, and some may have more severe consequences than others. By reverting changes quickly, we can minimize the negative impact of a deployed bug.
To ensure we are prepared to revert a merged PR swiftly, one approach is to create a new PR that includes the revert commit. This way, the revert PR is ready to be merged if the need arises. If we determine that the application is stable and the revert is not required, we can simply close the revert PR. This proactive approach allows us to be prepared for any potential issues while maintaining flexibility based on the stability of the application.
While it is not necessary to add revert PRs for every merge, it is advisable to consider them for significant changes or modifications that directly impact our critical path.
In certain situations you might need to urgently revert without the luxury of creating a revert pull request. In such cases, you can rely on your deployment infrastructure to deploy a specific historical commit. For instance, utilizing Heroku’s releases and rollbacks functionality empowers you to quickly deploy or rollback to a previous working version of your application.
By being prepared to revert changes promptly when needed, we demonstrate agility in addressing production issues and minimize any adverse effects on our application and users.
Use Feature Flag Effectively
Feature flags provide a powerful mechanism to introduce conditionality into our application, allowing us to selectively enable or disable specific features. This concept is particularly useful when upgrading a Rails application.
While we strive to merge as many backward-compatible changes into production as possible, an alternative approach is to encapsulate all upgrade-related code behind feature flags. However, this alternative introduces a significant number of conditional statements that need to be maintained.
It is generally preferred to minimize the usage of feature flags and keep changes directly deployable to production. This approach ensures that the code we deploy has undergone proper testing and validation, reducing the risk of hidden bugs that may only surface once the feature flags are removed or toggled on.
By judiciously implementing feature flags and striking a balance between their usage and direct deployment, we can effectively manage the complexity of conditional code, maintain code quality, and ensure a reliable production environment.
Conclusion
In conclusion, upgrading a Ruby or Rails application with low test coverage may present challenges, but it should not deter you from moving forward. By implementing the strategies we discussed you can mitigate the risks associated with low test coverage and proceed with confidence.
Defining a critical path or smoke test that encompasses essential user interactions is a crucial initial step. By leveraging CI, bug tracking software, a reliable staging environment, and gradually increasing test coverage, we can effectively identify and address bugs during the upgrade process.
To facilitate a smoother upgrade process, it is important to adopt small, focused PRs that can be quickly reverted if needed, along with well-selected feature flags to manage the rollout of new features or changes. These practices enhance the overall efficiency and effectiveness of the upgrade process.
If you need any assistance or guidance with your Ruby or Rails upgrade project, please don’t hesitate to reach out to us . We understand that low test coverage can be a concern, but we hope that this blog post has provided you with the confidence to proceed. Upgrading your application is an important step, and with the right strategies in place, you can navigate the process successfully.