3 Sanity Checks You Should Conduct Before Choosing A Ratio For Your Analysis
I ❤ ratios.
A correctly defined ratio can provide you with important information about the interaction between two key metrics in your business, providing you with an information-rich summary of how your metrics are interacting with each other as well as a way of monitoring changes very efficiently.
However, while ratios are valuable for analysis, it is difficult to choose the right ratio and devastating to your analytics efforts if you choose wrong.
Nowadays, in the age of “lean analytics”, businesses are told to pay attention to 5–10 key metrics in their business. This means that using the wrong ratio as a metric can have an effect as small as confusing your analytics team or as large as providing you with inaccurate recommendations that lead your business astray.
The goal of this short article is to provide you with a quick three step test to determine whether the ratio you are using for your analysis is truly valid and valuable for your business.
After introducing each of these tests, I am going to use “% new sessions” as utilized in Google Analytics, as an example to show you why a seemingly important ratio is, in fact, not a metric you should utilize.
Let’s start.
The Sanity Test: Does the ratio make sense?
A ratio that “makes sense” has the following attributes:
It is composed of two and only two raw metrics
Ratios should only be composed of two raw metrics or else it would be extremely difficult to make sense of the information the ratio is trying to convey.
While this may seem obvious, it is worth noting because of the fundamental nature of this guideline.
The most important thing to pay attention to when conducting this test is to make sure that the components of your ratio are, in fact, both raw metrics. That is, it shouldn’t be made up of numbers that are already ratios such as “bounce rate” or “conversion rate”.
If you use a metric that utilizes a ratio as a denominator or numerator (such as “bounce rate per user”), your ratio will inevitably be composed of more than two raw metrics and, while those metrics may occasionally provide useful insights, oftentimes they are too complex for effective anaylsis.
2. It tells you something about the interaction of its components
Ratios are used to illustrate or summarize the interaction between components instead of focusing on the components themselves. Therefore, it is important to make sure the interaction is a meaningful one.
For example, while “bounce rate” (bounces / total sessions) is a clear indicator of your users’ interaction with your website, in most cases there is no clear and present value to summarizing the interaction between conversion and page views.
Let’s now take “% new sessions” as an example of how to use these sanity checks.
The “% new sessions” ratio consists of two raw metrics ( total number of sessions made by new users and total number of sessions).
It also provides you with an overview of the composition of your traffic, a piece of information that you cannot access without the existence of the ratio.
Therefore, “% new sessions” passes the sanity check as a metric that makes sense (unfortunately this is the only test that it passes).
Relevance Test: Is this ratio relevant to your business objective?
Now that we have a metric that makes sense, let’s take a step forward and ask a harder question: does the metric provide information that is relevant to your specific business objective?
There are two types of business objectives that metrics can help you with: macro objectives and micro objectives.
Macro objectives define the overall success of your business or campaign. From the perspective of your company this is likely revenue, recurring revenue, or profit.
Good ratios that can help you with macro business objectives include, but are not limited to, conversion rate, revenue contribution per customer, and customer lifetime value (CLV).
Micro objectives are metrics that measure the effectiveness of your efforts to reach macro business objectives rather than measuring the success of your business as a whole. These objective include getting users to visit your website, ensuring their engagement with your material, and encouraging them to add your products to their cart.
Good ratio metrics that can help you with micro business objectives include, but are not limited to, “bounce rate” (this measures user engagement with the landing page), “pageviews/ session” (this measures whether users are engaged with your website), and “CTR of Facebook ads” (this measures the effectiveness of your ad copy and targeting).
Notice that micro objectives should be closely related to one more macro objectives in order to make sure that you are on the correct track with your marketing activities.
For example, bringing people from Facebook to visit your ecommerce website is a great micro objective as it pushes your customers towards buying a product. On the other hand, getting more likes on your Facebook page is not usually a good micro objective because it is not directly linked with the customer’s likelihood to buy product on your website.
Now let’s once again relate these insights to “% new sessions”.
An argument can be made that “% new sessions” passes the test of relevance because it measures the amount of new visitors that visit your website and this is a relevant measure of the micro objective of new visitor acquisition.
However, in measuring the number of new visitors that visit your website, the “# new sessions” is a much better metric than “% new sessions.” What “% new sessions” really measures is the composition of your traffic. This is difficult to draw direct business insights from.
For these reasons, “% new sessions” does not pass the relevance test.
Time Test: Can you compare this ratio across time and get meaningful insights?
Finally, a ratio metric is only valuable if you can monitor its changes over time and adjust your business actions accordingly based on what it tells you.
A metric can withstand the “test of time” if it has two attributes:
It will change over time as a result of your actions
You need to make sure that the ratio you choose will change significantly over time as a result of your business actions.
For example, the cost/ revenue ratio of your product might be a relevant metric to your business but if you are not conducting any significant cost-cutting activities this ratio will most likely remain stable for an extremely long period of time. As a result, the metric will not provide you with any meaningful information.
On the other hand, you can see a significant increase or decrease of the click through rate (CTR) of your Facebook or Google Ads if you adjust the copy or targeting of your campaign. This sensitivity to your actions makes CTR a great metric to compare over time.
2. The ratio is “directional”
A “directional” metric is one that indicates positive or negative implications for your business when the metric rises or falls over time. In other words, it helps tell you whether an increase or decrease in a metric is good or bad for your business
Let’s use “bounce rate” as an example. The reason it is such a good metric is because it is almost always better to have a low bounce rate on your website than a high bounce rate.
Let’s now relate these insights to “% new sessions”.
While “% new sessions” may change over time based on your actions, it is not “directional”.
An increase in “% new sessions” might indicate an increase in new traffic or a decrease in returning traffic. This ambiguity makes it hard to tell whether it is a good or bad thing for your business.
Therefore, “% new sessions” fails the “test of time” as well.
This article provides you with three quick tests that will help you decide whether a given ratio will be a good fit as a key metric for your business.
To be honest, I am thoroughly surprised that “% new sessions”, one of the 8 key metrics in Google Analytics, does not pass the sanity check explained in this article however this reality is important to note.
It is for this reason that we don’t analyze “% new sessions” as a metric in our Automated Google Analytics Tool but instead replace it with “# new sessions” and “# returning sessions”, two non-ratio metrics that are much better at measuring the traction and composition of your traffic over time.
Speaking of our tool, our new prototype is now live! If you are interested in beta-testing our Automated Google Analytics tool, please feel free to reach out at bill@humanlytics.co. Your comments and suggestions on how to make our tool more relevant and efficient are greatly appreciated.
Action Checklist
We’d love to hear your feedback/suggestions on our blog content: bit.ly/HMLsurvey
Email us at bill@humanlytics.co (and sign up for our newsletter) if you’d like to test out the prototype for free with your website data.
Follow us on Twitter, Facebook, Linkedin, Twitch, and Medium to help us make data analytics accessible to small businesses.
This article is produced by Humanlytics. At Humanlytics, we build tools for SMBs that not only help them answer their business questions and track metrics in real time, but also tell them what questions they should be asking in the first place — all with the goal of teaching them how to implement solutions.