Navigating customer analytics in the age of big data
As overused as the term may be, for the past 10 years, “big data” was only most relevant to large enterprises with terabytes of data and teams of data scientists.
For smaller players like most of you reading this article, big data is significantly less relevant due to several key reasons:
We do not have enough data in the first place. Most of the data we collect (such as orders, advertising performance, etc.) are extremely small and non-scalable, massing up to a couple of gigabytes at most every year.
Even with that couple of gigabytes of performance data, we lack both talents and infrastructure to deeply analyze the data and understand what kind of value we can actually derive from it.
The impact of big data analytics simply does not justify the cost - a 5% increase in conversion rate may mean millions of dollars to companies like Amazon, but only thousands for a million-dollar business like us.
However, the emergence of more mature and easy-to-use data infrastructures services such as Klaviyo and other CDPs (customer data platform) has made big data analytics a possibility for companies of smaller size.
This recent development made big data analytics more and more relevant for us in recent years (even though I still believe that it should not be an immediate priority for your business).
The purpose of this article is to walk through three primary points related to implementing big data analytics at your company:
What are some of the potential benefits of big data analytics for your business?
What kind of data infrastructure is required to make your company “big data ready”?
What technology/talents you will need to research and invest in today to make it have a positive ROI?
The “Big” Benefits of big data
In every area of business, there is a “case” to use big data as long as there is sufficient data to be collected, but the key challenge is to make the “case” worthy of the investment and truly translate to a positive bottom line.
In my opinion, the strongest case of big data in smaller organizations is relating to customer engagement, that is, finding and better serving high-quality customers so there are 1) more of them, and 2) they come back and buy more from you.
Relating to getting more high-quality customers, big data can help you better and deeper understand the action patterns of your highest paying customers, so you can facilitate more of those behaviors on your website by people who are not your customers yet.
This vague definition deserves an example to illustrate.
Let’s say that you run a simple e-commerce website that sells food and snacks for pets, ranging from daily food, nutritional supplements, to snacks.
The “non big data” way of obtaining more customers would be locating an email list of your top 100 customers, and drop those lists into either Facebook or Google to create a “lookalike audience” group, so the algorithm of those platforms can help you target people with similar interests.
With help of more advanced analytics, you can now understand much more about those high spenders.
For example, by collecting all website interaction data of all of your existing customers through a service like mixpanel or klaviyo, you can now understand specifically which pages and products your customers saw before committing to make a purchase.
In the case of our examples, we found that most of those highest paying customers visited the “about us” page of the website, and over 50% of them purchased the nutritional supplements for pets first before buying more products from us.
Information collected above can help us create a much more complex advertising funnel, with a mix of CPM campaigns to increase impression on the about us page, more focused creatives and spend around pet supplements, and a remarking infrastructure that focused on targeting people who have seen the “about us” page and visited the “pet supplement” product page.
There are also more advanced use cases of creating a “conversion scoring system” that dynamically scores potential customers “value” to your business based on their behavior conducted on the website, but that’s beyond the base use cases we are describing today.
Now, moving onto the second “customer engagement” use case, which is getting existing customers to buy more.
Let’s say that from the same pet supply store, you have also discovered that after buying the initial nutritional supplement products if a customer comes back to buy another product within 60 days of their initial purchase, they are much more likely to become a high paying customer.
Then you can create a dedicated email flow, and perhaps a marketing campaign that specifically targets people who have made a purchase of the nutritional supplement product line, so these people have a higher likelihood of becoming a high-value customers.
While all of those use cases explained above sound interesting and valuable, there are a lot of hidden traps along the way that require constant monitoring and attention from at least one dedicated person and preferably a team.
For the sake of completing this article, I am just going to mention an obvious trap: the observation bias.
With a surprisingly deep link with the physics phenomenon, the observation bias can be best described as “when you start observing and enhancing an existing phenomenon, its behavior changes drastically”.
Maybe when we start our efforts to target people who have purchased a nutritional supplement product, it is very well within the imagination that it in fact resulted in very few high-paying customers, and our initial hypothesis only works for people who naturally come back after 60 days of initial purchase without any advertising or promotional intervention.
Then, we will need to flip our hypothesis entirely and look back into the data to see what other areas of opportunity can work with promotional activities.
All I am trying to get at here is that the activity of big data analytics is by nature continuous, and you will need a healthy technology and talent pipeline to make the insights flowing, then you can truly obtain value from your big data activities.
Data infrastructure to make your organization “big data ready”
An ancient Chinese proverb says “If you want to be skillful at your trade, you must get the correct tool first.” And big data analytics definitely is an area where this saying rings true.
One of the biggest big data challenges I have seen for most companies under 100m USD in annual revenue (roughly) is incomplete data infrastructure.
This challenge can manifest in many forms include but are not limited to:
Separate, non communicating IT systems
Lack of a structured way to collect clickstream or customers activity data
Using IT system that does NOT provide full data access to the company for analytics
Lack of talents that can design and build a great data infrastructure
Luckily, it is becoming easier and easier to build those data infrastructures in recent years with the introduction of the concept of CDP, or customer data platform.
Very similar to CRMs but much more expansive in vision, CDP is designed to be a storage place for all of your customer data ranging from their basic information (such as Email, name, etc.), to all of their interactions with your company online (from the website), to offline (from your customer service department).
The introduction of CDP, or at least products that seek to achieve the “CDP vision”, makes it easier for companies to obtain a complete solution for customer data management without shopping for multiple services at the same time and worry about integrating them.
With that said, CDP solutions are by no means cheap and out-of-the-box. Significant knowhow and budget are still needed to fully set up a CDP system within your company that best satisfies your needs - but hey, it is much less painful now compared with 10 years ago.
A good place to start exploring CDP solutions for your company (especially) e-commerce, is Klaviyo, a basic but powerful CDP system disguised as an email marketing platform.
Personally, I found myself adding a lot of detailed customer interaction data into Klaviyo and use the API to full all event data for advanced analytics, and the process is rather simple with a reasonable cost to compare with other more “fancy” CDP solutions.
Now you have the data, how to analyze them?
Getting the data infrastructure ready is only your ticket into the big data playing field, now you have to start playing the “big data” game by investing and researching both the correct technology and talents for your company.
The first decision you need to make after setting up your data collection infrastructure is how you want your data analytics infrastructure to look like.
To put in perspective, when analyzing customer clickstream and interaction data, we are dealing with data that masses up to 100 gigabytes for only a year or maybe even a month depending on how much data you are collecting.
With this large volume of data, excel is definitely out of the question, and traditional SQL is sometimes not a good fit to analyze those interaction data because they are usually stored in a document format and are not structured enough to fit into a traditional relational database schema (though you can definitely find a way to do it).
This is when we need to bring out the big gun of big data analytics -> No SQL databases and apache spark.
No SQL databases are databases that are specifically designed to store and process less-structured data like clickstream data, text data, and anything that does not naturally warrant a row-by-row SQL structure.
Then, to process and analyze those data, a service called Apache Spark is one of the most popular solutions on the market right now to make the analytics processes finish in a reasonable amount of time.
To put a final layer of complications on the situation, you will also need experienced data engineers and business analysts to know what kind of analyses are possible given the technology stack that you have established above, and what kind of solutions can be offered to best benefit your current need.
The bottom line is this: it is not easy, and it is not cheap, but it is getting more and more valuable to reach the level of viability especially if you are achieving a high scale.
The key is preparation and know what you can do right now
This article is designed to give an overview of big data in smaller organizations because I have been getting more and more questions from clients related to the topic above.
In the concluding remark, my stance on big data in smaller organizations remains relatively the same: it is still very costly, it still requires a lot of talent and attention, while the outcome is uncertain.
However, it does not mean that you should not worry about improving the data infrastructure in your company.
For example, you should start by adding key customer interaction touchpoints (Add to cart, login, register) into your email platforms so you can better understand the behavior of your existing customers.
It is also not a bad idea to invest in some data integration services so your CRM, online stores, and email platforms are talking to each other rather than being indifferent silos.
Building a successful data infrastructure is for sure a tall mountain to climb, but unless we start today, we will never reach it.