On Making Behavioural Analytics Scalable
At Breathe Life we take data privacy and security very seriously. We hold ourselves accountable to the highest standards and make sure that our users and clients can use our products in complete safety. The data described in this article, behavioural data, is completely anonymized at the source. In other words, as we collect this data, we exclude all potentially identifying information. The resulting dataset contains a wealth of anonymized information about how our platform is being used by end-users, yet we have no way to link the behaviour to individuals.
In this article, I lay out some of the challenges that we faced at Breathe Life while implementing tracking on a scaling SaaS platform. I then cover the collection of tools, processes and mindset that we adopted to overcome these challenges.
Why We Do Behavioural Analytics
Breathe Life was founded on a strong data culture that has led us to quickly invest in the creation of quality data, including tracking data.
Tracking is hugely important for us because we use it to understand product performance and user behaviour. However, it also matters to our carrier clients, because they too have access to this data. They use it to power major business decisions. Among other use cases, carriers leverage this data to identify friction points in their distribution process, help advisors optimize their sales performance, design optimal recommendation systems, etc.
How It Became Hard
We instrument our SaaS platform with precise tracking. Historically we have done this by studying the underlying product features and deciding which system behaviours would be important to understand in the future. For example, when we launched a filter feature in our leads management module, we started recording user behaviour around list filtering.
Over time, this methodology became challenging. As our product grew bigger and stronger, the amount of time required to instrument tracking increased.
Our team was spending more and more time studying the engineering backlog, designing new tracking specifications to fit the new features, and manually checking the flows in production to see what had actually been implemented.
To cover it all we decided to divide and conquer. Each data team member was monitoring a different platform module, and we independently defined the tracking specs for the new features developed on our respective modules.
Our tracking plan was growing in size. It lived on a spreadsheet with over 50 events, including duplicates and names that did not follow our conventions. Not only were we spending a lot of time on tracking, but we were also headed on a dangerous path towards data integrity problems.
Our tracking plan in 2019 lived on spreadsheets.
What We Did About It
We studied other companies’ tracking specs (i.e. Sendgrid’s, Segment’s suggested templates), tracking strategy articles (i.e. Creating a Tracking Plan, What Happens When You Can’t Trust Your Data?), and data governance tools like Iterative.ly, Protocols and Typewriter.
With these in mind we designed a plan that, once completed, would allow us to scale our tracking code with minimal human intervention.
The plan is based on 2 foundation principles:
Business Questions First
Historically, even if we had meticulously implemented precise tracking on our SaaS, it was easy to ignore the end goal for our tracking. Why did we want to know that a user was filtering a list? The product team was not planning to tackle this optimization in the coming months, so why implement the event right away?
As we reviewed tracking through a scalability lens, we changed the way we work and reversed the process: we started asking the business question first. No business question? No event.
This was scary in many ways. We had to trust our business instincts, and stop relying on “just in case” code.
As we cleaned up our tracking plan we were letting go of many events that had seemed important not too long ago.
2. Quality Over Quantity
Asking the business question first was a big step forward, but it wasn’t enough. If we wanted to reach our goal of reducing the implementation workload and keeping the tracking code sustainable over time, we had to downsize. A smaller plan would be easier for developers to understand, and for us to test.
To emphasize quality over quantity, we reviewed all of our existing events and properties to check if other existing data could already answer the business question behind each event. When we found another way to answer our questions, we deprecated the related event or property.
For example when an end-user skips a step while filling out their insurance application, we used to trigger Skipped step. But during this review process we found that we would be able to detect skipped steps by combining 2 of our other existing events: Viewed step and Completed step, so we deleted Skipped step.
Now that we’ve deprecated 52% of the original events and 62% of the original properties, we are extra careful when it comes to adding new events and properties. We would only let something new in if there was a really good business question that could not be answered otherwise.
Making Sure The Fix Stays
After the cleanup we also added new tools to our stack that would maintain the quality and simplicity of our tracking plan over time. We moved our plan from a spreadsheet into a Github repository where we now enforce peer reviews before accepting plan modifications. There, the plan lives in a collection of YAML files.
Our GitHub repo where we collaborate to make changes to the tracking plan
We then wrote a custom python script to transform the collection of YAML files into a single JSON file that contains the full tracking plan. The script also checks that our events don’t use deprecated properties, and uploads the plan into Protocols via the Segment’s config API. In Protocols, the tracking plan can be used to validate the events we receive from our application.
Our custom package that generates the plan in JSON file from the collection of YAML files.
Next, our engineers implemented Typewriter which can pull the latest plan version from Protocols to generate a Segment analytics library. They also created a Slack channel where we send notifications whenever the tracking plan is modified. This allows the whole team to stay updated on the latest version of the tracking plan.
Our Slack channel where we broadcast changes to the tracking plan.
With all these tools and processes in place, we can trust that our tracking data will continue to be a reliable source of information, even as our SaaS grows in size and complexity.
And our journey does not stop here. The future of behavioural analytics at Breathe Life includes:
Automated workflows to push the plan updates to the developers’ code libraries
Increased auto-testing on data in staging and production environments
Ownership of the tracking plan by the product team
And probably more, but that story belongs in another article.