A Primer on Startup Metrics – Which Analytics Tool to pick.

Jan 6, 2014 · 12 min read · 23,410 views

As a analytics consultant I am working with a lot of startups and help them in the topics of metrics, analytics, retention and growth. Most of the questions i get are based on a few misunderstandings.

Eg. What kind of analytics tools you should use depends what you are trying to do with those tools. They have (more or less) specific purposes.

Disclaimer: This post focuses on web products - but you could apply the same concepts for mobile.

The Segmentation of web analytics

So let’s get into this…

First: Please lower your expectations

If you are hoping to add one tool and be ready to go, i need to stop you right here. Please readjust your expectations.

There are a few hard truth facts we need to accept:

All tools are good for a few things, but tend to suck for most of the other - the trick is to know what you are looking for.

Also: Any tool you will integrate will most likely be: not optimal for all of your usecases, not specific enough, most likely be broken or incorrect a good amount of your time and contradicting every other tool you have.

But that’s ok. If you know how and what to use them for.

Let’s roll…

I personally split usage of analytics into two axis:

  1. exploration / accounting.
  2. external / internal

Exploration vs Accounting

Exploration are those ad-hoc questions.

Exploration are all those quick questions people come up with and are very often related to some intuition we have.

  • eg “What feature is most used?”
  • or “Are users who use feature X more likely to upgrade to plan Y?”

Conversion Rates, Landing Page Optimizations, Funnel Optimizations, Viral Loops, Quick questions and insights are they typical examples of usecases of exploration.

Accounting are the metrics we track over time.

FYI the term accounting was coined by Eric Ries as innovation accounting covering more metrics than the typical product metrics, but the term kept sticky in normal web analytics as well.

If you have ever kept record of KPIs in your excel sheets (eg for investors) that’s a typical report we want for accounting.

But it also includes reports here we tend to use several times a week.

Cohort Retention Tables, Customer Happiness Reports, Traffic Referral Reports or Revenue Reports are typical examples for accounting reports.

About Data integrity

In case of exploration it usually doesn’t matter too much if we have only little historic data, or that we need to jump through a few hoops (aggregate from several tools, export to excel/R, etc). We need as much data as we need to invalidate or validate our hunch/gut-feeling or current objective. Very often this insights can and will be challenged by the team and need further verifications.

These insights usually lead to experiments (eg a/b tests).

In the case of accounting it is very important to have historic data and data integrity is very important - we can’t really tolerate too much noise in the data we are using for accounting. We can’t really tolerate broken reports nor can we have our team not trusting those reports.

This insights usually lead to strategic decisions. (eg switch sales focus)

While it is very useful to distinct between these two areas we need to keep in mind that in reality the lines are a bit blurry.

The second axis i tend to look at is external and internal analytics.

External vs Internal Analytics

  • External Analytics covers all public facing pages and all passive actions by users (eg views).
  • Internal are all actions done by (persistently hopefully) identifiable users, that potentially reflect if your product provides value to them.

When people speak of “product insights” they tend to mean that internal view of your product.

Many product have a very clear distinction between external and internal - eg. SaaS tools provide a value that i need to register for.

Sometimes it’s a bit blurry - Some products provide usecases to non-identified users as well - eg. content pages like pinterest or 9gag where a huge amount of the product is public and most of the visitors value is provided passive.

In a nutshell

Which Tools

There is roughly a million of analytics tools out there. I will focus in this article on the most common ones for web products: Google Analytics, KISSMetrics and Mixpanel.

Google Analytics:

GA is great for everything external. Especially traffic analysis and referral optimization work perfect in GA.

I am a huge Google Analytics geek and to cover it properly would be at least 3 own blog posts.

If you are new into it here are three blogposts i would recommend you to read:

The most common usecase for GA is referral optimization. It enables you to understand what sources bring you the most valuable traffic and where you should double down (as said above learn how to use weighted sorting).

If you use GA for all external stuff you will be very happy. For product internal insights GA tends to be useless.

The main reason is that GA doesn’t think in users and events. It thinks in visits and sessions.

You can force GA and focus more on visitors, you can segment down to the pages that are product internal, people that are registered and you could look mainly at events. But even if you do all of that then you will have a hard time using that data.

GA wasn’t made for product insights in first place.

GA is awesome for product-external exploration and very good for a lot of product-external reporting/accounting (eg weekly referral/traffic/seo reports).

But it stops being spot-on useful where most products actually start. (activation/engagement)

Ad Mobile: I am speaking obviously mainly about GA for web here. The GA team currently pushes their mobile analytics tool. Everyone who has used will notice this weird feeling that the tool wasn’t meant for this originally in first place. But given their current development speed I wonder if i will need to come back here in half a year to update this part.

KISSMetrics and Mixpanel

Both - KMS and MXP - come from the same main concept. They think in “people” and “events”.

Both tools are great and to cover them well would need several blogposts by themselves so please excuse my brevity.

At first they look very similar and i am pretty sure you could substitute every feature of the one in the other. But to me they both tend to have a bit different focus. They have been build by teams with a bit different values.

Mixpanel is awesome for product-internal exploration and good for product-internal reporting. It’s the tool i would pick if i focus more on product improvements (eg engagement). Personally I also have better experience with its stability.

KISSMetrics on the other hand feels like it has one foot in the product-external world.

It’s revenue tab makes it very easy to see what sources brought customers of what kind of life time value. You can easily see what your customers where doing and so on.

Of course MXP is catching up and offers a revenue tab as well. But nether the less KISSMetrics would be the tool i’d pick if I would focus a lot on revenue improvements (eg through content marketing) while trying to improve the product itself (especially as a SaaS startup)

Actually KISSMetrics fits so well as a hybrid between product-internal and external usecases - especially for content marketing - i wouldn’t be amazed if they come up with a specialised content marketing tool or somehow else integrate with Google Analytics to have a easier angle to the market.

Specialised tools

There are a lot of really good specialised tools and again not possible to cover all of them here. My personal four favorites are

And of course there are many more.

The advantage of specialised tools is that they understand (or expect) enough of your product internals that they can provide value insights right out of the box.

The big problem with all tools

Most tools are integrated through push events. Basically your site will send an event through javascript or your backend and let their service know.

There is a big problem with this approach.

Data Integrity

Sooner or later your integration will break for a day or two. Because someone on their or (more likely) your site made a mistake.

Sooner or later you will change how you track certain events and your comparison with your historic data will no longer work.

Did you ever see different numbers in different tools? Eg different amount of registrations between MXP and your database? or Revenue in KMS and in your stripe dashboard? Well that’s what i am speaking about.

In case of exploration this usually tends not to be a big problem. Anyway the data i work with tends not to be optimal (sample noise, dataschmutz, skewed, variance etc) and I am just looking for anchor points for future experiments.

In case of accounting this is a big problem. We cannot really have our revenue reports incorrect. This is unfortunately where the fun ends.

And there is a other problem…

Focus on product internal aspects

Very often product-internal reports tend to be quite specifically tied to your product.

  • If you run a market place you want to report improvements on both sides separated.
  • If you sell not to people but to companies you want to aggregate your users into those company buckets.
  • If you have different levels of users you might want to treat them different (eg drill down your best paying or engaged segment)

To figure the right metrics is a blogpost by itself but to in a nutshell you want to tie your product assumptions to quantified results.

All of the mentioned segmentations above are doable in some tools mentioned, but it will require you to jump through several loops.

And given the fact that very often data integrity tends to be a problem the logical conclusion is to sooner or (if possible) later build those highly specific reports by yourself.

Work with your database for accounting

When it comes to accounting i would highly recommend to work with your database.

As a rule of thumb: The more you need to zoom in - the better you work based on your own database.

Depending on the level of complexity you need you can:

  • Export to excel
  • Run a script that sends the most important numbers
  • Build a simple dashboard

In most cases i would recommend to start with the first one and do it for the first months. And then start step by step automating the most time consuming report parts to scripts and from scripts you will move to a simple dashboard view over time. Usually i would recommend stopping here.

If the dashboard itself turns out to be too time consuming (eg most likely because you put too many expectations into it) try using a tool like chart.io

If you pre-run into performance problems move the analytics to a own server and run the scripts only once a day.

Doing that you can introduce reports like Customer Happiness Index and Health Dashboards (red/green lights (or kpis) per customer or segment)

Disclaimer I am not advising you to build your own analytics tool. I not recommending to measure non-logged-in data. I advice you to segment your existing database data for reporting. There is a big (costly) difference.

I won’t go into dashboard design right now - but if you are interested in this topic please let me know via twitter happy to write a blogpost about it.

So to answer the question – which tool would i recommend

These are the swiss army knifes. Depending on stages i would recommend other tools additionally or instead.

If i have to decide between MXP and KISSMetrics i would - usually - go with Mixpanel (but as explained above it depends a bit on the specific product).

Hope this blogposts helps you a bit to get clarity about metrics even though i couldn’t cover all aspects nor all tools.

If you are still unsure which tools to pick i would recommend using Segment.io as a wrapper for your API calls. That way you can simply replace your tools whenever needed.

If i can help you anyhow with your analytics setup - let me know - i am doing weekly open office skype sessions - feel free to hit me up on twitter

Until then - measure/build/rock

Andreas