Like most other startups using the lean startup methodology, we’ve become pretty obsessed with tracking data. We track all kinds of stuff, from internal product metrics to external referrals to conversions via the various funnels we’ve set up. We rely on these numbers to help us make key product decisions, to tell us where we should focus our marketing efforts, and generally to find out what’s working and what’s not. For a company still wiggling our way into product/market fit, these numbers are our currency, and it’s important that we get them right.
We use several sets of tools to measure data, and each has its own strengths and weaknesses. In order to ensure validity, we often cross-reference these data sets against each other to make sure we’re getting the most accurate numbers possible. While I often wish there were a single solution to this problem that could provide reliable and accurate data in one place, I do find some sort of sick comfort in knowing that outliers and tracking errors can be mostly avoided by the built-in redundancy in our tracking methods. But however much satisfaction, validation, and useful insight comes from looking at the compiled data sets, there’s no doubt that tracking and gathering data can be a huge pain in the ass.
Here’s a rough outline of what we track and how:
- Traffic acquisition: We use Google Analytics for most of the basic stuff, and when I’ve taken the time to check it against some of our other sources the numbers on a high-level basis seem to be fairly accurate. Lately it’s seemed to have a hard time grabbing accurate absolute referral numbers, but the search term referral information can’t be beat.
- Conversion: This is a huge metric for any startup so it makes sense that we use every tool in the arsenal on this one. Some high-level goals are tracked through Google Analytics, but the process of hacking javascript events to resemble pages and then setting up URL-oriented funnels really doesn’t work well for us, since almost all of our pages are dynamically loaded and changed. This is where KISSmetrics really shines, and we’re using it a lot to track our specific conversion funnels. Since we load different value props based on referral, it’s really important to have Javascript-based tracking, especially at the top of conversion funnels. We can use database events to construct records for true conversion events, but the views and clicks that lead up this are really easy to define in KISS Metrics. Like I said, we cross-check a lot of these numbers against our own internal tracking of insertions and server requests, but so far we’ve had little to no discrepancy between our numbers and KISS Metrics. Which is sweet. Check out this killer post if you want a more detailed breakdown of what KISS can do.
- A/B and Multivariate Testing: If conversion is our most important metric, then A/B data is our most important weapon. Google Website Optimizer rules the roost on this one and it’s a great tool. One thing we do that’s really helped to accommodate easier implementation of GWO is to write the page variations dynamically using PHP. By adding fields to the database we’re calling that are set up specifically to handle the GWO tracking code, as well as any elements we want to test we can create new experiments without ever having to touch the actual code. It’s not rocket science, but it certainly makes it easier to quickly set up an experiment.
- Retention, Engagement, and Referral: Most of this we track using our own internal systems, but we’re using KISS Metrics again to track some very specific action funnels (how far users get in various setup processes, the various ways in which they send invitations) that are primarily Javascript-based. Again, we could probably use Google Analytics for this, but it’s hard to resist the simplicity of the KISS Metrics reporting interface. We do use Google Analytics for some basic pageview numbers and for some of our external link tracking, to give us a broad sense of how people are using the product. When we identify something we want to take a closer look at, we’ll build a little component or move to an event-based tracking system.
There’s more to it, but those are the basics. For what it’s worth, here’s a quick analysis of the two tools we use the most, as I see them:
- Google Analytics:
- PRO Powerful. I mean, wow. You can do pretty much whatever you want with this tool, provided you can figure it out.
- CON It’s impossible to really figure out all the way. I’m sure there are some gurus out there using every single feature, but I often spend so much time navigating the interface that’s hard to remember what I was actually trying to track.
- CON Constant fear of reliability failure. We’ve had to reset our tracking code a couple times for no apparent reason, and occasionally get data that reflects strange outliers. This probably happens with any analytics suite, but the fear of data voodoo sometimes keeps me up at night.
- CON Kills load times. Watch the activity monitor in almost any site you visit. That ‘13 of 14 items, still waiting’ text? Yeah, that’s the GA code.
- Conclusions: A lot like owning a Ferarri with mediocre tires that sometimes won’t start. In other words, lots of power but hard to steer and just unreliable enough to keep you on your toes.
- KISSmetrics
- PRO Event-based tracking. It’s built from the ground up to accommodate dynamic sites and rich web interfaces. Simple Javascript syntax makes tracking any event a cinch.
- PRO Reporting interface. They built it for regular people to actually use (an amazing concept), and they give you data where the main metric is ‘People’. As in ‘20 people clicked on this button.’ So simple, makes such a big difference.
- PRO Customer service. They’re a startup, and they roll lean-style, which means you can get answers to your questions. And quick. My last inquiry on twitter was answered in about 15 mins.
- CON Limited functionality. Ideally I want this simple model to be able to do some more powerful things. But they’ll get there soon, no doubt.
- Conclusion: More like owning a slick new Mini in Beacon Hill. Perfect for what it does, looks great, easy-to-use; probably not the right tool for bigger jobs.
Tags: KISSmetrics, Lean Startup, Pintelligence, Pinyadda, Product