– Being lean is about doing things that make sense and about being efficient, not necessarily about following every step of the ‘methodology’ to a T.

The lean startup movement/philosophy/methodology has lots to teach businesses and entrepreneurs of all shapes and sizes, but it’s not a live-or-die guide to building a successful company. In building Pinyadda we’ve done some things a little backwards and maybe a few others a little out of order, but for us it’s become clear that being lean is about lightweight, fast product iteration and hands-on customer development. The exact methods we use to achieve these goals have changed and will continue to change and evolve along with us.

Did we smoke test hundreds of landing pages? No. Did we build a product with the absolute minimum feature set? No. Have we learned and improved by integrating customers early and continuously iterating based on their feedback? Yes. The main point here is that it’s a good idea to read and absorb as much as you can, but don’t drive yourself crazy trying to implement the latest and greatest trends. Every business will have different customers and different products that demand different tactics and different solutions. Do what’s best for you.

– Measurement takes many forms.

With most Web 2.0 – style products, what you’re building is not so much a ‘product’ in the traditional sense as an experience. In this vein, while it’s really important to measure things like clicks and actions and pageviews that can be quantified, it’s also important to realize that actions are often dictated by psychological and emotional reactions which cannot be easily quantified.

A careful reading of most lean materials will reveal that qualitative data (surveys, interviews, observations) can often be as useful or more useful than all the numbers in the world. There are certainly instances where the reverse is also true, but the main point is not to throw feedback out the window because it’s not numeric in nature. Some of the most important feedback we’ve ever received about Pinyadda has come in casual conversation over a few beers.

– There’s no magic button.

At first glance, it can seem like the lean startup approach makes for clean periods of structured action, where one piece fits nicely into the next. The truth is that it’s nearly impossible to suddenly declare yourself “done” with one stage and simply transition cleanly to the next step. Often (especially in the early stages) things are muddled and it’s very difficult to tell exactly where you are in the life-cycle of the company (this isn’t made any easier by the fact that most entrepreneurs are, by nature, somewhat delusional). And even if you can get a good sense for where you are, trying to plot a course into the next phase can be daunting.

Going back to the first point – there’s no formula for this stuff. We have a general idea about where Pinyadda stands with our customers and we’ve defined some tangible goals for ourselves that should manifest over the foreseeable term (for us, about 3-6 weeks at a time). But we don’t sit around wondering if we’re in Phase 1.3 or 1.4. Set a goal, use your core philosophies as a guide, and then do everything necessary to achieve it.

Here’s the video of the presentation. Thanks to Matthew Mamet of EditMe for the recording. Matt’s original post on the meetup is worth a read as well.

Boston Lean Startup Circle – Pinyadda from Matthew Mamet on Vimeo.

Turns out this stuff is useful for people who are pirate-neutral, maybe even those with an aversion to pirates (I don’t understand you). I could sum it up  but you’re much better off checking out the presentation or watching a video of Dave giving the basic talk. But the gist is pretty simple: measure the stuff that matters, don’t measure the stuff that doesn’t. This might seem like an easy thing to do, but in reality it can be pretty easy to confuse getting a lot numbers with getting the right numbers.

We’ve measured a whole bunch of stuff over the course of our time at Pinyadda, and we’re still zeroing in on exactly what we need and what the best ways are to quantify those metrics. What works for our business almost certainly won’t work for yours or anyone else’s, and it takes time to figure out the magic concoction that can quickly and easily provide accurate insight into what’s really going on.

In our current phase, we’re focusing on two key aspects of the AARRR model – Activation and Retention. At the moment, these are the two most important things that we need to figure out: how and when are people getting to the ‘aha’ moment with Pinyadda, and what keeps them coming back? By drilling down and taking a really close look at these specific aspects of our product over the last month, we’ve been able to do a couple of key things that have kept us focused on making the right product iterations instead of spreading feature lists all over the whiteboards and then spinning around in tight circles:

1) We asked some really specific but straightforward questions about how people were using our product. These are things like “how many people does someone need to follow before they start Pinning articles consistently,” and “of the people who login most, do they follow more people, sites, or topics?” We’ve had this data all along but we’d always cast it in terms of feature use or product analysis. Spinning the questions this way, to make them centered on the people instead of the product, makes the information more useful, easier to understand and much more actionable.

2) We invested a small but invaluable amount of time in building some reporting features for ourselves. Raw data is wonderful and we have lots of it, but if it takes an hour harvest and organize it’s not really useful to us on the daily. Taking hours away from pure product development to work on this is a little hard to swallow, but sacrificing a few Friday nights to do it might save weeks of wasted effort later if you can avoid building a single doomed feature. Plus, it’s a good environment to mess around with some new toys (I used a lot of CSS3, which I don’t use much in production, and also got comfortable with a great visualization library called Flot). Here’s a sneak peak:

3) We set some tangible goals for user activity and retention. We’ve always done goal setting but narrowing the focus has helped everyone on the team look at their area of focus from a common perspective. When you’re as small as we are it can be hard not to get lost in your own little corner. Keeping our eyes on a small and focused goal set makes it easier to stay on the same page. The results have been great and it’s been awesome to feel the team catch its collective stride.

McClure’s language may be a bit blue, but the message should be taken to heart by small teams everywhere. It’s helped us not because we follow it to a T or take every word as truth, but because it just makes sense. Every business will have to find its own way, and what makes sense to measure will vary widely. But by thinking critically about metrics before spewing numbers all over the place, coloring the things you measure in terms of people and not faceless features, and keeping team members aligned toward small but focused tasks, you can channel the spirit of pirates and strive forward in the search for internet booty.

We use several sets of tools to measure data, and each has its own strengths and weaknesses. In order to ensure validity, we often cross-reference these data sets against each other to make sure we’re getting the most accurate numbers possible. While I often wish there were a single solution to this problem that could provide reliable and accurate data in one place, I do find some sort of sick comfort in knowing that outliers and tracking errors can be mostly avoided by the built-in redundancy in our tracking methods. But however much satisfaction, validation, and useful insight comes from looking at the compiled data sets, there’s no doubt that tracking and gathering data can be a huge pain in the ass.

Here’s a rough outline of what we track and how:

- Traffic acquisition: We use Google Analytics for most of the basic stuff, and when I’ve taken the time to check it against some of our other sources the numbers on a high-level basis seem to be fairly accurate. Lately it’s seemed to have a hard time grabbing accurate absolute referral numbers, but the search term referral information can’t be beat.

- Conversion: This is a huge metric for any startup so it makes sense that we use every tool in the arsenal on this one. Some high-level goals are tracked through Google Analytics, but the process of hacking javascript events to resemble pages and then setting up URL-oriented funnels really doesn’t work well for us, since almost all of our pages are dynamically loaded and changed. This is where KISSmetrics really shines, and we’re using it a lot to track our specific conversion funnels. Since we load different value props based on referral, it’s really important to have  Javascript-based tracking, especially at the top of conversion funnels. We can use database events to construct records for true conversion events, but the views and clicks that lead up this are really easy to define in KISS Metrics. Like I said, we cross-check a lot of these numbers against our own internal tracking of insertions and server requests, but so far we’ve had little to no discrepancy between our numbers and KISS Metrics. Which is sweet. Check out this killer post if you want a more detailed breakdown of what KISS can do.

- A/B and Multivariate Testing: If conversion is our most important metric, then A/B data is our most important weapon. Google Website Optimizer rules the roost on this one and it’s a great tool. One thing we do that’s really helped to accommodate easier implementation of GWO is to write the page variations dynamically using PHP. By adding fields to the database we’re calling that are set up specifically to handle the GWO tracking code, as well as any elements we want to test we can create new experiments without ever having to touch the actual code. It’s not rocket science, but it certainly makes it easier to quickly set up an experiment.

- Retention, Engagement, and Referral: Most of this we track using our own internal systems, but we’re using KISS Metrics again to track some very specific action funnels (how far users get in various setup processes, the various ways in which they send invitations) that are primarily Javascript-based. Again, we could probably use Google Analytics for  this, but it’s hard to resist the simplicity of the KISS Metrics reporting interface. We do use Google Analytics for some basic pageview numbers and for some of our external link tracking, to give us a broad sense of how people are using the product. When we identify something we want to take a closer look at, we’ll build a little component or move to an event-based tracking system.

There’s more to it, but those are the basics. For what it’s worth, here’s a quick analysis of the two tools we use the most, as I see them:

- Google Analytics:

• PRO Powerful. I mean, wow. You can do pretty much whatever you want with this tool, provided you can figure it out.
• CON It’s impossible to really figure out all the way. I’m sure there are some gurus out there using every single feature, but I often spend so much time navigating the interface that’s hard to remember what I was actually trying to track.
• CON Constant fear of reliability failure. We’ve had to reset our tracking code a couple times for no apparent reason, and occasionally get data that reflects strange outliers. This probably happens with any analytics suite, but the fear of data voodoo sometimes keeps me up at night.
• CON Kills load times. Watch the activity monitor in almost any site you visit. That ’13 of 14 items, still waiting’ text? Yeah, that’s the GA code.
• Conclusions: A lot like owning a Ferarri with mediocre tires that sometimes won’t start. In other words, lots of power but hard to steer and just unreliable enough to keep you on your toes.

- KISSmetrics

• PRO Event-based tracking. It’s built from the ground up to accommodate dynamic sites and rich web interfaces. Simple Javascript syntax makes tracking any event a cinch.
• PRO Reporting interface. They built it for regular people to actually use (an amazing concept), and they give you data where the main metric is ‘People’. As in ’20 people clicked on this button.’ So simple, makes such a big difference.
• PRO Customer service. They’re a startup, and they roll lean-style, which means you can get answers to your questions. And quick. My last inquiry on twitter was answered in about 15 mins.
• CON Limited functionality. Ideally I want this simple model to be able to do some more powerful things. But they’ll get there soon, no doubt.
• Conclusion: More like owning a slick new Mini in Beacon Hill. Perfect for what it does, looks great, easy-to-use; probably not the right tool for bigger jobs.