Notes on Lean analytics

Reading in progress. Notes under construction.

Part One: Stop Lying to Yourself

Ch1: We’re All Liars

Let’s face it: you’re delusional.
We’re all delusional—some more than others. Entrepreneurs are the most delusional of all.

You need to lie to yourself, but not to the point where you’re jeopardizing your business.
That’s where data comes in.

Case Study - Airbnb Photography

Insight: professional photography would help their business.

Action: Use Minimum Viable Product to test their hypothesis.

Ch2: How to Keep Score

A Good Metric: Overview

  • Comparative
  • Understandable
  • Ratio / Rate
  • Actionable (Changes the way you behave)

A Goog Metric: Elaborated

Qualitative versus Quantitative Metrics

Vanity versus Actionable Metrics

Exploratory versus Reporting Metrics

Figure 2.1

Case Study: Circle of Moms

  • Circle of Friends at first, huge amount of users but low activity
  • Analysis found that moms are the most sticky groups among the users
  • Pivot to Circle of Moms
  • Finding an “unknown unknown”

Leading versus Lagging Metrics

  • Leading metrics: better, not possible at the early stage
  • Lagging metrics (e.g. churn): somewhat too late but still useful

Correlated versus Causal Metrics

Finding a correlation between two metrics is a good thing. Correlations can help you predict what will happen. But finding the cause of something means you can change it.

Find a correlation -> Run an experiment to test it

Moving Targets

Sometimes there’s a huge gulf between what you assume and what users actually do. You might think that people will play your multiplayer game, only to discover that they’re using you as a photo upload service. Unlikely? That’s how Flickr got started.

Case Study: HighScore

  • Line in the sand not reached even after incremental improvements
  • CEO picked up the phone to find out that the initial baseline of usage they set wasn’t consistent with how engaged customers were using their products

Testing

  • Segmentation
  • Cohort Analysis: compares similar groups over time
  • A/B Testing: cross-sectional studies
  • Multivariate Analysis: save time

Figure 2.2

Summary

Figure 2.3

Ch3: Deciding What to Do With Your Life

Markets that don’t exist don’t care how smart you are.

Lean Canvas

Figure 3.1

  1. Problem: Have you identified real problems people know they have?
  2. Customer segments: Do you know your target markets? Do you know how to target messages to them as distinct groups?
  3. Unique value proposition: Have you found a clear, distinctive, memorable way to explain why you’re better or different?
  4. Solution: Can you solve the problems in the right way?
  5. Channels: How will you get your product or service to your customers, and their money back to you?
  6. Revenue streams: Where will the money come from? Will it be onetime or recurring? The result of a direct transaction (e.g., buying a meal) or something indirect (magazine subscriptions)?
  7. Cost structure: What are the direct, variable, and indirect costs you’ll have to pay for when you run the business?
  8. Metrics: Do you know what numbers to track to understand if you’re making progress?
  9. Unfair advantage: What is the “force multiplier” that will make your efforts have greater impact than your competitors?

What Should You Work On?

Figure 3.2

Ch4: Data-Driven Versus Data-Informed

Data is a powerful thing. It can be addictive, making you overanalyze everything. But much of what we actually do is unconscious, based on past experience and pragmatism. And with good reason: relying on wisdom and experience, rather than rigid analysis, helps us get through our day. After all, you don’t run A/B testing before deciding what pants to put on in the morning; if you did, you’d never get out the door.

Math is good at optimizing a known system; humans are good at finding a new one. Put another way, change favors local maxima; innovation favors global disruption.

How (Not) to Think Like a Data Scientist

Note that these are pitfalls:

  1. Assuming the data is clean
  2. Not normalizing
  3. Excluding outliers (e.g. biggest fans)
  4. Including outliers
  5. Ignoring seasonality
  6. Ignoring size when reporting growth
  7. Data vomit (too much information)
  8. Metrics that cry wolf (thresholds too sensitive)
  9. The “Not Collected Here” syndrome (should mash up data with data from other sources)
  10. Focusing on noise

Learn Startup

We sometimes remind early-stage founders that, in many ways, they aren’t building a product. They’re building a tool to learn what product tobuild.

Be Lean. Don’t be small.

Some people believe Lean Startup encourages that smallness, but in fact, used properly, Lean Startup helps expand your vision, because you’re encouraged to question everything.

Part Two: Finding the Right Metric for Right Now

Ch5: Analytics Frameworks