TELLING YOUR PRODUCT STORY WITH METRICS Govind Ramanathan We communicate as a Product Manager Customers • • • • New Features Feature Improvements Performance Business Value Executive Team • • • • Product Vision Roadmap Revenue Resource Utilization Stakeholders • • • • Current Status Roadblocks Timeline Partner Issues Marketing • • • • • Vision Customer Impact Innovations Target Markets Competitive Advantage Product Team • • • • • Overall Vision Feature Priority Overall experience User story Roadmap We run as fast as possible Market Requirement Document Feature Specifications Think Words to Numbers With Documents • Vision • Build a platform that creates quality leads to our customers at low cost With Metrics • Vision • Increase conversions to 30% • Increase quality leads to 55% • Reduce cost per acquisition to 20% • Status • The Feature to screen quality leads is 50% done • The APIs needed to measure conversion is defined • Status • Conversions are at 15% • Quality of leads increased to 40% • Cost per acquisition is 45% Metrics • It is not… • Replacement to a Vision statement • Metrics are derived from your vision statement. It helps to enforce the vision. • Replacement to Scrum or the engineering progress • Replacement to Roadmaps • It is… • Language to communicate progress accurately • Goal for the teams to focus on • Improve it and feel good about the work • Method to filter the vision into numbers • Means to force us to think about actual benefit/value we are creating • To measure and communicate success pre and post product launch • Tool to reduce chatter on details and focus on the bigger picture Types of Metrics • Feature Metrics • For large feature measure the progress • Example • Product: Inventory management system. Own fulfilling orders and replenishing inventory. • Metric: • Average time to fulfill order is less than or equal to 2 days • Orders fulfilled greater than 2 days is less than 10% • Buffer inventory <= 30% Types of Metrics • Product Metrics • Directly measures product improvements • Measures the product in isolation • Example • Product: Sell Textbooks Online • Metric: • Decrease online ordering time to less than 2 minutes • Reduce shipping average shipping cost to less than $3 Types of Metrics • Company Metrics • Measures the entire suite • Reports on the improvement of the eco-system • Forces compromises on individual products for the overall Product Suite • Example • Company: WeSellLotOfStuffOnline.com • Metric: • Increase Revenue by 30% to $30M per quarter • Decrease Operating Cost by 10% to $20M per quarter • Increase customer base by 50% to 5M customers How to define Metrics? • Identify the entity of value • Identify the audience for the metric • Avoid metrics that may be influenced by externalities • Metrics should be impacted only by what it measures • Look back at your vision • Would improving this metric satisfy your vision • If more than one metrics satisfies the vision then pick the most important • You can have more than one but make sure they are mutually exclusive • Look at the top-level metrics to determine if lower-level metrics are targeted reasonably Things to watch out… • Always monitor over optimizing for a metric • Evaluate if counter metrics needs to be created • Evaluate if new metrics are needed • Define and evangelize how you obtain the data for your metrics • Including pre and post processing • Data is never perfect…but we can get close • Resist the temptation to add revenue as a metric • This is only useful at the company level. At the product level you lose focus • Never count product bugs or support calls as a way to monitor product improvements Continue measuring the product pre and post launch Days to Fulfill Order 16 14 12 10 8 Product Launch 6 4 2 0 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec THANK YOU “In God we trust; all others must bring data.” W. Edwards Deming