TELLING YOUR PRODUCT
STORY WITH METRICS
Govind Ramanathan
We communicate as a Product Manager
Customers
•
•
•
•
New Features
Feature
Improvements
Performance
Business Value
Executive Team
•
•
•
•
Product Vision
Roadmap
Revenue
Resource Utilization
Stakeholders
•
•
•
•
Current Status
Roadblocks
Timeline
Partner Issues
Marketing
•
•
•
•
•
Vision
Customer Impact
Innovations
Target Markets
Competitive
Advantage
Product Team
•
•
•
•
•
Overall Vision
Feature Priority
Overall
experience
User story
Roadmap
We run as fast as possible
Market Requirement
Document
Feature
Specifications
Think Words to Numbers
With Documents
• Vision
• Build a platform that creates
quality leads to our customers at
low cost
With Metrics
• Vision
• Increase conversions to 30%
• Increase quality leads to 55%
• Reduce cost per acquisition to
20%
• Status
• The Feature to screen quality
leads is 50% done
• The APIs needed to measure
conversion is defined
• Status
• Conversions are at 15%
• Quality of leads increased to
40%
• Cost per acquisition is 45%
Metrics
• It is not…
• Replacement to a Vision statement
• Metrics are derived from your vision statement. It helps to enforce the vision.
• Replacement to Scrum or the engineering progress
• Replacement to Roadmaps
• It is…
• Language to communicate progress accurately
• Goal for the teams to focus on
• Improve it and feel good about the work
• Method to filter the vision into numbers
• Means to force us to think about actual benefit/value we are creating
• To measure and communicate success pre and post product launch
• Tool to reduce chatter on details and focus on the bigger picture
Types of Metrics
• Feature Metrics
• For large feature measure the progress
• Example
• Product: Inventory management system. Own fulfilling orders and replenishing
inventory.
• Metric:
• Average time to fulfill order is less than or equal to 2 days
• Orders fulfilled greater than 2 days is less than 10%
• Buffer inventory <= 30%
Types of Metrics
• Product Metrics
• Directly measures product improvements
• Measures the product in isolation
• Example
• Product: Sell Textbooks Online
• Metric:
• Decrease online ordering time to less than 2 minutes
• Reduce shipping average shipping cost to less than $3
Types of Metrics
• Company Metrics
• Measures the entire suite
• Reports on the improvement of the eco-system
• Forces compromises on individual products for the overall Product Suite
• Example
• Company: WeSellLotOfStuffOnline.com
• Metric:
• Increase Revenue by 30% to $30M per quarter
• Decrease Operating Cost by 10% to $20M per quarter
• Increase customer base by 50% to 5M customers
How to define Metrics?
• Identify the entity of value
• Identify the audience for the metric
• Avoid metrics that may be influenced by externalities
• Metrics should be impacted only by what it measures
• Look back at your vision
• Would improving this metric satisfy your vision
• If more than one metrics satisfies the vision then pick the most
important
• You can have more than one but make sure they are mutually exclusive
• Look at the top-level metrics to determine if lower-level metrics are
targeted reasonably
Things to watch out…
• Always monitor over optimizing for a metric
• Evaluate if counter metrics needs to be created
• Evaluate if new metrics are needed
• Define and evangelize how you obtain the data for your metrics
• Including pre and post processing
• Data is never perfect…but we can get close
• Resist the temptation to add revenue as a metric
• This is only useful at the company level. At the product level you lose focus
• Never count product bugs or support calls as a way to monitor
product improvements
Continue measuring the product pre and post launch
Days to Fulfill Order
16
14
12
10
8
Product
Launch
6
4
2
0
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
THANK YOU
“In God we trust; all others must bring data.”
W. Edwards Deming