>> Nachi Nagappan: Thank you for coming here today. We are very happy to have with
us Narayan Ramasubbu from the University of Pittsburgh from the School of Business
there. Narayan has actually been working in empirical software engineering for several
years now. He's written several award-winning papers [inaudible] and FSE. He actually
got his PhD from the University of Michigan, Ann Arbor, and previously he was actually
at Singapore Management University.
>> Narayan Ramasubbu: Thanks. Thank you for the opportunity to share my work. As
Nachi was saying, I'm actually faculty in the business school very at work in software
engineering and related areas. The talk today is not about a specific project but to reflect
on my experiences traveling to, sometimes, different approaches to the same sort of
questions we have in business and in software engineering research. And it's sort of a
call for research from the community here to see that we have to build a common ground
for important questions and to alleviate inquiring software engineering strategic concerns
of a firm, not only questions about programs and project managers but that really we can
answer questions about firm strategy.
That's the sort of ambition I have for this talk, and I'll reflect on and show examples of a
few strategies where I have tried to build such a common ground. What's the motivation
for building such a common ground? And as we live today we can see that software is
fused into every wide range products and services. It's not just high tech firms that worry
about software. Banks worry about software. It's also strategically important in the sense
that when do we release an app for a banking application on the mobile phone? The
timing of it, the quality issues of it all have implications to firm strategy and your stock
prices. And it's almost -- The term is that software and IT are fused into almost all realms
of a firm's operation, firms not only like Microsoft but a wide range of services and
industries.
So given that context, all the tradeoffs we talk about in software engineering, cost versus
quality, short-term benefits versus long-term issues, all of have strategic consequences,
how we handle those tradeoffs. So it is in this context that I would like you to see the
strategies I present and how we can address questions related to these tradeoffs. And it
is my observation that there is very little common ground between the literatures that talk
about these tradeoffs. In management, especially in technology management research,
they act as if they are blissfully unaware of all advances in software engineering.
And software engineers research most of the time acts as if there is no other research
out there talking about these tradeoffs. Let's say in projects in operations management
talk about these tradeoffs all the time and they model it, but we don't talk to each other.
That's my observation, that there is no common ground and we have to attempt to build
such a common ground and there is value in it.
So the outline for my talk is in the context of whatever I have said, about this common
ground. I'll show you three studies where we have worked, and I have tried to pursue an
agenda of getting both the attention of software engineering research and also
simultaneously in technology management and how it has panned out. Let's start with
the first study. The context for this is I was invited to this company. It's not a real name
but it's a firm in the test and measurement industry. They are a fairly technology and
research-oriented firm, and by nature of their business they always stay one step ahead
of consumer products because they produce the test and measurement equipment for
these other firms to build these consumer products.
Now the context of this firm is that -- So the regular software development lifecycle: you
know, you build a product, release it to customers. And this is the vendors', the software
companies, route. Now users of this company have the situation that they can use a
product as-is or customize it extensively. The need for customization is because in the
test and measurement [inaudible] the standards evolve at a rapid pace. In this context,
this was five years ago when the Bluetooth standards were evolving rapidly. Not every
firm wants to use the same set of standards, but the products have their own limitations
so every customer wants to customize it to their needs.
In doing so they raise new requirements which the firm now has to address in the
evolution of its products. So that's the operational context of the scenario we have. The
question is: if a customer requirement comes to a manager, I could standardize it in the
next release of the product -- I could incorporate that feature in the product -- or say,
"No, we are not going to do that. If you want that need, we will give you some ways to
implement that feature for you alone." Whether you have to standardize it or let go and
let people hack your product to build their own custom version is the question.
So the dilemma is you get a customer's request, what do you do with it, yu know,
whether we standardize it -- And that means it's part of our work in the software firm -- or
let people or some part of our team customize it further. Now that has strategic
consequences. The question is: if we standardize it, when are we going to release that
feature? If we let others customize it and if we get involved, at what cost are we doing
this? And if we do so for a certain type of customers, should we standardize that feature
at some point for everyone?
So that's the dilemma in the decision making for the product manager in this setting.
Now how do we give insights? I was invited to provide insights on, "What do we do in
this context?" And the questions from a research point of view are how do we model the
standardization or customization question, and how do we take decades of work on
software estimation to provide guidance on customization costs?
Those were the research questions for me. And we talked about the empirical context:
that this firm has a software test and measurement product that has a hardware
component, software component and users. So data collection on this, I was fortunate to
collect over five years of data of the product's evolution. When I was invited it was in the
second year of its release, so from the first year we collected archival data along with -- I
was there at there at this firm for about six months to do interviews and collect rich data
on the entire period of this product's evolution which was from 1999 to 2003.
Before we dive into the modeling, this is what I noticed at [inaudible] level. From a
developer's point of view most of the customizations that are requested violate
standards, that's why they ask for this customization and request that if they are part of
standard, a standard offering would have addressed those needs. Now customers have
genuine requirements that actually violate these standards. If we were to address those
standards -- These bars are the standards, no violation, simple to do stuff. And as you
can see the time to develop it and to get that into the standard features is less. Any time
there is a violation of standards request, it takes a long time to satisfy those
requirements. But there are genuine customer requirements.
The more you postpone it, if the competition offers those features, it's likely that people
are going to shift to those products. So again here right from managers to the
programmers everybody is sensitive to this thing that whether a competitor offers it or
not. So you can see that whatever a competitor is offering those requests are
immediately satisfied. What does it mean? Now the behavior of programmers and the
team here is largely driven by market forces and customers. Now from a complexity
point of view and design point of view that may not be always a good thing because you
are building products and architecturally in some sort of a logical way adding features in
an incremental fashion, in a way that we can manage complexity. But there are these
external forces that impact these programmers and teams, so under this context is
where the modeling comes in the picture of what happens when these multiple forces
are impacting.
So some of the influential factors from just observation and talking to people are: the
standards are evolving; competition puts out features that we need to satisfy; and then,
sometimes the programmers want and the team wants to preserve compatibility across
different product versions; and the nature of change these customers are requesting;
and all forms of complexity -- logical complexity, data complexity -- that's involved.
So then I went to formally model this. This is just a boxes and arrows representation of
what is undergoing on all this. The decision here is this question of whether we should
standardize or customize a request. And then all these features impact current
customers whatever we do, the constraints imposed by standards and our own product
features such as preserving compatibility; all these have an impact. And as I said before,
market forces the desire to have some specific market share or our current market share
in this market.
And then, the standard control variables that we know from prior software engineering.
So that was our first modeling strategy. Yes?
>>: There's a question [inaudible] a couple slides ago, about the competition, the
requests getting fixed faster when there's competition. You said in the bullet point below
people were sensitive, [inaudible] sensitive to staying ahead of the competition. But if
they're adding features that their competition already has, that doesn't really put them
ahead does it? That just puts them less behind?
>> Narayan Ramasubbu: Right. So the tension here, they want to add features ahead of
competition but they are held back by these constraints that if the firm or the policy of the
team is to say, "We don't break standards or we don't break compatibility with our prior
versions," those are held back because the managers decide, "No." And then,
competition releases that and then now we want to respond very quickly to those sort of
features that we missed out on. So despite having a plan to build this product, suddenly
these twists happen and then, forgetting what we are going to do right now, we
reprioritize to go after those features we missed out on. So that’s how that cycle goes
on. They are sensitive to this but we can't stay ahead of competition all the time with all
the features. Some competitors don't care about or they have different constraints for
their policies. So that's the tension we want to model and how it plays about.
>>: Did you get an understanding of why it took less time when you had competition
versus if you didn't? Like, what was the reasoning, what was the driver there of the
difference?
>> Narayan Ramasubbu: So the context is the test and measurement industry, right?
So if we don't provide -- So for example this was in the Bluetooth protocol analysis and
someone would say, "I want to know more details about it." Maybe we are not offering
that feature. Now if we really want that, to build their product, it's test and
measurements. And they're going to say, "We'll go with someone else who provides
that." So the customers will defect essentially. They won't renew their licenses of the
product and they would go with a competitor. So loss of install base -- It's not like a big
locking mechanism you have with a customer. They can switch easily. Whoever
provides the details they want, they go with that.
>>: I was just trying to get an understand of like why there was the difference in the time
to develop the features when there was competition or if there wasn't.
>> Narayan Ramasubbu: That's what we want to find out. So that was at the aggregate
level. When I collected the data and plotted it, that's what I saw. And the hypothesis for
me is maybe this firm is not making the right decisions for all the questions. So there are
all these tensions are not helping them make optimal decisions, so that's what is leading
to all these difference. And hence the formal model to understand what's going on.
>>: Do you have any insight in which scenario the firm goes for customized solution?
>> Narayan Ramasubbu: Yeah, we want to get to those insights. The approach here is
we build a model, validate it and then, we can derive insights to say under what
conditions would the firm standardize or customize.
>>: And, sorry, another question. So you're talking about standards and new features,
and they're different things.
>> Narayan Ramasubbu: Right.
>>: All right. So in the standard constraints or in the standards, where do the new
features come here or features that the competitors have, in which box do they belong to
in this model?
>> Narayan Ramasubbu: The unit of analysis is that feature addition. So you're trying to
change your current code to add a feature. So we say, "What's the nature of that
change?" Are you re-architecting the product or are you...
>>: So control, right?
>> Narayan Ramasubbu: Right. Exactly. We control for the extent of change mandated
by addressing that feature. So when I say unit of analysis in our data, the customer
request to add a feature, that's its role in our data set. So you'll have hundreds and
thousands of those requests and see how the firm responds to those things. That's the
boxes and arrows. So what is the data set?
So the pro-customization features: this team as their market share grows they tend to
customize features rather than standardize. So as their market share grows, the idea is
the install base is so huge that can't immediately standardize things because it's going to
break for someone else if you don't do a careful analysis. So it's going to delay. Hence,
"Let's just fix and do custom things for people who are requesting, take our time and
then, build it into the standard feature." Again if the request seems to violate any of the
current standards, we don't standardize those features. We let people do hacking workarounds to build those features. Again, if there is a customer who is a strategic partner,
we can't say no to them. So that also leads to custom solutions. Now every time you
build custom solutions what it means is you are increasing the code base to maintain
variations in the code base, so that's complexity for the team. So all these factors are
driving the firm to make the team make more and more variations of the code base for
them to maintain.
The factors that help them reduce that variation or some interactions factors here. So the
market share is big but if you are willing to break compatibility with previous version and
if you can move all customers to the new versions without worrying about compatibility
then that helps the firm standardize. The product obsolescence in this case is we don't
worry about previous version; let them expire. Like how in the Microsoft scenario, we
don't worry about what we did for Windows 95 anymore or whatever we add now we
don't worry about people having those previous versions. So I'll give you more. Formally
how do we do this? We calculate the probability of the firm, of the team addressing the
request as a custom feature or a standardized feature. And all these variables, we use
regressions to predict the impact of each and every variable on that probability.
So the answer after the regression that we perform is represented in this table. Now
what I would like to highlight is we have market share here. The more the market share
the probability that a request will be standardized is lower. However, you'll see as we
impose compatibility constraints and standards constraints, the probability actually flips
to be a positive one. So if normally as the market share grows, let's not hurry into
standardizing things. But as soon as some of these constraints which are policies are
imposed, the behavior of the team changes.
That's some evidence to show that there is this interaction and tension between
engineering teams and other people who are not engineering, for example marketing
folks and project management folks. And using the entire model, my recommendation for
the team for when a new custom request comes, depending on the request, is if we can
say what would be the probability of this team to address this and release this standard
feature or customization. So for the marketing folks that means as soon as they get a
customer requirement they know, they can predict, they can have a probability of, "Okay,
how is this team going to respond?" and then say whether they should relax their
policies or alter the priorities of some customer request accordingly. So that's how I sort
of helped the firm to reprioritize the way in which they can address customer requests.
So that was good to address that question of standardization and customization. But
now let's say the most optimal thing was to customize a request. The next obvious
question for a manager is to say, "What's the cost of doing that optimization? How much
of that cost should be born by us and by the customer because we're adding more
complexity to it?" The first approach for me was to use these software cost estimation
techniques that are built by Barry Boehm and the COCOMO and others. But it wasn't
totally satisfying prediction by those models for a variety of reasons in this context
because we don't have these variables we saw, the compatibility constraints or, for
example, how much we know about this customer and their context. Those variables
weren't exactly captured in the current model, so I had to build a custom model after
taking COCOMO and add all these other variables of interest.
That's the next model that I built for this firm to predict customization costs in that
COCOMO style adding in the other features we know from the first model. So here,
again, it's the same -- The contribution is to just add the contextual thread of variables
into this to help the firm predict customer costs. I don't think you would have any
questions here. The more complexity, the more the cost. If you're enforcing a particular
compatibility constraint, that's the biggest driver of cost. So if the customer asks, "We
need a feature but it needs to be compatible with all other things that we have," that's the
prime driver of how much driver it's going to have for this company.
Once we have that together what this body of work contributes is it sheds light on some
of the standardization and customization decisions and highlights the tradeoffs that are
involved in this. And qualitatively you can relate it to the tensions between the
engineering team and the marketing team on what should be the priority of customer
requests and so on. The papers are for your reference. And pursuing the same strategy
for me to have a foot in both software engineering and management, part of the work
was published in, I think, FSE and the journal that ran it was a management journal and
extended version of this work.
So that was the first starting point for me to think about how actions of programmers
have an impact on the marketing division of the firm and how the actions of these two
interact to inform important things for the firm. From there I want to move on to -- So as I
thought about it, I needed a tool to analyze these actions in way that we can shed more
light on it, to answer questions of what are the impacts of programmers on managers.
Here the concept I'm going to present first is a bit of an abstract concept and then how
we can utilize it.
The abstract concept here I want to introduce is this thing called design move. Now what
are design moves? The definition of it is any actions that a programming team makes
that alters the structure of the code. So if you're deleting lines of code and that alters the
structure then that's the design move you are making. If you are adding a component,
that's a design move. If you're refactoring the whole thing of the base of code to reduce
let's say Cyclomatic complexity, that's a design move. The notion of moves is -- Actually
I was motivated from linguistics. So in linguistic analysis they have identified this thing,
the duality between the structure of a piece of work and the moves that writers make,
how they introduce characters, how they take or kill characters, how do they move from
one part of a story to another.
So that's where the first notion of moves is there in the literature. It was picked up by
some organization theorist to see, for example, moves like you are encountering now
which is restructuring the firm. So that's the move and a manager is making that move.
Now I wanted to use that notion of moves for programmers and see how we can relate
those things. So there is this duality between what happens in a firm to the structure of
the firm. And now we can see a similar thing: the structure of code to what programmers
do with the code. So the idea is we are able to trace design moves in a team then we
can see more about what is going on in the team.
More than having cross-sectional views of regressions or cross-sectional things about
metrics, we want to be able to observe and trace these moves over a period of time. And
by doing that my claim is we can talk more about the connections between programmers
and management decisions. So let me illustrate at a very high level the design moves
concept. So here is a product, an ERP product in this case. It has a server, clients,
database and features: accounts receivable, payable, assets, pricing features. So now
let's say the team builds a configuration engine for this ERP as a product development.
That's a design move; they build it. Then they decide that in every implementation of this
product, at clients, "We will start logging the configurations of them." So these are users
implementing this product at their site and through the configuration engine, the
company starts logging how people are using and installing this software.
Now using that archival data, if a new company comes and wants to install this software,
from mining the database we can say, "Given your situation, this is the most optimal
configuration of this product." Now for ERP systems especially, installing configuration is
the biggest cost of making the people use the software. So by automating that process
this particular team is able to reduce that time by saying, "Someone else has this
configuration. That's the most apt configuration for you, so why don't you just adopt that."
That's all technical work and those are all design moves. Now there's the strategic
consequence of the move to the firm; you're able to reduce the cost of implementation
which is the biggest cost in this industry for this setting. Now you're also able to reduce
the reliance on these third party functional consultants. So let's say you are selling ERP
products. The more you are able to reduce these functional consultants from outside, the
clients are going to face lower cost of implementation then they are going to have a
positive experience with your products. So all those -- If we are to trace those moves
and connect it to performance measures here then we can see what is the value of all
that work those programmers have done to the firm. So that's the idea of connecting
design moves to performance measures we don't think of usually in software
engineering.
You know we would have thought about complexity metrics, productivity of
programmers, quality and defects. But we want to make an argument about our work as
programmers and if we connect it to these things then that becomes a powerful ability to
make a case of what type of work we want to do. That's the idea of this whole concept of
design moves.
Now there are more examples, if you want me to illustrate, of how we can connect
design moves. Do you want to hear more examples of connecting design moves to a
higher level strategy or we can just skip to more hypotheses and the data collection
part?
>>: I think examples are good.
>> Narayan Ramasubbu: Examples are good. Okay. So we saw the configuration
engine was the first design move we studied. Then, the team makes these extensions to
these features that are components that are in-built. The idea of extensions is we will
develop APIs for other people to build extensions. It need not be us. So all it requires is
good well defined interfaces and APIs for other people to build. So once we have those
extension points release, people can create their own code to generate whatever reports
they want out of this product. We did nothing for that; people can create their own
reports and people can even modify some of the processes through well defined
interfaces.
Normally if we are to do that, product managers are going to, say, push back and then
say, "Why would you want to do all of those things? You can just satisfy customers
through our standard releases." But if are to connect the design move to cycle time of
fulfilling customer requests, like we saw in the first [inaudible], if we are to reduce cycle
time then we can connect the value of this design move to firm-level performances of
cost reductions and cycle time. Now all this was a positive experience, right? So the
design moves: I purposefully illustrated only good things about this. But it might backfire,
some of the moves, as always in real life.
So here let's say we found a bug and we want of fix the bug. The minute we have all this
custom-written code, we have a problem. We don't have the source code with us. We do
not know how people have implemented all of these extension points. So a design
move, which it has positive things going on, might actually break certain things also.
Those are the negative consequences of some of the moves we make. We do not have
control over the quality of these custom things that people build which can --. Now when
people experience those things, all the errors are still going to be logged to us as the
vendor. And end-user facing trouble is going to just log it to the vendor not to their own
team to say, "What we did broke the code."
Design moves, if we study at that unit of analysis then we can see positive things,
benefits and also bad things. So in the literature, here is where the bad things are being
termed technical debt. Whatever we do, sometimes we accumulate complexity that can
prevent us from doing good things in the future.
In the evolution of this some refactoring happens in this product and we have the
programmers doing other things. So in this example of refactoring, again, the problem
we faced before of bad quality code that is causing us trouble, the developers are trying
to address that by having some sandbox, well controlled things for third-party
components in their next design move which can be good because now you are able to
reduce quality impacts of third-party applications onto your code.
So we saw three design move examples. For this piece of work I just illustrated from one
case but in the paper I report four different companies and the moves I observed over a
period of time. To organize all of these moves I created a conceptual map. On the X axis
is what I call an option value. An option value is the benefit, the potential benefit that can
be reaped from a design move a programmer makes. So this goes back to the idea of
real options theory which is, like in financial options, any component or rebuild might
have a potential benefit later on as I illustrated in the design move.
At the same time it might add to complexity and that's what we call technical debt. So
any move a programmer makes we can map it to say whether that increases the options
value of the product or it increases the debt of the product. Now if you see that in that
fashion -- It resells in four quadrants. So let me start with the easiest quadrant here.
Normally when the product is new early in the lifecycle you'll see not many options, not
many features built in the product, but the product is also relatively less in complexity. So
we have an option constrained state: low on value, low on debt.
And the other quadrants here, everybody wants to be on this quadrant. Right? We want
a product that is rich in options but low on technical debt because that's the most
desirable state. But we can't be in that state all of the time because of all the tensions we
talked about. So we might move across these states. When the debt is high and the
options are high, what we have is a situation of lock-in. We have good options but the
complexity is so high that we constrained by our debt. Here is a bad state; our
complexity and technical debt is high and we don't have many options to move around. It
is a nightmare for programmers to be in this state. Most desirable state, we often might
end up here because we build rich features but end up in complexity. So for all the firms
I studied over, I plotted the moves they made on this map.
This is the raw data I observed on the moves: what they did, what was the state of the
design capital. It's difficult for you to read but to show that it was systematically done for
all of the design moves. The resulting picture is something like this.
>>: Like what was the scale, how did you measure? It seems kind of like a -- It seems
like a hard thing to quantify.
>> Narayan Ramasubbu: Right. So that's one of the challenges. We need to have -- We
can do this only after time. So now I had the luxury to go back in time and then say, "I
see that the structure of the code has changed. What did you do to the code?" And then,
we analyzed each of those major changes to the code to...
>>: [inaudible] why has the code changed or something concrete [inaudible]....
>> Narayan Ramasubbu: So the first thing is lines of code. When I [inaudible] two
versions and see big changes then I reason about it. What is that? Similar deletions of
components also draw my attention. But then we need programmers to help us identify
why this is, so that's why this is a messy process, identifying the designs. And there is
some aggregation too. Like when I see two or three things going on at the same time, I
lump it into one design move. So they might have built two or three features on the same
bunch of changes to the lines of code and it's very difficult to segregate that.
This is the resulting picture of that analysis of the moves. You'll see here the circling
here means that the product team is trapped in this state. They're trying to get out. Now
let's take this company which is in a cell phone apps situation. This is clearly a bad state
and it's good that the firm is never in this state. They start with low options and low debt
and then they immediately, by this move they are making, go to the high quality state
which is good options and low on debt.
But it's transitionary. As they start adding more features their debt increases but the
options don't. Now from this state they keep on -- whatever they do, it's difficult to get out
of that state for this team. So when we do the circling, it requires tremendous
investments in this move to get them back on this state. So we all these transitions that
happen here show the behavior of the team and the interaction of where they putting
money in to resolve the situations. So what this helps -- This is a historical perspective.
We can plot such a map for Windows or Windows Phone and say, "Well, historically this
is how we've been transitioning around." Well, what's the goal for your team? At what
state are you right now? Where do you want to go?
Then, we can reason about what sort of investments you have to make to get the team
going there. So -- Yeah?
>>: In the fourth figure...
>> Narayan Ramasubbu: Yes.
>>: Why have they started with [inaudible] like low quality?
>> Narayan Ramasubbu: Yeah. So in that company basically they acquired a product.
>>: I see.
>> Narayan Ramasubbu: From someone else. After the acquisition they realized that
actually, "This is a bad situation." And it was a merger case where two firms were
merging and then their products, they are trying to consolidate. So your starting point in
any of these cases might be in different places depending on which window we are
studying for the product.
>>: And how did you measure the value? And were you measuring the value of each
move or just like the total value of the product?
>> Narayan Ramasubbu: No. For each move. Now how do we assess the move? I
illustrated the design moves and everything had a strategic consequence, right? So like
if we had a custom extension point we can significantly correlate it to a reduction in cycle
time. Now I asked the managers, "I see this correlation, is that high value or low value
for you?" because I don't know in their situation whether that was a significant thing or
not. I asked the managers to record whether it is high or low value by giving them
evidence. Some of it is they don't know because it was a long time ago, so I give them
snippets of my evidence from data and make them record it. But the debt I can record
because I see how much complexity is in-built.
>>: Do you measure debt in terms of bugs? Or how?
>> Narayan Ramasubbu: No, debt is in this case -- So the definition here is how much
does it prevent the team from adding a feature. So it's increase in complexity. The
quality -- We can correlate debt with quality as I will show in the next study. But for this, I
did not do that just increase in complexity basically on the code base.
>>: I don't know if this question has been asked but in this diagram, is it significant that
some of your -- or is it meaningful that within each quadrant it's a little bit closer
[inaudible] axis?
>> Narayan Ramasubbu: No, it is not. It's just a representation. It doesn't matter where
you are here. That's just my error of starting somewhere. It has nothing -- All of this is
equal state space. Only the transitions matter.
>>: And then in your analysis did you have a sense that something would be further or
closer to the axis? Do you think that would be meaningful to really dig into that more or -?
>> Narayan Ramasubbu: Yeah, so that's the future work. So right now this is a
qualitative piece. Although it's based on data, I haven't done any formal modeling of that.
>>: You kind of give a rough estimation.
>> Narayan Ramasubbu: Exactly.
>>: [inaudible]
>> Narayan Ramasubbu: Exactly. So it's the binary division, low-high. I haven't done
more work to say relative low or relative high. That's what we want and that's what I'm
encouraging. I'm telling it as a common ground. If we do that then software engineering
research can tell me exactly how we can do that and we can inform values for
managers.
>>: Is it on purpose that they all finish as -- the story ends well? They all have high
quality and low debt?
>> Narayan Ramasubbu: Yeah, so all of these four firms that are here, like each of
those four firms are successful firms. And in this circle story here all of them want to
build a product platform, and all the product platforms I observe are all successful
platforms. Like this company has a mobile application platform that's hugely successful
and so on. That company is a services company that's hugely successful. That's the
indication that all of them ended up at the time I ended my study in successful states but
they did not have the similar paths to success. So the idea here is that the paths to
success informs us about something, the struggles that they face to get there. So if you
see this company, they go in cycles. It's cyclical path towards where they want and so
on.
So if we extend our window of observation, everything is transitioning. There is no one
permanent place. Right now I can tell you right now this firm is starting phase out from its
success, the peak, and now they have to make some moves to get back to this state.
>>: So I was sort of asking this as in, can you identify a company that has probably
some design issues because they picked the wrong features and they keep yo-yoing
between low quality with lots of debt, little debt, lots of debt, little debt. And it just yo-yos
[inaudible]...
>> Narayan Ramasubbu: So that's that situation here. Like whenever this firm -- I call it
a debt averse firm. Like, whenever they sense that their technical debt is increasing,
they want to get back to some other state. But remember because of so many tensions
you can't just avoid [inaudible] so they go back and forth. But I don't have in my sample
where the firm just goes here and doesn't transition to the right side. That clearly shows
this is not a successful team and like any research that's the hardest thing for them to
disclose to me and let me observe. And when I went in here where all were successful
and I tell them, "Let us observe your history," they're okay to let me do things. That's a
limitation of this research.
>>: Is there concern about kind of self-reporting bias? I mean [inaudible] qualitative and
stuff like that. You know, everybody wants to feel probably like you're in a better place
than you used to be. Like it's pretty rare for a company to say, "No, we're terrible. We
used to do things great and now we do them really crappy." But when in reality that may
actually be the case. Like companies don't stay healthy forever; eventually they degrade.
And you mentioned the [inaudible] company was starting to lose their ability to stay on
[inaudible]...
>> Narayan Ramasubbu: Right. So...
>>: Like they still probably self-reported. It would take kind of hindsight for them to ever
see the defects when they actually recovered. So how does that impact kind of the bias
of these kinds of reportings?
>> Narayan Ramasubbu: The limitation here is in my window of observation, the ending
point was successful. That's what you're all pointing to. But in their history there is not
bias. Like you will see that people -- These two firms especially are in the worst states in
these places. And in hindsight I was able to point to that fact. Now if I had gone while
that was happening, maybe it's difficult for me to identify that. So collectively I don't see
that the sample is bias in that perspective, but the limitation is, yes, this sample does not
have a complete failure where the firm is just stuck in this quadrant all the time and not
able to move.
>>: Like kind of how I -- It's somewhat relative, too, really. Over the course of five years
the company may be in a high-quality state and observe a high-quality but three years
down the line they may realize that that was insufficient. And they're even more
successful and so they move from what they considered at the time as high quality to
even more high quality. So their hindsight has not changed their view. It looks like
[inaudible] like how can you tell whether [inaudible] high quality or low quality when the
actual [inaudible] shift from where they are.
>> Narayan Ramasubbu: So we cannot. And that's why there's, you know, the desire to
do more formal modeling between different states in this quadrant. It's all relative. But we
need to start somewhere of conceptualizing all of that. And I'm hoping this is a good
starting point, to see things like this. And then, now we can encourage research to say,
"Where's the next step after this?" How do we do things to understand those
transitioning factors and dynamics behind it? And I'll show you a little example in the
next study where I am attempting to do such a thing.
>>: My idea was just that you take it purely from a subjective perspective, where did the
firm see itself? Did it see itself in a high quality state? And that affects its choice to try to
move what it perceives as a higher quality state. Like if they feel like they're stuck...
>> Narayan Ramasubbu: Right.
>>: Or you can take it from a more, I guess, objective perspective if you have a concern
about the criteria for what is high quality.
>> Narayan Ramasubbu: Right. Now the only qualification for what you saying is that
this is not totally subjective, like the technical debt. I drew a line to say from the
complexity what is low and high, but that doesn't mean it's subjective. We did observe
code bases and we built from there. And this codification of low and high is arbitrary now
of what is low and what is high, but it is built upon evidence from data. So it's not
arbitrarily done, you know, like that. But we can build more relatively robust metrics to
formalize this classification.
Okay, so now from a theory point of view using that and these transitions we observe
here, I laid out some propositions. So when do programmers reduce debt? When do
they, you know, increase options? So based on just qualitative observations we can see.
The question here is I used the term resource munificence. What this means is there
enough money for the programs to do slack effort or is it a conducive environment? So if
you want to reduce technical debt that means you have to invest in refactoring which
doesn't yield immediate business benefits for a product manager, right? So do they give
the slack to do that refactoring?
We can say under this condition of high or low availability of resource and capability of
the team: if a firm is stuck in this situation, how would they respond? So if they have
good resources but low capability, these four teams seem to create more options without
reducing debt which is still better state. That means they would go here into this state
but not the perfect idea. Always we want to end up in that quadrant of low debt and high
quality. Again, the limitations of these propositions are based on qualitative observations
of the four cases. But in the future what I want to do is use data to validation these
propositions and hypothesis in a better way.
Another thing that I want to point out that's an interesting dynamic in this is your ability to
transfer debt. Now what does this mean? We build a product but we transfer the
responsibility to reduce complexity to other partners. For example, a technique that one
of the firms uses is they build a product and let the open source community deal with the
product. To further build new variations they'll let the community reduce the debt for
them. So that's the strategy of this firm, to remain in that state, to build products very
quickly and then let maintenance, through either open source communities or with
clients, -- they would transfer the responsibility to them. They don't take any risks. So
how do you stay in that position is a strategic question. So I call that, does the firm have
the ability to transfer debt? To remain in that state it's impossible to always keep debt
low. So whenever it accumulates, what's the strategy to reduce it or transfer it to
someone else?
And the paper, I don't have a software engineering paper for this but the paper is
published in a management journal, this design capital map on which you can observe
these transitions. I think it's an interesting idea for the software engineering community.
Now I want to finish the talk by highlight a most recent work of mine based on the
previous conceptualization of debt. The idea is to build a model of technical debt, an
evolutionary model that can highlight costs and benefits of technical debt. Think about
technical debt like taking a mortgage. You know, you buy a house, have a mortgage
that's debt, but at the same time you enjoy the house; that's the benefit. Instead of
having this only negative connotation about technical debt, I think we need to start
thinking about debt in terms of benefits and costs. So that's my aim to build a model like
that. The starting point for the model is this widely accepted and validated pattern of
customer adoption. So it generalized the results in lots of management studies across
products ranging from shoes, iPods, refrigerators. Anything consumers are involved in
seems to follow this pattern, an S shaped curve.
There is a point in time in the evolution of a product where this sudden growth happens
and that I call the take off point. And an inflection point is exactly the halfway of this S
curve; it's just a theoretical point. But the saturation is an observable point where the
interest in the product, the growth tapers off. So that’s the generalized that we know.
The argument is, can we apply this for software products too and for software
engineering? If we know the products are going to grow like this, features are going to
grow like that in a product then we can make use of that pattern to assess the cost and
benefit of debt. The theory I have here is: so this is the same S pattern that we saw. Now
imagine a team is taking on big time technical debt in order to deliver features faster
than this curve. So we want to expect that we're accumulating debt here, but you have to
pay interest on this debt some time. Now how we would model that is we are expect
because of this technical debt, high technical debt you're ability to keep on adding
features is not going to grow all the time; it's going to taper off far more quickly than you
would have done on this pattern.
So if we model it like that then this the benefit of your debt but that's the cost we'll have
to pay for taking on this debt. Conversely, you’re a firm that does not like technical. You
don't want to accumulate debt. In real life you're not fulfilling product growth as much as
the market is because you don't want to take this debt. Now that's the cost of the debt
because you're constantly lower than what the market is doing. But the benefit you could
realize if suddenly you shift your strategy at some point in time towards the end of the
product growth, your complexity is so low that you can start adding features in a faster
way. If we are able to do that then you could realize the benefits of this late surge in
features.
So that's the theory. And the S curve formally would be a logistic curve, and these
curves can also be modeled as logistic curves or an exponential saturation or that one is
an exponential growth curve here. Once we have these distributions now then we can
model and study products. So the one I studied empirically is this ERP product. The
base version is put out by the vendor. I observe across their customers some are low
debt customers; they don't want to take debt. Those are the low debt version. And
opposite to that is we have a high debt version where customers take this base product,
hack it, customize it, build a lot of other features they want. So across a sample of 48
customers I observed I had this variation here: 11 of them are just following the base
version that means the S curve, and 15 are in low debt versions, and 22.
So I plotted all of them together to see whether we can draw some insights. This is the
base version. You will see that on the Y axis I have the cumulative functionality at any
point in time, so that's ten years worth of data. And each month I see what are the
features embedded in this product that they have. Now accumulating the features and
measuring is a tricky business. We do not know what a features is, right? So in this
product they call it a transaction. A transaction would be like a command line executable
that you would see. So a transaction, for example, would be, "Open a sales order," or,
"Reconcile accounts." If your users [inaudible] enter that transaction code and then it
would run. So cumulative transactions that can run on this software are on the Y axis.
And time? After ten years. Month-wise aggregation is on the X axis. Question?
>>: Yeah, was it something that the team would say they consider a feature or
functionality or did you put [inaudible]?
>> Narayan Ramasubbu: No, the customer. So all these transactions are -- They record
those transactions. I just take the cumulative number of transactions that are available.
This is not a programmer point of view but an end user's point of view like how we would
normally do on function points. The end user facing view of functionality, not a
programmer's view of features.
>>: So that's just a total transaction, right? This curve seems a little -- I mean, it's just
kind of adoption, right? Like if they use the same feature over and over and over again
more frequently in the future then the curve goes up because you [inaudible]
transactions [inaudible]...
>> Narayan Ramasubbu: Oh, these are unique transactions [inaudible]...
>>: So the number of unique transactions?
>> Narayan Ramasubbu: Yes.
>>: Okay.
>> Narayan Ramasubbu: Like early the cumulative transaction keeps growing, that
means the product is evolving and programmers are adding transactions. But it hits the - You know, the slope of the curve changes over a period of time. It's slowing down. So
that's our S curve that we could -- You can model this curve using the same logistic
curve.
>>: So I don't know at all about the domain of the product, I guess, but could it be that -So is one thing that sort of debt, right, like they can't add anything because they just
screwed themselves, but the other one is maybe they considered the product was just
complete in some way or like as a whole. And they were like...
>> Narayan Ramasubbu: That's right here. That happens there, right?
>>: But is it because -- Like will they stop adding features because their product is
complete and they don't have debt and they just keep a couple people for maintaining it
or adding eventual plugins? Or is it because they took so much debt before that now
they have [inaudible] a year to refactor and take over again?
>> Narayan Ramasubbu: Yeah, so we'll see that.
>>: Okay.
>> Narayan Ramasubbu: But this is the base version, like how the vendor is putting it
out. So most likely this saturation here is like how you pointed out. The vendor feels the
product is complete, any more needs to be a new version of the product. "We are done
with this, and for the next three or four years we are just going to maintain the product."
So 11 of the customers in our sample just followed the pattern. They don't do anything
more than what the vendor does: take it, use it. You can build your transactions. That's
where the debt comes into play of how you build those transactions.
So these were the vendors and you'll see the 11 who closely follow the vendors' idea.
Now let's move on to -- Here is a pattern of customers' installations. It goes like this. This
curve is a vendor and the 11 customers I showed you. These are the variants that are
above the curve here up to the inflection point and after that they are on the lower things.
So here, like you pointed out before, an inference on this pattern could be that why this
is tapering off early and remains below the average product. Now we do not know
exactly why but one interpretation could be that they are burdened by debt possibly.
>>: Or that they consider it complete.
>> Narayan Ramasubbu: Yes. Now...
>>: By complete you really just mean that the return on the investment for new features
no longer makes financial sense so they stop adding features.
>> Narayan Ramasubbu: Exactly.
>>: It's too expensive or they don't enough value.
>> Narayan Ramasubbu: So it's good that you are all thinking about that because that's
the next state of analysis. So right now I'm just showing patterns and then, we can say,
"What about quality? What about customer satisfaction at different points in time?"
>>: So in that base version in that case they're getting the features from the original
vendor that's [inaudible] that. And in this case they're not able to roll out those new
versions...
>> Narayan Ramasubbu: Yeah.
>>: ...at some point? Oh, okay.
>> Narayan Ramasubbu: Because roll out is now your -- Remember I said transfer debt.
So the vendor is not going to do anything for you. You hacked the product. You add debt
now; now it's...
>>: You're on your own.
>> Narayan Ramasubbu: ...your responsibility. In my base versions I have more
features but now you're not able to use those new features because you're on a different
trajectory. You have to undo all the way here and then build up to go there, if you want to
do that.
But here they did get benefits, right? They build far better than the base vendor could do
using their platform. So that was the idea of there is a benefit to debt. Now we have to
quantify that benefit. How do we do it? That's the next question.
>>: Right. Because, I mean, immediately I originally thought that your curve was like
market share or...
>> Narayan Ramasubbu: No, no. This is just cumulative transactions.
>>: So then you don't know at this point whether one of those is like because they
hacked it they got a big jump and they [inaudible].
>> Narayan Ramasubbu: I can make informed guesses and I'm going to do that in a
minute. So the other story just for -- If I plot everything in one, it's going to be crowded.
That's the only reason I'm splitting it into different things. So here is another group of
customers. You'll see that they are below. That means they choose not to install those
components for a variety of reasons.
>>: They didn't even take the time to install the new version?
>> Narayan Ramasubbu: No.
>>: Got it.
>> Narayan Ramasubbu: So they are going to -- So in this product you can choose
components to install, like you can choose to install a Windows patch or not. You can...
>>: They didn't even deploy.
>> Narayan Ramasubbu: Yep. They don't deploy. So they don't deploy but then they
start surging beyond also. So unrolling does happen. You'll see here that they're here
but then they take it off. They unroll the features they built. So you'll see all these curves
not smooth because of all those actions. They install some [inaudible] and then, they
take it back. So here's a group of customers that go like that.
From this pattern we go for ideas like that. You know, you can calculate this area under
the curve here, area under the curve here and here to make some calculations for us to
see.
>>: Is it that they're doing some modifications? Is that how they continue to go beyond?
>> Narayan Ramasubbu: So this shift here, they made more modifications. So here
from a strategic angle, beyond this point the vendor is not doing justice to the cost. They
are not adding features for almost five years; they are just maintaining the product. "You
want to benefit from all of this then it's on your own," then those guys are able to do. And
here these guys aren't. They are even behind the basic thing available.
>>: Got it. It's understanding the person that's building the platform you're working on. If
they're investing at a large level, you scale back your investment and you just ride that
wave.
>> Narayan Ramasubbu: Right.
>>: And if they're not then you do it yourself....
>> Narayan Ramasubbu: Exactly. But you have to be in a position to do that and that's
our argument about debt has implications. So now we want to answer all these good
things you were mentioning, is it really valuable and what not. For that we need metrics.
And here it's still a work in progress. I'm going to show you two sorts of metrics: one on
quality, all the things about bugs and errors resolved, time to resolve an error, that sort
of metric; and then the customer satisfaction metric.
The customer satisfaction metric is relevant because this is answered by the end user
not the programmers. And this is collected by the vendor, not by me. So they collect
every month these surveys to see, "How satisfied are you with our product?" So, "You
want to install a new feature? First you have to answer this question and then you will be
able to install those features."
Imagine it in a case like, for installing hard patches from Windows you have to answer a
survey first before you'll be able to install any of those features. So that's how I collected
it from the vendor this satisfaction information. Then I plotted it at three points in time.
And this is my first take at the data. So I showed you the take off point, the inflection
point and the saturation point. So I'm showing you a snapshot of the data across these
three points. This is on the quality metrics, the three metrics: the processing time of
errors, number of errors that are still unresolved, and the percentage of errors exceeding
service level agreements. On all these metrics you will see that the high debt people are
doing bad which is what we would expect. The firm build a lot of debt, technical debt,
that means one of the major implications of that is quality is going to suffer and that's
what you see as the bad part of the debt.
And the low debt, the benefit is you're able to resolve it. And it's normalized; that's why
you'll see zero means they're equal and negative means they're so much relatively better
than that.
>>: So when does this graph come from? Five years down the road after launch or
fifteen years down the road?
>> Narayan Ramasubbu: Well at three points. The first is the take off point which is the
month here. So the take off point is this T1. The inflection point is here. And then, the
saturation point. So these are months for us. You'll see the month for take off is 37 and
the inflection point is about 60. And that's 73 months.
We can do better but right now in the first card I'm just doing three points.
>>: So that means that after three years the companies that go for high debt...
>> Narayan Ramasubbu: Are going to do so much poorer on quality. And they remain
poorer throughout. So let's go to customer satisfaction which tells a different pattern, an
interesting pattern. People who are in high debt initially are super satisfied; the
customers are very happy because they get new features despite poor quality. But it
starts going downwards soon. So about midway into growth, it goes down from super
satisfied to unsatisfied to totally unsatisfied. Now the low debt forks initially super
unsatisfied. No features, you know, they are not even installing good features available
on the product. The growth phase is even bad but then I noticed in the data the switch
happens that towards the end of the product line they are the most happiest people. So
that's what this data shows. Now again the limitation is I just saw three cross-sections.
We have to see more closely to see how that switch happens, what is the magic that
suddenly happens for this switch.
But what I am seeing are interesting patterns here that we need to look more closely
towards. So far what I've done is in the aim to build this connection between what
programmers are interested in and issues related to programmers to higher level
metrics. And in the first study I introduced complexity and how complexity influences
standardization and customization decisions. And then, a concept on the design capital
map using technical debt and options. And the third study takes that concept even
further to say now how do we study technical debt and the value of it. And then, I want to
go ahead and build models for options and debt together and understand these patterns
in a better way.
That's my aim going forward. Now I showed you pure cases but obviously there are
challenges. Now you'll see transitions. Here some customers are above but they then
transition to some other trajectory. So those are challenges for us in this research on
how to handle these transitions. There are more interesting transitions. So here you'll
see this is a base version. Someone is above all this but they'd refactor the code to get
back on trajectory. I don't have exact data on the three metrics to show for these
transitional cases. But all I know is all of these transitions, even if they get to the better
trajectory, are poorer. So the data shows that every time a transition happens there is a
fall in customer satisfaction even if you transition to a better state in the code.
So that's says something about the change in management and how to handle users,
but that's my current work to analyze that more deeply. That brings us to end of my talk.
And the goal, again, for my research agenda is to build this common ground of issues
and software engineering and connect it theories and questions of interest to managers.
And hopefully I get, through my research, both of the communities excited to the
concepts I'm build. Questions?
>> Nachi Nagappan: Let's first thank the speaker.
[applause]