Onboarding, Finding experts,
maintaining awareness
Kun Niu
Spring, 2011
Feb. 22nd
05-899D Human Aspects of Software
Development (HASD)
The Information Networking Institute
is a cooperative endeavor of:
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
College of Engineering
School of Computer Science
Tepper School of Business
Heinz College
1
Agenda
 Overview
 Paper discussion



Onboarding
Finding experts
Maintaining awareness
 Summary
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
2
Agenda
 Overview
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
3
Concepts
Onboarding
- The process of helping new employees become productive members
of an organization
Finding expert
- The process of finding the person who is expert in a specific part of the
software project
Awareness
- An understanding of the activities of others to provide a context for
one’s own activities
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
4
Why do people do research in these
fields?
• Find different facts hindering newcomers in joining the
project efficiently – Onboarding
• Find experts in the software development group and
make best use of them – Find experts
• Keep track of what software developers did in order to
know what the developers have done to the code and
why they did that – Maintaining awareness
All in all, make software development team more
productive.
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
5
Sample target for studies
• Students in the university who are ready to work in a
software intensive company
• Developers who work in a collaborative environment
(open source software or commercial software or
academic research, etc.)
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
6
Agenda
 Overview
 Paper discussion
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
7
Agenda
 Overview
 Paper discussion

Onboarding
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
8
Struggle of New College Graduates in
their First Software Development Job
• Andrew Begel, 2008
• Discover what occurs during the beginning of the
transition period from college graduate to experienced
software engineer
• Spent 85 hours observing 8 new software developers
(NSDs) in their first six months of employment at
Microsoft Corporation
• Result
•
Five ways in which recent graduates struggle to be effective:
communication, collaboration, technical, cognition, and orientation
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
9
Moving into a New Software Project
Landscape [Barthélémy, 2010]
• Grounded Theory
•
A qualitative research approach that employs theoretical sampling
and open coding to formulate a social theory “grounded” in the
empirical data (exploratory study)
• Three instruments for research
•
•
•
Initial Survey (of people in IBM Quality of Software Engineering)
Project Sketch (Asked them to describe their projects)
Phone interview (One-hour semi-structured)
• Participants
•
3 women and 15 men, experienced programmers and novices
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
10
Landscape features related
to orientation aids and
obstacles
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
11
Comparison Between Papers
• Similarities
•
•
•
Coupled with orientation and collaboration
Struggle in using document to gather information
Difficulty in working in a corporate environment
• Differences
•
•
•
•
•
Field study vs. multiple stages
Novice vs. Novice mixed with Experienced programmers
Where Novices get stuck
Without feedback vs. with feedback
Microsoft vs. IBM
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
12
Agenda
 Overview
 Paper discussion


Onboarding
Finding experts
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
13
A Degree-of-Knowledge Model to Capture
Source Code [Thomas Fritz, 2010]
• The degree-of-knowledge (DOK) Model
•
The DOK values are computed for a developer automatically by
combining authorship data from the source revision system and
interaction data from monitoring the developer's activity in the
development environment.
• Goals
•
•
Investigate whether DOK values can support finding who is an
expert in particular parts of a code.
Investigate whether DOK values can help familiarize (onboard) a
new team member onto a particular part of the development project.
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
14
Exploratory Case Studies
•
•
Site 1 involves seven professional developers building a Java
client/server system and using IBM Rational Concert as the source
repository
Site 2 involves 2 developers building open source frameworks for
Eclipse as part of their work and using CVS as source repositories
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
15
Factors that might correlate with
expertise
• Degree-of-Authorship (DOA)
Determined by three factors: first authorship (FA), the number of
deliveries (DL) and the number of acceptances (AC)
• Degree-of-Interaction (DOI)
The amount of interaction — selections and edits — a developer has had
with a source code element
• Degree-of-Knowledge (DOK)
Formula linearly combines the factors contributing to DOA and DOI:
DOK = α FA * FA+ αDL * DL+ α AC * AC + βDOI * DOI
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
16
Analysis and Results
• Formula factor computation
•
•
DOK = 3.293 + 1.098 * FA + 0.164 * DL – 0.321 * ln(1 + AC) + 0.19 *
ln(1 + DOI)
Negative values of DOI indicate usage that is not recent
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
17
Case study results: Finding experts
• Method
•
•
•
•
Two projects from Site 1
Most members of the team interacted with these two projects
Calculate DOK for class-developer pair
Calculate DOK for package-developer pair
• Results
•
All six developers stated that the result was reasonable, using
phrases like it is “close” and it reflects [reality] correctly.
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
18
Case study results: Onboarding
• Method
•
•
Three randomly-chosen developer
Interview and survey
• Results
•
•
Elements with high DOK values are not considered useful
Elements with high DOK may be a good starting point to find useful
elements
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
19
Expert Recommendation with Usage
Expertise [David, 2009]
• Assess the viability of usage expertise concept
• Evaluate expert recommendations for the ASPECTJ
and ECLIPSE projects
• Collect data according to usage context, depth,
breadth….
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
20
Experiment
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
21
Results
• Context improves accuracy of usage expertise
•
This means only a handful of developers contribute the bulk of
contributions for most methods.
• On Recommending Across Projects
•
There must be sufficient overlap in the calls to external libraries. By
extension this means that recommendations are at the very least,
possible.
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
22
Expertise Browser: A Quantitative Approach
to Identifying Expertise [Audris, 2002]
 Application range


Locate experts in a particular technology or tool
Locate experts in a particular part of product
 Interviews with members of distributed project teams
 Calculation using EA (“experience atom”)


Elementary units of experience
The simplest unit of experience that could be observed in projects
using change management systems is the atomic change (delta)
made to source code or to documentation.
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
23
Learning curve
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
24
Expertise Browser (Java Applet)
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
25
Analysis of Expertise Finding
• All of the papers try to quantify expertise analysis
• Are the selected data really helpful?
• Does change of code really reflect the expertise in the
code?
• Does it apply to other parts of the software
development group? (product design, QA, customer
service…)
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
26
Agenda
 Overview
 Paper discussion



Onboarding
Finding experts
Maintaining awareness
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
27
Hipikat: A Project Memory for Software
Development [Davor, 2005]
•
•
•
•
Provide developers efficient and effective access to the project
memory
Focus on open-source software projects, a subset of virtual teams
that typically have extensive public archives of all artifacts
relevant to the project
The tool forms a project memory from artifacts that were created
during a software development project (source code,
documentation, and electronic communication records)
Hipikat recommends to a developer artifacts selected from the
project memory that it considers to be relevant to the task being
performed
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
28
HipiKat Server Architecture
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
29
Eclipse newcomer case study
• Goals
•
•
•
If project memory about past modifications helpful to the developers
When and from which artifacts are queried
Evaluate the recommendation result
• Sample
•
•
Have adequate programming experience in the programming
language of the system under study
Have no previous experience as developers on the system itself
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
30
Eclipse newcomer case study (cont)
• Design
 Easy task – Display break points property
 Difficult task – Interactive modification during version operations on a
group of files
• Procedure
 Training
 Programming
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
31
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
32
Results
• Less access during the easy task than the difficult task
• Focusing on solving task rather than understanding
code
• Using recommendations to find the code could be
reused
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
33
FASTDash: A Visual Dashboard for Fostering
Awareness in Software Teams [Jacob, 2007]
• Use electronic tools to operate in a shared virtual
environment
•
Hard to observe activities
• Lack adequate tools to support sufficient ongoing
awareness
•
Hard to determine who is working on which file
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
34
Fastdash Interface
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
35
Field Study
•
•
Observational study of a software team’s collaborative behavior before and after
the introduction of FASTDash
Leverage the earlier work to develop a new coding that was a synthesis of the
most relevant parts of existing schemes and basic requirements
Coding categories and classificatoin
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
36
Observation Log
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
37
Result
Communication category distribution for pre- and postconditions
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
Use of shared artifacts
School of Computer Science
Heinz College
38
Group Awareness in Distributed
Software System [Carl, 2004]
•
•
•
•
•
•
A study of open source teams to determine whether developers need to stay
aware of one another
Interviewed fourteen developers on three different well-established OSS projects,
examined email and chat archives, and analyzed project artifacts such as sourcecode repositories, web pages, and official project documentation
Expected that projects would be set up to reduce awareness requirements, with
each software module carefully partitioned and protected from others.
Found that official partitioning is limited, and that developers can contribute to any
part of the code – an organizational approach that increases awareness
requirements
Found that the developers were able to maintain a good general awareness of
other developers and their activities, and were able to find more detailed
information about people’s activities when they needed to
The main mechanisms for maintaining group awareness were simple text
communication tools – developer mailing lists and text chat
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
39
Situational Awareness Support to Enhance
Teamwork in Collaborative Environments
• [Olga, 2008]
• Situational Awareness (SA) definition
•
•
•
Previous knowledge and understanding of the situation
Detection of relevant information from the environment
Interpretation of these information and reconfiguration
• Research Approach (Exploratory study)
•
Included contextual observations, interviews and task analysis, have
been translated into requirements for support of multidisciplinary
teamwork in life sciences
• Conclusion
•
Designing systems that support situational awareness is of great
importance to ensure that a collaborative environment enables
efficient and effective team coordination and decision making.
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
40
A Comprehensive Evaluation of Workspace
Awareness in Software Configuration
Management Systems [Anita, 2007]
• Workspace tool – Palantir
•
Tool used to monitor code, which is usually manually merged,
change
• Method – Experiment
•
•
Eclipse & CVS vs. Eclipse & CVS & Palantir
The same as above plus UML diagram
• Result
•
•
Subjects clearly monitor awareness cues
Filters are useful
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
41
Palantir
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
42
Conflict Detection and Resolution for Direct and Indirect Conflicts for Control and Experimental Groups
Time to completion
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
43
Indirect conflicts detection
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
44
Empirical Evidence of the Benefits of
Workspace Awareness in Software
Configuration Management [Anita, 2008]
• Method – Experiments ( two-stage)
•
•
Edit text file experiment
Edit Java code experiment
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
45
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
46
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
47
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
48
Results
• The tool is helpful in many ways
• Threats
•
•
How to generalize the result
How to make the experiment closer to real development
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
49
Comparison of Papers:
Awareness & Collaboration
• Awareness improves collaboration
• Collaboration needs communication interface (Partial
awareness?)
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
50
Agenda
 Overview
 Paper discussion



Onboarding
Finding experts
Maintaining awareness
 Summary
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
51
Session Summary
• Onboarding & Finding expert & Awareness
•
•
•
Tools tend to provide quantitative data to help manager’s judgment
How to generalize the result is still a problem (Can be solved?)
How to improve the accuracy of result?
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
52
Thank you! ^_^
The INI is a cooperative endeavor of:
College of Engineering
Tepper School of Business
School of Computer Science
Heinz College
53