Agile Approach in Data Intrinsic Development
developerWorks article template using
Microsoft Word
Type of Submission: Article
Title: Agile Approach in Data Intrinsic Development
Keywords: Agile Methodology, RUP
Topics:
Product Family: Rational
Products: Rational Unified Process
Skill Level: 1
Given: Dinesh
Family: Mohata
Job Title: Senor Consultant
Email: dinesh.mohata@in.ibm.com
Dinesh Mohata is a Senior Consultant in the Oracle Retail Practice within Distribution Sector.
The practice is responsible for providing project management, end to end services in retail
industry, and business solutions. Dinesh has over 9 years of IT & consulting experience in
software design, development, deployment and testing. He has retail industry experience of
over 5 years. Dinesh has played a role of Technical Architect.
His areas of interest include Agile Development Methodology, Design Patterns &
Implementation Data Cutover. He has earlier published a paper “Application Agility: A Design
Perspective” with Oracle. Dinesh can be reached at dinesh.mohata@in.ibm.com
Company: IBM India Private Limited
Photo filename: DineshMohata.jpg
Abstract:
This paper details out various Agile Software development methodologies like XP, SCRUM,
RUP etc. The second half of paper details out agile approach taken in one of the real life
projects. It covers the background of the project, approach taken, benefits gained, challenges
and conclusions. This paper will benefit the audience in providing varied perspective for agile
adoption. The tailoring done to the agile approach for data intrinsic development will definitely
provide users with some tips to take and may implement in their projects as well.
Page 1 of 13
Agile Approach in Data Intrinsic Development
Table of Contents
1.
2.
3.
4.
5.
6.
7.
8.
Introduction ............................................................................................................................. 3
Agile Software Development Methodologies .......................................................................... 3
2.1.
Extreme Programming (XP) ........................................................................................... 3
2.2.
SCRUM .......................................................................................................................... 4
2.3.
Agile Modeling (AM) ....................................................................................................... 5
2.4.
Agile Data (AD) .............................................................................................................. 5
2.5.
Rational Unified Process (RUP) ..................................................................................... 5
2.6.
Open Unified Process (OUP) ......................................................................................... 5
Agile Approach ........................................................................................................................ 5
Case Study: Traditional to Agile Approach in Data Intrinsic Development ............................. 6
Benefits Gained .................................................................................................................... 12
Challenges ............................................................................................................................ 12
Conclusion ............................................................................................................................ 12
References ............................................................................................................................ 13
Page 2 of 13
Agile Approach in Data Intrinsic Development
1. Introduction
Agile software development challenges traditional software development approaches. Rapidly
changing environments characterized by evolving requirements and tight schedules require
software developers to take an agile approach. These methods include practices such as short
iterations, frequent releases, simple and emerging design, peer review, and on-site customer
participation.
Traditional software development methodologies (SDMs), also referred to as software
development processes, are being replaced by light or agile SDMs. Agile SDMs are characterized
by iterative development, continuous code integration, and the ability to handle changing
business requirements [1].
Extreme Programming (XP) is the most well-known agile methodology. XP is based on a series of
concepts that include: having the business customer on-site, pair programming, collective code
ownership, continuous code integration, small releases, designing tests before writing code,
standup meetings, refactoring, and 40-hour work weeks [2].
Disciplined agile software delivery processes are Dynamic System Development Method (DSDM),
Open Unified Process (OpenUP), and agile instantiations of the IBM Rational Unified Process
(RUP) process framework. All of these methodologies function differently from traditional SDMs
and help organizations meet the challenges of the digital economy [1].
This paper covers the background of the project, approach taken, benefits gained, challenges
and conclusions.
2. Agile Software Development Methodologies
Agile methodologies are used to produce higher quality software in a shorter period of time. Agile
methodologies were developed to streamline the development process and remove barriers to
accepting business requirement changes during the development process. Agile methodologies
do not require that business requirements and design details be locked in for the duration of
development [3]. Agile SDMs share several features including prototyping, iterative development,
and minimal documentation [4].
2.1.
Extreme Programming (XP)
XP is a collection of practices for software construction, include refactoring, test-first design, pair
programming, on-site-customer, continuous integration, whole team, and collective ownership.
XP Practices
o
Test driven development:
o Focusing on Thinking about what the code should do before doing
 High Quality and defect minimization
– Regression test baseline
– Create confidence in development team
o Refactoring:
Page 3 of 13
Agile Approach in Data Intrinsic Development
o
o
2.2.
– Improve quality, reducing code entropy
– Re-organize design, to lessen the cost of change
Simple Design:
– Avoid paying the cost of potentially useful, but not business critical
feature.
– The design evolves according to client business requirements
Pair Programming
– High Quality standards through constant Code Review
– Cross Training of the whole team
SCRUM
The focus of Scrum is project leadership and requirements management. Scrum proponents
claim that it is a process framework, but if so then it is a sparse and incomplete one at best. A
more accurate description is that it is a high-level lifecycle for construction iterations (what Scrum
calls “sprints”), see Figure 1, and several practices such as daily stand-up “Scrum” meeting,
product owner, product backlog, iteration/sprint planning, and potentially shippable software.
Figure 1 Scrum Construction Lifecycle
alt=Agile lifecycle as used in SCRUM
The above figure has been referenced from http://www.ambysoft.com/essays/agileLifecycle.html
Scrum Approach

Accept:
Page 4 of 13
Agile Approach in Data Intrinsic Development
o
o
This is a complicated, unpredictable world and development environment.
You can't predict or definitively plan what you will deliver, when you will deliver it,
and what the quality and cost will be.

Assume:
o You can estimate, and then negotiate according to various risks as you proceed.

Understand:
o Following any cookbook approach won't improve the definition of "best", and will
only hinder appropriate responsiveness to the complexity and unpredictability.
o the delivery will be the best possible given the circumstances
The above Scrum Approach content has been referenced from http://www.controlchaos.com/oldsite/philo.htm
For implementing scrum visit www.implementingscrum.com for further details.
2.3.
Agile Modeling (AM)
AM is a collection of practices for light-weight modeling and documentation, including
requirements envisioning, executable specifications, active stakeholder participation, prioritized
requirements, and prove it with code. See www.agilemodeling.com for details.
2.4.
Agile Data (AD)
AD is a collection of practices for database development, including agile data modeling, database
testing, database refactoring, and continuous database integration. See www.agiledata.org for
details.
2.5.
Rational Unified Process (RUP)
RUP is a comprehensive process framework for iterative software delivery which can be
instantiated anywhere from a very agile form to a very traditional form as your situation warrants
[6]. RUP practices include risk-value lifecycle, whole team, test-driven development (TDD), and
business process. See http://www-01.ibm.com/software/awdtools/rup for details.
2.6.
Open Unified Process (OUP)
OpenUP combines and extends practices from Scrum, XP, AM and RUP for co-located agile
teams which are building business applications. OpenUP practices include whole team, daily
standup meeting, prioritized work items, risk-driven lifecycle, TDD, active stakeholder
participation, and continuous integration. See www.eclipse.org for details.
3. Agile Approach
Agile Approach has to be tailored to the needs of the project one is working upon. Agile process
adopted in this project was culmination of various precepts pronounced in various agile SDMs.
Page 5 of 13
Agile Approach in Data Intrinsic Development
Various precepts applied in the current work with respected to Agile SDM were as follows:










Requirements Envisioning
Redefining Stake Holder Participation
Incorporation of Late Requirements
Work Prioritization
Continuous Integration
Pair Programming
Refactoring
Cross Training of whole team
Applying TDD
Technical Excellence
The above approach is explained in detail with all due examples in the subsequent case study
section of this paper.
4. Case Study: Traditional to Agile Approach in Data
Intrinsic Development
A retailing client made their first foray into ERP implementation by implementing Oracle packaged
solution a.k.a. Oracle Retail. This program has been aimed to implement various modules of
Oracle Retail suite formerly called Retek. Oracle Retail Suite will form the technological backbone
of their retail venture. The process of this retailer is quite ad-hoc in nature. The IT infrastructure
required to make this has to be robust and scalable to meet the dynamic changing requirements
for the customer. This project will transform and streamline the business and IT processes for this
retailer in lines with world class best practices in Retail industry
Retail is all about business agility. Requirements change on the fly. To beat the competition
retailer has to change the dynamics of its organization at rapid pace. This requires systems and
processes to be agile. IT has to be an enabler for this happen. This in turn requires IT systems to
follow sound agile process to deliver value to the customer in the minimum time possible. Retail
being highly margin sensitive area retailer has to capture lot of data to optimize its pricing engine
to get the best value to the customer and in turn protect its margins. This requires huge data
churning within the retail organization. This required defining and proving the agile processes
work in packaged implementation. Moreover, the sound agile processes adopted during the
implementation will become the sound backbone for the retailer for future.
This section details about the various facets of agile software development applied for this retailer.
The tenets of agile development have been picked and experience around this has been penned
down in the paragraphs below.
Requirements Envisioning
Ascendant Methodology was adopted to do requirements envisioning. During inception of the
project a high-level requirements modeling was done to understand the scope and identify the
initial stack of requirements. This project being a packaged implementation, it was critical to
understand the fitment of the customer requirements with the product. Any deviations were duly
captured and communicated to the customer as gap. This also made the implementation team
aware of the gaps coming forward beforehand and just in time.
Page 6 of 13
Agile Approach in Data Intrinsic Development
Redefining Stakeholder Participation
Agile approach in traditional environment is really tough. People are not willing to understand the
agile approach as such. Customer is not fully aware of the approach and has no willingness to
here it as well. Approach taken was internally we will satisfy our customer i.e. functional
consultants using agile approach. Agile approach was to internal customers of the project team.
Stake holder for the development team is the internal teams which certify the product before
releasing to the external stakeholder.
Incorporation of Late Requirements
Initial requirements were prototyped. The design was envisaged keeping in mind the changes
coming late in the project to be incorporated. Each report was driven by an Oracle Database View.
Report were predefined by customer in ms excel templates. These templates were uploaded in
the database. Any additional column required a change in template and subsequent upload a
CLOB database object type. Since each report was based of database view it was just matter of
adding the additional column in the view from the table already in use in the view definition.
Work Prioritization
Changing the stakeholder definition to internal customer helped team to prioritize the work. Entire
team was co-located and any suggestions on work priority were discussed right at the moment to
decide team’s future course of action. Real stakeholders were intimated on the details of the work
scheduled for the team. Communication to real stakeholders was done only after having internal
stakeholder approval.
Continuous Integration
Continuous integration was key approaches adopted in the data intrinsic development. In all the
instances the code was deployed based on few iterations of develop, build and test cycle. The
development was done based on requirements provided by the customer. The requirements were
then prioritized. Based in these prioritizations the development would work with iterations of
development. The end product was continuously integrated with the bigger system to ensue this
integration does not break the larger system. This resulted in smooth transition during the
movement of products in production environment.
There was an instance wherein application was developed in the corporate environment using
intranet network. This application was supposed to be deployed at a remote location for the
customer. This application was working perfectly fine when the server was accessed using the
local area network (LAN). But application just was not working when deployed in the remote
location. The connectivity to the server was not appropriate for certain protocols required in the
application. This was a serious issue since without opening up this protocol in all the routers to
server this application wouldn’t work. Design was re-organized to attain this in two step process.
This design was also thought through with customer in mind. The process using the protocol was
replaced with a two step process. The first step was making a direct database entry of the data to
be transported from local client to the server. The second step was creation of the file at the
server side using database server file utilities. These two steps isolated the protocol error
occurring in the application. Cost of change was quite less since this kept the other steps in the
application intact and re-designed a portion of the application process.
During the one of the demonstrations of the code developed during the project the agile approach
was quite handy. Two team members from the team were executing the demonstration to the
Page 7 of 13
Agile Approach in Data Intrinsic Development
customer. One of programmers was displaying the features of the code and performing the initial
tests for the customer on the developed application. This demonstrator was connected to the
projector for the entire audience. Another programmer was sitting just beside and noting the
comments of the audience. There was couple of enhancements requested by the customer
during this session. While the first demonstrator was demonstrating the application, the second
developed already coded these new features in the code and deployed the same code during the
recess break of the session. Post the recess break once more the application walkthrough was
conducted. This walkthrough resulted a huge success maximum of changes requested by the
customer was successfully implemented and demonstrated within a very short duration.
Pair Programming
There was instance in the project were in we had to deploy an unknown application. This
definitely required some initial training to get the job done. Customer arranged for online CBT
training but this did not help. We did form a two member team to get the job done. There was one
programmer doing the typing effort and another person religiously followed the application
deployment guideline to guide the typist. This combined development effort paid off and we were
successfully able to implement the application in very short duration. As a side note I would
emphasize that this required good resources with a good attitude to get the job done.
Cross Training of whole team
All the members in the team were provided an opportunity to work within other modules of the
project. The whole team was collocated in a single venue. Team sitting in a single location
actually helped in various ways. Discussions between team members of various modules by and
large happened in the same room. This resulted in each and every individual having a fair amount
of idea on each module being implemented. As in IBM servers, it applies to humans as well. Not
all the time each individual is fully engaged. IBM virtualization concept utilizes spare processing
power from servers in the stack which are not in use. Applying the same concept in the colocated, it helped in filling any immediate requirement from a module which would require some
extra effort for any dynamic requirement. This in turn also gave some amount of knowledge to the
cross team member who approached to solve the issue.
Applying Test Driven Development
TEST DRIVEN DEVELOPMENT (TDD) LIFE CYCLE
The below diagram explains the TDD process lifecycle.
Figure 2 Test Driven Development
Page 8 of 13
Agile Approach in Data Intrinsic Development
ADD A TEST
Pass
RUN ALL TESTS
Fail
WRITE/REFACTOR CODE
Pass
Fail
RUN THE TEST
alt=Test Driven Development Lifecycle
The above content has been referenced from http://www.agiledata.org/essays/tdd.html
Add a test
In test-driven development, each new feature begins with writing a test. This test must inevitably
fail because it is written before the feature has been implemented. In order to write a test, the
developer must understand the specification and the requirements of the feature clearly. In all
the developments developer was provided with use cases and stories to cover requirements and
exception condition. This made developers aware of the requirement before writing the code.
Page 9 of 13
Agile Approach in Data Intrinsic Development
An instance of test in current scenario wherein a development of Retail application for financial
accounting was developed. User story of this application was beforehand given to the developer.
The requirement was to develop application that sends data on daily basis of certain attributes
and thereafter updation of flag at the subsriber end. The test condition was written by the
developer stating the attributes of the database required to pull the data from. If any of the
columns required fo certain table does not exist then this test condition will fail. The code had to
be rewritten to take the appropriate column from the table for the test to pass.
Run all tests and see if the new one fails
This validates that the test harness is working correctly and that the new test does not mistakenly
pass without requiring any new code.
The new test should also fail for the expected reason. This step tests the test itself, in the
negative: it rules out the possibility that the new test will always pass, and therefore be worthless.
Write some code
The next step is to write some code that will cause the test to pass. The new code written at this
stage will not be perfect and may, for example, pass the test in an inelegant way. That is
acceptable because later steps will improve and hone it.
It is important that the code written is only designed to pass the test; no further (and therefore
untested) functionality should be predicted and 'allowed for' at any stage.
Run the automated tests and see them succeed
If all test cases now pass, the developer can be confident that the code meets all the tested
requirements. This is a good point from which to begin the final step of the cycle.
Refactor code
Now the code can be cleaned up as necessary. By re-running the test cases, the developer can
be confident that refactoring is not damaging any existing functionality. The concept of removing
duplication is an important aspect of any software design. In this case, however, it also applies to
removing any duplication between the test code and the production code.
Repeat
Starting with another new test, the cycle is then repeated to push forward the functionality. The
size of the steps can be as small as the developer likes, or get larger if s/he feels more confident.
If the code written to satisfy a test does not fairly quickly do so, then the step-size may have been
too big, and maybe the smaller testable steps should be used instead.
The first step is to quickly add a test, basically just enough code to fail. Next you run your tests,
often the complete test suite although for sake of speed you may decide to run only a subset, to
ensure that the new test does in fact fail. You then update your functional code to make it pass
the new tests. The fourth step is to run your tests again. If they fail you need to update your
functional code and retest. Once the tests pass the next step is to start over (you may first need
to refactor any duplication out of your design as needed, turning TFD into TDD).
I like to describe TDD with this simple formula:
Page 10 of 13
Agile Approach in Data Intrinsic Development
TDD = Refactoring + TFD. [5]
The above content has been referenced from http://www.agiledata.org/essays/tdd.html
This Test Driven Development approach was followed across on new developments done in the
project.
Technical Excellence
Customer had proposed for development of 16 reports as part of SOW at an implementation
project at a large India based retailer. Since the reports were not classified for its complexity the
customer proposed very complex reports for development.
The challenges for the report development
1. Each report was culmination of three to four medium complex reports
2. Excel output was required for the reports which was not in purview of development
effort/methodology
3. Huge Technical Challenges
Based on the requirements the report development was started developing the skeleton or
framework for one of the reports. Brainstorming was conducted on technical challenges of the
report and work for the same started. The working report was ready in a day. There were issues
of performance in the report as far as the data display in the screen was considered. The data
was displayed in the spreadsheet on a cell by cell basis. Oracle’s Webutil technology was
leveraged to write data in the excel sheet.
Following steps were taken to foresee any bottlenecks and combat well in advance

Since each report was based on a database view, SQL tuning the underlying view
database query was indispensable.

Server Statistics report collected in daily basis check the server health and apply
appropriate check if necessary

Database performance was continuously monitored by modifying database parameters

Collecting the data base statistics reports.

Each SQL session and application session was traced. Oracle database utilities were
used to analyze the performance impact of session parameters
This approach of continuous integration and collective ownership as a team was adopted to
benchmark the technical approach in an agile way.
Page 11 of 13
Agile Approach in Data Intrinsic Development
5. Benefits Gained
By following the agile-tional approach to software development for data intrinsic development
following was gained

















Design and Build of reports were prototyped resulting in creation of working software
before discussing requirements further with customer
Effort spent on creating this working software was restricted to two days only
New alternatives/suggestions were advised to customer on first meeting itself
Realistic expectations were set for the customer from day one based on the technical
limitations of the development
Functional fitment done well in advance because of requirements envisioning
Ability to take any additional touch-ups on the code as along as the framework of the
development does not change
Customer was very satisfied on the approach of creating prototype in initial phase
Requirements late in the project were easily incorporated
The working reports were used by the customer for extensive data analysis from their
end.
They used the reports to further verify the data in their system
On the fly change during presentation using Pair Demo as part of continuous integration
was just excellent and was highly appreciated by the customer
Pair Programming resulted in huge customer satisfaction
Quick Results
High Quality standards were set through refactoring codes
Good Clean code that works
Agility in the Software development process
Huge confidence in the development team
6. Challenges
There were many challenges faced in journey of applying agile-tional approach to development





It was difficult to make team agree on the agile approach since the school of thought
was more towards waterfall model
Getting the buy in from the Project Management was tough and hence workaround
stakeholder idea was devised
Technical Challenge of publishing an excel report from Oracle 10g server
Performance of the reports in general and this was dealt with continuous monitoring
of database and application
Regression Testing of deliverable upon incorporating late feedback
7. Conclusion
Software development can be done in agile way bringing delight to the customer as an end result.
It provided quick turnaround time to deliver quality product to the customer. Willingness from the
all stakeholders is indispensable for successful agile methodology implementation. Requirements
late in the game can also be incorporated with agile SDM. Customer is fully aware of the end
product getting delivered to him.
Page 12 of 13
Agile Approach in Data Intrinsic Development
Agility is more of perspective. It’s really just a mindset. It’s about adaptability to changing
business requirements. Based on the points highlighted in this paper, and supporting points given,
it could be very well stated that agile SDM can be adopted in an organization whilst providing
huge benefits and requiring few cultural changes that will greatly impact the implementation.
8. References
1. Boehm, B. & Turner, R. Management challenges to implement agile processes in traditional
development organizations. IEEE Software. 22(5), 30-40. 2005.
2. Theunissen, W., Boake, A., & Kourie, D. In search of the sweet spot: Agile open collaborative
corporate software development. Proceedings of the 2005 Annual Research Conference of
the South African Institute of Computer Scientists and information Technologists on IT
Research in Developing Countries. White River, South Africa. 268-277.
3. Lindstrom, L. & Jeffries, R. Extreme programming and agile software development
methodologies. Information Systems Management. 21(13), 41-53. 2005.
4. Holmstrom, H., Fitzgerald, B., Agerfalk, P., & Conchuir, E. Agile practices reduce distance in
global software development. Information Systems Development. 23(3), 7-18. 2006.
5. Agile Database Techniques by Scott Ambler
6. Kroll, P. and MacIsaac, B. (2006). Agility and Discipline Made Easy: Practices from OpenUP
and RUP. Boston: Addison-Wesley
Page 13 of 13