CONTINUOUS INTEGRATION AND TEST Stephen Oglesby - CSCI577b – April 2011 Introduction 2 Who am I? What is Continuous Integration? Continuous Integration + Automated Testing Requirements for successful CI&T Scaling options CI&T within CSCI577ab “Cake, and grief counseling, will be available at the end of the testing period.” - GLaDOS 4/27/2011 Continuous Integration 3 “Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.” – Martin Fowler Born out of eXtreme Programming (XP), not limited to it. 4/27/2011 What does that mean? 4 Code commit detection Per check-in compilation (yes you should do this anyway, but now it’s automatic) Automated build solution Expandable to include static code analysis Results reported to the team Immediate notification of integration errors Faster resolution 4/27/2011 CI & Test 5 Automated unit & acceptance testing is a natural extension of CI Reduces test time and cost Improve communication & project visibility Updated and available test results throughout development Bug Tracking System integration for automatically logging new defects Reduce risk 4/27/2011 Requirements 6 Single source repository Automated build Automated self-test Tests run on production environment clone 4/27/2011 Requirements (cont.) 7 Fast builds 4/27/2011 Scaling Up from XP 8 Add Hierarchy Execute build at each level of integration Label version units based on integration test depth Copper Silver Copper Gold Copper Silver Copper Add Builds Stable, Beta, Dev, Nightlies 4/27/2011 CI&T in CSCI577ab 9 Requires bringing test activities into the first semester Test case definition during requirements gathering/WikiWinWin Still requires test case dependencies, even more critical since automation environment needs them Standardized test procedures and results, automatically generated Automation Framework + CI Server should be largely reusable across most projects 4/27/2011 CI&T in CSCI577ab (cont.) 10 Provides quantitative assessment criteria Percentage of development time with a failed build Number of overnight build failures Number of deleted test cases Percentage of requirements with corresponding test cases Percentage of methods not tested Increases team collaboration, code quality, developer productivity 4/27/2011 Open Source CI&T Solutions 11 Automated Test Frameworks Framework for Integrated Test (Fit + Fitnesse) Software Testing Automation Framework (STAF) Selenium Continuous Integration Server CruiseControl + Subversion “At the end of the experiment you will be baked, and then there will be cake” - GLaDOS 4/27/2011 References 12 Sean Stolberg, “Enabling Agile Testing Through Continuous Integration”, Agile Conference 2009, IEEE, pg 369-374, 2009 Hehui Liu, Zhongjie Li, Jun Zhu, Huafang Tan, and Heyuan Huang, “A Unified Test Framework for Continuous Integration Testing of SOA Solutions”, 2009 International Conference on Web Services, IEEE, pg 880-887, 2009 Eun Ha Kim, Jong Chae Na, and Seok Moon Ryoo, “Test Automation Framework for Implementing Continuous Integration”, Sixth International Conference on Information Technology: New Generations, IEEE, pg 784-789, 2009 Eun Ha Kim, Jong Chae Na, and Seok Moon Ryoo, “Implementing an Effective Test Automation Framework”, 23rd Annual IEEE International Computer Software and Applications Conference, IEEE, pg 534-538, 2009 Jon Bowyer and Janet Hughes, “Assessing Undergraduate Experience of Continuous Integration and TestDriven Development”, 28th International Conference on Software Engineering, ACM, pg 691-694, 2006 Fabrizio Cannizzo, Robbie Clutton, and Raghav Ramesh, “Pushing the Boundaries of Testing and Continuous Integration”, Agile 2008 Conference, IEEE, pg 501-505, 2008 Christopher G. Jones, “Test-Driven Development Goes to School”, Journal of Computing Sciences in Colleges, Volume 20, Issue 1, pg 220-231, 2004 Martin Fowler, “Continuous Integration”, http://www.martinfowler.com/articles/continuousIntegration.html, 2006 4/27/2011 13 Questions? “Well done. Here are the test results: You are a horrible person. I'm serious, that's what it says: A horrible person. We weren't even testing for that.” - GLaDOS 4/27/2011