Adam Guida Introduction Adam Guida - Senior QA Automation Developer Work History: - 2011 – Present: - Deutsche Bank Global Technologies Cary, NC - 2006-2011: - lulu.com - Raleigh, NC Email: guidadic@gmail.com 2 Automated Testing is a Liability Unless you have Stable Test Environments Deployment Automation Strong Communication Collaboration Technical Expertise The Truth About Automated Testing Automated Testing: When done correctly, it can significantly increase the repeatability and effectiveness of testing efforts Augments classical test methodology but does NOT replace manual testing Increases regression consistency Reduces tedium Allows human testers a greater focus on new and creative ways of breaking things However: Automated testing is ineffective without an approach that considers all facets of the development automation ecosystem Purpose of this lecture Lessons Learned Poorly written test automation isn’t sustainable Writing good tests alone isn’t enough Continuous Automation in QA Test Strategies Best Practices Environments Tools & Technologies Communication Continuous Automation Practices Continuous Integration (CI) A practice where code is automatically compiled and unit tested on every developer code check-in Continuous Deployment If CI phase passes, application(s) are automatically deployed to one or more pre-production environments, triggering additional layers of automated testing Continuous Automation public class Feature { Unit test failed! I’ll take a private int 1; look at broke and Latest build ontry public My void bad!doSomething(){ I’ll what check-in ais fixup for that if (a right == away. b){ again CI for testing! return true; } Sweet! All tests passed! DEV & QA Let me take a look. I see! You’re Feature: Test A and B are True This one is my fault. LooksEquals like I B Scenario: Hey, can youVerify takeLet’s athat lookAat the last Awesome! forgot to adjust some“Feature” testsgrab after Given I open run? Looks like a the regression test that last build. I’ll check-in a fix & When Ilunch! click on “A” and “B” some failed. restart the automation Then the result is “…….” Dude, I’m having a problem getting this test working. The web service is returning a 404 Test SCM the wrong resource. I’ll IM App calling SCM you the correct name Automation Pipeline Build + Unit Test Test Environment Deploy Automated Testing Report Continuous Automation Collaboration Establish partnerships & comradery with your development team Treat QA & Developers as a single team Close low-level knowledge gaps in functional testing Develop workflows for how DEV & QA address test failures as a team Foster a culture that values cross-team learning, exchange of knowledge and professional growth Continuous Automation Best Practices Demo Continuous Automation Best Practices Automated Test Suite should be run as often as possible Automated tests should run……………………...automatically On every successful automated deployment Automation executes using the fewest clicks possible Start Small and Simple – It won’t all happen overnight Automation Test Team: Strategies Delegation - Separate the technical from test responsibilities for a given task Increases efficiency and focus by reducing context switching Pair-testing - Pair technical & non-technical testers together for greater impact Increases technical experience and promotes mentorship of non-technical testers Increases usability & readability of tests Reporting - Test Lead/Manager - keep a close eye on automated test reporting. If tests start failing, seek to understand the underlying causes, then try to assist test team by removing roadblocks Understanding daily/weekly trends in test results helps guide risk assessments Continuous Automation Best Practices Treat Your Test Suite like an Enterprise Application Automated testing *IS* a form of programming and bound by the same laws as software development Strive to be the best programmer you can & adhere to software development best practices Use a Source Control Management tool Unit Test your test automation (Yes, that means testing your tests) Practice Code reviews Seek architectural guidance and input from your development team Defect tracking (automated tests apps get bugs too!) Automated Test Strategies The Perfect Automated Test Fast – Speed impacts every aspect of a quality automation suite Reusable – Reuse reduces code complexity and increases simplicity of a test suite Relevant – Test indicate a problem we care about and is not redundant Independent – Is self sufficient and does not depend on the ordering of other tests Focused – Tests a specific aspect of the application without being too broad Clear – Determines a pass or failure result automatically, ideally narrowing down the exact cause of the failure Understandable – Is written in such a way that documents behavior other stake- holders can understand Flexible – Change is the only constant – write tests in ways that can handle changes within reason Automated Test Strategies Managing the Complexity Versus Coverage Tradeoff System Integration / End-to-End –Provides the most comprehensive test coverage at the cost of speed and complexity. Unit – Isolated testing of individual classes, methods and functions Component – Tests are isolated to the component itself. Inputs/Outputs are simulated through test adapters or mock Automated Test Strategies Example of Component Testing Data Transformer Testing MOCK DATABASE H2 End-2-End TEST 1 User Database Product Database Marketing Aggregator WEB SERVICE CALL Web Server Browser Order Database TEST 2 Web Service Testing TEST 3 MOCK AGGREGATOR UI Testing Automated Test Strategies Managing the Complexity Versus Coverage Tradeoff Diversify your test portfolio using a combination of Unit, Component and System testing Promote Unit Testing w/ Devs - allows you to start trusting some basic elements of the system, so you can focus on more challenging test scenarios Minimize System/End-to-End tests - to fewer high-level smoke tests based on areas of importance Maximize your Component Testing - focus on lower level functionality, API’s and data model interactions Use Test Adapters – for both greater control & reliability of the application input/outputs Choosing the right Tools & Technologies Don’t Reinvent the Wheel – someone has already solved your problem Carefully Evaluate All Technologies – Require experience prior to adoption. Invest time in evaluating the pros & cons of a variety of different solutions. Build POC’s and give stakeholders (developers) a role in decision making Straightforward Integration – Utilize tools which accept open formats for read/write – ensure each tool can fully integrate with others as necessary Active Community of Users – Favor technologies where an active online support community exists over managed (paid) support solutions Technologies Programming Language Rule of thumb: Test Automation should be based on the same programming language of your application Enables sharing of development resources, support and contribution Shared libraries and other resources between testing & dev Should be capable of utilizing a highly functional IDE’s such as Eclipse, Intellij, Visual Studio, etc Tools: IDEs (Integrated Development Environment) Intellij IDEA / RubyMines Fully integrated support for Cucumber Syntax highlighting Code Editing/Refactoring Supports many languages Build-in database browser Debugger Tools: Continuous Integration Server CI (Continous Integration Server) More than just a build agent Manages the end-to-end flow of your automation pipelines and reporting Is both the glue and front-end that binds the continuous deployment, testing and reporting processes Tools: VMs - Virtual Machines VirtualBox Virtualized OS’s Linux Windows Android OS X Contained sandbox environments Backup/Restore, share and take snapshots of your VM environments Tools: Create and configure lightweight, reproducible, and portable development environments Easy to setup/tear down and deploy to a standardize test environment Works with VirtualBox and other virtual environment products Support integrated into Intellij IDEA via plug-in Tools: Deployment Orchestration Help automate aspects of environment provisioning & deployment Install & Configure Servers & other resources Helps facilitate continuous deployment of your application Many great frameworks to choose from - each has it’s own set of strength & weakness Tools & Technology: Database Management •Treats SQL like version-able code • XML, JSON, YAML and generic SQL syntax supports multiple database technologies • Flexible runtime: Maven Plugin and standalone support 24 Final Thoughts Automation is an investment, that when managed effectively, provides a particular set of benefits which increase efficiency and repeatability of testing. AUTOMATION IS NOT A BENEFIT IN ITSELF Treat automation as a 1st class citizen of your project If you’d like a copy of this presentation, please send me an email at: guidadic@gmail.com