Critical factors in agile software projects according to people, process and technology perspective Karla M. B. Silva Simone Santos Informatics Center Federal University of Pernambuco Recife, Brazil kmbs@cin.ufpe.br Informatics Center Federal University of Pernambuco Recife, Brazil scs@cin.ufpe.br Abstract—Agile Software Development have become very important over the years for organizations of the modern world; however, software development itself is a very complex process. There are so many factors that affect project success or failure. This paper is a systematic review on the critical success factors of agile software development projects. We classified fourteen critical factors into three categories based on the pillars of Information Systems: people, process, and technology. In addition, primary studies were selected to identify the profile of agile development organizations, projects’ characteristics, and implications of agile studies in these projects. Keywords—critical factors; agile software development; systematic review software development; I. INTRODUCTION Software development has been the focus of many researches, managers, and engineers in the last years; however, software development is not a simple process. Several software projects are not delivered on time or budget. In addition, some of them do not give real value to the client. In order to improve the software development process, 17 specialists created The Agile Manifesto in 2001. Some of the Manifesto principles are: effective communication among team members and between the team and its clients and users; variable scope, in which changes in requirements during the project are accepted as intrinsic to software development; rapid delivery of software to the client; and team self-management, to provide flexibility and efficiency in the development [01]. In addition, there are some software development methods that are called “agile”; for instance, Scrum, Extreme Programming (XP), Lean, Crystal, and Kanban. These are the methods used in the companies cited in this paper. Although, according to the Standish Group [10], software development projects continue to fail even with the existence of communities of methodology practice such as PMI and Agile that promote best practices. Our research aims to identify and provide insights into the critical success factors that help agile software development projects to succeed. We decided to explore and categorize the critical factors in agile software projects according to people, process and technology perspective. In addition, we started to investigate the profile of organizations in agile environments, their projects’ characteristics, and implications of CSFs studies in software development companies. Our studies included both in-house and outsourced software development projects. The broad objective of this study is to answer the following research question. What to consider when developing software in agile environments? More specifically, this study focuses on four specific questions: RQ-01: What are the critical factors in agile software development under people, process and technology perspective? RQ-02: What kind of organizations work with software development in agile environments? RQ-03: What are the implications of critical factors studies in agile software development field for the organization’s projects? RQ-04: What are the main characteristics of the developed projects, according to number of collaborators by team, project duration, domain, and platforms used? Answering these questions, this review will help companies that work with agile to understand the critical factors in ASD, and how they can improve the success rate in their projects. The next section gives an overview of the research method used in this study, and section 3 describes the results of this research. After all, section 4 concludes and provides recommendations for further research in the agile software development field. II. RESEARCH METHODS This research has been carried out by following Kitchenham [04] procedures for conducting a Systematic Review (SR), which involves several activities. Kitchenham summarizes the activities into three major phases: planning, conducting, and reporting the review. The planning and conducting stages have several minor phases. On the other hand, reporting is a single stage phase. Table I shows the complete review process proposed by Kitchenham. According to Kitchenham [03], many of these stages involve interaction. For example, some activities are initiated during the protocol development stage, and refined when the review proper takes place. TABLE I. Phases Planning REVIEW PROCESS Java to get the search result on the databases. These scripts are available online in a public repository at GitHub [08]. At the end of stage 4, we were left with 23 papers. Stages Identification of the need for a review Development of a review protocol Identification of research Selection of primary studies Conducting Study quality assessment Data extraction & monitoring Data synthesis Reporting N/A A. Data Sources and Search Strategy The search strategy included only electronic databases, and papers that are written in English. The following electronic databases were searched: Fig. 1. The selection process of primary studies. ACM Digital Library (www.acm.org) IEEE Xplore (www.ieee.org/ieeexplore) ScienceDirect (www.sciencedirect.com) Scopus (www.scopus.com) The search on these databases was based on the three keywords identified in this paper: (1) critical factors, (2) agile software development, and (3) software development. With the major search terms, we could identify synonyms (Table II) that will compose the search string. TABLE II. Keywords Critical factors Agile software development Software development KEYWORDS AND SYNONYMS Synonyms Critical success factors, CSF, crucial factors, risky Agile, ASD, lean, scrum, XP, kanban, crystal Software project, software process development The search string was combined by using the Boolean “AND” operator to connect the three categories above. ("critical factors" OR "critical success factors" OR CSF OR "crucial factors" OR risky) AND ("agile software development" OR agile OR ASD OR lean OR scrum OR XP OR kanban OR crystal) AND ("software development" OR "software project" OR "software process") This search strategy resulted in a total of 1581 results that included 1387 unduplicated papers. Fig. 1 shows the complete review process including the number of papers identified at each stage. To analyze and store the data from all stages, we used a Google Sheet that is stored at Google Drive [09]. In addition, in stage 1, our study used Selenium scripts written in B. Abbreviations and Acronyms Inclusion and exclusion criteria are used to exclude studies that are not relevant to answer the research questions [05]. In general, we included all the studies related to agile software development which passed the following criteria: The studies must be written in English; They must be completely and available on the electronic databases searched in this paper; The studies should be about agile software development practices, critical factors, or companies that use agile. In addition, we excluded studies which focus or main focus was in a specific framework, or related to very specific fields in the software development area; for example, user interface and architecture studies. III. RESULTS A. Overview of Studies We selected 23 primary studies, and we categorized the studies into four main groups according to their methodology: case study, conceptual, research, and review. Papers were classified as case study if there was an implementation of a research method involving a detailed examination of agile development practices, as well as its related contextual conditions. Conceptual papers are the ones primarily based on theoretical considerations. In addition, research papers are the ones that use surveys as input. If the study tends to summarize, discuss, or criticize, they were classified as a review. However, some studies fit more than one category, so we created a new group to contain all these mixed studies. Fig. 2 shows that most of the primary studies are researches. RQ, but it was incomplete, and “3” indicates that the study completely answered the question. TABLE IV. PRIMARY STUDIES BY RESEARCH QUESTIONS RQ-01 RQ-02 RQ-03 RQ-04 Total PS-01 3 1 2 2 8/12 PS-02 2 1 2 1 6/12 PS-03 3 1 1 1 6/12 PS-04 3 1 2 3 9/12 PS-05 1 3 1 3 8/12 Fig. 2. Primary studies by category. PS-06 1 1 1 3 6/12 These categories helped us to understand the primary studies, and answer the research questions. Equally important, Table III shows that the number of studies in the agile development field are not always increasing through the years. PS-07 2 1 1 1 5/12 PS-08* 1 1 1 1 4/12 PS-09 2 3 1 3 9/12 PS-010 2 1 2 1 6/12 PS-011* 1 1 1 1 4/12 TABLE III. PRIMARY STUDIES BY YEAR Year Number of Paper % PS-012 1 2 1 1 5/12 2005 1 4,35% PS-013 3 2 1 2 8/12 2008 2 8,70% PS-014 2 1 1 1 5/12 2009 1 4,35% PS-015* 1 1 1 1 4/12 2010 2 8,70% PS-016 2 1 1 1 5/12 2012 4 17,39% PS-017 1 2 1 2 6/12 2013 4 17,39% PS-018 2 1 1 2 6/12 2014 6 26,09% PS-019 3 1 1 2 7/12 2015 3 13,04% PS-020 2 1 1 1 5/12 PS-021 2 1 3 1 7/12 PS-022 2 1 2 1 6/12 PS-023 2 2 2 2 8/12 Total 44/69 31/69 31/69 37/69 To put it differently, Fig. 3 shows the amount of primary studies by database and year of publication. * Indicates that the study did not answer any research question. Fig. 3. Primary studies by year of publication and database. In a moment that we finished stage four from the selection process, our next step was to extract the data from the selected studies, and address the research questions. Evidences and answers were found for the four questions discussed in the introduction section of this paper. Table IV shows the correspondence between primary studies and research questions. The studies were graded according to the following criteria: “1” indicates that the paper does not answer the research question, “2” means that the study helps to answer the B. What are the critical factors in agile software development under people, process and technology perspective? This question has the highest score on Table IV because most of the selected studies answered or helped us to answer the research question 1 (RQ-01). While analyzing the primary studies, we find out that nine papers [PS-01, PS-03, PS-05, PS07, PS-017, PS-018, PS-019, PS-021, and PS-023] use Chow and Cao study [PS-04]. This reference is a survey study with 109 agile projects, and these studies mention Chow to provide a list of main success and failure factors in agile software development. The critical success factor technique to identify and measure an organization’s performance was developed by Rockhart [02], and was refined and became well established few years later [07]. According to [05], CSFs are issues that if addressed appropriately, it can increase the chances of project success. However, there are many interpretations for definition of success in software development. In this study, we will indicate that success is associated with time, scope, cost, and quality variables [PS-01, PS-03, PS-04, PS-013, PS-022, PS023]. The failures and success factors from [PS-04] listed in [PS-07 and PS-023] contain a total of 55 items grouped into 4 and 5 categories. Different from [PS-04], we decided to regroup these factors into three main categories: people, process, and technology. From the people perspective, we will consider team members of the development team, customers, end users, or any other human resources. By the same token, process is related to the operation of ASD, and technology is related to all technical and tech procedures. Table V shows the most common critical success factors found in [PS-01, PS-03, PS-04, PS-07, PS-010, PS-013, PS016, PS-018, PS-019, PS-020, PS-021, PS-022, and PS-023] in agile environments. It is important to realize that the primary studies not cited above did not have an explicit list of success or failures factors. TABLE V. CRITICAL FACTORS BY PEOPLE, PROCESS, AND TECHNOLOGY PERSPECTIVE People Competence and expertise Executive support Team and user motivation Team size User participation Process Agile practices Delivery most important features first Right amount of documentation Strong communication Technology Appropriate technical training As show above, some factors are cited more frequently in studies; for example, team and user motivation. Chow and Cao [PS-04] found that “team members with great motivation positively influenced the perceived success of the agile software development projects”. Although, [PS-013] disagrees with some of the Table V factors. The author explains that the methodology of asking a few people for their opinion without a mediated focus group methodology has its limitations. The study defends that team size, technical competency, and communication are not significantly related with success. C. What kind of organizations work with software development in agile environments? Only 6 studies provide contents to answer the question 2 (RQ-02) because it is a very specific question. In our research, we have a broad range of companies. This range includes startups [PS-017], small and large outsourced software development companies, and companies that their business are not software development [PS-05]. Two papers [PS-08, PS011] are not specific about success and failures factors in agile, so their cases will not be counted in this question. In addition, [PS-017] includes some educational and governmental organizations. Primary study [PS-017] conducted a survey with 200 companies to identify the agile practices in these companies. In this survey, the author identified that most of the companies are medium size (less than 100 employees). Fig. 5 shows [PS-017] research results. Integration testing Simple design Tool supports Well defined coding standards Based on our analyses, we identified 14 factors as the most frequently cited across 13 studies. To put it differently, Fig. 4 shows the number of occurrences of each factor from Table V. Fig. 5. Survey results from [PS-17] Similarly, [PS-05] presents 3 cases, and the company of case 1 has over 500 IT employees. As can be seen in Table VI, the second case from [PS-05] presents a well-established company with more than 10 years delivering e-commerce and infrastructure services. Table VI shows the summary of the cases from the [PS-05] study. TABLE VI. SUMMARY OF THE CASES FROM [PS-05] Company 1 Financial corporation Number of employers over 500 IT employees Company 2 Software company 120 developers Company 3 Internet content and access provision 200 developers Domain Fig. 4. Factors from Table V by total citation count. In contrast, study [PS-09] argues that “we have to be aware of large companies with well-defined structures and processes”. The author explains that the transition process to agile might not be easy. The article suggests to try to use agile practices in small projects in these bureaucratic companies. Must be remembered that all of these companies use agile methodologies and frameworks; for example, Crystal, XP, Scrum, and Kanban. Most of these companies’ sprints are around four weeks where each interaction is a complete development life cycle. For agile practitioners, the most important points are a collaborative environment and organizational culture adaptive to changes. According to them, the agile approach requires self-organizing teams that have the power to coordinate their work without direct supervision. Fig. 6 shows the summary of the companies’ profile. Fig. 6. Number of primary study by companies’ size. D. What are the implications of critical factors studies in Agile Software Development field for the organization’s projects? Study [PS-04] argues that failures and success factors “can contribute to the understanding of how to avoid certain serious pitfalls that are critical to the success of a project”. In Table V, we classified 14 factors into people, process, and technology categories. According to [PS-023], these factors can help managers to prioritize tasks better and improve resource allocation. In the final analysis, the general implications of these factors are related to project success or failure. Although, it can be more slowly or not directly associated; for example, internal conflicts and duplicated work. A team with internal communication issues will have problems to share information among team members, so each member will need to find their our solution. As a result, collaborators will spend a lot of time and money with this approach. E. What are the main characteristics of the developed projects, according to number of collaborators by team, project duration, domain, and platforms used?? Primary study [PS-09] discusses the myth that “agility helps in any case”. The author explains the variety of applications, domains, and risks are different to each project; however, for [PS-09], the main reasons to use agile practices are to achieve shorter development cycles, be able to cope with changing requirements in a cost-effective manner, and have a close cooperation with the customer. The first minor question was about the number of collaborators by team. According to [PS-013], “the number of members in a team greatly influences the degree of communication that can be had between team members. For instance, in a team that has a large number of members, it is quite likely that frequent, informal, and rapid interactions will become difficult. The pace of informal communications can be made faster in small teams”. Our research contains small and large teams. In addition, these teams could be located in different countries, and working on the same projects [PS-06 and PS-023]. Primary studies [PS-04, PS-013, and PS-017] focus on the small teams, but they do have bigger teams where the number of team members are equal or greater than 20. Only studies [PS-05 and PS-016] contain just small teams. Fig. 7 shows the number of primary for each size of companies. Communication is present in our fourteen factors list. Based on our research, we defend that companies need to understand that communication is very critical for any project. In [PS-01], the author says that internal project communication between projects’ members can decrease the amount of team conflict and keeps the team stable. It is important to point out that instability can result in a project being delayed and exceeding budget. Software development activities require collaboration between team members, and these interactions can improve the knowledge management in the project. Fig. 7. Number of primary study by companies’ size. Similarly, study [PS-03] argues that people is a very important key in the software development process. Competence, expertise, and user participation are CSFs listed in the people category. In [PS-01], the author explains the relation between these two factors and inconsistent requirements. According to them, these factors are relevant to develop a software product that meets the client’s expectations. In addition, it will decrease process performance. Similarly, for project duration, we have short and long projects. In [PS-016], we found really small projects; for example, 2 or 3 months duration. [PS-05] contains projects which duration was about 2 years. In contrast, [PS-08], which does not comment about agile projects, contains projects that took more than four years to be completed; however, all of them encourage user participation in the development process to obtain rapid feedback. Our primary studies do not comment a lot about the domain or platforms used. In [PS-05], we found more details about the projects; for example, it says that the project 1 was a redevelopment of an existing system for the financial market, project 2 was a new e-commerce service, and project 3 was the maintenance of a recommendation system for products from several virtual stores. By the same token, [PS-016] comments about three cases. The first one is a finance critical software for public use. The second case contains two kinds of projects: a video codec product for mobile applications, and customer tailoring for the video codec product. The last case was a mobile version of the internet security system, and a monitoring system for the internet security system. Because we have a broad range of projects in terms of number of team members, project duration, domain, and complexity, we could not create a single profile of these projects. Our studies included both in-house and outsourced software development projects. In addition, the companies use several agile methodologies and frameworks; for instance, Crystal, XP, Scrum, and Kanban. CONCLUSION This systematic review identified 1387 unduplicated papers from the search on four electronic databases. We selected 23 to conduct this research, but only 20 contribute to answer our research questions. This study has been carried out in three main steps as follows: an extensive literature review to identify CSFs and factors related to agile software development, analysis of identified CSFs and the agile environment, development of a CSFs list based on people, process, and technology and explanation about organization's profile, projects’ characteristics and implications of CSFs. The list of critical success factors contains the most commons factors found in our primary studies. For people perspective, they are: competence and expertise, executive support, team and user motivation, team size, and user participation. Similarly, we have 4 factors for process list: agile practices, delivery most important features first, and right amount of documentation. The technology category is composed by: appropriate technical training, integration testing, simple design, tools supports, and well-defined coding standards. General implications of these factors are related to project success or failure. Our primary studies had a mix of companies and projects. Most of them were small companies with less than 10 employees, but they do have large cases. Our studies included both in-house and outsourced software development projects. As a result, we had companies from different markets. Although, according to the articles, the most important characteristics are collaborative environment and adaptive culture. We could not provide more evidences for the RQ-02 RQ-04 because the number of case studies was too small. This review has identified several attractive research challenges that need to be addressed in future researches. A clear finding of the review is that we need to increase the number of case studies in agile software development. In particular, to understand how companies of the modern world are using agile methodologies. Talk about future work in others databases APPENDIX A. PRIMARY STUDIES INCLUDED IN THIS REVIEW [PS-01] Ahimbisibwe, A., Cavana, R., & Daellenbach, U. (2015). A contingency fit model of critical success factors for software development projects. Journal Of Ent Info Management, 28(1), 7-33. doi:10.1108/jeim-08-2013-0060 [PS-02] Bano, M., & Zowghi, D. (2015). A systematic review on the relationship between user involvement and system success. Information And Software Technology, 58, 148-169. doi:10.1016/j.infsof.2014.06.011 [PS-03] Bermejo, P., Zambalde, A., Tonelli, A., Souza, S., Zuppo, L., & Rosa, P. (2014). Agile Principles and Achievement of Success in Software Development: A Quantitative Study in Brazilian Organizations. Procedia Technology, 16, 718-727. doi:10.1016/j.protcy.2014.10.021 [PS-04] Chow, T., & Cao, D. (2008). A survey study of critical success factors in agile software projects.Journal Of Systems And Software, 81(6), 961-971. doi:10.1016/j.jss.2007.08.020 [PS-05] de O. Melo, C., S. Cruzes, D., Kon, F., & Conradi, R. (2013). Interpretative case studies on agile team productivity and management. Information And Software Technology, 55(2), 412-427. doi:10.1016/j.infsof.2012.09.004 [PS-06] Drury-Grogan, M. (2014). Performance on agile teams: Relating iteration objectives and critical decisions to project management success factors. Information And Software Technology, 56(5), 506-515. doi:10.1016/j.infsof.2013.11.003 [PS-07] França, A., da Silva, F., & de Sousa Mariz, L. (2010). An empirical study on the relationship between the use of agile practices and the success of Scrum projects. Proceedings Of The 2010 ACMIEEE International Symposium On Empirical Software Engineering And Measurement - ESEM '10. doi:10.1145/1852786.1852835 [PS-08] Ghobadi, S., & D’Ambra, J. (2012). Coopetitive relationships in cross-functional software development teams: How to model and measure?. Journal Of Systems And Software, 85(5), 1096-1104. doi:10.1016/j.jss.2011.12.027 [PS-09] Hochmüller, E., & Mittermeir, R. (2008). Agile process myths. Proceedings Of The 2008 International Workshop On Scrutinizing Agile Practices Or Shoot-Out At The Agile Corral - APOS '08. doi:10.1145/1370143.1370145 [PS-10] Jorgensen, M. (2014). Failure factors of small software projects at a global outsourcing marketplace.Journal Of Systems And Software, 92, 157-169. doi:10.1016/j.jss.2014.01.034 [PS-11] Lehtinen, T., Mäntylä, M., Vanhanen, J., Itkonen, J., & Lassenius, C. (2014). Perceived causes of software project failures – An analysis of their relationships. Information And Software Technology, 56(6), 623643. doi:10.1016/j.infsof.2014.01.015 [PS-12] Madsen, K. (2005). Agility vs. stability at a successful start-up. Companion To The 20Th Annual ACM SIGPLAN Conference On Object-Oriented Programming, Systems, Languages, And Applications OOPSLA '05. doi:10.1145/1094855.1094966 [PS-13] Misra, S., Kumar, V., & Kumar, U. (2009). Identifying some important success factors in adopting agile software development practices. Journal Of Systems And Software, 82(11), 1869-1890. doi:10.1016/j.jss.2009.05.052 [PS-14] Moe, N., Dingsøyr, T., & Dybå, T. (2010). A teamwork model for understanding an agile team: A case study of a Scrum project. Information And Software Technology, 52(5), 480-491. doi:10.1016/j.infsof.2009.11.004 [PS-15] Petersen, K., Khurum, M., & Angelis, L. (2014). Reasons for bottlenecks in very large-scale system of systems development. Information And Software Technology, 56(10), 1403-1420. doi:10.1016/j.infsof.2014.05.004 [PS-16] Pikkarainen, M., Salo, O., Kuusela, R., & Abrahamsson, P. (2011). Strengths and barriers behind the successful agile deployment—insights from the three software intensive companies in Finland.Empirical Software Engineering, 17(6), 675-702. doi:10.1007/s10664-011-9185-5 [PS-17] Rodríguez, P., Markkula, J., Oivo, M., & Turula, K. (2012). Survey on agile and lean usage in finnish software industry. Proceedings Of The ACM-IEEE International Symposium On Empirical Software Engineering And Measurement ESEM '12. doi:10.1145/2372251.2372275 [PS-18] Senapathi, M., & Srinivasan, A. (2012). Understanding postadoptive agile usage: An exploratory cross-case analysis. Journal Of Systems And Software, 85(6), 1255-1268. doi:10.1016/j.jss.2012.02.025 [PS-19] Senapathi, M., & Srinivasan, A. (2013). Sustained agile usage. Proceedings Of The 17Th International Conference On Evaluation And Assessment In Software Engineering EASE '13. doi:10.1145/2460999.2461016 [PS-20] Shahane, D., Jamsandekar, P., & Shahane, D. (2014). Factors influencing the agile methods in practice - Literature survey & review. 2014 International Conference On Computing For Sustainable Global Development (Indiacom). doi:10.1109/indiacom.2014.6828020 [PS-21] Sheffield, J., & Lemétayer, J. (2013). Factors associated with the software development agility of successful projects. International Journal Of Project Management, 31(3), 459-472. doi:10.1016/j.ijproman.2012.09.011 [PS-22] Shrivastava, S., & Rathod, U. (2015). Categorization of risk factors for distributed agile projects.Information And Software Technology, 58, 373-387. doi:10.1016/j.infsof.2014.07.007 [PS-23] Stankovic, D., Nikolic, V., Djordjevic, M., & Cao, D. (2013). A survey study of critical success factors in agile software projects in former Yugoslavia IT companies. Journal Of Systems And Software,86(6), 1663-1678. doi:10.1016/j.jss.2013.02.027 REFERENCES [01] Beck K., Beedle M., Bennekum A., Cockburn A., Cunningham W., Fowler M., et al. (2001). Agile Manifesto. Retrieved 16 July 2015, from http://www.agilemanifesto.org/ [02] Bullen, C., & Rockhart, J. (1981). A primer on critical success factors (Working Paper No. 69), Massachusetts Institute of Technology, Sloan School of Management, Center for Information Systems Research, Cambridge, Massachusetts. [03] Hossain, E., Babar, M., & Paik, H. (2009). Using Scrum in Global Software Development: A Systematic Literature Review. 2009 Fourth IEEE International Conference On Global Software Engineering. doi:10.1109/icgse.2009.25 [04] Kitchenham, B. (2004). Procedures for Performing Systematic Reviews. [05] Nasir, M.H. & Sahibuddin, S. (2011). Critical success factors for software projects: a comparative study. Scientific Research and Essays, 6 (10), 2174-2186. [06] Petersen, K., Feldt, R., Mutjaba, S., & Mattson, M. (2015). Systematic Mapping Studies in Software Engineering. [07] Rockhart, J., & Crescenzi, A. (1984). Engaging top management in information technology. Sloan Management Review, 25 (4), 3–16. [08] Silva, K. (2015). Scripts to get articles info from ACM and Science Direct. GitHub. Retrieved 16 July 2015, from https://github.com/karlambsilva/get-articles-info-selenium [09] Silva, K. (2015). Selection Process :: Karla Silva. Google Sheets. Retrieved 14 July 2015, from https://docs.google.com/spreadsheets/d/1tfmuqZBv48OrlSSvpDYjI_qC R0yOUVeaGWLNkHCTuzI/edit?usp=sharing [10] Standish Group International Inc. (2012), “CHAOS 2012 report agile projects successful 3X more than non-agile projects”, Standish Group International Inc, Boston, MA, available at: www.sdsconsulting.com/blog/agile-projects-successful-3x-more-non-agileprojects (accessed August 2013). [11] Standish Group International Inc. (2012). CHAOS Manifesto 2012. Standish Group International Inc. Retrieved 20 July 2015, from http://www.versionone.com/assets/img/files/CHAOSManifesto2012.pdf