Introduction to Agile and Scrum Agile Methodology for Project Management Michael Vollmer School of Computing, University of Kent Table of contents 1. Module Information 2. Development Methodologies 3. Scrum Development Process 4. Summary Michael Vollmer Introduction to Agile and Scrum 1 / 25 Module Information COMP5590 Module Project • The groups will implement part of a general practitioner (GP) booking system • Groups are formed of four or five people from the same class slot. You can form your own groups • Interfaces: Patient (assigned to Groups A and D), Doctor (assigned to Groups B and E), Admin/Receptionist (assigned to Groups C and F) • Each group will also implement authentication and logging features • For A1, groups will write eight (four people groups) or 10 (five people groups) user stories • For A2, groups will then implement those user stories in three stages Michael Vollmer Introduction to Agile and Scrum 2 / 25 COMP5590 Assessment 1: User Stories Each group produces the following: 1. One user story for authentication 2. One user story for authorisation checks 3. Six (for a group of four) or eight (for a group of five) additional user stories depending on the features for your assigned interface At this point, the features of the project are intentionally described vaguely. You should explore these features within your group and come up with the corresponding user stories. Groups A and B present on Week 17. Groups C and D present on Week 18. Michael Vollmer Introduction to Agile and Scrum 3 / 25 Development Methodologies Why not just be perfect? Proposal: All software development should start with a precise specification of what exactly the software should do. Development then proceeds using formal methods to produce highly reliable software that matches the specification as closely as possible. Questions: • What’s wrong with this approach? • In what sorts of situations would this approach be appropriate? • In what sorts of situations would this approach not be appropriate? Michael Vollmer Introduction to Agile and Scrum 4 / 25 What might software engineering be? Mathematical culture • Mathematical proofs • Abstract data types • Structured programming Managerial culture • Control the workforce • Structured management of teams • Requirements and specifications Michael Vollmer Introduction to Agile and Scrum 5 / 25 Software development methodology When deciding how to go about developing software, let’s consider: • Should we follow some sort of structured process? • If so, what sort of process should we use? • How might we evaluate different processes? How do we avoid mindless rule-following? Michael Vollmer Introduction to Agile and Scrum 6 / 25 Considering different methodologies Is the methodology focused on engineers or managers? • Engineer: focus on tools and code • Manager: focus on team structure and bureaucracy Is the methodology focused on product or process? • Product: work based on a specification • Process: continually adapt as work goes on Michael Vollmer Introduction to Agile and Scrum 7 / 25 Waterfall methodology Product and manager focused, inflexible. Multi-stage process: 1. Gather requirements 2. Analyze and design 3. Programming 4. Testing 5. Deployment Michael Vollmer Introduction to Agile and Scrum 8 / 25 Cleanroom methodology • Work with exact specification, for high reliability software • Meant to invoke the “cleanrooms” used in the electronics industry to prevent introduction of defects during fabrication of semiconductors • Relies on formal verification and formal methods Michael Vollmer Introduction to Agile and Scrum 9 / 25 Chrysler Comprehensive Compensation System (C3) • Rapid software system development (in Smalltalk) • Replacing old COBOL systematicality • Focused on process and engineering • First project using Extreme Programming (XP) • Payroll system implemented in the mid-90s, development led by Kent Beck • After a few failed attempts, project succeeded and was live from 1997 to 1999, responsible for payroll handling over 10,000 people • Eventually rolled back, reverting to old COBOL system Michael Vollmer Introduction to Agile and Scrum 10 / 25 Extreme Programming • Engineering practices drive the process • Programming in pairs, extensive code review • Frequent releases • Test all code! Test-Driven Development (TDD) • Don’t implement features until they are needed (YAGNI) • Continuous integration, always have a running system • Heavy reliance on frequent communication Michael Vollmer Introduction to Agile and Scrum 11 / 25 The New New Product Development Game In today’s fast-paced, fiercely competitive world […] [c]ompanies are increasingly realizing that the old, sequential approach to developing new products simply won’t get the job done. Instead, companies in Japan and the United States are using a holistic method—as in rugby, the ball gets passed within the team as it moves as a unit up the field. — Takeuchi & Nonaka (1986) Michael Vollmer Introduction to Agile and Scrum 12 / 25 SCRUM methodology • Following the Rugby terminology • Maneger and process oriented • Empirical process • Three pillars: • Transparency: Everybody knows what’s going on • Inspection: Check your work as you are doing it • Adaptation: Adjust the direction when needed Michael Vollmer Introduction to Agile and Scrum 13 / 25 Manifesto for Agile Software Development (2001) We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: • Individuals and interactions over processes and tools • Working software over comprehensive documentation • Customer collaboration over contract negotiation • Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more. Michael Vollmer Introduction to Agile and Scrum 14 / 25 Scrum Development Process Scrum Roles Who does what in the team? • Product owner: customer representative, defines stories, prioritizes tasks, business focused • Scrum master: process facilitator, mentors team, removes obstacles, controls process • Development team: cross-functional team, contribute to coding + testing + debugging + planning • Executives: usually just get in the way, needed for the money Michael Vollmer Introduction to Agile and Scrum 15 / 25 Scrum Process Big ideas: • Backlog of user stories to implement • Planning based on value × effort • Sprints add functionality in one to two weeks “With scrum, a product is built in a series of iterations called sprints that break down big, complex projects into bite-sized pieces” — Megan Cook, Atlassian Michael Vollmer Introduction to Agile and Scrum 16 / 25 Product Backlog The Product Backlog is an emergent, ordered list of tasks necessary to complete the project. Product Backlog Refinement is the act of breaking down large tasks in the backlog into smaller, more manageable tasks. The Product Goal describes a future state of the product that the team is aiming for, which will be reached once the Backlog is complete. The Backlog emerges to define what will fulfill the Goal. Michael Vollmer Introduction to Agile and Scrum 17 / 25 Scrum Ceremonies Daily scrum 15 minute meeting • What you did yesterday • Pick tasks for today • What is blocking you Sprint-related meetings • Planning to choose tasks for next sprint • Review to engage with stakeholders • Retrospective to continuously improve Michael Vollmer Introduction to Agile and Scrum 18 / 25 Sprint Planning • Decide what can be done in the sprint, and how it will be done • What: Product owner describes ogoal of sprint and what backlog items contribute to that goal. Scrum team decides what subset of that can be done during the sprint. • How: Development team works out how to actually do the necessary work to achieve the goal. • Items from the backlog may be scheduled to be done during the sprint only if they are small enough to be completable during a single sprint. Otherwise, refinement is necessary. • The main input going into the meeting is the backlog, as it provides the list of tasks from which the sprint’s goal will be determined. • The output of the meeting should be the selection of the goal for the sprint, made visible via the backlog. Michael Vollmer Introduction to Agile and Scrum 19 / 25 Burndown Chart Graphical representation of work left to do vs. time. Remaining work is on the y-axis, elapsed time is on the x-axis. Michael Vollmer Introduction to Agile and Scrum 20 / 25 Workflow and Kanban boards Invented in Japan in the late 1940s at Toyota, kanban boards are used to visualize workflow. Tasks move through steps, visualized as columns. Michael Vollmer Introduction to Agile and Scrum 21 / 25 GitLab workflow Michael Vollmer Introduction to Agile and Scrum 22 / 25 Roadmap A Roadmap is a flexible plan of action for achieving your product vision. Michael Vollmer Introduction to Agile and Scrum 23 / 25 Lightweight requirement capture • Agile methodologies • Formal specifications are too hard • How do we get customers involved? • What is a good way of talking to them? • User stories • Not requirements, but stories • Use paper or Word or whatever works • As a user I want something so that benefit • Fixed story format: user and benefit Michael Vollmer Introduction to Agile and Scrum 24 / 25 Summary Agile Development Development methodologies Waterfall, extreme programming, SCRUM Managerial vs. engineering; process vs. product SCRUM development methodology Product owner, scrum master, development team Backlog, sprint meetings, daily scrum meeting, burndown chart Michael Vollmer Introduction to Agile and Scrum 25 / 25 Code Quality Readable, Reliable, and Maintainable Code Michael Vollmer School of Computing, University of Kent Table of contents 1. Big Idea 2. Structure 3. Style 4. Documentation 5. Java-specific 6. Summary Michael Vollmer Code Quality 1 / 29 Big Idea Software Maintenance “Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live” — John Woods You don’t just write code once and leave it alone. Software systems evolve over time, and as they evolve they grow more complex. Michael Vollmer Code Quality 2 / 29 Readability “We want to establish the idea that a computer language is not just a way of getting a computer to perform operations but rather that it is a novel formal medium for expressing ideas about methodology. Thus, programs must be written for people to read, and only incidentally for machines to execute.” — Structure and Interpretation of Computer Programs Michael Vollmer Code Quality 3 / 29 Software Development Remember: software development is a social process. Michael Vollmer Code Quality 4 / 29 Structure GOTO Considered Harmful “Go To Statement Considered Harmful” Famous letter by Edsger W. Dijkstra, printed in Communications of the ACM, Vol. 11, No. 3, March 1968. Michael Vollmer Code Quality 5 / 29 GOTO Considered Harmful Compare this: 10: I = 1 20: PRINT "HI" 30: I = I + 1 40: IF I <= 10 THEN GOTO 20 To this: FOR I = 1 TO 10 DO PRINT "HI" NEXT Michael Vollmer Code Quality 6 / 29 Structured Programming • Coined by Edsger W. Dijstra, first associated with early programming language ALGOL (Algorithmic Language). • Three basic control structures: • Sequence: Ordered statements executed in sequence. • Selection: One or more statements/blocks executed conditionally based on program state. Common forms include if , case, etc. • Iteration: A statement/block executed repeatedly until the program reaches a certain state. Common forms include while, for, etc. • Subroutines: Callable units such as procedures, functions, methods. • Blocks: Enable groups of statements to be treated as if they were one statement. Usually wrapped in curly braces { ... } Michael Vollmer Code Quality 7 / 29 Structured Programming Theorem The structured programming theorem provides the theoretical foundation for structured programming. According to the theorem, the three basic control structures (sequence, selection, and iteration) are sufficient to express all possible computable functions. Michael Vollmer Code Quality 8 / 29 Single Exit Point • Rule: A function/method should only ever return from one location. In other words, there should only be a single exit point. • Sometimes called the “single-entry/single-exit” rule. • Not always followed, there are exceptions. • Predates Java, arguably made even more sense in older languages. • Still often good advice, but don’t mindlessly follow rules. Michael Vollmer Code Quality 9 / 29 Single Exit Point: Example p u b l i c bool containsElement ( i n t element ) { for ( int i = 0; i < containerSize ; i ++) { i f ( check ( i , element ) ) { return true ; } } return false ; } p u b l i c bool containsElement ( i n t element ) { bool found = f a l s e ; f o r ( i n t i = 0 ; i < c o n t a i n e r S i z e && ! found ; i + + ) { i f ( check ( i , element ) ) { found = t r u e ; } } r e t u r n found ; } Michael Vollmer Code Quality 10 / 29 Single Exit Point: Example p u b l i c bool containsElement ( i n t element ) { // a c q u i r e access to r e s o u r c e Resource r = g l o b a l R e s o u r c e . g e t ( ) ; bool found = f a l s e ; f o r ( i n t i = 0 ; i < c o n t a i n e r S i z e && i f ( check ( i , element , r ) ) { found = t r u e ; } } ! found ; i + + ) { r . r e l e a s e ( ) ; // r e l e a s e access to r e s o u r c e LOG . record ( element , found ) ; // log r e s u l t r e t u r n found ; } Michael Vollmer Code Quality 11 / 29 When to Exit Early • Exceptions! • Guards/checks at top of function • Shorter methods p u b l i c S t r i n g check ( i n t i ) { i f ( i < 10) { return ” a ” ; } e l s e i f ( i == 0 } { return ”b” ; } else { return ” c ” ; } } Michael Vollmer Code Quality 12 / 29 Early Exit: Example p u b l i c bool containsElement ( i n t element ) { i f ( i n v a l i d I n p u t ( element ) ) { r e t u r n f a l s e ; // e a r l y e x i t guard } bool found = f a l s e ; f o r ( i n t i = 0 ; i < c o n t a i n e r S i z e && i f ( check ( i , element ) ) { found = t r u e ; } } ! found ; i + + ) { r e t u r n found ; } Michael Vollmer Code Quality 13 / 29 Reduce Unnecessary Nesting for ( int i = 0; i < n ; i ++) { foo ( a [ i ] ) ; while ( . . . ) { ... } } for ( int i = 0; i < n ; i ++) { foo ( a [ i ] ) ; other ( a , i , n ) ; } void other ( char [ ] a , i n t i , i n t n ) { ... } Michael Vollmer Code Quality 14 / 29 Reduce Unnecessary Nesting if (a) { i f (b) { // . . . } } i f ( a && b ) { // . . . } Michael Vollmer Code Quality 15 / 29 Keep program logic simple Compare this: i f ( i s A d m i n i s t r a t o r ( user ) | | isOwner ( user ) ) return true ; else return false ; To this: r e t u r n i s A d m i n i s t r a t o r ( user ) | | isOwner ( user ) ; You don’t need the conditional at all! Michael Vollmer Code Quality 16 / 29 Remove Duplicate Code • Duplicate code makes program longer and harder to read. • Duplicate code makes further changes harder and more likely to introduce errors, because changes need to be made in multiple places. • Duplicate code makes bug fixes harder, because changes need to be made in multiple places. • Avoid duplicate code by factoring it: new method, class, etc. Michael Vollmer Code Quality 17 / 29 Aside: Higher-order Functions Structured programming is procedural. An alternative paradigm of programming is functional programming. Procedure calls were once thought too expensive to be widely used, whereas GOTO statements and special iteration control forms were thought to be cheap. This was debunked in the classic paper, Lambda: The Ultimate GOTO, by Guy Steele Jr. All the control forms of structural programming can be implemented with only lambda. Michael Vollmer Code Quality 18 / 29 Style Naming • Class names use UpperCamelCase. • Method names use lowerCamelCase. • Constant names use UPPER_SNAKE_CASE. • Non-constant field names, local variable names, and parameter names use lowerCamelCase. • Type variable names are either a single capital letter (for example, A) or a class name followed by the letter T (for example, ListT ). • Always try to use descriptive names, rather than just x1, x2, etc. Michael Vollmer Code Quality 19 / 29 Blocks and Curly Braces • Kernighan and Ritchie style (“Egyptian brackets”) • No line break before the opening brace, except as detailed below. • Line break after the opening brace. • Line break before the closing brace. • Line break after the closing brace, only if that brace terminates a statement or terminates the body of a method, constructor, or named class. For example, there is no line break after the brace if it is followed by else or a comma. Michael Vollmer Code Quality 20 / 29 K & R style example r e t u r n new MyClass ( ) { @Override p u b l i c void method ( ) { i f ( condition ( ) ) { try { something ( ) ; } catch ( ProblemException e ) { recover ( ) ; } } else i f ( otherCondition ( ) ) { somethingElse ( ) ; } else { lastThing ( ) ; } } } Michael Vollmer Code Quality 21 / 29 Indentation, line wrapping, whitespace • Indent should match block scope. • Each time a new block or block-like construct is opened, the indent increases by one unit. • How much should the indent be? Project dependent, but should be consistent. Good default: 4 spaces. • On line wrap, the wrapped line should be indented at least +4 spaces past original line. • For readability, use a single space between some constructs like if , for, etc., and the first open parenthesis. • Similarly, surround binary operators with single spaces (so like a + b, not a+b). • Horizontal alignment of variable declarations, comments, etc., is allowed but not required. Michael Vollmer Code Quality 22 / 29 Auto-formatting Most IDEs have some sort of auto-format function. The configuration can be saved and shared across a team. Michael Vollmer Code Quality 23 / 29 Documentation In-line Comments For implementation details, different from Javadoc comments (see next slide). /* * This i s * okay . */ // And so // i s t h i s . / * Or you can * even do t h i s . */ Block comments are indented at the same level as the surrounding code. They may be in /* ... */ style or // ... style. For multi-line /* ... */ comments, subsequent lines must start with * aligned with the * on the previous line. Michael Vollmer Code Quality 24 / 29 Javadoc / ** * M u l t i p l e l i n e s o f Javadoc t e x t are w r i t t e n here , * wrapped normally . . . */ p u b l i c i n t method ( S t r i n g p1 ) { ... } / ** An e s p e c i a l l y s h o r t b i t o f Javadoc . */ Each Javadoc block begins with a brief summary fragment, not a complete sentence. So, instead of “This method returns the item”, use “Returns the item.” You should try to have a Javadoc for every public class, method, and field. Michael Vollmer Code Quality 25 / 29 Java-specific Overriding Always use the @Override annotation whenever you can. // mark method as a s u p e r c l a s s method // t h a t has been o v e r r i d d e n @Override i n t overriddenMethod ( ) { // . . . } Do this so you can take advantage of the compiler’s static checking. Michael Vollmer Code Quality 26 / 29 Caught exceptions It is very rarely correct to do nothing in response to a caught exception. What do you do instead? • Handle it. • Log it. • If it is “impossible”, rethrow it as an AssertionError. • If no other option, explain why in a comment. try { i n t i = I n t e g e r . p a r s e I n t ( response ) ; r e t u r n handleNumericResponse ( i ) ; } catch ( NumberFormatException ok ) { // i t ’ s not numeric ; t h a t ’ s f i n e , j u s t co nt i n u e } r e t u r n handleTextResponse ( response ) ; Michael Vollmer Code Quality 27 / 29 Static members When referencing a static class member that needs to be qualified, try to use the class’s name. Foo aFoo = . . . ; Foo . aStaticMethod ( ) ; // good aFoo . aStaticMethod ( ) ; // bad somethingThatYieldsAFoo ( ) . aStaticMethod ( ) ; // v e r y bad Michael Vollmer Code Quality 28 / 29 Summary Code Quality Make code straightforward Code is meant to be read, not just run. Care about the people who have to read it, including yourself! Coding conventions There are standard rules for writing and formatting Java code, though the exact details are not set in stone. Michael Vollmer Code Quality 29 / 29 Lecture 7 Design Patterns Dr. Michael Vollmer m.vollmer@kent.ac.uk CO871—Advanced Java for Programmers University of Kent, School of Computing 2022-10-10 Mon Outline Overview Interpreter pattern Visitor pattern Brief tour of some other patterns Presentation agenda Overview Interpreter pattern Visitor pattern Brief tour of some other patterns Reusable software I Goal of software engineering, especially object-oriented software engineering I Factor objects into classes I Define class interfaces and inheritance hierarchies I Identify relationships I Don’t solve every problem from first principles I Reuse solutions I Particularly patterns of classes and communicating objects Design patterns I Describe a common recurring problem and the core of a solution to that problem. I Make it easier to reuse successful designs and architectures I Help to: I Choose alternatives I Avoid compromising reusability I Improve documentation I See: Design Patterns: Elements of Reusable Object-Oriented Software by Gamma, Helm, Johnson, and Vlissides How do you use design patterns? I You’ve used off-the-shelf libraries and frameworks before, right? I Design patterns do not go straight into your code, they first go into your brain. I You first need to learn what they are and how they work. I Then you can identify problems that fit with those that a particular design pattern aims to solve. I Then you can apply them in your design. Essential elements I Name – Increases vocabilary and allows design at a higher level, discussion, etc. I Problem – When to apply the pattern, conditions that must be met, how to represent the algorithm as objects, etc. I Solution – The elements that make up the design, their relationships, collaborations, responsibilities. A template that can be applied in many different situations. I Consequences – Results and trade-offs, impact on flexibility, extensibility, and portability. Common design patterns I Observer I Strategy I Decorator I Singleton I Factory I Command I Adapter I Facade See: Head First Design Patterns Example problem: arithmetical expressions, like 3 ⇤ (4 + 5) As part of a compiler project, we want to be able to handle simple arithmetical expressions. At the very least, we want to be able to: I evaluate them, and I pretty-print them. In the future, we may wish to: I perform more computations (cost metrics, common sub-expression elimination, etc), and I include new arithmetic operators. We require: I a data representation, and I a way of interacting with it. Presentation agenda Overview Interpreter pattern Visitor pattern Brief tour of some other patterns The pattern The pattern I Intent – Define a representation for a language grammar (an abstract syntax tree) and an interpreter for sentences in the language. I Participants I classes for each node in the AST, derived from abstract class AbstractExpression I concrete classes for each TerminalExpression, which implement interpret() for their terminal symbol I classes for each NonterminalExpression in the AST. Given a rule R ::= R1 R2 . . . Rn in the grammar, a NonterminalExpression will have an AbstractExpression attribute for each of the Ri . Typically interpret() calls itself recursively on each of these. The pattern I Context – Any information used by all the interpret() operations. I Client – Parses a sentence of the language and builds the AST from the TerminalExpression and NonterminalExpression classes. It then calls the interpret() method on the root of the AST. The implementation public abstract class SimpleNode { public abstract eval(); } public class ASTInteger extends SimpleNode { // value public int eval() { return getValue(); } } public class ASTAdd extends SimpleNode { // left and right public int eval() {return left().eval() + right().eval();} } ASTStart ast = parser.start(); //parse the expression int result = ast.eval(); Applicability, collaboration, and consequences I Language to interpret; abstract syntax tree I Client builds the AST, initialises the context and invokes interpret on the root I Interpret uses context to access state of interpreter I Consequences I Easy to change and extend the grammar I Easy to implement the grammar I Large grammars hard to maintain – multiple classes I Hard to add new computations over the expression—need to change every class Presentation agenda Overview Interpreter pattern Visitor pattern Brief tour of some other patterns The pattern The pattern I Intent – Represents a computation to be performed on an object structure. I Participants I ObjectStructure: Classes derived from abstract class or interface Element, which defines an abstract accept() operation whose argument is a Visitor. I Concrete classes for each different ConcreteElement (node) in the object structure (AST). Each such class must implement accept(). I Abstract class or interface Visitor that defines an abstract visit() computation for each ConcreteElement in the object structure. I A specific ConcreteVisitor class can be derived from Visitor to implement each computation to be performed on the object structure. I The Client parses a sentence of the language and builds the AST from the TerminalExpression and NonterminalExpression classes, as before. It then creates a new ConcreteVisitor object. The visitor is passed as the argument to the accept() method of the root of the AST. The implementation public interface SimpleVisitor { public int visit(ASTAdd node); public int visit(ASTInteger node); } public class SimpleEvalVisitor implements SimpleVisitor { public int visit(ASTAdd node) { return node.left().accept(this) + node.right().accept(this); } public int visit(ASTInteger node) { return node.getValue(); } } The implementation public interface Element { public int accept(SimpleVisitor v); } public class ASTInteger implements Element { public int accept(SimpleVisitor v) { v.visit(this); } } ASTStart ast = parser.start(); SimpleVisitor v = new SimpleEvalVisitor(); int result = ast.accept(v, 0); Applicability, collaboration, and consequences I Applicability I Object structure contains many classes I Object structure rarely changes but new computations are common I Collaboration I Client builds AST, creates visitor and asks root to accept the visitor. I Consequences I Easy to add new computations over the expression I Visitor gathers related computations and separates unrelated ones I Can visit across class hierarchies I Can accumulate state I Hard to add new elements—need to change every visitor class I Breaks encapsulation (visitor reads internal state of objects it visits) Presentation agenda Overview Interpreter pattern Visitor pattern Brief tour of some other patterns Decorator Decorators are used to modify the functionality of objects at runtime. public interface Computer { public void build(); } public class CheapPC implements Computer { public void build() { buildPC(); } } public abstract class ComputerDecorator implements Computer { private Computer c; public ComputerDecorator(Computer c) { this.c = c; } public void build() { this.c.build(); } } Decorator public class GamingPC extends ComputerDecorator { public GamingPC(Computer c) { super(c); } public void build() { super.build(); System.out.println("adding gaming GPU"); System.out.println("adding RGB lights"); } } public class Workstation extends CompterDecorator { public Workstation(Computer c) { super(c); } public void build() { super.build(); System.out.println("adding ECC RAM"); } } Singleton The singleton pattern resticts the instantiation of a class, ensuring only one instance of the class exists. public final class MySingleton { private static final MySingleton instance; private MySingleton() {} public static MySingleton getInstance() { if (instance == null) { instance = new MySingleton(); } return instance; } // ... } This pattern is controversial. Some people say it should not be a pattern at all. Build Systems Automatically building software artifacts Michael Vollmer School of Computing, University of Kent Table of contents 1. The Problem 2. Dependency Graphs 3. Make and Makefiles 4. Maven and Dependency Management Michael Vollmer Build Systems 1 / 26 The Problem Multi-file projects Most software has more than one source code file! Large projects may have hundreds or thousands. The question of “how do you compile all the source code files” becomes complicated at such a large scale. Michael Vollmer Build Systems 2 / 26 Incremental builds A naive approach would be some sort of script that searches for all the source files in the project and invokes the compiler on all of them. Can we do better than this? Michael Vollmer Build Systems 3 / 26 Incremental builds A naive approach would be some sort of script that searches for all the source files in the project and invokes the compiler on all of them. Can we do better than this? If we only change a single file, do we really need to re-compile all the other files? Michael Vollmer Build Systems 3 / 26 Extremely long build times • Even with incremental builds, it can take a long time to build big programs! • Steep hardware requirements, multi-hour build processes • Software companies these days often rely on build farms • Ongoing research in optimising build times Michael Vollmer Build Systems 4 / 26 Dependency Graphs Graph (1 of 4) A B C D E G H F Michael Vollmer Build Systems 5 / 26 Graph (2 of 4) A B C D E G H F Michael Vollmer Build Systems 6 / 26 Graph (3 of 4) A B C D E G H F Michael Vollmer Build Systems 7 / 26 Graph (4 of 4) A B C D E G H F Michael Vollmer Build Systems 8 / 26 Make and Makefiles What is make and why use it? make is a widely used build tool for unix/linux environments. It is a versatile tool: • Works for different programming languages (and even non-programming tasks!) • Allows for complex patterns and dependencies • Can execute arbitrary commands or even invoke itself recursively • You can build code without knowing or caring how it works Michael Vollmer Build Systems 9 / 26 You need a Makefile Write this to a file named Makefile hello: echo "Hello, World" We can execute this with: $ make hello echo "Hello, World" Hello, World However, if we create a file called hello and try again, nothing happens: $ touch hello $ make hello make: Nothing to be done for 'hello' Michael Vollmer Build Systems 10 / 26 Makefile targets Your Makefile will consist of a series of rules of this form: targets: prerequisites command command command Read this rule as: we will build target by runing commands, and it depends on prerequisites. For example: hello.o: hello.c gcc -c hello.c -o hello.o This rule will only run if hello.o doesn’t exist, or if hello.c has been modified more recently than hello.o. Michael Vollmer Build Systems 11 / 26 More example Makefile rules foo.o: foo.c gcc -c foo.c -o foo.o bar.o: bar.c gcc -c bar.c -o bar.o main: main.c foo.o bar.o gcc main.c foo.o bar.o -o main Assuming foo.c, bar.c, and main.c all exist and haven’t been compiled, what will happen if we run make main? What if, after that, we edit foo.c and want to recompile main. What will happen? Michael Vollmer Build Systems 12 / 26 Makefile variables You can define and reference variables in a Makefile. You can assign lists of strings to variables. files := foo.o bar.o main: main.c $(files) gcc main.c $(files) -o main Michael Vollmer Build Systems 13 / 26 Makefile automatic variables There are some special built-in “automatic” variables. The $@ variable refers to the current target. This is useful when there are multiple targets in a rule: foo.o bar.o: echo $@ The $^ variable refers to the whole list of prerequisites, while the $? rule refers to the list of prerequisites that are newer than the target. Michael Vollmer Build Systems 14 / 26 PHONY targets: all and clean We often have targets like all and clean like this: clean: rm -f *.o all: a b c However, what happens if there actually is a file called all or clean? .PHONY: clean all Michael Vollmer Build Systems 15 / 26 Makefile wildcards You can use the wildcard function and * to match multiple files. sources := $(wildcard *.c) You can also use the % wildcard to do matching: %.o: %.c gcc -c $^ -o $@ This is a common rule that appears in Makefiles for C programs. Michael Vollmer Build Systems 16 / 26 Makefile static patterns We can define static patterns based on lists of targets. targets...: target-pattern: prereq-patterns ... commands For example, if we wanted a rule to handle a particular list of object files: objects = foo.o bar.o $(objects): %o: %s gcc -c $^ -o $@ all: $(objects) Michael Vollmer Build Systems 17 / 26 Makefile filters We can use filter to filter out strings from lists. sources := foo.c bar.c baz.s ugh.h foo: $(sources) cc $(filter %.c %.s,$(sources)) -o foo This says that foo depends on foo.c, bar.c, baz.s, and ugh.h, but only foo.c, bar.c, baz.s should be specified in the command. Michael Vollmer Build Systems 18 / 26 Do I need a Makefile for my Java program? Short answer: probably not, but it’s still important to learn this stuff. Slightly longer answer: • Some people do use make with Java, but it is unusual these days • make is widely used with other programming languages (C, C++, etc.) • Java-specific tools like Ant and Maven are preferred for Java development Michael Vollmer Build Systems 19 / 26 Maven and Dependency Management Building Java specifically We can use make for Java, but there are other specialised tools we could use instead. For example, Java code follows a rigid structure: folders are packages, classes are individual files, etc. A tool for building Java programs doesn’t need to be told where to find the Java classes or what they’re named. Michael Vollmer Build Systems 20 / 26 External dependencies An issue that becomes very important when building large systems is handling external dependencies, such as frameworks and libraries. Many languages these days, including Java, have a standard way of declaring and installing these sorts of external dependencies. A project will declare what it depends on, and the tool will fetch the required artifacts. This is complicated by versions. If you have two Java programs that both depend on a particular library, but a different version of the library, how do you build them both? Michael Vollmer Build Systems 21 / 26 Maven • Maven is a tool for building Java programs • Maven can do lots of things! Build JARs, fetch library dependencies, run tests, etc. • Maven repositories are a database of build artifacts and dependency details • Maven is opinionated: it has default settings which will usually work for a reasonable project Michael Vollmer Build Systems 22 / 26 Maven example (1/4) Assume we have a HelloWorld class at path src/main/java/hello/HelloWorld.java: package hello; public class HelloWorld { public static void main(String[] args) { Greeter greeter = new Greeter(); System.out.println(greeter.sayHello()); } } And a Greeter class at path src/main/java/hello/Greeter.java: package hello; public class Greeter { public String sayHello() { return "Hello world!"; } } Michael Vollmer Build Systems 23 / 26 Maven Example (2/4) We need to create a pom.xml (Project Object Model) file: <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi=" http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https: //maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>uk.ac.kent</groupId> <artifactId>example</artifactId> <packaging>jar</packaging> <version>0.1.0</version> </project> Michael Vollmer Build Systems 24 / 26 Maven Example (3/4) We can automatically compile our Java code with mvn compile $ mvn compile [INFO] Scanning for projects... [INFO] [INFO] ----------------------------< uk.ac.kent:example >-----------------------[INFO] Building example 0.1.0 [INFO] --------------------------------[ jar ]----------------------------------[INFO] [INFO] --- maven-resources-plugin:2.6:resources (defaultresources) @ example --... At this point, the *.class files are in the target/ directory. If we want to build a JAR file, we can run mvn package. Michael Vollmer Build Systems 25 / 26 Maven Example (4/4) If we want to add an external dependency like JUnit 4, we can insert this into the pom.xml: <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> <scope>test</scope> </dependency> This tells Maven which version of JUnit we want, and what scope it should be available in (here, the “test” scope). Maven will automatically fetch JUnit 4.12 when needed. Michael Vollmer Build Systems 26 / 26 Open Source Software Michael Vollmer School of Computing, University of Kent Table of contents 1. Background 2. Open source models 3. Contributing to open source software 4. Summary Michael Vollmer Open Source Software 1 / 21 Background MIT Hacker Culture TMRC: Tech Model Railroad Club Access to computers should be unlimited. All information should be free! Michael Vollmer Open Source Software 2 / 21 Free Software Free as in freedom, not as in free beer. GNU GPL (1989) • Restrictive: You must release changes using the same license. • Hacker-centric: User is also a programmer. Michael Vollmer Open Source Software 3 / 21 Windows vs. Linux • Commercial software • Business approach to software • Source code owned by corporation • Funded by selling software • Free software • Hacker culture of software • Code is free to modify and reuse • Funded by additional services Michael Vollmer Open Source Software 4 / 21 Windows + Linux • Commercial interests • Not selling software but providing services • For-profit interests sometimes no longer opposed to free software • New generation of open source software • Open source without the “restrictive” licenses • Corporate-friendly open source Michael Vollmer Open Source Software 5 / 21 Open source software It’s much more than free vs. commercial license • A lot of software is not a final end-user product • Search for sustainable open source funding • Who is in control of open source software? • Contributing and accepting contributions Michael Vollmer Open Source Software 6 / 21 Sustainable open source funding Crucial safe internet infrastructure receives donations of $2000 per month to pay one developer. Michael Vollmer Open Source Software 7 / 21 Welcoming community “Please just kill yourself now. The world will be a better place.” – Linus Torvalds Becoming a contributor is hard without guidance. Attracting contributors takes a lot of effort and energy. Michael Vollmer Open Source Software 8 / 21 Code of conduct • “Be nice” is not enough. • Clear set of rules to be followed, even by “key” contributors • Process for resolving and reporting issues that arise. Michael Vollmer Open Source Software 9 / 21 Open source models Free software model • Philosophy and ideals • Built by hackers for hackers • Pay not with money, but with code • Open decisions (rarely the case) • Licenses and tools • GPL license requires publishing changes • LGPL allows linking in proprietary software • Distributed version control (git) Michael Vollmer Open Source Software 10 / 21 Dual licensing model • Philosophy and ideals • Free for hackers, paid otherwise • Pay with either money or code • Commercial, but open ownership • Licenses and tools • AGPL closes service provider loophole • Using database forces you to share your code • Commercial license if you don’t want to Michael Vollmer Open Source Software 11 / 21 Consulting model • Philosophy and ideals • Share code and allow all uses • Find alternative funding source • Commercial services, training • Licenses and tools • MIT, BSD, Apache • Allow unrestricted use without requirements Michael Vollmer Open Source Software 12 / 21 Corporate open source model • Philosophy and ideals • Developer tools should be open • Even those from companies • Supports other profitable business • Project ownership remains corporate • Licenses and tools • MIT, BSD, Apache (again) • Share code on GitHub for community reasons Michael Vollmer Open Source Software 13 / 21 Community open source model Alternative for community projects Free as in free beer (with donations) Like corporate model without corporations Is this sustainable? Michael Vollmer Open Source Software 14 / 21 Contributing to open source software Contributing to open source • What open source projects need to live • Maintainer: 1 hour a day, week, or month. Someone to merge changes, build releases, explain issues. • Core developers: irregular, but more time. Implements major new features in the software. • Contributors: A couple of hours or days. Improvements, bug fixes, better documentation. • Even small things matter • Minor bug fixes and typos • Documentation improvements • Good bug reports • How to do it well • Follow style and contributing guidelines • Report issues with minimal example • Small to large builds trust & communicate well Michael Vollmer Open Source Software 15 / 21 Good contributing Read the guidelines: On GitHub, file often called CONTRIBUTING.md. Follow guidelines: Where to discuss, how to comment and explain, etc. Follow coding style: Code indentation, brackets, etc. Explain things well: Good title and detailed description. Michael Vollmer Open Source Software 16 / 21 Reporting issues is also contributing Writing a good issue is a valuable skill! • Minimal: As little code and data as possible. • Complete: Provide all relevant parts of code. • Verifiable: Describe and test the behavior Michael Vollmer Open Source Software 17 / 21 Minimal, complete, verifiable example (1/3) Broken function to check palindromes: function palindrome(s) { for(var i = 0; i < s.length / 2 - 1; i++) if (s[i] != s[s.length-1-i]) return false; return true; } Mostly works, but not always! palindrome("step on no pets") == true palindrome("hello world") == false palindrome("step onto pets") == true // ???? Michael Vollmer Open Source Software 18 / 21 Minimal, complete, verifiable example (2/3) “I tried calling palindrome(text) and it didn’t work!” Not complete: What is the text you used? Not verifiable: in what way does it not work? Michael Vollmer Open Source Software 19 / 21 Minimal, complete, verifiable example (2/3) “I tried calling palindrome(text) and it didn’t work!” Not complete: What is the text you used? Not verifiable: in what way does it not work? “I tried calling palindrome("step onto pets") and it didn’t work!” Complete: sample input is now included Not verifiable: in what way does it not work? Michael Vollmer Open Source Software 19 / 21 Minimal, complete, verifiable example (3/3) “When I call palindrome("step onto pets") I get true but the input string is not a palindrome!” Complete: sample input is included Verifiable: gives expected and actual result Not minimal: input is longer than necessary Michael Vollmer Open Source Software 20 / 21 Minimal, complete, verifiable example (3/3) “When I call palindrome("step onto pets") I get true but the input string is not a palindrome!” Complete: sample input is included Verifiable: gives expected and actual result Not minimal: input is longer than necessary “When I call palindrome("x12x") I get true but the input string is not a palindrome!” Michael Vollmer Open Source Software 20 / 21 Summary Open source software • Recent history of open source software • How Microsoft adopted open source§ • Sustainable funding and welcoming community • Open source philosophies and funding • Free software or dual licensing model • Consulting, corporate, and community model • Contributing to open source software • Following style, code, and comment guidelines • Reporting minimal, complete, verifiable examples Michael Vollmer Open Source Software 21 / 21 Software Engineering Principles Michael Vollmer School of Computing, University of Kent Table of contents 1. Motivation 2. Complexity 3. Work 4. Cures 5. Summary Michael Vollmer Software Engineering Principles 1 / 21 Motivation Fundamental Knowledge What knowledge about software will remain relevant in 100 years? Michael Vollmer Software Engineering Principles 2 / 21 Fundamental Software Engineering Knowledge • The Halting Problem • It is impossible to give a program Θ that, for any other program p, decides whether p terminates or not. • Critically important concept in computer science. • Software Engineering • What is impossible to build? • Not a formal mathematical problem • Can we say anything certain at all? Michael Vollmer Software Engineering Principles 3 / 21 Historically Situated Knowledge Learning from the successes and the failures of the past. Michael Vollmer Software Engineering Principles 4 / 21 Learning from other Disciplines Is software engineering like real engineering, writing, or urban planning? Michael Vollmer Software Engineering Principles 5 / 21 Complexity Which software system is more complex to build? Chess engine that can consistently win against grandmasters? vs. Accounting system that calculates and pays VAT in UK and France? Michael Vollmer Software Engineering Principles 6 / 21 IBM System/360 Os for a very wide range of IBM machines Led by Fred Brooks Mythical Man-month Reflections on its problematic history Michael Vollmer Software Engineering Principles 7 / 21 Product: General, debugged, tested System: Set of compatible tools Principle #1: Building programming systems product is 9× harder than building a program Michael Vollmer Software Engineering Principles 8 / 21 Sources of Software Complexity No Silver Bullet (Brooks, 1986) “Much of the complexity [software engineers] must master is arbitrary complexity, forced without rhyme or reason by the many human institutions and systems to which [their] interfaces must conform.” Michael Vollmer Software Engineering Principles 9 / 21 No Silver Bullet • Essential complexity • Large with no two parts alike • Complexity of logic is essential • Non-linear nature of software • Accidental complexity • Imperfect programming tools • Unless this is more than 9/10, order of magnitude improvement is impossible Michael Vollmer Software Engineering Principles 10 / 21 Principle #2 “There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity.” Michael Vollmer Software Engineering Principles 11 / 21 Work IBM System/360 Significantly over budget – but it did not destory IBM Significantly delayed – but it shipped eventually IBM hired more programmers – but it did not help Michael Vollmer Software Engineering Principles 12 / 21 Workload balancing Unpartitionable work: Work that takes the same amount of time regardless of the number of people assigned: t = c. Perfectly partitionable; Time decreases with number of people: t = w/n. Requiring communication: Workers need to coordinate and be trained: t = (w + k)/n Complex task: Each worker needs to coordinate with each other: t = w/n + k ∗ n ∗ (n − 1). Michael Vollmer Software Engineering Principles 13 / 21 Principle #3 Adding manpower to a late software project makes it later. Michael Vollmer Software Engineering Principles 14 / 21 Further sources of complexity We do not even know what software should do! • Every problem is unique with little repetition • There might not be a stopping rule • Solutions are not right or wrong • Social scientists call this Wicked Problems Michael Vollmer Software Engineering Principles 15 / 21 Cures Mythical Man-month Second system effect: What happens to a designer’s second attempt at designing a system. Throw one away: The first system may be too slow, too big, awkward to use, or all three. Michael Vollmer Software Engineering Principles 16 / 21 No silver bullet • Buy versus build • The most radical solution for constructing software is to not construct it at all • Can I use off-the-shelf solutions? • Do we buy rather than build? • Nobody builds their own database • Many tasks done using spreadsheets • Still, no silver bullet! Michael Vollmer Software Engineering Principles 17 / 21 Information hiding Should all information be hidden or exposed? • Brooks (1975) – Regular info updates (microfiche!) • Parnas (1972) – Modules should hide their internals • OSS (1980s) – All source code should be available Michael Vollmer Software Engineering Principles 18 / 21 Conceptual coherence Simplicity proceeds from conceptual integrity. Each part must reflect the same philosophies and the same balancing of desiderata. Conceptual integrity dictates that the design must proceed from one mind, or a very small number of agreeing minds. Michael Vollmer Software Engineering Principles 19 / 21 Summary Software engineering principles 1. Building a programming systems product is 9× harder than building a program. 2. No single development will lead to order-of-magnitude improvements in productivity and reliability 3. Adding manpower to a late software project makes it later Michael Vollmer Software Engineering Principles 20 / 21 Software engineering principles • Three principles of software engineering • Essential and accidental complexity • What can be done about it Michael Vollmer Software Engineering Principles 21 / 21 User Stories and Use Cases Determining the Requirements of a Software System Michael Vollmer School of Computing, University of Kent Table of contents 1. User Stories 2. Detailed Use Cases 3. Examples 4. Summary Michael Vollmer User Stories and Use Cases 1 / 22 User Stories Clients and Providers Typical Client • Experts in their domain • Do not know how computers work • Not sure what is possible or realistic Typical Software Provider • Hopefully knows how to build software systems • Likely little or no domain knowlege (for example, not an expert on regulations for electronic health records) • Rare exceptions sometimes happen! Michael Vollmer User Stories and Use Cases 2 / 22 Learning Unfamiliar Domain • Put knowledge on paper: Specification that programmers can understand • Keep knowledge in brains: Not scalable, but developers can use intuition and contribute ideas Michael Vollmer User Stories and Use Cases 3 / 22 Lightweight requirement capture • Agile methodologies • Formal specifications are too hard • How do we get customers involved? • What is a good way of talking to them? • User stories • Not requirements, but stories • Use paper or Word or whatever works • As a user I want something so that benefit • Fixed story format: user and benefit Michael Vollmer User Stories and Use Cases 4 / 22 User Story Format As a student I want to purchase a parking pass so that I can drive to school. Priority: should Estimate: 4 Need to know: • User: so that we know who needs this • Benefit so that there is a reason to do it Michael Vollmer User Stories and Use Cases 5 / 22 User stories Three critical aspects: 1. Card: artefact representing requirement 2. Conversation: exchange of ideas with customer 3. Confirmation: customer-defined acceptance test Michael Vollmer User Stories and Use Cases 6 / 22 User story example: saving a shopping cart AS AN online customer, I WANT TO save my cart, SO THAT I can complete the purchase later. Q: For how long should the cart be saved? A: For 3 days from the last access. Q: What is an item in the cart becomes unavailable? A: When item becomes unavailable, remove it and warn the user. Q: Can the user modify the cart after saving? A: Yes, the user can modify the cart after saving. Michael Vollmer User Stories and Use Cases 7 / 22 Where do requirements come from? Who is the customer? • Product Owner in SCRUM • Actual system customer in XP Being a customer is hard! • Will Extreme Programming kill your customer? (2001) • Marie DeArment in the Chrysler project Michael Vollmer User Stories and Use Cases 8 / 22 How to write user stories Important details to consider: • What is the definition of “done” for this story? • What steps need to be taken and who is responsible for each of them? • How long will it take to complete? • Can this story be broken up into several stories (for example, a story for each step in a larger process)? User stories are sometimes “scored” based on their assumed complexity or time to completion. An epic is a big chunk of work which consists of multiple stories. A complicated enough story may need to become an epic. Michael Vollmer User Stories and Use Cases 9 / 22 Conversations • After obtaining a user story, you move on to the next step: conversation • User story alone doesn’t contain all the information you need • Need to have a conversation between developers and stakeholders/customers • Conversation must address questions like “what”, “why”, “when”, “who” • Conversation happens during Sprint Planning Meeting • Take notes during conversation, summary Michael Vollmer User Stories and Use Cases 10 / 22 Confirmation • Each story needs some acceptance criteria, “definition of done” • Acceptance criteria should be testable • Written in the form: given/when/then Given (something), when (action taken), then outcome of action taken. Michael Vollmer User Stories and Use Cases 11 / 22 Detailed Use Cases Use Cases vs. User Stories Developers should ultimately use both user stories and use cases. User stories are: • Short (ideally one sentence) • From the user’s perspective • No technical detail Use cases are: • Longer (more detail) • User interaction/flow • System requirements and technical detail Michael Vollmer User Stories and Use Cases 12 / 22 Identify Actors An actor could be: user, database system, server, cloud platform, etc. • What groups will use the system to do their jobs? • What groups are needed to execute the system’s primary functions? • What groups are needed to execute the system’s secondary functions? (maintain/administer system) • What external groups will the system interact with? (hardware/software) Michael Vollmer User Stories and Use Cases 13 / 22 Primary vs. supporting actors • A primary actor initiates an interaction with the system • The system initiates interactions with secondary actors A user of the application selects a category of products. The applications sends an SQL query to a database system. The database responds with the list of products in the category. The application formats and displays the list of products to the user. • The user is the primary actor. They initiated the interaction with the application • The database system is the secondary actor. The application initiates the interaction by sending the query Michael Vollmer User Stories and Use Cases 14 / 22 Identify Scenarios A scenario is a description of how actors interact in a system. For each scenario, there is a: • A use case • An acceptance test case For a scenario we need to know: • What tasks need to be performed? And by/for what actors? • What data/resources do the actors need? • What external events might affect the system? How are the changes communicated to the system (by an actor)? Michael Vollmer User Stories and Use Cases 15 / 22 Use Case Diagrams • Simplified, does not show full detail of use case • Summarises some of the relationships between use cases, actors, and systems • Does not show the order steps are performed in • Typically contains: • Actor, stick figure • Use case, circle • Interaction, line • System, box • May contain structured elements • Extend, arrow with dotted line (labelled “extend”) • Include, arrow with dotted line (labelled “include”) • Generalisation, solid arrow Michael Vollmer User Stories and Use Cases 16 / 22 Examples Example user stories Remember, stories are of the form: AS A (user), I WANT TO (action), SO THAT I (reason). • As a University Lecturer, I want to view my class schedule, so that I am not late to my classes. • As a taxi passenger, I want to add my credit card to my profile, so that I can pay for a ride faster. • As a content creator, I want to block annoying followers, so that they are not able to bother me. • As an editor, I want to review articles before they are published, so that I can ensure they are free from errors. Michael Vollmer User Stories and Use Cases 17 / 22 Example user story conversation As an editor, I want to review articles before they are published, so that I can ensure they are free from errors. Developer: What do you mean by “review” in this context? Product owner: After the author finishes and submits the text, the system places it in a queue and the editor will read it and modify it to correct errors. Developer: How will the editor access the article text? Product owner: The editor will request a list of all queued articles from the system and select an article. Developer: What about when the editor completes their editing? Product Owner: Then the system should remove the article from the queue and publish it on the website. Michael Vollmer User Stories and Use Cases 18 / 22 Example use case Imagine that we are building a software system to run an online store. We can define an example use case (simplified) for viewing orders. • Name: ViewOrders • Actors: initiated by administrator users, communicates with database • Description: Admin users will be able to view a list of all orders placed by customers • Preconditions: User is authenticated and has admin permissions • Flow: 1. The admin user clicks the “View All Orders” button in the user interface 2. Database is queried to request all order records 3. The admin user is shown a list containing all order records • Other requirements: Should respond within 30 seconds Michael Vollmer User Stories and Use Cases 19 / 22 Use Case Diagram (from “Visual Paradigm”) Michael Vollmer User Stories and Use Cases 20 / 22 Use Case Diagram (from “Visual Paradigm”) Michael Vollmer User Stories and Use Cases 21 / 22 Summary Requirements Capturing requirements as stories As a COMP5590 lecturer, I want you to remember this, so that you can use user stories in your projects! A user acts, the system responds Describe how actors interact with the system, and how the system interacts with actors. Michael Vollmer User Stories and Use Cases 22 / 22 Errors and Testing When software goes wrong Michael Vollmer School of Computing, University of Kent Table of contents 1. History of Testing 2. Kinds of Errors 3. Handling Errors 4. Approaches to dealing with errors 5. Kinds of Testing 6. Examples Michael Vollmer Errors and Testing 1 / 27 History of Testing Debugging-oriented Period 1940s to 1956 Program checkout involves debugging or testing Interpretive checking routines and post-mortem dumps Michael Vollmer Errors and Testing 2 / 27 Demonstration-oriented Period 1957 to 1978 Last step in the Waterfall methodology Can we exhaustively cover all cases? Michael Vollmer Errors and Testing 3 / 27 Destruction-oriented Period 1979 to 1980s “Program testing can be used to show the presence of bugs, but never to show their absence.” – Dijkstra, 1970 Professional testers focus on finding bugs through tests Michael Vollmer Errors and Testing 4 / 27 Evaluation-oriented Period 1980s to now Testing as measurable part of development process New theories and tools to support testing Michael Vollmer Errors and Testing 5 / 27 Kinds of Errors How did it happen? • Mistake – following a bad plan This algorithm isn’t actually going to sort a list. • Failure – error in translation from plan to reality The algorithm is correct but I implemented it incorrectly. • Slip – mechanical error when writing code Syntax error or off-by-one error in array access. • Malfunction – hardware failure or error in dependencies Wrong data received in network communication. Michael Vollmer Errors and Testing 6 / 27 When do we find out? • Static error – when developing the software • Immediate runtime error – right when it occurs • Latent runtime error – sometime after it occurs Michael Vollmer Errors and Testing 7 / 27 Runtime errors Input errors Operation cannot be completed because the inputs are wrong. // no character at index 99 "hello world".charAt(99) // null always represents invalid value "hello world".replace("world", null) Output errors There is no way of producing a valid result. // No valid result if file doesn't exist readFile("c:/temp/non-existent.txt") Michael Vollmer Errors and Testing 8 / 27 Handling Errors Engineering approach to errors Errors are inevitable • Programmers are only human • Generally can’t check everything • Time and budget are limited Good practices • Do not assume errors will not happen • Make it easy to find out what went wrong Michael Vollmer Errors and Testing 9 / 27 Indicating errors Ways of indicating errors • Error code – return null or -1 • Exception – use language mechanism to throw exception • Option type – wrap result in an Option or Maybe type Why not error code? • Cannot be confused with valid result • Propagate automatically (language-level exceptions, Maybe monad, etc) • Name explains reason for error Michael Vollmer Errors and Testing 10 / 27 Ways of handling bad inputs: ignore bad input What if we assume the input is valid? public static String niceName(String name) { return name.charAt(0).toUpperCase() + name.substring(1); } When will this method not work? What if name is null? What if name is "" (empty string)? Michael Vollmer Errors and Testing 11 / 27 Ways of handling bad inputs: recover from bad input What if we try to do the best we can with the bad input? public static String niceName(String name) { if (name == null || name.length == 0) { return ""; } else { return name.charAt(0).toUpperCase() + name.substring(1); } } What happens on bad inputs? You get an empty name as the result. Is this what you want? It depends… Michael Vollmer Errors and Testing 12 / 27 Ways of handling bad inputs: fail on bad input What if we give up on bad input? public static String niceName(String name) { if (name == null) { throw new Exception("Null name") } else if (name.length == 0) { throw new Exception("Empty name") } else { return name.charAt(0).toUpperCase() + name.substring(1); } } Terminating execution immediately Most languages today support some kind of exception mechanism. When throwing an exception you can signal what was wrong with the input and make the caller deal with it. Michael Vollmer Errors and Testing 13 / 27 Defensive Programming “Better safe than sorry” • Assume that your code might not be called correctly • Protect your code by throwing on bad inputs • Use extra security measures in low-level code • Check for null values when necessary Michael Vollmer Errors and Testing 14 / 27 Approaches to dealing with errors Mathematics and formal methods Prove programs correct Testing can only prove presence of bugs, not their absence. Beyond testing, we should prove programs are correct. • Most common approach: use types to specify correct behaviour • More precise types to capture properties like non-nullability • Verification tools for critical software Michael Vollmer Errors and Testing 15 / 27 Let it Fail The “kill yourself” strategy When the process does not know how to react, it should shut itself down and let someone else deal with it. • Error handling used in Erlang • Program consists of many small processes • Processes communicate via messages • Supervisor processes handle failures Michael Vollmer Errors and Testing 16 / 27 Test-based engineering approach Test-Driven Development (TDD) Write tests first! A software development practice where you create test cases before writing actual code. • Tests as a light-weight specification • Test that we get correct results for sample inputs • Test expected behaviour for bad inputs • Test early, test often Michael Vollmer Errors and Testing 17 / 27 Kinds of Testing Manual vs. Automated Testing Manual Testing Human testers manually work through different features of the software trying to identify bugs, evaluate usability, etc. Automated Testing Test scripts, frameworks, etc., automate the testing of software, and programmers use these systems to develop a suite of tests that can be run regularly. Michael Vollmer Errors and Testing 18 / 27 Functional vs. Non-functional Testing Non-functional Testing Tests how well software works by evaluating performance, accessibility, user interface design, etc. Functional Testing Testing the “functionality” of software, ensuring it works properly and exhibits the right behaviour. Michael Vollmer Errors and Testing 19 / 27 Unit Tests • The kind of testing we will use the most in this module • Individual units of source code (modules, functions, classes, methods, etc) are tested individually • Popular JUnit library for testing Java programs • Unit test provides a kind of contract that a piece of code must satisfy • Isolate each part of the program, try to determine if individual parts are correct • Often involves creating fake/mock objects Michael Vollmer Errors and Testing 20 / 27 Integration Tests • Individual software units are combined and tested as a group • Takes modules that have been unit tested, groups them into larger aggregates, and applies tests defined in an integration test plan • Detect inconsistencies or conflicts between units when integrated/aggregated into a group • Often involves creating and populating objects, connecting to resources, etc. Michael Vollmer Errors and Testing 21 / 27 Systems Tests and Acceptance Tests • Systems Tests • Testing conducted on a complete integrated system, end-to-end • Include external dependencies • Example: testing the user’s experience with an application by launching it and clicking on stuff • Often done by testing engineers • Acceptance Tests • Tests conducted to determine if the system satisfies some acceptance criteria • The user/customer determines whether they accept the system • Example: beta testing Michael Vollmer Errors and Testing 22 / 27 Property-based Testing • Related to unit testing and integration testing. • Instead of specific examples, generate lots of random input parameter combinations that satisfy predefined properties • Potentially more thorough than ordinary example-based unit tests • Pioneered by QuickCheck in Haskell, available in other languages like Java and Python Michael Vollmer Errors and Testing 23 / 27 Examples Example Unit Tests Procedure: Adder We want to build an adder, which is a component that can add two integers together. • Given 1 and 1, the adder should return 2. • Given 1 and 2, the adder should return 3. • Given 2 and 2, the adder should return 4. • Given 1 and 0, the adder should return 1. • Given 1 and -2, the adder should return -1. Michael Vollmer Errors and Testing 24 / 27 JUnit in Java import static org.junit.Assert.assertEquals; import org.junit.Test; public class TestAdder { // ... } Michael Vollmer Errors and Testing 25 / 27 JUnit in Java public class TestAdder { // ... // Given 1 and 1, the adder should return 2. @Test public void testSumPositiveNumbersOneAndOne() { Adder adder = new Adder(); assertEquals(2, adder.add(1, 1)); } // Given 1 and 0, the adder should return 1. @Test public void testSumPositiveNumbersOneAndZero() { Adder adder = new Adder(); assertEquals(1, adder.add(1, 0)); } // ... } Michael Vollmer Errors and Testing 26 / 27 JUnit-Quickcheck in Java public class TestAdder { // ... // Any number plus 0 equals that number. @Property(trials = 10) public void testSumAddZero(int number) { Adder adder = new Adder(); assertEquals(number, adder.add(number, 0)); } // ... } Michael Vollmer Errors and Testing 27 / 27 Theory of Testing Writing Effective Tests Michael Vollmer School of Computing, University of Kent Table of contents 1. Notes on A1 and User Stories 2. Review of Testing 3. Test Engineering 4. Examples 5. Summary Michael Vollmer Theory of Testing 1 / 15 Notes on A1 and User Stories What is a user story? Template As a [type of user] I want to [do some action] so that I [get some benefit] • Stories should be specific • The parts of the story should be closely related to each other • Should be relevant to the project! Michael Vollmer Theory of Testing 2 / 15 User story example: saving a shopping cart AS AN online customer, I WANT TO save my cart, SO THAT I can complete the purchase later. Q: For how long should the cart be saved? A: For 3 days from the last access. Q: What is an item in the cart becomes unavailable? A: When item becomes unavailable, remove it and warn the user. Q: Can the user modify the cart after saving? A: Yes, the user can modify the cart after saving. Michael Vollmer Theory of Testing 3 / 15 Review of Testing Role of Testing • Verification • Software conforms to specification • Are we building it right? • Defect testing to check for bugs • Validation • Software does what user wants • Are we building the right thing? • Validation testing to check requirements Michael Vollmer Theory of Testing 4 / 15 Test-driven Development • TDD in practice • Write tests to capture requirements • Implement code satisfying tests • Test automatically and frequently • Development process • Structured approach to development • Mostly focused on engineering side • Red-green-refactor method Michael Vollmer Theory of Testing 5 / 15 Red-green-refactor Method • Red – think about what you want to develop • Green – think about how to make your tests pass • Refactor – think about how to improve your existing implementation Process When you write a test it will initially fail (red). After, you will write the code to make the test pass (green). Eventually, you will update/modify the code while making sure the tests still pass (refactor). Michael Vollmer Theory of Testing 6 / 15 Test Engineering Questions About Testing • How do we know our tests are good? • How can we design good tests? • What tools and theories can help us? Michael Vollmer Theory of Testing 7 / 15 White and Black Box Testing • Black-box (functional) • Without looking at the code • Tests based only on specification • Focuses on representative inputs • White-box (structural) • Access to the code structure • Test based on program logic • Cover as much code as possible Michael Vollmer Theory of Testing 8 / 15 Equivalence Class Partitioning • Equivalence classes • Subsets of inputs that are equivalent • Program works the same way • Code follows the same path • Designing tests • Identify ranges of values for which the behavior is the same • Pick one sample input from each class Michael Vollmer Theory of Testing 9 / 15 Boundary Value Analysis • Heuristic for finding good partition representatives • Idea: use “bounary” values Michael Vollmer Theory of Testing 10 / 15 Code Coverage • White-box testing metric • How much code is run by tests? • The more the better! • Still no absolute guarantee • Coverage metrics • Statement coverage – which individual statements are run • Decision coverage – which branches of conditionals are run Michael Vollmer Theory of Testing 11 / 15 Examples Example: months of the year Let’s say we have a function where input values are months of the year, expressed as integers. What are some valid/invalid inputs to the function? Can we come up with the equivalence classes? Michael Vollmer Theory of Testing 12 / 15 Example: months of the year Let’s say we have a function where input values are months of the year, expressed as integers. What are some valid/invalid inputs to the function? Can we come up with the equivalence classes? Michael Vollmer … -2 -1 0 1 2 3 … 11 12 13 14 … invalid valid invalid Theory of Testing 12 / 15 Example: months of the year Let’s say we have a function where input values are months of the year, expressed as integers. What are some valid/invalid inputs to the function? Can we come up with the equivalence classes? … -2 -1 0 1 2 3 … 11 12 13 14 … invalid valid invalid What are the boundaries? Michael Vollmer Theory of Testing 12 / 15 Example: months of the year Let’s say we have a function where input values are months of the year, expressed as integers. What are some valid/invalid inputs to the function? Can we come up with the equivalence classes? … -2 -1 0 1 2 3 … 11 12 13 14 … invalid valid invalid What are the boundaries? Good test values are: 0, 1, 12, 13. Michael Vollmer Theory of Testing 12 / 15 Example: city names Let’s say we have a function where input values are names of cities, expressed as strings. What are some valid/invalid inputs to the function? Can we come up with the equivalence classes? Michael Vollmer Theory of Testing 13 / 15 Example: city names Let’s say we have a function where input values are names of cities, expressed as strings. What are some valid/invalid inputs to the function? Can we come up with the equivalence classes? Michael Vollmer "" null "New York" invalid invalid valid Theory of Testing 13 / 15 Example: city names Let’s say we have a function where input values are names of cities, expressed as strings. What are some valid/invalid inputs to the function? Can we come up with the equivalence classes? "" null "New York" invalid invalid valid What about non-empty, non-null strings that are not really names of cities? Michael Vollmer Theory of Testing 13 / 15 Example: tests to cover code Let’s look at the following example code: public int foo(int value) throws Exception { if (value < 0) { throw new Exception(); } if (value > 5) { return value + 5; } else { return value; } } Michael Vollmer Theory of Testing 14 / 15 Example: tests to cover code Let’s look at the following example code: public int foo(int value) throws Exception { if (value < 0) { throw new Exception(); } if (value > 5) { return value + 5; } else { return value; } } … -2 -1 0 12345 67… invalid valid valid Boundary values: 0, 1, 5, 6. Michael Vollmer Theory of Testing 14 / 15 Summary Testing Theory • What can you do with testing? Validate, verify, find bugs. • Agile approach: tests are specification, write tests first • Break possible inputs into classes • Consider testing values on boundaries of classes Michael Vollmer Theory of Testing 15 / 15 Unit Tests Practical Testing with JUnit Michael Vollmer School of Computing, University of Kent Table of contents 1. Review of Unit Tests 2. What is JUnit? 3. JUnit Annotations and Assertions 4. JUnit Example 5. Demo Michael Vollmer Unit Tests 1 / 15 Review of Unit Tests Unit Tests • The kind of testing we will use the most in this module • Individual units of source code (modules, functions, classes, methods, etc) are tested individually • Popular JUnit library for testing Java programs • Unit test provides a kind of contract that a piece of code must satisfy • Isolate each part of the program, try to determine if individual parts are correct • Often involves creating fake/mock objects Michael Vollmer Unit Tests 2 / 15 Example Unit Tests Procedure: Adder We want to build an adder, which is a component that can add two integers together. • Given 1 and 1, the adder should return 2. • Given 1 and 2, the adder should return 3. • Given 2 and 2, the adder should return 4. • Given 1 and 0, the adder should return 1. • Given 1 and -2, the adder should return -1. Michael Vollmer Unit Tests 3 / 15 What is JUnit? Origins of JUnit JUnit was born on a flight from Zurich to the 1997 OOPSLA in Atlanta. Kent was flying with Erich Gamma, and what else were two geeks to do on a long flight but program? The first version of JUnit was built there, pair programmed, and done test first (a pleasing form of meta-circular geekery). – Martin Fowler Michael Vollmer Unit Tests 4 / 15 What is JUnit? • Automated unit testing framework for Java • Very important in the history of test-driven development and automated testing • Most popular framework for testing in Java Michael Vollmer Unit Tests 5 / 15 JUnit Annotations and Assertions Basic Test Annotation The main annotation is @Test @Test public void test() { // unit test body } This is a single unit test. Michael Vollmer Unit Tests 6 / 15 Basic Assertions Various assertions, like assertTrue, assertEquals, etc. @Test public void test() { boolean b = foo(); // some computation assertTrue(b); // assert that b should be true } If the assertion succeeds, the test passes. Michael Vollmer Unit Tests 7 / 15 Before/After Annotations Before and after every test: @Before and @After. Static methods to execute once before/after all tests in class: @BeforeClass and @AfterClas. @Before public void before() { // set up for tests } Michael Vollmer Unit Tests 8 / 15 JUnit Example Example: MyAdder Specification Adder class keeps track of an internal accumulator. Consumer adds numbers to the accumulator, requests the accumulator value, and resets the accumulator. // Don't implement yet, just sketch out public interface public class MyAdder { public void add(int n) {} public void reset() {} public int getAcc() { return 0; } } Michael Vollmer Unit Tests 9 / 15 Example: MyAdderTest public class MyAdderTest { MyAdder adder; @Before public void initAdder() { adder = new MyAdder(); } @Test public void testCreateAdder() { assertEquals(adder.getAcc(), 0); } @Test public void testAddOne() { adder.add(1); assertEquals(adder.getAcc(), 1); } @Test public void testAddOneTwice() { adder.add(1); adder.add(1); assertEquals(adder.getAcc(), 2); } } Michael Vollmer Unit Tests 10 / 15 MyAdderTest results (red) ------------------------------------------------------T E S T S ------------------------------------------------------Running org.example.MyAdderTest Failed tests: testAddOneTwice(org.example.MyAdderTest): expected:<0> but was:<2> testAddOne(org.example.MyAdderTest): expected:<0> but was:<1> Tests run: 3, Failures: 2, Errors: 0, Skipped: 0 Michael Vollmer Unit Tests 11 / 15 MyAdderTest results (red) (in an IDE) Michael Vollmer Unit Tests 12 / 15 Implement MyAdder public class MyAdder { public MyAdder() { acc = 0; } public void add(int n) { acc += n; } public void reset() {} // no tests, so not implemented! public int getAcc() { return acc; } private int acc; } Consider: what if we implemented add() like this instead? public void add(int n) { acc += 1; } Michael Vollmer Unit Tests 13 / 15 MyAdder and MyAdderTest fixed (green) ------------------------------------------------------T E S T S ------------------------------------------------------Running org.example.MyAdderTest Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.06 sec Results : Tests run: 3, Failures: 0, Errors: 0, Skipped: 0 Michael Vollmer Unit Tests 14 / 15 MyAdderTest results (green) (in an IDE) Michael Vollmer Unit Tests 15 / 15 Demo