Software Engineering Lecture 1 Introduction to Software Engineering Code of Conduct Software Engineering is a collaborative activity. You are encouraged to work together, but ... Some tasks may require individual work. Always give credit to your sources and collaborators. Good professional practice: To make use of the expertise of others and to build on previous work, with proper attribution. Unethical and academic plagiarism: To use the efforts of others without attribution. Projects Project teams, about 3 to 5 peoples. Select your own project, any branch of software engineering Real project for real client who intends to use the software in production. Feasibility study and plan: during semester Presentations: • requirements • design • final Project Selection Some suggested projects Recitation section to suggest projects Contact potential clients: Gain idea of their expectations Estimate scope and complexity of the project Discuss business decisions Assemble project team Advertise on the web site Previous Experience Your background Biggest program that you have written? Biggest program that you have worked on? Biggest project team that you have been part of? Longest project that you have worked on? Most people who have used your work? Longest that your project has been in production? My background Course Themes 1. Leadership of large software projects Software as a product Clients and their needs Quality Requirements and specification Usability Evolution Project management Personnel management Economic, legal, and social factors Course Themes 2. Large and very large systems Software design Software architecture Object-oriented design Dependable systems Reliability Verification Legacy systems Characteristics of Software Products General characteristics Usability Maintainability Dependability Efficiency Good software products require good programming, but ... Programming quality is the means to the end, not the end itself. Software as a Product Software is expensive!! Every software project has a trade-off between: Functionality Resources (cost) Timeliness Example: Accounting Management System Client (a.k.a Customer) The client provides resources and expects some product in return. Client satisfaction is the primary measurement of success. Question: Who is the client for Microsoft Excel? Variety of Software Products Examples? -Operation System -Database Management System -Embedded System -Games -Application Software -… Categories of Product Categories of client and software product: Generic (e.g., Microsoft Excel) Bespoke (customized) (e.g., IRS internal system) Many systems are customized versions of generic packages (e.g., Cornell's payroll system) Variety of Software Products Software products are very varied --> Client requirements are very different --> There is no standard process for software engineering --> There is no best language, operating system, platform, database system, development environment, etc. A skilled software developer knows about a wide variety of approaches, methods, tools. The craft of software engineering is to select appropriate methods for each project and apply them effectively. Professional Responsibility Organizations put trust in software developers: Competence: Software that does not work effectively can destroy an organization. Confidentiality: Software developers and systems administrators may have access to highly confidential information (e.g., trade secrets, personal data). Legal environment: Software exists in a complex legal environment (e.g., intellectual property, obscenity). Acceptable use and misuse: Computer abuse can paralyze an organization (e.g., the Internet worm). Software Engineering Lecture 2 The Software Process Books Frederick P. Brooks, Jr. The Mythical Man Month. Addison-Wesley, 1972. Ian Sommerville, Software Engineering, 6th edition. Addison-Wesley, 2000. Grady Booch, James Rumbach, Ivar Jacobson, The Unified Modeling Language. Addison-Wesley 1999. Software Process Fundamental Assumption: Good processes lead to good software Good processes reduce risk Risk Management What can go wrong in a software project? How can the risk be reduced? The Software Process (Simplified) Feasibility and Planning Requirements Design Implementation Operation and Maintenance The Waterfall Model Requirements Definition System and Software design Programming and Unit Testing Integration and System Testing Operation and Maintenance Requirements Analysis and Definition The system's services, constraints and goals are established by consultation with system users. They are then defined in a manner that is understandable by both users and development staff. This phase can be divided into: Feasibility study (often carried out separately) Requirements analysis Requirements definition Requirements specification System and Software Design System design: Partition the requirements to hardware or software systems. Establishes an overall system architecture Software design: Represent the software system functions in a form that can be transformed into one or more executable programs Unified Modeling Language (UML) Programming and Unit Testing The software design is realized as a set of programs or program units. (Written specifically, acquired from elsewhere, or modified.) Individual components are tested against specifications. Integration and System Testing The individual program units are: integrated and tested as a complete system tested against the requirements as specified delivered to the client Operation and Maintenance Operation: The system is put into practical use. Maintenance: Errors and problems are identified and fixed. Evolution: The system evolves over time as requirements change, to add new functions or adapt the technical environment. Phase out: The system is withdrawn from service. Discussion of the Waterfall Model Advantages: Process visibility Dependence on individuals Quality control Cost control Disadvantages: Each stage in the process reveals new understanding of the previous stages, that requires the earlier stages to be revised. Feedback in the Waterfall Model Requirements Definition System and Software design Programming and Unit Testing Integration and System Testing Operation and Maintenance Iterative Refinement (Evolutionary Development) Concept: Initial implementation for user comment, followed by refinement until system is complete. Vaporware: user interface mock-up Throw-away software components Dummy modules Rapid prototyping Successive refinement Iterative Refinement Evaluation Requirements Implementation (prototype) Design Iterative Refinement Concurrent Activities Requirements Outline Description Design Implementation Initial Version Intermediate Versions Final Version Iterative Refinement & Software Process Concurrent Activities Outline Description Requirements Design Implementation Final Version Iterative Refinement When is iterative refinement appropriate? Iterative Refinement + Waterfall Model: Graphics for Basic Outline Description: Add vector graphics to Dartmouth Basic. Phase 1: Extend current language with a preprocessor and run-time support package. (1976/77) Phase 2: Write new compiler and run-time system incorporating graphics elements. (1978/80) Iterative Refinement + Waterfall Model: Graphics for Basic Design Issues: Pictorial subprograms: coordinate systems, window/viewport User specification of perspective Design Strategy: (Iterative Refinement) Write a series of prototypes with various proposed semantics Evaluate with a set of programming tasks Iterative Refinement + Waterfall Model: Graphics for Basic Phase 1: Implementation (Waterfall) When the final specification was agreed, the entire preprocessor and run-time support were recoded. The system was almost entirely bug-free. Phase 2: New compiler (Waterfall) Phase 1 was used as the requirements definition for the final version. Observations about Software Processes Completed projects should look like the Waterfall Model but ... the development process is always partly evolutionary. Risk is lowered by: Prototyping key components Dividing into phases Following a visible software process Making use of reusable components Software Engineering Lecture 3 (a) Feasibility Study (b) Requirements Definition Feasibility Study Before beginning a project, a short, low-cost study to identify • Client • Scope • Potential benefits • Resources needed: staff, time, equipment, etc. • Potential obstacles Where are the risks? How can they be minimized? Feasibility Study A feasibility study leads to a decision: go ahead do not go ahead think again In production projects, the feasibility study often leads to a budget request. In research, a feasibility study is often in the form of a proposal. CS 501: Client In CS 501, you have two clients: • The client for the project • The professor for the course Can you satisfy them both? Scope What are the boundaries of the project? CS 501 Examples: • Static web pages with open access on the Web [Web Profiler] • Used by the general public [Digital Collections] • Varying data formats [Legal Information] • Thousands of sensors [Data mining] • Support for Windows, Mac, Unix [SALSA] Potential Benefits Why are you doing this project? Examples • • • • • Create a marketable product Improve the efficiency of an organization Control a system that is too complex to control manually New or improved service Safety or security • Get a good grade on CS 501 Resources Examples: CS 501 Staff: 5 to 7 students, with some help. How many hours per week? What skills do people have? Time: Must be completed by end of semester, including operational system, documentation, presentation Equipment and software: What special needs are there? Client: Will the client be sufficiently available and helpful? Obstacles CS 501 projects Start-up time. Creating a team, scheduling meetings, acquiring software, learning new systems, ... Business considerations. Licenses, trade-secrets, ... Too ambitious. Nothing to show at the end of the semester. Changing circumstances. Client leaves the university, ... What else? How to Minimize Risk? CS 501 Projects • Several target levels of functionality: required, desirable, optional • Visible software process: intermediate deliverables • Good communication within team and with Teaching Assistant Good processes lead to good software Good processes reduce risk Feasibility Report A written document • For a general audience: client, financial management, technical management, etc. • Short enough that everybody reads it • Long enough that no important topics are skipped In CS 501, I am looking for a well written, well presented document. Requirements Definition and Analysis Requirements Definition System and Software design Programming and Unit Testing Integration and System Testing Operation and Maintenance Example: Library of Congress (A Partial Failure) Outline Description The Library of Congress requires a repository system to store and make accessible very large amounts of highly varied material over long periods of time. Chronology 1993-94 CNRI carries out research on architectures for digital libraries 1995-97 CNRI implements prototype repository for Library of Congress 1998 CNRI and Library of Congress carry out requirements definition The Repository Repository Users Identification System Search System Storage and Representation of Complex Objects Data Several representations: thumbnail image reference image archival image Metadata Each representation may have its own metadata Repository: Research Achievements 1. CORBA implementation of repository access protocol. 2. Integration of persistent naming through handle system. 3. Use of structural metadata to describe complex objects, elementary typology. 4. Access management framework and implementation. 5. Applet-based middleware for user interfaces. 6. Information visualization program to view the structure of large collections. Good Discoveries During Prototype • Structuring complex information in digital libraries • Data driven digital library interfaces • Comparison of object-oriented, relational, and file based storage systems • Naming and identification of library objects • Boundaries of required repository system Bad Discoveries During Prototype • Resistance to change within Library of Congress • Technical weakness of Library of Congress • Gaps in CNRI architecture Mistakes • Confusion of objectives (research and implementation) • Failure to involve all stakeholders • Over-ambitious (no proper feasibility study) The Requirements Process Feasibility Study Requirements Analysis Requirements Definition Feasibility Report System Models Requirements Specification Definition of Requirements Requirements Document Specification of Requirements Requirements Definition High-level abstract description of requirements: • Specifies external system behavior • Comprehensible by customer, management and users Should reflect accurately what the customer wants: • Services that the system will provide • Constraints under which it will operate Library of Congress Requirements Study Team (all experienced): Librarian, Software Engineer (CNRI), Computing Project Leader (Library of Congress), + 2 others Advisors: Mailing list of about 20 knowledgeable stakeholders. Timetable: Preliminary report (2 months). Final report (1 month). Functional Requirements Example: Library of Congress repository • Support for complex digital objects • Access management • Identification • Information hiding • Open protocols and formats • Integration with other systems (scope) DRAFT OVERVIEW OF ITS SUPPORT FOR NDLP PRODUCTION AND DELIVERY OF AMERICAN MEMORY NDLP collections already released Coolidge collection (for repository test) NDLP collections in conversion Future NDLP collections Other applications and materials NDLP Workflow Tracking Support Current Storage Structure (in Unix files, by aggregate) Object Administration System ILS Repository Index Generation (including pre-processing) American Memory User Interface (retrieval, navigation, & display) AM user interface plus access management for objects/collections Other User Interfaces (e.g. RLG, OCLC, DLF partners) ILS OPAC Interface Supporting infrastructure Handle assignment & registration Handle-server NOW Handle resolution FUTURE Non-functional Requirements Environment: • • Estimates of sizes, numbers of users, etc. Reliability and performance measures and targets Preferred: Example: Library of Congress repository • Hardware and software systems (e.g., IBM/Unix) • Database systems (e.g., Oracle) • Programming languages (e.g., C and C++) Evolution of Requirements • If the requirements definition is wrong, the system will be a failure. • With complex systems, understanding of requirements always continues to improve. Therefore... • The requirements definition must evolve. • Its documentation must be kept current (but clearly identify versions). Software Engineering Lecture 4 Management I: Project Management The Aim of Project Management To complete a project: • On time • On budget • With required functionality • To the satisfaction of the client • Without exhausting the team The Project Manager • • • • Create and maintain the schedule Should track progress against schedule Keep some slack in the schedule Be continually making adjustments: Start activities before previous activity complete Sub-contract activities Renegotiate deliverables • Keep senior management informed Project Planning Methods The Critical Path Method, Gantt charts, Activity bar charts, etc. are roughly equivalent. These methods are best when: • Model is updated regularly (e.g., monthly) • The structure of the project is well understood • The time estimates are reliable • Activities do not share resources [Critical Path Method is excellent for large construction projects.] Example: An Open University Course Deliverables: 16 8 8 4 1 4 Written texts (bound in pairs) Television programs Radio programs Computer programs Home experimental kit (scientific calculator) Assignments and sample solutions Flexibility Schedule: Dates for broadcasting TV and radio programs are fixed. Printing and mailings can be accelerated if overtime is used. Functionality: The course team can decide what goes into the components of the course. Resources: The size of the course team can be increased slightly. Scheduling: Critical Path Method An activity A dummy activity An event A milestone Critical Path Method other activities START Revise Unit 3 Edit Unit 3 Print Unit 3 Mail Unit 3 END Critical Path Method other activities Revise Unit 3 Edit Unit 3 START other activities Revise Unit 4 Edit Unit 4 Typeset Unit 3 Print Units 3/4 Typeset Unit 4 Mail Units 3/4 Critical Path Method Script TV 2 Edit Unit 3 Make TV 2 Mail Delivery START Edit Unit 4 Prototype Computer 1 Document Computer 1 Program Computer 1 Time Estimates for Activities (Weeks) 4 6 1 3 1 2 3 1 12 12 1 3 3 2 4 4 2 8 Earliest Start Dates 1 1 12 26 4 3 6 15 1 2 17 3 12 0 22 1 23 1 3 12 12 3 2 17 19 2 4 4 4 17 8 25 Latest Start Dates 11 1 12 26 4 3 6 15 2 1 17 3 12 0 23 1 24 1 3 12 14 3 2 17 20 2 4 13 4 17 8 25 Critical Path 1/11 12/12 26/26 15/15 17/17 22/23 0/0 12/14 4/13 17/17 19/20 17/17 23/24 25/25 Slack 1/11 26/26 10 10 12/12 0 0/0 3 15/15 0 17/17 0 0 1 22/23 0 2 12/14 2 17/17 9 4/13 19/20 1 1 0 9 23/24 25/25 1 5 0 17/17 Key Personnel In computing, not all people are equal: • The best are at least 5 times more productive • Some tasks are too difficult for everybody Adding more people adds communications complexity • Some activities need a single mind • Sometimes, the elapsed time for an activity can not be shortened. What happens to the project if a key person is sick or quits? Key Personnel: Schedule for Editor Earliest Start Date Activity Weeks 15-16 Weeks 17-18 Weeks 19-20 Weeks 21-22 Edit Unit 3 Edit Unit 4 Edit Unit 5 Edit Unit 6 Week 15 Week 17 Week 19 Week 21 Review draft of Unit 7 Review draft of Unit 8 Check proofs of Unit 3 Check proofs of Unit 4 Weeks 18-19 Week 22 Vacation Out sick Start-up Time On a big project, the start-up time is typically three to six months: • Personnel have to complete previous projects (fatigue) or recruited. • Hardware and software has to be acquired and installed. • Staff have to learn new domain areas and software (slow while learning) • Clients may not be ready. Experience with Critical Path Method Administrative computing department at Dartmouth used the Critical Path Method for implementation phase of major projects. Experience: Elapsed time to complete projects was consistently 25% to 40% longer than predicted by model. Analysis: • Some tasks not anticipated (incomplete understanding) • Some tasks had to be redone (change of requirements, technical changes) • Key personnel on many activities (schedule conflicts) • System ZZZ (non-billable hours) CS 501: Software Engineering Lecture 5 (a) Documentation (b) Requirements Analysis Assignments September 13 October 4 October 16 November 8 Nov 29 - Dec 1 Exam week Feasibility and plan Group RequirementsGroup/individual Midterm exam Individual Design Group/individual Project presentations Group Final examination Individual Details are subject to change. Assignment 1 Wednesday, September 13: Project plan due -- report. Title of project Client/customer Team members Outline description Current status (e.g., previous work) Plan (e.g., major stages, assignment to tasks, technical environment, schedule, etc.) Any other relevant information Documentation • Reasons for documentation: visibility (e.g., project plan, interim report) user support (e.g., user manual) team communication (e.g., interface specifications) maintenance and evolution (e.g., requirements) • Characteristics of documentation: accurate and kept current appropriate for audience maintained online (usually) simple but professional in style and appearance Documentation is expensive --> Quality not volume Form of Documentation External • Printed • Web site Internal • Program documentation • Program context (e.g., copyright notices) Requirements Definition and Analysis Requirements Definition System and Software design Programming and Unit Testing Integration and System Testing Operation and Maintenance The Requirements Process Feasibility Study Requirements Analysis Requirements Definition Feasibility Report System Models Requirements Specification Definition of Requirements Requirements Document Specification of Requirements Requirements Analysis 1. Understand the requirements in depth: • Domain understanding Examples: science research, application… • Stakeholders Example: companies, ministries, Danang City… Viewpoint Analysis Example: University Admissions System • Applicants • University administration Admissions office Financial aid office Special offices (e.g., athletics, development) • Computing staff Operations Software development and maintenance • Academic departments Interviews with Clients Clients may have only a vague concept of requirements. • Prepare before you meet with them • Keep full notes • If you don't understand, delve further • Small group meetings are often most effective Clients often confuse the current system with the underlying requirement. Requirements Analysis 2. Organize the requirements: • Classification into coherent clusters (e.g., legal requirements) • Recognize and resolve conflicts (e.g., functionality v. cost v. timeliness) Example: Dartmouth general ledger system Requirements Analysis 3. Model the requirements: • Informal Prose • Systematic Procedural models Data-centric models Object models • Formal models Procedural Models: Flowchart Operation Decision Manual operation Report Flowchart: University Admissions Form received New? T Database record Notify student F Update database Complete? T F Notify student Evaluate Procedural Models: Pseudo-code Example: Check project project plan check_plan (report) if report (date_time) > due_date_time then error (too_late) if report (client) = none then error (no_client) if report (team) < min_team or > max_team then error (bad_team) if error() = none then comments = read_report (report) return (comments (text), comments (grade)) else return error() Data-Flow Models External entities Processing steps Data stores or sources Data flows Example: University Admissions Rejection Application Completed form Receive application Evaluate application Applicant Offer Example: University Admissions Assemble Application Stage Acknowledgment Application form Receive Applicant Completed application Acknowledgment AND Initiate evaluation Evaluation request AND Supporting information Pending database Applicant database Example: University Admissions Process Completed Application Stage Rejection Evaluation request Acceptance Evaluation Special request Applicant database Financial aid Offer Requirements Analysis v. System Design Dilemma. • Requirements analysis should make minimal assumptions about the system design. • But the requirements definition must be consistent with computing technology and the resources available. In practice, analysis and design are interwoven. However, do not to allow the analysis tools to prejudge the system design. CS 501: Software Engineering Lecture 6 (a) Requirements Analysis (continued) (b) Requirements Specification The Requirements Process Feasibility Study Requirements Analysis Requirements Definition Feasibility Report System Models Requirements Specification Definition of Requirements Requirements Document Specification of Requirements Requirements Analysis Methods for data modeling and design • Data flow diagrams • Entity-relation diagrams • Data dictionaries • Object models Many of these methods blur the distinction between analysis and design. Entity-Relation Model A Design Methodology for Relational Databases • A database of entities and relations • Tools for displaying and manipulating entityrelation diagrams • Tools for manipulating the database (e.g., as input to database design) Warning: There is much confusion about definitions and notation Entity-Relation Diagram An entity A relation between entities An entity or relation attribute An inheritance relation Example: CS 501 Project Major Client 0:n 1 Student Project Person 0:n 1 CS501 Student 5 to 7 Member of 0:n Tech contact MARC Format for Monographs (Books) 001 245 260 650 650 700 89-16879 r93 Campus strategies for libraries and electronic information {Bedford, Mass.} : Digital Press, c1990. Academic libraries--United States--Automation. Libraries and electronic publishing--United States. Arms, Caroline R. (Caroline Ruth) Entity-Relation Diagram for MARC Book 0:n Author of 0:n 1 0:n Editor of Describes 0:n Catalog record Short title 1:n Is about Control numb Creator 0:n 0:n Subject heading Data Dictionaries A data dictionary is a list of names used by the system • Brief definition (e.g., what is "date") • What is it (e.g., number, relation) • Where is it used (e.g., source, used by, etc.) • May be combined with a glossary As the system is implemented, the data dictionary in the requirements is input to the system data dictionary, which is a formal part of the system specification. A Note on Object Models This course teaches object models as a tool for design. Some people recommend object models for requirements analysis, but it is difficult to use them without constraining the system design. Non-Functional Requirements Product requirements performance, reliability, portability, etc... Organizational requirements delivery, training, standards, etc... External requirements legal, interoperability, etc... Examples of Non-Functional Requirements Privacy (Mercury digital library) Functional requirement: Usage data for management of system Non-functional requirement: Usage data must not identify individuals Minimizing records (NeXT) Functional requirement: Retain all required records Non-functional requirement: Discard all other records Unspoken Requirements Example: Resistance to change at XXX Requirements Specification What is the purpose of the Requirements Specification? Requirements Specification: Purpose 1. It describes the requirements to the stakeholders • Expressed in the terms that the stakeholders understand • Comprehensible from many viewpoints • Reviewed by stakeholders so that they understand implications • Must be clear about assumptions (things left out) Requirements Specification: Purpose 2. It describes the requirements to the implementers • As precise and specific as possible • Expressed in terms that they understand • Comprehensible to new team members Requirements Specification: Purpose 3. It records the requirements for the future • An essential part of system evolution 4. If may be a contractual document • See you in court! Requirements Specification: Approaches • Natural language • Structured natural language • Design description language • Requirements specification language • Graphical notation • Formal specification See Sommerville, Chapter 7. CS 501: Software Engineering Lecture 7 Management II Business and Legal Aspects of Software Engineering Legal Environment Software is developed in a complex legal and economic framework. Changes in laws follow changes in technical world. Jurisdictions: • Vietnamese laws • International treaties • Federal and state statues • Precedents • Supreme Court • Cost of establishing precedent Legal Topics • International • Intellectual property (copyright, patent, contract) • Tort (e.g., liability of Internet service provider) • Privacy • Free speech and its limitations (government secrets, obscenity, blasphemy, hate) Legal Information Institute: http://www.law.cornell.edu/ Copyright A copyright gives the owner the exclusive right to: • reproduce • distribute • perform • display • license Gradually extended to cover text, music, photographs, designs, software, ... Copyright Copyright at creation • Works for hire • Contracts and licenses • First sale • Fair use • Infringement (contamination) International differences • • Moral rights Copyright registration Software Patents • Should be: non-obvious, novel, useful • 17 years from award (20 years from application) • Poor quality of examining can lead to broad patents for routine computing concepts • International differences Copyright applies to the expression of ideas, patents to the ideas themselves. Contracts and Licences Contracts allow intellectual property to be sold or licensed • • • • • • • Promise in exchange for adequate consideration Written document with signature Permanent or temporary, whole or part Exclusive or non-exclusive Termination, problems and difficulties Terms and conditions as agreed Enforceable by courts Derivative Works When software is derived from other software: • New code is owned by new developer • Conditions that apply to old code apply to derived work If you write S, which is derived from A, B, C and D, you can not distribute or licenses S unless you have right to distribute each of A, B, C and D. To create a software product, you must have documented rights to use every component. Privacy Invasions of privacy: • • • • intrusion appropriation of name or likeness unreasonable publicity false light Be very careful about collecting personal data without the knowledge of the individual Software Business Questions • You are employed for company X writing software. When you leave, who owns your work? What use can you make of the work? • You work free-lance for company X. When you finish, who owns your work? What use can you make of the work? • Read the contract! Your Next Job ... • Employment contract may restrict your next job (not working for competitors, etc.) • Trade-secret information (non-disclosure agreement) Ask when you are interviewed! Trade Secrets and Non-Disclosure Agreements Trade Secret "... information, including a formula, pattern, compilation, program, device, method, technique, or process that derives independent economic value from not being generally known and not being readily ascertainable and is subject to reasonable efforts to maintain secrecy." Uniform Trade Secrets Act Non-Disclosure Agreement Legal agreement not to disclose trade secrets. Some Business Models • Software developed in-house • Package licensed to customer, binary only (Microsoft model) • Package licensed to customer, source code for customer's modifications • Bespoke software for customer (may be owned by supplier or customer) • Software bundled with hardware product (PalmPilot) Free-Lance Software Development You and a few friends create a company to develop software. How much should you charge per hour? You plan to work 40 hours a week for 50 weeks of the year and want to earn $50,000. Hourly rate = $50,000 / (40 x 50) = $25 But ... Free-Lance Software Development Salary Taxes and benefits Rent, equipment, etc. Fees, services, etc. Travel and misc. TOTAL EXPENSE Hours worked less administration less marketing BILLABLE HOURS $50,000 $15,000 $10,000 $15,000 $10,000 $100,000 2,000 400 350 1,250 Hourly rate = $100,000 /1,250 = $80 Fixed and Variable Cost: Packaged Software Example: • The initial development cost of a software product is $10 million. • The cost of packaging and distribution of each copy is $5. • Technical support costs average $15 per copy. • The package sells for $200 per copy. Fixed cost = $10 million Variable cost = $20 Fixed and Variable Costs: Profit or Loss $15M $10M $5M 2,500 5,000 7,500 Unit sales Community Development • Shareware • Open source (e.g., Linux, Apache, Perl, etc.) -> Shared development -> Market penetration Example: TCP/IP for Vax/VMS Software may be open source, but packaging and services can be profitable businesses Open Source • Free redistribution • Source code • Derived works • Integrity of the author's source code • No discrimination against persons or groups Open Source • No discrimination against fields of endeavor • Distribution of license • License must not be specific to a product • License must not contaminate other software http://www.opensource.org/osd.html Practical Advice Be aware of the law, but do not pretend to be a lawyer. Use a professional for: • Contracts and licenses • Troubles (complaints, injunctions, subpoenas, etc.) • Personnel issues • When in doubt, ask help! Source Code Management Or Configuration Management: How I learned to Stop Worrying and Hate My Co-workers Less Source Code Management Also known as Configuration Management Source Code Managers are tools that: – Archive your development files – Serve as a single point of entry/exit when adding or updating development files Why You Want A Source Control System Supports concurrent development Manage diverging source code bases Records file/release versions Easy access to all previous revisions Can record why a revision was made Optimal disk space usage You’ll end up doing something equivalent anyway so it may as well be automated Source Code Management Tools Are Not A substitute for project management A replacement for developer communication How They Work Central database of source code, documentation, build tools Each file stored only once - all other versions are diffs of that one copy To Make a Change – Check out the latest version of a file – Make the changes – Update the database What should be in the database Source Code Documentation Build Tools – Often need old versions of the tools to build old versions of the software – Ensures software is rebuilt exactly as the customer received it Test Suites Anything else you might want later Version Control Companies ship several products from the same source base (i.e. Win NT and Windows 2000 versions of MS Office) When tracking down bugs you want to examine the code as it was when the product shipped Code Sharing Multiple people can work on the same source base without colliding – (1) Locks individual files so only one person at a time can modify it *OR* – (2) Allows multiple people to modify a source file and the system will automatically merge the changes (usually) Locking Only one person can work on a file at once Works fairly well if developers work on different areas of the project and don’t conflict often Problem 1: People forget to unlock files when they are done Problem 2: People work around locking by editing a private copy and checking in when the file is finally unlocked - easy to goof and lose changes Merging Several people can work on a file at once Before committing changes, each user merges their copy with the latest copy in the database – This is normally done automatically by the system and usually works, but you should not blindly accept the result of the merge Labeling Label all the files in the source base that make up a product at each milestone Just before and just after a major change (e.g.. changing several interfaces) When a new version ships Version Trees Each file in the database has a version tree Can branch off the version tree to allow separate development paths Typically a main path (trunk) for the next major version and branches off of shipped versions for maintenance Branching When a new version ships, typically create a branch in the version tree for maintenance Double update: fix a defect in the latest version and then merge the changes (often by hand) into the maintenance version Also create personal versions so you can make a change against a stable source base and then merge in the latest version later Examples RCS – Solaris: man rcsintro CVS – Solaris: man cvs – www.cyclic.com/cvs/info.html Visual SourceSafe – msdn.microsoft.com/SSAFE ClearCase – www.rational.com RCS File management only Transaction model – check out and lock – edit – check in and unlock Little support for binaries CVS Built on top of RCS – Therefore little support for binaries Database can be remote No locking: merge before commit Fast Integrates with emacs SourceSafe Microsoft’s entry into the field Project-based Checkout-edit-checkin model Built-in web site creation tools Integrates with MSDEV Clearcase Clearcase is configuration management on steroids You create a view of the database with a config spec, which describes how to select files from the database. When you set a view, Clearcase creates a virtual filesystem containing only those versions of the files selected by the config spec Clearcase Features Distributed System – Several groups at different locations can work on the same database Can install triggers – Example: e-mail the author of a file when some one makes a change to it Uses merging model like CVS, but can also lock files More Clearcase Features Integrates with MSDEV Build Management – Knows to rebuild out-of-date files even if your makefile doesn’t Slow and a bit buggy Helpful Rules for Version Control Bliss Archived Files Should Always Compile Code Review Files Before Check-in Compile and run latest archived files *as a set* before Check-in No Cheating (even “simple bug fixes” need to undergo this process) Software Engineering Lecture 10 Formal Specification Formal Specification Why? • Precise standard to define and validate software Why not? • May be time consuming • Methods not suitable for all applications Formal Specification Ben Potter, Jane Sinclair, David Till, An Introduction to Formal Specification and Z (Prentice Hall) 1991 Jonathan Jacky The Way of Z (Cambridge University Press) 1997 Mathematical Specification Example of specification B1, B2, ... Bk is a sequence of m x m matrices 1, 2, ... k is a sequence of m x m elementary matrices B1-1 = 1 B2-1 = 21 Bk-1 = k ... 21 The numerical accuracy must be such that, for all k, BkBk-1 - I < Specification of Programming Languages <unsigned number> ::= <unsigned integer> | <unsigned real> <unsigned integer> ::= <digit> {<digit>} <unsigned real> ::= <unsigned integer> . <digit> {<digit>} | <unsigned integer> . <digit> {<digit>} E <scale factor> | <unsigned integer> E <scale factor> <scale factor> ::= <unsigned integer> | <sign> <unsigned integer> <sign> ::= + | Pascal number syntax Formal Specification Using Diagrams unsigned integer digit unsigned number unsigned integer + . digit unsigned integer E - Two Rules • Formal specification does not guarantee correctness • Formal specification does not prescribe the implementation Example: Z Specification Language Informal: The function intrt(a) returns the largest integer whose square is less than or equal to a. Formal (Z): intrt: N N a : N• intrt(a) * intrt(a) < a < (intrt(a) + 1) * (intrt(a) + 1) Example: Algorithm 1 + 3 + 5 + ... (2n - 1) = n2 Example: Program int intrt (int a) /* Calculate integer square root */ { int i, term, sum; term = 1; sum = 1; for (i = 0; sum <= a; i++) { term = term + 2; sum = sum + term; } return i; } Finite State Machine A broadly used method of formal specification: • Event driven systems (e.g., games) • User interfaces • Protocol specification etc., etc., ... Finite State Machine Example: Therapy control console [informal description] State Transition Diagram Select field Enter Patients Enter Fields Start (ok) Setup Beam on Ready Stop (interlock) Select patient State Transition Table Select Select Enter Patient Field Patients Fields Patients Setup Patients Fields Ready Patients Fields Beam on ok Start Stop interlock Fields Setup Ready Beam on Setup Ready Setup Z Specification STATE ::= patients | fields | setup | ready | beam_on EVENT ::= select_patient | select_field | enter | start | stop | ok | interlock FSM == (STATE X EVENT) STATE no_change, transitions, control : FSM Continued on next slide Z Specification (continued) control = no_change transitions no_change = { s : STATE; e : EVENT • (s, e) transitions = { (patients, enter) (fields, select_patient) s} fields, patients, (fields, enter) setup, (setup, select_patient) patients, (setup, select_field) (setup, ok) ready, fields, (ready, select_patient) patients, (ready, select_field) fields, (ready, start) beam_on, (ready, interlock) setup, (beam_on, stop) ready, (beam_on, interlock) setup } Schemas Schema: • The basic unit of formal specification. • Describes admissible states and operations of a system. LibSys: An Example of Z Library system: • Stock of books • Registered users. • Each copy of a book has a unique identifier. • Some books on loan; other books on shelves available for loan. • Maximum number of books that any user may have on loan. LibSys: Operations • Issue a copy of a book to a reader. • Reader return a book. • Add a copy to the stock. • • • • Remove a copy from the stock. Inquire which books are on loan to a reader. Inquire which readers has a particular copy of a book. Register a new reader. • Cancel a reader's registration. LibSys Level of Detail: Assume given sets: Copy, Book, Reader Global constant: maxloans Schemas Describing Operations Naming conventions for objects: Before: plain variables, e.g., r After: with appended dash, e.g., r' Input: with appended ?, e.g., r? Output: with appended !, e.g., r! Operation: Issue a Book • Inputs: copy c?, reader r? • Copy must be shelved initially: c? shelved • Reader must be registered: r? readers • Reader must have less than maximum number of books on loan: #(issued {r?}) < maxloans • Copy must be recorded as issued to the reader: issued' = issued {c? r?} • The stock and the set of registered readers are unchanged: stock' = stock; readers' = readers Domain and Range X ran m y dom m x m:X Y Y dom m = { x X : y Y x y} ran m = { y Y : x X x y} Operation: Issue a Book Issue stock, stock' : Copy Book issued, issued' : Copy Reader shelved, shelved': F Copy readers, readers' : F Reader c?: Copy; r? :Reader [See next slide] Operation: Issue a Book (continued) Issue [See previous slide] shelved dom issued = dom stock shelved' dom issued' = dom stock' shelved dom issued = Ø; shelved' dom issued' = Ø ran issued readers; ran issued' readers' r : readers #(issued {r}) < maxloans r : readers' #(issued' {r}) < maxloans c? shelved; r? readers; #(issued {r?}) < maxloans issued' = issued {c? r?} stock' = stock; readers' = readers LibSys: Schema for Abstract States Library stock : Copy Book issued : Copy Reader shelved : F Copy readers: F Reader shelved dom issued = dom stock shelved dom issued = Ø ran issued readers r : readers • #(issued {r}) < maxloans Schema Inclusion LibDB stock : Copy Book readers: F Reader LibLoans issued : Copy Reader shelved : F Copy r : Reader • #(issued {r}) < maxloans shelved dom issued = Ø Schema Inclusion (continued) Library LibDB LibLoans dom stock = shelved dom issued ran issued readers Schema Decoration Issue Library Library' c? : Copy; r? : Reader c? shelved; r? readers #(issued {r?}) < maxloans issued' = issued {c? r?} stock' = stock; readers' = readers Schema Decoration Issue Library c? : Copy; r? : Reader c? shelved; r? readers #(issued {r?}) < maxloans issued' = issued {c? r?} stock' = stock; readers' = readers The Schema Calculus Schema inclusion Schema decoration Schema disjunction: ^ AddKnownTitle AddNewTitle AddCopy = Schema conjunction: ^ EnterNewCopy AddCopyAdmin AddCopy = Schema negation Schema composition Software Engineering Lecture 11 Object-Oriented Design I What is in a Requirements Document? Example (Web Butler and Web Site Profiler) • Run web data collection in real time or batch mode How are jobs started? • Job parameters How are the parameters set up (interactive, edit file, ...)? What are the parameters (specify)? Can job parameters be stored and used again? If so, how? • Job monitoring What feedback is given while job is running? Can the user pause or break a job? If so, are the results retained? What is in a Requirements Document? Remember • The requirements document specifies the functionality that you plan to deliver to the client • It must be comprehensive and detailed. Everything must be written out -- no hand waving! The requirements document is likely to be several times as long as Assignment 1. Assignment 2 -- Individual Parts One approach: With your document, include a list of who contributed what part to the Requirements study, e.g., Person A Requirements analysis for database design (member of team of 3), wrote Section 3.1 of document, worked with client to identify software needs. Person B Prepared visual aids for presentation, edited entire document, specified the security needs and wrote Section 4.2. The Waterfall Model Requirements Definition System and Software design Implementation and Unit Testing Integration and System Testing Operation and Maintenance Useful Texts Grady Booch, James Rumbaugh, Ivar Jacobson, The Unified Modeling Language. Addison-Wesley 1999. Grady Booch, Object-Oriented Analysis and Design with Applications, second edition. Benjamin/Cummings 1994. Rob Pooley, Perdita Stevens, Using UML Software Engineering with Objects and Components. Addison-Wesley 1999. The Importance of Modeling • A model is a simplification of reality. • We build models so that we can better understand the system we are developing. • We build models of complex system because we cannot comprehend such a system in its entirety. Models can be informal or formal. The more complex the project the more valuable a formal model becomes. BRJ Principles of Modeling • The choice of what models to create has a profound influence on how a problem is attacked and how a solution is shaped. • Every model can be expressed at different levels of precision. • The best models are connected to reality. • No single model is sufficient. Every nontrivial system is best approached through a small set of nearly independent models. BRJ The Unified Modeling Language UML is a standard language for modeling software systems. • Serves as a bridge between the requirements specification and the implementation. • Provides a means to specify and document the design of a software system. • Is process and programming language independent. • Is particularly suited to object-oriented program development. Notation: Classes Window origin size open() close() move() display() name attributes operations A class is a description of a set of objects that share the same attributes, operations, relationships and semantics. Notation: Interface ISpelling An interface is a collection of operations that specify a service of a class or component, i.e., the externally visible behavior of that element. Notation: Collaboration & Use Case Chain of responsibility A collaboration defines an interaction, i.e., a society of roles and other elements that work together to provide some cooperative behavior. Place order A use case is a description of a set of sequence of actions that a system performs that yields an observable result. Notation: Active Class EventManager eventlist suspend() flush() An active class is a class whose objects own one or more processes or threads and therefore can initiate control activity. Notation: Component & Node orderform.java A component is a physical and replaceable part of a system that conforms to and provides the realization of a set of interfaces. Server A node is a physical element that exists at run time and represents a computational resource. Notation: Behavioral Things: Messages & States display An interaction is a behavior that comprises a set of messages exchanged among a set of objects within a particular context to accomplish a specific purpose. Waiting A state machine is a behavior that specifies the sequence of states an object or an interaction goes through during its lifetime in response to events. Notation: Grouping and Annotation Business rules A package is a general-purpose mechanism for organizing elements into groups. return copy of self A note is a symbol for rendering constraints and comments attached to an element or a collection of elements. Notation: Relationships A dependency is a semantic relationship between two things in which a change to one may effect the semantics of the other. 0..1 employer * employee An association is a structural relationship that describes a set of links, a link being a connection among objects. Notation: Relationships (continued) child parent A generalization is a specialization/generalization relationship is which objects of the specialized element (child) are substitutable for objects of the generalized element (parent). A realization is a semantic relationship between classifiers, wherein one classifier specifies a contract that another classifier guarantees to carry out. Diagrams in UML A diagram is the graphical representation of a set of elements, usually rendered as a connected graph of vertices (things) and arcs (relationships). • Class diagram shows a set of classes, interfaces, and collaborations with their relationships. • Object diagram shows a set of objects and their relationships. • Use case diagram shows a set of use cases and actors (a special kind of class) and their relationships. Diagrams in UML (continued) • Interaction diagram shows an interaction, consisting of a set of objects and the relationships, including the messages that may be dispatched among them. => A sequence diagram emphasizes the time ordering. => A collaboration diagram emphasizes the structural organization of the objects that send and receive messages. Diagrams in UML (continued) • Statechart diagram shows a state machine consisting of states, transitions, events, and activities. • Activity diagram is a statechart diagram that shows the flow from activity to activity within a system. • Component diagram shows the organization and dependencies among a set of components. • Deployment diagram shows the configuration of processing nodes and the components that live on them. The HelloWorld Example class name HelloWorld operations paint() Abstraction for HelloWorld class name HelloWorld annotation operations paint() g.drawString ("HelloWorld", 0, 10)" The "Hello, World" Example import java.awt.Graphics; class HelloWorld extends java.applet.Applet { public void paint (Graphics g) { g.drawString ("Hello, World!", 10, 10); } } Example from: BJR Class Diagram Applet generalization Note that the Applet and Graphics classes are shown elided. HelloWorld paint() dependency Graphics Class Inheritance Diagram Object Panel interface Component ImageObserver Applet Container HelloWorld Packaging Classes java HelloWorld applet Graphics awt lang package Notation for Classes and Objects Classes AnyClass attribute1 attribute2 operation1() operation2() or AnyClass Objects anObject:AnyClass or :AnyClass or anObject The names of objects are underlined. Software Engineering Lecture 12 Object-Oriented Design II Requirements: the Long Term Believe that your software will be in use 5 years from now. • What happens at end of semester? Packaging and hand-over Client's technical preferences (C++, Java) • Some system decisions based on short-term considerations • Which formats, protocols, etc. do you think will last? (IIOP, RMI, SNMP, ...) Requirements, Design and Implementation Remember the definitions. Example: Consistency between two players of a board game • The requirement is ..... • The design is ..... What is a requirements specification? Modeling Classes Given a real-life system, how do you decide what classes to use? • What terms do the users and implementers use to describe the system? They are candidates for classes. • Is each candidate class crisply defined? • For each class, what is its set of responsibilities? Are the responsibilities evenly balanced among the classes? • What attributes and operations does each class need to carry out its responsibilities? Noun Identification: A Library Example The library contains books and journals. It may have several copies of a given book. Some of the books are reserved for short-term loans only. All others may be borrowed by any library member for three weeks. Members of the library can normally borrow up to six items at a time, but members of staff may borrow up to 12 items at one time. Only members of staff may borrow journals. The system must keep track of when books and journals are borrowed and returned and enforce the rules. Noun Identification: A Library Example The library contains books and journals. It may have several copies of a given book. Some of the books are reserved for short-term loans only. All others may be borrowed by any library member for three weeks. Members of the library can normally borrow up to six items at a time, but members of staff may borrow up to 12 items at one time. Only members of staff may borrow journals. The system must keep track of when books and journals are borrowed and returned and enforce the rules. Candidate Classes Library Book Journal Copy ShortTermLoan LibraryMember Week MemberOfLibrary Item Time MemberOfStaff System Rule the name of the system event measure repeat book or journal abstract term general term general term Relations between Classes Book Journal Copy LibraryMember Item MemberOfStaff Is Item needed? is an is an is a copy of a Item Item Book is a LibraryMember Operations LibraryMember borrows Copy LibraryMember returns Copy MemberOfStaff borrows Journal MemberOfStaff returns Journal Item not needed yet. Class Diagram MemberOfStaff LibraryMember 1 1 on loan on loan 0..12 Journal 0..* Copy is a copy of 1..* 1 Book Rough Sketch: Wholesale System A wholesale merchant supplies retail stores from stocks of goods in a warehouse. What classes would you use to model this business? Rough Sketch: Wholesale System RetailStore Order Merchant Product Warehouse Invoice Shipment Rough Sketch: Wholesale System RetailStore name address contactInfo financialInfo Merchant Warehouse Order Product Reversals Invoice Shipment damaged() return() wrongItem() Responsibilities -track status of shipped products responsibility (text field) Expanding a Class: Modeling Financial Information RetailStore association 1 * Transaction Which class is responsible for the financial records for a store? Payment Invoice Modeling Invoice Shipment ??? RetailStore invoiceRecord goodsShipped Invoice invoiceNumber adornments +goodsShipped() + public -sendInvoice() - private PartsList Lessons Learned Design is empirical. There is no single correct design. During the design process: • Eliding: Elements are hidden to simplify the diagram • Incomplete: Elements may be missing. • Inconsistency: The model may not be consistent The diagram is not the whole design. Diagrams must be backed up with specifications. Levels of Abstraction The complexity of a model depends on its level of abstraction: • High-levels of abstraction show the overall system. • Low-levels of abstraction are needed for implementation. Two approaches: • Model entire system at same level of abstraction, but present diagrams with different levels of detail. • Model parts of system at different levels of abstraction. Component Diagram executable component hello.hml HelloWorld.class hello.jpg hello.java Actor and Use Case Diagram • An actor is a user of a system in a particular role. BookBorrower Borrow book An actor can be human or an external system. • A use case is a a task that an actor needs to perform with the help of the system. Use Cases and Actors • A scenario is an instance of a use case • Actor is role, not an individual (e.g., librarian can have many roles) • Actor must be a "beneficiary" of the use case (e.g., not librarian who processes book when borrowed) In UML, the system boundary is the set of use cases. Use Cases for Borrowing Books Borrow copy of book BookBorrower Return copy of book Reserve book Extend loan Relationships Between Use Cases: <<uses>> Extend loan BookBorrower Borrow copy of book <<uses>> Check for reservation <<uses>> Relationships Between Use Cases: <<extends>> BookBorrower <<extends>> Borrow copy of book Refuse loan Use Cases in the Development Cycle • Use cases are a tool in requirements analysis • Intuitive -- easy to discuss with clients • Use cases are often hard to translate into class models • Scenarios are useful to validate design Software Engineering Lecture 13 Object-Oriented Design III Comments on Presentations Presentation • • Standard of graphics has been high Some text too small (diagrams, screen dumps) Content • • Level of detail Requirements v. design The client defines the requirements Well done, but time is short. What is your critical path? Modeling Dynamic Aspects of Systems Interaction diagrams: set of objects and their relationships including messages that may be dispatched among them • Sequence diagrams: time ordering of messages • Collaboration diagrams: structural organization of objects that send and receive messages Activity diagram: flow chart showing flow of control from activity to activity Statechart diagram: models a state machine Bouncing Ball Diagrams Example: http://www.cs.cornell.edu/ domain name TCP connection HTTP get Client Servers Actions on Objects returnCopy(c) call okToBorrow() return send local status notifyReturn(b) create <<create>> destroy <<destroy>> asynchronous signal stereotypes Links LibraryMember 1 on loan 0..* Copy association +borrowCopy() +returnCopy() class libMem:LibraryMember object message borrowCopy(c) c:Copy link Sequence Diagram: Change in Cornell Program :MEngStudent Cornellian 1 : getName() 1.1 : name 2: new PhDStudent(name) :PhDStudent 3: <<destroy>> sequence numbers added to messages Sequence Diagram: Borrow copy of a Book libMem: LibraryMember BookBorrower theBook:Book theCopy:Copy borrow(theCopy) okToBorrow borrow borrow Class Inheritance Diagram Object Panel interface Component ImageObserver Applet Container HelloWorld Sequence Diagram:Painting Mechanism :Thread run :Toolkit run :ComponentPeer callbackLoop handleExpose paint target:HelloWorld Activity Diagram: Process Modeling Release work order branch [materials not ready] Reschedule [materials ready] Assign tasks guard expression Activity Diagram: Parallel Activities start state Decompress fork Stream video join stop state Stream audio State Diagram returned() not borrowable returned() borrowable borrowed()[last copy] guard expression borrowed()[not last copy] State diagram for class Book Implementation Modeling Subsystem A grouping of elements that specifies what a part of a system should do. Component (UML definition) "A distributable piece of implementation of a system, including software code (source, binary, or executable) but also including business documents, etc., in a human system." A component can be thought of as an implementation of a subsystem. Component Diagram executable component hello.hml HelloWorld.class hello.jpg hello.java Components and Classes agent.dll AgentAction PatternSearch Policy Components and Classes agent.dll Realizes AgentAction PatternSearch Policy extended component Components and Classes Classes represent logical abstractions. Components represent physical things. Components may live on nodes. Classes have attributes and operations directly. Components have operations that are reachable only through interfaces. Interfaces render.java simulation.exe IRender dependency realization interface Application Programming Interface (API) API is an interface that is realized by one or more components. simulation.exe IRender IModels ILighting Components and Replaceability Components allow system to be assembled from binary replaceable elements. • A component is physical -- bits not concepts • A component can be replaced by any other component(s) that conforms to the interfaces. • A component is part of a system. • A component provides the realization of a set of interfaces. Software Engineering Lecture 14 System Architecture I Data Intensive Systems System Architecture The overall design of a system: • • • • • • Computers and networks (e.g., monolithic, distributed) Interfaces and protocols (e.g., http, CORBA) Databases (e.g., relational, distributed) Security (e.g., smart card authentication, SSL) Operations (e.g., backup, archiving, audit trails) Software environments (e.g., languages, source control tools) Data Intensive Systems Examples • Electricity utility customer billing • Telephone company call recording and billing • Car rental reservations (e.g., Hertz) • Stock market brokerage (e.g., Charles Schwab) • Web sales (e.g., Amazon.com) Example 1: Electricity Utility Billing First attempt: Transaction Data input Master file Each transaction handled as it arrives. Bill Criticisms of First Attempt Where is this first attempt weak? The requirements have not been specified!!! Transaction Types • • • • • • • • Create account / close account Meter reading Payment received Other credits / debits Check cleared / check bounced Account query Correction of error etc., etc., etc., Typical Requirements • All payments to be credited on day received • Customers must be able to query account by telephone • Cutting off service for non-payment requires management authorization • Data input staff should process n transactions per day per person • Error rate must be below 0.01% • System available 99.9% of business hours Batch Processing: Validation errors Edit & validation Incoming transactions Data input read only Master file Validated transactions Batch Processing: Master File Update Validated transactions in batches errors Reports Sort by account Master file update Bills Instructions Benefits of Batch Updating • All transactions for an account are processed together • Backup and recovery have fixed checkpoints • Better management control of operations • Efficient use of staff and hardware Online Inquiry Customer service read only Transactions Data input Master file Bills Example 2: A Small-town Stockbroker • Transactions Received by mail or over telephone For immediate or later action • Complex customer inquiries • Highly competitive market A Database Architecture Database(s): • Customer and account database • Financial products (e.g., account types, pension plans, savings schemes) • External databases (e.g., stock markets, mutual funds, insurance companies) Database Architecture Products & services database Customer & account database External services Real-time Transaction Real-time transactions Products & services database Customer & account database External services Real-time Transactions & Batch Processing Real-time transactions Products & services database Data input Batch processing Customer & account database External services Architectural considerations • Real-time service during scheduled hours + batch processing overnight • Combine information from several databases • Database consistency after any type of failure two-phase commit reload from checkpoint + log detailed audit trail • How will transaction errors be avoided? • How will transaction errors be corrected? Example: Merger of Two Banks Each bank has a database with its customer accounts. The databases are used by staff at many branches and for back-office processing. The requirement is to integrate the two banks so that they appear to the customers to be a single organization and to provide integrated service from all branches. Merger of Two Banks: Options A ??? ??? B Merger of Two Banks: Architectural Options I. Convert everything to System A. convert databases retrain staff enhance System A (software and hardware) discard System B II. Build an interface between the databases in System A and System B. III. Extend client software so that it can interact with either System A or System B database. Distributed Computing: General Problem An application that is running on one computer wishes to use data or services provided by another: • Network connection private, public, or virtual private network location of firewalls • Protocols point-to-point, multicast, broadcast message passing, RPC, distributed objects stateful or stateless • Quality of service Network Choices Public Internet: Ubiquitous -- worldwide Low cost Private network: Security Predictable performance Choice of protocols (e.g., IBM's SNA) Quality of Network Services Performance Maximum throughput Variations in throughput Real-time media (e.g., audio) Business Suppliers Trouble shooting and maintenance Upgrades Firewall Private network Public network Firewall A firewall is a computer at the junction of two network segments that: • Inspects every packet that attempts to cross the boundary • Rejects any packet that does not satisfy certain criteria, e.g., an incoming request to open a TCP connection an unknown packet type Software Engineering Lecture 15 System Architecture II Distributed and Real Time Systems Comments on Requirements Report Audience • • Client and design team Will be updated over time Content • • • Level of detail -- will be used to validate the implementation Requirements, not design Precise, but not legalistic Sequence Diagram: Notation libMem: LibraryMember BookBorrower theBook:Book theCopy:Copy borrow(theCopy) okToBorrow borrow borrow dotted line shows object lifetime rectangle shows focus of control Sequence Diagram: Branching libMem: LibraryMember theBook:Book theCopy:Copy BookBorrower 1:borrow(theCopy) 2:okToBorrow [not ok]3:noborrow [ok]3:borrow 4:borrow branch Example: Distributed Database two copies of the same data Distributed Data and Replication Distributed Data Data is held on several computer systems. A transaction may need to assemble data from several sources. Replication Several copies of the data are held in different locations. Mirror: Complete data set is replicated Cache: Dynamic set of data is replicated (e.g., most recently used) With replicated data, the biggest problem is consistency. Example: Broadcast Search User User interface server Databases Example: UseNet Stateless Protocol v. Stateful Stateless protocol Example: http Open connection Send message Return reply Close connection State in http must be sent with every message (e.g., as parameter string or in a cookie) Stateless Protocol v. Stateful Stateful (session) protocol Example: Z39.50 Open connection Begin session Interactive session End session Close connection Server remembers the results of previous transactions (e.g., authentication, partial results) until session is closed. Firewall Private network Public network Firewall A firewall is a computer at the junction of two network segments that: • Inspects every packet that attempts to cross the boundary • Rejects any packet that does not satisfy certain criteria, e.g., an incoming request to open a TCP connection an unknown packet type The Domain Name System First attempt to resolve www.cs.cornell.edu .edu server 1 2 3 cornell.edu server cs.cornell.edu server Discussion of the First Attempt Problems? The Domain Name System Better method local DNS server 1 almaden.ibm.com cornell.edu Local ece.cmu.edu cache ibm.com acm.org .edu .edu server 2 cornell.edu server 3 cs.cornell.edu server Real Time System A real time system is a software system whose correct functioning depends upon the results produced and the time at which they are produced. • A soft real time system is degraded if the results are not produced within required time constraints • A hard real time system fails if the results are not produced within required time constraints Example: Web Server http message daemon TCP port 80 spawned processes The daemon listens at port 80. When a message arrives it: spawns a processes to handle the message returns to listening at port 80 Embedded Systems Software and hardware are combined to provide an integrated unit, usually dedicated to a specific task: • Digital telephone • Automobile engine control • GPS • Scientific instruments The software may be embedded in the device in a manner that can not be altered after manufacture. Example: Autonomous Land Vehicle GPS Steer Sonar Model Laser Control signals Throttle Controls Sensors Signal processing Other Applications Response critical • • Network router Telephone switch • Seat bag controller Shared systems • • Multi-user data processing Time sharing Techniques • Special purpose hardware • Multi-threading and multi-tasking • Parallel processing => digital signal processing • Interrupts => levels and priorities Multi-Threading Several similar threads operating concurrently: • Re-entrant code -- separation of pure code from data for each thread • Testing -- single thread and multi thread May be real time (e.g., telephone switch) or nontime critical Real Time Executive Schedules and dispatches tasks in a real time system • Real time clock • Interrupt handler • Scheduler • Resource manager • Dispatcher Must be extremely reliable Timing Timing mechanisms • Synchronous (clocked) -- periodic stimuli • Asynchronous -- wait for next signal Example: Communications protocols may be synchronous or asynchronous Hardware v. Software Design of embedded systems requires close understanding of hardware characteristics • Special purpose hardware requires special tools and expertise. • Some functions may be implemented in either hardware of software (e.g., floating point unit) • Design requires separation of functions Distinction between hardware and software may be blurred. Example: Dartmouth Time Shared System master processor Communications processor I/O Mulitplexor Central processor Central processor Communications processor Central processor Software Considerations Resource considerations may dictate software design and implementation: • Low level language (e.g., C) where programmer has close link to machine • Inter-process communication may be too slow (e.g., C fork). • May implement special buffering, etc., to control timings Example: CD Controller 4 Input block 7 3 2 5 6 Circular buffer 1 Output block Continuous Operation Many systems must operate continuously • Software update while operating • Hardware monitoring and repair • Alternative power supplies, networks, etc. • Remote operation These functions must be designed into the fundamental architecture. Routers and Other Network Computing • Interoperation with third party devices • Support for several versions of protocols • Restart after total failure • Defensive programming -- must survive => erroneous or malicious messages => extreme loads • • Time outs, dropped packets, etc. Evolution of network systems Software Engineering Lecture 15 System Architecture II Distributed and Real Time Systems Administration Assignment 2: Requirements • • • Grades -- presentation, report, individual Comments at presentation Comments from teaching assistant Assignment 3: Design Comments on Requirements Report Audience • • Client and design team Will be updated over time Content • • • Level of detail -- will be used to validate the implementation Requirements, not design Precise, but not legalistic Sequence Diagram: Notation libMem: LibraryMember BookBorrower theBook:Book theCopy:Copy borrow(theCopy) okToBorrow borrow borrow dotted line shows object lifetime rectangle shows focus of control Sequence Diagram: Branching libMem: LibraryMember theBook:Book theCopy:Copy BookBorrower 1:borrow(theCopy) 2:okToBorrow [not ok]3:noborrow [ok]3:borrow 4:borrow branch Example: Distributed Database two copies of the same data Distributed Data and Replication Distributed Data Data is held on several computer systems. A transaction may need to assemble data from several sources. Replication Several copies of the data are held in different locations. Mirror: Complete data set is replicated Cache: Dynamic set of data is replicated (e.g., most recently used) With replicated data, the biggest problem is consistency. Example: Broadcast Search User User interface server Databases Example: UseNet Stateless Protocol v. Stateful Stateless protocol Example: http Open connection Send message Return reply Close connection State in http must be sent with every message (e.g., as parameter string or in a cookie) Stateless Protocol v. Stateful Stateful (session) protocol Example: Z39.50 Open connection Begin session Interactive session End session Close connection Server remembers the results of previous transactions (e.g., authentication, partial results) until session is closed. Firewall Private network Public network Firewall A firewall is a computer at the junction of two network segments that: • Inspects every packet that attempts to cross the boundary • Rejects any packet that does not satisfy certain criteria, e.g., an incoming request to open a TCP connection an unknown packet type The Domain Name System First attempt to resolve www.cs.cornell.edu .edu server 1 2 3 cornell.edu server cs.cornell.edu server Discussion of the First Attempt Problems? The Domain Name System Better method local DNS server 1 almaden.ibm.com cornell.edu Local ece.cmu.edu cache ibm.com acm.org .edu .edu server 2 cornell.edu server 3 cs.cornell.edu server Real Time System A real time system is a software system whose correct functioning depends upon the results produced and the time at which they are produced. • A soft real time system is degraded if the results are not produced within required time constraints • A hard real time system fails if the results are not produced within required time constraints Example: Web Server http message daemon TCP port 80 spawned processes The daemon listens at port 80. When a message arrives it: spawns a processes to handle the message returns to listening at port 80 Embedded Systems Software and hardware are combined to provide an integrated unit, usually dedicated to a specific task: • Digital telephone • Automobile engine control • GPS • Scientific instruments The software may be embedded in the device in a manner that can not be altered after manufacture. Example: Autonomous Land Vehicle GPS Steer Sonar Model Laser Control signals Throttle Controls Sensors Signal processing Other Applications Response critical • • Network router Telephone switch • Seat bag controller Shared systems • • Multi-user data processing Time sharing Techniques • Special purpose hardware • Multi-threading and multi-tasking • Parallel processing => digital signal processing • Interrupts => levels and priorities Multi-Threading Several similar threads operating concurrently: • Re-entrant code -- separation of pure code from data for each thread • Testing -- single thread and multi thread May be real time (e.g., telephone switch) or nontime critical Real Time Executive Schedules and dispatches tasks in a real time system • Real time clock • Interrupt handler • Scheduler • Resource manager • Dispatcher Must be extremely reliable Timing Timing mechanisms • Synchronous (clocked) -- periodic stimuli • Asynchronous -- wait for next signal Example: Communications protocols may be synchronous or asynchronous Hardware v. Software Design of embedded systems requires close understanding of hardware characteristics • Special purpose hardware requires special tools and expertise. • Some functions may be implemented in either hardware of software (e.g., floating point unit) • Design requires separation of functions Distinction between hardware and software may be blurred. Example: Dartmouth Time Shared System master processor Communications processor I/O Mulitplexor Central processor Central processor Communications processor Central processor Software Considerations Resource considerations may dictate software design and implementation: • Low level language (e.g., C) where programmer has close link to machine • Inter-process communication may be too slow (e.g., C fork). • May implement special buffering, etc., to control timings Example: CD Controller 4 Input block 7 3 2 5 6 Circular buffer 1 Output block Continuous Operation Many systems must operate continuously • Software update while operating • Hardware monitoring and repair • Alternative power supplies, networks, etc. • Remote operation These functions must be designed into the fundamental architecture. Routers and Other Network Computing • Interoperation with third party devices • Support for several versions of protocols • Restart after total failure • Defensive programming -- must survive => erroneous or malicious messages => extreme loads • • Time outs, dropped packets, etc. Evolution of network systems Software Engineering Lecture 16 System Architecture III Distributed Objects Real-Time: Software Considerations Resource considerations may dictate software design and implementation: • Low level language (e.g., C) where programmer has close link to machine • Inter-process communication may be too slow (e.g., C fork). • May implement special buffering, etc., to control timings Buffering Example: CD Controller 4 Input block 7 3 2 5 6 Circular buffer 1 Output block Continuous Operation Many systems must operate continuously • Software update while operating • Hardware monitoring and repair • Alternative power supplies, networks, etc. • Remote operation These functions must be designed into the fundamental architecture. Example: Routers and Other Network Computing • Interoperation with third party devices • Support for several versions of protocols • Restart after total failure • Defensive programming -- must survive => erroneous or malicious messages => extreme loads • • Time outs, dropped packets, etc. Evolution of network systems Example: Transaction Monitor messages Transaction monitor processes A transaction monitor: monitors transactions, routes them across services, balances the load, restarts transactions after failure. Software Reuse: Application Packages • Package supports a standard application (e.g., payroll, user interface to Internet information, mathematical algorithms) • Functionality can be enhanced by: => configuration parameters (e.g., table driven) => extensibility at defined interfaces => custom written source code extensions Reuse: Object Object Oriented Languages Example: Java is a relatively straightforward language with a very rich set of class hierarchies. • Java programs derive much of their functionality from standard classes • Learning and understanding the classes is difficult. => Java experts can write complex systems quickly => Inexperienced Java programmers write inelegant and buggy programs Reuse: Objects - Basic Definitions • An object is a piece of code that owns attributes and provides services through methods. • The methods operate on instance data owned by the object. • A class is a collection of like objects. Reuse: Objects - Characteristics • Encapsulation. An object has a public interface that defines how other objects or applications can interact with it. methods public instance data • Inheritance. Subclasses can be derived from parent classes. They inherit or override the parents' methods and instance data. • Polymorphism. The effect of a method can vary depending on the class that implements it (e.g., display_object) Reuse: Objects - Object Binding Binding is the linking of the software interface between two objects. • Static binding: The interface is determined at compile or build time. Straightforward Allows type checking • Dynamic binding or late binding: The link is established at run time. Flexible and extensible Complex Reuse: Objects - Distributed Objects Objects on separate computers interact through method calls and instance data. Major systems: • CORBA (Common Object Request Broker Architecture) • Microsoft family: OLE, COM, DCOM, Active X ... Desirable Properties of Distributed Objects • Different languages and operating environments • Reusable code: components • Architecture can be extensible • Future changes can be localized • Standard tools used for client/server interactions Example: Fedora IDL A research project to explore extensibility: -- very simple Interface Definition Language -- powerful tools for extensions -- interoperability, Cornell and CNRI http://www.cs.cornell.edu/cdlrg/fedora.html Object Request Broker (ORB) C C++ Cobol Objects IDL IDL IDL Interface IDL Java Client Other IDL Server Object Request Broker Interface Definition Language module <identifier> { <type declarations>; <constant declarations>; <exception declarations>; { Naming context interface <identifier> [:<inheritance>] { See next slide } Define a class interface <identifier> [:<inheritance>] { ..... } Define a class Interface Definition Language (continued) interface <identifier> [:<inheritance>] { <type declarations>; <constant declarations>; <exception declarations>; Define a class [<op_type] <identifier>(<parameters>) Define a method [raises exception] [context]; .... [<op_type] <identifier>(<parameters>) Define a method [raises exception] [context]; .... } ORB: Programmer's View Client Invoke a on object X Server Invoke a on object Y Object X Object Y a a Object Request Broker Object Request Broker (ORB) An ORB lets objects make requests to and receive response from other objects located locally or remotely. • Static and dynamic method invocations • High-level language bindings • Self-describing system • Local/remote transparency • Inter-ORB protocols Internet Inter-ORB Protocol (IIOP) ORB: System View Interface repository Client Dynamic Client IDL ORB invocation stubs interface Object Implementation implementation repository Static Dynamic Object skeletons invocation adapter Object Request Broker CORBA Services • • • • • • • • • • • • • Naming service Event service Concurrency control service Transaction service Relationship service Externalization service Query service Life cycle service Persistence service Licensing service Properties service Security service Time service Distributed Objects and the System Life-Cycle All large systems change with time. • Dynamic binding of objects combined with polymorphism permits the addition of extra object types, incremental changes, etc. to be localized. Development environments change with time. • Language bindings and IIOP permit changes. Production environments changes with time. • Code can be reused in different environments. Software Engineering Lecture 17 Design for Usability I Q2: Finite State Machine The cruise control system on an automobile is controlled by a master switch and three buttons. Initially, it is turned on by the master switch. The master switch can be turned off at any time. When first turned on, the system enters stand-by mode. When the system is in stand-by mode, the driver of the automobile can press Button A to engage the cruise control at the current speed of the automobile. When the cruise control is engaged, if the driver presses the brake or presses Button B the system will be disengaged and return to stand-by mode. After returning to standby mode, the driver can press Button C to engage the cruise control at the speed that it was set at previously. (After the system is first turned on, Button C has no effect.) When the cruise control is engaged, the driver can press Button A to increase speed by one mile per hour or Button C to decrease speed by one mile per hour. State Transition Diagram A C MS-On A Off Standby A Engaged C B-Brake MS-Off Standby1 State Transition Table MS on Off MS off A B Brake C Standby Standby Off Engaged Engaged Off Engaged Standby1 Engaged Standby1 Off Engaged Engaged Question 4 When software is written, who owns the copyright? How can somebody else be permitted to use the software? How can copyright be transferred to somebody else? Question 4 When software is written, who owns the copyright? The person who writes the software Except work for hire -- the employer How can somebody else be permitted to use the software? By permission from the copyright owner (usually a license) How can copyright be transferred to somebody else? Copyright is property that can be sold or given away (usually a contract) Question 4 You are employed for company X writing software. When you leave, who owns your work? What use can you make of the work? Question 4 You are employed for company X writing software. When you leave, who owns your work? The company (work for hire) What use can you make of the work? None without permission of the copyright owner Question 4 You work free-lance for company X. When you finish, who owns your work? What use can you make of the work? Question 4 You work free-lance for company X. When you finish, who owns your work? It depends on the circumstances Have a written contract What use can you make of the work? If you hold the copyright -- unrestricted Otherwise -- none without agreement Distributed Objects and the System LifeCycle All large systems change with time. • Dynamic binding of objects combined with polymorphism permits the addition of extra object types, incremental changes, etc. to be localized. Development environments change with time. • Language bindings and IIOP permit changes. Production environments changes with time. • Code can be reused in different environments. Design for Usability Usability of a computer system is a combination of factors: • User interface design • Functionality • Performance • Help systems and documentation • Freedom from errors Anything else? Iterative Design Evaluation Requirements Implementation (prototype) Design Methods for Evaluation of Usability • Observing users (user protocols) • Focus groups • Measurements effectiveness in carrying out tasks speed • Expert review • Client's opinions • Competitive analysis Levels of Usability interface design conceptual model functional design data and metadata computer systems and networks The Conceptual Model The conceptual model is the user's internal model of what the system provides: • The desk top metaphor -- files and folders • The web model -- click on hyperlinks • The library model -- search and retrieve • The form filling model -- fill form, submit Example: The Mercury page turner Interface Design The interface design is the appearance on the screen and the actual manipulation by the user • Fonts, colors, logos, key board controls, menus, buttons • Mouse control or keyboard control? • Conventions (e.g., "back", "help") Example: Screen space utilization in the Mercury page turner Principles of Interface Design Interface design is partly an art; there are general principles: • Consistency -- in appearance, controls, and function. • Feedback -- what is the computer system is doing? why does the user see certain results? • Users should be able to interrupt or reverse actions • Error handling should be simple and easy to comprehend • Skilled users offered shortcuts; beginners have simple, well-defined options The user should feel in control Disabilities • What if the user: is visually impaired or color blind? does not speak English? is a poor typist? • There is a tradition of blind programmers • Navigation of web sites need not be only visual You may have a legal requirement to support people with disabilities Functional Design The functional design, determines the functions that are offered to the user • Selection of parts of a digital object • Searching a list or sorting the results • Help information • Manipulation of objects on a screen • Pan or zoom Same Functions, Different Interface Example: The desk top metaphor • Mouse -- 1 button (Macintosh), 2 button (Windows) or 3 button (Unix) • Close button -- left of window (Macintosh) right of window (Windows) Data and Metadata Data and metadata stored by the computer system enable the functions and the interface • The desktop metaphor has the concept of associating a file with an application. This requires a file type to be stored with each file: -- extension to filename (Windows and Unix) -- resource fork (Macintosh) • Data validation often requires that a user interface has access to a database (e.g., names and addresses) Computer Systems and Networks The performance, reliability and predictability of computer systems and networks is crucial to usability • Response time instantaneous for mouse tracking and echo of key stroke 5 seconds for simple transactions • Example: Pipelined algorithm for the Mercury page turner • Quality of Service for real time information Design Tensions in Networked Systems • Client computers and network connections vary greatly in capacity • Client software may run on various operating systems; it may be current or an earlier version • System designers wish to control clients; users wish to configure their own environments Usability and Cost • Performance may be expensive in hardware or special software development • User interface development may be a major part of a software development project • Costs are multiplied if a user interface has to be used on different computers or migrate to different versions of systems Web browsers provide a general purpose user interface that others maintain Extensibility in Web Browsers • Data types: helper applications, plug-ins • Protocols HTTP, WAIS, Gopher, FTP, etc. proxies • Executable code CGI scripts at server JavaScript at client Java applets • Style sheets Software Engineering Lecture 18 Design for Usability II Q5: Object Oriented Design A system generates weather maps using data collected from unattended weather stations. Each weather station collects meteorological data and produces summaries of the data. On request, it sends the summary information to an area computer. The area computer uses a database of digitized maps to generate a set of local weather maps. Noun Identification: A Library Example The library contains books and journals. It may have several copies of a given book. Some of the books are reserved for short-term loans only. All others may be borrowed by any library member for three weeks. Members of the library can normally borrow up to six items at a time, but members of staff may borrow up to 12 items at one time. Only members of staff may borrow journals. The system must keep track of when books and journals are borrowed and returned and enforce the rules. Q5: Noun Identification A system generates weather maps using data collected from unattended weather stations. Each weather station collects meteorological data and produces summaries of the data. On request, it sends the summary information to an area computer. The area computer uses a database of digitized maps to generate a set of local weather maps. Candidate Classes Library Book Journal Copy ShortTermLoan LibraryMember Week MemberOfLibrary Item Time MemberOfStaff System Rule the name of the system event measure repeat book or journal abstract term general term general term Q5: Candidate Classes System general term WeatherMap Data same as MeteorologicalData WeatherStation is this a general term? MeteorologicalData how does this relate to WeatherStation? DataSummary how does this relate to DataSummary? AreaComputer hardware Database general term DigitizedMap Q5: Observations about the Candidate Classes WeatherMap is a DigitizedMap is derived from 1...* DataSummary WeatherStation has a set of MeteorologicalData MeteorologicalData DataSummary is derived from MeteorologicalData DigitizedMap Can Meteorological Data be an attribute of WeatherStation? Can DataSummary be combined with WeatherMap? Q5: Attributes and operations WeatherStation location metereologicalData collectData() getSummary() DigitizedMap location geographicData printMap() WeatherMap location date-time geographicData weather gatherData() printMap() Or should MetereologicalData be a separate object? Class Diagram MemberOfStaff LibraryMember 1 1 on loan on loan 0..* 0..12 Journal Copy is a copy of 1..* 1 Book Q5: Class Diagram DigitizedMap WeatherStation location metereologicalData collectData() getSummary() WeatherMap 1 1...* summary location date-time geographicData weather gatherData() printMap() Command Line Interfaces User interacts with computer by typing commands • Allows complex instructions to be given to computer • Facilitates formal methods of specification & implementation • Skilled users can input commands quickly • Requires learning or training • Can be adapted for people with disabilities • Can be multi-lingual • Suitable for scripting / non-human clients Direct Interaction User interacts with computer by manipulating objects on screen • Can be intuitive and easy to learn • Users get immediate feedback • • • • Not suitable for complex interactions Does not require typing skills Straightforward for casual users, slow for skilled users Icons can be language-independent • Difficult to build scripts • Only suitable for human users Design for Direct Manipulation • Conceptual models, metaphors, icons => there may not be an intuitive model • Navigation around large space • Conventions are growing over the years => scroll bars, buttons, help systems, sliders => good for users, good for designers Menus • Easy for users to learn and use • Certain categories of error are avoided • Enables context-sensitive help Major difficulty is structure of large menus • • • • Scrolling menus (e.g., states of USA) Hierarchical Associated control panels Menus plus command line Users prefer broad and shallow to deep menu systems Information Presentation Information to be displayed Presentation software Display Information Presentation • Text precise, unambiguous fast to compute and transmit • Graphics simple to comprehend uses of color shows variations Help System Design Help system design is difficult! • Must prototype with mixed users • Categories of help: => Overview and general information => Specific or context information => Tutorials (general) => Cook books and wizards => Emergency ("I am in trouble ...") • Must have many routes to same information Never blame the user! System Considerations of User Interfaces • Personal computer cycles are there to be used • Any network transfer involves delay • Shared systems have unpredictable performance • Data validation often requires access to shared data • Mobile code poses security risks Usability and Cost • Performance may be expensive in hardware or special software development • User interface development may be a major part of a software development project • Costs are multiplied if a user interface has to be used on different computers or migrate to different versions of systems Web browsers provide a general purpose user interface that others maintain Extensibility in Web Browsers • Data types: helper applications, plug-ins • Protocols HTTP, WAIS, Gopher, FTP, etc. proxies • Executable code CGI scripts at server JavaScript at client Java applets • Style sheets Web User Interface: Basic Web browser Web servers • Static pages from server • All interaction requires communication with server Web User Interface: CGI Script User interface tables CGI Scripts Web browser Web servers • Scripts can configure pages • Scripts can validate information • All interaction requires communication with server Web User Interface: JavaScript html Java Script Web browser User interface tables CGI Scripts Web servers • JavaScripts can validate information as typed • Some interactions are local • Server interaction constrained by web protocols Web User Interface: Applet Any server Applets Web browser Web servers • Any executable code can run on client • Client can connect to any server Device-Aware User Interfaces • Examples of devices: desk-top computer, fast network connection laptop computer, intermittent connectivity PalmPilot, intermittent use of cradle Smart telephone Digital camera, camcorder • Device-aware user interfaces are aware of: => Performance of device => Connectivity The Importance of Design Good support for users is more than a cosmetic flourish • Elegant design, appropriate functionality, & responsive system: => a measurable difference to their effectiveness • A system that is hard to use: => users may fail to find important results, or mis-interpret what they do find => user may give up in disgust A computer system is only as good as the interface it provides to its users Software Engineering Lecture 19 Performance of Computer Systems Moore's Law Original version: The density of transistors in an integrated circuit will double every year. (Gordon Moore, Intel, 1965) Current version: Cost/performance of silicon chips doubles every 18 months. Moore's Law and System Design Design system: Production use: Withdrawn from production: 2000 2003 2013 Processor speeds: Memory sizes: Disk capacity: 1 1 1 1.9 1.9 2.2 28 28 51 System cost: 1 0.4 0.01 Moore's Law: Rules of Thumb Planning assumptions: Every year: cost/performance of silicon chips improves 25% cost/performance of magnetic media improves 30% 10 years = 100:1 20 years = 10,000:1 Parkinson's Law Original: Work expands to fill the time available. (C. Northcote Parkinson) Planning assumptions: (a) Demand will expand to use all the hardware available. (b) Low prices will create new demands. (c) Your software will be used on equipment that you have not envisioned. False Assumptions Unix file system will never exceed 2 Gbytes (232 bytes). AppleTalk networks will never have more than 256 hosts (28 bits). GPS software will not last 1024 weeks. Nobody at Dartmouth will ever earn more than $10,000 per month. etc., etc., ..... Moore's Law and the Long Term What level? Within your working life? 1965 2000? When? Predicting System Performance • Mathematical models • Simulation • Direct measurement All require detailed understanding of the interaction between software and systems. Queues arrive wait in line service Single server queue depart Queues service arrive wait in line Multi-server queue depart Mathematical Models Queueing theory Good estimates of congestion can be made for singleserver queues with: • arrivals that are independent, random events (Poisson process) • service times that follow families of distributions (e.g., negative exponential, gamma) Many of the results can be extended to multi-server queues. Utilization: Rule of Thumb mean service time utilization = mean inter-arrival time When the utilization of any system component exceeds 30%, be prepared for congestion. Behavior of Queues: Utilization mean delay utilization 0 1 Simulation Model the system as set of states and events advance simulated time determine which events occurred update state and event list repeat Discrete time simulation: Time is advanced in fixed steps (e.g., 1 millisecond) Next event simulation: Time is advanced to next event Events can be simulated by random variables (e.g., arrival of next customer, completion of disk latency) Timescale Operations per second CPU instruction: 400,000,000 Disk latency: read: 60 25,000,000 bytes Network LAN: dial-up modem: 10,000,000 bytes 6,000 bytes Measurements on Operational Systems • Benchmarks: Run system on standard problem sets, sample inputs, or a simulated load on the system. • Instrumentation: Clock specific events. Serial and Parallel Processing Single thread v. multi-thread e.g., Unix fork Granularity of locks on data e.g., record locking Network congestion e.g., back-off algorithms Example: Performance of Disk Array Each transaction must: wait for specific disk platter wait for I/O channel signal to move heads on disk platter wait for I/O channel pause for disk rotation read data Close agreement between: results from queueing theory, simulation, and direct measurement (within 15%). The Software Process Requirements Definition System and Software design Programming and Unit Testing Integration and System Testing Operation and Maintenance Software Engineering Lecture 20 Coding Standards Tools for Debugging 1 Coding Standards Or How to Pound all of your odd-shaped programmers into a one size fits all hole I think there may be a bug in Joe’s Code - Please Fix func GreenEggsNHam(Not SamIAm, Green EggsNHam) foreach Green TryThem in SamIAm do EatThem(TryThem) = false NotInACarNotOnABus(EggsNHam) func NotInACarNotOnABus(Green EggsNHam) EatThem(EggsNHam) = true NotOnAPlane(EggsNHam) foreach NotLikeThem SamIAm of EggsNHam do if not EatThem(SamIAm) then NotInACarNotOnABus(SamIAm) IDoNotLikeThem(EggsNHam) Joe’s Code Following a Sane Coding Standard . . . func DepthFirstSearch(graph G, vertex v) foreach vertex w in G do Encountered(w) = false RecursiveDFS(v) func RecursiveDFS(vertex v) Encountered(v) = true PreVisit(v) foreach neighbor w of v do if not Encountered(w) then RecursiveDFS(w) PostVisit(v) What are Coding Standards Coding standards are guidelines for code style and documentation. The dream is that any developer familiar with the guidelines can work on any code that followed them. Standards range from a simple series of statements to involved documents. Areas Typically Covered Program Design Naming Conventions Formatting Conventions Documentation Possibly Even Licensing Why Have Coding Standards Greater consistency between developers Easier to develop and maintain Saves time and money Prime Directive Document every time you violate a standard. No standard is perfect for every application, but failure to comply with your standards requires a comment Ambler’s Law of Standards Industry Standards > organizational standards > project standards > no standards The more commonly accepted a standard the easier it is for team members to communicate Invent standards when necessary, but don’t waste time creating something that you won’t be able to use later. All languages have recommended coding standards available. It is well worth your effort to find and use industry standards Push for organizational standards whenever possible Good Coding Style Names Use full English descriptors Use mixed case to make names readable Use abbreviations sparingly and consistently Avoid long names Avoid leading/trailing underscores Documentation Document the purpose of every variable Document why something is done not just what Accessors use getVar() and setVar() functions on all class variable unless class is being used solely as a data structure (OOP) Member Functions Documentation What and why member function does what it does Parameters / return value How function modifies object Preconditions /Postconditions Concurrency issues Restrictions Internal Documentation Control Structures Why as well as what the code does Difficult or complex code Processing order Three Rules Coding standards needn’t be onerous - find a standard that works for your team. Standardize early - the effort to bring your old work into the standard will be too great otherwise. Encourage a culture where standards are followed. Examples of Coding Standards http://www.ambysoft.com/javaCodingStandards.html http://www.swtech.com/java/codestd/ http://ccs.hst.nasa.gov/ccspages/policies/standards/coding_standar ds.html http://www.scriptics.com/doc/styleGuide.pdf Software Engineering Lecture 21 Dependable Systems I Reliability Software Reliability Failure: Software does not deliver the service expected by the user (e.g., mistake in requirements) Fault: Programming or design error whereby the delivered system does not conform to specification Reliability: Probability of a failure occurring in operational use. Perceived reliability: Depends upon: user behavior set of inputs pain of failure Reliability Metrics • • • • • • Probability of failure on demand Rate of failure occurrence (failure intensity) Mean time between failures Availability (up time) Mean time to repair Distribution of failures Hypothetical example: Cars are safer than airplane in accidents (failures) per hour, but less safe in failures per mile. Reliability Metrics for Distributed Systems Traditional metrics are hard to apply in multi-component systems: • In a big network, at a given moment something will be giving trouble, but very few users will see it. • A system that has excellent average reliability may give terrible service to certain users. • There are so many components that system administrators rely on automatic reporting systems to identify problem areas. User Perception of Reliability 1. A personal computer that crashes frequently v. a machine that is out of service for two days. 2. A database system that crashes frequently but comes back quickly with no loss of data v. a system that fails once in three years but data has to be restored from backup. 3. A system that does not fail but has unpredictable periods when it runs very slowly. Cost of Improved Reliability $ Up time 99% 100% Will you spend your money on new functionality or improved reliability? Specification of System Reliability Example: ATM card reader Failure class Example Metric Permanent System fails to operate non-corrupting with any card -- reboot 1 per 1,000 days Transient System can not read non-corrupting an undamaged card 1 in 1,000 transactions Corrupting Never A pattern of transactions corrupts database Principles for Dependable Systems The human mind can encompass only limited complexity: => Comprehensibility => Simplicity => Partitioning of complexity Principles for Dependable Systems High-quality has to be built-in => Each stage of development must be done well => Testing and correction does not lead to quality => Changes should be incorporated into the structure Quality Management Processes Assumption: Good processes lead to good software The importance of routine: Standard terminology (requirements, specification, design, etc.) Software standards (naming conventions, etc.) Internal and external documentation Reporting procedures Quality Management Processes Change management: Source code management and version control Tracking of change requests and bug reports Procedures for changing requirements specifications, designs and other documentation Release control Design and Code Reviews • Colleagues review each other's work: can be applied to any stage of software development can be formal or informal • The developer provides colleagues with: documentation (e.g., specification or design), or code listing talks through the work while answering questions • Most effective when developer and reviewers prepare well Benefits of Design and Code Reviews Benefits: • • • • • Extra eyes spot mistakes, suggest improvements Colleagues share expertise; helps with training An occasion to tidy loose ends Incompatibilities between modules can be identified Helps scheduling and management control Fundamental requirements: • Senior team members must show leadership • Must be helpful, not threatening Process (Plan) Reviews Objectives: • To review progress against plan (formal or informal) • To adjust plan (schedule, team assignments, functionality, etc.) Impact on quality: Good quality systems usually result from plans that are demanding but realistic Good people like to be stretched and to work hard, but must not be pressed beyond their capabilities. Statistical Testing • Determine the operational profile of the software • Select or generate a profile of test data • Apply test data to system, record failure patterns • Compute statistical values of metrics under test conditions Statistical Testing Advantages: • • • Can test with very large numbers of transactions Can test with extreme cases (high loads, restarts, disruptions) Can repeat after system modifications Disadvantages: • • • Uncertainty in operational profile (unlikely inputs) Expensive Can never prove high reliability Example: Dartmouth Time Sharing (1980) A central computer serves the entire campus. Any failure is serious. Step 1. Gather data on every failure • 10 years of data in a simple data base • Every failure analyzed: hardware software (default) environment (e.g., power, air conditioning) human (e.g., operator error) Example: Dartmouth Time Sharing (1980) Step 2. Analyze the data. • Weekly, monthly, and annual statistics Number of failures and interruptions Mean time to repair • Graphs of trends by component, e.g., Failure rates of disk drives Hardware failures after power failures Crashes caused by software bugs in each module Example: Dartmouth Time Sharing (1980) Step 3. Invest resources where benefit will be maximum, e.g., • Orderly shut down after power failure • Priority order for software improvements • Changed procedures for operators • Replacement hardware Factors for Fault Free Software • Precise, unambiguous specification • Organization culture that expects quality • Approach to software design and implementation that hides complexity (e.g., structured design, object-oriented programming) • Use of software tools that restrict or detect errors (e.g., strongly typed languages, source control systems, debuggers) • Programming style that emphasizes simplicity, readability, and avoidance of dangerous constructs • Incremental validation Error Avoidance Risky programming constructs • • • • • • Pointers Dynamic memory allocation Floating-point numbers Parallelism Recursion Interrupts All are valuable in certain circumstances, but should be used with discretion Defensive Programming Murphy's Law: If anything can go wrong, it will. Defensive Programming: • Redundant code is incorporated to check system state after modifications • Implicit assumptions are tested explicitly Defensive Programming Examples • Use boolean variable not integer • Test i <= n not i = = n • Assertion checking • Build debugging code into program with a switch to display values at interfaces • Error checking codes in data, e.g., checksum or hash Some Notable Bugs • Built-in function in Fortran compiler (e0 = 0) • Japanese microcode for Honeywell DPS virtual memory • The microfilm plotter with the missing byte (1:1023) • The Sun 3 page fault that IBM paid to fix • Left handed rotation in the graphics package Good people work around problems. The best people track them down and fix them! Software Engineering Lecture 22 Dependable Systems II Validation and Verification Defensive Programming Murphy's Law: If anything can go wrong, it will. Defensive Programming: • Redundant code is incorporated to check system state after modifications • Implicit assumptions are tested explicitly Defensive Programming Examples • Use boolean variable not integer • Test i <= n not i = = n • Assertion checking • Build debugging code into program with a switch to display values at interfaces • Error checking codes in data, e.g., checksum or hash Terminology Fault avoidance Build systems with the objective of creating faultfree systems Fault tolerance Build systems that continue to operate when faults occur Fault detection (testing and validation) Detect faults before the system is put into operation. Fault Tolerance Basic Techniques: • After error continue with next transaction • Timers and timeout in networked systems • Error correcting codes in data • Bad block tables on disk drives • Forward and backward pointers Report all errors for quality control Fault Tolerance Backward Recovery: • Record system state at specific events (checkpoints). After failure, recreate state at last checkpoint. • Combine checkpoints with system log that allows transactions from last checkpoint to be repeated automatically. Fault Tolerance General Approach: • Failure detection • Damage assessment • Fault recovery • Fault repair N-version programming -- Execute independent implementation in parallel, compare results, accept the most probable. Validation and Verification Validation: Are we building the right product? Verification: Are we building the product right? In practice, it is sometimes difficult to distinguish between the two. That's not a bug. That's a feature! Cleanroom Software Development Software development process that aims to develop zero-defect software. • • • • • Formal specification Incremental development with customer input Constrained programming options Static verification Statistical testing It is always better to prevent defects than to remove them later. Example: The four color problem. Static and Dynamic Verification Static verification: Techniques of verification that do not include execution of the software. • May be manual or use computer tools. Dynamic verification • Testing the software with trial data. • Debugging to remove errors. Static Validation & Verification Carried out throughout the software development process. Validation & verification Requirements specification Design Program Static Verification: Program Inspections Program reviews whose objective is to detect faults • Code may be read or reviewed line by line. • 150 to 250 lines of code in 2 hour meeting. • Use checklist of common errors. • Requires team commitment, e.g., trained leaders So effective that it can replace unit testing Inspection Checklist: Common Errors Data faults: Initialization, constants, array bounds, character strings Control faults: Conditions, loop termination, compound statements, case statements Input/output faults: All inputs used; all outputs assigned a value Interface faults: Parameter numbers, types, and order; structures and shared memory Storage management faults: Modification of links, allocation and deallocation of memory Exceptions: Possible errors, error handlers Static Analysis Tools Program analyzers scan the source of a program for possible faults and anomalies (e.g., Lint for C programs). • Control flow: loops with multiple exit or entry points • Data use: Undeclared or uninitialized variables, unused variables, multiple assignments, array bounds • Interface faults: Parameter mismatches, non-use of functions results, uncalled procedures • Storage management: Unassigned pointers, pointer arithmetic Static Analysis Tools (continued) • Cross-reference table: Shows every use of a variable, procedure, object, etc. • Information flow analysis: Identifies input variables on which an output depends. • Path analysis: Identifies all possible paths through the program. Test Design Testing can never prove that a system is correct. It can only show that (a) a system is correct in a special case, or (b) that it has a fault. • The objective of testing is to find faults. • Testing is never comprehensive. • Testing is expensive. Testing and Debugging Testing is most effective if divided into stages: • Unit testing at various levels of granularity tests by the developer emphasis is on accuracy of actual code • System and sub-system testing uses trial data emphasis is on integration and interfaces • Acceptance testing uses real data in realistic situations emphasis is on meeting requirements Acceptance Testing Alpha Testing: Clients operate the system in a realistic but non-production environment Beta Testing: Clients operate the system in a carefully monitored production environment Parallel Testing: Clients operate new system alongside old production system with same data and compare results The Testing Process System and Acceptance Testing is a major part of a software project • It requires time on the schedule • It may require substantial investment in datasets, equipment, and test software. • Good testing requires good people! • Management and client reports are important parts of testing. What is the definition of "done"? Testing Strategies • Bottom-up testing. Each unit is tested with its own test environment. • Top-down testing. Large components are tested with dummy stubs. user interfaces work-flow client and management demonstrations • Stress testing. Tests the system at and beyond its limits. real-time systems transaction processing Test Cases Test cases are specific tests that are chosen because they are likely to find faults. Test cases are chosen to balance expense against chance of finding serious faults. • Cases chosen by the development team are effective in testing known vulnerable areas. • Cases chosen by experienced outsiders and clients will be effective in finding gaps left by the developers. • Cases chosen by inexperienced users will find other faults. Test Case Selection: Coverage of Inputs Objective is to test all classes of input • Classes of data -- major categories of transaction and data inputs. Cornell example: (undergraduate, graduate, transfer, ...) by (college, school, program, ...) by (standing) by (...) • Ranges of data -- typical values, extremes • Invalid data, reversals, and special cases. Test Case Selection: Program Objective is to test all functions of each computer program • Paths through the computer programs Program flow graph Check that every path is executed at least once • Dynamic program analyzers Count number of times each path is executed Highlight or color source code Can not be used with time critical software Program Flow Graph if-then-else loop-while Fixing Bugs • Isolate the bug Intermittent --> repeatable Complex example --> simple example • Understand the bug Root cause Dependencies Structural interactions • Fix the bug Design changes Documentation changes Code changes Moving the Bugs Around Fixing bugs is an error-prone process! • When you fix a bug, fix its environment • Bug fixes need static and dynamic testing • Repeat all tests that have the slightest relevance (regression testing) Bugs have a habit of returning! • When a bug is fixed, add the failure case to the test suite for the future. Regression Testing Applied to modified software to provide confidence that modifications behave as intended and do not adversely affect the behavior of unmodified code. • Basic technique is to repeat entire testing process after every change, however small. Real Time Software Development Testing and debugging need special tools and environments • Debuggers, etc., can not be used to test real time performance • Simulation of environment may be needed to test interfaces -- e.g., adjustable clock speed • General purpose tools may not be available Software Engineering for Real Time The special characteristics of real time computing require extra attention to good software engineering principles: • Requirements analysis and specification • Development of tools • Modular design • Exhaustive testing Heroic programming will fail! Some Notable Bugs • Built-in function in Fortran compiler (e0 = 0) • Japanese microcode for Honeywell DPS virtual memory • The microfilm plotter with the missing byte (1:1023) • The Sun 3 page fault that IBM paid to fix • Left handed rotation in the graphics package Good people work around problems. The best people track them down and fix them! Staying Out of Prison in the Information Economy Lecture Caveats I am not a lawyer and do not have any formal legal training This lecture is made up of my observations of the legal system to make you aware of important issues concentrating on an information technology workplace Cardinal Rule: Be aware of the law, but always consult an attorney if/when you become involved with it. Law Caveats Some people read the text of the law and think they know it. Things are never so easy. If you have questions ask a lawyer. Others ignore the law relying on corporate lawyers in case something goes wrong. This is not a good idea. As in any other system, catching problems in the design phase is always better than in the debugging phase. Talk Overview Life for Lawyers Vs. Life for Engineers Patents, copyright, trademarks, trade secrets reviewed Defamation ISP Liability Privacy Jurisdiction Issues Life for Computer Professionals Binary Problem solutions either work or not. Little room for gray areas. Physical and mathematical laws ultimate authority when disputes arise Guiding Philosophy - “Tell me what you need and I will create a system with appropriate trade-offs at least cost to solve your problem.” Life for Lawyers Gray Effort and intent often matter as much as results Supreme court ultimate authority when disputes arise Guiding Philosophy - “Laws are passed based on how society should run - even if enforcement and legal interpretation issues have yet to be nailed down.” When Worlds Collide . . . Legal community always behind the technology curve Lawyers and politicians typically have poor technical backgrounds As a result, analogies often made between new technological paradigms and old world systems - some more easily defended than others. Different interpretations would result in different laws Patents Embodiment of a specific methodology Competing products must use different method for achieving same task to avoid payments Definite lifespan beyond which patent information freely available for use by the public Copyright Specific work Automatically held when work is created, but easier to defend if it is registered Definite lifetime beyond which the work is freely available to the public Trademark Specific name or phrase Generic terms cannot be trademarked Trademarks can be lost if they are not defended Lost trademarks: aspirin, kleenex Held Trademarks: Coke, Pepsi Trade Secrets Does not expire - as long as it is kept secret Competitors may not use secrets obtained through extraordinary means Example: Walled chemical plant layout learned through helicopter use Defamation Publishing damaging statements you cannot prove about others The publisher and author are both liable Slander is a less serious, but similar, crime where damaging statements that cannot be proven are made in a public arena Bally Total Fitness Vs. Faber A “Bally Sucks” web site was created by Faber complaining about Bally fitness centers The trademarked Bally seal was placed on the site overlaid with the words “Sucks” Bally sued Faber making claims of trademark infringement, dilution, and unfair competition. Bally Case Decision No trademark infringement - little possibility of confusion No dilution - the defendant did not sell a competing product and did not convey confusion about the author’s identity No dilution (lessening ability of the plaintiff’s mark to identify its goods and services) since defendant was not marketing a competing product Incidentally - no slander, negative opinions protected under the first amendment ISP Liability What is an Internet Service Provider Like? Phone Company: Route information flows between individuals Newspaper: Package content for distribution in a public forum Answer determines ISP’s legal liability The rules have been in a constant state of flux in recent years Ancient History (~Decade Ago) Defamatory posting on Prodigy (Stratton Oakmont Vs. Prodigy Services 1995) Prodigy a large ISP Claimed to be “family friendly”. Prodigy advertised that internal newsgroups monitored for bad/inappropriate language Role of a publisher - hence, Prodigy like a newspaper CompuServe did not monitor users activity - like a telephone company (Cubby Inc. Vs. CompuServe Inc. in 1991) Modern Era Communications Decency Act ISP may monitor user activity (according to policy) If statement to the effect that ISP does not take responsibility for user traffic in place then no ISP liability, BUT Area for complaints must be available Complaint response must happen in a timely fashion DMCA Digital Millennium Copyright Act If a copyright infringement is claimed a web site must be taken down (however tenuous the claim may be) Web site can only be reinstated after an appeals process. Near Future? . . . European Computer Crime Treaty may be created by the end of this year ISP’s may be required to monitor user traffic with a 40 day data-log. ISP’s not explicitly exempt from liability Hacker/Security Tools Illegal Citizens must provide passwords for data seized by police Privacy in the Workplace Test for employers/employees - “Do you have a reasonable expectation of privacy?” A case can be made that private e-mail on business machines still private, but this is not the law Work-related material on business machines is definitely not private Privacy in E-mail Legally, e-mail is like a postal letter Expectation of privacy in transit Mail loses its special protected status once it leaves the letter carrier's grasp For e-mail, Expectation of privacy while signal travels over Internet E-mail loses its protected status at the mail server whether you have read it or not Spam and Address Spoofing Matthew Seidl v. Greentree Mortgage Co. (1998) Greentree hired third party to send mass e-mail to potential customers (spam) Return address spoofed to read nobody@localhost.com (an actual address) Over 7,000 complaints sent to nobody resulting in denial of service for 3 days Libel case dismissed since third party was a contractor. Likely that third party would, in fact, be vulnerable to a lawsuit. Business E-mail Electronic Communications Privacy Act (1986) says all business communication belongs to that business Deleting e-mail can be ruled spoliation (intentionally destroying company records) Archive worthless if it cannot be indexed effectively (in effect, saving everything can be equivalent to saving nothing) What about Privacy at Home? A lot of public information is considered private. An increasing amount of public information available on the Internet Reverse phone lookups Campaign Contributions Housing prices (Thwarted) Driver’s license information and photographs Data Collection Data collection has few boundaries in U.S. Check privacy policy (can change!!) EU Safe Harbor agreement may change things in the future (TRUSTe web site privacy seal program) Jurisdiction “The Internet has no boundaries” Is that really true? If you break a law in Finland, but you were on the Internet in the United States, what happens to you? What if you are in California and you break a law in Minnesota? E-Commerce Big Questions Did you sell an illegal item to a resident of community X? Did you try to stop the flow of illegal sales into X? An easy example of where this might come up is found in the on-line pornography boom. Obscene or Offensive? Indecent speech and offensive speech protected under the 1st Amendment Obscene speech is not But what is obscene speech? Miller Test for Obscenity (1) Whether “the average person applying contemporary community standards”, would find that the work, taken as a whole, appeals primarily to prurient interest. (2) Whether the work depicts or describes, in a patently offensive way, sexual conduct specifically defined by applicable state law. (3) Whether the work, taken as a whole, lacks serious literary, artistic, political, or scientific value. Federal Court System 94 US District Courts (89 in the 50 states) 13 Judicial Circuits, each with a court of appeals Supreme Court ultimate appellate court Jurisdiction can be a determining factor in case outcomes US V. Thomas (1994) Mr. And Mrs. Thomas ran a pornographic BBS in California State officer paid a membership fee and downloaded pornography in Tennessee Couple tried in Federal court in Tennessee and lost their case International Jurisdiction Extradition over civil suits unlikely Big Question #1: Do you have assets in the country in question? Big Question #2: Will you ever try to enter country X? Godfrey Vs. Dolenga Dolenga was a Cornell Biochemistry Master’s student from British Columbia Godfrey, a nuclear physicist from London, made anti-Canadian remarks in a newsgroup Dolenga responded by flaming Godfrey Godfrey notified Cornell of the offensive remarks, but they were not removed (First Amendment) Godfrey filed defamation suits against Dolenga and Cornell in Britain (one of at least seven such cases) Dolenga Did Not Defend Himself . . . Dolenga was found guilty by default in English court BUT - Dolenga does not have assets in England and it is unlikely that American courts will enforce the British judgement. Cornell Did Defend Itself Cornell has assets in England (the Cornell abroad program) The suit was for roughly 80,000 pounds. The University could have settled, but chose to take the case to court The suit was brought to a successful conclusion (for Cornell) Lessons to be taken away from this . . . Conclusions . . . The law is constantly changing and never as simple as it seems You should try to be familiar with the law to protect yourself (corporate lawyers are like a fire department, not like a seeing eye dog) Even so, you DO need the help of someone with formal training when dealing with legal issues Software Engineering Lecture 25 Management III Managing People Administration Return of laptops and wireless cards -> Dates for return will be announced on "Notices" -> All equipment must be returned before the examination. Bring the receipt to the exam. -> If an extension granted, (e.g., independent research) must return and be issued again If any repairs needed, please swap for replacement since warranty runs out on December 15. Administration Early examination December 7, 10:00 to 11:30, Upson 5130 Send email to rosemary@cs.cornell.edu if you plan to take the early examination, by December 5 All laptops and wireless cards must be returned before the examination Managing People Theoretical: • Organizational behavior • Industrial psychology Group behavior • Cognitive fundamentals Economic motivation Maslow's Hierarchy of Needs Self-realization needs Esteem needs Social needs Safety needs Physiological needs Software Engineering Basics • Professional staff are the major cost of software • Professional staff vary greatly in productivity => Ability => Education and training => Motivation => Interaction with colleagues and leaders => Work environment • People are productive when happy and happy when productive Software is Built by Teams • Best size for a team is 3 to 8 people • Team members may include: developers (from trainee to expert) domain experts graphic or interface designers software librarians testers • Teams must have: administrative leadership (manager) technical leadership Group Working 20% non-productive 30% working alone 50% interaction with others Communication • Informal Kitchen, smokers' doorway, after work, etc. Walkabout (tours) Ad hoc meetings • Staff meetings (non-technical) Example: Tektronics • Technical meetings Facilitation Record of decisions Administrative Leader (Manager) • Personnel Assigning tasks Hiring, promoting, etc. • Resources Budgets Space, facilities Equipment • Project management Relationships with other teams and clients Project plan and schedule Hiring Criteria Productivity is a combination of: • • • • Analytic ability Verbal ability and communication skills Education Application domain knowledge • Adaptability and inquisitiveness • Personality and attitude • • Platform experience Programming language experience Staff Retention • Technically interesting work up to date hardware and software opportunities to learn and experiment • Feeling of appreciation management recognition money and promotion • Working conditions space, light, noise, parking flexibility • Organizational dynamics Firmness Managers must be firm when needed: • Assignment of tasks must be equitable and open; everybody will have to tackle some of the dreary tasks • Carrots are better than sticks, but poor performance must be addressed. • Nobody is indispensable; nobody should be allowed to think that they are indispensable Technical Challenges • Canceling projects Example: the Andrew window manager • Changes of environment Example: the World Wide Web • Technical tinkering v. needed re-engineering Turning a Group Around To turn a weak group into a strong one is the greatest challenge of leadership • The art of the possible • Promotion of the best over the old leaders • Using opportunities to reorganize • Resignations and terminations • Respect people who try, yet refuse to accept problem areas Brutal and abrupt rarely equals persistent and firm How to be Led As a junior member of a team, what can you do to make it productive? Software Engineering Lecture 26 Risk in Software Engineering Failures and Risks Software development projects can fail in many ways: Bad software engineering • Late, over budget • Lack of function, full of bugs, bad performance Changing circumstances • Changing markets • Better alternatives • Changes of management The biggest single source of problems is poor understanding of requirements Managing Risk Manage projects to avoid risk: • Open and visible software process => Avoid surprises • Continual review of requirements • Willingness to modify or cancel projects Canceling a Project Example: Andrew Window Manager (wm) • Technically superior to X (MIT's Athena project) in 1986 but ... Digital Equipment Corporation turning X into a product with massive support nobody ready to support wm • Therefore wm cancelled in 1986, Andrew user interface and applications ported to X Failure to Cancel a Project Example: University F developed a novel programming language. • From 1985 to 1989, this was a promising language for simple programming of window-based applications • By 1990, clearly not gaining acceptance beyond University F • But development continued for many more years (about $500K) Not cancelled because ... Too Big to Cancel! Example: University A has antiquated administrative systems. Senior management decides to replace them all with commercial packages from X. The timetable and budget are hopelessly optimistic. • Staff get dispirited. • The Chief Information Officer finds another job. • A new Chief Information Officer is appointed. What should she do? We are doing it the Wrong Way! Example: University B has a (big) joint project with Company Y to develop a new computer operating system. After two years work, a junior software developer persuades the university leader that the technical approach is wrong. • What should the university do? • What should the company do? How to Stop Gracefully • It is harder to cancel a project than to start it. • It is harder to withdraw a service than introduce it. Considerations • The proponents of the system must now reverse their public stance. => Management of expectations • Users of the service need a migration strategy. • Technical staff must have a graceful path forward. Time to Complete a Software Project Large software projects typically take at least two years from start to finish • Formative phase -- changes of plan are easy to accommodate • Implementation phase -- fundamental changes are almost impossible Yet many things can change in two years. A Sense of Urgency Example: A not-for-profit corporation is developing a system for a government organization. • By 1996 all research had been completed and the system demonstrated successfully with real users. • In 2000, the system is still not in full production Reasons: => Incremental improvements to the software => Repeated requests for more functionality => Reluctance to reorganize clerical staff Nobody had a sense of urgency Overtaken by Events Example: University C has a project to develop a digital library system, with funds from Company Z , private foundations and the government. • By 1993 an extensive system is running at the university and Z is marketing the technology to its customers. • By 1994 it is clear that web browsers and web formats (though technically weak) are becoming widely adopted. => What should the university do? => What should the company do? Changing Requirements and Design Example: The CNRI Handle System -- a high performance, distributed system to map names to resources (1994-99). • • • • In 1994 only web browser was Mosaic In 1994 Java did not exists In 1994 mirroring and caching utilities were not available In 1994 commercial interest was limited Design decisions made in 1994 had to be changed. Software was rewritten and greatly improved in 1998/9. If a job's worth doing, it's worth doing twice! Changes of Leadership Many projects are wasted because of management changes Example: In 1988, Company W gave University D $1,000,000 to port a new operating system to its personal computers. The work was well done, on time. • Company W changed its president and senior technical staff during the project. The work was wasted. • A decade later and several presidents later, Company W is releasing a modern version of the same operating system. A graduate student from University D is now Senior Vice President of Company W! Client Oversight When work is out-sourced, the client must be vigilant. Example: Company G was the world's leader in software for optimization (e.g., linear and integer programming). G had implemented several packages for various manufacturers. • An operating system Company H contracted with G to develop an optimization package for its new operating system. • The package was late, performed badly and disliked by customers. What went wrong? What can we learn? Too Difficult! Example: A development team at University E was given government funds to build a high-performance gateway from protocol x to protocol y. • A promising young developer was hired and assigned to this task • The project was too difficult for him, but he hid his problems for many months. • The project produced nothing of value. What can we learn from this experience? Engineering and Marketing Corporate engineering & marketing divisions at cross purposes: Examples: • Xerox's Palo Alto Research Center pioneered window managers, Ethernet, graphical user interfaces, font managers, etc, => Apple, Adobe, Digital, etc. brought them to the market • IBM would not bring its first Unix workstation to the market until the software had been largely rewritten => Sun's early workstations were unreliable but sold well Senior Management Dynamics • Directors and shareholders appoint the President => The President does not want to admit failures • The President appoints the Chief Information Officer => The CIO does not want to admit failures • The CIO appoints the computing managers => The computing mangers do not want to admit failures • The computing managers appoint the developers => The developers do not want to admit failure Everybody pretends that things are going well Senior Management Dynamics At last the troubles can not be hidden ... • Directors and shareholders try to blame the President • The President tries to blame the Chief Information Officer • The CIO tries to blame the computing managers (and grumbles about the President) • The computing managers try to blame the developers (and grumble about the CIO) • The developers grumble about their managers What can we do better? Sobering Thoughts • Major computing projects are very complex. Inevitably there are delays and failures. • Few organizations know how to manage risk & uncertainty. • The best CIO's => Manage to minimize risk => Have the confidence of their staff who keep them truthfully informed => Have the self-confidence to keep their seniors truthfully informed Software Engineering Lecture 27 Software Engineering as Engineering The Y2K Problem: Saving Memory • In 1967 memory cost $1 per byte The Air Force used single digit dates • If 2-digit dates saved 1% of memory... savings over 20 years $16 to $24 million per gigabyte The Y2K Problem: Saving Memory • By 1980s, memory was much cheaper, but 2-digit dates were standard. Why incur the cost of changing standards? 1970 The mortgage industry 1990 The Social Security Industry moved towards 4-digit dates • On January 1, 2000 2-digit dates stopped working! Where's the Problem? • A simple bug: dates of the form 19xx have been encoded xx • A simple fix: find every occurrence of the bug modify the code recompile Where's the problem? Find Every Occurrence ... • What computers do we use? data processing control embedded systems personal devices • What programs do they run? in-house development packages and libraries firmware, microcode, hardware Who wrote this program? Where is the source code? Where's the Problem? Computers fail everyday. What's special about this bug? • What if they all fail at the same time? • What if we lose telephone, electricity, radio, etc.? • Traffic signals, elevators, The greatest worry was uncertainty. Social Consequences Worry creates its own problems: • Wal-Mart forecast lower profits in Q1 2000 • Legislation to limit law suits • Opportunities for computer fraud and sabotage • Trading partners Organizational Procedures • Ostrich => do nothing => buy insurance • Bureaucratic => fill in forms that programs are compliant • Subcontract => hire Y2K specialists • Do it yourself => in-house computing department Y2K Validation Request from Library of Congress to confirm that our code is Y2K compliant: Our code is fine .... but it depends on ... which depends on ... Yes. Our code is fine. Request from DARPA to confirm that our code is Y2K compliant: It's been validated by another part of the US government Thank you! Technical Strategies • Replace noncompliant applications with compliant ones (e.g., new versions of packages) • Repair noncompliant applications (e.g., in-house applications) • Terminate noncompliant programs on an as-needed basis • Mask the data exchange between applications • Object code interception New Bugs If it's not broke don't fix it. • 10 billion lines of code checked (often automatically) • 10 million new bugs introduced accidentally • ?? security holes, errors, etc. introduced accidentally or deliberately Is all the Money Going to Y2K? Y2K as a great excuse to have the computing budget increased: • Upgrade the operating system • Replace the old package • Sell something to your customers What boss will turn turn a request for Y2K funds? What systems administrators will not install Y2K upgrades? Profiteering • Buy gold, wood stoves, bottled water • Y2K specialists • Pundits, consultants, writers • Religious cranks Final Thoughts on Y2K We create computer systems that are more complex than our understanding of them: • We over estimate our ability to validate systems • We under estimate our ability to adapt and respond Software engineering usually thinks of systems as independent. Will the long-term benefit of the Y2K problem be a greater understanding of how software systems interact with each other and with our social systems? The Need for Software Engineering Software as a product: => Awkward to use => Full of errors => No chance to try it out => No guarantees Not much of a product What is Engineering? What is Engineering? The profession of: ... creating cost-effective solutions ... ... to practical problems ... ... by applying scientific knowledge ... ... and established practices ... ... building things ... and taking responsibility for them! Crafts, Science, Engineering Science Production Commercial Craft From: Shaw and Garlan Professional Engineering Crafts, Science, Engineering algorithms data structures Science Production Craft From: Shaw and Garlan software development methodologies Commercial compiler construction Professional Engineering Software Engineering as Engineering? • Part craft -- part engineering • Embryonic scientific basis • Evolving body of knowledge • Too much flux for the apparatus of a profession (e.g., accreditation) Example: Texas and the ACM The End • Good process leads to good software: the limits of heroic efforts • Minimize risks: visible process function v. time v. cost • The importance of people Requirements, requirements, requirements!