Front cover Lotus Instant Messaging andd Web Conferencing ing Features and benefits of running Sametime 3.1 on Solaris 9 Installation and configuration hints and tips Performance-tuning recommendations William Tworek Michael Lee ibm.com/redbooks Redpaper International Technical Support Organization Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 March 2004 Note: Before using this information and the product it supports, read the information in “Notices” on page v. First Edition (March 2004) This edition applies to Lotus Instant Messaging and Web Conference (Sametime), Release 3.1 This document created or updated on March 12, 2004. © Copyright International Business Machines Corporation 2004. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .v Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii The team that wrote this Redpaper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii Chapter 1. Introduction to Sametime 3.1 on Solaris 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Overview of Lotus Sametime 3.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Features and Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.2 Sametime architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.3 Sametime terms and definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.4 Sametime 3.1 requirements on Solaris . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.5 Other requirements for Sametime 3.1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Introduction to Solaris 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Why Solaris? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 Highlights of Solaris . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.3 System Requirements for Solaris 9. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.4 For more information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 2 2 4 5 5 5 5 6 7 7 Chapter 2. Installing and running Sametime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.1 Preparation for installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.1.1 Basic requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.1.2 Create group and user accounts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.3 Preinstallation considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2 Install and Setup Domino . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.3 Installation and set up of Sametime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.3.1 Pre-install tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.3.2 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.3.3 Integrating LDAP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.3.4 Verifying server functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.4 Extending Sametime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.4.1 Enabling SSL for Sametime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.5 Maintaining your Sametime environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.5.1 Uninstalling Sametime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Chapter 3. Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Tuning methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Step 1: Observe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.2 Step 2: Hypothesize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.3 Step 3: Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Monitoring performance on Solaris . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 The prstat command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 The vmstat command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 The mpstat command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.4 The iostat command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.5 The netstat command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.6 The ps command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . © Copyright IBM Corp. 2004. All rights reserved. 39 40 40 40 41 41 42 43 44 46 47 48 iii 3.2.7 The pstack command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.8 The SE ToolKit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.9 Solaris internals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Lotus Domino 6 performance monitoring options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Tuning Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Solaris kernel tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Solaris file system tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 Network tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.4 Other tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.5 Domino Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Sample testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.1 Performance load description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.2 Performance testing and tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.3 Performance results after tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.4 Testing conclusions: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 49 49 49 49 51 52 54 56 57 58 61 62 63 66 68 Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces. © Copyright IBM Corp. 2004. All rights reserved. v Trademarks The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: Eserver® Eserver® Redbooks (logo) Eserver™ ibm.com® ™ Domino® IBM® Lotus Notes® Lotus® Notes® Redbooks™ Sametime® Tivoli® The following terms are trademarks of other companies: Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Other company, product, and service names may be trademarks or service marks of others. vi Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Preface IBM® Lotus® Instant Messaging and Web Conferencing (Sametime)™ 3.1,with more than 9 million users, is the market-leading instant messaging and Web conferencing solution for business. Sametime provides presence awareness, instant messaging, and Web conferencing. This allows it to help speed communication, reduce travel expenses, and produce results. Using Sametime, people can (1) discover whether others are available to communicate and collaborate, (2) correspond quickly through instant messaging, and (3) present information and improve customer satisfaction through Web conferencing. Today's business teams consist of colleagues, suppliers, partners, and customers that are dispersed across campuses, across countries, and around the world. Lotus Sametime provides the flexible, efficient, and instantaneous workforce communication that these conditions demand. The Solaris 9 Operating Environment is the foundation for Sun systems, from smaller departmental servers to massive, clustered servers with more than 100 CPUs. Designed for multiprocessing and 64-bit computing, Solaris software delivers a consistent computing environment that scales to handle heavy traffic, huge data sets, and CPU-intensive problems. By minimizing planned and unplanned downtime, reducing administration errors, and simplifying troubleshooting, the Solaris Operating Environment (OE) keeps mission-critical applications available while ensuring high-speed and reliable access to data. The Solaris 9 OE is scalable, available, manageable, and secure. This IBM Redpaper describes how to leverage the Sun Solaris 9 environment as a stable and scalable platform for Sametime 3.1. Topics covered in this Redpaper are: Basic installation and setup guidelines for Sametime 3.1 on Solaris 9 General Solaris performance monitoring techniques Performance-tuning suggestions for optimizing a Sametime 3.1 install on Solaris 9 The results of basic performance benchmarks demonstrating the scalability of Sametime 3.1 on Solaris 9 Because Sametime 3.1 executes on top of a basic Domino™ 6 server, readers should be somewhat familiar with the Lotus Domino platform. Those new to this platform may want to first review the following IBM Redbooks and IBM Redpapers: Lotus Domino R5 for Sun Solaris 8, SG24-5969 http://www.redbooks.ibm.com/abstracts/sg245969.html Upgrading to Lotus Notes and Domino 6, SG24-6889 http://www.redbooks.ibm.com/abstracts/sg246889.html Upgrading to Lotus Domino 6: The Performance Benefits, REDP-3634 http://www.redbooks.ibm.com/abstracts/redp3634.html © Copyright IBM Corp. 2004. All rights reserved. vii The team that wrote this Redpaper Michael Lee (Lead Author) is a Software Engineer with the Lotus® Technical Support organization, based out of Atlanta, Georgia. He primarily supports Lotus technologies running on UNIX® platforms. William Tworek (Project Leader) is a Project Leader with the International Technical Support Organization, working out of Westford, Massachusetts. He provides management and technical leadership for projects that produce Redbooks™ on various topics involving IBM and Lotus Software technologies. Prior to joining the ITSO, he was an IT Architect in the consulting industry working for Andersen Consulting/Accenture, followed by work at IBM Software Services for Lotus. His areas of expertise include collaborative technologies and business portals, system integration, and systems infrastructure design. The authors would also like to thank the following people for their contributions to this project: IBM Lotus Software contributors: Charles Smith, Dave Curley, Mark Phillips, Roger Eames, ZengYu Du, Jeff Miller, and Ben Caldwell SUN Microsystems Contributors: Bob Somers and Craig Swain IBM International Technical Support Organization contributors: John Bergland and Maragaret Ticknor Become a published author Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You will team with IBM technical professionals, Business Partners, and/or customers. Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you'll develop a network of contacts in IBM development labs and increase your productivity and marketability. Find out more about the residency program, browse the residency index, and apply online at: http://ibm.com/redbooks/residencies.html Comments welcome Your comments are important to us! We want our Redpapers to be as helpful as possible. Send us your comments about this Redpaper or other Redbooks in one of the following ways: Use the online Contact us review redbook form found at: http://ibm.com/redbooks viii Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Send your comments in an Internet note to: http://ibm.com/redbook@us.ibm.com® Preface ix x Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 1 Chapter 1. Introduction to Sametime 3.1 on Solaris 9 Lotus Sametime, now known as IBM Lotus Instant Messenging and Web Conferencing, is the industry leader in real-time collaboration software. Reasons for its success are based on its ease of integration, scalability, and flexibility. With Sametime, it is possible to provide your organization with the ability to interact in real-time without regard to distance. Many leading IT research groups believe that some 70 percent of enterprises will use instant messaging technology in some form in 2003. Thus, with the integration of real-time technology becoming ubiquitous, real-time collaboration is no longer an option for many businesses but a competitive necessity. This chapter introduces this Lotus real-time collaboration solution as it is available on the Sun Solaris 9 platform. It also provides an introduction to Sun Solaris 9 itself. Those already familiar with both Lotus Sametime and the Sun Solaris platform may want to skip directly to the next chapter in this Redpaper. © Copyright IBM Corp. 2004. All rights reserved. 1 1.1 Overview of Lotus Sametime 3.1 Lotus Instant Messaging and Web Conferencing (Sametime) is the IBM offering for real-time collaboration in the office or on the road. As a critical component of the Lotus Workplace, Sametime helps organizations, their customers, and their business partners be better coordinated, better informed, and more agile. Sametime offers the following functions: Presence awareness: This key functionality is often overlooked because of its basic nature. But this feature is an extensive component of the collaboration facility. It can be used in applications and virtual rooms to speed collaboration, provide status information, be extended to other wholly separate instant messaging systems, or even be available from a wireless device. Instant messaging: This property probably is what comes to mind first when Sametime is mentioned. Users can communicate through text or audio/visual components and send files. Web conferencing: This increases the range and power of real-time collaboration. One can schedule meetings, use the whiteboard facility, make presentations, and share applications and documents. 1.1.1 Features and Benefits Sametime features and benefits include the following: Extensibility and ubiquity: Integration with Notes/Domino already exists, along with the ability to integrate with other applications through its extensive toolkit. Security: Data is encrypted to protect meeting content and chat transmissions. Open gateway: Users from external communities that are not necessarily Sametime-based can be added to awareness lists and communicate with Sametime users securely through support of SIP (Session Initiation Protocol) and SIMPLE (SIP for Instant Messaging and Presence Leveraging Extensions). Scalability: A single server has a definite limit. However, you can increase the number of users being served through Sametime communities and meetings without a tangible limit by using a variety of multi-server solutions made available through the software. Directory support: Sametime supports Domino and LDAP directories. Market support: Sametime is the market leader in real-time collaboration. Management tools: The Sametime environment can be managed through extensive functions that include statistics, tuning, debugging, logging, security, and broadcasting. Client support: A native windows client and a Java™-based client are provided. Browser support: Sametime supports the Internet Explorer and Netscape browsers. Proxy support: Sametime supports reverse proxies. 1.1.2 Sametime architecture Sametime is made up of three basic components. Community services: This component encompasses awareness, chat, instant meetings and instant messaging. Essentially this provides the opportunity to interact and make communication as easy as a chance encounter. Meeting services: This component offers a broad range of collaboration tools that includes the whiteboard, presentation facilities, and provisions for sharing programs and 2 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 documents. It also covers the scheduling and meeting maintenance. This service greatly expands the ability to explicate and demonstrate concepts and data by providing all the features necessary to substitute for an in-person meeting. Toolkits: This component provides API/toolkits that enable the integration of real-time capability into separate applications. Users can do this through HTML (Sametime Links), COM, C++, or Java. Sametime Community Services communication Sametime Meeting Services is built on to run on top of the Domino HTTP engine. Thus, the communication architecture being implemented is familiar for many administrators. Community Services is usually something new and the various pieces can be confusing. It’s important to understand how Sametime communicates information so we can better understand its components. Basic Sametime communication is depicted in Figure 1-1. Figure 1-1 Basic Sametime communication A server application is defined as an add-on or enhancement that connects to the server and declares what services are being provided. These services can be developed through the Sametime toolkit. Sametime natively supplies these services: Who Is Online, also known as a Buddy List Authentication Resolution, which resolves a full name to a list of IDs Directory browsing N-way chat The server manages entities such as users, server applications, multiplexers, and notifications. The multiplexer limits the amount of I/O and I/O maintenance required by the server. This can be a standalone machine or machines that do all the connection initialization, polling, termination, and even protocol translation. Probably most importantly, it also can (1) distribute data from a single source to a number of recipients and (2) receive data from multiple sources and concentrate the data into a single stream. Thus, you can use a number of multiplexers to relieve congestion that can occur from large I/O requirements. Chapter 1. Introduction to Sametime 3.1 on Solaris 9 3 Sametime Processing When you configure or manage it, Sametime only needs a user population from your environment. This population can exist in an Domino directory or an LDAP directory. Sametime provides a structure that users can use to be aware of the people in their organization(s) and then use to communicate and collaborate without being limited by the constraints imposed by distance. Similarly, the amount of data that Sametime needs from your organization is limited. The easiest and most flexible use of Sametime will be dependent upon directory information already existing in your environment. If you think of Sametime as a communication engine, the only fuel you need is your users. 1.1.3 Sametime terms and definitions The following key terms and definitions are important when working with Sametime: Sametime community: This is similar to the idea of a Domino domain in that it groups servers for a set of users. However, the Sametime community groups servers that specifically provide Sametime services. Whiteboard: This is similar in presentation to a slideshow. The whiteboard is a tool that allows for interactive presentation. One participant can control the presentation, while the other participants can view and annotate the image. Of course, this is all done remotely. Application sharing: This function is otherwise known as screen sharing. It gives multiple remote users the ability to work with a single application on a single user’s computer. Chat: A chat session can consist of two (or more) users exchanging instant messages Instant Message: An instant message is text-based data to be presented immediately to a client or clients Awareness: Awareness is allso known as presence. This function provides users with the ability to detect when other users are online and to determine their status. Instant meeting: Instant meetings are initiated from the contact list and launched in the Sametime Meeting Room client on a user's machine. The Sametime Meeting Room client contains collaborative components that support screen-sharing, whiteboard, send Web page, polling, chat, and audio/video collaborative activities. Audio/Visual: Sametime Audio/Video Services support interactive IP audio and video capabilities. They also enable clients with the appropriate hardware (sound card, microphone, speakers, and camera) to transmit and receive real-time audio and video during a Sametime meeting. SIP/SIMPLE: Session Initiation Protocol (SIP, RFC 3261) is a request response protocol for initiating an interactive user session that involves elements such as video, voice, chat, and more. SIMPLE stands for SIP for Instant Messaging and Presence Leveraging Extensions. It is an open standard that extends SIP to provide a common means of providing online presence and instant messages. Meetings: A meeting can be either scheduled or instant. A meeting provides a place for multiple users to gather and use the full range of Sametime collaborative features. These features include screen sharing, whiteboard, question and answer polling, sending Web pages, Meeting Room chat, and audio and video capabilities. Print Capture: This utility provides a means of creating a whiteboard file separate from a conversion server. The Sametime Print Capture operates much like a printer driver and enables end users to print output from any Windows® application to the file format required by the whiteboard. 4 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Conversion Services: This server provides the automated capability to convert a file to be used in a meeting. Record and Playback: This enables users to record meetings for future reference. The meeting recording is stored in a .RAP file that a future authorized user can play. Sametime Links: This toolkit allows Web developers to Sametime-enable their Web pages and applications with presence, chat, meeting capabilities, and more. 1.1.4 Sametime 3.1 requirements on Solaris The following requirements exist for Sametime when it is used on the Sun Solaris platform. Hardware: Sun Sparc architecture Memory: 1 GB RAM recommended; 512 MB RAM minimum Disk space: 1 GB recommended; 512 MB minimum Operating system: Solaris 8 or Solaris 9 (in 64bit mode) Domino version: Domino 6.0.2 CF1 with the following components: – Java Runtime Environment 1.1.4 or higher (for installation) – X-Windows-based GUI for installation purposes (i.e. CDE) – Root access to the server for installation purposes 1.1.5 Other requirements for Sametime 3.1 As will be described in additional detail later in this paper, Lotus Sametime is built on top of the Lotus Domino platform. Therefore, the availability of a workstation running the Lotus Notes® Administration client is preferrable to fully adminster and configure Sametime. Many administrative tasks that require access to Domino/Notes databases can be performed from the web browser interface of Domino. However, using the Lotus Notes client can simplify these tasks. 1.2 Introduction to Solaris 9 As the foundation for Sun systems, the Solaris Operating Environment enables an IT organization to deliver on the promise of massive scale, continuous real-time computing, and secure systems—all the while increasing service levels, reducing risk, and decreasing costs. With Solaris 9 software, you can manage resources efficiently and provide a higher level of service. Solaris 9 helps reduce complexity by providing easy-to-use interfaces and comprehensive administration tools, all designed to help lower your IT risk. Finally, the Solaris 9 Operating Environment helps you minimize costs and lower the total cost of ownership by optimizing resources and enabling them to scale to meet demand. 1.2.1 Why Solaris? Sun offers a wide range of servers from a miniature single CPU system to the ultrahigh-end Sunfire 15k with 106 CPU’s. Additionally, Domino benchmark testing has demonstrated some of impressive response times and capacity available on the Solaris system. Some reasons for choosing Solaris might be: High rank for performance History of stability Extremely large feature set and tools to support enterprise computing Strong ties to Java Chapter 1. Introduction to Sametime 3.1 on Solaris 9 5 More details on Solaris as compared to other UNIX OSs as a platform for Domino/Sametime can be found in the Lotus Developer Domain article “Choosing a platform for Domino 6: Hardware platform vendors” at the following address: http://www-10.lotus.com/ldd/today.nsf/0/48557f0e4f6993f485256c0800162731?OpenDocument 1.2.2 Highlights of Solaris Solaris platform is distinguished by the following highlights: Overall Support for one million simultaneous processes on a single system Support for up to 128 CPUs in a single system and 848 CPUs in a clustered environment Support for up to 576 GB of memory Support for more than four billion network connections Support for up to 252 TB file systems with Sun StorEdge QFS Presence of IPv6, enabling a 128-bit IP address space Presence of a 64-bit Java virtual machine Highlights of Sun Solaris 9 It combines power, stability, and predictability with complete backwards compatibility. Offers reliability, availability, and serviceability comparable to that of a data center, but at a fraction of the cost of a mainframe. Assures superior availability through a smaller, more stable kernel design and increased load balancing across multiple processors. Scales to handle heavy traffic, huge data sets, and CPU-intensive problems. Tightens an already secure environment with increased support for major security protocols as well as new technologies. Offers a complete global solution with extensive support for 37 languages and 123 locales. Provides the premier deployment platform for Java technology. Enhances ease of use through simplified software installation and setup, plus comprehensive integration capabilities. Supports the latest networking protocols and adheres to all major industry standards. Key new features in Solaris 9 An improved thread model Optimized memory manager Unix File System enhancements New and improved data management tools New and improved system management tools 6 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 1.2.3 System Requirements for Solaris 9 SPARC 32-bit and 64-bit platforms x86 (32-bit) Sun and third-party platforms (See http://sun.com/bigadmin/hcl for a hardware compatibility list.) Note: Lotus Sametime 3.1 is supported on the SPARC version of Solaris 9 only (running in 64 bit mode). Memory: 64 MB minimum for x86 platforms; 128 MB minimum for SPARC platforms Note: Disk Space: 600 MB for desktop systems; 1 GB for servers 1.2.4 For more information More details on Sun Solaris 9 can be found online at: http://wwws.sun.com/software/solaris/ds/index.html Chapter 1. Introduction to Sametime 3.1 on Solaris 9 7 8 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 2 Chapter 2. Installing and running Sametime This chapter provides a basic walkthrough of installing and working with Sametime 3.1 on the Sun Solaris 9 platform. In this chapter, we include some tips and tricks to help deal with issues that an administrator may encounter. The chapter also contains an overview of the different components and architectural points that come with planning a Sametime environment. Lastly, there is a section on some basic troubleshooting and maintenance that will provide a good starting point for managing an environment. © Copyright IBM Corp. 2004. All rights reserved. 9 2.1 Preparation for installation Before you perform the installations, you should Satisfy basic requirements Create group and user accounts Work through pre-installation considerations 2.1.1 Basic requirements Tip: As any administrator knows, a great deal of time can be saved through proper preparation. Some key points are: 1. Make sure that you have full root access before starting. Sudo and other workarounds are not recommended. 2. Install the required Java version (Java 1.1.4 or above). Java 1.4 comes with the default Solaris 9 install. You can verify this with the command - # java -version 3. Use the recommended X-windows interface. Although you can install Sametime through a command line interface, the default installation and operation requires a graphical interface. 4. Set the kernel parameter msgtql to 1024 or higher. You can verify this through the command - # sysdef -f|grep MSGTQL 5. Use the required version of Domino. Sametime 3.1 requires Domino 6.0.2 CF1. Before starting the Domino 6 installation, make sure to meet following requirements: Apply the latest recommended patches for your Sun Solaris 9. You can type #showrev -p to view the sytem’s current patch list. During Domino installation, checkos will validate that the required patches are installed before the actual installations (see Figure 2-1 on page 11). There are no patches currently required for Solaris 9. See the Domino 6.0.2 Release Notes for the minimum patch requirements. Note: Unix platforms are case sensitive. Be careful to match case when running commands, setting configuration, or referring to files. 10 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Figure 2-1 Example of checkos Check the msgtql parameter. This sets the maximum number of messages that the system can handle at the interprocess communication level. Domino requires that this be increased to 1024 to for purposes of partitioning and fault recovery. An example of the command run to check the msgtql parameter is shown in Example 2-1: Example 2-1 Check of the msgtql parameter # sysdef -i|grep MSGTQL 1024 system message headers (MSGTQL) If the value shown is less than 1024, you can set this parameter in the /etc/system file by adding or changing an already existing line: set msgsys:msginfo=1024 Note: In Solaris 9, Sun has increased the default value of rlim_fd_max to 65536. Therefore, it is unnecessary to set rlim_fd_max as in previous versions of Solaris. Ensure that your Solaris system is running in 64-bit kernel mode. Domino 6.x does not support Solaris 32-bit kernel mode. To test whether you are running a 64-bit kernel, type the following: $ /usr/bin/isainfo -kv 64-bit sparcv9 kernel modules If you are not running a 64-bit kernel, you can type the following to boot to 64-bit mode: #/usr/sbin/eeprom boot-file="" X-windows based support of at least a 256 color resolution is necessary for the installation of Sametime. Although the Domino install does not require a graphical user interface, Sametime uses a Java-based GUI to install the necessary filesets. Based on this requirement, if you have a smaller Sun server that supports a directly-attached video card, then it is important to make sure that the system is configured to boot up into the native CDE display and has Java installed. Because Solaris 9 boots up into CDE and comes with Java 1.4 by default, nothing should be needed at this point. However, if you have a larger Sun system that does not support a directly-attached video display, then the usage of a remote X-windows interface is recommended. Therefore, a workstation with available x-windows emulation software would be required. Chapter 2. Installing and running Sametime 11 In general, this Redpaper assumes that your server has video capabities and that it will run the x-windows user interface locally. Note: X-windows, or as a purist may call it, X or X11, is the engine Unix systems use to draw to the screen and thus provide a graphical user interface. It abstracts the low-level details of hardware communication. Therefore, a program only needs to be able to communicate in a much simpler language that is distilled into basic routines to perform graphic operations regardless of hardware. X-windows does the actual hardware instruction. Meanwhile, the program only needs to speak to X-windows to draw to the screen (even if that machine is located across the network). CDE and other desktop environments like Gnome and KDE are examples of window manager programs that provide a graphical look and feel to the user. Thus, one can think of CDE as program that handles the design and interaction of the interface for the user although X-windows does the actual drawing. Solaris uses CDE as its default desktop environment, but it also has a simple environment called X desktop. You may hear CDE, X, and X-windows used for the same thing, but people essentially are simply referring to the GUI component of the operating system. For more information about the requirements and noted issues of the Sametime 3.1 and Domino 6.0.2 products, see the following resources: Domino 6.0.2 release notes http://www-10.lotus.com/ldd/notesua.nsf/e18d5eb0b8be97d9852567e50052ad16/811bd9436321312 585256d36004f581e?OpenDocument Sametime 3.1 release notes http://www-10.lotus.com/ldd/notesua.nsf/e18d5eb0b8be97d9852567e50052ad16/811bd9436321312 585256d36004f581e?OpenDocument Sametime 3.1 for Solaris readme http://www-12.lotus.com/ldd/doc/uafiles.nsf/docs/ST31sol/$File/st31solarisreadme.htm 2.1.2 Create group and user accounts When the above requirements are fulfilled, you need to create a group and a user account for Domino setup. The user account will be used to operate the Domino server, so you must have a dedicated user for Domino server. In Solaris 9, you can use admintool or create a user via the command line if you are more comfortable. But the admintool is being phased out and will give way to the Solaris Management Console. To start the console, type: # smc & This command will bring up a graphical interface that handles most of your operating system administration, as shown in Figure 2-2 on page 13. 12 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Figure 2-2 Solaris Management Console To create a new group: 1. In the Navigation window, click This Computer →System Configuration; then click Users. 2. Click the Groups icon in what is called the View pane. You can see instructions on how to use the SMC tool to manage groups in the Information pane at the bottom. In accordance with the instructions, perform the following actions on the menu bar with the Groups icon selected: a. Select Action →Open. b. Select Action →Add group. c. Type the group name and unique group number in the appropriate fields as shown in Figure 2-3. Figure 2-3 Creating a group with SMC d. Click OK. To create a user: 1. Click the User Accounts tool under the System Configuration Users section. a. Select Action →Open from the drop-down menu. b. Select Action →Add User →With Wizard, also from the drop-down menu. c. Follow the fields and forms presented by the wizard. You can refer to the help on the left-hand window, as shown in Figure 2-4. Figure 2-4 Adding a user with SMC Chapter 2. Installing and running Sametime 13 Note: In our test environment for this Redpaper, we used User name: notes Group name: notes However, you can pick any name you want to use. It is not required to name the user account notes for Domino. For more details on creating users and groups using Admintool, see Chapter 2.5.1 Creating users and groups in the IBM Redbook, Lotus Domino R5 for Sun Solaris 8, SG24-5969 In this Redpaper, notes user and notes group refer to the user account and the group you created above for Domino. You can also refer to the Sun Administration Guide for more about managing users and groups in Solaris 9. Refer to “System administration Guide: Basic administration” at: http://docs.sun.com/db/doc/806-4073 2.1.3 Preinstallation considerations Prior to starting the installation process for any Domino 6.x server, some basic planning is recommended. Specifically, consider the following: Deployment options: Various deployment options and choices are available with Sametime. Before you start, considering the following: – Domino domain: Decide whether to install Sametime in an existing Domino domain or a dedicated Domino domain. Because of Sametime’s ability to work with other directories, you do not need a preexisting domain or migration of existing users to work. – LDAP or native Domino directory: Determine whether or not LDAP or the native Domino directory will be used. – Architecture: Decide on the overall architecture of your Sametime deployment. You can utilize Domino cluster technology, Sametime multi-server meetings, mux, and even a third party load balancer. Security considerations: As with any application, proper security planning is crucial. Sametime actually provides multiple types and layers of security; the installation- and setup-related aspects are: – Physical and platform security: The hardware and operating system itself must be properly secured. – Protocol security: You can protect communications using secure socket layer encryption (SSL) and Notes/Domino port encryption. – Authentication: To allow users to have single sign-on (SSO) capabilities across multiple servers, the Lightweight Third-Party Authentication (LTPA) token-based Domino multi-server single sign-on (MSSO) must be enabled. For more information about security features and options, refer to Working with Sametime security section in the Sametime 3.1 Administrator’s Guide at: http://www-12.lotus.com/ldd/doc/sametime/3.1/sthelpad.nsf/769e1b028f863f84852566c1003584 67/46bec4c3a1a34e8885256d33007da8b3?OpenDocument&AutoFramed 14 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Naming conventions: Establish naming conventions for the names of your Sametime servers, the OS User IDs, and the OS Group names you plan to use. Having a clear server and ID naming strategy can greatly simplify system management. Disk allocation strategy: Unlike a Domino mail server, the Sametime server will run most of its activity on just a few databases. This makes the allocation of I/O run a little differently. Assuming that you have a good number of users, it is possible to spread the I/O manually across disks. However, with only ten databases running and most of the activity being done with one or two databases, RAID strategies become more effective. Installation architecture: It is highly recommended to install Sametime on a dedicated Domino instance/partition. Although it is possible to run other Domino applications such as mail and HTTP on the same Domino partition, it is not recommended. However, if your Sun hardware has adequate capacity, other Domino partitions can exist on the same physical box to support these additional Domino capabitiles (for example, mail and HTTP). Tip: Isolating the functionality of Sametime adds value outside of just resource allocation. Not running other tasks such application and mail serving allows the administrator much more flexibility in administration. Uninstalling, upgrading, and general maintenance are made vastly easier. 2.2 Install and Setup Domino 1. Identify or create a file system on the Solaris system to be used for the install. Tip: We recommend using file systems spread across multiple physical devices, or I/O channels, to optimize I/O performance/throughput. 2. Log in as Solaris root user. 3. Insert the Domino installation CD-ROM 4. Change to the sunspa directory: #cd /cdrom/cdrom0/sunspa 5. Run the install program. Type #./install. 6. Press the Tab keys to continue. 7. Follow instructions and answer the prompts. Here is quick reference: a. Select Setup type: We are using the Domino Enterprise Server because it is offers the full feature set of Domino. Each installation depends on the particular Sametime license purchased. You will need to choose the appropriate one for you. b. Set current program directory: Specify the location where Domino executables are located. By default this is /opt/lotus. c. Determine whether you want to run more than one server based on this installation: If you want to have domino partitioning on this server, you need to answer Yes here and follow the instruction. In our example, we selected No. Chapter 2. Installing and running Sametime 15 d. Set current data directory: Specify the location where all of your Domino data will be located (for example, /notesdata). e. Set user and group: Specify the user and group you created for Domino. 8. Then, you will see Select Server setup method prompt. There are 3 options available for you. – Manual server setup: Manually starting the server to initiate server setup. – Local Server Setup: Automatically utilizing the server’s setup program at the end of install. – Remote Server Setup: Running the setup program from any Domino Administrator Client that has the Remote Server Setup tool option installed. We use Remote Server Setup (as in Figure 2-5) in this paper. Figure 2-5 Remote Server Setup 9. After you have completed all the prompts, verify your selections n the confirmation screen. See Figure 2-6 on page 17. 16 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Figure 2-6 Installation settings 10.Press Tab to install the Domino server code. At this point, the installation program validates OS patches and starts installation. After the installation completes, depending on your installation choice, the script will either complete (manual installation), start up the Domino setup program (local installation), or automatically start listen mode to setup the Domino server (with the Remote install option). See Figure 2-7. Figure 2-7 Server setup listen mode At this point, the actual server code installation has been completed. However, the server needs to be configured to start up. The server is now in listening mode. It is waiting for the Remote Server Setup tool to finish the configuration. Chapter 2. Installing and running Sametime 17 11.Start Remote Setup from your workstation. This is an installable option with the Domino Administration client. In Windows XP, simply choose Start →All Programs →Lotus Applications →Remote Server setup. When the set-up program first starts, it looks like Figure 2-8. You can ping the server to confirm that the server is listening on the default port of 8585. Type the hostname (FQDN or the IP address) and click OK. Figure 2-8 Remote Server Setup tool Note: Each installation mode has the same look to the interface. Therefore, the following screen shots will appear the same for whatever installation option you choose. 12.Follow the set-up procedure. a. Verify the Organization, Server name, Administrator, network, and ports options. b. Make sure that any necessary changes are completed. c. Remember to memorize passwords. (You may want to change default passwords to something more meaningful to you.) Tip: Putting Sametime in its own domain, separate from any existing Domino infrastructure that you may have, can make managing the server much easier with no loss of functionality. This approach simplifies server configuration records and administration. However, if you set up multiple Sametime servers, it make sense to have them all participate in the same domain. In addition, it is extremely useful to store a copy of the administrator ID on the server. It allows you to run the command line console and retrieve it in case of an issue with Domino security. However, you must keep the file secure. 13.You can choose to customize these services, as shown in Figure 2-9 on page 19. For Sametime, you need the LDAP and HTTP server. This is true whether you decide to use Meeting services or LDAP or not. If you do not select it here, the Sametime install will configure it for you later. 18 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Figure 2-9 Customizing Domino services 14.Click Finish. Domino server configuration begins. The process asks whether you want to remove the server from the listening mode. Click Yes. At this point, your server should be ready. 15.You must be logged in as the notes user to start the server. Then, change the directory to the notes data directory: $cd /notesdata Note: If you decide to switch to the notes user using su, use the following command: - su - <notes_user> Running this command ensures that the notes user’s environment settings take effect. You can verify that you now have a fully initialized notes user session by issuing the following command: $ pwd The following command returns the directory that you are currently in, which should be the notes user’s home directory: $ whoami This returns the user you are currently. It should return the notes user. 16.The command to start Domino is located in /opt/lotus/bin directory. Issue the following command as the notes user from the data directory: $ /opt/lotus/bin/server" s/b "$ ./start The Domino console will start up with the lines in Example 2-2: Example 2-2 Domino console start up Lotus Domino (r) Server, Release 6.0.2CF2, July 23, 2003 Copyright (c) IBM Corporation 1987, 2003. All Rights Reserved. Chapter 2. Installing and running Sametime 19 Starting Sametime in the background An alternative to starting Sametime as just described is to start Sametime so that it runs in the background. To run Sametime in the background, you will need to prepare your environment to run the X Virtual Frame Buffer (Xvfb). Xvfb emulates a dumb framebuffer using virtual memory. The primary use of this server was intended to be server testing, but other uses for it have been found, including: Doing batch processing with Xvfb as a background rendering engine Providing an unobtrusive way to run applications that don't really need an X server but insist on having one available anyway. This is the case for Sametime. To install and configure Xvfb for Sametime, performing the following steps: 1. Install the Xvfb. Xvfb is already installed on Solaris 9 in /usr/openwin/bin. Solaris 8 users will need to obtain a separate implementation of Xvfb. 2. Login in from a terminal shell as root and run the text in Example 2-3: Example 2-3 Root log in /usr/openwin/bin/xhost <host name or ip address> /usr/openwin/bin/Xvfb :1 -screen 0 1280x1024x8 & You might get a message stating "No such file or directory," but this is normal. You can assign any number except 0 in place of the number 1in the example above. This is the display number you wish to have associated with this instance of the XVFB. 3. You can verify that the Virtual Frame Buffer (VFB) is running by typing: ps -ef | grep vfb You should see the Xvfb process running. 4. Now that the VFB is running, you should set up Sametime to run in the background. You can enable the startup of the Xvfb automatically upon boot by creating a script in the /etc/rc3.d directory with the lines in step 2. 5. Log in as the notes user and make changes to ~/.profile or equivalent to include the text in Example 2-4: Example 2-4 Configuring Sametime to run in the background LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${BINDIR} PATH=$PATH:${BINDIR} DISPLAY=<hostname>:<display> export LD_LIBRARY_PATH PATH DISPLAY <hostname> is the hostname of your machine. <display> is the display number used in Step 2. In that example it was 1. However, it can be any display number. Of course, as always, different unix shell profiles can have different syntax. Therefore, this should be verified with your shell. After VXvfb has been properly configured, Sametime can be started in the background by performing the following: 1. Login as the notes user you will use to run the Sametime instance. 2. From the data directory, run the command: 20 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 nohup ./ststart < /dev/null > /dev/null 2>&1 & You may exit the telnet session or the x-windows session. To have Domino/Sametime startup automatically, create a script with this command and place the file in the /etc/rc3.d/ directory. 2.3 Installation and set up of Sametime This section describes the actual set up and installation of Sametime. 2.3.1 Pre-install tasks This section describes the steps necessary to configure the underlying Domino server code that is required for Sametime 1. Verify that the server document is using its fully qualified name as shown in Figure 2-10. This is important for authentication and identification purposes. Simply using a host name causes name resolution issues. Figure 2-10 Implementation of a fully qualified name 2. Verify that the fully qualified hostname and localhost are listed in the /etc/hosts file. Also, make sure the notes data directory is included in the PATH environment variable of the notes user, as shown in Example 2-5. These are known issues and requirements for making Sametime 3.x on Unix to work properly. Example 2-5 Sample /etc/hosts file 127.0.0.1 9.33.85.89 localhost loghost sametime.cam.itso.ibm.com sametime You can change the default PATH for your notes user by changing or adding the PATH variable in the user’s environment profile. The example in Example 2-6 assumes that the user’s shell is ksh and that the notes user is modifying his or her own .profile. Example 2-6 Sample text to add to $HOME/.profile of the notes user PATH=/usr/bin:/usr/ucb:/etc:/notesdata:. export PATH 3. In the Server document, be sure to include “Sametime Development/Lotus Notes Companion Products” in the “Run unrestricted methods and operations” field in the Security section, as shown in Figure 2-11. This allows proper execution of the methods designed by Sametime Development. Figure 2-11 Required security settings Chapter 2. Installing and running Sametime 21 4. Put the fully qualified hostname in the Hostname field and enable Bind to host name”in HTTP section of the Server document. This does two things: – First, it sets the definitively sets the hostname to be used by the HTTP service. If left blank, it would be set by the platform’s hostname (which could cause a confusion later). – Second, it binds the HTTP service to the IP specified by the Hostname field rather than to all IPs that may be assigned to the system. This removes the possibility of confusion. In addition, you may want to take this opportunity to change the value in the Home URL field to the Sametime home page that will exist after Sametime is installed (which defaults to http://server.domain/stcenter.nsf), as shown in Figure 2-12. This can make life much easier for your users because they will automatically be directed to the Sametime homepage as the default URL when accessing this server. They will not need enter it manually. Figure 2-12 HTTP settings 5. Next, bring the core Domino server down by typing Quit at the Domino console Note: Three possible sticking points to watch for are: 1. It is important the Domino server has been fully started and brought down at least once at this point. Shortcuts to access the Domino directory and populating the above fields can yield unpredictable results 2. Make sure that the bin directory (as in /opt/lotus/bin) is not in the PATH. 3. Check the hosts file to verify that the normal IP loopback address is listed and pointing to 127.0.0.1. 4. Log into the Solaris root account. 5. Make sure that you have rights to use the display. To disable security, enter the following command. #/usr/openwin/bin/xhost + 6. Set your DISPLAY variable, as shown in Example 2-7: Example 2-7 DISPLAY variable setting #ksh #DISPLAY=:0.0 #export DISPLAY Note: As mentioned earlier, this Redpaper assumes your server has video capabities and that it will run the x-windows UI locally. If you are installing using a remote X-windows interface, some of the steps would be slightly modified to ensure proper exection of the install UI remotely. 22 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 2.3.2 Installation 1. Mount the cd-rom drive with the first CD out of the Sametime package: # cd /cdrom/cdrom0 2. Start the installation program, as shown in Figure 2-13. Figure 2-13 Starting the Sametime installation program The user interface will then start, as shown in Figure 2-14. Figure 2-14 Initial install screen 3. Follow along with the wizard. It will ask for the Domino Lotus Directory. This is the location of the executable portion of the Domino installation. This is /opt/lotus in our Domino installation example. 4. The next couple of screens will ask questions such as the fully qualified hostname, the data directory, and language. 5. The installation program will ask which type of directory you wish to use. This will be the directory Sametime uses to authenticate. Either a Domino Directory, or an external LDAP directory, is supported. However, it is important to note that whatever directory choice is made at this point cannot be easily changed after installation. See Figure 2-15. Figure 2-15 Choosing the authentication directory Chapter 2. Installing and running Sametime 23 Note: It is important to note that only users in the local Domino Directory can be administrators. Therefore, when Sametime administrators log in to the Sametime administration interface with an ID that has Sametime administrator rights, they still are authenticated through the local Domino directory, regardless of whether an LDAP or Domino directory is used. By default, the Domino administrator created during the Domino server setup is configured as a Sametime administrator. 6. Once all the choices have been made, you can choose to start the actual copying of files and installation routine. After all files have been copied and processed, the install program will notify you of successful completion. See Figure 2-16. Figure 2-16 Successful completion Tip: If there was a failure during the install, the first places to look for information regarding the failure will be the /tmp/install.log and stsetup.log in the Domino data directory. Additionally, if you have a Sametime environment of multiple servers you must ensure that certain Sametime settings are consistent across all of the Sametime servers in the community. See the Technote titled Sametime 3.0: What Configuration Documents Should Be Replicated Between Servers in a Community? for more details on this issue. http://www-1.ibm.com/support/docview.wss?rs=0&q1=What+Configuration+Documents+Shou ld+Be+Replicated+Between+Servers+in+a+Community&uid=swg21089634&loc=en_US&cs=utf-8 &cc=us&lang=en 2.3.3 Integrating LDAP If you choose to use an LDAP directory, you will need to specify that during the installation routine, and then configure Domino and Sametime to use LDAP. However, as mentioned earlier, it is important to note that whatever directory choice is made at this point cannot be easily changed after installation. See Figure 2-17 on page 25. 24 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Figure 2-17 Choosing to use LDAP during the install routine Configure Domino LDAP authentication first needs to be configured through Domino. This configuration requires you to create a directory-assistance database. If the Domino/Sametime server is not already started at this point, you need to start it. Starting the Sametime is the same as starting Domino. Sametime is now configured to start when Domino does. Simply log in as the Domino user and switch to the data directory and run the server command, as shown in Example 2-8. Example 2-8 Starting Sametime $ cd <data directory> $ /opt/lotus/bin/server Alternatively, start up Sametime in the background as described in the “Starting Sametime in the background” on page 20. Using the Notes client from the drop down menu, choose File →Database →New. See Figure 2-18. Figure 2-18 Creating a directory assistance database Using the Directory Assistance template, create a new database. The database should open automatically after being created. Click the Add Directory Assistance button in the view window. On the Basics tab, enter the following: – Domain type: LDAP – Domain name: A unique name that is not a Domino Domain – Company name: A descriptive name – Search order: Usually 1, because it is the first directory to search Chapter 2. Installing and running Sametime 25 – Make this domain available to: -> Check Notes Clients & Internet Authentication/Authorization – Group Authorization/Nested Group expansion: -> Recommended – Enabled: Select Yes Leave the default wildcard entry (*) in all of the fields on the Naming Context Rules tab. Using these wildcards opens up the restrictions so that all entries can be searched. Set the Enabled and Trusted for Credentials fields to Yes. See Figure 2-19. Figure 2-19 Naming Context rules tab On the LDAP tab, set the following: – Hostname: Hostname or IP address of the LDAP server – Base DN for search: The base distinguished name to search from (for example, o=ibm). The other settings depend on your individual configuration. Refer to the section “Configure the LDAP Directory settings” in the Sametime Admnistrator’s Guide for more information about each setting. Configure Sametime to use LDAP Open the Sametime server’s Sametime Configuration database (stconfig.nsf) with the Notes client. From the file dropdown menu, select Create →Other. Choose LDAPServer as the document type and click OK. Enter the IP address or the hostname of the LDAP server in the Network Address of LDAP Connection field. Verify that the rest of the settings match those entered into the Directory Assistance document created in the previous section. Change the Sametime Directory parameter in the <data dir>/sametime.ini to LDAP. Restart the Sametime server to let these changes take effect. Tip: To test LDAP connectivity, run the following command from the Domino console, type: >load ldapsearch -b <base dn> -h <hostname> <search criteria> For example, to search for mail addresses that start with michael, with a base dn of ou=itso,o=ibm on the server ldap.ibm.com, type: >load ldapsearch -b ou=itso,o=ibm -h ldap.ibm.com mail=michael* 26 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 2.3.4 Verifying server functionality After the installation is finished, you will want to connect to the server with the Sametime Connect client and start a meeting to verify that everything works properly. With the Sametime up and running as previously described, open up a web browser on any machine in the network and point it to the Sametime server. The address will be in the form http://<server>/stcenter.nsf. Testing the connect client At this point, the functionality for users is the same across server platforms. The Java connect client that can be launched from the web page implements the same basic chat functionality found in the Windows version of the connect client. Thus, it will be a good example of authentication, awareness, and instant messaging functionality 1. Click the link ”Launch Sametime Connect,” as shown in Figure 2-20. Figure 2-20 Launching the Sametime Java Connect client 2. You will be brought to a login screen, as shown in Figure 2-21 on page 28. Click the Connectivity button to make sure that the correct Sametime server is listed. Tip: Sametime configures Single Sign-On by default on the Domino server. One side effect of this is that the default security requires the fully qualified name be used in the URL for authentication. However, this can be changed by modifying LDAP search filters if an LDAP server is being used. Refer toother IBM Redbooks available on the Redbooks website for more details on Sametime and LDAP configurations. Chapter 2. Installing and running Sametime 27 Figure 2-21 Sametime Java Connect client log-in window 3. Change the Host field if it is not already populated with the proper Sametime server. In most cases when the connect client is launched from the Sametime server UI, these settings will be correct. Click OK to continue. See Figure 2-22. Figure 2-22 Preferences window 4. Enter your username and password and log on to Sametime. By default, the Domino administrator will be a good ID to test. If all is successful you should see your user in the awareness list as green (green signifying availability), as shown in Figure 2-23 on page 29. 28 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Figure 2-23 Successful test of authentication and awareness Note: If the Java client doesn’t appear to be working, you may find that resizing the client helps. Testing Meeting services 1) From the server’s home page, click the “Schedule a Meeting” link. See Figure 2-24. Figure 2-24 Scheduling a meeting 5. Log in using the same Domino administrator used when testing the chat client, as shown in Figure 2-25. Figure 2-25 Logging into Sametime through the New Meeting page 6. Fill out the New Meeting form, as shown in Figure 2-26 on page 30. Be sure to click the Start Now radio button so the meeting will occur immediately. Click Save. Chapter 2. Installing and running Sametime 29 Figure 2-26 Meeting form 7. You should automatically be forwarded to the meeting and be able to see yourself as a member of the meeting. An additional test would be to then log in as another valid user in the directory and try to join the same meeting. Then, you can determine whether both users successfully joined and are able participate in the meeting. See Figure 2-27. Figure 2-27 Test meeting 1. If you have any problems with the installation, review the setuplog.txt in you data directory for errors. 2.4 Extending Sametime One of the great values of Sametime is it’s configurability to support large enterprises. You can have multiple servers working together to provide seamless service. This includes add-on features, scaling for larger business requirements, and integrating with and adding functionality to existing messaging pieces. This section provides a brief overview and does not offer a comprehensive list. It covers Sametime and SSL, Sametime conversion services, Sametime Mux, SIP gateway, and scaling/clustering solutions. 30 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 2.4.1 Enabling SSL for Sametime Sametime supports the usage of Secure Sockets Layer (SSL) technology to safely secure all instant messaging and meeting traffic. Of course, the usage of SSL encryption must always be weighed against the performance impacts of enabling it. Below is a quick review of the key steps involved in setting up a Sametime server for SSL. In general, SSL must both (1) be enabled within the Domino server for SSL encryption of the web interface to Sametime and (2) also be enabled with the Sametime java code via modification of the sametime.ini file for encryption of general meeting and chat traffic. More details can be found in within other Sametime 3.1 oriented Redbooks, as well as in the Sametime Administration Guide. 1. Create a certificate authority database: a. From a Notes client, use the Domino Certificate Authority advanced template to create the ca.nsf database. Open ca.nsf in the Notes client. b. Click 1. Create Certificate Authority Key Ring & Certificate. If you receive an error, open the database again and try this step again. Then, complete the following steps: i. Accept the default Key Ring File (CAKey.kyr). ii. Enter the Key Ring Password and verify it. iii. Set the Key Size to 1024. iv. Enter the common name (a descriptive name such as Servername CA), an organization, a state or province, and a country in the appropriate fields. v. Click the Create Certificate Authority Key Ring button. c. Click 2. Configure Certificate Authority Profile. Accept all defaults and click Save and Close. d. Click 3. Create Server Key Ring & Certificate and complete the following steps: i. Accept the default Key Ring File Name (keyfile.kyr). ii. Enter the Key Ring Password and verify it. iii. Accept the default key size. iv. Enter the CA Certificate Label (for example, servername CA). v. Enter the common name (the fully-qualified name of the server), an organization, a state or province, and a country in the appropriate fields. vi. Click the Create Server Key Ring button. vii. Enter the password for the authority that you entered when you clicked 1. Create Certificate Authority Key Ring & Certificate. 2. Enable SSL on the Domino server: a. Using a Notes client, open names.nsf and then open the Server document for the Sametime server. Follow the steps below: i. Go to Ports →Internet Ports. ii. Ensure that the "SSL key file name" matches the default key ring file name listed in Step 2 above (keyfile.kyr). iii. Set the TCP/IP Port Status to Disabled. iv. Set the SSL Port Status to Enabled. v. Click Save and Close. Chapter 2. Installing and running Sametime 31 3. Get the certificate authority text from the server: a. Open a browser and go to http://<servername>/ca.nsf. (This URL is case sensitive.) Follow the steps below: i. Click Accept This Authority In Your Server on the left. The certificate text appears on the right. ii. Copy the certificate text from BEGIN CERTIFICATE to END CERTIFICATE. iii. Paste this text into a new text document and save it; for example, save the text as ca.txt. When saving the text document, insert a new line after the END CERTIFICATE line. 4. Transfer files to the server: a. FTP the following files from their current locations (listed below) to the data directory of your server (for example, the /notesdata directory): i. keyfile.kyr and keyfile.sth, both of which are located in the data directory of your client ii. ca.txt, which is located in the directory you chose when creating this file in step 4 above. 5. To create a key file, run the program Ikeyman on the server. To run Ikeyman: a. Open the file ikeyman.sh. This file is located in the server's binary directory (for example, /opt/lotus/notes/latest/ibmpow). b. If your Sametime location is different than /opt/lotus/notes/latest/ibmpow, set the variable SAMETIME_HOME to the correct value. Otherwise, do not modify the ikeyman.sh file. c. Change your directory to the server's binary directory and run ikeyman.sh (ensuring that the file has execute privileges). d. When ikeyman starts, select Key Database File →New. e. Accept the default setting for the key database type (for example, Key database type = JKS). f. Enter key.jks in the File Name field. g. Enter /notesdata (the server's data directory) in the Location field. h. Click OK. i. When prompted, enter a password for the key file. Reenter the password to verify it. The key file is created and some information about the file appears. Do not close the Ikeyman program. j. Import the server certificate authority into the key file as follows: i. Click the Add button on the right. ii. In the dialog box that appears, make sure that the Data Type field contains the text Base64-encoded ASCII data. iii. Set the Certificate file name to the name of the text file you created in Step 4 (ca.txt). iv. Enter the location where ca.txt was transferred in Step 5 (for example, /local/notesdata) in the Location field. v. Click OK. vi. Enter a label such as "servername CA" for the Certificate. vii. Click OK. 32 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 k. Close Ikeyman after the program finishes importing the certificate.` 6. Stop the server. 7. Modify the Sametime.ini file: Open the Sametime.ini file (located in the server's data directory) and make the following changes under the [Config] section of the file. If a setting is already in the file, change the value as indicated below: – ConfigurationPort: 443 – ConfigurationSSLEnabled: true – SSLManagerClassName: com.lotus.sametime.configuration.IBMJSSE12Manager – javax.net.ssl.keyStore: /local/notesdata/key.jks – javax.net.ssl.trustStore: /local/notesdata/key.jks – javax.net.ssl.keyStorePassword: PASSWORD (where PASSWORD is the password for the key file you created in step 6) – javax.net.ssl.trustStorePassword: PASSWORD (where PASSWORD is the password for the key file you created in step 6) – javax.net.ssl.trustStoreType: JKS – javax.net.ssl.keyStoreType: JKS 8. Restart the server. If you removed staddin from the notes.ini task list in Step 1, be sure to reinsert staddin before restarting Sametime conversion services The Sametime Conversion services expand the ability of your Sametime server(s) to display attachments in the whiteboard of your meetings. This greatly simplifies the creation and content management of meetings. Once the file is attached to the meeting, it is then sent to the attachment conversion server. This server returns a converted file that can be viewed via the whiteboard. When this feature is installed, users no longer have to prepare each meeting attachment manually. Once installed, attachment conversion is an invisible process. This feature comes on the second CD of the Sametime package. It requires a separate machine running Windows 2000 or XP. Installation is done simply by running the application in the STConv directory. Once stconv.exe is installed, simply direct your Sametime server to it by: 1. Opening the meetingserver.ini file in the Domino data directory 2. Locating the line: NumberOfAttachmentConversionServers. 3. Entering the following values: a. Software: SOFTWARE\Lotus\Sametime\MeetingServer\ConversionServices b. NumberOfAttachmentConversionServers:=0 c. AttachmentConversionServer1: 0.0.0.0:19610 d. AttachmentConversionServer2: 0.0.0.0:19610 e. AttachmentConversionServerTimeout: 120 Change the value to the number of conversion servers that will be used. Point the AttachmentConversionServer# parameters to the appropriate servers. Chapter 2. Installing and running Sametime 33 Attachment conversion services are started through Windows, and the Sametime server is restarted for the settings to take effect. Sametime Mux The Sametime server has a component called the Community Services multiplexor (referred to as MUX). This manages connections from Sametime clients to the Community Services on the server. It is possible to install the MUX by itself on a separate machine. The MUX program does not rely on the pre-existence of software other than Windows. When implemented, a client could then connect to the MUX server or the Sametime server. All client communication is then passed through a single connection between the MUX and the Sametime server. This allows for another server or possibly multiple servers to handle communication, awareness, and I/O issues. This separation of tasks can be useful for increasing flexibility in administration and infrastructure. Of course, it also would increase performance, since each task is focused on separate machines. Note: The Mux component must be installed on a Windows NT®, Windows 2000, or Windows XP machine. SIP Gateway SIP Gateway support is new in Sametime 3.1 on Solaris. By default, it is installed as a component of the Sametime server. Connecting to an external SIP-enabled community requires that the SIP gateway can connect to the SIP Connector. The SIP Connector must be installed on a separate host. Once installed and operating, it then manages the connections for the SIP gateway. Installing and configuring SIP functionality is beyond the scope of this redpaper, but it is covered within other Sametime 3.1-oriented Redbooks, as well as in the Sametime Administration Guide. Note: The SIP Connector can only be installed on a Windows platform. Scaling and Clustering Solutions Sametime 3.1 for Solaris provides multiple scaling and clustering capabilities. These capabilities allow you to deploy a more advanced production system than the basic installation covered in this Redpaper. These advanced features allow you to: 1. Install multiple Sametime partitions on the same machine. 2. Add additional servers that provide Community services to an unlimited number of users. By adding another server to the Sametime community, it possible to just keep adding new machines as it becomes necessary. All servers have the ability to serve chat, instant meetings, and awareness to all other users in the community. Just register another server in the Domino domain and install Sametime on that server. 3. The Community services cluster provides all the same functionality of adding another Sametime community server. It also adds automatic failover and load balancing for community services. This feature builds on top of Domino clustering capabilities. 4. It is possible to use multiple Sametime servers to deliver a single meeting. This allows for greater geographic flexibility, more efficient usage of resources, and greater ease of administration. All it requires to enable this is to add the appropriate Sametime connection documents and invite the meeting server to the specified meeting. 34 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 5. For clustered meeting services that provide failover and load balancing, there is a separate product called Sametime Enterprise Meeting Server (EMS). 2.5 Maintaining your Sametime environment There are many tasks that you will want to consider to properly maintain your Sametime environment, such as performing adequate backups and ensuring that the OS has the latest security patches. However, this section covers two topics that will be crucial to all environments: namely, (1) how to add new users and (2) a listing of key files and tools for troubleshooting. Registering new users Sametime is extremely lightweight in its address book requirements. If you are using a Domino directory, the only options that you need to ensure are created for any new users are the Username and Internet Password fields in the Domino person document. If you happen to be using an LDAP directory, you can use almost any identifier that you wish to specify the user, as long as you make sure that you have the appropriate mapping in your Sametime LDAP configuration. Below is a simple example of the process used to create a user when using the Domino directory to authenticate. You can register users either Lotus Domino Administrator client or Web administrator client(webadmin.nsf). You can type: http://servername/webadmin.nsf to use Domino Web Administrator. To register new users: 1. Select the People & Groups tab. 2. Click People on the far right of the tab. 3. Select Register. 4. Select Certifier ID and close the certifier window. You can use CA process as well. 5. Ignore the warning message regarding certifier recovery information. Note: An Internet password is required for Sametime. 6. After make all necessary selections, click Add Person. 7. Click Register at the bottom of the page. 8. Click Done. Tip: If you want to register a large number of users at once through a script, see the section titled “Registering users from a text file“ in the Domino Administrator 6 Help database In addition, Domino supports migrating users from a variety of sources including MS Exchange, Windows NT domains, Active Directory, and LDAP. Chapter 2. Installing and running Sametime 35 Troubleshooting tools A short list of the troubleshooting and maintenance tools available for maintaining your Sametime environment follows: nsd: Probably the most valuable tool in identifying problems associated with crashes and performance. The executable is located in the /opt/lotus/bin directory by default. It takes a snapshot of the system, including Domino stack traces and memory allocations. Console log: To enable logging of all console output to a text file, add the parameter console_log_enabled=1 in the notes.ini (located in the data directory). All output is sent by default to <Domino data directory>/IBM_TECHNICAL_SUPPORT/console.log file. SametimeDiagnostics.properties: Used to configure the log output of Sametime. Currently the default level of logging is set to disable System.Out prints and at the INFO level. Greater detail can be displayed by setting log levels to DEBUG. Output can be captured in files specified here and by enabling System.Out prints and enabling console logging as mentioned above. Domlog.nsf: Enabled through the Server document →Internet protocols tab →Domino Web Engine subtab. This tool can provide valuable information about HTTP performance and activity. Stlog.nsf: Sametime activity information stored in a Domino database. sametime.log: Process information related to Sametime stored in a text file. communityConfig.txt: Contains connection information regarding IP addresses, ports, cluster membership, and login mapping to other community servers. Trace files: Traces can be enabled by setting parameters such as VP_TRACE_ALL=1 in the sametime.ini. This will create trace files in the /opt/lotus/bin/notes/latest/sunspa directory by default. Take extra care in making notes of settings changes. If you want to enable another server, you need to match them up 2.5.1 Uninstalling Sametime One final consideration is how to remove Sametime if needed. To do so, simply delete the binary and data directories from the server. You can do this as the root user with the command in : Example 2-9 Removing Sametime #rm -rf <Domino program directory, i.e. /opt/lotus> #rm -rf <Domino data directory, i.e. /local/notesdata> Note: “rm -rf” is a recursive delete that does not ask for deletion confirmation. It deletes all files and directories targeted. Files deleted in this manner cannot be recovered. Every Unix administrator I know has at some time accidentally deleted something with this command. Use it with care. Outside of meetings and configuration, Sametime does not house much in the way of stored data. If you want to keep your configuration information when uninstalling, you can save the following files: stconf.nsf: Where meetings are stored 36 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 stconfig.nsf: Sametime Administration data vpuserinfo.nsf: User buddy list data sametime.ini: Generally not changed unless advised by Support meetingserver.ini: Generally not changed unless advised by Support stlog.nsf: Sametime logging information Chapter 2. Installing and running Sametime 37 38 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 3 Chapter 3. Tuning Solaris is an extremely powerful operating system. When you talk about performance, you are generally talking about resources in terms of I/O, computing power (CPU), and memory. Overall, Solaris offers high performance in terms of: The amount of resources it can handle The flexibility in the types of resources it manages The efficiency with which it uses those resources The power it gives an administrator in managing those resources With all of these capabiltiles, comes the ability to a host a wide range of computing tasks from database operations, web serving, graphics processing, and so on. However, each task is different in the requirements it has for resources and how it uses them. Thus, it falls on the administrator to tune the system to use those resources as effectively as possible. In this chapter, we examine the proper methodology and tools to actually see what is going on with your system from a performance standpoint. Then, we review some of tuning options available to you and what effect they can do for you. Next, we run through a test scenario to show what this methodology might look like. Finally, we consider how Sametime uses resources and the value that tuning can bring. © Copyright IBM Corp. 2004. All rights reserved. 39 3.1 Tuning methodology Tuning can have a variety of implementations. Users can tune a system and get fairly good results without any knowledge of what the changes they made do or why they made them. This approach works fine unless there is a problem. When a problem does occur, the best one can hope for is sub-optimal results. The worst that can occur is an unstable server that is prone to crashing. Sadly enough, this can very easy be the case. Also, there is no set list of tunings for every situation. The things that need to be tuned, what things can be tuned, and how one can implement tuning vary from application to application (and, to some degree, from box to box). Therefore, a more open methodology will provide a better understanding of the problem and flexibility in confronting differing situations. A good methodology centers around applying the scientific method to the changes that must be made to the system. The value in doing this comes from: Developing knowledge Being able to apply understanding over and over again Increasing the ability to handle more complexity and a greater variety of situations Achieving greater optimization Maintaining stability This step-by-step approach to performance tuning (observing, hypothesizing, and testing) may take longer to start, but it will produce better results and use your time more efficiently. 3.1.1 Step 1: Observe Observe the starting behavior of the system by using standard performance monitoring tools. A list of some of the tools used to collect data on Solaris systems is provided later in this chapter. 3.1.2 Step 2: Hypothesize Addressing the question of “what is the problem?” requires an understanding of the system and the changes that can be made to it. For our purposes, it is best to think of problems that fall into the category of the Big Three: I/O, CPU, and memory. I/O: Refers to a software or hardware device that sends and receives information to and from the computer. Think of it in terms of a pipeline. There is the question of how much can go through at once and how fast it goes through. However, we are talking about information rather than water or oil. That information can flow to and from the keyboard, a hard drive, or the network. CPU: Also a pipeline in a certain sense. However, in this case we are talking about the ability to process data rather than just move it. The processing of information becomes limited by how much data can be processed at once and how fast it can be processed. The question of how efficiently data is being processed is also critical for our purposes. Memory: Memory issues with regard to performance generally center around maximizing the speed at which data can be accessed. In that regard, it is better to think of memory in terms of pages that the system needs to keep processing information. Physical memory (also known as RAM) is much faster than a storage device (such as a hard drive). Thus, performance centers on trying to maximize the use of RAM through swap devices, file caching, and flushing memory to disk. So the questions to ask are: 1. Is one of the Big Three categories being affected? 40 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 2. Is there something I can do about it? 3. What can I expect from these changes? Use this basic structure to build a hypothesis about what is causing the performance bottleneck and what can be done to avoid it. 3.1.3 Step 3: Test Try to limit the number of variables that you will modify to isolate the effects of your changes and serve to provide evidence of how effective those changes are. At the very least, try to isolate the changes so that they only affect one of the three areas discussed above. Be sure to have specific criteria that you will be looking at to judge whether the testing was effective or not. Points to remember: The first 80% of performance tuning is usually achieved quite easily. After that point, it becomes more and more difficult to wring more performance out of your box. Often people will try to get the 80% level quickly and tune from there. At some point, though, you will need to decide when enough is enough. Trying to get too much performance out of limited resources increases both risks and costs. Pushing a box to the limits of it’ abilities, leaves less of a buffer to handle unexpected or more demanding situations. Do not make adjustments unless there is clear value to them. To some degree, each tuning point represents a risk and may have an unintended effect. You not only need to justify the benefits of any changes. You also must be able to account for the tertiary effects of those changes. Try to minimize your changes to the system for each test. The value of tuning is best maximized when you can test the clear relationship of the change being made to the system. Although this may not always be feasible, greater adherence to this rule will save time in the long run. Always keep a backup plan whenever changes are made to a system. One thing that you can count on is that the more valuable your data is and the more difficult it is to get back to where you started, the more likely something unexpected is going to happen to it. Hope for the best, but prepare for the worst. 3.2 Monitoring performance on Solaris Unix systems have a very strong core set of performance monitoring tools that have been around for some time. Sun has built on these tools and added a few of its own to maximize your ability to monitor and analyze a Solaris system. Table 3-1 presents a brief overview of the different tools available and their usefulness. Table 3-1 Key Solaris performance monitoring commands Command Description prstat Provides a summary of process activity vmstat 30 180 Provides statistical information from the Solaris virtual machine. The 30 sets the collection/reporting interval to 30 seconds and the 180 specifies the number of iterations to be repeated. This will cause vmstat to report statistics every 30 seconds for 1.5 hours. Chapter 3. Tuning 41 Command Description mpstat 30 180 Provides statistics on each of the multiple processors. This helps us determine whether some CPUs are working harder than others. The parameters are the same as vmstat. iostat -x 30 180 Provides statistics on the I/O generated on the system. This helps us determine whether some disks are more heavily utilized than others. The -x option provides extended statistics. netstat Provides statistics about the network. structures, and activity ps Returns a process list pstack Runs a process debugger Tip: Examining memory statistics and usage should always be your staring point. Memory issues can manifest itself in many different ways because the system itself is responsible for memory maintenance. So it’s possible that CPU issues and even I/O issues can be related to the overhead required to manage memory. For example, if the system is swapping, you may see high utilization of disks or CPU. 3.2.1 The prstat command This command shows what processes are active on the system, along with how much CPU time they are using. It shows what processor the processes are bound to, their size in memory, and their priority. If you have used the freeware tool top, this command works very similarly to it. Enter the command as follows: # prstat Unlike other stat commands, prstat is not required to have any argument. The output will be refreshed every 5 seconds by default and appear very similar to the illustration shown in Figure 3-1. Figure 3-1 prstat example 42 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 3.2.2 The vmstat command The vmstat command is probably the easiest and most often used way to get a quick look at the performance of your system. It provides a great balance of conciseness and completeness. In one snapshot, you get a valuable overview of each major aspect of your system. The name is a bit of a misnomer since it provides more information than just the virtual memory on your box. The command generates information about process queues, disk utilization, type of system calls, cpu utilization, page faults, swap utilization, and free memory available. The listing that vmstat generates contains a number of columns that display data about the state of the system. These columns cover the processes, memory, page, disk, faults, and CPU. Figure 3-2 shows sample vmstat output from a mostly idle system. There are 0 processes waiting to be executed in the run queue (column r), minimal paging (pi and po columns), no disk activity, and minimal CPU in use (all idle). Domino/Sametime usage is tracked as user CPU (that is, the us column). This command is a simple way to look at memory: # vmstat 5 You can simply specify a time interval to vmstat, and then let it run until you press Ctrl-C to stop it. Figure 3-2 A vmstat 5 example Tip: Always ignore the first line of any *stat command since it is just a summary since the system has been up. Accordingly, it doesn’t give you any indication as to the use of the system during the time period of your interest. The most useful vmstat outputs are summarized in Table 3-2. Table 3-2 Useful vmstat output columns Column Meaning r Number of runnable processes (waiting for CPU time) b Number of blocked processes (such as waiting for I/O and paging) w Number of runnable but swapped-out processes (normally 0) swap The amount of available swap space in kilobytes free The amount of available free physical memory in kilobytes re Page reclaims (memory pages taken from other processes) Chapter 3. Tuning 43 Column Meaning mf Minor faults pi Kilobytes paged into memory from the swap device po Kilobytes paged out of memory to the swap device fr Kilobytes of memory made available de Anticipated memory shortfall sr Pages sacnned by page-out scanner (should be close to 0) disks Number of disk operations per second. (Each sub-heading represents a device.) us Percentage of CPU time spent in user mode sy Percentage of CPU time spent in system mode id Percentage of CPU time spent idle A more in-depth examination of the columns and values is as follows: – Normally the r, b, and w columns contain fairly low numbers, if not 0. Larger numbers in these columns usually indicates a negative performance implication. The general rule of thumb is that if the run queue (r column) is greater than four times the number of CPUs in the system, you probably have a CPU constraint. This is described in the book Sun Performance and Tuning, by Adrian Cockcroft and Richard Pettit. – The swap, free, re, mf, pi, po, fr, de, and sr columns give information about the activity of the virtual memory manager. Solaris will keep a threshold for free memory so that it will always be able to service requests. Optimal conditions indicate there is an abundance of memory in the free column without much going on in the re, mf, pi, po, fr, de, and sr columns. If this is not the case, that means the virtual memory manager is having to work to maintain an availability of memory. – The us, sy, and id columns should give you a good snapshot of what the CPU utilization is like. Understand that the us or user column refers to user processes, of which Domino is one. To get a detailed view of each CPU, use mpstat. – You want the sr to be zero. A non-zero scan rate indicates that the system has encountered a page fault and does not have enough free memory to service it. When the scan rate column is regularly above 0, you may have to add more memory or reconfigure memory utilization so that there will be free pages for any faults that occur. Fortunately, the Solaris 9 system has come a long way in managing memory so that you do not have to perform this reconfiguration. 3.2.3 The mpstat command The mpstat command is a great way to see exactly what is going on with processor usage. The vmstat command gives you a good overview. However, if you need more specific information per processor, mpstat is the way to go. The mpstat command is a more accurate and detailed description of what is going on with each processor, while vmstat is an approximation of the whole. On a larger system that may house a couple of separate and distinct systems called Sun domains, there may be a good number of processors. In such a case, getting a look at the system per processor can be extremely useful for tuning and balancing. 44 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 The mpstat command generates a listing showing what each CPU is spending its time doing: for example, the distribution of system, user, wait, and idle time; system calls made; lock contention; interrupts; faults; and cross calls. There is some Solaris tuning that can be performed to balance the CPU load if it is unbalanced, but these tunings are not commonly needed. Instead, mpstat gives you a good idea of the efficiency at which the CPU is able to run. You should use mpstat to see what the actual CPU load is on the system, since vmstat will only report on the most heavily used CPU. The output shown in Figure 3-3 shows the key columns of interest for our purposes: Figure 3-3 An mpstat 5 example The interrupts are spread unevenly across the CPUs (intr column). An uneven distribution may point to a problem with the system’s “SBus” configuration (that is, the most common Solaris system bust). The number of failed mutex enters (smtx column) shows internal lock contention. Low numbers are common, but when it gets into the thousands or higher—or if there appears to be a correlation between high system time and high smtx levels—then you have some contention that should be addressed. Monitor the system calls (syscl column). System calls are the basic functions upon which the kernel runs. Some people use a rule of thumb of about 5000 per processor, but this may not yield a very good picture. The best practice is to see whether the load you are placing on the system corresponds well with the number of calls being generated. If not, this could be indicative of a problem. Use percentage statistics (the total of the usr and sys columns) to identify additional CPU processing needs. The time spent in the wait (wt column) determines whether the system has to wait in order to complete processing. You want to minimize this because it means that CPU utilization is essentially being wasted This can often be caused by I/O not being available or by having to wait on other processes. As you see from Table 3-3, this command produces a lot of columns. But you may care most about the following columns: Table 3-3 Useful mpstat output columns Column Meaning xcal Interprocessor cross-calls intr Interrupts csw Context switches icsw Involuntary context switches Chapter 3. Tuning 45 Column Meaning smtx Spins on mutex locks usr Percent user time sys Percent system time wt Percent wait time idl Percent idle time 3.2.4 The iostat command The iostat command provides information about disk utilization. It is entered as follows: #iostat -x 5 The main columns to look at are svc_t and %b. The svt_t column describes the average service time (in milliseconds) required to complete an I/O request. The %b column indicates the percentage of time spent servicing requests. Using the two statistics, you can get a good picture of the I/O utilization for each device. See Figure 3-4. Figure 3-4 An iostat -x 5 example Utilization numbers may often reflect close to 100% utilization; however, this does not necessarily indicate a problem. See the Sun article, What does 100% busy mean? for more details. You can find this article at: http://www.sun.com/sun-on-net/itworld/UIR990801perf.html Sample iostat output is shown in Figure 3-5for an idle system. Figure 3-5 Sample iostat output for an idle system 46 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 3.2.5 The netstat command The netstat command provides information about the network system. There are three different areas you can use to identify problems of network activity. First, you can look at the number of packets being transferred and the number of collisions being experienced by the network interface. netstat -i or netstat -i <interval> Both forms of this command and their sample outputs are shown in Example 3-1. Example 3-1 Netstat -i command and output # netstat -i Name Mtu Net/Dest lo0 8232 loopback hme0 1500 9.33.85.89 # netstat -i 5 input hme0 packets errs packets 3826075 1684 3762737 3 0 1 5 0 2 3 0 2 4 0 2 Address localhost 9.33.85.89 output errs colls 0 0 0 0 0 0 0 0 0 0 Ipkts Ierrs Opkts Oerrs Collis Queue 5863614 0 5863614 0 0 0 3825995 1684 3762715 0 0 0 input (Total) packets errs packets 9693029 1684 9629691 858 0 856 51 0 48 51 0 50 48 0 46 output errs colls 0 0 0 0 0 0 0 0 0 0 The first example gives the total statistics since the network interface was started. The second example gives a running set of statistics over time. By looking at the number of packets being transferred and the number of collisions being experienced, you can get a good idea of whether the network interface may be overloaded and whether the current rate is above or below the average. netstat -k Although this form of the netstat command provides a wealth of information, we will focus solely on the ability on the nocanput variable. This counter indicates that packets were not accepted inside the kernel. Under these conditions, it would be important to increase the size of the streams queue. To see the data, use the command # netstat -k |more Then search for noncanput errors, which are the relevent statistics related to your interface, as shown in Example 3-2. Example 3-2 Netstat -k output hme0: ipackets 3844344 ierrors 1684 opackets 3765380 oerrors 0 collisions 0 defer 0 framing 0 crc 0 sqe 0 code_violations 0 len_errors 0 ifspeed 100000000 buff 0 oflo 0 uflo 0 missed 1684 tx_late_collisions 0 retry_error 0 first_collisions 0 nocarrier 0 nocanput 0 allocbfail 0 runt 0 jabber 0 babble 0 tmd_error 0 tx_late_error 0 rx_late_error 0 slv_parity_error 0 tx_parity_error 0 rx_parity_error 0 slv_error_ack 0 tx_error_ack 0 rx_error_ack 0 tx_tag_error 0 rx_tag_error 0 eop_error 0 no_tmds 0 no_tbufs 0 no_rbufs 0 rx_late_collisions 0 rbytes 1498527458 obytes 2011478816 multircv 0 multixmt 0 Chapter 3. Tuning 47 netstat -sP tcp We are looking for any network buffer overload errors caused by not allocating enough space to the TCP/IP buffers of the network. A properly running system allocates the minimal amount of memory to network interfaces to support load on the system We are specifically interested in any dropped frames from the TCP/IP stack (such as tcpTimRetransDrop, tcpHalfOpenDrop, tcpListenDropQ0, and tcpListenDrop), as highlighted in Figure 3-6. Figure 3-6 Sample netstat -sa output 3.2.6 The ps command The ps command gives a listing of the processes running on the system and attribute information regarding them. This can be quite helpful not only in seeing basic information, but also in getting a snapshot of CPU, memory, time spent running, and more. The two specific commands are shown in Example 3-3: Example 3-3 Specific ps commands ps -elf ps -e -o user,pid,ppid,pcpu,vsz,pmem,psr,osz,pset,fname Tip: For long command lines, you can set up an alias or a shell script that acts as a wrapper for the command. To set the alias, add the following line to your .profile alias bigps=’ps -e -o user,pid,ppid,pcpu,vsz,pmem,psr,osz,pset,fname’ To create a wrapper script, enter: # echo “ps -e -o user,pid,ppid,pcpu,vsz,pmem,psr,osz,pset,fname” > bigps # chmod 755 bigps 48 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 3.2.7 The pstack command The pstack command is a bit advanced, but we will consider it very briefly just because it can be so useful. If you want to get a closer look at what a process is doing and possibly why there may be a bottleneck, use the command: # pstack <pid> Successive snapshots of the process may be able to give you a better idea if any particular stack is having a problem processing. 3.2.8 The SE ToolKit The SE toolkit is a freely-downloadable software package developed by Sun performance experts. In addition to using stat commands, this toolkit can help you to collect more “intelligent” performance information. You can download the toolkit and its documents from http://www.setoolkit.com/ 3.2.9 Solaris internals The Solaris Internals website offers free tools that expand upon the tools mentioned here. The additional tools cover memory, file systems, and kernel debugging. Generally these tools are most valuable when more specific information is needed or when a problem needs to be addressed. http://www.solarisinternals.com/si/tools 3.3 Lotus Domino 6 performance monitoring options Domino continuously generates statistics that you can use to monitor system activity and platform use, and it includes many server-monitoring features that work together to inform you about the processes, networks, and use of the Domino system. You can monitor the system using one of three tools: the Domino Administrator, the Web Administrator, or the server console. For example, you can use the Domino server monitor and statistics charts from the Domino Administrator to view graphical representations of system status. Another example is the server console, which provides a representation that uses your predefined colors and text attributes to illustrate the status of a process. The Domino Administrator includes these system-monitoring tools that you can use to configure, view, and track the Domino system: Domino server monitor The Domino server monitor displays real-time statistics and provides a visual representation of the status of servers and their tasks. You can view all servers or a subset of servers. Finally, you can view the status by state or by timeline. Statistics Domino gathers statistics that show the status of processes currently running on the system. You use these statistics, along with the predetermined statistics thresholds, to monitor both your Domino system and platform statistics. The Domino 6 platform statistics tool is especially useful in eliminating switching between Domino and Solaris statistics. Domino platform statistics can collect the same information generated by Solaris statistics such as kstat, iostat, vmstat, and netstat. In addition, the generated information is integrated with monitoring configuration (Events4.nsf) and reports(Statrep.nsf). This platform statistics function is automatically enabled in Domino 6. Chapter 3. Tuning 49 The platform statistics are grouped into five categories: – Logical disk statistics – Memory statistics – Network statistics – CPU statistics – Miscellaneous system statistics For some statistics, average, minimum and peak values are calculated. There are sets of Domino console command to control the platform statistics. The commands are: – To view platform stats > show stat platform or you can also specify an object name that you are interested in how stat platform.<object>.* – To set sampling interval > platform time 15 This means that you are setting the interval to 15 minutes. The default is 1 minute. – To reset counters > platform reset – To pause gathering > platform pause – To resume gathering > platform resume A general guideline for some platform statistics that are especially meaningful to the Solaris platform are summarized in Table 3-4. Table 3-4 Healthy statistics ranges cheat sheet for Solaris Platform Stat name Normal Value LogicalDisk.ServiceTime Single/RAID < 40 msecs LogicalDisk.PctUtil < 20 % Memory.RAM.PctUtil < 5% Network.PctUtilBandwidth < 30% of total bandwidth Network.PctCollisionRate < 2% of transmitted packets Process.ActiveDomino.TotalCpuUtil < 80% of combined CPU util System.CPUQueueLen < 1 per processor System.PctCombinedCpuUtil < 95% System.PctTotalUserCpuUtil < 75% of Total CPU utilization Memory.PagingFile.PctUtil < 40% Monitoring server tasks You can collect and record information about the Domino system. The Event Monitor task determines whether an Event Handler has been configured for the event. If it has, the task routes the event to the specified person, database, or server-management program for 50 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 processing. The Statistic collector task gathers Domino server statistics and creates statistics reports in the Monitoring Results database (STATREP.NSF) or in another database that you specify. The ISpy task executes TCP server and mail-routing event generators. Monitoring configuration documents These documents define and configure what constitutes an event and how the event is handled. They also allow you to customize the messages that appear on the console when an event occurs. Monitoring databases These databases store monitoring documents, information, and results. The Monitoring Configuration database (EVENTS4.NSF) stores the documents that you use to set up monitoring. It also includes information about statistics, statistic thresholds, and event messages. The Monitoring Results database (STATREP.NSF) stores the gathered statistics reports and can be configured to store information about logged events. The log file (LOG.NSF) stores the server's log documents. Tip: To customize the appearance of the Domino server console: Create a Server Console configuration document for the server you are monitoring, which allows you to specify the text, background, and color attributes that the Domino server console uses to display monitoring information. By default, the Domino Administrator server console uses the same attributes, but you can override the defaults and customize the appearance of the Domino Administrator server console. IBM Tivoli® Analyzer for Lotus Domino IBM Tivoli Analyzer for Lotus Domino includes two integrated system management tools: the Sever Health Monitor, which offers real-time assessment and recommendations for server performance, and the Activity Trends tool, which provides data collection, data exploration, and resource balancing. You can use these tools to manage servers and databases to ensure better server performance and build a baseline for current and future needs. Note: The IBM Tivoli Analyzer for Lotus Domino requires a separate license. See more details about IBM Tivoli Analyzer in the LDD Today article, Start using Domino 6 Server Health Monitoring Now by Carol Zimmet, available at: http://www-10.lotus.com/ldd/today.nsf/62f62847467a8f78052568a80055b380/453d9e87c12cda080 0256bce003f3c6e?OpenDocument&Highlight=0,domino,server,health,monitoring 3.4 Tuning Considerations Solaris 9 represents the simplest version of Solaris to tune because many parameters are now dynamically tuned by the OS or preconfigured. This section reviews the important tuning parameters are included in Solaris 9 for Domino 6. You also should consult the articles at the Domino for Sun Solaris website at http://www.lotus.com/dominosolaris Chapter 3. Tuning 51 These articles cover broader topics and different versions of Solaris. 3.4.1 Solaris kernel tuning This section covers several variables that are often tuned in the Solaris kernel. msgsys and rlim_fd_max As mentioned in Chapter 2, the parameters in Example 3-4 must be implemented at their respective levels. However, the default value of rlim_fd_max has been increased to 65536 in Solaris 9. Example 3-4 Default value of rlim_fd_max set msgsys:msginfo_msgtql=1024 set rlim_fd_max=65536 fsflush You can tune the system deamon, fsflush, with these two parameters: – tune_t_fsflushr: This controls how frequently fsflush runs. It specifies the number of seconds between invocations. – autoup: This parameter is used in tandem with tune_t_fsflushr. It controls how much memory is examined at each fsflush execution. Example 3-5 depicts an fsfluch tuning. Example 3-5 set tune_t_fsflushr=1 set autoup=100 Some experts recommend setting tune_t_fsflusher to about four or five instead of one. The thing to watch is excessive CPU consumption by the fsflush process. If fsflush is burning large amounts of CPU time constantly, consider increasing the value for tune_t_fsflusher. However, do not make it too large. You may get periodic spikes of load when the fsflush process runs. segmap_percent The kernel maps 12% of physical memory by default to be used for file cache. Segmap_percent adjusts the percentage of memory that the kernel will map into its address space for the file system cache. Non-segmapped memory is considered free. Thus, there will be a balance between the efficiency of the cache utilization and need for process memory. Experiment with this value, starting with values around 25, as follows: set segmap_percent=25 A read/write call that is segmapped can increase performance and reduce I/O. Enter the command #netstat -k|more to measure the cache hit rate and determine whether poor utilization could be the cause of the performance degradation, as shown in Example 3-7 on page 53. 52 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Figure 3-7 Segmapped performance measurement Search for the segmap module. Segmap efficiency can be computed by ((get_reclaim1-get_reclaim0)+(get_use1 - get_use0))/(getmap1-getmap0) Alternatively, one can watch the amount of page-in activity as a quick and dirty means of identifying possible value from increasing this value. Tip: Before making any changes in the /etc/system file, make a backup copy of the file, as follows: # cp /etc/system /etc/system.good If the change in /etc/system causes the system to become unbootable, you can recover with following command: ok boot -a Finally, you can specify the name of /etc/system file that you want to use. You can specify system.good here: Name of system file [/etc/system]: /etc/system.good Dispatch Table The kernel scheduler manages when a process is scheduled to run. Classically, CPU intensive processes are pushed to a lower priority. This can occur because the time quantum allocated to the process expires with the process still running. The OS assumes that it would be more beneficial to run other processes to maintain a high level of interaction. This can cause a process using a good deal of CPU to hold resources while running at a lower priority. Increasing the time quanta for each process can may be beneficial in avoiding this problem. To see your dispatch table, run the command Chapter 3. Tuning 53 # dispadmin -c TS -g There are a existing dispatch tables that can be implemented instead of the defaulted table. You can download an implementation from: http://www.solarisdatabases.com 3.4.2 Solaris file system tuning For purposes of explication, this book focuses entirely on using UFS as the file system. Solaris supports many different type of file systems, each with their own benefits and costs. Many of the points touched upon here are transferable to other file systems in concept, if not in syntax. Logging Logging gives you the ability to adding journaling capabilities to a UFS file system. Logging forces the system to store transactions before they are applied to the file system. This prevents the file system from becoming inconsistent and eliminates the need to run fsck after a system crash. Logging has also been found to create much better overall file system performance. To implement this function, use the logging option in the /etc/vfstab file. /dev/dsk/c4t1d0s6 /dev/rdsk/c4t1d0s6 /notesdata ufs 1 yes logging noatime Domino maintains it’s own access time date so it isn’t necessary to rely on the file system to maintain the atime metadata. This is implemented best through the /etc/vfstab file so that the mount options are used automatically. For example: /dev/dsk/c4t1d0s6 /dev/rdsk/c4t1d0s6 /notesdata ufs 1 yes noatime tunefs Tip: To see existing settings, use the command: # fstyp -v <device> | more for example: # fstyp -v /dev/rdsk/c0t0d0s0 | more Making changes to a file system can have a very positive results when you uncover a system bottleneck. Be warned, however, that each system’s needs are very different and that the implementation of file system tuning is very much a system-by-system process. You can use the tunefs command to make changes to a file system. These settings vary widely between systems. The syntax of the tunefs command is: # tunefs -a maxcontig -d rotdelay-e maxbpg -m minfree -o [space | time] raw-device a maxcontig The maximum number of logical blocks, belonging to one file, that will be allocated contiguously before inserting a rotational delay. Increasing this value can improve read-ahead performance. However, a larger value can have a negative impact for transaction processing (such as using smaller reads). 54 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 To see the current settings, use the a maxcontig command. d rotdelay The expected time to service a transfer completion interrupt and initiate a new transfer on the same disk. This is used to decide on the rotational spacing to place between successive blocks in a file. This setting should only be modified at the instructions of the hardware manufacturer. e maxbpg The maximum number of contiguous blocks a file can allocate out of a single cylinder group. This is used avoid the possibility that a file will use all of the blocks in a cylinder, thus causing delays for all other files. In a file system with a few large files, you may get a performance benefit by allowing the file to use more of cylinder group before having to perform unnecessary seeks. m minfree The percentage of file system held back from normal users. This can be set to 0%. However, practice this can limit the superuser’s ability to maintain files and drastically reduce throughput. o [space | time] Optimizesthe file system for space or access time. This is done by either minimizing fragmentation or minimizing the time spent allocating blocks. Eliminate fragmentation UFS is a block-based file system that can build up disk fragmentation over time. Disk fragmentation can cause read/write performance issues. If you suspect fragmentation, use the command shown in Example 3-6 to run a disk-to-disk dump restore. Example 3-6 #ufsdump 0f - . | ( cd dest_dir && ufsrestore xf -) #mv dest_dir notes_dir ufs_HW For systems with high-speed I/O devices, it may be useful to increase the number of waiting processes for the disk. The default for Solaris 9 is low enough that this may be beneficial for high-performance hardware underlying the filesystem. Each process that is blocked from the write queue increments the ufs_throttle’s counter. Example 3-7 depicts the command to check this: Example 3-7 #adm -k<<EOT > ufs_throttles/D > EOT physmem 1f011 ufs_thottles: ufs_throttles: 0 The system will keep blocking processes until it reaches the low-water mark (ufs_LW). By default they are set at 16 MB for ufs_HW and 8 MB for ufs_LW. Adding the following lines in the /etc/system file sets the values at their default levels, as shown in Example 3-8 on page 56: Chapter 3. Tuning 55 Example 3-8 set ufs_LW=8388608 set ufs_HW=16777216 Spreading I/O RAID configurations can be used to improve general overall performance and reliability. However, you do not necessarily need a RAID to spread out the I/O for a system. Unix offers the ability to configure links so that certain files or directories can exist on another disk. This allows for more power in allocating I/O and better overall performance. This feature also exists within Domino, but the ease of configuration, low overhead, and flexibility that Unix allows with this makes it a very attractive choice. Use the command in Example 3-9 to implement this feature: Example 3-9 Configuring file or directory links # mv <data dir>/<file> <target dir> # ln -s <target dir>/<file> <data dir>/<file> The greatest limitation with this approach occurs when you have a single database that overwhelms the I/O on a device. Because you cannot divide the database between different devices using links, it becomes impossible work around this limitation. 3.4.3 Network tuning The key topics for tuning the network include the following: TCP tuning parameters Tuning TCP buffering Turning off network routing TCP tuning parameters /usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q 2048: The default maximum number of pending TCP connections waiting to be accepted by a TCP listener /usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q0 2048 The default maximum number of incomplete pending TCP connections for a TCP listener /usr/sbin/ndd -set /dev/tcp tcp_time_wait_interval 60000 Time in milliseconds before a connection in TIME_WAIT is dropped /usr/sbin/ndd -set /dev/tcp tcp_xmit_hiwat 32768 The size of the transmit buffer /usr/sbin/ndd -set /dev/tcp tcp_recv_hiwat 32768 The size of the receive buffer /usr/sbin/ndd -set /dev/tcp tcp_slow_start_initial 2 The initial size of the congestion window−useful when networking with Windows 56 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Note: The ndd commands can be run dynamically or run upon system restart. To execute them automatically after each system reboot, you can write a script and name it something like network-tuning and create a link to a file in /etc/rc2.d. For example, # ln -s network-tuning /etc/rc2.d/S99network-tuning set tcp:tcp_conn_hash_size=1024 Add to the /etc/system file (increases the size of the hash table that holds the TCP connections). An appropriately sized table can speed connection lookups. Tuning TCP Buffering Unpredictable network performance and errors can result from dropped packets at the kernel. Verify that the syncq has enough capacity through the use of netstat -k (in monitoring tools). If so, it may be wise to increase the queue. Sun advises testing in increments of 10 messages at a time. An example follows: set sq_max_size=12 Turning off network routing By default, a system with more than one network card will turn on routing between the cards. You can disable this setting by creating a a notrouter file. Run the following command and routing will be disabled the next time the system starts: # touch /etc/notrouter 3.4.4 Other tuning You also can tune the swap partitions and turn off unused tasks. This section covers these tuning options. Swap Properly configured swap is extremely important during periods when there may not be enough memory. The swap partition can require some of the greatest I/O to maintain decent performance. On the positive side, it is quite easy to manage swap partitions and swap files. To check your swap configuration use: # swap -l To implement a swap partition, simply mount that partition as a swap device. To implement the swap device so that it loads automatically on system start, add the following line to the /etc/vfstab file: <disk device> - - swap - no - - - swap - no - For example: /dev/dsk/c0t1d0s5 Tip: Swap uses a round-robin method to allocate pages. Thus can be useful to implement on different spindles. Additionally, be careful when sizing swap. Too much swap can create a condition where hard pageouts occur too easily. A general rule of thumb is to use two times the physical memory. Chapter 3. Tuning 57 Turning off unused tasks To disable services that you aren’t using, move their startup scripts out of the /etc/rc3.d and /etc/rc2.d directories. Below is a sample listing of scripts that may not be useful to you: S73cachefs.daemon: - improves network file system performance through caching. S73nfs.client: a remote file system client S88sendmail: email S15nfs.server: remote file system server S76snmpdx: Simple Network Management Protocol master agent S89sshd: Secure Shell daemon S50apache: Apache webserver S80mipagent: Mobile IP agent S90samba: SMB/CIFS (Windows fle sharing) 3.4.5 Domino Tuning This section covers some tuning options within Domino. There is no magic formula that will work for everybody. You need to experiment with these options to achieve peak performance. Remove any unused server tasks Many server tasks are enabled by default. If you don’t need a task, disable it. It will simplify the things that Domino has to do and increase the available resources. Enable Run web agent concurrently? To improve concurrency of web agents, make sure the Run web agents concurrently? option is enabled. To find this setting, open the server document and select to the Internet Protocols and Domino Web Engine tab. Then, you will find Web Agents section. See Figure 3-8. Figure 3-8 Web Agents Increase the number of HTTP threads When web server responses slow, you may increase the number of threads. It can help increase the concurrency and performance of the web server. The downside is that increasing the number of threads increases the memory used by the HTTP stack. Therefore, you should only use this if there is a significant benefit. The default is 40, and this setting is located in your server document. Select the Internet Protocols and HTTP tab and look for the Basics section, as shown in Figure 3-9 on page 59. 58 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Figure 3-9 HTTP number of active threads Increase Maximum cached users Domino Server stores successfully authenticated users information such as name, password, and groups in memory. The default is 64. If your web server is accessed by a multiple of thousands of users concurrently every day, you may consider increasing this number. This setting is also located in the server document. Select the Internet Protocols and Domino Web Engine tab. Then look for the Memory Caches section. See Figure 3-10. Figure 3-10 Maximum cached users Increase HTTP listenbacklog queue Users may experience connection time-outs from a browser to a Domino server. Increasing this number may help alleviate the time-out problems. This setting is in the server document. Go to the Internet Protocols and HTTP tab. Look for the Network Settings section. See Figure 3-11. Figure 3-11 Listen queue size Chapter 3. Tuning 59 Attention: In addition to the listen queue size setting, you must also increase tcp_conn_req_max_q tcp_conn_req_max_q0 to at least value that you specified in the listen queue size setting. Disable the browser caching Browser caching can create a bottleneck if your site is not frequently revisited by the same user. Set the settings in Figure 3-12 in the Web User Preferences section of the Domino Web Engine tab in the Server document or Web Site document: Figure 3-12 Disabling browser caching PercentAvailSysResources=<percentage> ConstrainedSHMSizeMB=<megabytes> These parameters are implemented in the notes.ini file. Their function is to limit the memory Domino sees as being available to use. Choose whichever is more convenient. Domino is a 32-bit application that runs on the 64-bit Solaris operating system. Because of the limits of 32-bit architecture, Domino can only address a maximum of 4 gigabytes. If your system has 4 gigabytes or more, it will help to limit the amount of memory Domino tries to address. The default recommendation is 3 gigabytes, which would leave a gigabyte of overhead. Also, if your system has less that 4 gigabytes of memory, there is value in limiting the memory Domino sees as being available so that other applications and even the OS itself do not have to contend for the same resources. Notes_PRIVATE_DPOOLSIZE Memory can become fragmented and more handles need to be created as old handles cannot be reused. Eventually the total number of memory handles can be exhausted. These issues can be addressed by creating a large private dpool. To implement this, enter: # Notes_PRIVATE_DPOOLSIZE=26214400 # export Notes_PRIVATE_DPOOLSIZE JavaMaxHeapSize=<size in bytes> Add this to the notes.ini file to increase the maximum size of your Java heap. A larger heap will decrease the overhead associated with memory maintanence activities. JavaStackSize=<number_of_bytes> The default value is 409600 (400 K) which is fine in most cases for the Java execution stack. However, increasing the stack size may be important for deeply-nested routines. 60 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Additional features While these options do not really fall under the heading of tuning, their implementation can greatly increase performance. Transaction logging When implemented correctly using a separate disk and controller, the logging of transactions to be applied later in a batch can increase performance significantly. Other benefits include: – A shorter recovery time after process failure because consistency checking isn’t necessary – The ability to use third-party software to backup individual transactions to get exact point-in-time recovery. Sametime Mux Sametime Mux (as described earlier) is software that allows for the connection management of Community Services to be handled by another single machine or a group of machines. Connections and their management are very expensive in terms of resources. Mux relieves the Sametime server of the burden of having to deal with numerous connections because all communication is channelled into a single connection. Partitioning and clustering Partitioning and clustering offer the ability to take advantage of resources and provide a more scalable service. For example, the 4 GB limit for 32-bit applications may make it necessary to implement additional partitions to take advantage of hardware capabilities. These partions can be used as a cluster mate to provide a more scalable service on the same physical machine. Architecturally, there are a number of strategies that use these solutions to improve your community’s performance, availability, and reliability. 3.5 Sample testing To show some of these tuning concepts and details in practice, we performed some basic system tuning for this Redpaper. The specific details of the test environment utilized in this section were: Hardware: Sun Ultra 60 with dual sparcv9 processors operating at 450 MHz. Physical memory was one gigabyte. OS Configuration: Swap space was configured for 512 megabytes on the primary disk. The system had two SCSI hard drives configured to operate independently, with both Sametime/Domino and the OS on the same drive. The system was equipped with a 100 Mbps Full Duplex ethernet interface. We installed Sun Solaris 9, and did not apply any Sun maintenance/fixes/patches. Software configuration: We then installed Domino 6.0.2 CF1 with out-of-the box functionality. Sametime 3.1 was then installed on top of this configuration. Note: It is important to note that this performance testing was exectuted to showcase tuning concepts and methodologies only. This testing was not intended to demonstrate the scalability of Sametime on Solaris in any way. The hardware actually used was a rather old workstation-classed Sun system, which is not remotely representative of the performance of current Sun server-class equipment. . Chapter 3. Tuning 61 3.5.1 Performance load description To put a load on our system, we used an in-house tool that simulated a whiteboard meeting. The workload performed its tasks in three main stages: schedule meetings, prepare the background load, and simulate an active server. Phase 1: Schedule meetings The server workload ran before the test and scheduled the meetings. When the meetings were scheduled, a 5 MB Freelance presentation with 21 slides was also uploaded to the server. Although each meeting used the same presentation, each received a unique copy of the presentation. Phase 2: Join the meeting We started the users over a given ramp-up period. Then, we waited for the meeting to begin. Phase 3: Simulate an active meeting server We ran the whiteboard workload for 30 minutes. This period represented the pure meeting load. The presenter flipped through all of the pages of the presentation at the rate of one per minute. The actual whiteboard presentation started as soon as the presenter joined the meeting. During some of the tests, two stages were used where Phases 2 and 3 were repeated. This activity simulatee a background load after the first stage had completed the meeting activity. We used single and multiple drivers to run the test against the server. Each test user performed the following tasks: 1. Connect to the server. 2. Log on to the server. 3. View the list of active meetings. If there are too many meetings to fit into a single page, download the next page of meetings. 4. Open a link to the meeting. 5. Download the meeting applet. 6. Start the meeting client on the driver. (The standard applet was downloaded, but the test ran the custom client.) 7. Start turning pages on the whiteboard (Presenter). A previous study using the same tools had been conducted by the Lotus Product Introduction Engineering team. That study found that CPU performance was the bottleneck that ultimately prevented having more users on the system. CPU utilization was by far the heaviest during the join period for those tested, as shown in Figure 3-13 on page 63. 62 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Figure 3-13 Sample performance profile from an Intel® server Note: The performance results shown above are not from a Sun system. Instead, they were taken from an Intel-based server. This chart is included to highlight the points during the simulated load scripts in which the system would be expected to see the highest impacts. 3.5.2 Performance testing and tuning This section provides the results of our actual performance tests, and details the tuning settings/parameters that we applied to improve performance. Our overall tuning strategy was as follows: Test the capabilities of a baseline system using “out-of-the-box” default settings and parameters. Apply performance tunings based on the baseline results. Re-test the “tuned” system to verify performance improvements. Out-of-the-box performance numbers A single Sametime server was installed. Three separate loads of 200 and 250 concurrent users were tested on the system. We found that performance was extremely limited. These utilized two drivers, which effectively limited the joining of meetings to two users at a time. The stress test caused the system to consistently fail serving connections and not handle the overall load gracefully. System calls were extremely high for the load. (This result can be obtained through vmstat or mpstat.) The average for this period was over 11,000. This is indicates that the kernel made numerous calls for what is essentially a low load. The average CPU usage was 84%. If we looked at usage on a 1-10 scale corresponding to percent usage (so that 5% was on the first level and 85% on ninth level), the cpu ran at peak capacity more than any other level. Chapter 3. Tuning 63 There were 3866 tcpListenDrop errors over the course of the test. Less than 55% of the simulated users completed successfully. Resource utilization followed the profile of previous studies. Joining the meeting was the greatest stress on the system by far. Paging activity consistently reached the 300s throughout the join period Over 100 MB of memory was left unused during the test. Average service was under relatively normal, but variance was quite high. Table 3-5 Average service times for users two-driver system Service Average Times Attach file 1453 ms Schedule meeting 2013 ms Connect to meeting center 320 ms Total time to join meetings 24123 Activate a page turn 1252 ms Note: The data in this table has little statistical significance. Because of the number of failed user simulations in each test, the data cannot be used as a measure of comparative performance. Average service times tended to degrade as the servers load increased. Thus the service times trended higher as more users were placed on the system. Two additional loads of 100 and 200 were run from only a single driver with better results. There were no dropped connections. All simulated users were served. The average CPU utilization during the period users joined the meeting was was 63%. Paging activity was limited to approximately half of the activity experienced in the two driver test. Table 3-6 Average service times for users on single-driver system Service Average Times for 100 users Average times for 200 users Attach file 1974 ms 2214 ms Schedule meeting 2043 ms 2163 ms Connect to meeting center 804 ms 908 ms Total time to join meetings 23747 ms 22409 ms Activate a page turn 1392 ms 1863 ms Note: The service times from the 100 user sample and the 200 user sample did not represent a significant statistical difference in any of the categories. 64 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Data Summary and Tuning CPU We can certainly see that CPU is used heavily, although not necessarily to the maximum level. The two questions the data may lead us to ask are: 1. Can you do anything to make the application work more efficiently? 2. Is there something outside of the application causing unneeded work? To answer these questions, some changes to the application that may help the application level need to be considered. The second point will be dealt with by looking at possible problems in how the other resources are used that could cause the extra CPU usage. Sametime Meeting Services are built primarily using a Java as the application engine. To minimize the need to thrash for resources, the following notes.ini parameters were implemented to increase: – JavaStackSize=512000 – JavaMaxHeapSize=314572800 To increase the performance of the HTTP engine, the following tunings were applied: – Disabled browser caching – Increased the number of HTTP threads to 60 – Increased the Listen Backlog to 1024 The following change was made to Domino: – Implemented Notes_PRIVATE_DPOOLSIZE=26214400 in the notes.ini I/O It is interesting to note that performance from a user standpoint was statistically the same whether there were 100 or 200 users on the system. The amount of I/O wasn’t large enough to indicate that the hardware was anywhere nearly overwhelmed. Given this fact and the fact that CPU utilization percentage was in the mid-80s, you might conjecture that the main bottleneck was a problem handling the amount of network I/O traffic: the dropped connections were certainly a good indication of that. The key two priorities were (1) to increase the amount of I/O the system can handle by giving it the structures it needs and (2) to speed the disk I/O along faster by making it more efficient. The following ndd parameters were implemented in a startup script to match up with the HTTP settings made earlier and to increase data transfer efficiency: – /usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q 2048 – /usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q0 2048 – /usr/sbin/ndd -set /dev/tcp tcp_xmit_hiwat 32768 – /usr/sbin/ndd -set /dev/tcp tcp_recv_hiwat 32768 The data directory was moved to the unused disk. The stlog.nsf, vpuserinfo.nsf, stauths.nsf and the names.nsf were linked back to the original disk to spread the I/O. Filesystem logging and noatime options were implemented to reduce CPU usage and I/O usage and to increase I/O speed, as follows: /dev/dsk/c0t1d0s7 /dev/rdsk/c0t1d0s7 /notesdata ufs 1 yes noatime,logging Chapter 3. Tuning 65 Memory The amount of paging activity indicates that memory was not being used effectively. The unused memory and the activity led to the conclusion that more efficiently using memory could lead to performance gains by alleviating I/O and CPU utilization. The following were implemented in the /etc/system file. • • • Set segmap_percent=45 Set autoup=100 Set tune_t_fsflushr=1 3.5.3 Performance results after tuning Loads with a single driver with concurrent loads of 100 and 200 users were run to compare against the earlier tests. These tests show that there was a significant improvement in performance. Service times showed overall significant improvement No significant paging activity was observed. The average for page out was 1.4 k/sec and less than 1 k/s for page ins. CPU utilization for the loads were half of the pre-tuned levels. Average CPU usage during the most intensive period was 32%. And normalized maximum CPU usage was never higher than 51%. The were no indications of I/O bottlenecks Table 3-7 Performance results after tuning Service Average Times for 100 users Average times for 200 users Attach file 1510 ms 1437 ms Schedule meeting 1958 ms 1922 ms Connect to meeting center 358 ms 416 ms Total time to join meetings 19336 ms 18959ms Activate a page turn 1011 ms 875 ms Further testing with two and more drivers loaded completed with high success and yielded some interesting results. There was a strong relationship between the CPU utilization during the periods of joining meetings and the rate at which users are joining meetings. Shorter ramp-up times or the use of more drivers had a definite additive effect on the CPU. Maintaining active users did not directly decrease modal performance (what the user was most likely to experience) in I/O heavy areas such as turning pages on the whiteboard. However, it did increase the likelihood of experiencing single instances of bad performance. Further tuning of network I/O could possibly alleviate this issue. A greater number of users did not have a strong effect on CPU processing. The CPU utilization rose significantly only when direct activity associated with the users or maintenance required limited resources. The was a direct relationship and seemingly linear relationship between the number of concurrent users and memory usage. 66 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Testing eventually ended without reaching limits of the scaling abilities of Sametime on Solaris. The last stress test had 1000 users concurrently attending meetings using 4 drivers. CPU utilization during the join phase averaged 69%. Memory eventually decreased over the course of the meeting simulation until paging activity started to be affected. This occurred during the page-turning portion of the meeting at the end of the test. The effect was to increase the usage of the CPU, but it is difficult to say that it affected performance because service times steadily decreased during that time. See Figure 3-14. Figure 3-14 CPU utilization over the course of the stressing the system with 1000 users During the course of stressing the system, service times remained stable. It is logical to think that that overall performance may falter as more concurrent users on the system joined. However, service times remained stable during this phase. See Figure 3-15. Figure 3-15 Total time to join a meeting Chapter 3. Tuning 67 Resource consumption and utilization appeared stable for each part of the overall test. The only phase that seemed to indicate a possible problem was the page turns. Performance remained good throughout, but CPU consumption increased at the very end of the test. At that point, it appeared that memory consumption was greater than availability and drove the server to start paging. More testing may indicate that we were very close to a memory bottleneck that would have affected performance. If that was the case, the next course of action could have been to further tune the system to limit the memory consumption or test the point that the CPU can handle more, More tuning could have definitely have been pursued and could have further enhanced the performance of the system. See Figure 3-16. Figure 3-16 The service time of turning pages during t e testing 3.5.4 Testing conclusions: Overall, we have reached several conclusions from our basic performance testing: Sametime scales well on Solaris. Tuning can have a major impact on performance. Servicing more instant messaging users is relatively cheap on resources, but servicing more meetings is relatively expensive. Other than joining meetings, the CPU usage was taxed very little by Sametime activity. Overhead associated with maintaining resources can be considered the second most likely activity to cause a bottleneck for the CPU. Due to the fact that meeting times will be more distributed in a real environment, memory or I/O is likely to be the cause of a bottleneck in production environment. 68 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Non-concurrent meetings did not appear to affect each other’s performance. This matches the results found by the earlier testing by the Product Introduction Engineering team I/O issues are more likely to manifest themselves in discrete instances of lack of service and long service times (rather than a continuous problem). Indications of poor I/O management are not always continuous and obvious. Network performance relies heavily on having available processing power. However, it is again important to note that this performance testing is only executed to showcase tuning concepts and methodologies. This testing is intended not to demonstrate the scalability of Sametime on Solaris in any way. The hardware used is not remotely representative of the performance of current Sun server-class equipement. Therefore, the number of concurrent users/response times shown are not representative of what one should expect from a server-class Sun system running Sametime. Chapter 3. Tuning 69 70 Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Solaris 9 Back cover Lotus Instant Messaging and Web Conferencing (Sametime) 3.1 for Sun Features and benefits of running Sametime 3.1 on Solaris 9 IBM® Lotus® Instant Messaging and Web Conferencing (Sametime)™ 3.1, with more than 9 million users, is the market-leading instant messaging and Web conferencing solution for business. Installation and configuration hints and tips The Solaris 9 Operating Environment is the foundation for Sun systems. Designed for multiprocessing and 64-bit computing, Solaris software delivers a consistent computing environment that scales to handle heavy traffic, huge data sets, and CPU-intensive problems. Performance-tuning recommendations This IBM Redpaper describes how to leverage the Sun Solaris 9 environment as a stable and scalable platform for Sametime 3.1. Topics covered in this Redpaper are: Basic installation and setup guidelines for Sametime 3.1 on Solaris 9 General Solaris performance monitoring techniques Performance tuning suggestions for optimizing a Sametime 3.1 install on Solaris 9 Results of basic performance benchmarks demonstrating the performance tuning of Sametime 3.1 on Solaris 9 ® Redpaper INTERNATIONAL TECHNICAL SUPPORT ORGANIZATION BUILDING TECHNICAL INFORMATION BASED ON PRACTICAL EXPERIENCE IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment. For more information: ibm.com/redbooks