Tools of Software Projects Canvas Pages Spring 2020 1. Automation, Security, Licensing - Choosing Technology Why automate? ● ● ● ● Software Development Process Release Process Time-to-Market Fix-to-Deploy Shell Programming: POSIX: https://en.wikipedia.org/wiki/POSIX (Links to an external site.) UNIX Shells, Configuration: https://en.wikipedia.org/wiki/Unix_shell (Links to an external site.) Shell Invocation: https://en.wikipedia.org/wiki/Shell_script (Links to an external site.) Shell programming generals: http://gsd.web.elte.hu/lectures/tools/tools-2/ Links to an external site. Bashisms: https://wiki.ubuntu.com/DashAsBinSh (Links to an external site.) Regular expressions: Regular Expression Cheat Sheet Regular Expression Samples Regular expressions: http://gsd.web.elte.hu/lectures/tools/tools-1/ Links to an external site. Regexp in more detail: https://en.wikipedia.org/wiki/Regular_expression (Links to an external site.) Play with regexp: https://regexr.com/ (Links to an external site.) Choosing Technology: ● Software Licenses: https://en.wikipedia.org/wiki/Software_license ● (Links to an external site.) ● What about data? Security: ● Development in Cloud environment ● Typical security breaches ● Passwords ○ https://www.xkcd.com/936/ ○ (Links to an external site.) ○ Use Password Managers! ● Do we need Anti-Virus Software? ● Keep Best Practices at all times! Useful stuff for non-technical friends: https://switching.software/ (Links to an external site.) Useful stuff for technical gurus: https://prism-break.org/ 2. Version Control Systems VCS = Version Control System Why do we use VCSs? ● ● ● ● ● Getting back most up-to-date version, keeping track of changes. Documenting and storing change history. Avoiding conflicts between developers. Supporting different workflows. Supporting multiple versions / releases. There are a lot: https://en.wikipedia.org/wiki/Comparison_of_version_control_software (Links to an external site.) Centralized Workflows: ● VCSs: Subversion, CVS, Perforce Distributed Workflows: ● VCSs: Git, Mercurial, GNU Bazaar, BitKeeper Subversion (SVN): ● SVN Introduction: svn.html ● SVN Quick Guide: https://www.tutorialspoint.com/svn/svn_quick_guide.htm ● (Links to an external site.) ● Git: ● Git Introduction Slides: git-elte-2.pdf ● Thorough and Accurate Quick Guide for Everything: https://www.tutorialspoint.com/git/git_quick_guide.htm ● (Links to an external site.) 3. Build Systems History - from Autotools to CMake Build Systems ● ● ● ● Why a Build System? History Software Delivery & Deploy Cross-Platform Build Outlook on Build Automation: https://en.wikipedia.org/wiki/List_of_build_automation_software (Links to an external site.) Make: Wiki: https://en.wikipedia.org/wiki/Make_(software) (Links to an external site.) GNU Make Manual: https://www.gnu.org/software/make/manual/make.html (Links to an external site.) Autotools: Slides on Autotools: https://www.lrde.epita.fr/~adl/dl/autotools.pdf (Links to an external site.) Autotools Mythbuster: https://autotools.io/index.html (Links to an external site.) CMake: CMake Guides: https://cmake.org/cmake/help/latest/#guides (Links to an external site.) CMake Reference: https://cmake.org/cmake/help/latest/ (Links to an external site.) Ninja build system: https://ninja-build.org/manual.html 4. Build Systems - Ant, Maven, Gradle + Artifact Repositories Build Systems for Java - Ant, Maven, Gradle + Artifact Repositories Java Build Systems (slides) Artifact Repositories (slides) ● Software Repository ● Artifact Repository (Binary repository manager: https://en.wikipedia.org/wiki/Binary_repository_manager ● (Links to an external site.) ● Software Package Management Systems: https://en.wikipedia.org/wiki/List_of_software_package_manageme nt_systems ● (Links to an external site.) Advanced Build Summary (slides) 5. Testing Basics Testing Basics ● Why? CI / CD? https://en.wikipedia.org/wiki/Continuous_integration ● (Links to an external site.) ● Simple Levels of Testing (triangle of testing) ○ 0. asserts / different builds ○ 1. unit ○ 2. integration ○ 3. system ● Real Ways of Testing ○ Types: https://www.tutorialspoint.com/software_testing/software _testing_types.htm ○ (Links to an external site.) ○ Methods: https://www.tutorialspoint.com/software_testing/software _testing_methods.htm ○ (Links to an external site.) ○ Levels: https://www.tutorialspoint.com/software_testing/software _testing_levels.htm ○ (Links to an external site.) ● Unittest frameworks ○ JUnit / JMock: See testing_java.zip and https://junit.org ○ (Links to an external site.) ○ & http://jmock.org/ ○ (Links to an external site.) ○ GTest / GMock See testing_cpp.zip and https://github.com/google/googletest ○ (Links to an external site.) ○ Nosetests https://nose.readthedocs.io ○ (Links to an external site.) ● Test Expectations: gtesting.cpp.html 6. CI / CD ... towards DevOps Advanced Builds Multiple Builds because of ● ● ● ● ● Debug / Release / ReleaseWithDebug (+Obfuscation) Free / Commercial / With Extra Feature Release per Branch (eg: 1.4 maintenance, 2.0 new release) Release per Platform Special Builds ○ Coverage ○ Memory / Leak Checking ○ Thread Checking CI/CD & Automation towards DevOps - ci.pdf ● Jenkins https://jenkins.io/ 7. CI Extra Tools Tools for Code Verification "The earlier it is caught, the cheaper it is to fix." ● Compiler Settings / Flags (warnings, warnings as errors -> No Warning Policy) ● Static Analyzers ○ List of Tools per Language https://en.wikipedia.org/wiki/List_of_tools_for_static_code _analysis ○ (Links to an external site.) ○ FindBugs http://findbugs.sourceforge.net/ ○ (Links to an external site.) ○ also see http://www.methodsandtools.com/tools/findbugs.php ○ (Links to an external site.) ○ Cppcheck http://cppcheck.sourceforge.net/ ○ (Links to an external site.) ○ also see Demo ○ PMD https://pmd.github.io/ ○ (Links to an external site.) ○ also see Documentation ■ CPD = copy-paste-detector - code duplication detection ○ SonarQube https://www.sonarqube.org/ ○ (Links to an external site.) ○ Clang Static Analyzer https://clang-analyzer.llvm.org/ ○ (Links to an external site.) ○ Clang Tidy https://clang.llvm.org/extra/clang-tidy/ ○ (Links to an external site.) ○ CodeChecker (@ELTE) https://github.com/Ericsson/codechecker ○ (Links to an external site.) ○ Pylint https://www.pylint.org/ ○ (Links to an external site.) ● Coverage Analysis ○ EclEmma http://www.eclemma.org/ ○ (Links to an external site.) ○ gcov https://gcc.gnu.org/onlinedocs/gcc/Gcov.html ○ (Links to an external site.) ○ and gcov.cpp.html ● Profiling ○ List of Performance Analysis Tools https://en.wikipedia.org/wiki/List_of_performance_analysi s_tools ○ (Links to an external site.) ○ VisualVM https://visualvm.github.io/ ○ (Links to an external site.) ○ Gprof https://en.wikipedia.org/wiki/Gprof ○ (Links to an external site.) Tools for Code Maintainability ● Documentation Generators ○ Comparison of Documentation Generators https://en.wikipedia.org/wiki/Comparison_of_documentati on_generators ○ (Links to an external site.) ○ Javadoc https://en.wikipedia.org/wiki/Javadoc ○ (Links to an external site.) ○ Doxygen http://doxygen.nl/ ○ (Links to an external site.) ● Code Styles & Standards / Auto-Formatting ○ pep8 https://pypi.org/project/pep8/ ○ (Links to an external site.) ○ flake8 https://pypi.org/project/flake8/ ○ (Links to an external site.) ○ Checkstyle http://checkstyle.sourceforge.net/ ○ (Links to an external site.) ○ also see list of checks http://checkstyle.sourceforge.net/checks.html ○ (Links to an external site.) ○ ClangFormat https://clang.llvm.org/docs/ClangFormat.html ○ (Links to an external site.) ○ (no standard, you decide and configure) ● Code Metrics ○ Kind of Static Analysis, see tools there (Pylint, SonarQube...) 8. Containers and Virtualization Why to talk about Virtualization? ● You need the cloud ● But the cloud is complex ○ Rivaling technologies ○ Different ecosystems (sometimes closed) ○ Multiple levels (as Service Models) https://en.wikipedia.org/wiki/Cloud_computing ○ (Links to an external site.) ■ IaaS - Infrastructure as a Service ■ PaaS - Platform as a Service ■ SaaS - Software as a Service ● Lack of understanding the basics Why to Virtualize? ● Multiple small (guest) machines on one big (host) machine controlled and monitored by a program (hypervisor) ● Pros ○ Cost saving ○ Energy saving (Green Computing) ○ Easy inspection and control ○ Automated deployment and shutdown ○ Live migration (Uptime) ● Cons ○ Less stable performance ○ Sharing of resources ○ Time sharing (Response Time) Here, it is not ● Application Virtualization (from Wine to JVM, Mono) ● Virtual Memory ● Emulation (QEMU, DOSBox, VICE) Full Virtualization ● ● ● ● ● ● ...also called Hard Virtualization Different guest / host opsys Not always possibel -> special CPU instruction set needed Performance problems Hardware-assisted virtualization Paravirtualization (modified guest) ● KVM, Xen, Virtualbox Operating-system-level Virtualization ● ...also called Containerization ● ...also called Soft Virtualization ● Multiple user spaces separated by the same kernel ○ jails ○ partitions ○ containers ○ virtual environments ● Low overhead ● Knowledgeable allocation of resources (RAM, storage) ● File system isolation (chroot - 1982) ● Copy-on-write storage (union file systems) -> images (like in Docker) ● LXC, OpenVZ, Docker (deliver app?), LXD (deliver machine?) Container Orchestration ...Ansible, Chef, Puppet... Docker ● ...on Windows: Windows container, or Linux with Hyper-V (Full Virtualization) ● Because OS-level-virtualization is so light, put one app in a container ● Flexible and environment-independent configuration and deployment ● https://docs.docker.com/get-started/ ● (Links to an external site.)