Talking Stats with Firemen Highlights from StatCom's first year Outline ● What is StatCom? Why should you get involved? ● Project #1: Shoreline Fire Department ● Project #2: ACLU of Washington State ● Q&A What is StatCom? ● ● Student-run organization which provides statistical consulting services to non-profit community and governmental groups Serves clients external to the university community who do not have the resources to hire professional statistical consultants A brief history ● ● ● ● 2001: StatCom founded at Purdue University 2005: Purdue seeks to expand StatCom, contacts other universities. Assaf Oron decides to create StatCom chapter @ UW 2006: StatCom featured as Amstat News cover story. First joint meeting of StatCom chapters at JSM, Seattle. 2007: 5-6 active chapters (Purdue, UW, Cornell, Michigan, Ohio State, UHasselt (Netherlands)) Services ● Study design ● Questionnaire design ● Data analysis ● Statistical literature review ● etc StatCom @ UW ● ● Members: – Students from Stat/Biostat, Genome Sciences – Recent graduates (Epi, Math) currently working at UW and Microsoft Research Meetings: – ● Roughly monthly, depending on member availability and workload Structure: – Non-hierarchical, decisions reached by consensus StatCom @ UW (cont'd) ● Project teams – Usually involve 3-4 StatCom members – Often led by a more senior/experienced student – Autonomous, report on progress at general StatCom meetings What's in it for me? ● Real-world consulting experience ● Potential for publications ● Good karma Example Projects Project: Shoreline Fire Dept. ● Serves a 13-square mile area with a population of 50,000 ● 100 employees ● Handles 8,000 calls annually for – fire suppression – technical rescue – emergency medical and advanced life support – fire prevention, education, and inspection – etc SFD Project: Initial problem ● How to compare peer evaluation ratings of candidates for promotion to paramedic from different station houses? Paramedics Station House #1 Station House #2 Station House #3 SFD Project: Initial solution ● ● ● Fit a linear mixed effects model to account for an individual-specific “rater” effect OK, but... the peer evaluation system as a whole needed an overhaul So... StatCom took on a larger challenge: Design and help implement an improved peer evaluation system for the Shoreline Fire Department SFD Project: Problems to fix ● ● ● Questionnaire: Peer eval questionnaire much too long Rater assignment: Employees volunteer to rate candidates of their choosing Analysis: Possibility for “halo” effect by station SFD Project: Solutions ● Questionnaire: Run psychometric analysis tools on data from last year's evaluations to identify/eliminate questions which – are redundant – correlate poorly with total score – have small variance SFD Project: Solutions (cont'd) ● ● Rater assignment: Introduce two-stage rating process: – Stage 1: Raters identify candidates whose abilities they are familiar with – Stage 2: Raters are matched to candidates via semi-random algorithm and asked to complete full questionnaire Analysis: Linear mixed effects model as described above SFD Project: Current status ● ● ● ● Two-stage design proposal adopted Shortened questionnaire adopted in principle, still a work in progress Design being rolled out for current round of evaluations of Drivers First-stage (matching) data in within two weeks. Second-stage (questionnaire) data in for analysis at end of October