Respectful Relationships Evaluation
– Rounds 1 and 2
Preliminary Draft Report
Prepared for:
Department of Families, Housing Community Services and
Indigenous Affairs (FaHCSIA)
P11007
The University of Queensland
Institute for Social Science Research
ABN: 63942 912 684
October2011
The University of Queensland
Institute for Social Science Research (ISSR)
DIRECTOR
Professor Mark Western, PhD, BA (Hons)
DEPUTY DIRECTOR (RESEARCH)
Professor Paul Boreham, PhD, BEcon. (Hons)
The University of Queensland
Level 4, General Purpose North 3 (Building 39A)
Campell Road, St Lucia Brisbane
Queensland 4072 Australia
Telephone +61 7 3346 7344
Facsimile +61 7 3346 7646
Email issr@uq.edu.au
The University of Queensland Institute for Social Science Research Website
Respectful Relationships Evaluation
–Rounds 1 and 2
Preliminary Report
October 2011
Reference: P11007
Printed
Last saved
File Name
Authors:
Director Research
Project team members
Name of Project:
Name of organisation
Document Status
October 2011
October 2011
Robyne Le Brocque, Caroline Crothers, Silke Meyer,
Warren Laffan,
Paul Boreham
Robyne Le Brocque, Caroline Crothers, Silke Meyer,
Warren Laffan
Respectful Relationships Evaluation
FaHCSIA
Draft
Level 4, Building 39A, GP North 3
The University of Queensland
Brisbane 4072 Queensland
Telephone: +61 7 3346 7344
Facsimile: +61 7 3346 7646
Email: issr@uq.edu.au
The University of Queensland Institute for Social Science Research Website
The University of Queensland ABN: 63942 912 684
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
Table of Contents
Tables .............................................................................................................................. b
Executive Summary .................................................................................................................... i
Selection Criteria ........................................................................................................................ i
1.
Introduction ...................................................................................................................... 1
2.
Process Evaluation ............................................................................................................. 1
3.
Outcome Evaluation ........................................................................................................... 1
4.
Methodology Respectful Relationships projects – rounds 1 and 2 ............................................ 2
4.1
Descriptive Meta-analysis of Funded Projects: ................................................................... 2
4.2
Evaluation of project content ........................................................................................... 2
4.3
Evaluation of project design and project fidelity ................................................................. 3
4.4
Examination of projects against the NASASV Standards for Sexual Assault Prevention through
Education................................................................................................................................. 3
5.
Method for descriptive meta-analysis: .................................................................................. 3
5.1
Manual Analysis ............................................................................................................ 3
5.2
Software Analysis (using Leximancer) .............................................................................. 4
5.3
Considerations for meta-analysis ...................................................................................... 4
6.
Semi-structured interviews with key informants:.................................................................... 4
6.1
Methodology for semi-structured interview ....................................................................... 5
6.2
Analysis of semi-structured interviews .............................................................................. 6
6.3
Summary of processes and content evaluation.................................................................... 9
7.
Criteria considerations for round 3 projects ..........................................................................12
7.1
Criterion consideration: The project should demonstrate that it will be based on a clearly
articulated framework or logic ...................................................................................................12
7.2
Criterion consideration: The project must demonstrate that it is inclusive, relevant and culturally
sensitive..................................................................................................................................12
7.3
Criterion consideration: The project should include where appropriate a whole school approach
(WSA) 12
7.4
Criterion consideration: The project where a whole of school approach is incorporated must
involve school staff ..................................................................................................................12
7.5
Criterion consideration: The project where a whole of school approach is incorporated must
consider the curriculum content .................................................................................................12
7.6
Criterion consideration: The project where a whole of school approach is incorporated must
consider the curriculum delivery ................................................................................................13
7.7
Criterion consideration: The project where a whole of school approach is incorporated must
consider the curriculum structure ...............................................................................................13
Page a
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
7.8
Criterion consideration: The project must provide for evaluation and set performance measures
as part of the framework ...........................................................................................................13
7.9
Criterion consideration: Management capacity and capability .............................................13
7.10
Criterion consideration: Project timelines and milestones ...................................................13
7.11
Criterion consideration: Project budget and justification .....................................................13
7.12
Criterion consideration: Project target audience and implementation....................................14
7.13
Criterion consideration: Project staff and relevance of experience and qualifications .............14
8.
Selection Criteria ..............................................................................................................14
8.1
Project description and implementation ...........................................................................14
8.2
Project approach, development and design .......................................................................15
8.3
Primary prevention expertise, experience and qualifications of staff ....................................15
8.4
Timelines, including duration and budget implications .......................................................16
8.5
Management capacity and capability................................................................................16
8.6
Evaluation of project......................................................................................................16
Appendix A: Projects in rounds 1 and 2..........................................................................................18
Appendix B – Catalogue of Documents Rounds 1 and 2 ...................................................................19
Appendix C – Consent Form .........................................................................................................23
Appendix D: Semi-structured questionnaire ....................................................................................24
Appendix E: NASASV Standards outline .......................................................................................30
Tables
Table 1 Characteristics by Project and attributed items ................................................................................ 9
Table 2 Project Key .................................................................................................................................... 11
Page b
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
Executive Summary
This preliminary draft report of the Respectful Relationships Evaluation project is prepared for the
Department of Families, Housing, Community Services and Indigenous Affairs (FaHCSIA) and outlines
findings from the preliminary evaluation of the projects from rounds 1 and 2. Included in these
preliminary analyses are projects where appropriate documents were available and follow-up interviews
were conducted with nominated representatives of organisations involved with the projects.
The following selection criteria considerations have been derived from the preliminary evaluation
findings of nine of the projects from rounds 1 and 2. These considerations along with the National
Association of Services Against Sexual Violence (NASASV) standards have led to the formulation of the
selection criteria produced below for round 3 projects. The selection criteria considerations were:

The project should demonstrate that it will be based on a clearly articulated framework or logic.

The project must demonstrate that it is inclusive, relevant and culturally sensitive.

The project should include, where appropriate, a whole of school approach (WSA).
o
The project should involve school staff where a WSA is incorporated.
o
A WSA project should consider the appropriateness of content.
o
A WSA project should consider the suitability of delivery.
o
A WSA project should consider the suitability of structure.

The project should provide for evaluation and set performance measures as part of the
framework.

The project should describe its management capacity and capability.

Project timelines and milestones should be provided and clearly articulated.

A detailed project budget, including budget justification, should be provided.

A clearly defined project target audience and implementation plan should be provided.

The project needs to demonstrate that project staff has the relevant training, experience, and
qualifications to undertake the project.
Selection Criteria

Project description and implementation
Describe the project, the location, activities, demographic of participants (including age, gender, ethnicity
and expected numbers) and how the project is to be implemented. The proposed budget needs to be
described, outlining the milestones of the project and costs associated with each milestone. A project plan
should be provided that demonstrates how the project will be implemented and matched against the
milestones and how the delivery provides value for money to the Community and Government in
achieving respectful relationship goals and objectives.
Level of importance: High

Project approach, development and design
The project should demonstrate that it will be based on a clearly articulated framework or logic and that
Page i
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
this is incorporated into the approach of the project. The approach should outline how and why the target
group(s) was selected and provides research-based evidence to support this selection process. Aims and
objectives of the project need to be clearly articulated. The project should be developed within a
theoretical framework, following principles around theories of change. Projects need to demonstrate that
their approach is based on good practice for primary prevention of violence and/or respectful relationships
education. Projects need to describe how their approach includes an understanding of gender and power
and includes an understanding of the social, cultural and individual factors that lead to interpersonal
violence; how it aims to develop skills in areas of positive behaviour in relationships and responsibilities;
and how it produces an understanding of differences between attitude and behaviour change. Projects
need to describe whether their program incorporates a component of consultation with community leaders
or representative from cultural groups, how this will be undertaken and how it considers the needs of
marginalised groups.
Level of importance: High

Primary prevention expertise, experience and qualifications of staff
Demonstrate that your organisation has the experience and expertise to develop or conduct a respectful
relationships project. Show how your organisation will ensure that staff delivering violence prevention
programs or activities will have relevant qualifications, training, expertise and supporting supervision to
conduct the activity.
Level of importance: Medium

Timelines, including duration and budget implications
Provide a project timeline including but not limited to areas such as the planning phase, community
consultation, promotion of the program, and recruitment of participants, development and implementation
of support mechanisms, evaluation design, running the program, evaluation results, and community
feedback. Identify major milestones and link these to the planning phase, consultation, promotion,
recruitment, development and implementation of the project. Progress reports submitted throughout the
life of the project need to describe how the project is tracking, what contingencies have been planned if
issues arise, what the expected milestone deliverables are and whether the project is on track to deliver on
time and to budget.
Level of importance: Medium

Management capacity and capability
Demonstrate that your organisation has the ability and experience to manage a respectful relationships
project to a high standard and can demonstrate previous experience in working alongside other
organisations in similar projects.
The following dot points are provided to assist applicants with their responses:
Level of importance: Medium

Evaluation of project
Round 3 projects will be evaluated using longitudinal quantitative data to examine the effectiveness of
individual projects in changing attitudes and behaviours surrounding interpersonal violence. The overall
evaluation will examine the efficiency of the implementation of the project; how the project delivered
against the National Association of Services against Sexual Assault’s (NASASV) National Standards for
Page ii
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
Prevention of Sexual Assault through Education and the effectiveness of the project in achieving the
desired awareness, attitude and behaviour change in the target population.
As such, you will be required to monitor the progress of the project against clearly defined aims and
objectives and; to work with an independent evaluation team to facilitate a formal, independent
evaluation.
The evaluation (predominantly survey based where culturally appropriate**) will collect participant’s
demographic information including age, gender, ethnicity, (including Aboriginal or/ and Torres Strait
Islander status), sexual orientation (where age appropriate), SES and psychosocial risk (potentially
including: socioeconomic disadvantage, non-nuclear family structure, parental risk characteristics, family
dysfunction, and stressful life events). Among treatment group participants, information will also be
collected regarding project attendance and self-reported motivation to attend and complete the project. A
combination of items from different inventories will be used to examine participants’ attitudes and beliefs
around abusive behaviours in dating/ intimate relationships and past/ current exposure to domestic
violence. The prevalence of abusive behaviour and experienced abuse in past/ current dating and intimate
relationships will also be measured/ captured where age appropriate as well as exposure to family and
domestic violence. For the projects targeting younger participants the research team will conduct a
literature review on existing educational projects to identify age-appropriate measures used in other
studies. The survey instruments and the collection method used will be tailored to the participant groups
based on cultural consultation and a review of the age and cultural appropriateness.
As such, please demonstrate your organisation’s experience in monitoring and evaluating a project of this
kind, including working with an independent evaluator. Describe how the evaluation results will be used
to inform the next steps beyond the life of the funding, and contribute to broader understanding and the
evidence base for best practice.
** If participation in an independent evaluation is not appropriate i.e. due to cultural factors, the nature of the target group, projects will need to
identify this and outline why it is not possible to participate.
Level of importance: High
Page iii
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
1. Introduction
This preliminary draft report of the Respectful Relationships Evaluation project is prepared by the team
from the Institute for Social Science Research (ISSR) at The University of Queensland for the
Department of Families, Housing, Community Services and Indigenous Affairs (FaHCSIA) and outlines
the preliminary evaluation of a selection of projects from rounds 1 and 2.
Material generated within each project from rounds 1 and 2 were provided to FaHCSIA as part of the
process. However, as these materials were needed by ISSR to undertake the analysis and evaluation of
each project, it was necessary for FaHCSIA to seek consent to pass this material to ISSR. Only those
projects providing consent have been included in the rounds 1 and 2 evaluation process.
This process of evaluation was complemented with follow-up interviews from nominated projects where
the majority of materials were available; approximately eleven projects were selected for the initial
reporting requirements. Of these projects only nine could be interviewed and this aspect incorporated in
the preliminary draft report.
The project involved an outcome and process evaluation of the relevant educational projects funded under
the Respectful Relationships scheme since 2009 and an evaluation of the effectiveness of these projects in
achieving the desired awareness, attitude and behaviour change in the target population.
Findings from the evaluation of projects funded in rounds 1 and 2 have been used to inform FaHCSIA’s
selection criteria for projects to be funded in round 3 in 2012. The focus of the current research project
has two components:
1. Process evaluation
2. Outcome evaluation
2. Process Evaluation
The process evaluation for rounds 1 and 2 was based on analysis of the content and implementation for
projects running over the funding period and will be used for evaluating projects in round 3. The process
evaluation analysed the content and actual implementation of projects to evaluate the efficiency of the
project model and execution (including challenges and benefits experienced throughout the
implementation and facilitation process).
The process evaluation was based on project content and implementation to inform the:
1. identification, evaluation and synthesis of project content;
2. evaluation of the extent to which implemented projects meet the NASASV Standards for
Sexual Assault Prevention Education.
3. efficiency of project and model implementation (including challenges and benefits
experienced throughout the implementation and facilitation process).
An analysis of project content and implementation will be crucial in the development of recommendations
around best practice models around future project preparation, implementation and facilitation. The
process evaluation will occur for all three project funding rounds.
3. Outcome Evaluation
The outcome evaluation will be based on the measurable impact of the projects commencing during and
after March 2012 on awareness and attitude and behaviour change pre and post project participation.
Projects included in the outcome evaluation will largely be from round 3 of project funding. Projects
Page 1
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
commencing prior to March 2012 (rounds 1 and 2) will not be included in the outcome evaluation due to
the lack of baseline data. These projects will undergo process evaluation only. The outcome evaluation
will examine the effectiveness of project participation in:
1. changing attitudes and behaviour around violence in dating and intimate relationships to
reduce and prevent sexual, physical and emotional abuse among project participants.
2. changing the attitudes and behaviours of the community at large.
An analysis of the effectiveness of each project in changing attitudes and behaviour around violence in
intimate and dating and intimate relationships will inform future funding of projects aimed at reducing
and preventing sexual, physical and emotional abuse among project participants.
4. Methodology Respectful Relationships projects – rounds 1 and 2
The process evaluation for this part of the project was based on project implementation of projects in
rounds 1 and 2 funding (see Appendix A – Projects in rounds 1 and 2). The process evaluation analysed
the content and actual implementation of projects to evaluate the efficiency of project and model
execution (including challenges and benefits experienced throughout the implementation and facilitation
process); and to evaluate the extent to which implemented projects meet the NASASV Standards for
Sexual Assault Prevention through Education.
There are two central components to the research methodology for the Process Evaluation for rounds 1
and 2:
1.
Descriptive meta-analysis of project content as reflected in individual grant
applications, project resources, final reports and related documents from funded projects.
2.
Qualitative evaluation of projects with project service providers and teachers/ educators.
4.1 Descriptive Meta-analysis of Funded Projects:
In order to evaluate the design and content of rounds 1 and 2 funded projects, a descriptive meta-analysis
of all documentation relating to the projects was undertaken including original grant applications, project
manuals, project resources such as workbooks and session plans, reports and related documents. The
descriptive meta-analysis aims to identify the intended content in each project, compare and contrast each
projects approach to the mode of delivery and design and, evaluate if the content addressed guidelines
outlined in the NASASV Standards for Sexual Assault Prevention through Education.
4.2 Evaluation of project content
A number of approaches are used to promote Respectful Relationships. These include: the social
construction of gender, ecological and multi-risk factors, development of Respectful Relationships social
norms and values, social learning and intergenerational transmission of Respectful Relationships values,
evaluation and development of sexual ethics, values-based development, and other identified approaches.
The meta-analysis was designed to identify, evaluate, and synthesise project content relating to:
a.
communication, conflict resolution, negotiating sexual consent and relationship
behaviour
b.
awareness and knowledge around sexual assault, intimacy, ethical behaviour, and
equality and respect in dating and intimate relationships
c.
critical examination of peer norms, attitudes and beliefs that sustain violence against
women
Page 2
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
4.3 Evaluation of project design and project fidelity
The project design and delivery was evaluated against the content template (above) to evaluate its
efficacy in addressing the intended content. There are many issues to consider in the development and
delivery of violence prevention projects such as different presentation methods, the duration of projects,
the targeted populations, the settings for project delivery, sex segregation in project delivery, peer
education, project implementation in the social context and adapting projects for local conditions. One of
the key strategies in this primary prevention project has been the flexibility to deliver a variety of projects
to young people both in the main school system and those who are not currently enrolled. The initial
evaluation of funded projects will, therefore, compare and contrast the delivery mode across projects and
the targeted participants. Comparisons were made between projects with wide-spread implementation (i.e.
the whole school/ community) and those with narrower coverage (i.e. a classroom or youth group) to
evaluate the efficacy of project approaches in regard to inclusion versus specificity.
4.4 Examination of projects against the NASASV Standards
A major component of the process evaluation is the comparison of guidelines outlined in the NASASV
Standards for Sexual Assault Prevention through Education with the aims and objectives of each project
funded under the ‘Respectful Relationships’ scheme. This included a comparison of any association
between the implementation of NASASV standards and project outcomes across all projects (but
particularly those in rounds 2 and 3) included in the evaluation. The following standards have been
adapted from the National Standards for effective sexual assault prevention through education to include
domestic and family violence prevention:
1.
2.
3.
4.
5.
6.
Using coherent conceptual approaches to project design
Demonstrating the use of a theory of change
Undertaking inclusive, relevant and culturally sensitive practice
Undertaking comprehensive project development and delivery
Using effective evaluation strategies
Supporting thorough training and professional development of educators
A description of the national standards is found in Appendix E.
5. Method for descriptive meta-analysis:
Two approaches for the meta-analysis of project content were undertaken for the analysis of each
project’s documentation:
5.1 Manual Analysis
Information from the projects were read, analysed, and classified according to the summary template
which includes the project content outlined above. Research support staff read and classified all
documents under the supervision and guidance of the project managers. The initial classification was
conducted independently. Secondly, in collaboration with reviewers, researchers reviewed the
summarised data, highlighting similarities and differences. This report was prepared by the research team
and identifies the content of each project included in this report, indicates if the intended content was
addressed in the design of the project and relating documents, and compares and contrasts each projects
approach to the delivery of the project. This method was time-consuming and subjective as the
classification of project content was prone to research bias in its exploration and interpretation.
Page 3
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
5.2 Software Analysis (using Leximancer)
Leximancer is specialist analytics software for unstructured, qualitative textual data. Leximancer
examines word co-occurrence information in text and identifies the characteristic distribution of words
based on two measures; frequency and prominence. The frequency and prominence of words informs the
basis of concept and theme identification.
Software analysis (using Leximancer) consisted of uploading documentation of all project plans, progress
reports other project information into a dataset. An analysis of word prominence (co-occurrence with
other frequent words) and frequency (the number of times a word is identified in-text) was undertaken
with the aim to identify each project’s aims, objectives, design and scope. The identification of these
were manipulated manually to analyse facets and stages of the structure of the relevant projects, overall as
an aggregated intervention module, by each individual project and by the stage/ facet of interest of a given
project. This enabled comparisons between projects, emergent themes/ concepts at the various stages of a
given project and an analysis of all projects at large.
Where project specifics are not identified, documents were searched and information pulled to inform
possible deficits in qualitative analysis. However, one of the benefits of using Leximancer was that it
analyses large volumes of unstructured text and hence, variation between project documentation were
easily navigated. Only those projects providing consent and material relevant for this analysis are
included. The Leximancer analysis will be included in the final report/evaluation of rounds 1 and 2
following receipt of complete project documentation of all projects undergoing document evaluation.
5.3 Considerations for meta-analysis
This method assumes that each project was able to fully articulate and document the content of their
project in the design and grant application phase as well as in the implementation phase.
There are some projects that had difficulty in providing documentation that adequately informed the
content summary template proposed. Furthermore, analysis of documentation did not adequately
document the complexity/ diversity of project content, implementation and design. Examples of these
types of considerations include projects that did not conform to the design proposed in project
documentation, projects that were based on the changing needs of the participants; projects that were
highly culturally specific, or projects that were designed to service a variety of population groups and
therefore the documentation of content was not overly prescriptive. Finally, due to the lack
standardisation of project documentation, some projects simply did not provide enough detail to inform
an adequate understanding of the content and delivery.
Therefore, the secondary component of the process evaluation, a qualitative evaluation, was designed in
part, to respond to the paucities of information surrounding project complexities (discussed above) as well
as document perceptions of efficacy as they relate to the implementation, facilitation and perceived
outcomes of each project.
6. Semi-structured interviews with key informants:
The second component of the process evaluation incorporated a semi-structured interview (see Appendix
D) with key informants from each project, including community members and professionals involved in
the development, implementation and facilitation of individual projects and curricula. The aim of the
semi-structured interviews was to examine the efficiency of project and model implementation (including
challenges and benefits experienced throughout the implementation and facilitation process). Qualitative
Page 4
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
interviews were undertaken with project service providers, teachers and organisational representatives via
telephone for rounds 1 and 2 projects.
6.1 Methodology for semi-structured interview
Semi-structured interviews were conducted via telephone where feasible to minimise evaluation costs.
Project team members initially contacted project personnel and the relevant organisations, introduced the
research team, and explained the evaluation process. A time to conduct the telephone interview with the
nominated project member was then arranged and signed consent was organised. The interviews with key
personnel addressed perceptions of the Respectful Relationships project, including its challenges and
benefits around project implementation, facilitation and outcomes.
The qualitative interviews were undertaken with a collaborative focus to help improve outcomes for the
target group and to assist in developing future funding strategies. The interviews document the
perceptions of project providers with regard to the project's development and implementation, noting
perceived difficulties encountered and associated explanations, as well as suggested solutions to inform
the foci and development. This information is crucial in the development of recommendations around
Good Practice modelling around future project preparation, implementation and facilitation.
Interviews for this report were semi-structured and consisted of a series of specific focus questions
(Appendix D). The protocol for these interviews was prescriptive to ensure consistency and validity
between projects. The list of eligible projects considered for analysis and reporting in the preliminary
draft report is listed in Table 2. This table also indicates which of these projects are included in the semistructured interview process for this report.
The focus questions for the qualitative evaluation interview with project service providers and
teachers/educators addressed the following key areas:
a.
What are the key factors that enabled and, conversely, inhibited implementation of the models?
What improvements are possible?
b.
For those projects delivered in schools, how well was a whole-of-school approach
implemented?
c.
How important was the training of teachers/educators in effectively delivering the project
models?
d.
How could the training of teachers/educators be improved to increase capacity to deliver
effective violence prevention projects?
e.
To what extent are particular models appropriate and effective in achieving outcomes for young
people from diverse groups, such as, Indigenous, culturally and linguistically diverse
backgrounds, regional or remote locations, sporting clubs, or specific age and/ or gender
cohorts?
f.
To what extent have the particular models generated unintended consequences, positive and
negative?
g.
To what extent did the capacity of teachers/educators affect each of the models in achieving its
outcomes?
h.
In what ways can better outcomes from violence prevention projects be achieved?
Page 5
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
i.
To what extent would particular models, strategies and practices be recommended as good
practice?
j.
In what ways do the projects contribute to the evidence base about effective interventions to
reduce interpersonal violence?
k.
What are the rates of participant attrition? What is the trajectory of participant retention versus
attrition at the various stages of the project? And, what are the perceived causes of participant
attrition?
6.2 Analysis of semi-structured interviews
Interview findings will be incorporated into the required reporting components for each individual project
funded under the Respectful Relationships scheme in the second draft report, due 30 November 2011.
While data collected for each project will be identifiable by organisation/ agency, full anonymity of
individual interview participants will be ensured to project staff and representatives. A summary of
project characteristics for this report is provided in Table 1 below. Other issues that have emerged from
the evaluation and analysis of rounds 1 and 2 projects and are considered to be relevant for assessing new
projects to be funded under round 3 cover the following matters:





Staff: Comments received indicate that the lack of consistency in staff/ project-related contact
persons within FaHCSIA was an important aspect that complicated seeking advice, following up
issues or providing and receiving feedback. The considerable movement of staff was described as
challenging from the participants point of view.
Record keeping: Projects often suffered from a lack of good practice in record keeping. Guidelines
for a standardised approach to maintaining appropriate records and reporting would be beneficial.
Inconsistency in the quality, quantity and content of project documentation presented problems for
the ease of information accessibility and undermined good practice in the tracking of project progress
and project transparency. Suggestions for reporting of content include clearly detailing the duration
of the project including the frequency and length of the sessions, the number of participants involved,
the recruitment method, demographic information and the rate of participant attrition across the
duration of the project. Often the intention or purpose of the project was clearly articulated however,
the way this intention was executed via the content of the project was less so for some projects. It is
recommended that projects provide detail on each of the project sessions including information on
what was done (including the activities and educational content) and the purpose/ intention of this.
Funding: The initial funding of projects was greatly appreciated by organisations and agencies since
it provided an opportunity to develop and produce programs that foster respectful relationships.
Funding often ceased at a critical or promising point of the project implementation and projects often
found it difficult to secure alternative sources to fund the project further. This was experienced as
frustrating by some projects which felt that by the time the project had been fully developed and
implemented it could no longer be delivered to the relevant target groups.
Period of program: Leading on from the funding comments, a number of participants who were
interviewed commented that the project, although invaluable in concept, suffered from a lack of
continuance over a greater period of time and lacked the opportunity to be extended to a wider group
of young people outside the initial project participants. Projects that were part of an existing program
would have wider implications and benefits to target groups and the community.
Longevity of programs in Indigenous Communities: The continuance of programs in Indigenous
communities is of particular importance in facilitating engagement and impact. This is in part, due to
Page 6
Rounds 1 and 2 Evaluation – Preliminary Draft Report






P11007
the mechanisms of participant recruitment operant in Indigenous communities (largely via word of
mouth and extended family networks) and the trajectory of participant engagement. The uptake of
Indigenous participation was reported typically to be slow initially and increased exponentially via
word of mouth recruitment and community familiarity. As such, it is recommended that the longevity
of programs in Indigenous communities be considered as a mechanism by which to increase project
engagement and impact.
Attrition: The duration of the project needs to be considered given the particular characteristics of the
group. Longer term duration is preferable, however, may not be suitable given the particular
participant group. Participant retention and engagement, particularly among high risk participant
groups, was a reported challenge.
Evaluation (standard format & framework): Most participants could have benefited from having a
template with standard sets of criteria to consider the evaluation of projects. It was apparent that some
evaluation processes applied to rounds 1 and 2 projects did not necessarily meet rigorous standards or
practices when undertaking evaluation work. The validity of the instruments used and the match up
with the program content needs more consideration. Evaluation findings were included in progress
and final reports by some projects. Without the relevant rigorous evaluation standards in place these
findings can, however, be misleading and need to be regarded with care.
Reporting (standard format): Providing a standard format for reporting requirements was also
identified as desirable by a number of projects. Standardised guidelines that address all items of
interest, including project design and implementation along with benefits to community and
individuals need to be provided to projects by FaHCSIA when advising successful applicants of
funding outcomes. Providing project outcomes in a standardised would be beneficial to FaHCSIA in
assessing and finalising accountability and monitoring requirements expeditiously.
Workshop/conference possibly: A number of participants indicated that workshops or a conference
open to all RR projects during the life of the projects would have been beneficial in providing
opportunities to discuss the variety of projects, the outcomes, aspirations and lessons learnt and
experiences in running the project such as the areas design and implementation strategies that may or
may not have worked.
Networking (ideas exchange): Similar to the workshop/conference concept a number of participants
considered that a network of participants and experts in the field of respectful relationship issues
could have added to the experience and provided guidance in the approaches used for the various
projects in these rounds. A board of experts available to support participating projects could add to
the overall value of individual projects as well as all RR projects as a whole and would have been
beneficial in communicating how these projects contributed to communities and individuals.
Existing cultural and professional service relationships within the target community: Qualitative
interviews identified differences in the success of projects that were funded as part of a program of
interventions compared to those funded as singular new projects. Projects that existed within a
demonstrated program of engagement with the target community, be it current programs with schools
or specific indigenous communities, appeared to be more successful in achieving their goals
compared to other projects. Projects that were funded to initiate new projects in communities where
they did not have an existing program reported that most of their time was consumed with building
and establishing relationship needed for effective engagement with the targeted communities. Projects
funded without this support found that much of their time and resources were directed towards
developing these relationships before being able to establish their program. Communities and target
groups were less willing to engage with new projects offered by people they did not know.
Page 7
Rounds 1 and 2 Evaluation – Preliminary Draft Report




P11007
The role of a ‘cultural broker’ (for projects implemented in Indigenous communities): It was
reported by Indigenous projects that the appointment of a person who is connected with the
community and can negotiate cultural issues and convey these to the project facilitators in highly
beneficial. It was suggested that recruiting Indigenous people from the community to the project as
cultural brokers was integral to the success of the project because cultural brokers play a significant
role with language and with cultural protocols and also symbolise the legitimacy and place of cultural
knowledge in the context of the program and thereby, increase community receptivity. The use of a
cultural broker was said to foster inclusion and increase the cultural relevance of the project among
indigenous participants and the community at large. It is for this reason, that it is beneficial that
projects offer mentorship opportunities to local Indigenous people recruited to these roles.
Standardised training modules for project facilitators and staff: Some projects identified staff
training as an area for improvement. The lack of standardised training led to inconsistencies in the
knowledge and skill bases between project facilitators and other staff. Informal training was provided
when deficits in knowledge/ skill bases were identified, often, after the fact. Project fidelity and
delivery mechanisms may have been improved by the use of standardised training prior to the
commencement of the project.
Preventative versus punitive delivery timing: It was reported that there was a certain amount of
opposition/ reactivity to the project content among participants who had exhibited violent behaviour
previously. Evidence suggests that participants receive information better when it is delivered using a
preventative and collaborative approach rather than addressing what the participants are doing wrong.
This raises questions about the timeliness of violence prevention programs among participants who
are at higher risk of exhibiting violent behaviour and suggests that interventions among high risk
groups may need to be implemented earlier.
Awareness piggy-backing: One project was completed during National Nonviolence week. The
timing of this was strategic and had the dual effect of raising the profile of the school based project
within the community and also increased the perceived relevance of violence interventions among the
participants themselves.
Page 8
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
6.3 Summary of processes and content evaluation
Table 1 Characteristics by Project and attributed items
3
foci of project
sexual

theoretical approach
7
9
10
14
Consultation
only
16


family




preventing negative
behaviours/ attitudes
fostering positive
behaviours/ attitudes
feminist



















systemic

social




cultural
individual
22



20

domestic
other
approach
5




other
model of change
ethnic composition
attitude change



skill building


behaviour change
















Indigenous



Diverse
White


Institute for Social Science Research (ISSR)/CONROD



Page 9
Rounds 1 and 2 Evaluation – Preliminary Draft Report
3
gender composition
P11007
5
7
9
10
mixed
Location
20









at risk










remote
Culturally specific
Culturally specific
scope of intervention
whole of community


low
urban
22

high
rural
n.a.



















**

whole of school
NASASV standards
16

all female
all male
SES
14
1. Used coherent
conceptual approaches
to program design


2. Demonstrated the use
of theory of change
3. Undertaken the
inclusive, relevant, and
culturally sensitive
practice




4. Undertaken
comprehensive program
development and
delivery
5. Used effective
evaluation strategies
6. Supported thorough
training and
professional
development of
educators



Institute for Social Science Research (ISSR)/CONROD


















**





Page 10
Rounds 1 and 2 Evaluation – Preliminary Draft Report
Exposure
Duration
Frequency
Intensity
P11007
3
5
7
9
10
14
16
20
22
variable
*
10
sessions
6 weeks
1 year
12 months
10
sessio
ns
40 weeks
1 day
4-6 session
*
*
1/week
10
sessions
Sporadic
1/week
Once
2 hours
*
2 hours
1 hour
*
1 hour to
3 days
3 hours
1 day
1 hour
* no information available at time of reporting.
** does not meet this classification generally however, some provisions have been made for this.
Table 2 Project Key
Project Organisation
Key
Interviewed
University of Western Sydney
1
Available after 10 October 2011
La Trobe
3
Completed
NT
5
Completed
University of NSW
6
Available after 5 October 2011
SHINE SA
7
Completed
Youth and Family
9
Completed
Wandiliya
10
Completed
Akeyulerre-
14
Completed
Youth and Family Logan
16
Completed
Australian Red Cross
20
Completed
Relationships Australia
22
Completed
Institute for Social Science Research (ISSR)/CONROD
Page 11
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
7. Criteria considerations for round 3 projects
The following points that have emerged from the initial document analysis and interviews conducted with
project staff for this report. They are listed by general aims and the more specific meaning behind each
point if considered as a selection criterion. They further incorporate relevant aspects from the National
Association of Services Against Sexual Violence (NASASV) guidelines, which should be considered
during the selection process for round 3 projects.
7.1
Criterion consideration: The project should demonstrate that it will be based on a clearly
articulated framework or logic (It is based on a theory of change or can demonstrate that its aim is to
provide opportunity for change, that it includes an understanding of gender and power and includes an
understanding of the social, cultural and individual factors that lead to interpersonal violence; that it
aims to develop skills in areas of positive behaviour in relationships and responsibilities; that it produces
an understanding of differences between attitude and behaviour change).
Level of importance: High
7.2
Criterion consideration: The project must demonstrate that it is inclusive, relevant and
culturally sensitive (Assumptions within the program should be acknowledged and discussed and the
consultation with community leaders or representative from cultural groups will be undertaken when
appropriate and consider the needs of marginalised groups).
Level of importance: High
7.3
Criterion consideration: The project should include where appropriate a whole school
approach (WSA) (The project should involve and engage the whole school community, develop
relationships between the school, parents/family and relevant local agencies and services; the program
should where possible review and communicate school policies and support them in practice and address
violence supportive norms and behaviours in the school ethos and environment).
Level of importance: High
7.4
Criterion consideration: The project where a whole of school approach is incorporated
must involve school staff (The project should be supported and resourced for staff to carry out their
role(s), identify the information skills they need, provide training and skills development to ensure they
are equipped to deliver programs, encourage networking with community partners, have a clear rationale
for the use of community/peer educators and have a critical understanding of the involvement of female
and male educators).
Level of importance: High
7.5
Criterion consideration: The project where a whole of school approach is incorporated
must consider the curriculum content (It should address the following areas: the various forms of
violence, targeting the root causes of violence and violence supportive attitudes, teach commitment to and
Page 12
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
skill in non-violence and avoid focusing specifically on avoiding or minimising the risk of victimisation).
Level of importance: Medium
7.6
Criterion consideration: The project where a whole of school approach is incorporated
must consider the curriculum delivery (Consideration should be given to an interactive and
participatory approach that uses quality resources, addresses cognitive, affective and behavioural
domains, is matched to stages of change, provides specific attention to skills development and responds
appropriately to disclosures of victimisation/perpetration).
Level of importance: Medium
7.7
Criterion consideration: The project where a whole of school approach is incorporated
must consider the curriculum structure (The project should be of sufficient duration and structure to
bring about change and be designed and timed to be age appropriate and to suit developmental needs;
the project should have a clear rationale for using single sex or mixed sex groups and understand the
drawbacks and merits of each and should minimise the risk of vicitimisation).
Level of importance: Medium
7.8
Criterion consideration: The project must provide for evaluation and set performance
measures as part of the framework (Projects should follow a template and value-add program
evaluation process which ensures that indicators of success are identified in program planning and
design, identifies clear and realistic processes for program evaluation and build in strategies for long
term follow-up; the project should reflect on program logic and framework and disseminate knowledge
and learning widely).
Level of importance: High
7.9
Criterion consideration: Management capacity and capability (Organisational governance
structures need to be in place and provide support for the management of the project during its life and
that the organisation’s board and key staff are suitably qualified; the organisation should be able to
demonstrate how they will be able to work with other organisations to ensure success of the project).
Level of importance: High
7.10
Criterion consideration: Project timelines and milestones (Realistic timelines need to be put in
place with measurable milestones with key performance indicators for evaluating progress and success of
the project and that management aims are clearly stated to ensure deliverables are met).
Level of importance: Medium
7.11
Criterion consideration: Project budget and justification (Organisations are required to
demonstrate that the project has been carefully planned and costs associated with resources and
timeframes have been well considered; that the project in its merit also provides value for money and
Page 13
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
social capital benefits to a community; the proposal provides accountability and recognition of reporting
requirements to FaHCSIA).
Level of importance: Medium
7.12
Criterion consideration: Project target audience and implementation (The project needs to
identify its target population and demographics relevant to this group. The project further needs to
outline how the respectful relationships program will address cultural needs and provide and inclusive
framework and implementation plan with the above in mind).
Level of importance: High
7.13
Criterion consideration: Project staff and relevance of experience and qualifications (The
project team should be able to demonstrate that the organisation has the experience and expertise to
develop and conduct a respectful relationships project; the team and organisation need to demonstrate
that a violence prevention program or activities will be able to produce a high quality and effective
project; the project team should be able to provide previous primary prevention/respectful relationships
projects, details of skills and qualifications or ability to recruit personnel with relevant skills and
experience and/or appropriate qualifications; the organisation should be able to demonstrate how team
members will be supported and supervised appropriately and have the relevant mandatory reporting and
complaints management processes in place).
Level of importance: High
8. Selection Criteria
8.1
Project description and implementation
Describe the project, the location, activities, demographic of participants (including age, gender, ethnicity
and expected numbers) and how the project is to be implemented. The proposed budget needs to be
described, outlining the milestones of the project and costs associated with each milestone. A project plan
should be provided that demonstrates how the project will be implemented and matched against the
milestones and how the delivery provides value for money to the Community and Government in
achieving respectful relationship goals and objectives.
The following points are provided to assist applicants with their responses:
1. Please provide an estimate of all project costs in the budget table provided.
2. Please note the activities, participants, and locations in the provided activities table. Please describe
what the activities are and how they will be implemented for participants, particularly in an inclusive,
relevant and culturally sensitive practice.
3. What is the implementation plan for the project including a timeline and major milestones?
4. How will the project be promoted and participants found for the project?
5. Include any issues to consider/resolve prior to implementing the project.
6. Key performances need to be developed for evaluation of budget outcomes.
Level of importance: High
Page 14
Rounds 1 and 2 Evaluation – Preliminary Draft Report
8.2
P11007
Project approach, development and design
The project should demonstrate that it will be based on a clearly articulated framework or logic and that
this is incorporated into the approach of the project. The approach should outline how and why the target
group(s) was selected and provides research-based evidence to support this selection process. Aims and
objectives of the project need to be clearly articulated. The project should be developed within a
theoretical framework, following principles around theories of change. Projects need to demonstrate that
their approach is based on good practice for primary prevention of violence and/or respectful relationships
education. Projects need to describe how their approach includes an understanding of gender and power
and includes an understanding of the social, cultural and individual factors that lead to interpersonal
violence; how it aims to develop skills in areas of positive behaviour in relationships and responsibilities;
and how it produces an understanding of differences between attitude and behaviour change. Projects
need to describe whether their program incorporates a component of consultation with community leaders
or representative from cultural groups, how this will be undertaken and how it considers the needs of
marginalised groups.
The following dot points are provided to assist applicants with their responses:
1. What theory or model is the project approach is this based on?
2. What outcomes do you expect for participants? How will the project approach support these
outcomes? What is the theory of change that guides the project approach, and how will the project
make this theory work for participants? If this project is based on a previous project describe the
outcomes for participants.
3. What research information is the project based on?
4. Why was this approach chosen for this target group?
5. Will the project be delivered exactly as described in this application or adapted according to ongoing
experience and feedback from the project as it is delivered?
6. What good practice information for primary prevention of violence/relationships education is this
project based on?
7. Key performances need to be developed for evaluation of project outcomes.
Level of importance: High
8.3
Primary prevention expertise, experience and qualifications of staff
Demonstrate that your organisation has the experience and expertise to develop or conduct a respectful
relationships project. Show how your organisation will ensure that staff delivering violence prevention
programs or activities will have relevant qualifications, training, expertise and supporting supervision to
conduct the activity.
The following dot points are provided to assist applicants with their responses:
1. Describe your experience in running previous primary prevention /respectful relationships projects.
2. Provide details of either the skills and qualifications of current staff who would work on the project,
or the skills and qualifications the organisation would recruit to support the project.
3. How would staff be supported, and provided with supervision to support the project?
4. Describe the structures your organisation has for mandatory reporting and complaints management.
Page 15
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
Level of importance: Medium
8.4
Timelines, including duration and budget implications
Provide a project timeline including but not limited to areas such as the planning phase, community
consultation, promotion of the program, and recruitment of participants, development and implementation
of support mechanisms, evaluation design, running the program, evaluation results, and community
feedback. Identify major milestones and link these to the planning phase, consultation, promotion,
recruitment, development and implementation of the project. Progress reports submitted throughout the
life of the project need to describe how the project is tracking, what contingencies have been planned if
issues arise, what the expected milestone deliverables are and whether the project is on track to deliver on
time and to budget.
The following dot points are provided to assist applicants with their responses:
1. Describe the project with an overview of all major components from design to outcomes.
2. Provide details in how the components fit into the budget requirements and milestones.
3. Provide timelines against major components and deliverables with milestones incorporated against
the deliverables.
4. Describe how contingency planning and risk management elements of the project are incorporated
with budget and outcome expectations.
Level of importance: Medium
8.5
Management capacity and capability
Demonstrate that your organisation has the ability and experience to manage a respectful relationships
project to a high standard and can demonstrate previous experience in working alongside other
organisations in similar projects.
The following dot points are provided to assist applicants with their responses:
1. What organisational governance structures will be in place to support the management of the project,
including financial management infrastructure and budget oversight processes?
2. Demonstrate that your organisation’s Board and key staff (including financial management) have
appropriate experience and qualifications.
3. Describe how your organisation would work with other relevant organisations to make the project
successful.
Level of importance: Medium
8.6
Evaluation of project
Round 3 projects will be evaluated using longitudinal quantitative data to examine the effectiveness of
individual projects in changing attitudes and behaviours surrounding interpersonal violence. The overall
evaluation will examine the efficiency of the implementation of the project; how the project delivered
against the National Association of Services against Sexual Assault’s (NASASV) National Standards for
Prevention of Sexual Assault through Education and the effectiveness of the project in achieving the
desired awareness, attitude and behaviour change in the target population.
Page 16
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
As such, you will be required to monitor the progress of the project against clearly defined aims and
objectives and; to work with an independent evaluation team to facilitate a formal, independent
evaluation.
The evaluation (predominantly survey based where culturally appropriate**) will collect participant’s
demographic information including age, gender, ethnicity, (including Aboriginal or/ and Torres Strait
Islander status), sexual orientation (where age appropriate), SES and psychosocial risk (potentially
including: socioeconomic disadvantage, non-nuclear family structure, parental risk characteristics, family
dysfunction, and stressful life events). Among treatment group participants, information will also be
collected regarding project attendance and self-reported motivation to attend and complete the project. A
combination of items from different inventories will be used to examine participants’ attitudes and beliefs
around abusive behaviours in dating/ intimate relationships and past/ current exposure to domestic
violence. The prevalence of abusive behaviour and experienced abuse in past/ current dating and intimate
relationships will also be measured/ captured where age appropriate as well as exposure to family and
domestic violence. For the projects targeting younger participants the research team will conduct a
literature review on existing educational projects to identify age-appropriate measures used in other
studies. The survey instruments and the collection method used will be tailored to the participant groups
based on cultural consultation and a review of the age and cultural appropriateness.
As such, please demonstrate your organisation’s experience in monitoring and evaluating a project of this
kind, including working with an independent evaluator. Describe how the evaluation results will be used
to inform the next steps beyond the life of the funding, and contribute to broader understanding and the
evidence base for best practice.
The following dot points are provided to assist applicants with their responses:
1. Tell us about how the project will be evaluated, including the evaluation model, what data will be
collected and how the results will be used and reported. Include how participant involvement will be
monitored, including rates for not completing activities.
2. Describe your organisation’s experience in collecting data for evaluating respectful relationships or
similar projects (particularly on sensitive topics).
3. Describe the ethical issues that you will need to consider in collecting data from potentially
vulnerable or disadvantaged participants, and how you will ensure that this will fit in with the
requirements of the Privacy Act 1988 and mandatory reporting in your state/territory.
4. Describe your organisation’s experience in working with an independent evaluator.
** If participation in an independent evaluation is not appropriate i.e. due to cultural factors, the nature of the target group, projects will need to
identify this and outline why it is not possible to participate.
Level of importance: High
Page 17
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
Appendix A: Projects in rounds 1 and 2
RR Projects as of 27 September 2011
ROUND 1
ORGANISATION
PROJECT
1. University of Western Sydney
Sex and ethics
2. CASA House & CRC Centre
SAPPSS
3. La Trobe University
Living safer sexual lives
4. SA Government
Keeping
safe
child
protection curriculum
5. NT Department of Education &
Training
COMMENCED
COMPLETED
12-May-09
28-Mar-10
1-Jun-09
17-Jun-09
NOTES
Final report
Consent
[30-Jun-12]
Ongoing
Consent
1-Jun-11
Final report
Consent
19-Jun-09
10-Dec-10
Final report
Consent*
Modified Keeping safe
child protection curriculum
19-Jun-09
30-Jun-11
Final report
Consent
6. University of NSW and NRL
Sex and ethics
22-Jun-09
31-Mar-10
Final report
Consent
7. SHINE SA
RR Curriculum
Final report
Consent
8. WA Government
SHINE RR curriculum
Project
Discontinued
ROUND 2
ORGANISATION
PROJECT
9. Youth & Family Focus
The Mersey RR project
COMMENCED
15-Mar-10
10. Wandiliya
Murrung Program
11. Dirtywork comedy
No Means No
12. Baptist Community Services
RR program
13. Kurbingui Youth Development
Healthy Relationships
14. Akeyulerre
Akeyulerre RR program
15. Link-Up (NSW) Aboriginal
COMPLETED
NOTES
31-Jul-11
Final report
Consent
31-May-10
30-Jun-11
Final report
Consent
31-May-10
[30-Sep-11]
Ongoing
Consent*
2-Jun-10
[31-Jul-12]
Ongoing
Consent
11-Jun-10
[31-Oct-11]
Ongoing
Consent
18-Jun-10
30-Jun-11
Final report
Consent
Link-up program
21-Jun-10
30-Jun-11
Final report
Consent*
16. Youth & Family Service Logan
RR program
21-Jun-10
31-Jul-11
Final report
Consent
17. Swinburne University
Mumgu-dhal tyama-tiyt
22-Jun-10
[31-Dec-11]
Ongoing
Consent
18. UnitingCare Wesley Adelaide
RR program
22-Jun-10
[30-Apr-12]
Ongoing
Consent
19. Sexual Assault Resource Centre
RR education
23-Jun-10
[20-Jun-12]
Ongoing
Consent
20. Australian Red Cross
Indigenous RR programs
25-Jun-10
31-Jul-11
Overdue#
Consent
21. AFL
Respect & Responsibility
28-Jun-10
[30-Aug-12]
Ongoing
Consent
22. Relationships Australia
Love Bites
21-Jul-10
31-Jul-11
Final report
Consent
ROUND 3 (Note: Round 3 anticipate selection 2011 and commencement 2012)
TBA
ADDITIONAL PROJECTS
23. NAPCAN
Love Bites
Kerryn to contact
24. YWCA
Relationship Things
Kerryn to contact
25. Ruby Gaea
Page 18
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
ROUND 1
ORGANISATION
PROJECT
COMMENCED
COMPLETED
NOTES
SAPPSS
Kerryn to contact
26. Casa
SAPPSS
Kerryn to contact
Notes:
* Consent not obtained – do not contact project
# Final report not received
Appendix B – Catalogue of Documents Rounds 1 and 2
1. University of Western Sydney
2. CASA House & CRC Centre
3. La Trobe University





























Consent cover
Front page
Progress report 15 June
Progress report 30 Aug
Final report
Proposal
Letter of variation 2009
Funding agreement
PR 2- report CASA house June 2010
PR 3- 18mnth report- SAPPSS Dec 2010
CASA progress report
Summary of evaluation of CASA house
No means no show
CASA progress report 1- development and
planning
CASA house implementation plan
CASA house proposal
CASA funding agreement 300
Consent cover
Final funding agreement
Final report
Progress report 1
Progress report Dec 2010
Proposal
Progress report 2
Respectful relationships project plan
Program manual
Overview of program
Consent cover
Evaluation report








Consent cover
Final letter of funding
Govt January 10 report
Govt August 09 progress report
Govt- FINAL Report June 2011
NT RR Proposal
NT Keeping Safe Update
Progress Report 2
4. SA Government
5. NT Department of Education &
Training
Page 19
Rounds 1 and 2 Evaluation – Preliminary Draft Report
1. University of Western Sydney
6. University of NSW and NRL
7. SHINE SA
8. WA Government
P11007

Consent cover

Front page

Progress report 15 June

Progress report 30 Aug

Final report

Proposal

Letter of variation 2009

Funding agreement

Respectful Relationships proposal

Variation of agreement

Trainer evaluation forms

Briefing proposal – DRAFT

Evaluation of DET’s

Consent cover

Lumby RR report 08

NRL RR proposal

Final report

Final funding agreement

Final report

Implementation and funding proposal

Consent

Letter of offer

Proposal

Variation of agreement

Evaluation contacts
As above
ROUND 2
9. Youth & Family Focus
10. Wandiliya











Funding agreement
Progress report 10.12.10
Activity work plan
Original application
Consent cover
Final project report
Consent cover
Final report June 2011-09-20
Funding agreement
Risk tool
Performance report 1.12.10











Variation funding agreement June 2011
Life care progress report Dec 2010
Project roles and responsibilities
Activity work plan June 2010
Consent cover
BCS signed RR R2 agreement
Letter of variation deed
Interim project delivery analysis
Cover letter
Original application
Original application





Consent cover
Final report
Funding agreement
Original application
Risk tool
11. Dirtywork comedy
12. Baptist Community Services
13. Kurbingui Youth Development
14. Akeyulerre
Page 20
Rounds 1 and 2 Evaluation – Preliminary Draft Report
1. University of Western Sydney









Consent cover
Front page
Progress report 15 June
Progress report 30 Aug
Final report
Proposal
Letter of variation 2009
Funding agreement
Performance report

















































Attachment C Facilitators guide
Consent cover
Original application
Evaluation report
Final report
Progress report 2010
Attachment D Student Information Book
Correspondence with FACSHIA
Marden facilitators guide
Progress report 31 May
Memorandums of understanding
Session 1 to 10 (9 documents in total)
Group session plan session 1
Letter to facilitators (4 pages doc)
Youth and family service Logan city xls
Control strategies
Original application
Funding agreement
Variation
Evaluation design layout
Progress report Jan 2011
Activity work plan July 2010
Consent cover
Progress report Dec 10
Activity work plan
Funding agreement
Consent cover
Original application
Consent
Progress report 2011
Variation June 2011
ARCS_FA_executed
Progress report Dec 2 2010
Activity work plan
Implementation plan
Control strategies
Australian Red Cross Society WA
Consent
Original application
Progress report 2011
Progress report 2010
Consent
Original application
Signed agreement
AFL Activity Work Plan
AFL Activity Work Plan – Nov10
Activity Sites
Consent cover
Data teacher training dec.
P11007
15. Link-Up (NSW) Aboriginal
16. Youth & Family Service Logan
17. Swinburne University
18. UnitingCare Wesley Adelaide
19. Sexual Assault Resource Ctr
20. Australian Red Cross
21. AFL
22. Relationships Australia
Page 21
Rounds 1 and 2 Evaluation – Preliminary Draft Report
1. University of Western Sydney













P11007
Consent cover
Front page
Progress report 15 June
Progress report 30 Aug
Final report
Proposal
Letter of variation 2009
Funding agreement
Student data year to date
Progress report Dec 2010
Original application
Funding agreement
Napcan growing respect brochure
Page 22
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
Appendix C – Consent Form
Informed Consent Respectful Relationships Project Representatives
Research Team:
Dr Silke Meyer
Project Leader and Chief Investigator, Institute for Social Science Research
Contact phone: 3365 6071
Contact email: s.meyer@uq.edu.au
Dr Robyne Le Brocque
Co-Chief Investigator, Centre of National Research on Disability and Rehabilitation Medicine
Prof Justin Kenardy
Co-Chief Investigator, Centre of National Research on Disability and Rehabilitation Medicine
Prof Lorraine Mazerolle
Co-Chief Investigator, Institute for Social Science Research
Ass Prof Michele Haynes
Co-Chief Investigator, Institute for Social Science Research
Project Title: Evaluation of Respectful Relationship (RR) Projects
Please read the following statements and indicate whether you understand each one and agree
with it.

I have been informed about the purpose and nature of this research.
Y /N

I agree to participate in a telephone interview (~60min) with a researcher from the University of Queensland.
Y /N

I agree for this interview to be audio-recorded.
Y/N

I understand that this interview will cover information around the RR project I represent, including the planning,
implementation and delivery phase.
Y/N

I understand that research staff employed by the University of Queensland will protect the privacy and confidentiality
of all information collected throughout this interview and that no identifiable information will be used in the write-up of
findings.
Y/N

I have agreed to represent my RR project for the purpose of this interview of my own free will.
Y/N

I am aware that I may withdraw my consent from participating in this interview at any time without any penalty
Y/N
Name of RR project representative:
Name of RR project represented by interviewee:
Signature of project representative:
Witnessed by:
Date:
Page 23
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
Appendix D: Semi-structured questionnaire
The University of Queensland
Respectful Relationships Evaluation Project
Round 1 and Round 2 Semi-Structured Interview
Round:
Project:
Contact Person:
Position:
Contact details:
Summary of project:
Commenced:
Delivery mode:
Focus:
Wide: Whole of school/ community
Narrow: Classroom or group
The aim of this interview is to examine the efficiency of project and model implementation and the
effectiveness of the project to meet the goals of the Respectful Relationships Program. We are interested
in your views. This information will be used to inform recommendations relating to the development of
Good Practice models for future project preparation, implementation and facilitation. Participation in this
evaluation will assist us to help improve outcomes in the prevention of violence and to assist in the
development of future funding strategies.
This questionnaire is divided into five parts. In the first section we will be talking about how the project
came about and the initial planning stages of the project. In the second section we will be asking you
about setting up the project and training. We will then ask you about your ideas on the success of your
project, and the impact of the project on your organisation and participants. We will also ask about the
funding process and finally, your recommendations from the project.
Page 24
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
Project Description
Describe the project and their role
Describe the primary focus of the project:
Sexual:
Domestic:
Family:
Other (specify):
Describe the team and the setting (initial team, those employed, training, skills level)
Describe any community consultation and the outcomes from this process
Describe the level of parent and other key stakeholders support and involvement in the project
Planning and initiation
How did the project come about?
What was the perceived need?
Did your organisation support the project?
What were the challenges at this early stage?
Have you run a project like this before and is this project part of a larger program/focus?
Project design
What was the main focus of the project?
Behavioural change model:
Attitude change:
Skill building:
Behaviour change:
Page 25
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
How did the design and presentation method come about?
Did the design come from a model or conceptual approach you have previously used?
Were you working with a particular theory?
Theoretical considerations:
Social:
Cultural:
Individual:
Other (specify):
Explore the target group. What are the characteristics of the population?
Why did you choose the target group?
(Explore issues associated with gender/cultural/age/attitudes)
What are the challenges in working with this population?
Did this impact on the delivery model in the design?
Did you have to modify your design to address any culturally specific issues and what were
they?
In what way did the project fit with the goals of the RR program?
NASASV
The National Association of Services Against Sexual Violence (NASASV) developed the
national standards for the primary prevention of sexual assault through education. These were
released in 2009. The next funding for the Respectful Relationships Program round will be
evaluated against these standards.
Have you heard of the NASASV standards?
Did these standards inform the design or evaluation of your project?
Have you integrated these standards into your project in any way? Describe.
For reference:
1.
2.
3.
4.
5.
6.
Using coherent conceptual approaches to program design
Demonstrating the use of theory of change
Undertaking the inclusive, relevant, and culturally sensitive practice
Undertaking comprehensive program development and delivery
Using effective evaluation strategies
Supporting thorough training and professional development of educators
Page 26
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
Project Implementation
Setting up the project
What are the key factors that enabled and, conversely, inhibited implementation of the models?
What improvements are possible?
(For those projects delivered in schools/ whole of community) how well was a whole-of-school/
community approach implemented?
How did the project change between design and implementation? Why?
How did you assess project fidelity? Describe.
Staffing and Training
Describe the training that was undertaken for the project
How important was the training of teachers/educators/leaders in effectively delivering the project
models?
How could training be improved to increase capacity to deliver effective violence prevention
projects?
Project Evaluation
The project
In what ways was your project successful?
Did you see any changes in attitudes, behaviours, or risk factors?
What were the barriers?
What were the strengths and weaknesses of your program?
Did you identify any inherent biases in the project such as gender or cultural assumptions?
What would you change if you were to do this project again?
Overall do you think you achieved what you wanted to achieve and why?
What would you do differently next time?
Page 27
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
Other outcomes
To what extent have the particular models generated unintended consequences, positive and
negative?
What unforeseen benefits have come from your project?
What problems has the project uncovered? Were other key factors that need to be
addressed at the project level such as alcohol abuse, child abuse identified?
Were there any mechanisms for addressing the needs of participant who became distress or
experienced trauma due the project content?
Where there any mechanisms for addressing the needs of staff who became distress or
experienced trauma due the project content?
How can the program be modified to be used in other settings or with other targeted
participants?
Your participants
Did the target population enjoy the project? Was this project appropriate for your target
population?
How did you recruit participants?
Did participants participate in the whole program? Describe. What are the rates of participant
attrition? What is the trajectory of participant retention versus attrition at the various stages of
the project? And, what are the perceived causes of participant attrition?
What feedback did you get from participants?
Did you need any additional support or referrals for participants such as following disclosures or
other issues?
Page 28
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
Your organisation and partnerships
What could your organisation do to support these projects more?
Did you have appropriate access to resources such as computers?
What support did you have for facilitators?
What partnerships did you have to facilitate the project? How well did that work?
What has your organisation learned from undertaking this project?
What would grant recipients and organisers change in their program to provide improved
outcomes in terms of violence prevention?
Grant application
Did the grant help you to achieve your goals?
Did the grant result in you modifying the design of the project?
Were there restrictions from funding that made it difficult for you to achieve what you wanted to
achieve?
What feedback would you give the funding body about the grant and the process of applying?
Did you receive enough information and ongoing support once the grant had been awarded?
Overall Evaluation
Project Evaluation
Did you see any changes in attitudes, behaviour, or risk factors during or after participating in
the project?
How could you have evaluated the success of your project?
In what ways can better outcomes from violence prevention projects be achieved?
Finally do you have any other feedback you would like to share to help inform the next funding
round?
Page 29
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
Appendix E: NASASV Standards outline
Examination of projects against the NASASV Standards for Sexual Assault Prevention through
Education
A major component of the process evaluation is the comparison of guidelines outlined in the NASASV
Standards for Sexual Assault Prevention through Education with the aims and objectives of each project
funded under the ‘Respectful Relationships’ scheme. This included a comparison of any association
between the implementation of NASASV standards and project outcomes across all projects (but
particularly those in rounds 2 and 3) included in the evaluation. The following standards have been
adapted from the National Standards for effective sexual assault prevention through education to include
domestic and family violence prevention:
1.
2.
3.
4.
5.
6.
Using coherent conceptual approaches to project design
Demonstrating the use of a theory of change
Undertaking inclusive, relevant and culturally sensitive practice
Undertaking comprehensive project development and delivery
Using effective evaluation strategies
Supporting thorough training and professional development of educators
Using coherent conceptual approaches to project design
Using coherent conceptual approaches to project design relates to the articulation of the theoretical
approach upon which the project is based, demonstrating a clear rationale and research evidence relevant
to the target population.
Indicators
•
•
•
A quality project would include a coherent articulation of one or more recognised theoretical
concepts relevant to the purposes of sexual, domestic and family violence prevention.
Theoretical approaches will include an understanding of the gendered nature of society and the
over representation of men among perpetrators of sexual violence.
Theoretical approaches should support achievement of positive behaviours in relationships, as
well as responsibility for behaviour.
Demonstrating the use of a theory of change
Demonstrating the use of a theory of change relates to maximizing consistency between project aims and
the attitude change, skills development, or behaviour change strategies used in projects.
Indicators
•
•
•
•
An understanding of the social, cultural and individual factors that may result in
sexual, domestic and family violence occurring.
Articulation of the project’s role in working towards primary prevention.
Articulation of the behaviour change theory models influencing the project and the
logical relationship with addressing the factors identified with the occurrence of
sexual, domestic and family violence.
Understanding of the differences between attitude change, skill and behaviour
change and their impact on achieving primary prevention.
Undertaking inclusive, relevant and culturally sensitive practice
Page 30
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
Undertaking inclusive, relevant and culturally sensitive practice relates to ensuring the specific needs of
different and significant population groups are central to building primary prevention models and
projects.
Indicators
•
•
•
•
•
•
Explicit discussion and description about the assumptions within a project which are inherent at
both surface and deep structure levels.
Development of an initial profile of the target group.
Consultation with mentors, community leaders or representatives from the population group
leading to a consideration of the specific content needs that are relevant to the population group.
This may lead to surface and/or deep structure changes.
Ensuring the specific needs of population groups are embedded in the theoretical approach,
theory of change, content and delivery, evaluation, and the training and development of
prevention projects and educators.
Development of distinct educational projects for ‘selective interventions’ with at risk groups.
Ensure evaluation methods specifically support collection of data about the degree to which the
specific project design met the needs of the target population group.
Undertaking comprehensive project development and delivery
Undertaking comprehensive project development and delivery relates to developing projects based on
best practice research evidence from international and local literature, and practice knowledge.
Indicators
•
•
•
•
•
•
•
•
•
•
•
•
Education activities are linked to theory of change and key concepts.
Project activities are sufficiently diverse and engaging to achieve educational outcomes*.
Decisions regarding duration and intensity of projects to be made explicit.
Decisions regarding target populations of projects are made explicit.
Rationale provided for decisions regarding settings of projects.
Mechanisms for addressing the needs of survivors of sexual, domestic and family violence** are
provided in the project.
Rationale for decisions regarding gender of participants and facilitators is provided.
Discussion of how diversity is addressed by the project is provided.
Rationale for decisions regarding staffing of project (in addition to gender of facilitators) is
provided.
Understanding of context and engaging key people in the setting where a project will be
conducted, including building partnerships and consultation on local needs, is demonstrated.
Rationale provided for the context of the project and how this may affect the project’s delivery
and effectiveness.
Perceived benefits or impacts of project adaptation are demonstrated.
Using effective evaluation strategies
Using effective evaluation strategies relates to the collection of adequate data that indicates the
effectiveness of a project in achieving its stated objectives, leads to recommendations for refinement
and/or future rollout, and gauges its impact on participants and contribution to primary prevention.
Indicators
Page 31
Rounds 1 and 2 Evaluation – Preliminary Draft Report
•
•
•
•
•
•
•
P11007
Articulation of clear and realistic processes and intended project outcomes to be evaluated.
Demonstration of how evaluation is built into project design.
Discussion of evaluation approaches to be used and rationale for use is evident.
Ideally, both quantitative and qualitative methods should be used.
Provision of a strategy for long term evaluation follow-up, or which identifies barriers to such a
strategy, is documented.
Consideration of contextual matters that may influence evaluation outcome is documented.
Identification of methods to be used to disseminate findings beyond reporting to funding bodies is
documented.
Supporting thorough training and professional development of educators
Supporting thorough training and professional development of educators relates to ensuring that sexual,
domestic and family violence prevention through education projects are delivered by well prepared and
supported professionals or peer educators.
Indicators
•
•
•
•
•
•
•
•
Projects need to identify how educators will be resourced with knowledge of sexual, domestic
and family violence including a gender analysis, knowledge and skills to address survivors of
sexual, domestic and family violence and how to access support services, and knowledge of
prevention education theories and practices.
Projects need to demonstrate how educators will access skills based training to prepare educators
to deliver prevention projects, where the facilitators do not already have these skills.
Training provided to educators needs to include both education skills and moral/ethical stance to
work.
Projects using peer educators need to provide a rationale for their use, adequate training and
methods for ongoing support.
Projects need to demonstrate their rationale for choice of facilitator and the facilitator’s
qualifications and experience relevant to project delivery.
Projects delivered by teachers need to address their specific needs and articulate methods to work
in partnership with community-based violence prevention workers.
Projects need to demonstrate ongoing supervision of workers and attention to their safety.
Projects need to encourage networking with other educators doing similar work or with mentors.
Addendum to NASASV standards for purposes of the current evaluation
Project activities are sufficiently diverse and engaging to achieve educational outcomes*.
Education outcomes in sexual, family and domestic violence prevention programs are largely
based on the observable/measurable impact of program participation in regard to 1) awareness,
2) attitudes and 3) behaviour. An example of programs that focus on awareness raising and
attitude change often include providing statistics and definitions, and doing activities such as
‘debunking rape myths’. However, there is little evidence that attitude change and awareness
raising can alone, instigate behaviour change. The challenge of how to bring about actual
behaviour change through such programs is an important issue for project evaluation. As such,
the third facet of the propensity of each of the programs to machinate behaviour change will be
evaluating the educational approaches to changing behaviour through skills building activities.
These could include teaching negotiation skills, problem solving, decision making, critical
Page 32
Rounds 1 and 2 Evaluation – Preliminary Draft Report
P11007
thinking, coping with stress, coping with emotions, communication skills, interpersonal
relationship skills, conflict resolution skills, teaching positive relationship behaviours, help
seeking and bystander training.
Mechanisms for addressing the needs of survivors of sexual, domestic and family violence ** are provided
in the project**.
While not stipulated in the NASASV standards, it is important that the mechanisms for addressing
the needs of survivors of sexual, domestic and family violence also be extended to facilitators/
educators and non-participants at large. This addendum is provided following consideration of
the possible wide reaching impact of project participation beyond the scope of the survivors of
violence who are participating in the project and also, beyond the project participants in general.
In consideration of this, the extent to which projects promote social support, help-seeking and
provide accessible avenues in which to access these— for survivors of violence, project
educators, participants and the community at large will be examined.
Page 33