Value For Money Report
2014:
Transformation
www.bakertilly.co.uk
www.bakertilly.co.uk
Value For Money Report
2014:
Transformation
Ordered by the House of Commons to be printed on 2nd December 2014.
This report is presented to the House of Commons pursuant to Paragraph 26(3) of Schedule 2 to the Budget Responsibility and
National Audit Act 2011.
Value For Money Report 2014: Transformation
HC 855
www.bakertilly.co.uk
© Parliamentary Copyright House of Commons 2014
www.bakertilly.co.uk
Table of Contents
Section 1
Executive Summary
2
Section 2
Context
6
Section 3.1
Work Programmes
7
Section 3.2
Financial Savings
8
Section 3.3
Staff Engagement / Culture
9
Section 3.4
Quality
15
Section 3.5
Skills
19
Section 3.6
Structure
22
Section 3.7
Processes
25
Section 3.8
Outputs
27
Section 4
Appendices
29
www.bakertilly.co.uk
1
1. Executive Summary
High Level Scope
Each year, as part of the NAO’s annual accountability to Parliament, the Board on behalf of TPAC
selects a topic for review to assess the extent to which the organisation is achieving value for
money in its use of resources. This year the review focusses on transformation and seeks to
establish whether the NAO’s Transformation Programme has:
•
Delivered increased skills in key strategic areas;
•
Led to more effective and efficient deployment of staff;
•
Widened the delivery mechanisms and interventions to the NAO clients;
•
Started to change the leadership and management culture within the NAO; and
•
Achieved the intended savings.
Findings
We found strong evidence that the NAO has started to transform. There is real commitment from
the Leadership Team to the vision, and a genuine desire to improve the performance of the
organisation. We found a strong sense of passion for the people in the organisation, and a desire to
create a new approach that reflects corporate rather than individual performance.
Talking to directors from across the organisation, there was complete support for the principles of
the change. All recognise that the NAO needed to change and modernise with all seeing positive
benefits of the transformation. Most could point to some tangible evidence of change, but not all
Directors felt that the progress was as significant as presented by the Leadership Team and felt that
there is a great deal more to be done to embed the changes.
We found evidence to support the positive sense that the transformation programme has started
and is already delivering some change including:
•
A new integrated work programme that seeks to draw in expertise from across the
organisation;
•
Tough decisions taken to move out some of the people who had been at the organisation for
a long time;
•
Bringing in new talent from industry and the public sector;
•
A commitment to a cluster structure with a focus on joining together themes from previously
separate parts of the organisation
•
A wider range of ‘products’ and evidence that some reports are now drawing on experience
from across the organisation; and
•
Evidence that teams focus on systematic issues in government departments to help them
improve processes, bringing in the experience gained from working across government.
We found two key areas where the NAO needs to do more if it is to deliver the longer-term
benefits envisaged from the Transformation, summarised on the following page:
www.bakertilly.co.uk
2
Staff Perceptions
One of the major challenges that the NAO will need to address to achieve embedded change is to
convince its staff.
The 2013 people survey showed a fall in the confidence of staff in the Leadership Team. However
this survey was undertaken over the period when the main organisational changes were being
made and redundancies were happening. To test these trends, Pulse Surveys were undertaken
during 2014 and then the latest annual staff survey was undertaken in August 2014. These surveys
show a steady improvement in the positive answers given by staff. This is encouraging, particularly
the answer relating to an appreciation of the impact of clusters, which suggested that staff have a
much deeper understanding now than they did a year ago.
However the organisation scored poorly on some headings which will be crucial to successful
implementation of the Transformation Programme. Whilst there was a small positive improvement
in the trend, the level of confidence in the Leadership Team is too low with only one in three of the
staff having confidence and only the same proportion believing that they can challenge the way
things are done. The Leadership Team should set a target for a more rapid rate of improvement of
these indicators, which are crucial if the goals of the Transformation Programme are to be achieved.
As a minimum the organisation should aim for at least a majority of the people who work for the
NAO stating that they have confidence in the Leadership Team by the time of the 2015 Staff Survey.
Evidence collected from interviews and our own observations led to the conclusion that the
Leadership Team can appear somewhat defensive at times, and this may transmit itself to staff. This
could be a factor behind the confidence scores.
Management response: Implicit in the Baker Tilly findings is the point that the Leadership Team
should do more to improve the way they are perceived across the Office. While some of the 2014
People Survey scores on aspects of leadership have improved, overall they remain too low. However,
there are some contradictory movements in some scores with people acknowledging the increased
responsiveness and visibility of the Leadership Team without this translating into similar movements
in terms of overall confidence. We need to do more to analyse the survey results so that we
understand what might drive these contradictory messages and gear our response so it addresses
any underlying concerns.
Accountability
Whilst accountability for directorate matters could be traced from the Board to the individual
director, we found that the new team based cluster accountability was still to be bedded in and that
directors with cluster roles found it hard to articulate their accountability for cluster tasks, or how
the leadership team would hold them to account for this element of their work.
More recently we have seen evidence that the Board have reflected on this point (first quarterly
accountability meeting) and have agreed steps to monitor the implementation of these new
arrangements over the coming year. The ongoing implementation of these arrangements is an
important matter to address culturally, as a team based cluster model will only succeed if there is a
clear shift in accountabilities from the individual directors to the clusters. There is a strong desire to
make the clusters work, but we sense a nervousness about what this means and, not unsurprisingly,
a reluctance to completely let go and so this change will need constant re-enforcement over the
coming months.
www.bakertilly.co.uk
3
Conclusions
Our overall conclusion is that the NAO has made a solid start to its transformation. A number of
steps have been successfully completed and these enabling actions provide a platform for the
future. The next stage, which is part of the NAO Leadership Team's plan, is embedding the cultural
changes within the wider organisation.
To achieve the embedding of the changes, confidence of the organisation in the Leadership Team is
crucial. There is evidence that this is improving, but there is much further to go. The NAO needs to
also continue with the programme of helping define and explain how cluster accountability will
function alongside the traditional directorate accountability.
We also reached the following more detailed conclusions:
• Skills – New skills have been recruited in key strategic areas. There is a need for a ‘phase 2’ skills
strategy that seeks to equip staff with the key competencies that they will need to work as part
of cluster teams, thus embedding the new ways of working in cluster teams
• Deployment of staff – we found some evidence of a small number of high profile assignments
where the NAO is drawing staff from across teams and using the cluster model as a tool to break
down the ‘silos’ that have existed between directorates. The clusters are a mechanism that will
allow the NAO to move towards making this the norm rather than the exception, however this
needs on-going focus and attention from the Leadership Team to re-enforce this message.
• Delivery mechanisms and interventions to the NAO clients – there is some evidence that the
new cluster approach is leading to a broader range of services and a new way of delivering these
as interventions to help Government improve its performance. A start has been made with the
creation of cluster dashboards, quarterly accountability meetings and the emerging work to
prioritise the role of the functional leads within the clusters. These changes have been
experienced by some of the most senior people, but to make a real difference in the delivery
mechanisms and interventions, staff at all levels of the NAO need to experience the effects of
the transformation.
• Leadership and management culture within the NAO – the leadership team and other senior
managers support the principles of the transformation and improvements have started to take
effect. Most have started to work differently and the cluster model encourages a fresh
approach. At the time we collected our evidence, the accountability trail through the clusters
was insufficiently articulated and some directors we interviewed were skeptical that the
Leadership Team fully intended to delegate to the Clusters in the ways described. Consequently
there needs to be a constant reinforcing of the thrust to empower clusters to develop the
solutions that will then be supported by the organisation and also an explicit plan for members
of the Leadership Team to change and adapt their individual ways of working. The new quarterly
review of the accountability arrangements should serve to help keep a focus on this key issue.
• Savings – Staff savings have been secured in 2013-14 at a higher level than the initial target. In
this first year the costs of redundancies outweigh the savings as expected, however the NAO
remains on target for a payback within 2 years as planned. Given this, management should
continue to focus on the cost saving target to ensure that the payback is achieved.
www.bakertilly.co.uk
4
Recommendations
Our strategic recommendations are:
1. We understand that to move the organisation forward, there was a need to focus on team
goals rather than outputs. This was a logical first step, but to move to the next stage, the
NAO’s staff need to understand how they will be appraised and rewarded within the new
organisation. Therefore the NAO needs a stronger definition of the accountability thread
running from the Board, through the clusters, to individual directorates. This needs to
articulate the role of the leaders within clusters so that these are positions with authority
and accountability. Reviewing the recent papers on the Board review of the accountability
arrangements, it is clear that this point is accepted and seen as a high priority for the
organisation and so our recommendation is that the Leadership Team are tasked with the
ongoing implementation of these new arrangements and that the Board monitors the
impact, seeking evidence that objectives are set and monitored at a cluster level.
Management Response: Accepted. The LT will monitor along with the Board the impact of
the new accountability arrangements. There have been two cluster accountability reviews
and a further review is planned for December. The LT will undertake a full year accountability
review in March. The LT will then undertake an evaluation of cluster accountability as part of
the benefits realisation in May 2015. This will be reported to the NAO Board.
2. A significant step-forward has been made with regard the development of skills. We
recommend a training programme that helps staff members develop new working methods
and techniques that will be adopted in delivering more integrated work. We found some
good examples of reports developed using a more integrated approach and these could be
used as part of that training programme. Management Response: Accepted. The new ways
of working will be embedded in existing training and the new ways of working emphasised in
training courses and through the appraisal system. This will be supported by on the job
management to reinforce the new working methods.
3. Success depends on a consistent style and a clear focus on the goals of the Transformation
Programme being displayed by all, at all times. If transformation is to occur, the Leadership
Team need to change themselves. The Leadership Team need to ensure that the messages
they give directors and staff are entirely consistent with the culture of empowerment sought
and are consistent between one another. We recommend that the Leadership Team
members are set personal targets to work in this new way and their performance is
monitored through 360 degree feedback to test the extent of their consistency with one
another and with the stated goals of transformation. Management Response: Implemented.
The LT has agreed new objectives which clearly link to the goals for transformation and the
extent to which these objectives have been achieved is actively monitored at Leadership
Team meetings and will be critical in evaluating individual and collective performance at the
year end. The Leadership Team will also implement 360 degree feedback in 2014.
We also make seven more minor recommendations for consideration, described in the body of
the report and in Appendix D.
www.bakertilly.co.uk
5
2. Context
This report is the annual review of the value for money achieved by the NAO.
This year, the topic chosen was a review of the Transformation Programme. The scope of the
work was to report to TPAC and the Board on the progress the NAO is making with it’s
transformation, whether it is on course to deliver the benefits envisaged and to make
recommendations that will help the NAO drive through the cultural change it seeks. Detailed
objectives were set in order to focus on whether there is evidence of change in terms of work
programmes, financial savings, staff engagement, quality, skills, structure, process and outputs.
The NAO has a vital job to do and it needs the best and most highly motivated people to work
with Parliament to ensure that they are acting with probity and are working in the best and most
effective ways. The vision seeks to empower NAO people to work in innovative ways to facilitate
the delivery of the highest quality and best value for money services possible.
To do this the NAO has designed a model that moves from a traditional hierarchy to a more
modern team-based approach, so that the NAO talent works to maximum effect to deliver these
important interventions. This is shown in the chart below:
This value for money assessment was to test the extent to which the organisation has changed,
is changing and plans to change in the future. The report aims to describe those steps that have
already been taken, and those still to be navigated.
www.bakertilly.co.uk
6
3.1 Analysis (Work Programme)
Background
The NAO had reported in the January 2014 update to the Board (item 4) that there is now a
more integrated work programme which is designed to position the NAO up well in line with the
next general election in 2015.
Work Programmes
An integrated Work Programme has been set up in relation to 2014/15 which describes the jobs
which are to be undertaken and the applicable responsible cluster with the named responsible
leads defined on the MKi system. We obtained the Work Programme which was available and
can confirm that the layout is consistent with the expected headings in line with good Project
Management methodologies and is similar in format to PRINCE2 project management practices.
Business rules have been set up for the work programmes with the management of budgets,
people and reporting of the programme all being set out in a Change Control Framework. All
changes need to be approved by the Executive Lead from the applicable cluster and this is
consequently captured by the IPO which provides monthly returns on changes to the
programme. The Work Programme details a number of output products for the upcoming year
which supports the NAO’s aims to develop more integrated products – these are described in
more detail in section 3.8 of this report.
Develop Strategic Themes
It was confirmed during the review that the movement towards an integrated Work Programme
was a significant piece of work which the NAO undertook and this is the first of its kind within
the organisation, as such this is evidenced by the 2014/15 work programme.
Conclusions
We found that significant work has been undertaken in developing the cluster structures. These
new structures will aid integration. In financial year 2014/15 work programmes were integrated
into a more formal plan which is forward looking rather than retrospective.
We conclude that the Work Programme has developed into a more robust document. The Work
Programmes could still be developed further and as such we have made one recommendation in
Appendix D. The programmes do not detail a senior lead and as such there is a limited traceable
responsibility for the production and review of the work, unless the work programme is used in
tandem with the MKi system, this was stated by the Head of Policy and Legal. It is recognised
that the Work Programme assigns tasks to a particular cluster but a named individual is not
clearly identified until the MKi system is used. This should not deter from the fact that there has
been significant improvements made in the development of the Work Programme and there is a
clear process now in place with regards to amendments being made and the monitoring of the
programme.
www.bakertilly.co.uk
7
3.2 Analysis (Financial Savings)
Background
The savings that are proposed from the Transformation Programme are all to be achieved
through staff restructuring. According to the original transformation bid submitted in March
2013, there would be an expected £2.5m staff cost saving per annum over the life of the
Transformation Programme with an overall expected payback period of 23 months. During
review of the January 2014 update to the Board it is stated that there had been a saving to date
of £1.3m through the restructuring process and that the estimated saving would be
approximately £2.8m.
Findings
We identified that the net staff salary saving for 2013/14 was £1,388k and the 2014/15 saving is
expected to be £2,880k. The 2013/14 figures were tested to staff leaver documentation to
confirm staff have left. During the various Board updates which were reviewed it was stated that
there is a part year saving of £1.3m due to the restructuring happening mid year, which we
confirmed through testing. It should be noted that we found some inconsistencies in the update
reports where the £2.5m permanent saving appears to reflect the savings made in 2013/14
rather than savings for 2014/15 and beyond, however we can confirm that this is made clear in
the supporting Annex of the NAO’s update document that the 2013/14 saving is £1.3m.
Redundancy / Severance costs
In March 2013 the NAO bid for £4.67m to cover the redundancy costs for the Transformation
Programme. Upon review of a spread sheet submitted by the NAO in order to confirm saving
values we identified that total exit costs were approximately £4.2m. When this value is
compared to the restructuring value stated within the April 2014 update we identified that there
is a variance of £0.2m as it is reported that the total spend was £4.4m with £0.37m being
returned to the Exchequer. This was confirmed as being a last minute change with regards to
pension rights for an employee. We can confirm that the NAO is on track with regards to the 23
month payback period of the redundancy costs when the £2.5m permanent savings have been
taken into consideration. It was confirmed by the Financial Controller that as the exit costs were
less than expected, this difference was returned to the Exchequer, although this has not been
reported to date.
Conclusion
We identified that the NAO has achieved permanent annual savings of £2.8m from 2014/15
onwards and that £1.3m savings have already been achieved with respect to 2013/14 as
reported in the January 2014 update presented to the Board.
www.bakertilly.co.uk
8
3.3 Analysis (Staff Engagement / Culture)
Background
The Transformation update in January 2014 states that the NAO is moving into a cluster working
structure whereby six clusters are formed, each of which have a more empowered and
independent working style and also individual strategic aims. Five key areas are targeted in order
to implement this, these are; Empowerment; Process light; Coaching; Data led management;
and Visible Leadership and accountability.
The NAO has always conducted annual Staff Surveys but at the second half of financial year
2013/14 the organisation ran “Pulse Surveys” to collect early evidence on staff attitudes and
their thoughts over the quality of the leadership. Pulse Surveys were introduced in light of
disappointing results in the annual Staff Survey undertaken in September 2013. A series of
events were held during November 2013 (called NAO2014) whereby the results of the Staff
Survey were considered which led to a recognition that the concerns of people about the
Transformation process were wider than expected and also that work needed to be undertaken
to rectify this. In this report we have drawn on the results of the Staff Surveys (2011-2014) and
Pulse Surveys (January and April 2014) to gauge the opinions of staff and whether attitudes are
improving or declining.
We were advised against undertaking further independent surveys since our initial fieldwork
coincided with the peak season for a significant proportion of NAO staff. As such we focused our
interviews on the Leadership Team and Directors. This meant that we relied upon the Staff and
Pulse Surveys for evidence of the staff culture throughout all levels of the NAO and how the
culture of transformation has permeated the organisation. There was also evidence provided
that there has been some involvement from staff focused on the future of the NAO, including
cluster and grade sessions.
Interviews
As a part of the assessment of culture change attributable to transformation, interviews were
undertaken to determine the perceived level of progression made. Key findings include;
•
The transformation has resulted in instances of integration across teams with numerous
interviewees noting the use of VfM staff on financial audits and vice versa;
•
The Leadership Team reflected that in explaining to staff the reasons for the transformation
programme, greater focus should have been given to the external drivers for change;
•
The transformation has resulted in the development of the Skills Strategy which identifies
the needs of clusters. It was recognised that there is still work to be done in this area, in
particular with regards to technical skills, such as commissioning.
Please note that Director and Leadership Team quotes captured during the interviews are
captured in Appendix A.
www.bakertilly.co.uk
9
Workshop
In addition to the interviews we also ran a workshop whereby we assessed the Leadership Team
capabilities and the leadership culture within the organisation. We did this by targeted
Leadership Effectiveness Analysis. This comprised a series of online competency, behavioural
and cultural profiling tools. A brief was sent out to the leadership team on the 14th April 2014 in
order to provide details as to the scope of the workshop and what the perceived and intended
outcome of the session would be. Another output from the workshop was to review evidence
produced by management to demonstrate a comprehensive articulation of the change in
Leadership the organisation is targeting. This included consideration of the structure,
competencies, behaviours, processes and risk appetite. Results of the workshop concluded that
the attendees believe that the Transformation Programme has revitalised a stagnated
directorate with a noticeable increase in collaborative working and empowerment.
The workshops also discussed the accountability of staff within the new structure and the
communication of the progress of Transformation. It was noted that accountability of the
clusters is being monitored by the quarterly accountability meetings and that these meetings
review the achievements of the clusters, strategy achievements and also general comments
surrounding the clusters.
Workshop Presentation and Agenda
The workshop was held on the 3rd June 2014 which consisted of a two hour concise assessment
of the Leadership Team and focused on three key areas. These included; case studies of
Transformation from other organisations; Challenges to Leadership Team and Reflections of the
Leadership Team. This workshop allowed a forum to provide honest feedback and an
identification of the challenges which face the Leadership Team in light of the Transformation
Programme. Examples of the presentation are shown below;
www.bakertilly.co.uk
10
As a part of the workshop we developed a model which details the hallmarks which comprise a
healthy Transformation Project. The model (below) allows the NAO to gain insight into the key
factors which we believe are required to successfully implement a programme of
Transformation. Each of section shows a number from a scale between 1 – 5, reflecting
development progress assessed through our observations of the Leadership Team. This is the
scale of maturity within the organisation with 5 representing full maturity and 1 representing
low maturity. Projects only pass the from the formation stage to developed once they have
scored the maximum of 5, signifying fully formed/developed, and so on through the model.
With regards to the headings in the table below, these represent the stages of a Transformation
Project over time from formation, to development, to being embedded and finally to
stagnation. Note these are our views on where the NAO is on its journey of transformation:
HALLMARKS OF EFFECTIVE
TRANSFORMATION PROJECT
Formation
Developed
Establish a sense of urgency for change
5
1
Form a powerful coalition to lead the change
4
Create and communicate a change vision.
5
2
Empower others to move change forward.
5
2
Celebrate short-term “wins” and recognise those who help
1
Build on success; align people and systems with new ways
3
Keep the message consistent; champion the vision.
3
Embedded
Stagnated
This shows that the Transformation is at an early stage with good progress in terms of formation,
but further work to complete the development of the implementation plan and so far no
evidence that changes are embedded.
The workshop also assessed leadership within the NAO. In order to fully implement a
Transformation Programme there needs to be a strong element of leadership and strategic
direction which the organisation will have confidence in and be prepared to follow. The model
below shows “Best Practice” which we would expect to see in a strongly led organisation. Each
of the sections contains a number on a scale between 1 – 5 as before:
BEST PRACTICE
Formation
Developed
A shared vision of the future
5
3
Clarity about ownership and accountability
5
1
Supportive culture and shared values
4
Motivation and inspiration
5
2
Well formed leadership team
5
4
Alignment of performance across organisation
3
www.bakertilly.co.uk
Embedded
Stagnated
11
This table shows that we found a well formed Executive Leadership team with a strong vision and
values and a sound understanding of the alignment of processes and people. In addition we believe
that the Transformation Programme has been developed for a strategic purpose, although the
implementation of this programme requires more work. Further clarification is needed on the role of
individuals in clusters and their wider contribution to NAO aims and objectives.
Staff Surveys and Pulse Surveys
Staff Survey results are compared annually to monitor progress and changing attitudes, with areas of
development identified. Usually the NAO conducts one Staff Survey per year, the last two of which
were undertaken in July 2013 and July 2014 with the results published in August and September
respectively. These are undertaken using an independent, expert provider. The more recent Pulse
Surveys were undertaken in-house during 2014 to provide additional, more frequent information.
2013 Staff Survey
Overall the results of the Staff Survey 2013 showed a decline in the confidence of staff in the
leadership when compared to the results from 2012. This was largely attributed to the introduction
of the Transformation Programme and in particular the restructuring and the redundancy
programme that was introduced at this time. Staff felt that there were issues surrounding
communication, the clarity of the Transformation Programme and how this would affect their roles
on an individual level.
Pulse Surveys
In light of the disappointing results from the 2013 Staff Survey, additional Pulse Surveys took place in
January 2014 and April 2014 to gauge opinion of the crucial first year of the Transformation
Programme:
• There were improvements in the scores of the Pulse Surveys in both January and April 2014 as the
average score in the 2013 People Survey was 42% agreement, the January Pulse Survey this rose
to 48% and April this was 49%
• In terms of clusters the most positively scoring cluster for both January and April 2014 was
Corporate Services with average scores of 56% and 60% respectively. The cluster with the biggest
positive improvement in scores was that of Delivery Through Networks which rose from 39% in
January to 50% in April
• In terms of the most improved answer to a question in the survey, this was for “I understand the
strategic issues which are the focus of my cluster”. The score for this question improved from 62%
in January to 73% in April
• The question that fell the most was that “I have a strong personal attachment to the NAO”. This
fell by 5% from January to April
• The lowest scoring questions as a whole remained the same from January to April 2014, although
it should be noted that in all instances the scores increased by between 3 and 5%;
o Overall, I have confidence in the decisions made by the Leadership Team
o Progress with the strategy is being communicated clearly
o The Leadership Team is taking appropriate action on the results of the 2013 people survey
o I would recommend the NAO as a great place to work
o The Leadership Team makes an effort to listen to staff and to take account of our views
• This suggests that in April 2014 there was evidence of an improving trend of opinion amongst the
staff, but still work to do to return confidence to the pre-transformation level.
www.bakertilly.co.uk
12
2014 Staff Survey
The results of the 2014 survey show some positive signs of progress but also that there still
remains a lot to be done to embed the changes. The vast majority of scores were more positive
than in 2013 with 55 of the 59 comparable questions showing an improvement. The overall
averages show that 54% of the staff agreed with the statements in 2012, 47% in 2013 and 52% in
2014. Below we extract some of the questions that are key to this review of the Transformation
Programme:
• 72% of the staff say that they have a clear understanding of the NAO’s objectives, but only
55% of staff feel they are committed to them. This is up from 52% in 2013 but down from
2012 where it stood at 67%.
• Confidence in day to day managers is high, for example ‘my manager motivates me to be more
effective in my job’ is 69% up from 62% in 2012 and 2013 and ‘My manager is open to my
ideas’ is up to 80% from 77% last year. However, staff seem less clear about the organisation’s
objectives as only 46% say that ‘My manager helps me understand how I contribute to the
NAO’s objectives’ although this is an improvement from 2013 when it stood at 42%.
• In terms of Values, a large proportion (69%) say that they understand them, however only 31%
believe that the Leadership Team leads the organisation in line with these objectives, up from
a low of 25% last year but not yet back to the pre-transformation level of 40%. Slightly more
(44%) are of the view that Directors lead in accordance with these values, so staff seem to
have a lower opinion of the Leadership Team than the Director group.
• Visibility scores for the Leadership team have improved, back to 49% which was the score in
2012. But only 39% believe that the Leadership Team has a clear vision of the NAO (up from
2013 when it was 32% but not yet back to the pre-transformation score of 49%). Only 30% say
that they have confidence in the Leadership Team, which is up from 20% in 2013 but still
below the 2012 score of 49%. There was however a large increase in those who felt there was
evidence of the Leadership Team taking ‘appropriate’ action on the basis of the last survey, up
to 38% from just 20% last year.
• These low scores for confidence in the Leadership Team contrast with ‘Overall I have
confidence in the decisions made by my Director or Director-General’ which stood at 75%,
largely unchanged.
• Only 35% of staff feel that ‘I think it is safe to challenge the way things are done in the NAO’
although this is an improvement from 2013 when the score stood at 25%, but not back to the
pre-transformation score of 41%.
• There was a significant improvement in the number who say that they understand the
strategic issues which are a focus of their cluster, up to 57% from 44% last year.
• Only about half of the staff (53%) say that they feel motivated to help the NAO meet its
objectives, slightly down from last year.
• Only 35% feel that the NAO has a culture of collaboration across teams, although this is up
from 28% in 2013 and 32% in 2012.
www.bakertilly.co.uk
13
Conclusion
The staff survey results paint a mixed picture.
On the one hand there some positive findings. There was a significant dip in staff confidence in
2013, a period where the effects of the redundancy programme were being felt across the
organisation, and consequently an impact on how positive people viewed the NAO. Over the
Pulse Surveys in January and April 2014 and the annual Staff Survey in August 2014, there have
been steady rises in confidence, suggesting that the concerted effort to communicate more
widely, to promote the clusters and to move to a culture with more empowerment, is starting to
have an effect.
A particularly important trend is that staff are starting to develop an appreciation of the cluster’s
goals, and that will help bind them more strongly into the transformation.
There are however a number of conclusions that highlight that there is a very long way to go to
embed the cultural changes. It is a concern that confidence in the Leadership is so low and that
majority of staff do not appear to believe it is ‘safe’ to challenge. Only just over half of the NAO’s
people say they are committed to its objectives and more than two thirds say that they do not
believe that the Leadership Team apply the values set for the organisation.
So there is a lot to do to embed these changes. The Leadership Team have taken on board the
concerns of staff as expressed in 2013 and that is having some positive impact, but the changes
are small and we would have expected that by this point the confidence in the Leadership Team
would have improved more significantly than it has.
www.bakertilly.co.uk
14
3.4 Analysis (Quality)
Background
During the update presented in January 2014 it is reported that “the quality of our work has
improved over the last four years, confirmed through independent feedback from our
stakeholders and external quality assessment”. In addition to this it is reported that this increase
in quality work which has assisted in the formation of a new role for the NAO in relation to the
abolition of the Audit Commission. The NAO has also taken expert advice from Boston
Consulting in order to assist in the Transformation Programme.
Transparency Reports
Transparency reports act as an internally produced but externally reported statement of audit
quality for the applicable year. To make an assessment as to the audit quality we reviewed
Transparency Reports for 2011 to 2013. The 2011 report states that additional assurance was
undertaken by the Audit Inspection Unit of the Professional Oversight Board. Other key findings
include;
•
Reports focus on the three E’s (of VfM) and statutory audit rather than the proposed full
array of reports;
•
Quality Control is built into the work through policies on recruitment, training and
continuing professional development, the Code of Conduct, and through specific
operational guidance; and
•
All staff are required to undertake Continuing Professional Development (CPD) and to take
responsibility for updating their technical knowledge.
With regards to the December 2012 Transparency Report, key findings include;
•
The governance structure of the NAO had changed since the previous report insomuch as
the NAO had become an individual corporate entity. As such there is more prominence on
internal management of NAO resources; and
•
Issues surrounding audit quality arose, these include; Independence and ethics; local
accountability; policy reforms; and financial reporting in the public sector.
The report also stated that the following issues were addressed from the previous year’s report;
•
The internal cold review process found potential improvements; and
•
The policy on secondments was strengthened with mechanisms in place to increase
independence.
The final key finding is that the report refers to a number of quality assurance mechanisms in
place which include; external review by Oxford University; internal reviews on a sample of
reports; post project reviews and requesting of feedback from the clients. See Appendix M for
the AQP framework which outlines the quality expectations from the NAO.
www.bakertilly.co.uk
15
The Transparency Report for 2013 was also reviewed. We identified that there is an increased
drive in developing members of staff and leading to better quality reports, in addition to more
dimensions of work being undertaken. These include; performance improvement; support to
Parliament; and internationally. The report also states that internal and external reviews are
undertaken on the output products in order to ascertain whether the quality has increased and
also that the review process is consistent with previous years whereby a senior member of the
engagement team reviews the audit work and then an Engagement Director reviews the work.
External Assurance
External Assurance is provided to the NAO and is recorded centrally on a spread sheet which
contains a number of links to the actual output reports, these reports are held centrally on the
NAOTank intranet system. We assessed the external assurance which is split into 13 assessment
criteria. These are shown on three separate charts below, please note that these refer to all
report types which were submitted to us on the central spread sheet and as such this was
completed on a sample basis.
The chart shows that the external assurance opinion of the
quality of the reports, in all instances, fell in 2012. Since the
introduction of the transformation programme the quality of
the scope, financial analysis and qualitative analysis has
increased with the quality of the administration decreasing.
This chart (left) shows there is a less obvious trend in the
quality elements of the report. The only element that has
shown year on year improvement is that of the synthesis of
analysis and VfM improvement which has increased from 2.82
in 2011 to 3.36 in 2013. The quality of methods declined from
2011 – 2013 but then shows an improvement in 2014. Graphs
and stats improved from 2012 – 2013 where the quality
assessment peaked before a drop in 2014. Finally there was a
steep two year decline in the quality of the structure before a
steep increase in 2014.
For the remaining assessment criteria, and particularly the
average rating of the reports, there is a more obvious and
identifiable trend in the quality. The average report quality
dropped to 3.10 in 2012 and has risen to 3.35 in a two year
period, peaking in 2014. In terms of the VfM assessment
criteria there was no external assurance data available for 2013
and 2014, as such we identified that the score had dropped
from 2011 (3.13) to 2013 (2.75) and consequently is an area
for potential targeted improvement.
www.bakertilly.co.uk
16
As the chart (right) shows the year which contains
the best quality reports is 2014 (average score is
3.35 out of a possible 5) with the lowest year being
2012 (average quality score of 3.10). This suggests
that the Transformation Programme has led to the
increase in report output quality.
However it should be noted that the increase in the
average score from 2012 to 2014 is 8.1%. As such it
could be argued that the Transformation
Programme, although contributing to the
improvement of report quality, is not having the
expected impact.
Client Feedback
In addition to the external feedback we also analysed and assessed the client attitude to the
reports which have been produced.
For each of the reports the client had the opportunity to grade the auditor on a number of
questions which are asked in relation to the work, at each stage there is also an opportunity to
make a comment that supports the mark provided. This can allows more substance and is
particularly useful as the marks for each question are from 1 (good) to 3 (poor). In total 21
questions are asked of the auditor. We reviewed the client feedback and made a year on year
comparison of the average scores for each question in order to ascertain whether or not there
has been an increase in the perceived quality of the outputs. In addition to this we compared the
average score of the reports year on year in order to see, overall, whether quality has improved.
As the chart above shows the general
opinions of the clients with regards to the
questions is, on the whole, less positive in
2013/14 to 2012/13. The only exceptions to
these are for questions; 2.1, 3.1 and 3b with
the remaining 18 questions showing more
positive scores in 2013/14.
The chart above that the average score is
broadly unchanged.
www.bakertilly.co.uk
17
Conclusion
In terms of the external assessment we can confirm that there has been an improvement in
quality since the introduction of the Transformation Programme which is reflected in the scores
provided by the external assurance provider. However in terms of “improvement over the last
four years” we have identified aspects of the reports which have not improved and we also
confirmed that the average quality of work actually dropped from 2011 to 2012. In addition to
this the increase in quality (from 2012-14) has risen by 8%. We have ascertained that since the
start of the Transformation Programme there has been an overall increase in the opinion of the
clients in relation to the quality of the products which are produced by the NAO. This was an
improvement of 1.45% in the customer feedback scores.
Going forward we would expect to see further improvements in customer satisfaction and this
should be monitored by the NAO.
www.bakertilly.co.uk
18
3.5 Analysis (Skills)
Background
The NAO had reported in the January 2014 update to the Board (item 4) that “the core skill of
financial management and accounting is being integrated much more with in-depth sector
knowledge and targeted acquisition of skills”. In line with the new structure and employee mix at
the NAO there is a greater emphasis on the organic growth of staff, internal development and
empowering of individuals in order to emphasise that good work will be rewarded and that the
NAO will invest time and money into the correct candidate.
Skills Strategy
During review of the January 2014 update we identified reference to a “Skills Strategy” which
underpins the Transformation programme. In addition to this the report states that the
individual clusters are responsible for implementing the Skills Strategy in their own way, but each
cluster must follow the principles of the document. The Skills Strategy contains an action plan
(Appendix G) which details a lead officer and a challenge role which is in line with excellent
project management methodology and accountability. The Transformation Programme skills
development per cluster are being focused upon the model (see Appendix M). The model ranks
the importance of the applicable values of the Transformation Programme going forward with
core financial audit skills being the base of the skills mix. Although sector skills and local service
delivery is being placed above this in terms of importance and we can confirm this strategy is
being followed.
Competency Frameworks
Before the Transformation Programme begun the NAO had an online Competency Framework
(Appendix H) which was available to all members of staff on the organisation’s intranet site. As
Appendix H shows; the Pre-Transformation Competency Framework was very brief with bullet
points stated underneath a role and abilities which would be expected from a member of staff.
Since the incarnation of the Transformation Programme the Competency Framework has been
amended in order to incorporate the ideals, skills mixes and cultural integrations which the
programme is expected to deliver. This framework acts as a matrix as the staff members can see
the core skills, current skills for their role and also competencies for other positions. It is also
clear that there is greater emphasis on accountabilities, empowerment and self development in
order to organically grow into their role and see how to develop into other more senior roles.
Ownership and empowerment features frequently throughout the competency framework in
addition to focusing upon work of the cluster as an individual unit but also cross cluster working
as an organisation.
Staff CV’s
It was confirmed by the Interim Head of Governance that as an output of the Transformation
Programme each member of staff has been required to upload a CV onto the intranet in order
that the organisation has access to an individual’s professional skills and make the organisation
more transparent. It was stated by the Interim Head of Governance it is recognised that not all
CVs have been uploaded as it was estimated by the NAO that 70% of staff CVs had been
uploaded.
www.bakertilly.co.uk
19
Development case study
One objective of the Transformation Programme is to organically grow current members of staff
within the organisation and focus on developing deeper expertise within the clusters, The Head
of HR Policy confirmed that this will be embraced going forward. We obtained and reviewed the
details of an example of the NAO organically developing a member of staff with external training.
This specific case study refers to a member of staff that put forward a business case and
application in order to undertake an external training course and a professional qualification. The
business case, written by the candidate, describes the reasons as to why they believe they
should be put forward for the qualification and the training programme, many of these reasons
stated by the candidate seem to align with the overall skills objectives of the NAO (Appendix P).
This member of staff is currently on the training course and is progressing though it.
Line Manager Briefings
In terms of the training and support of staff we requested training slides of any appropriate
events which the NAO undertakes in order to steer and guide staff into accepting the
Transformation Programme. We obtained and reviewed an example from the Managing Major
Programmes and Projects (MMP) cluster which had a Line Management Briefing for the staff for
Human Resources related issues (starter, leavers, training, performance management etc…..) and
also for the cluster as a whole (for line managers, this acted as a training and feedback session
for the attendees).
The Line Manager briefings, both general and HR, are informative and relay information which
would be useful for the attendees to understand. We obtained this as an example presentation
and it was stated by the Diversity Manager that the same presentations were shown to each of
the clusters, and as such, it is reasonable to suggest that there would be little or no cluster
specific information presented. Below there are examples of slides of the Line Manager briefing.
www.bakertilly.co.uk
20
Recruitment Policy
The Recruitment Policy was reviewed in line with the January 2014 update which stated that
“financial management and accounting is being integrated ‘much more with in-depth sector
knowledge and targeted acquisition of skills”. We expect the Recruitment Policy will reflect the
targeted acquisition of skills. The Recruitment Policy contains information surrounding the
following areas; The requirement to recruit; The role of HR in recruitment process; Job
descriptions and person specification; Advertising; Agency workers; Interviews; Assessments;
and Offers. It should be noted that the Recruitment Policy was last updated on the 29th July 2013
and as such does not include many of the achievements and work milestones which have been
accomplished within the NAO during that period. It was confirmed by the Head of HR Policy that
there are plans to update and revise the Recruitment Policy in order to reflect the targeted
acquisition of skills in line with the Transformation Programme.
Conclusion
To conclude we can confirm that there has been good progress with relation to the “Skills”
element of the Transformation Programme and the movement towards the key goals of organic
growth of staff from within the organisation and also the increased empowerment of staff
members. The statement made in the January 2014 update suggests that this integration is an
on-going process which is consistent with our findings on the matter. We have made two
recommendations in light of the skills findings which can be seen in Appendix D.
.
www.bakertilly.co.uk
21
3.6 Analysis (Structure)
Background
The NAO had reported in the January 2014 update to the Board (item 4) that Cluster working has
led to a more integrated and flexible offering which has enabled each cluster to have a wider mix
of assurance products. The model of Cluster working is the primary method of organisational
restructuring which has been proposed and reported on and as such will be investigated in more
detail during this section of the report.
The new cluster (or team based) accountability aims to foster a culture of collective delivery,
staff engagement and collaboration as the new way of working. Monthly and quarterly reports
enable the leadership team to hold clusters to account for business delivery and outcomes. Each
cluster has a lead Director responsible for planning, staff development, quality and knowledge
and innovation. Although there are leads for each of these functions, all Directors share
responsibility for the effective delivery of them. They are supported by integrated Corporate
Services (such as the new Integrated Programme Office). At the quarterly meetings clusters are
held accountable by ELs for the operational performance of cluster, people engagement, skills,
client insight, progress on strategic outcomes and work programme deliveries.
New Expertise
Contained within the Skills Strategy is an additional structure diagram (Appendix O) which details
the strategic cluster based way of working and links this into the development of new expertise
within each area of the cluster. This diagram centres around the NAO strategic focus point with
methods of developing expertise surrounding this and feeding directly into the clusters. The
diagram shows how the methods for developing expertise has fed into four of the clusters with
examples of the new expertise which have arisen from the implementation of the cluster
structure. It should be noted that two of the clusters do not seem to have reported examples of
new expertise being developed and as such this structural benefit can be considered a “work in
progress”. Although we do recognise the work that has been achieved for more integrated
structural working.
Quarterly Cluster Reviews
As a part of the Transformation Programme there are “Quarterly Integration Review” (QIR)
meetings which began in July 2014 onwards as was agreed at the Leadership Team meeting in
May 2014. As a part of this QIR there will be an assessment of four key cluster enabling roles,
these include; Strategy & Planning; Quality; People; and Knowledge & Innovation. Each of these
roles has an Executive Lead assigned and as such there is an element of accountability to the
implementation and review of each of these roles. This will also provide a forum to “check and
challenge” each of these four key roles whereby issues in the cluster working can be quickly
identified and rectified.
www.bakertilly.co.uk
22
Before and After proposed structure
The diagram below is a visual depiction of the proposed amendments which will impact the
working structure of the NAO from a silo based working methodology to a more integrated and
communal method of work.
As the diagram describes, the pre-transformation structure consisted of a number of different
departmental hierarchies, each of these have a strategic direction which ultimately feed into the
Comptroller and Audit General through a number of Assistant Audit Generals. The key
characteristics of this structure confirm that although there was a delivery of business results,
there was little or no integration, this inhibited staff utilisation and also sharing of knowledge
across the NAO.
It should be noted that at present the NAO has only partly implemented the proposed changes
and as such will not be realising the full benefits of the structure until this is fully in place.
Additional Structural Assessment Surrounding Reporting Line Changes
The transformation process has resulted in a change in reporting lines for members of staff who
now have a Line Manager from their cluster who has developmental duties for that individual.
However, we observed some confusion from respondents as to when this change had taken
place and the details surrounding these changes. Enquiries with a HR Specialist and Business
Partner revealed that the change, which was implemented on 1st January 2014, was only
applicable to certain members of staff. We asked a sample of 16 members of staff whether they
had been notified by this change with 15 of these saying that they had. This is an example of
good communication.
www.bakertilly.co.uk
23
As the graphs below show, the majority of the reporting line changes which were sampled were
made by email and in December. These results again show the issues surrounding communication
of this important matter and could lead to a lack of understanding if not communicated
effectively.
Potential and Performance Grid (Appendix C)
The transformation process has resulted in the development of a Potential and Performance Grid
that is used as part of the appraisal process for members of staff. However, a number of issues
arose during interviews with regards to the use and application of this model, and as a result we
decided to perform further enquiries and testing in this area to clarify the detail.
Objectives
Transformation objectives were developed as part of the change process and it is important that
there is a ‘golden thread’ from these objectives through to the individual objectives of Directors.
We can confirm that all Directors have objectives assigned that are connected directly through to
the transformation objectives. We obtained and reviewed director objectives which are stated
within Appendix I which are reviewed three times each year. In addition to this we identified that
on-going accountability for personal and skills development based on clear personal objectives
linked to those of the business reviewed by ELs periodically throughout the year. All staff have
objectives and performance assessments. The area, in need of further development, is helping
Directors understand how to balance directorate and cluster responsibilities. We found that there
was a lack of clarity over the prime objective, so that those with cluster-wide roles were unsure
on the extent of their responsibilities over peers for matters that were historically director tasks.
Conclusion
We are able to confirm that there has been progress made with regards to the implementation of
a cluster working structure which seeks to avoid silo working and promotes a more integrated
organisation. It should be noted that this integration of cluster working is still in the progress of
being implemented and as such the full benefits of integrated working will not be experienced
until this structure is fully embedded within the organisation. The changes from individual to
team-based accountability demands a change in culture and there is further work to help all
directors understand this shift in emphasis and to reach the point where this new accountability
framework is truly embedded in the organisation.
www.bakertilly.co.uk
24
3.7 Analysis (Process)
Background
The NAO had reported in the January 2014 update that “the goal of the Transformation is to be
process light” and mentions that there is an intention to move away from templates and a move
towards a “business partnership model for corporate services”. In order to achieve this there
have been three primary driving factors which are set to contribute to achieve these objectives.
These are; removing senior layers; business partnering model (the formulation of the Integrated
Programme Office); and implementing new systems which should increase the efficiency of the
NAO in terms of output production and document storage.
Removing Senior Layers
Following our review of the financial savings made we can confirm that senior layers have been
removed which include 11 Directors; 9 Audit Managers and 20 Audit Principals. With regards to
the redundancy programme these senior layers equate to 44% of the total staff members made
redundant. The removal of layers enables the products to benefit from input from operational
staff with a more “hands on approach” and enables the publishing of their products in a more
efficient fashion.
Business Partnership Model
This model takes the form of the Integrated Programme Office (IPO) which was launched in April
2014. The IPO is required to undertaken the following key actions;
•
Bring together management information to support Leadership Team and Cluster decision
making, to improve integration and efficiency;
•
Integrating various corporate strands such as strategy, policy, financial and staffing and
working closely with the cluster support functions such as the Strategy and Planning
Directors;
•
Managing the NAO business cycle of the strategy, business plan, delivery of corporate
outcomes / objectives and developing the corporate business / performance reporting to
the Leadership Team and the Board;
We obtained and reviewed a Cluster Dashboard which has been in place at the NAO since May
2014. This dashboard reflects all current work in line with budgets and also reflects utilisation of
staff in the clusters. This is a robust mechanism for tracking the allocation of work within the
programmes and also records reports which have been made to PAC. The IPO also maintains the
work programme (section 3.1) and records amendments and alterations to this.
Implementation of new systems
In relation to the implementation of new systems within the NAO, three main projects are
underway;

Enterprise Resource Planning (“ERP”);

Business Intelligence (“BI”); and

Building integrated Knowledge Management Services (Enterprise Content Management).
www.bakertilly.co.uk
25
Enterprise Resource Planning
The ERP system is an electronic system which enables managers to visually see where staff are
allocated and also allows staff to see where they will be working, similar to an electronic
calendar. In addition to this the ERP is intended to be used as a financial planning tool. It is
believed that one system which can be used to undertake a number of functions will reduce
maintenance costs at the NAO and also lead to process efficiencies and thus a process light
organisation.
Business Intelligence and Enterprise Content Management (ECM)
The aim of the Business Intelligence system is to “enable the delivery of high quality management
information that presents a single version of the truth, that is delivered on time, is easily
accessible and is used by the right people”, this is being achieved through the use of the
Mkinsight system. However we also identified that Mkinsght has a module contained within it
called Template Designer which uses templated reports. This appears to be contradictory to the
January 2014 update which states that the NAO will place “Less reliance on templates” going
forward.
Reliance on Templates
The update of 2014 states that there will be “less reliance on templates” with regards to the
Transformation Programme which will contribute to the organisation being process light. This
aims to enable more initiative from staff to produce bespoke products which will be of benefit to
clients. We can confirm that pre-transformation templates were used in order to produce output
products. Examples of templates that have been used can be seen below;
Conclusion
The NAO, through the Transformation Programme, is in the process of becoming process light in
terms of the reduction of layers within the organisation and also a slight reduction in the use of
templates. Whilst we accept that reducing reliance of templates is a positive action, this needs to
be balanced with an overarching organisational methodology and this could be achieved through
development of a training programme which all staff could use to deliver work.
www.bakertilly.co.uk
26
3.8 Analysis (Outputs)
Background
The NAO had reported in January 2014 that outputs are evolving throughout the life of the
Transformation Programme and that the outputs deliver a wider range of advice to clients on
delivery mechanisms and interventions open to them. It should also be noted that at the time of
the review the NAO did not have a “Recommendation Tracking” electronic software system
which would increase the robustness of this process and allow for recommendation tracking to
be easily undertaken. A recommendation tracker could be used to assess the implementation of
recommendations from these output reports.
Developing Outputs
We selected a number of output reports over a period of time which fairly and accurately
reflects the life of the Transformation Programme in addition to a selection of outputs before its
commencement. During the update on the Transformation Programme in July 2013 and also
January 2014 there is reference to the “Fan of Assurance” (See Appendix N) which visually
details the assurance products which the NAO will attempt to roll out across the organisation.
This is due to be implemented in financial year 2014-15. The NAO will need to improve it’s
electronic filing arrangements in order that the range of suites can be properly and robustly
accommodated within Mkinsight. If this is not the case there is potentially an increased risk that
an inappropriate product could be used for an assignment. We reviewed the work programme
for 2014/15 in order to identify whether there will be a broader range of product types in line
with the Fan of Assurance. We can confirm that although there is still a focus on VfM reports,
there is a wider range of output products with nine product types from the fan of assurance
being planned. The chart below shows the breakdown.
www.bakertilly.co.uk
27
Fan of Assurance
The NAO has produced the “Fan of Assurance” (Appendix N) which details the potential
products which, post Transformation, will be available to the NAO. As the diagram details there
will be five Parliamentary reporting products, 12 MKI reporting products and 23 NAO reporting
products. In order to test the development of the types of outputs that the NAO has been
producing over the duration of the Transformation Programme we selected five reports which
were chosen at random and these are shown in Appendix L. The output products that are shown
were selected on a sample basis and this represents a cross section of reports from 2012 –
2014.
Example of developing output products
During the review we identified a report which was a good example of this development
towards integrated working. This report focused on the privatisation of the Royal Mail from 1st
April 2014. The Royal Mail report shows an example whereby the NAO bought in and made use
of a private sector expert in order to develop ideas and recommendations surrounding
privatisation. This example of increased use of sector wide expertise is the direction in which the
NAO are aiming to take for output development. However there is still a strong focus on value
for money which is shown in the conclusion (below right).
Conclusion
We identified that there have been improvements in the development of products during the
early implementation of the Transformation Programme. There is a move away from just
undertaking VfM reviews and assurance assignments and a move towards producing a wider
range of products which will also add value. We appreciate that this process of diversifying the
products is in its early state and this would be an area that would require monitoring by the NAO
in order that evidence can be easily obtained. Although our sample suggests only four product
types have been produced since the beginning of Transformation, the 2014/15 Work Programme
suggests there will be at least nine product types. This is a significant improvement. Three
recommendations have been made in Appendix D.
www.bakertilly.co.uk
28
Appendix A – Quotes from Directors and
Leadership
Team
“Clusters are collectively
starting to see synergies within their cluster”
“12 out of 50 is unprecedented by NAO standards, but they should have gone further”
“In terms of outcomes, we’ve got a lot further to go on that”
“We’ve got bright people here, why don’t we trust them to get on with it then”
“We are beginning to see changes in the staff culture”
“Transformation can only be good for the NAO going forward”
“If you don’t press the reset button, people don’t put away the accumulated baggage”
“In terms of accountability, it’s quite muddy and there needs to be someone in charge”
“We’re not good at accountability”
“The NAO is demonstrating greater influences”
“The NAO influences stakeholders and wider groups”
“Transformation – great idea, badly executed”
“In relation to comms, we made tactical errors, we were not clear on the external dynamics”
“We sold it as if we were in a position of strength, which implies it was optional, but we needed
change due to the external dynamics”
“We’re clear that we didn’t get it right to begin with”
“It’s an organisation of professional sceptics, therefore it makes it hard to promote change”
“We hear different messages from different members of the Leadership Team and it’s confusing”
“It’s difficult to carry messages if they’re not what you believe”
www.bakertilly.co.uk
29
Appendix B – Transformation Programme Timeline
Outline
benefits
realisation
targets
Identify
benefits
2012/13
Start of
program
me
2013/14
2014/15
Value
for
money
review
www.bakertilly.co.uk
2015/16
End of
program
me
Appendix C – Potential and Performance
www.bakertilly.co.uk
Appendix D – Recommendations
Ref.
Recommendation
Comment
Management Response
5.1.1
The NAO should ensure
that
the
Work
Programme document
assigns jobs to a
responsible officer in
order
to
increase
accountability and also
assist in the NAO’s goal
of increased delegation.
It is noted that this
accountability is already
identified through the
Mkinsight system.
We found through review of
the work programmes that
work programmes and jobs
do not state responsible
parties. This does not allow
for increased delegation,
increase accountability and
also enables less assessment
integration of the work
programmes. However, we
appreciate
that
this
accountability
can
be
discovered when accessing
the Mkinsight system.
Implemented. The work programme
and MKI routinely designates the
responsible cluster and lead director
for approved work.
5.3.2
Communications should
be
developed
in
standard format with a
1st section describing
organisational
wide
messaging and a second
section to cover cluster
specific issues
We identified that the NAO
has issues with regards to
communication to staff,
particularly
around
the
Transformation Programme.
This has led to a lack of
understanding from staff and
the feeling of not being
informed of changes.
Accepted. We communicate a wide
range of issues through many
different mediums and where
appropriate recognise the need to
communicate at both a corporate
and a cluster level. In addition, the
clusters have developed their own
ways to communicate cluster
specific issues and including
structured
face
to
face
communication and newsletters.
5.4.1
The NAO should set a
deadline for all staff
members to upload
their CV to the intranet
in order that the skills
mix is clearly visible to
all members of staff.
During the review we
identified that there are
some members of staff that
have not uploaded their CV
to the intranet. This means
that their skills cannot be
searched and potentially not
fully utilised.
Accepted. The LT has agreed that as
part of the culture change people
should be open about their
experience and the LT will consider
what impact that has on people’s
utilisation rather than make this a
compliance issue at this stage but
there would an expectation that all
staff members would have their CVs
on the intranet by 2015.
www.bakertilly.co.uk
Appendix D – Recommendations
Ref.
Recommendation
Comment
Management Response
5.5.2
The NAO should review and
update the Recruitment
Policy to include references,
examples and background as
to how the Transformation
will impact the recruitment
of staff going forward. This
should also include direct
reference to the targeted
acquisition of skills.
We identified that the
Recruitment Policy, last
updated June 2013, does not
reflect the progress made in
relation to the work
undertaken and the impact
that Transformation has had
within the organisation. As
such at the next available
juncture the progress and
achievements should be
reflected within the policy.
Implemented. HR have widened
the sources of new talent for the
Office,
including
inward
secondments, senior advisers and
temporary VFM staff and are
putting in place a range of
preferred supplier contracts to
ensure the Office can access new
skills more readily. Recruitment
campaigns are now being
conducted on a rolling basis
reflecting both the transformation
programme
and
wider
employment market conditions.
5.8.1
Ensure that there are
adequate and robust filing
systems on the Mkinsight
system in order that the type
of report can be found
appropriately and easily by
the product producer
Through review of the
Mkinsight
system
we
identified that there were
links and examples of output
products which should be
used, however it should be
noted that this was difficult
to use and find the
applicable report.
Accepted. We accept that MKi
does not give the best experience
for non financial audit users.
Improvements are being made to
MKi with further enhancements
planned to be completed by
January 2015.
5.8.2
With increased assurance
products being rolled out,
the NAO should consider
producing robust guidance as
there will be an increased
risk that an incorrect product
could be used at an incorrect
time.
This
could
be
implemented in line with the
recommended
training
programme.
With the range of products
set to increase to 23 from
the
current
amount
confusion could arise in light
of which products should be
used an any given juncture.
As such we believe that
guidance would be useful in
light of the range of
products.
Implemented. As new product
streams come on board we
provide detailed guidance and
protocols to teams on how to
consider whether the product is
applicable. We will also refresh
guidance for VFM work.
5.8.3
The NAO should evaluate (in
terms of costs and benefits)
whether
procuring
a
Recommendation Tracking
software would be suitable.
Such software will enable
more robust tracking to
occur in addition to improved
reporting arrangements..
It was ascertained during the
review that at present the
NAO does not have a
recommendation
tracking
piece of software. This
would be useful in order to
gauge the progress of
recommendations made as a
part of previous outputs.
Accepted. This functionality does
exist in Mki and implementation
will be included in the further
enhancements planned.
www.bakertilly.co.uk
Appendix E – Pulse Surveys Jan 2014
Local Effective
Mass
Influencing Managing
Corporate
service strategic market
&
major
services
delivery centre operations regulating programmes
Delivery
through
networks
Total
I am proud when I tell others I am part
of the NAO
75%
73%
72%
75%
68%
68%
55%
70%
I understand the NAO strategy
74%
74%
74%
60%
65%
68%
67%
69%
I understand the strategic issues which
are the focus of my cluster
38%
71%
79%
71%
60%
67%
47%
62%
I feel inspired to do the best in my job
71%
64%
56%
71%
50%
53%
45%
59%
I feel motivated to help the NAO
achieve its objectives
77%
60%
55%
53%
55%
49%
39%
56%
I am taking personal responsibility to
help drive the NAO strategy
60%
57%
53%
53%
47%
51%
37%
52%
I have a strong personal attachment to
the NAO
62%
46%
51%
53%
40%
53%
41%
50%
The Leadership Team has a clear
vision for the future of the NAO
60%
53%
52%
41%
44%
32%
39%
46%
The Leadership Team makes an effort
to listen to staff and to take account of
our views
51%
47%
45%
28%
34%
30%
33%
39%
I would recommend the NAO as a great
place to work
55%
33%
34%
35%
40%
32%
31%
38%
The Leadership Team is taking
appropriate action on the results of the
2013 people survey
46%
37%
28%
29%
23%
19%
31%
31%
Progress with the strategy is being
communicated clearly
28%
27%
26%
22%
23%
21%
24%
25%
Overall, I have confidence in the
decisions made by the Leadership
Team
32%
29%
33%
25%
21%
12%
20%
25%
Average per cluster
56%
52%
51%
47%
44%
43%
39%
48%
Appendix E – Pulse Surveys Apr 2014
Delivery
Managing
Effective
Local
Corporate through
major Influencing & Mass market
strategic service
services network
programme regulating
operations
centre
delivery
s
s
All
I understand the strategic issues which
are the focus of my cluster
49%
88%
86%
77%
72%
71%
73%
73%
I am proud when I tell others I am part
of the NAO
78%
68%
74%
81%
74%
64%
59%
71%
I understand the NAO strategy
84%
68%
72%
62%
70%
57%
59%
68%
I feel inspired to do the best in my job
56%
59%
46%
54%
64%
49%
59%
55%
I feel motivated to help the NAO
achieve its objectives
68%
51%
42%
61%
56%
44%
56%
54%
I am taking personal responsibility to
help drive the NAO strategy
66%
54%
56%
47%
54%
37%
61%
53%
The Leadership Team has a clear vision
for the future of the NAO
68%
39%
46%
36%
43%
44%
29%
45%
I have a strong personal attachment to
the NAO
54%
41%
49%
42%
43%
44%
41%
45%
The Leadership Team makes an effort
to listen to staff and to take account of
our views
59%
48%
44%
36%
41%
33%
29%
42%
I would recommend the NAO as a great
place to work
55%
43%
37%
52%
30%
37%
34%
42%
The Leadership Team is taking
appropriate action on the results of the
2013 people survey
47%
36%
38%
32%
28%
29%
22%
34%
Progress with the strategy is being
communicated clearly
46%
36%
30%
15%
25%
27%
27%
30%
Overall, I have confidence in the
decisions made by the Leadership
Team
51%
25%
30%
28%
25%
24%
15%
29%
Average per cluster
60%
50%
50%
48%
48%
43%
43%
49%
www.bakertilly.co.uk
Appendix F – Client Feedback
As previously stated there are 2 1 questions in total for the client feedback questionnaires, these
questions are as follows;
1. At the start of the audit, was the NAO team sufficiently knowledgeable?
2. Did the NAO team develop sufficiently its understanding of the organisation?
3. Keeping you informed about progress with the study;
4. Being clear about the purpose of information requests;
5. The amount of information requested during the study;
6. Comment on the scope of the study;
7. Comment on emerging findings during the course of the study;
8. Check and clear draft reports;
9. Comment on the NAO Press Notice;
10. The report had a clear purpose;
11. The report contained meaningful analysis;
12. The report was relevant in the context in which you operate;
13. How satisfied were you with the response to your comments;
14. In your view, was the VFM Conclusion well founded?
15. Draw on evidence in the report?
16. Identify clear actions for your department?
17. How do you consider the NAO Press Notice;
18. To what extent did the study provide new insights for your work?
19. Professionalism;
20. Their level of skills to carry out the audit;
21. We would value a discussion with you about any of the questions.
www.bakertilly.co.uk
Appendix G - Skills Strategy Action Plan
No.
Action\ Responsibilities
Timing
Lead
Design a richer composite measure to track strategic and operational
value and risks to report monthly to the Leadership Team and annually to
the Board through the yearly HR report.
Aug - Dec 2013 EL:D&I
Extend the scope and value of benchmarking we undertake across public
and private sector audit.
Aug - Dec 2013
Focused LT responsibility for delivering this skills strategy
Aug 2013 –
Mar 2015
4.
Introduce a new assurance training programme
2 Sep 2013
5.
Develop and maintain specialisms within core audit
Aug-Sep 2013
6.
Ensure our analyst capability can address our client’s future strategic
issues.
Sep 2013 –
Mar 2015
7.
Put greater emphasis on continuing to develop technical skills post
qualification
Sep 2013 –
Mar 2015
8.
Develop single architecture to integrate all professional, technical,
leadership, mandatory and induction training
Oct 2013
Continue to provide all mandatory training as a priority
Aug 2013 –
Mar 2015
Continue to develop our two leadership development programmes in line
with the needs of the transformed NAO
Aug 2013 –
Apr 2014
Design a new coaching programme for Directors.
Aug 2013 –
Sep 2013
Review performance management to ensure it rewards skills
development.
Manage deployment and development of skills effectively using an
integrated business planning platform.
Mar 2015
1.
2.
3.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
Develop clear roles and expectations of sector experts and specialists.
Aug – Dec
2013
Set up a new network of sector experts and specialists.
Sep – Dec
2013
Develop a structured and longer term specialist development programme Sep 2013 – Apr
for staff
2014
Demonstrate effective deployment and development of specialist skills
and community building across the NAO.
Nov 2013 +
Every member of staff will publish their CV on our corporate intranet.
Sep – Dec
2013
Aug 2013 +
Spend time talking to staff about skills, aspirations & ideas
20.
Help staff develop personal knowledge management skills and a greater
understanding of their clients.
Sep 2013 +
21.
Each cluster will design a structured and longer term sector development
programme
Aug 2013 –
Apr 2-14
Define skills needed for meeting immediate and longer term programme
needs.
Clusters have responsibility to match staff skills to work allocations.
Aug – Dec
2013
11 Nov 2013
Develop clear business requirements for secondments
Aug – Dec
2013
23.
24.
COO
www.bakertilly.co.uk
Who is an
active
collaborator
L&D Manager
& team
Cluster
Development
Directors
Director
General
Martin
Sinclair
Michael Kell
Cluster
Development
Directors
Manager L&D
EL: D&I
Cluster
Development
Directors
HR Director
COO
Cluster
Development
Directors
L&D Manager
& team
Leads of
specialist
communities
of practice
EL:D&I
Cluster
Development
Directors
Cluster
development
directors
EL:D&I
Directors &
Audit
Managers in
each cluster
Mar 2015
19.
22.
Challenge
role
Appendix H - Competency Framework
Previous Competency Framework
www.bakertilly.co.uk
Appendix H - Competency Framework
Current Competency
Framework
www.bakertilly.co.uk
Appendix I – CVs and Director Objectives
CV location
Director Objectives
To work as a leadership team to deliver the strategic objectives of the major projects cluster,
including delivering our work programme as set out in the business plan and the influence we
are targeting through our benefits realisation plan”
“To be able to demonstrate an improvement in our external engagement to secure influence,
based on better internal engagement; and in the growth and development of the capabilities
and behaviours that we need to succeed both as a cluster and in the wider NAO”
“To be a leader in delivering our vision for skills and development of our people in the long term,
as defined in our skills strategy and its action plan”
“To continue to enhance and develop wider cluster knowledge and expertise - particularly in
respect of supply chain understanding and accounting for complex long term schemes”
“ “To deepen and broaden my understanding of accounting for complex transactions, including
investment derivatives”
www.bakertilly.co.uk
Appendix I – CVs and Director Objectives
Lead behavioural and culture change across the MMO cluster that is consistent with the NAO’s
transformation strategy”
“Work with other cluster directors to lead staff through change, inspire and support them to
achieve their potential – measure through people survey and feedback mechanisms”
“As Cluster Director, collectively achieve at least the expected benefits as set out in the” Benefits
Realisation” (Appendix B for example timeline of benefits realisation)
“Build a confident, empowered and motivated DCMS team that is engaged in the area and
cluster”
“Deliver the DTN objectives and benefits, as outlined in our cluster objectives, benefits realisation
and strategies for skills, people, K&I and quality. This is a collective objective, to demonstrate
excellence and strong leadership for the DTN cluster and to engage collaboratively with our
teams”
“Act as a collegiate and inclusive leader of teams, embodying the values of a transformed NAO”
“Develop stronger and more collaborative working relationships and culture across the Home
Office area to ensure we develop effective insight and expertise and good quality networks
across our sectors”
“Build confident, empowered and motivated teams, so that they are engaged in their areas and
cluster and perform to their potential”
“To work with cross-Cluster colleagues to embed the transformation programme and to develop
a series of insights, increase our knowledge and skills base to build work plans that are relevant
and innovative across the Cluster”
www.bakertilly.co.uk
Appendix J – Pulse Survey Comparaison
www.bakertilly.co.uk
Appendix K - Testing sheet for output products
Date
Title
Type
18-Apr-12
Transparency
VFM: Standard
13-Jun-12
Central-local government
VFM: Standard
04-Jul-12
Affordable homes
VFM: Early stage
19-Sep-12
DfID multilateral aid review
VFM: Standard
21-Nov-12
Tax avoidance
VFM: Standard
18-Dec-12
HMRC customer service
VFM: Standard
31-Jan-13
MOD equipment plan
VFM: Standard
13-Mar-13
Integration of public services
VFM: Landscape review
24-Apr-13
Equitable Life
VFM: Standard
02-Jul-13
HMRC Standard Report 2011-12
Audit and assurance: long form report
04-Sep-13
Border Force
VFM: Standard
31-Oct-13
Emergency admissions
VFM: Standard
28-Nov-13
Student loans
VFM: Standard
13-Dec-13
Council Tax support
VFM: Standard
27-Feb-14
Personal Ind Payments
VFM: Early stage
www.bakertilly.co.uk
Appendix L – Output products
February
2014
October
2013
www.bakertilly.co.uk
Appendix L – Output products
April 2013
December
2012
July 2012
www.bakertilly.co.uk
Appendix M – Expectations
AQP what is expected of the NAO
The Transformation programme skills development per cluster
www.bakertilly.co.uk
Appendix N – Fan of Assurance
Appendix O – Cluster Diagram
www.bakertilly.co.uk
Appendix P - Supporting statement for
development application
“The NAO already recognises the need to do more to grow
its commercial and business skills base, in order to offer
the best possible service to Parliament and our clients in
government.”
“Increased technical knowledge – an expanded
commercial and financial toolkit to use in studies and pass
on to my teams and colleagues around the office.” and
“Increased professional credibility and fluency in business
language, building on my current understanding, to help
up the NAO’s game when engaging with private sector
experts.”
www.bakertilly.co.uk
Appendix Q – The team and scope
Purpose of the report
Baker Tilly were appointed to critically
review the Transformation Programme in
terms of the value for money of its key
objectives surrounding the programme at
the NAO.
This review consists of a critical appraisal of
stated progress with regards to the
Transformation Programme with evidence
being requested to support the progress
statements and Transformation updates.
The Value For Money assignment therefore
focuses on the following eight key areas:
• Work Programme;
• Financial Savings;
• Staff Engagement;
• Quality;
• Skills;
• Structure;
• Processes; and
• Outputs
Our Team
Our team principally comprised:
Paul Brown
Donal Laverty
Stephanie Harvie
Ken De Negri
George Powell
Partner and Engagement
Lead
Partner
Senior Consultant
Consultant
Consultant
Accordingly, for the avoidance of doubt, we
emphasise that the work we have performed
does not constitute an audit and that this
report does not imply the expression of an
audit opinion, or the satisfaction of an audit
opinion, statutory or otherwise. The sole
purpose of this report is to assist the NAO in
the appraisal of Value for Money with regards
to the Transformation Programme.
Use of the report
Approach
We undertook a review of the aspects
surrounding
the
Transformation
Programme. This was largely conducted
remotely but we also spent time on site
with key members of staff in order to
interview and test a number of key
statements made in the various published
Transformation updates.
We also set up a Workshop in order to
ascertain the qualities of the Leadership
Team and judge their ability to embrace the
Transformation culture.
This report is intended for the sole use of the
NAO and the Public Accounts Committee
(TPAC) of the House of Commons. We
understand that reports to TPAC are in the
public domain.
We cannot accept responsibility to any third
party in relation to our report and disclaim all
liability to any third party in connection
therewith.
This report should not be
disclosed to any third party without the prior
written consent of Baker Tilly.
www.bakertilly.co.uk