The Swedish Agency for Public Management

Evaluation of different areas. An analysis of sector-specific evaluation agencies (2014:7)

On behalf of the Government, Statskontoret has analysed autonomous sector-specific evaluation agencies. A central part of the assignment is that Statskontoret shall analyse and assess the conditions under which autonomous sector-specific evaluation agencies constitute an effective organisation of the state evaluation resources. In light of this analysis, Statskontoret shall assess the general strengths and weaknesses associated with autonomous sector-specific evaluation agencies in relation to alternative ways of organising state evaluation.

Sector-specific evaluation agencies - a heterogeneous group

Statskontoret's survey includes the Swedish National Council for Crime Prevention, the Swedish Fiscal Policy Council, Institute for Evaluation of Labour Market and Education Policy, the Swedish Social Insurance Inspectorate, the Swedish Agency for Cultural Policy Analysis, the Swedish Agency for Growth Policy Analysis, the Swedish Agency for Health and Care Services Analysis, the Swedish Council on Health Technology Assessment and Transport Analysis.

The survey shows that there are several similarities but also significant differences between evaluation agencies. Prominent similarities are primarily that they are permanently organised operations for evaluation within their respective areas. They also conduct personnel-intensive operations and rely on personnel with a high level of expertise. Most of them have the government and the Government Offices as their primary target group, and the agencies' activities usually consist of both self-initiated work and work with government commissions.

The differences are mainly that they operate within different policy areas with area-specific evaluation questions and objects of scrutiny. The agencies therefore require expert knowledge of an entirely different nature.

Several factors influence the evaluation agencies' effectiveness

The sector-specific evaluation agencies have been established to conduct analyses and evaluations within different areas. The activities and performances need to satisfy several requirements of effectiveness:

  • The reports that are produced need to be relevant and useful for the evaluation agencies' target groups.
  • The agencies need to be credible producers of reliable analyses and evaluations.
  • In their capacity as government agencies, the evaluation agencies need to conduct cost-effective operations.
  • Several factors are significant with regard to the evaluation agencies' conditions for working in an efficient manner. The evaluation agencies' relevance and usefulness for the Government and Government Offices depend largely on the clarity of the analyses and evaluations. The relationship between the commission-driven and self-initiated activities also impacts effectiveness.

The agencies must be perceived as credible and independent producers of knowledge and thus it is important that they have an autonomous position. Furthermore, they need to have, for example, good quality assurance processes and personnel with a high level of expertise within the policy area.

Effective organisation of governmental evaluation resources

The evaluation agencies are a heterogeneous group and therefore the analysis and evaluation focus on their commonalities.

When should a sector-specific evaluation agency be established?

Sector-specific evaluation agencies work in practice as one of several evaluation options in the relevant specialist areas. The question of when it is justified to set up a sector-specific evaluation agency therefore largely concerns the added value that the agency is expected to provide in relation to other state evaluation options.

A sector-specific evaluation agency adds value by providing the Government with access to in-depth methodological and specialised expertise whereby analyses and evaluations are continuously produced in the specific area. This in turn enables a long-term development of knowledge on the outcome and effects of policies, and on conditions of importance for policy formulation.

In order to assess whether a sector-specific evaluation agency should be established, the Government and the Government Offices should pose the following questions:

  • What need is there for continuous analyses and evaluations in order to strengthen the development of knowledge within an area?
  • Who will use the analyses and evaluations, and to what end?
  • How important is access to a permanent analytical capacity within the area for specific questions?
  • To what extent is it possible to attain expertise and long-term knowledge development within the area using other evaluation options?

The more immediate added value of setting up sector-specific evaluation agencies may differ and vary in size depending on which specialist area is involved. The establishment of sector-specific evaluation agencies therefore requires taking into account several different area-specific circumstances.

To justify the establishment of a sector-specific evaluation agency, it is also important that the agency's target groups can make use of the analyses and evaluations. The Government therefore needs to clearly state its purpose with the evaluations as well as who will utilise the resulting data.

What are the general advantages and disadvantages of using different state evaluation options?

In addition to sector-specific evaluation agencies, the Government has access to several other options, such as sector agencies, committees of inquiry, Statskontoret, universities and private consultants. The advantage of engaging a sector agency, for example, is that it has extensive knowledge of policy implementation and results. Sector agencies are also well placed to analyse complex issues that form the basis for the development of their operations. The disadvantage is that sector agencies are responsible for operations and are thus perceived to be less credible than the sector-specific evaluation agencies when it comes to complete objectivity when describing and analysing shortcomings and problems in their own operations.

Statskontoret is an independent evaluation agency with a general, sector-wide evaluation mission, and has the advantage of being able to accumulate knowledge within different areas in its evaluation activities. Statskontoret is also able to develop specialised expertise in cross-sectoral issues such as the work with steering, organising and streamlining government operations. However, Statskontoret does not have the same depth of knowledge as the sector-specific evaluation agencies and Statskontoret's analyses therefore risk becoming more general in nature. Correspondingly, there are also general pros and cons with committees of inquiry including expert groups, universities and higher education institutions, and private consultants.

What contributes to sector-specific evaluation agencies being an effective organisation?

In its steering and monitoring of evaluation agencies, the Government should take into account several factors that are important for the effectiveness of government operations. There is no evident control model that suits all evaluation agencies and the steering and monitoring should conform to what the Government wants to achieve. It is particularly important that the Government clearly defines the evaluation agency's tasks and target groups. Government commissions play an important role in the direction of the agency's activities, and with a structured dialogue on planned government assignments and self-initiated studies, the agency's operations can more easily gain relevance. The size of the operations is also very important for effectiveness. Larger operations are more often given time for external analysis and skills development training. They are also less vulnerable to, for example, unplanned personnel absences. To reduce vulnerability and satisfy the various needs for expertise, the evaluation agencies engage external expertise to varying degrees. An agency that frequently engages consultants may however find it more difficult to build up its own long-term competency.

The evaluation agencies' impact on sector agencies

There are different experiences of how much and in which way evaluation agencies affect the sector agencies that have comprehensive implementation responsibilities. In certain areas, the evaluation agencies contribute with a knowledge base that indirectly benefits the sector agencies' operational development. In other areas, the sector agencies perceive analyses and evaluations produced by evaluation agencies as not adding any new or useful knowledge of benefit to the operational development.