Blue Paper

Operational definitions of performance indicators Q6 and Q7

First Draft BP 14 July 2006

Synthesis Forschung Gonzagagasse 15/3 1010 Vienna Telefon +43 1 310 63 25 Fax +43 1 310 63 32 E-Mail office@synthesis.co.at http://www.synthesis.co.at SB Consulting GmbH Meldemannstrae 12-14 1200 Vienna Telephone +43-1-33 168-0 Fax +43-1-33 168-101 E-Mail info@oesb.at http://www.oesb.at

Preface

The operational definitions of the PES performance indicators Q6 and Q7 have been agreed upon by the Participating Partners at their second meeting in Vienna in March 2006. The paper presents the conclusions the team of consultants has drawn from the discussion during the meeting. This draft paper is subject to revisions. These revisions might turn out to be substantial.

For the team of consultants: Prof. Michael WagnerPinter Vienna, 27 July 2006

Content

1 2 indicators Q6 and Q7 3 Satisfaction 4 Are the data indicators available? already How part are of they the defined? PES information system? 7 6

Annex List of papers of the Working Group Impressum 8 11

Operational definitions of performance indicators Q6Q7

1 Satisfaction Survey Variations Standard Operational is in survey of Satisfaction indicators Q6 and Q7 definition job seekers indispensable survey scenario design ofand
employers indicator

The two performance indicators Q6 and Q7 are from an operational point of view quite the same. Both indicators aim at capturing the satisfaction with the overall service the PES provides to customers. Q6 is directed towards people (as potential job seekers), whereas Q7 is directed towards employers. The empirical information for Q6 and Q7 has to be survey based. Thus, the PES either has already established such a survey (and is therefore in a position to retrieve the information); or it has not (yet) established the survey and can therefore not retrieve the information. Of course, there might be a substantial variation in the way the survey is carried out (methods of sampling, frequency of the survey, type of questions asked), but this will not be dealt with at this stage of the project. There are, however, two issues that have to be addressed in the context of operational definitions. Let us start with the following standard scenario for a satisfaction survey. This standard survey includes the question How are you satisfied with the overall services the Public Employment Service has delivered? structures the answers to the questions into the items very satisfied satisfied unsatisfied very unsatisfied Within the scenario of the standard survey the operational definitions of indicators Q6 and Q7 are straightforward: (1) Count the number of all answers

Operational definitions of performance indicators Q6Q7

Best case, Deviations Overall School grading worst are case likely satisfaction system

(2) choosing the items very satisfied and satisfied. (3) Divide this number (4) by the number of all answers returned to this question. The indicators will assume the value 1 if all customers are satisfied; it will assume the value 0 if none is satisfied. It is likely that the practice of several PES among the Participating Partners will deviate from the standard survey scenario. In particular, two deviations might occur: Firstly, all questions asked refer to specific services delivered. Secondly, the answers are structured along a grading which reflects the grading used in the respective national school system. Regarding the first deviation, the overall satisfaction can be derived as a weighted average of the answers to the specific service questions. The weights used in this procedure should be documented when the information is forwarded. Regarding the second deviation, there is a simple solution. Just partition the set of grades in a top half and a bottom half if the number of grades is even; if the number of grades is odd, then delete the neutral grade between the top and the bottom set. Once this partition has been carried out, it is possible to use the operational definition given; it is only necessary to substitute the line (2) by choosing grades from the top half.

Operational definitions of performance indicators Q6Q7

2 3 Customer satisfaction (employers) (job seekers) Are the data indicators available? already How part are of they the defined? PES information system?

Q6: Achieving customer satisfaction (job seekers) Is this indicator already part of your management information system? Is there an indicator already part of your management information system that comes close to Q6 with regard to data provision

yes

no

Q7: Achieving customer satisfaction (employers) Is this indicator already part of your management information system? Is there an indicator already part of your management information system that comes close to Q7 with regard to data provision

yes

no

Please answer the following questions and tick the appropriate:

Do you have full access to data to calculate indicator? yes no

Please describe the measurement process:

Do you have full access to data to calculate indicator? yes no

Please describe the measurement process:

Operational definitions of performance indicators Q6Q7

List of Papers (latest draft)

ListPapers White Blue Red Yellow Green Papers of Papers Papers papers of the Working Group

All papers of the Working Group will be distributed by the group of consultants to each of the Participating Partner: There are several series of papers. Some of them can be used for general circulation, some of them are for internal use only; none of the papers are strictly confidential. White Papers deal with general topics and cover the topics of the final report of the Working Group (unrestricted circulation) Blue Papers list performance indicators and discuss their strong and weak points (unrestricted circulation) Red Papers report on harmonized data forwarded by the Members of the Working Group (circulation only with permission of the Members who provided data) Yellow Papers contain only information which serves to organize the work in progress (of no use to outsiders of the Working Group) Green Papers cover issues related to the choice of good practices and their presentations (unrestricted circulation)

Operational definitions of performance indicators Q6Q7

Type of paper White Papers Blue Papers

Reference WP 01 BP 01 BP 04 BP 06 BP 10

Title PES-Performance, Indicators, Good Practice Supporting Flowcharts Measuring effectiveness and operational efficiency of the Finnish PES A list of eight performance indicators (Second draft) Measurement Procedures: The specific approaches taken by the Participating Partners in calculating the performance indicators and the underlying populations (First Draft) Capturing Context: A radar chart approach (First draft) Operational definitions of performance indicators Q1Q3 (Fourth draft) Operational definitions of performance indicators Q4 and Q5 (Second draft) Operational definitions of performance indicators Q6 and Q7 (First Draft) Operational definition of performance indicator Q8 (First Draft) Public Employment Services in Europe: a large spectrum of missions Performance Indicators: A first survey on data availability and ranking PES Background and Content Information and Indicators in the year 2005 (Second draft) A tight schedule (Third draft) Transition from unemployment to employment: Indicator 1 Transition from unemployment to employment before unemployment lasts long: Indicator 2 Transition from measures to employment: Indicator 3 Looking for Good Practices (First Draft)
8

Date 19/01/06 23/01/06 03/03/06 03/04/06 29/05/06

BP 11 BP 12 BP 13

29/05/06 27/07/06 27/07/06

BP 14

27/07/06

BP 15

27/07/06

Yellow Papers

YP 01 YP 02

11/01/06 12/01/06

YP 06

12/05/06

YP 07 Red Papers RP 01 RP 02

06/06/06 30/05/06 30/05/06

RP 03 Green Papers GP 01

30/05/06 01/06/06

Operational definitions of performance indicators Q6Q7

List of Papers (previous drafts) Type of paper Blue Papers Reference BP 02 Title Transition from Unemployment to Employment (First draft) Transition from Unemployment to Employment (Second draft) A preliminary list of eight performance indicators (First draft) Operational definitions of performance indicators Q1Q3 (Third draft) Operational definitions of performance indicators Q4 and Q5 Operational definitions of performance indicators Q6Q8 A tight schedule (First draft) PES Background and Content Information and Indicators (First draft) A tight schedule (Second draft)

Impressum Date 10/02/06

BP 03

28/02/06

BP 05

06/03/06

BP 07 BP 08

03/04/06 03/04/06

BP 09

03/04/06

Yellow Papers

YP 03 YP 04 YP 05

20/01/06 06/03/06 06/03/06

Eigentmer und Verleger: Synthesis Forschung Gesellschaft m.b.H. Gonzagagasse 15/3 1010 Wien Fr den Inhalt verantwortlich: a.o. Univ.-Prof. Dr. Michael Wagner-Pinter Synthesis Forschung Wien 2006