Activity Title
2012
Employers perception survey about quality of graduates Duty Station Islamabad Reported to TESP Secretariat, Higher Education Commission, Islamabad Responsible / Monitored Human Resource Management Division, Higher by Education Commission, Pakistan Duration of activity 6 months from the date of sign of contract Expected Starting Date January 1, 2013 i.e. date of contract effectiveness Selection Method Selection Based on Consultants Qualification (CQS)
1.
BACKGROUND
In response to the efforts regarding implementation of Government Higher Education Development program, as a part of the second Medium term Development Framework for Higher education (2011-2015), Higher Education Commission and the World Bank jointly initiated Tertiary Education Support Program (TESP). Project Development Objective is to improve the conditions of teaching learning and research for enhanced access, quality and relevance of tertiary education. The Project will support Government reform in tertiary education under two components: Component 1 -Program Financing, and Component 2 - Capacity Building, Policy Design and Monitoring and Evaluation(M&E).Component 1 will consist of four subcomponents, aligned with the Governments overall program: (i)improved fiscal sustainability and expenditure effectiveness; (ii) enhanced quality and relevance ofteaching and research; (iii) improved equitable access; and (iv) strengthened governance and management. Component 2 will strengthen capacities for program implementation and M&E systems. It will finance essential technical assistance (TA) and capacity building activities, carefully selected and sequenced with the implementation of Component 1. 2. An important activity under Component 2 of the Banks support is
Employers perception survey on graduates employability to find out their perceptions on the competencies required for a good employment and the extent to which higher education institutions are developing those competencies among their students. This survey will help to improve responsiveness of the higher education system to the labor
Human Resource Management Division, Higher Education Commission, Pakistan Page 1
Tertiary Education Support Program (TESP)- Terms of Reference on Employers perception survey about quality of graduates
2012
market needs by identifying and instilling traits and skills essential for promoting graduates employability; in accordance with the aspirations of the employers.
2.
GOAL:
To determine the extent up to which the competencies produced by tertiary education offered in Pakistani institutions is in accordance with the aspirations of the employers so that a perfect match between the knowledge, skills and attitudes fostered by the educational institutions among the students and those that are desired and demanded by the local and global markets may be created.
3.
Specific objectives of the survey would be as under To unveil employer opinion on those graduate qualities/ competencies that they consider important for employment To attain employer feedback on the extent to which Pakistani graduates demonstrate those qualities/competencies To point out the areas where the development of graduate qualities could be refined To build a closer relationship and goodwill between key employers, HEIs and graduates of Higher education Institutions of Pakistan To obtain baseline data to tailor educational activities at higher level to make it more employment oriented To provide useful information to the students, parents and other stakeholders for better decision making regarding higher education To review and determine the preferred model in terms of the time that HE institutions may allocate between teaching/research and coaching of skills and traits essential for greater employability; vis--vis use of tutorials, practicing skill under regular curriculum teaching, or else. The survey will also attempt: To explore best practices for integration and development of high quality employability skills through consultation with key stakeholders including universities, business and industry To identify the best possible procedures for assessment and reporting of the quality of graduates in general and level of employability skills in particular
Page 2
Tertiary Education Support Program (TESP)- Terms of Reference on Employers perception survey about quality of graduates
To pinpoint practical and cost-effective methods that enable quality of the graduates and employability skills to be explicitly identified as part of the higher education assessment and reporting process To recommend assessment and reporting approaches for the recognition of graduate employability To forecast the impact of recommended approaches on the higher education sector
2012
4.
IMPORTANT CONSIDERATIONS
In order to achieve the objectives stated above a well planned, carefully designed and properly implemented survey is required. The important considerations in this context include but are not limited to the followings: i. ii. iii. iv. v. vi. vii. viii. ix. Who will conduct the survey? What will be the basic criteria for selection of the firm/organization to conduct the survey What will be the time frame for the survey? How the perceptions of the employers would be measured? Who will check the tool, constructed for the survey to ensure its validity and quality? How many and from which sectors employers would be included in this survey, whether overseas employees would also be involved if so how? As this may involve some legal issues. What will be the policy regarding the conduct of survey (what procedure will be adopted to get into real opinions of the employers)? Which measures would be taken to ensure quality and accountability of the survey? How formative and summative evaluation of the project will be done and by whom? How the information will be presented and what will be the specifications of the report?
5.
Methodolgy
5.1.
In order to get the complete picture of the Employer perception Survey, it is important to have adequate representation from the employers of all sectors (Public & Private) and all types of organizations., The managers of the following organizations / firm will constitute the population:-
Page 3
Tertiary Education Support Program (TESP)- Terms of Reference on Employers perception survey about quality of graduates
i. ii. iii. iv. v. vi. Government / Semi Government / Autonomous Organizations of Pakistan Public & Private sector higher education institutions of Pakistan Companies Incorporated Financial Services Companies Insurance Companies Banking Companies
2012
The stratified random sampling will be best technique for this survey. 25 % from each stratum will be taken as representative sample. If the survey firm is aware of any other suitable, recent, list of firms (e.g. business registries, censuses) as well as best fitted sample design for the survey, it will inform the HEC for decision. The classification of firms into small, medium and large will be decided based on the number of employees. Firms will be classified as small if they have between 5-19 employees, as medium 20-99 employees, and as large 100 or more employees. The survey will target the HR and Production Managers for larger firms/enterprises, the Owner/General Manager for the medium and small firms/enterprises. population. The firm is responsible to get the list containing the information on the name, address or location, phone number, size, and activity of the selected firms from respective Ministries / Departments or any other reliable resource. It will be the responsibility of the survey firm to contact the employers on the list and persuade them to provide information for the project. Sometimes this persuasion can be difficult to obtain, but this project relies on persistent interviewers who can artfully persuade the managers of the firms on the list to share some of their time. The HEC welcome other suggestions about facilitating firm contacts, which should also be detailed in the proposal. The following points should be observed for the sampling: Preventive measures should be established. The firms proposal should specify the measures it will implement in order to reduce non-response to a minimum. These may refer to the selection and training of interviewers; the definition of a realistic fieldwork plan, the motivation of respondents; and the monitoring of non-response rates of individual interviewers. Minimum 300 firms will be taken as sample from overall
Page 4
Tertiary Education Support Program (TESP)- Terms of Reference on Employers perception survey about quality of graduates
2012
Substitution should not be entrusted to interviewers. Only the survey core managers may manage the list of reserve firms to interview, and authorize substitution only as a last resort. Substitution should be properly documented. The survey datasets should contain information from the questionnaire cover page of all target firms/enterprises and all mobilized reserve firms. For substituted firms/enterprises, at least the reason of substitution, the number of the substitute firm/enterprise, and the codes of all responsible fieldworkers and survey managers should be reported. The rate of substitution for the survey as a whole should not exceed 50 percent of the firms in the target sample.
5.2 Instruments
The employer survey will be carried out using a package of inputs (core questionnaires, operations and interviewer manuals, forms for fieldwork) that will be developed by the survey firm in close collaboration with the HEC team. The questionnaire will be constructed in the light of study objectives, e.g. first part of the questionnaire will be based on describing the qualities/competencies that are essential for the employees of the certain organization. Second part will uncover the perceptions of the employees about the extent up to which employability skills are possessed by the Pakistani HEIs graduates of today. information on firms background characteristics. It is important to note that this survey is far from the typical public opinion or marketing survey. The questionnaire requests objective information on numbers of employees in different categories, for example, and on managerial evaluations of the skills of these employees. This difference has important implications for a number of aspects of the survey, including the need for active involvement and supervision by the HEC at all stages. In another part collects
Page 5
Tertiary Education Support Program (TESP)- Terms of Reference on Employers perception survey about quality of graduates
2012
(d) the interaction between the interviewer and the respondent, indications as to the latter being puzzled or reluctant, extent of cooperation and response, evidence of embarrassment or uncertainty on the part of the respondent in answering certain items; and (e) the most appropriate type of employee who would respond to various kinds of information; and After the pilot stage, the HEC team will evaluate the quality and the information in the pilot questionnaire, and will make changes to the questionnaire. The firm will also participate in this evaluation and make suggestions, and will submit a report detailing the results of the pilot test, no later than 2 working days after the completion of the tests. If necessary, a second pilot will be conducted to further improve the questionnaire. The firm will work in close collaboration with the HEC team in identifying areas that may require adjustments (wording, fluidity, translation, etc.) Associated adjustments to the implementation manual and training program for interviewers and field operators will be made upon completion to the adjustments to the instruments. The firm will be expected to update training accordingly using the adjusted questionnaire and implementation materials.
Page 6
Tertiary Education Support Program (TESP)- Terms of Reference on Employers perception survey about quality of graduates
2012
All interviewers will be paid for participating in the training, but those not passing the quiz will not be employed in the rest of the project. The WB team may also choose to exclude any interviewers from further participation at any time without specifying the reasons. Basic components of training include but not limited to the following: (a) A basic instruction guide to provide operational definitions for the data to be gathered (b) Emphasis on special concepts and definitions attached to the survey explained during an intensive training session (c) In order to ensure that interviewers are fully competent to assist respondents in recalling events and reporting accurate information, techniques, including those to be used if respondents have difficulties with questions or are reluctant to supply information, are demonstrated. (d) In cases of non-responses due to refusals, either partial or total, the interviewers should be trained to make all efforts to fully explain the purposes and importance of the survey in simple and persuading terms and to reassure the respondent about the confidentiality of information. Courtesy and patience is the best and last resource. After the training, the firm will allocate each interviewer a set of companies to be surveyed in the form of lists containing the names, addresses and phone numbers of the companies as well as their codes. For each interviewer, the enterprises to be contacted should be grouped based mainly on their geographical location, with the purpose of minimizing travel-related costs. The supervisors first approach of the companies is usually made as a phone call, in which they briefly introduce the survey and ask for a precise appointment with a key person, able to provide the survey information. Data on the first page of the questionnaire (containing general information about the company) should usually be pre-entered by the interviewer, based on these phone contacts made prior to the direct contact. The firm will be responsible for making copies of the questionnaires, and all related materials: interviewers letters, identification badges, etc. Data entry training: Data entry persons should participate in interviewer training because a thorough knowledge of the questionnaire is a valuable asset. There will be a 2 day training session on the data entry program at the end of the interviewer training.
Page 7
Tertiary Education Support Program (TESP)- Terms of Reference on Employers perception survey about quality of graduates
2012
Tertiary Education Support Program (TESP)- Terms of Reference on Employers perception survey about quality of graduates
2012
motivation for this complexity is that past experience shows that interviewers learn on the job and provide better quality if they are fully invested in a project, so the results are better if there are fewer interviewers carrying out more interviews each. At the other extreme, it is possible for interviewers to over-work, so the HEC task team should be asked for permission in case of any interviewer carrying out more than 20 interviews. In keeping with proper survey protocol, the HEC will be involved at all stages of field preparation and implementation, and data management.
5.7
Implementation Documentation
The survey firm will be expected to keep detailed information of the implementation of the activities in all the areas covered in the survey to keep track of the sampled firms, including rates and reasons of non-response. The survey firm will provide weekly written status reports to inform the HEC of implementation progress, milestones and other relevant information.
5.8
Coding
The firm will carry out a rigorous coding of open questions included in the questionnaire. Particular emphasis should be given to questions related to occupation and economic activity, both of which should be coded at the three digit level. In order to assure quality in the coding process, firms would be expected to propose suitable strategies. Issues and activities related to coding should be carried out in close coordination with the HEC.
5.9
Data Entry
The survey firm will develop a suitable data entry program. The structure and the list of checks incorporated will be discussed with the HEC.. The data entry will be done concurrently with field work, so that information from the data entry can inform the process of the survey. This data entry program will exploit the fact that the questionnaire is pre-coded and can provide immediate error messages in response to invalid entries. It will be tested during the pilot test. The survey firm is expected to ensure that all the data is entered (from both non-response and filled questionnaires), and to transmit the collected data weekly to the HEC. In addition to this,
Page 9
Tertiary Education Support Program (TESP)- Terms of Reference on Employers perception survey about quality of graduates
2012
the firm should develop a coding strategy in order to ensure each firm/enterprise has a unique code for data entry. Each interviewer, supervisor and data entry person should also have a unique code which should be recorded for all questionnaires. The start and end time of the whole survey and module by module should be recorded. The phone number of the firms respondent(s) should be recorded for possible follow up.
6.2 Deliverables
The following deliverables are expected: (i) Lists of names and contact information for all survey firm personnel involved in the project. While partial lists should be provided as necessary and as described above, a complete list should be provided at the end of the project. (ii) Electronic version in English of the questionnaires used for the survey (iii) Electronic version in English of the final questionnaire with adjustments based on the pilot phase Human Resource Management Division, Higher Education Commission, Pakistan Page 10
Tertiary Education Support Program (TESP)- Terms of Reference on Employers perception survey about quality of graduates
2012
(iv) Intermediate field reports and databases, complete codebook; final database. A database with refusals/non-contacts (including basic information on the firm, reasons for replacement, sample identification, interviewer/supervisor identification, date of visits). (v) Filled-in questionnaires. (vi) Field work report in English of field work protocols adhered and experiences, including a detailed account of compliance with the sample design and indicators of data quality (sample size, sampling, timing, geographical coverage, issues of concern, full disclosure of any problems encountered). (vii) SPSS/STATA/N-vivo data files containing the raw data from the Employer Survey, and the tracking forms (with substitution forms) (viii) Final report with a detailed description of the methodology used, description of the implementation strategy, and descriptive statistics of the data. (ix) Final technical report, analyzing survey data to examine skills demand by employers and their perception of the quality and relevance of higher education programs
7. Required Qualifications
The selected firm shall possess the following qualifications: (a) A legal status recognized by Pakistan, enabling the organization to perform the above-mentioned tasks. (b) Experience in applied qualitative / quantitate research techniques, particularly indepth interviews; (c) Demonstrated experience of organizing enterprise surveys in Pakistan region preferably or the South Asia region in general, within the past ten years. (d) Demonstrated strong capacity and experience in planning and organizing survey logistics, and proven record of delivering timely results. (e) Proven excellence in production of analytical work, both written reports and brief presentations of research; (f) Good network of experienced enumerators, supervisors and data entry persons. (g) Demonstrated strong capacity in data management and statistics. (h) Capacity to quickly begin field work directly after the signing of the contract (the technical proposal should include a work plan, with a timetable for finalizing survey instruments, recruitment and training, data collection and data processing). (i) Capacity for concurrent data entry in the field, as well as ability to provide a cleaned data set within 2 weeks of the end of field work. (j) Knowledge of the following software: SPSS or Stata, nvivo preferable . Human Resource Management Division, Higher Education Commission, Pakistan Page 11
Tertiary Education Support Program (TESP)- Terms of Reference on Employers perception survey about quality of graduates
(k) Strong interpersonal skills and a team oriented spirit.
2012
To obtain first-hand information on the assignment consultants are welcomed to visit the TESP Secretariat, HEC before preparing a proposal. 8.1 TECHNICAL PROPOSAL:
Page 12
Tertiary Education Support Program (TESP)- Terms of Reference on Employers perception survey about quality of graduates
2012
The Technical Proposal would include the followings but not limited to:i. ii. iii. iv. A brief description/ biography of the firm/ consultants/organization with an outline of recent experiences on assignments of a similar nature. A description of the methodology and work plan for performing each activity related to survey. The list of the proposed staff team by specialty, the tasks that would be assigned to each staff team member and their deployment schedule. Brief but comprehensive CVs; signed by the proposed professional staff and countersigning by the head of the firms / team employed are mandatory to be attached with the proposal. CVs should clearly indicate whether a professional is presently employed or is proposed. Key information should include number of years working for the firm/entity, educational qualification, experience, age and degree of responsibility held in various assignments etc. Estimates of the total staff input ( both professional and support staff; estimated staff time) needed to carry out the assignment, supported by bar chart diagrams showing the time proposed for each professional/ team member Any additional technical information as may be deemed necessary. The Technical Proposal shall not include any financial information. Financial Proposal
v.
It lists all the costs associated with the assignment, including but not necessarily limited to:i. ii. Remuneration for staff (local and foreign, in the field) Reimbursable expenditures such as subsistence (housing etc.), transportation (international and local, for mobilization and demobilization), services and equipment (vehicles, office equipment, furniture, and supplies), office rent, printing of documents, expenditures on information and communication technologies and training, if it is a major component of the assignment. All the reimbursement made for non-consumable items will be the property of the client. The Technical/Procurement Committee of TESP, HEC will evaluate Technical and Financial Proposals respectively in accordance with QCBS method in general and on the basis of following criteria:
Human Resource Management Division, Higher Education Commission, Pakistan Page 13
Tertiary Education Support Program (TESP)- Terms of Reference on Employers perception survey about quality of graduates
2012
a) Expertise and Capability of the firm/consultants/ organization including the Five Years Similar Experience Overall experience with government departments at the federal, provincial and local levels Independent experiences related to the conduct of surveys etc. Experiences of conducting similar exercises at international level Experience of working with the donor-assisted projects in Pakistan especially in Higher Education b) Proposed Personnel Experience in undertaking similar evaluation exercises Other relevant experiences with the government departments (federal, provincial and local) Experience of consulting with donors and donor-assisted projects International Experience of conducting similar exercises c) Management Plan Suitability and usefulness of the proposed plan of action Measures proposed by the bidder for assuring the quality of the activity d) Coverage of maximum employers from variety of industries and sectors. Maximum coverage of geographical area Note: (i) The Committee may decide any other criteria at the time of selection of firm / consultant. The weightage of each criteria will also be set by the Committee to be constituted for this purpose (ii) Technical and Financial proposal will be sought on standardized forms designed in line with QCBS method.
In the event that the financial offer surpasses the budget stipulated for the study, or if an agreement cannot be reached regarding the implementation details for the study, the negotiation process will be declared closed. In that case, the World Bank will open the financial proposal of the second ranked firm and a contract negotiation will follow. This process will continue to be used until an agreement is reached.
The contract will be a lump sum contract structured into three payments, as follows: (a) First payment will be made upon signing the contract (20% of total);
Page 14
Tertiary Education Support Program (TESP)- Terms of Reference on Employers perception survey about quality of graduates
2012
(b) Second payment will be made upon carrying out the pilots and submission, and Bank approval, of the initial report, final questionnaire and organization of the field work (50% of total); (c) The final payment will be made upon submission of the final clean dataset and the technical report, acceptable to the HEC (30% of total). The proportions of the total may be adjusted during the contract negotiations, based on an analysis of the costs in each of the stages of the proposed study.
10.
There will be 02 surveys viz. in 1st. and 3rd project year. The period for the completion of each survey shall be 6 months from the date of signing of the contract. The 2nd survey to be undertaken in 3rd project year i.e. after a period of two years of first survey will additionally compare and investigate how (and why) the findings are different from the first survey and if HE institutions implemented the measures/suggestions made as a result of first survey and, how and up to what extent the HE institutions got benefit of the initial survey findings.
11.
The activity will be governed under World Bank Guidelines for Selection and Employment of Consultants under IBRD Loans and IDA Credits & Grants and rules and regulations of the Higher Education Commission as applicable in this case. For all the operational and management purposes, the selected firm will seek guidance from the designated representative of TESP/HEC.
The adherence and progress towards completion of the activity/periodic work will be monitored and appraisals will be conducted by the TESP / HEC Working Group members meeting on a fortnightly basis with the selected firm during the survey period.
Page 15
Tertiary Education Support Program (TESP)- Terms of Reference on Employers perception survey about quality of graduates
2012
12.
All the materials (physical and non- physical) produced by this funding including, but not limited to, system, data, papers, methodologies of the surveys will belong to HEC. Key guidelines regarding different issues related to the activity are pre-set by the HEC and will be followed accordingly: these include: Shortlisting of Consultants and Designing of Contract, Receipt and Opening of Proposals and clarification or Alteration of Proposals, Evaluation of Proposals, Consideration of Quality and Cost including evaluation of the Quality and opening of Financial Proposals Negotiations and Award of Contract including publication of the Award of Contract, Debriefing by the Borrower Rejection of All Proposals and Re-invitation
Please also note that the costs of preparing the proposal and of negotiating the contract, including a visit to the Client, are not reimbursable
Page 16