Running Head: EVALUATION OF ADULT PROGRAMS

Evaluation of Adult Learning Programs Jaclyn N. Pagnotta Florida Atlantic University

EVALUATION OF ADULT PROGRAMS Introduction Early this semester I had the opportunity to interview an administrator in the field of adult and community education. For this interview I selected Ms. Elisa Moro, who is the Career Advisor for a non-profit intermediary organization. She facilitates workshops across the Palm Beach county school district and Palm Beach State College. I choose to interview Ms. Moro because as an intermediary organization administrator she has a unique perspective on both the school district and state college setting. Ms. Moro works for Prime Time of Palm Beach County, a nonprofit intermediary organization that serves afterschool programs and practitioners. The organization provides supports and resources that increase program quality to positively impact school-age youth. Prime Time provides a set of quality standards, a system for reaching these standards through assessment, technical assistance and resources, as well as a broad and diverse range of program enhancements to afterschool providers. They also offer a set of core competencies, professional development opportunities, and networking events for afterschool professionals across the county. Prime Time has partnered with Palm Beach State College to develop credit and noncredit educational pathways that are relevant to afterschool practitioners. Prime Time offers scholarship opportunities for practitioners who are interested in pursuing degrees or certificate programs relevant to the field and access to the WAGES Florida Project, which provides salary incentives for those who pursue related education. In addition to this, Ms. Moro offers professional career advising to afterschool practitioners who are considering furthering their education. Career advising provides career counseling, educational planning, assistance with course scheduling and financial aid applications, and more. Career advising is committed to empowering afterschool practitioners and providing ongoing support as they work to reach their educational and career goals. Ms. Moro describes the bulk of her administrative responsibilities to include program planning, recruiting, representational functions, marketing, and conducting relations with students including counseling, referring applicants, and reviewing performance. My hope was to uncover a critical issue during this interview, and the issue of evaluation arose. Ms. Moro expressed her main challenge as lack of program evaluation. I have

EVALUATION OF ADULT PROGRAMS selected this as my critical issue for the semester. A major concern Ms. Moro expressed is individual training programs are not evaluated effectively. Knowing the importance of evaluation, she is troubled by this but struggles to find the time to develop a tool to test out and

roll out to the organization. Many other organizations share this struggle. This paper will address evaluation tools and benefits for adult and community education programs. History Program evaluation is the use of social research procedures to systematically investigate the effectiveness of social intervention programs such as education and training (Stufflebeam & Shinkfield, 2007). The term program evaluation became popular in the midsixties (Ball, 2011) when those responsible to producing educational programs needed this kind of information to make decisions. The legal push for evaluation in adult education strengthened in the late 1980s. Condelli (1996) stated: The 1988 reauthorization of the Act increased state requirements evaluation should address in evaluation. The National Literacy develop indicators of program evaluation of their local programs. The success in recruitment, retention and improving Act also required the Department of Education to develop program quality to guide states in the development of the indicators (p. 106). Many forces have come together to make evaluation a critical issue. Political for local program evaluation by specifying six topic areas that and by mandating the use of standardized test scores Act of 1991 took a further step by requiring states to quality within two years and to use them in indicators were to assess programs students literacy skills. The model indicators of

EVALUATION OF ADULT PROGRAMS forces such as the Acts listed above and several others to follow paved the way by requiring legislative mandate. Financial forces have made evaluation a priority as well. Most funders now require data demonstrating the results of programs. They want to see the impact of a program to determine if they should fund that program and continue funding it. In grant applications evaluations plans are requested and in reports back to the funder, evaluation results are requested. In many cases, it is difficult if not impossible to receive funding without using evaluations. Aside from political and financial reasons for evaluation, a big emphasis is on enabling social transformation and improving the facilitating and learning experience for the participants (McNamara, 2010). Stakeholders The issue of evaluation affects many stakeholders. The groups most directly

impacted are the adult learners who participate in the program, the facilitator, the program itself, the organization that hosts the program, the funders that fund the program, and the communities that the participants serve. The funders role should be to require evaluation and certain criteria in order to receive funding. This will help ensure that the program is being responsible with their dollars. The adult learners role is to evaluate the program honestly to inform the program of tips to improve. The facilitators role is to self-evaluate to continually grow us a facilitator and provide the best learning experience for the program. The participants will use the training to better serve the community in whatever capacity they may be. Quality evaluations can help produce quality trainings. These professional development offerings will help produce a more effective community. This impact could lead to even better outcomes as a culture of evaluation and

EVALUATION OF ADULT PROGRAMS improvement continues.

Ms. Moro as an adult and community education administrator has a major stake in this issue. By not using evaluation, she is placing her programs at risk for lower performance and questioning from the informed public. As an administrator, Ms. Moro has a responsibility to stay informed and up to date and do all in her power to manage high quality programs. Alternative Directions Condelli (2010) states, Educational evaluation is not therefore an objective, external, value-free process, but rather is deeply influential in shaping educational philosophy and policy. The conceptual and ethical stance it adopts is influencing the educational debates to a significant degree. The worst thing that could happen is to not evaluate programs at all. This would leave programs without quality assurance or measurable program outcomes. The best thing that could happen is that adult programs everywhere embrace and adopt evaluation plans as a piece of the continuous cycle of planning adult education programs. There are many different options and approaches in selecting and determining the type of evaluation to use. Three popular approaches include: self-evaluation, formative and summative evaluations. The main difference between formative and summative evaluations is the timing in which it is administered. Summative evaluations are completed after the education is delivered. Formative evaluations are conducted during the educational program. Both formative and summative evaluations are needed in the development of a program (Rossi et al., 1999). Another approach, self-evaluation, seeks to move away from external evaluation and towards empowering the practitioner to evaluate themselves. Regardless of model or approach chosen, a number of key questions need to be

EVALUATION OF ADULT PROGRAMS answered when organizing a program evaluation (Thackwray, 1998): How is evaluation defined? What are the functions of the evaluation? What are the objects of the evaluation? What kinds of information should be collected regarding each object? What criteria should be used to judge the merit of an evaluation object? Whose interests should be served by the evaluation? What methods of enquiry should be used? Who should do the evaluation? By what standards should the evaluation be judged? How and when should the results be presented?

Recommendations Ms. Moros grant-funded programs are being evaluated by her research department using the pre-test, post-test format. These results are analyzed across the years of the funding period. The results are compiled into reports for presentations and deliverable requirements. In many cases, the results are compared with other programs across the county (sometimes even across the country) to produce richer data for larger, initiatives. However, her programs that do not receive grant funding are not evaluated like this, or at all for that matter. I recommend that Ms. Moro take into consideration the best practice of evaluating her departments programs, as evidenced by the literature. I have prepared three evaluation documents that can be used across many subject areas for trainings offered in the adult and community education arena. Ms. Moro could have her facilitators administer these assessments at the end of an educational offering. By capturing these results and analyzing the data, Ms. Moro will have a better idea of the effectiveness of the programs her department offers.

EVALUATION OF ADULT PROGRAMS

Ms. Moro expressed her main challenge as lack of program evaluation, the inspiration for this critical issue paper. Specifically, I took on the challenge of providing her with a generic evaluation tools that can be used across several programs. Though our interview and follow up questions, I believe her programs would be best evaluated by in person surveys. The surveys below have open ended questions as well as questions to be scored using a Likert scale. Completion of these evaluations could be made necessary in order to receive the certificate of completion for the course, training, or workshop. The first is an application of learning document that challenges the participant to create an individualized plan to apply what they just learned. By creating an action plan, the participant has a plan to follow to make the most of the content covered in the program. The second is an evaluation form for the participant to provide feedback of their experience. I have also included a long-term follow up assessment to be administered three months after the initial program was offered. This will serve as a tool to see the long-term benefits of skills learned and applied.

Application of Learning Create an individualized plan, specifying how you will apply what you just learned. Action Plan List knowledge, skills, attitudes, values and/or feeling learned.

Specify when, where, and how you want to apply what you have learned.

EVALUATION OF ADULT PROGRAMS Specify a time frame. Name people who could offer assistance, support.

List other resources that might be helpful (such as books, trainings).

Specify how you will know you are successful.

Review this plan with at least one other person, make changes as appropriate. Trainer will email participants to follow up and schedule a coaching session to go over your plan and help facilitate the process of initiating your application.

(Caffarella, 2002, p. 202) Participant Evaluation Form Training title Please circle the ratings that best describe your reaction to this session: 1= No 1. Were the session objectives clear? 2= Somewhat 3= Yes, definitely 1 2 3 Date

2. Were the instructional techniques and materials helpful

EVALUATION OF ADULT PROGRAMS in your learning of the material? 1 2 3

3. Did the instructor focus the presentation on the session objectives? 1 2 3

4. The overall session contributed to my knowledge and/or skill base 1 2 3

5. Please identify any information and/or skills you can use from the training:

6. Please suggest improvements for this session:

(Caffarella, 2002, p. 243)

3-Month Follow-up

FOLLOW UP SURVEY
Training Workshop Title: Instructors Name: Date of training:
For each item identified below, circle the number to the right that best fits your judgment of its quality. Use the rating scale to select the quality number.

Survey Item

Scale
D Neutral

EVALUATION OF ADULT PROGRAMS

is a g r e e
1. 2. 3. 4. 5. 6. 7. 8. 9.

10
A g r e e
2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5

Learned something new Applied knowledge 3 months later Course Contained Quality Content I learned/gained new knowledge of skills from this training I have applied what I learned from this training in my job Satisfied with Instructor's techniques Handouts and materials will be /useful Presenter promoted and encouraged constructive dialogue Course met my expectations

1 1 1 1 1 1 1 1 1 1 1

10. Training was a worthwhile investment for my employer and me 11. I would recommend this course to someone else

The first tool, the application of learning, is recommended to be administered during the training session, prior to dismissing the students (Cafarella, 2002). While in the classroom, students can interact and review the plan with their peers. This will help them create a community of practice where they can exchange ideas and help one another toward accomplishing their goals. The second tool, the participant evaluation form combines open and closed ended questions. This assessment could be passed out at the completion of the educational offering. Alternatively, this could be emailed out after the participants leave to keep records

EVALUATION OF ADULT PROGRAMS electronic. The third tool is a similar assessment to the first. However, this will be emailed out three months after the educational offering. This assessment will measure the learning transfer and application. It will allow programs to monitor the long-term implications of their training

11

and practicality of the objectives. Participants can evaluate how the training has impacted them in the months since they completed it. Tracking response rate would also be a useful measure for this assessment as well as the other two. According to Kells (1992) institutions and programs can be strengthened substantially through effective evaluation and the basis for choices about the future can be soundly established by a combination of internal self- assessment and unbiased, informed peer review. Reflection I respect Ms. Moro greatly for the work that she does and for taking time out of her busy schedule to accommodate the interview I had with her. I was eager to focus this paper on evaluation as my critical issue for the course in order to give back to Ms. Moro for the insight she shared with me. Identifying a generic tool for Ms. Moro to use across her programs would be of great benefit to her programs, her organization, and her funders. My hope is that she appreciates the effort to help her in her role. The three evaluation tools are simple to administer and would allow her to capture data on the programs offered. These three tools combined will help give her and her stakeholders a better idea of the effectiveness of the programs they offer.

EVALUATION OF ADULT PROGRAMS

12

References Ball, S. (2011). Evaluating Educational Programs. Princeton, NJ: Educational Testing Service. Caffarella, R. S. (2002). Planning programs for adult learners: A practical guide for educators, trainers, and staff developers (3rd ed.). San Francisco, CA: Jossey-Bass. Condelli, L. (1996). Evaluation systems in the adult education program: The role of quality ----------indicators. Washnington, DC: Pelavin Research Institute. Creswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th ed). Boston, MA: Pearson. Kells, H. (1992). Self-regulation in higher education: A multi-national perspective on collaborative systems of quality assurance and control. London, ENG: Jessica Kingsley.

EVALUATION OF ADULT PROGRAMS McNamara, G., Joyce, P., & OHara, J. (2010). Evaluation of adult education and training programs. Dublin, IE: Elsevier Ltd. Publication manual of the American Psychological Association (6th ed.). (2009). Washington, D.C.: American Psychological Association.

13

Rossi, P. H., Freeman, H. E., and Lipsey, M. W. (1999). Evaluation: A systematic approach (6th ed). Thousand Oaks, CA: Sage. Stufflebeam, D. & Shinkfield, A. (2007). Evaluation theory, models, and application. San Francisco, CA: Wiley. Thackwray, B. (1998). Effective evaluation of training and development in higher education. London, ENG: Kogan Page.