Systems Appraiser Guide: Academic Quality Improvement Program

Systems Appraiser Guide
Valuable Information for Systems Appraisers
Academic Quality Improvement Program

The Higher Learning Commission
Contents
The Systems Appraisal Process .............................................................................................. 3
Systems Appraisal Glossary ............................................................................................ 3
Identifying Accreditation Issues in the Systems Appraisal ................................................ 5
Dealing with Systems Portfolio Gaps ............................................................................... 7
Where evidence an institution is meeting the Commission’s
common Criteria for Accreditation is likely to appear in a Systems Portfolio ................ 9
Category Summary Statements ......................................................................................10
Systems Appraisal Worksheets ..............................................................................................10
Instructions for Completing the Systems Appraisal Worksheets ......................................10
Individual Worksheets 1 – 9............................................................................................12
Chart of the flow among worksheets 1 – 9 ......................................................................14
An MS Word version of the Worksheet Template is available for
download at www.AQIP.org. Select the Downloads link and look
under the AQIP Peer Reviewers category.
Steps in the Systems Appraisal Process................................................................................15
Systems Appraisal 1: Independent Review .....................................................................15
Systems Appraisal 2: Consensus Review .......................................................................16
Systems Appraisal 3: Consensus Conversation ..............................................................17
Writing Feedback in Appraising a Systems Portfolio ............................................................19
Content Guidelines.........................................................................................................19
Independent Worksheets ....................................................................................21
Effective Comments for Results (R) Questions....................................................22
Consensus Worksheets ......................................................................................23
Strategic and Accreditation Issues Analysis (Draft, Proposed, and Feedback) .....23
Writing “Feedback Ready” Comments.................................................................24
Writing “Actionable” Comments...........................................................................25
Style Guidelines .............................................................................................................25
Grammatical Considerations ..........................................................................................26
Frequently Asked Questions about Systems Appraisals.......................................................28
What’s New in the Systems Appraiser Guide:
• Using the institution's Self-Evaluation in preparing feedback, page 17.

• New content guidelines, page 19.
• New content in Strategic and Accreditation Issues, page 24.
• The current Appraiser Schedule has been removed from the Guide and is now
available in the AQIP Peer Reviewers section of Downloads on the AQIP Website.
• The contact information for the current Systems Appraisal Process manager can be
found in the Staff Contacts section on the AQIP Website.
©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08.
2
The Systems Appraisal Process

The AQIP Systems Appraisal is a consistent, cost-effective process designed to provide an
AQIP organization with professional feedback representing the consensus view of a team of
higher educators and others experienced in continuous quality improvement and systems
thinking. The process begins when an organization informs AQIP its Systems Portfolio is ready
for review, and ends 12 weeks later with the delivery of a 35-45 page Systems Appraisal
Feedback Report that includes three components: a Critical Characteristics Analysis, which
shows the organization how the team understood its distinctive mission, context, and goals;
Category Feedback on each of the nine Categories, identifying what the team sees as the
organization’s strengths and opportunities for improvement; and a Strategic and Accreditation
Issues Analysis, in which the team identifies what it views as the highest strategic priorities for
the organization’s future. In addition, the team will provide the organization with a potentially
publishable two-three page Appraisal Summary that captures the team’s appraisal of the
organization’s developmental maturity on each of the nine Categories.
The goal of the Systems Appraisal Feedback Report and Appraisal Summary is to provide the
organization with actionable information it can use to improve its processes and performance.
AQIP expects organizations participating in the Program to maintain accredited status to
demonstrate overall improvement from one Systems Appraisal to the next. AQIP bases its
reaccredidation decisions every seven years on an organization’s overall pattern of
improvement (including information from Systems Appraisals, regular reports on Action
Projects, and other data), and thus leaves it up to the organization to decide whether, when, and
how to address the opportunities for improvement and Strategic and Accreditation issues
identified in a Systems Appraisal.
In the rare case where there is strong evidence that an organization is no longer committed to
continuous improvement or in a position to benefit from AQIP participation, the Systems
Appraisal team can alert AQIP. When such a case occurs, AQIP will work with the organization
to gather information that will allow the AQIP Standards and Admissions Panel to recommend
appropriate action — perhaps even the return of the organization to the more traditional PEAQ
accreditation process.
The appraisal team may also nominate a specific organizational process or system for AQIP
Outstanding Practices recognition, directing AQIP staff to gather additional information about
the nominated process. If selected for recognition, the organization will be given a formal
opportunity to share its success with the rest of the higher education community.
Systems Appraisal Glossary

Category Summary Statement. A statement, crafted by the team (beginning as each team
member completes each Independent Category Worksheet) that expresses the team’s overall
formative evaluation of the institution’s progress and developmental maturity to date on the
Category, and, in the team’s judgment, the institution would be best advised to make the focus
of its attention as it continues its journey to continuous quality improvement. These statements
should be positive, stressing what the institution has accomplished and explain the benefits of
where it needs to focus next. Statements should be written so that the group of nine Category
©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08
3
Summary Statements, together with the Strategic and Accreditation Issues Analysis, can be
released to the public as an Executive Summary of the Systems Appraisal.
Consensus Category Analysis. The final, revised version of the feedback on each Category
produced as a result of the Consensus Conversation. These Consensus Category Analyses are
based on the Consensus Category Drafts prepared before the telephone discussion, and
reflects the suggestions and changes of the team that occurred during that conversation. In
other words, each Consensus Category Draft, plus any changes from the Consensus
Conversation, will be summarized/combined for the organization into one Consensus Category
Summary, which will include a Category Summary Statement expressing in a few sentences the
Appraisal Team’s overall formative evaluation of the Category.
Consensus Category Draft. A worksheet completed by one team member, summarizing the
Independent Worksheets provided by all team members on one particular Category. The
Consensus Category Drafts for all nine Categories are the basis for the Consensus
Conversation.
Consensus Conversation. The conference call held by the team to reach consensus on the
Category Feedback and the Category Summary Statement for each Category, the Critical
Characteristics Analysis and the Strategic and Accreditation Issues Analysis that will be
provided to the organization in the team’s System Appraisal Feedback Report.
Consensus Review. The stage in the Systems Appraisal when team members (other than the
team leader) are assigned to summarize the Independent Worksheets for one or two of the
Categories in order to produce Consensus Worksheets - one Consensus Worksheet per
Category.
Critical Characteristics Analysis. Analysis produced by the team leader with any
modifications made after the Orientation Conversation. (The Analysis may be discussed and
modified during the Consensus Conversation.) This Analysis is part of the final Systems
Appraisal Feedback Report.
Critical Characteristics Draft. Summary consensus proposal prepared from the Critical
Characteristics Worksheets produced by the team leader. Produced and distributed to team
members in preparation for the Orientation Conversation.
Critical Characteristics Worksheet. Worksheet produced by each member of the team and
submitted to the team leader, after studying the Organizational Overview and skimming the
Systems Portfolio, that attempts to identify the key characteristics that make the organization
being appraised unique and distinct.
Independent Category Worksheets. Worksheets every team member produces on each of the
nine Categories during the Independent Review stage of Systems Appraisal.
Independent Review. The stage in the Systems Appraisal when each team member is
independently producing first a Critical Characteristics Worksheet - as a review of the
Organizational Overview - and second, a set of Independent Worksheets as a review for each of
the nine Categories.
Organizational Overview. A five-page introduction to the Systems Portfolio in which the
organization answers eight direct questions and describes the key factors and capabilities that
underlie its strategies for success. This is also the means for the organization to explain its
distinctive mission and context to the Systems Appraisal team.
4
Orientation Conversation. A conference call that occurs after the team leader has produced
the Critical Characteristics Summary. This call can identify questions the team needs answered
in order to complete its Appraisal.
Strategic and Accreditation Issues Analysis. Analysis produced by the team leader with any
modifications made after the Consensus Conversation. This Analysis is part of the final Systems
Appraisal Feedback Report.
Strategic and Accreditation Issues Draft. Summary of the worksheets produced by each
member of the team during the Independent Review stage, produced by the team leader as a
consensus proposal for the Consensus Conversation.
Strategic and Accreditation Issues Worksheet. Worksheet produced by each member of the
team and submitted to the team leader, in conjunction with the Independent Worksheets. The
worksheet attempts to identify the potential crosscutting or priority issues for the future of the
organization being appraised, and identifies any issues related to the organization’s compliance
with the Commission’s Criteria for Accreditation.
Systems Appraisal Feedback Report. The final report sent to the organization. It includes a
Critical Characteristics Analysis, Category Feedback, the Appraisal Summary and a Strategic
and Accreditation Issues Analysis.
Systems Appraisal Team Leader. The leader of the Appraisal team who is responsible for
coordinating the Appraisal team and ensuring that the reviews are completed by the deadlines.
The leader may be responsible for collecting any additional information from the organization
that the team needs in order to conduct its reviews. And the leader may also be asked to
deliver/present the Systems Appraisal Feedback Report to the organization.
Systems Appraisal Team. Typically a group of six (including the team leader) AQIP Peer
Reviewers who are assigned to complete an Independent and Consensus Review of an AQIP
organization’s Systems Portfolio. The team will produce a Systems Appraisal Feedback Report
for an organization.
Systems Portfolio. The description, created by the organization, of its key processes, their
performance effectiveness, and the way the organization improves its processes and
performance. The Systems Portfolio is written in response to the questions under each of the
nine Categories. The organization also prepares and submits an index to the Systems Portfolio
showing how the Portfolio presents evidence for each of the Commission’s Criteria for
Accreditation.
Identifying Accreditation Issues in the Systems Appraisal

Like all institutions accredited by the Higher Learning Commission, colleges and universities
participating in AQIP must demonstrate that they meet the Commission’s Criteria for
Accreditation. Every seven years, AQIP determines that a participating institution meets these
five Criteria through a formal process called Reaffirmation of Accreditation. This
determination is based on a pattern of evidence provided by the institution during the previous
seven years of its participation in AQIP.
A major source of evidence for the Reaffirmation of Accreditation judgment comes from the
comprehensive review of institutional systems and activities that constitute the Systems
Appraisal. Not only do institutions create a Systems Portfolio that describes the institution’s
current processes and performance results, but every AQIP institution also provides the
5
Systems Appraisal team with an Index to the Criteria for Accreditation correlating the evidence
in its Systems Portfolio with the Commission’s Criteria for Accreditation and their Core
Components. The purpose of this Index is to make it easier for the Appraisal team to determine
whether the institution has provided sufficient and persuasive evidence that it is meeting the
Criteria for Accreditation.
During their Independent Review of the Systems Portfolio, team members should identify
potential accreditation issues for the institution to address. As they complete their nine
Independent Category Worksheets (Worksheet #4), they should be on the alert for any
instances in which an institution’s description of its systems or results signals the institution’s
failure to meet an accreditation expectation. Although such instances should be rare, if one
occurs the Independent Reviewer should note it in drafting Worksheet #5, Strategic and
Accreditation Issues, highlighting the issue with an A (for accreditation) and identifying the
specific Criterion for Accreditation that evidence indicates is not met. The team leader will
receive these worksheets from each team member and prepare from them a single Strategic
and Accreditation Issues Summary (Worksheet #7) which will serve as a “consensus proposal”
for the team’s discussion during the Consensus Conversation. If the entire team agrees that an
issue identified validly threatens the institution’s accreditability, then that issue should appear in
the final feedback report on the Strategic and Accreditation Issues Analysis (Worksheet #9).
During its Consensus Conversation regarding accreditation issues, the team will also review the
institution’s Index to make certain the institution has presented, in its Systems Portfolio,
sufficient and persuasive evidence that it meets each of the Criteria for Accreditation. All team
members should review this Index prior to the Consensus Conversation, and note any areas
where they believe the evidence is insufficient to show that the institution meets a Criterion. If
the team’s consensus judgment is that there are significant gaps in the pattern of evidence
presented, then the team leader should be directed to add these gaps to Worksheet #9 as
accreditation issues (A), specifying which Criteria are in question, and why the evidence is
insufficient.
An institution will have ample opportunity to address and correct any accreditation issues
identified by a Systems Appraisal prior to its scheduled Reaffirmation of Accreditation. AQIP will
conduct a Checkup Visit to each institution two or three years prior to its Reaffirmation, and the
institution will have opportunity to provide evidence that issues that concerned the Appraisal
team are no longer relevant, either because the institution has provided evidence to fill gaps
identified by the team, or because the institution has taken action to correct problems identified
by the team. If accreditation issues remain “on the table” after the completion of the Checkup
Visit, the institution will have additional opportunities to augment and correct the evidence that it
meets the Criteria for Accreditation when its Reaffirmation of Accreditation takes place.
Thus the System Appraisal team’s signaling an accreditation issue, through serious, does not
imperil an institution’s accreditation because the institution has sufficient opportunity to explain
to AQIP why an issue is spurious (by providing the missing evidence) or how a valid issue has
been addressed so that it should no longer cause concern — before the Reaffirmation of
Accreditation occurs. However Systems Appraisers should use this mechanism judiciously,
signaling only accreditation issues (A) that rise to the level, which would threaten the
institution’s ability to remain accredited. Other important crosscutting issues for the institution
can be identified as strategic issues (S or SS) while issues of more limited scope will be
identified as opportunities for improvement (O or OO) in the review of specific Categories.
6
Dealing with Systems Portfolio Gaps

An institution must tackle every P, R, and I item in the AQIP Categories
AQIP expects participating higher education organizations to examine themselves on all of the
items under AQIP’s nine Categories. Any college or university committed to reaching its full
institutional potential must ask itself every one of these questions, seriously and regularly.
But not every institution is equally mature in its quality thinking, and the different circumstances
in which institutions find themselves force development in some areas before others. An
institution may still be behaving reactively in some processes while it develops proactive
approaches in others.
The Systems Portfolio provides an institution a means of formally stating answers to the
questions in each item. The institution should summarize its current level of maturity in the C, or
context, items that begin each Category. Its response to the C items should explain to the
Systems Appraisal team where it believes it currently stands in respect to the P, R, and I items
in that Category. Appraisers should confirm or contest this institutional self-understanding in
their analysis of the Category.
Examining itself on every item will help an institution discover the gaps that exist between what
the institution does now and what it could do to excel. Realistically, an institution won’t be able
to write a substantive response for every item and may give a brief “No reportable results”
response for some. An institution might not yet be able to describe stable, systematic
approaches for some processes, or it might discover that it has no performance result data for
certain processes. Many institutions discover that systematic improvement cycles appear only
after they have stabilized their processes, and collected and analyzed data on process
performance.
To ensure it gets useful feedback on the areas where it is currently working to improve, an
institution must write substantive responses to no fewer than one-third of the total P, R, and I
items within each of the nine Categories. (Institutions may write substantive responses for all
items, but must stay below the overall 100-page limit.)
In its Systems Portfolio, an institution addresses at length a subset of P, R, and I items

that it selects
AQIP encourages an institution to focus its Systems Portfolios on those P, R, or I items where it
has something substantive to say about its processes, results, or improvement cycles. For
example, if it is developing and deploying strong and effective processes in a particular area,
the institution will describe them under the relevant P items. Similarly, it will report performance
results under the relevant R items, and actual cycles of improvement under the I items.
If there is nothing to report, the institution should say so, clearly and concisely, ideally in a short
comment like “No reportable information is currently available about how we set our learning
goals for undergraduates. We recognize stabilizing the design of this process represents an
major opportunity for educational improvement.” In turn, the Systems Appraisal team may
acknowledge that the institution is aware of a gap in this area, and consequently that the item
represents an opportunity for improvement. An extensive comment in feedback may not be
required in such cases.
7
In its feedback report, the team should designate a “no reportable…” item with an O symbol
(indicating it represents an “opportunity for improvement”) and merely state “No reportable
information from the institution at this time.” If the team believes that the institution’s lack of
attention (or progress) in an area is critical, it should promote the O to an OO, and concisely
explain why continuing to ignore this item would be perilous for the institution.
Using the shorthand response of “No reportable…” allows an institution to focus the 75-100
pages of its Systems Portfolio on those areas where it can most benefit from feedback. By not
wasting space in describing things that do not now exist the team’s feedback becomes more
focused, and more useful.
Responding to an item with “No reportable…” does not exempt an institution from confronting
the issues the item covers. Every institution striving for systematic quality must deal with all of
the process, results, and improvement issues raised in the AQIP Categories. But there is a
natural developmental sequence to where an institution puts its efforts. Just as a child can’t
learn to run before it learns to walk, an institution can measure the effectiveness of a process it
hasn’t formulated, nor can it improve a process whose current performance is unknown. Thus it
makes no sense for an institution to write long responses to items it has not yet seriously
addressed.
At some point in the future, after the institution’s processes become more mature, it will have to
develop and use measures of those processes and implement systems for continuous
improvement.
Some examples
An institution examines itself on item 3P5 and concludes that it currently has no systematic or
dependable processes for deciding whether it should serve a new student or stakeholder
population. For item 3P5 in its Systems Portfolio, it states “No reportable processes in this
area.” The team concurs by assigning 3P5 an O, and stating, “No systematic processes for
identifying new students or stakeholders are currently in place.”
An institution has described good processes for leadership and communication, but examines
itself on item 5R1 and concludes that it has no dependable measures or indicators of these
processes’ effectiveness. It simply states “No reportable results in this area” for item 5R1 in its
Portfolio. The team decides that this is a major gap, assigns the symbol OO to it, and explains in
its comment why it is imperative that the institution design measures of leadership and
communication effectiveness and gather hard data on these processes.
An institution that has not yet developed effective processes for planning is unlikely to have data
on the performance of these processes; consequently, when it writes its responses to items in
Category 8, it decides to invest its Portfolio space in describing the current state of its
processes, thereby inviting feedback from the Appraisal team on that focus.
In addition to these correspondences, pay attention to the fact that:
• AQIP Category Context for Analysis (C) and Processes (P) responses often contain
evidence that relate directly to the Criteria’s Core Components.
• AQIP Results (R) responses may provide very strong evidence for any of the five Criteria
for Accreditation
• AQIP Improvement (I) responses may provide excellent evidence for Criterion Two and
Criterion Three in particular.
8
Where evidence is likely to appear in a Systems Portfolio that an institution is

meeting the Commission’s common Criteria for Accreditation
Commission’s
Student Learning
Preparing for the
Engagement and
Criterion Three:
Discovery, and
Criterion Four:
Criteria
Criterion Five:
Criterion Two:
Criterion One:
Application of
and Effective
Mission and
Acquisition,
Teaching.
for
Integrity.
Service.
Future.
Accreditation
AQIP Categories
1. Helping Students Learn x x x
2. Accomplishing Other
x x x
Distinctive Objectives
3. Understanding Students'
and Other Stakeholders’ x x x x
Needs
4. Valuing People x x
5. Leading and
x x x
Communicating
6. Supporting Institutional
x
Operations
7. Measuring Effectiveness x x x x
8. Planning Continuous
x
Improvement
9. Building Collaborative
x x
Relationships
9
Category Summary Statements

Earlier, AQIP used “rubrics” to communicate to institutions their level of maturity on each of the
nine Categories. Because these rubrics were standardized, some institutions interpreted them
as grades.
AQIP has replaced the rubrics with Category Summary Statements, in which teams craft a
separate statement for each of the nine Categories. Each Statement should be a short
paragraph consisting of 2-5 sentences that praise the institution for the progress it has made in
continuous improvement, including:
o identifying processes that need to be designed

o identifying ways to improve the design of existing processes
o deploying robust processes broadly
o clarifying the goals of existing processes
o specifying the performance measures used to evaluate how well processes are
achieving their goals
o analyzing the measures gathered from the operation of processes
o using measured data to improve processes and performance
Be obnoxiously positive — the goal is to help the institution develop its quality culture by letting
it build upon past successes.
Systems Appraisal Worksheets
Instructions for Completing the AQIP Systems Appraisal Worksheets

AQIP compiles all of these worksheets into the Systems Appraisal Feedback Report, and emails
each team member a copy. The team leader and team members should not send anything
directly to the institution.
AQIP System Appraisal feedback reports are formatted using two different paragraph styles.
These two styles are built into the following worksheet templates that AQIP provides to Systems
Appraisers.
The styles used are identified as CC and SAC. Either of these indicators will appear in the
formatting toolbar (which you can turn on using Word’s view…toolbars command. The
material on this instructions page, in other portions of the worksheets (such as titles), and in the
boilerplate sections of the feedback report appear in normal or headings styles. You cannot
make changes or additions to this page.
AQIP uses an unvarying CC style for the Critical Characteristics that Systems Appraisers
identify, and also for Strategic and Accreditation Issues Worksheets (#5, 7, and 9). Each Critical
Characteristic begins with the letter O, followed by a single digit that refers to one of the eight
questions each institution uses to organize its Overview, followed by a lowercase letter (a, b, c,
etc.), without intervening spaces). The lowercase letters help to distinguish different Critical
Characteristics that the team identified for the same questions.
10
The items in Strategic and Accreditation Issues Worksheets begin with a capital SI or AI, a tab,
and then the comment.
The paragraph format for style uses Arial 11 point type, has a tab set at 0.5 inch from the left
margin, a hanging indent of 0.5 inches, 6 points of leading before and after each paragraph, and
line spacing set to 1.5 lines.
O3b This is a sample of a critical characteristic identified by the Systems Appraisal team. The
letter indicates it is the second comment that the team identified as responding to
Overview question 3. You will use this format for Worksheets 1 – 3, and when reprinting
relevant Critical Characteristics on Worksheets 4, 6, and 8.
AI This is a sample of an accreditation issue as it might appear on worksheets 5, 7, or 9.
AQIP uses an unvarying SAC style for System Appraiser comments on each Category, the
comments appraisers identified as strengths (SS or S) or opportunities (OO or O). Each
comment begins with the Category item number (a single digit, then a letter — C, P, R, or I —
and then a one or two digit number, without intervening spaces). This item identifier is followed
by a tab, the rating — SS, S, O, or OO — another tab, and then the comment, written without
tabs, carriage returns, or extra spaces between words or sentences.
The paragraph format for style SAC that is employed for Appraiser comments uses Arial 11
point type, has two tabs set at 1 and 1.5 inches from the left margin, a hanging indent of 1.5
inches, 6 points of leading before and after each paragraph, and line spacing set to 1.5 lines.
1P2 OO This is a sample of a comment on a single Category item. Most appraiser

comments will look like this one. Style SAC is used to format System
Appraiser comments like these.
1P3-1P6 SS This is a sample of a comment on a range of adjacent Category items

that the institution has addressed with a single section of its Portfolio.
1P7,1R4 O This is a sample of a comment on two separate Category items that the
institution has addressed in a single section of its. This occurs rarely.
If in doubt (or if you mistakenly change the format) use the format painter (the little paintbrush
icon) to copy the format from the appropriate sample item above: highlight the sample item, click
the paintbrush, and paint it on all of the items you have created for which that style is
appropriate.
11
Individual Worksheets 1 – 9
1: Critical Characteristics Worksheet
Who prepares or completes it? Every team member (except the team leader).
When is it prepared? As soon as AQIP sends you the Systems Portfolio.
Then what happens? Team members email their completed Worksheets #1 to the team
leader.
2: Critical Characteristics Draft
Who? The team leader prepares this consensus proposal.
When? As soon as all team members email the team leader their completed Worksheets
#1.
Then? The team leader emails all team members Worksheet #2 so that they can prepare for
the Orientation Conversation, reaching agreement on the institution’s critical
characteristics.
3: Critical Characteristics Analysis
Who? The team leader incorporates the entire team’s agreements into this analysis.
When? After the Orientation Conversation, prior to the Independent Review of the Systems
Portfolio.
Then? The team leader emails all team members Worksheet #3 to guide their Independent
Review of the nine Categories as they prepare Worksheet #4.
4: Independent Category Worksheet
Who? Every team member (including the team leader).
When? After team leader distributes Worksheet #3, which marks the beginning of the
Independent Review stage of the Systems Appraisal.
Then? Each team member emails nine Worksheets #4 (one for each Category) to the team
leader. (Save the file for each worksheet with a name in this format: Crit2_ABC.doc or
Crit7_ABC.doc, where “ABC” represents the initials of the team member who prepared
the worksheet.)
5: Draft Strategic and Accreditation Worksheet
Who? Every team member drafts an independent list of these issues.
When? During the Independent Review stage, concurrently with the preparation of
Worksheets #4, team members prepare Worksheet #5, indicating strategic issues with
an S and accreditation issues with an A. (By definition, all issues that threaten an
institution’s accredited status are also strategic for that institution.) Accreditation issues
can include those where the evidence is missing that the institution meets one or more
accreditation criteria and those where the evidence indicates that the institution fails to
meet an accreditation expectation. Comments must refer specifically to one or more of
the Commission’s Criteria for Accreditation (and the relevant Core Components of each)
when identifying accreditation issues.
12
Then? Each team member emails Worksheet #5 to the team leader. (Save the file in this
format: Wksht5_ABC.doc, where “ABC” represents the initials of the team member who
prepared the worksheet.)
6: Consensus Category Draft
Who? Team members each prepare Worksheet #6 for one or two Categories, as assigned
by the team leader.
When? After the completion of the Independent Review stage, when all team members
have sent their Worksheets #4 to the team leader, the team leader will make consensus
assignments and distribute the appropriate Worksheets #4 to each team member.
Then? After completing one or two Consensus Category Proposals (#6), email them to all
team members, who will study them in preparation for the Consensus Conversation.
7: Consensus Strategic and Accreditation Draft
Who? The team leader prepares this consensus proposal from the Worksheets #5 prepared
by each team member.
When? Prior to the Consensus Conversation.
Then? The team leader emails all team members Worksheet #3 to guide their Independent
Review of the nine Categories using Worksheet #4.
8: Consensus Category Analysis
Who? The same team member who prepared Worksheet #4 is responsible for the
corresponding Worksheet #8.
When? The final version of Worksheet #8 is prepared following the Consensus
Conversation; it incorporates what the team decided in that discussion.
Then? The team member emails the completed Worksheet #8 to the team leader, who then
forwards it to AQIP.
9: Strategic and Accreditation Analysis
Who? The team leader completes Worksheet #9.
When? After the Consensus Conversation, which discussed Worksheet #7 (which the team
leader had prepared from the Worksheets #5 prepared by each team member), the
team’s decisions are incorporated into Worksheet #9.
Then? The team leader forwards the completed version of Worksheet #9 to AQIP.
Chart showing flow among the nine worksheets

The chart on the next page shows how the worksheets combine to make up the process for
Systems Appraisal.
13
Orientation Conversation
Each team member creates a Critical Team leader prepares a Critical Team agrees on a consensus Critical
❶ Characteristics Worksheet
➙ ❷ Characteristics Draft
➙ ❸ Characteristics Analysis
Consensus Conversation
Each team member creates nine Independent Team members combine all work into nine Team agrees on a final Consensus Category
❹ Category Worksheets
➙ ❻ Consensus Category Drafts
➙ ❽ Analysis for each Category
Each team member independently creates a Team leader prepares a Strategic and Consensus Conversation Team agrees on a final consensus Strategic and
❺ Strategic and Accreditation Issues Worksheet
➙ ❼ Accreditation Issues Draft
➙ ❾ Accreditation Issues Analysis
©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08
14
Steps in the Systems Appraisal Process

Systems Appraisal 1: Independent Review
(Estimated Total Time: 13 - 20 Hours)
1. Quickly skim the entire Systems Portfolio.
2. Carefully read the Organizational Overview (5 pages) and the organization’s responses to
the Context (C) questions for each of the nine Categories. In these sections, the
organization provides foundational information about its distinctive features, scope,
students, faculty, staff, facilities, technologies, environment, competitors, opportunities,
and vulnerabilities. Draft a 1-2 page Critical Characteristics Worksheet in response to this
reading in which each sentence should identify a single factor or characteristic that you
believe it is important for the Appraisal team to keep in mind as its members review the
Systems Portfolio. The goal here is twofold: (1) to make sure the Appraisal team, as it
appraises each Category, bears in mind a shared understanding of the most important
aspects of the organization’s identify, its current dynamics and the forces surrounding it,
and its internal momentum and aspirations; and (2) to make sure the organization
recognizes that the team understands and knows what makes the organization distinctive
when it receives the Systems Appraisal Feedback Report.
Number each sentence with the appropriate Organizational Overview identifier (i.e., if you
write two statements in response to what the organization says in item OO5, number
them OO5a and OO5b). Keep your statements in order, and skip a line (one carriage
return) between each statement.
3. Receive back, by email, a consensus Critical Characteristics Draft from your team leader,
and read it carefully. (In the interim, analyze the complete Systems Portfolio.) Participate
in a one-hour telephone Orientation Conversation, in which the team members agree on
(or agree to modify) the Critical Characteristics Analysis. If crucial questions emerge from
this conversation, questions whose answers are needed before the Independent Reviews
can proceed, the team leader will contact the institution and then communicate the
answers to all team members. The leader will then make any modifications to the Draft
and email each team member the Critical Characteristics Analysis.
4. Complete an Independent Category Worksheet for each Category. The Organization’s
Systems Portfolio is divided into Category sections, which should contain the answers to
each of the questions within that Category. Each answer should be identified with the
section/number of its corresponding question in the AQIP Principles and Categories for
Improving Academic Quality booklet. The organization’s Systems Portfolio probably will
not repeat the question so keep the Categories handy for reference. To complete an
Independent Category Worksheet for each Category section in the Portfolio,
• choose a Category section in the Portfolio to focus on, read the answers in that
section as a whole first.
• review the Critical Characteristics Analysis and indicate, on the worksheet, the 3 - 4
factors that you believe are most important in relation to that Category as a whole.
• examine each answer individually and write a series of bullets that respond to a single
answer or set of answers that the organization has grouped together to respond to the
corresponding question(s) found in the corresponding AQIP Category section of the
booklet.
15
• assign each bullet a symbol to identify whether it represents one of the organization’s
strengths in meeting the requirements of the Category (SS or S) or one of the
organization’s opportunities for improvement of its systems and processes that fall
under this Category (OO or O). Discriminate issues carefully, and decide whether
each is a strength or an opportunity for improvement. Do not write bullets treating a
single issue as both strength and an opportunity for improvement.
• choose one Category Summary Statement (from a formal, graded array of seven
Category Summary Statements) that best captures your perception of the
organization’s current overall level of achievement on that Category.
Considering everything you have learned and discussed about the organization,
complete a Strategic and Accreditation Issues Worksheet, identifying up to three
potential crosscutting or priority issues for the future of the institution. Email your nine
completed Independent Category Worksheets and your Strategic and Accreditation
Issues Worksheet to the team leader.
Systems Appraisal 2: Steps in the Consensus Review

(Estimated total time: 4 - 6 hours)
1. Your team leader will have assigned you either a single Category (Category 1, “Helping
Students Learn”) or two Categories (any two of Categories 2 - 9) for consensus, and
have emailed you the appropriate Independent Category Worksheets completed by the
five other members of your Appraisal Team. Your task is to combine the contents of the
Independent Worksheets corresponding to an assigned Category into a Consensus
Category Worksheet that will be your consensus proposal during the team’s Consensus
Conversation. You will complete one or two Consensus Category Drafts depending on
how many you are assigned.
2. To complete a consensus worksheet, first review the Critical Characteristics Analysis
section on each Individual Worksheet, and indicate on the worksheet the 3 -5 items that
reflect the consensus of the team as to which Critical Characteristics are most relevant
to evaluating that Category.
3. Read the bullets section on each Independent Worksheet and look for commonalities
among them. Craft a comment (1-2 sentences) for each strength and for each
opportunity for improvement, that all team members could support. Propose a symbol
(SS, S, O, OO) for each comment. Write your proposed comments and corresponding
symbols on the Consensus Worksheet.
Your comments should summarize the shared views of all team members concerning
the strengths and opportunities for improvement in this Category. To do this, sort all of
the bullets on the six worksheets first by the Category questions, and then (if there are
multiple bullets for a single question) by theme.
Comments suggested by only one or two team members should be captured on the
worksheet, but list them at the bottom and do not give them a symbol.
A good strategy is to select, from the assembled worksheets, the best comment for a
particular theme, and use it as the basis for your consensus comment, adding or
modifying it if necessary.
16
Discriminate issues carefully, and decide whether each issue overall is a strength or an
opportunity for improvement. Do not propose a comment that treats the same issue as
both a strength and an opportunity for improvement.
4. Record on the Consensus Category Draft (s) the independent reviewers who echoed
each consensus comment you have proposed by filling in the symbol (SS, S, O, OO)
each reviewer gave that comment. Put an “X” in the column for any reviewer who had no
comment similar to the one you are proposing. In the last row of the worksheet, fill in the
Category Summary Statement numbers suggested by each independent reviewer and
then fill in your proposed Category Summary Statement number (in the left column).
5. Email your one or two completed Consensus Category Drafts to the team leader who will
make sure all members of the team receive copies prior to the Consensus Conversation
during which the team will come to agreement on the feedback it will provide to the
institution. The team leader will combine the individual Strategic and Accreditation
Issues Worksheets into a Strategic and Accreditation Issues Draft, and all members of
the team will also receive copies of this document prior to the Consensus Conversation.
After receiving and studying all Consensus Category Drafts and the Strategic and
Accreditation Issues Draft, you will participate in a Consensus Conversation (probably a
teleconference) in which each Category is discussed, in turn, with the discussion led by
whichever team member crafted the Consensus Category Draft under discussion.
Following this conversation (estimated time: four hours), each team member will
incorporate suggested changes in each Category consensus proposal and send all
revised work to the team leader.
Systems Appraisal 3: Steps in the Consensus Conversation

(Estimated total time: 4 - 6 hours)
1. After receiving and studying the Consensus Category Drafts from the other members of your
team, you will participate in a Consensus Conversation in which each Category is
discussed, in turn, with the discussion led by whichever team member crafted the
Consensus Category Draft under discussion. After discussing each Category, the team
leader will lead a discussion of the Strategic and Accreditation Issues Draft. Your team
leader will arrange and communicate to you the time scheduled for the conversation (or
conversations, if the conversation must be scheduled in two installments).
2. Typically, the team will go through the nine Categories in a prearranged order suggested by
the team leader, and the team member who prepared the Consensus Category Draft will
lead the discussion on that Category, running quickly through:
• each comment that was crafted to capture the consensus view of the team, asking
for suggestions and objections from the rest of the team concerning each proposed
consensus comment and symbol (SS, S, O, OO)
• the set of comments rejected as not capturing the team’s consensus view, asking
team members who feel passionately that their comment should not be rejected to
argue for their inclusion.
Throughout these discussions, the goal is for team consensus — unanimous agreement or
agreement of all but one team member. With all team members well prepared in advance,
17
the discussion of each Category should

take about 20 minutes, with twice that
amount of time, perhaps, for Category 1,
“Helping Students Learn.”
Using the institution's Self-Evaluation
in preparing feedback
At this point in the process, in
preparation for the Consensus
Conversation, the Team Leader also
distributes to each team member copies
of the institution’s Self-Evaluation of the
responses it has provided in its Systems
Portfolio. (A sample of this two-page
form is pictured below). AQIP asks each
institution to rate (using SS, S, O, OO —
or “?” if it is unsure) how it currently
views its activity and performance for
each P, R, and I item to which it has
provided an in-depth response —
indicating whether it perceives this
areas is an institutional strength or
opportunity for improvement. (Download
Self-Evaluating Your Systems Portfolio
for AQIP’s instructions to institutions.) The Self-Evaluation’s goal is to stimulate each
institution to decide where it thinks its current processes and performance are good, and
where it thinks its improvement opportunities lie.
The institution’s Self-Evaluation should help your team give better and more useful feedback
to the institution by showing where team and institutional appraisals differ. For example, if
your team agrees with the institution that its personnel evaluation process is an outstanding
strength (OO), you may need to write only a short sentence or two explaining why you
agree. But if the institution thinks its system for new program development is a strength (SS
or S) but your team sees it as an opportunity for improvement O or OO), it is the team’s
responsibility to explain clearly why it sees the system as an opportunity that it is in the
institution’s interest to improve. Differences between the institution’s self-evaluation and
your team’s appraisals alert your team to explain, clearly and specifically, how and why it
reached a different judgment than the institution did. This is true whether your team’s
appraisal is more or less favorable than the institution’s.
At the end of the discussion for each Category, the team must also agree on a Category
Summary Statement that captures the team’s view of the organization’s maturity for that
Category. The team member who prepared the Consensus Category Draft (having
considered the independent ratings of all team members) suggests a Category Summary
Statement number, and others can argue for a higher or lower number until the team
reaches an agreement.
3. After all Categories have been discussed, the team leader facilitates a discussion of the
Strategic and Accreditation Issues Draft that has been distributed to all team members, and
the team reaches consensus on the contents of a Strategic and Accreditation Issues
18
Analysis that will be part of the final Systems Appraisal Feedback Report. The team should
strive to identify no more than 3-4 strategic issues that are crucial for the future of the
organization. The team also considers whether it should make any modifications in the
Critical Characteristics Analysis that will provide the organization assurance that the team’s
feedback is based on a clear understanding of its distinctive organizational identify and
dynamics.
4. Following the conversation, each team member incorporates suggested changes in the
comments for their assigned Categories, and sends a finished Consensus Category
Analysis to the team leader. The leader assembles these together with the Critical
Characteristics Analysis and Strategic and Accreditation Issues Analysis into the final
Systems Appraisal Feedback Report, and sends each team member and AQIP an electronic
copy.
Systems Appraisal Timetable

A timetable for the current years’ Systems Appraisals can be found in the AQIP Peer Reviewers
section of Downloads on the AQIP Website at www.AQIP.org.
Writing Feedback in Appraising an AQIP Systems Portfolio
Content Guidelines
Focus primarily on content in writing comments, on giving useful feedback to the organization
whose Systems Portfolio you are appraising. But how you say things can enhance — or
undercut — the credibility and value of the feedback you give. Here are some pointers about
content and style to guide your comment writing.
• Don’t write comments for Context items. These are included to provide the minimum
context the team needs to interpret the P, R, and I items. If the C items for a category
are inaccurate, incomplete, or unhelpful, provide constructive feedback in your Category
Summary Statement (e.g., “The Context items for Category One need to spell out the
specific current educational goals the institution has for all its students, even if these are
undergoing constant revision.” or “The student groups the institutions now serves and
wishes to serve are not clear in the Context for Category Three, making it difficult to see
whether the processes in place can assess stakeholder groups’ needs.)
• Draw linkages between the organizations responses to Categories questions, and
between a Category question response and the organization's Organizational Overview.
• Appraise whatever the organization has provided, giving honest and frank feedback. If
the organization has not responded in depth to a P, R, or I item, briefly identify this as a
future opportunity for improvement (O or OO), explaining to the organization why
addressing it now is urgent or whether it could be scheduled for some future
improvement cycle. Organizations should address in depth at least 1/3 of the total P, R,
and I items in each Category, and should signal where they want detailed feedback by
the items to which they devote the most attention. If you think the organization’s focus is
wrong, or that they’ve given a Category too little attention, say so in your Category
Summary Statement.
19
• Give short responses to items where the organization’s response is short. AQIP
encourages an organization to focus its Systems Portfolios on those P, R, or I items
where it has something substantive to say about its processes, results, or improvement
cycles. For example, if it is developing and deploying strong and effective processes in a
particular area, the organization will describe them under the relevant P items. Similarly,
it will report performance results under the relevant R items, and actual cycles of
improvement under the I items.
• If there is nothing to report, AQIP urges organizations to say so, clearly and concisely,
ideally in short comments like “No reportable information is currently available about how
we set our learning goals for undergraduates. We recognize stabilizing the design of this
process represents an major opportunity for educational improvement.” In turn, the
Systems Appraisal team may acknowledge that the organization is aware of a gap in this
area, and consequently that the item represents an opportunity for improvement.
Extensive feedback comments may not be needed in such cases.
• In its feedback report, the team should designate a “no reportable…” item with an O
symbol (indicating it represents an “opportunity for improvement”) and merely state “No
reportable information from the institution at this time.” If the team believes that the
organization’s lack of attention (or progress) in an area is critical, it should promote the O
to an OO, and concisely explain why continuing to ignore this item would be perilous for
the organization.
• Give the organization feedback on performance measurement in every Category. If an
organization provides data for Result items, comment on whether its performance is
“good enough” and suggest how it can set stretch targets for future performance. If an
organization provides no in-depth responses to any of a Category’s R items, then it must
provide an in-depth response to the final P item, the one that asks what measures it
uses or plans to use to measure its performance in that Category. If this response is
weak, suggest how it might articulate the goals and objectives of various processes in
that Category, and possible measures of the performance of key processes.
• Don’t copy groups of sentences or paragraphs directly from the Systems Portfolio into
your report. The organization already has a copy of its Portfolio — it wrote it! Quoting
short phrases from the Portfolio, using organizational terminology, or citing tables and
illustrations by number all provide the organization with assurance that your team has
read their Portfolio carefully. Colleges and universities want to know your reaction and
counsel. But quoting at length what they’ve written back to them provides little value. Yet
be like a mirror — assure them you see and understand who they really are.
• Remember that O’s and OO’s are not faults, but Opportunities for improvement (or
Outstanding Opportunities). Often, an organization has not yet dealt with an area
because it hasn’t yet reached the level of development where that area is on its radar.
For example, organizations that haven’t yet measured their own performance in, say,
advising, aren’t at fault because they don’t compare their performance with others’ —
they simply haven’t yet reached the point where such comparisons would be possible —
or useful to them. Telling them that such situations represent opportunities for
improvement is not negative, but rather a way of coaching the organization to reach the
higher levels of excellence that it wants. Getting actionable feedback on how to improve
is a major reason why organizations join AQIP. (OO is pronounced “Oh….OOOH!” with
rising intonation, the way someone who finally understands with wonder the
20
opportunities ahead — not “uh-oh,” which is what you might say to someone who made
and error. For shorthand, call OO’s “outstanding opps.”)
• Do not contradict other comments found elsewhere in the feedback. Contradictions are
most likely to occur when an appraiser does not clearly specify the strength or
opportunity.
• Avoid prescriptions that simply say, in effect, “do this because we say to”; instead, tell
them why following your advice makes good sense. Don’t be afraid to suggest — even
to prescribe or proscribe — things an organization should consider doing (or stopping),
so long as you explain why doing (or refraining from) something would benefit (or hurt)
the organization. An organization is free to accept or reject your counsel, but will find it
difficult to reject a shrewd directive that includes an explanation of why obeying it is in
the organization’s self-interest.
• Avoid subjective judgmental terms (good, terrible, repulsive, interesting) that serve
primarily to describe your internal reaction to what the institution describes. Instead,
state your observation in a factual manner (e.g., customer satisfaction rates have
increased over the past three years) or communicate the reason for your positive or
negative reaction (e.g., continuing declines in enrollment are likely to produce serious
budget shortfalls).
• Do not refer to individuals by name in your comments or feedback, and be hesitant to
identify positions (VPAA) held by individuals as the subject for personal praise or blame.
Critical Characteristics Analysis (Draft, Proposed, and Feedback)

• Limit your draft worksheet to 1-2 pages.
• Use phrases rather than complete sentences.
• Begin each comment with the number of the item (O1 – O8) to which it refers
followed by a tab, then the comment itself:
O4 The institution clearly identifies its common learning goals for
baccalaureate graduates, thereby aligning all faculty members’ efforts in
helping students learn essential skills and content.
Independent Category Worksheets

Expect every member of the appraisal team to do an Independent Review of the Critical
Characteristics and all nine of the AQIP Categories in a Systems Portfolio. Every team member,
including the team leader, must review every Category. Although it might seem quicker and less
burdensome to have each Category reviewed by only a subset of the team, doing so would
seriously weaken the quality of feedback the organization receives. It would limit the number
and quality of ideas the team generates in identifying strengths and opportunities for
improvement, and make it difficult for some team members to participate in the consensus
conversation (on Categories they did not independently analyze), thus making the feedback
report something other than the consensus of the team’s views.
• Complete one worksheet for each Category.
• Limit each worksheet to 1-3 pages.
21
• Do not write comments responding to C (or Context) questions, which are included
so that you can place the institution’s response to other questions in the appropriate
institutional context
• Use a single, simple, complete thought per comment.
• Identify each comment you write as a SS (outstanding strength), S (strength), O
(improvement opportunity), or OO (outstanding improvement opportunity).
• Judge things as you see them: don’t try to balance the number of strengths and
improvement opportunities.
• Begin each comment with the number of the item (e.g., 3P2) followed by a tab, then
an identifier (SS, S, O, or OO), another tab, and then the comment itself:
3P2 SS The institution clearly identifies its common learning goals for
baccalaureate graduates, thereby aligning all faculty members’
efforts in helping students learn essential skills and content.
• Sort your comments into numerical order (within P, R, and I batches), and leave
space between comments.
The easiest way to do this is to follow the built in styles within the Systems Appraisal
Template. CC for Critical Characteristics comments and SAC for Systems Appraiser
comments.
Effective Comments for Results (R) Questions. Well-written Results comments frequently
address the following questions:
• Has the organization analyzed trends? Is the trend direction positive or negative? What is
the desirable direction (i.e., more or fewer)? Are explanations provided for significant
positive or negative changes?
• In each Category are data presented for measures of the key processes that were described
in the Process questions? How do the results from one Category link to other Categories
(e.g., strategic processes, supplier and partner relationships)?
• Are all important results presented? Are data focused on the critical organization
performance results (e.g., customer requirements, compliance with regulatory
requirements)? Are there any gaps in the data?
• Is the amount of data provided sufficient (e.g., number of cycles of data for trend data,
percentage of stakeholder population)?
• Are the data appropriately segmented?
• Do the data represent both short- and long-term priorities?
• How does the organization measure effectiveness, and are these measures presented?
• Are comparative data presented, and are they appropriate?
• What are the standard measures in this field? Is there any significance to the lack of any of
these measures in the Portfolio?
• Are the data normalized (presented in a way that takes into account the various size
factors)?
22
Consensus Category Worksheets

• Create a “consensus proposal” for each Category assigned to you by suggesting which
comments belong in your team’s feedback report, and providing proposed wording for
each comment.
• Begin your work on each Category by sorting the comments from a set of Independent
Category Worksheets into groups according to the questions on which they comment.
• If multiple comments deal with a common issue, select the most thoughtful one and use
it as a foundation upon which to build. Add ideas and words from similar comments to
this base.
• If several (3 or more) members of your team commented on the same issue, it probably
belongs in the consensus proposal. If only one person commented on an issue, include
it In consensus only if the comment has clear value and will benefit the institution.
• Even if only one or two team members suggest comments on an issue, include the
comment on your Consensus Category Worksheet. But if you propose that the team’s
final feedback omit this issue, recommend no symbol (SS, S, O, OO) for the comment.
Your worksheet “consensus proposal” in effect suggests that the team endorse only
those items for which you include symbols.
• In addition to following the advice for the Draft Category Worksheet (above), work to
craft consensus comments that are feedback ready (see below).
Strategic and Accreditation Issues Analysis (Draft, Proposed, and Feedback)

• Provide 3 – 5 Strategic Issues for every organization. Strategic issues are not dings or
faults — they constitute advice on where the team believes an organization would be
wise to invest its energies in the immediate future to continue (and accelerate) its quality
journey. These items should not repeat single SS, S, O, or OO comments, but rather
should summarize a few key themes that run through the items and Category analyses
you have done.
• A Strategic Issue might be complimentary — a suggestion that capitalizing on a set of
strengths (rather than addressing a set of opportunities) could move the institution in
beneficial ways. But observations like this typically occur in Category Comments, not as
Strategic Issues.
• Fit Strategic Issues to the organization current place and culture. Strategic issues are
typically broad, and can generate a variety of actions (action projects, initiatives, plans,
tasks, or undertakings). If we expect an organization to concentrate its efforts on a few
broad strategic issues, addressing them through a variety of projects, then it makes
sense to limit the number of issues we identify as its highest priorities. Three or four is
thus probably a sensible number. The rare organization that is highly practiced at
continuous improvement, with a culture that can generate and attack new projects easily
and complete them quickly, may be able to deal with more than three or four strategic
issues.
• Identify any accreditation issue that needs attention by specifically referring to one of the
Commission’s five Criteria for Accreditation, and to the relevant Core Component of that
Criterion. Accreditation-related issues should be serious, and imperil the institution’s
23
accreditation if left unaddressed. Do not associate a strategic issue with accreditation

simply to get the institution’s attention. Most colleges and universities are using AQIP to
maintain their accreditation, so staying accredited is of the utmost importance. Any
opportunity or weakness that might endanger accreditation automatically becomes a
major strategic issue. Moreover, accreditation issues are almost invariably critical issues
in their own right — issues about clarity of mission, educational purpose and
effectiveness, sufficiency and effective allocation of resources, and similar core
concerns. An appraisal team that perceives a central issue of this sort needs to place it
high on the organization’s agenda. The team should explain how the issue might
endanger the organization’s accredited status, and why the substance of the issue
matters — how ignoring it would handicap the organization’s ability to accomplish its
mission, serve its constituents, and, perhaps, maintain its vitality and integrity. In your
feedback, use letters to identify strategic and accreditation issues. Put A before any
issue that jeopardizes accreditation (even if it is also of strategic importance), and tie it to
a specific Criterion and Core Component.
• An accreditation-related issue can be raised either (a) because there is evidence in the
Systems Portfolio that the institution is on a course that may result in its not meeting a
Criterion, or (b) because the evidence that the institution meets every Core Component
of all five Criteria is missing or insufficient. In the latter case, alerting the institution to
gaps in evidence provides it with an opportunity to submit additional evidence before its
Reaffirmation of Accreditation.
• Have the team leader talk with AQIP’s Systems Appraisal Process Manager — before
you submit your final Appraisal — if your team identifies a serious accreditation concern
that is a “sin of commission” (rather than an instance where the organization has simply
not yet provided enough evidence in its Portfolio to substantiate that it meets every Core
Component). The current Systems Appraisal Process Manager’s name and contact
information is listed on the AQIP website under Staff Contacts (www.AQIP.org/Staff).
• Label strategic and accreditation issues distinctively (e.g. S1, S2, S3, A1, A2, etc.) so
the institution can distinguish which is which.
• Limit the worksheet to 2-3 pages.
Writing Feedback Ready Comments

Comments that can easily be integrated with others make the team leader’s job easier, and
make the feedback report more valuable to the organization. Feedback ready comments:
• are in complete sentences, frequently containing a subject and a verb “lifted” from the
Categories question. For example,
o 5P5 asks, “How does communication occur between and among institutional
levels?” and a comment might respond, “Communication between administration
and faculty occurs when…”
o 6P1 asks, “How do you identify the support service needs of your students?” and
a comment might respond, “Each department identifies its students’ support
service needs through a monthly…”
[Italics are used in these examples only to emphasize the subject and verb lifted from
the question.]
24
• mention specific examples from the organization’s Systems Portfolio

• cite diagram, table, or figure numbers from the organization’s Systems Portfolio
• link to key factors that appear in the Organizational Overview section of the Portfolio
(and are likely noted in the team’s Critical Characteristics Summary)
• follow content and style guidelines
• include “So what’s”
• consist of 1-3 complete, actionable, nonprescriptive sentences
• cite specific examples from the Portfolio
• link to the organization's Critical Characteristics as captured in your team’s Critical
Characteristics Summary
• mention, as appropriate, AQIP’s Principles of High Performance Organizations (e.g.,
Learning, Collaboration, Agility, Integrity, etc.) as values that inform or reflect institutional
activities and strategies
• meet both the content and style requirements of the Comment Guidelines
Writing Actionable Comments

The Systems Appraisal’s goal is to give the organization feedback it can use to (a) take
actions that will help it sustain the good things it is already doing, and (b) take actions to
improve the things it could do better. In both situations, the stress should be on action. Without
being prescriptive, you need to give advice and feedback that the institution can translate into a
course of action.
• In citing a strength, say or illustrate (with an example) what it has accomplished for the
organization — the “so what?”
• Suggest how strengths could become more valuable by deployment throughout the
organization.
• Explain the “so what?’ by linking a strength with one of the AQIP Principles of High
Performance Organizations, and suggest how the strength can be leveraged into
organizational culture change.
• For opportunities for improvement, explain what positive consequences could happen (or
what negatives could cease happening) if the opportunity was exploited — the “so what?”
• Note the costs of not seizing an opportunity for improvement — particularly the cost of
standing still while others (the competition) is moving on improvement.
Style Guidelines
• Use the full name of the organization or its own shortened name to refer to the organization.
For variety, use the institution or the organization as synonyms.
• Echo the organization's terminology whenever appropriate. Make it clear that you have read
its Systems Portfolio and are writing in reaction to what it says.
• Keep your tone polite, professional, and positive.
• If something is unclear, explain what is missing and why its omission is important.
25
• Focus on the organization's substantive strengths or improvement opportunities, not its

writing style or graphics. Do not comment on writing errors or style (e.g., the Portfolio needs
a good proofreading or writing overuses the passive voice) in your feedback (although you
should alert AQIP staff if poor writing becomes an obstacle to your appraisal).
• If there are problems in understanding graphs, tables, or figures, write a single comment
summarizing the difficulties encountered (e.g., variables in tables are presented
inconsistently, making it impossible to distinguish counts from percentages or readers
cannot determine which direction represents improvement in line graph figure 2-3).
• Identify strengths or improvement opportunities according to where the issue falls in the
Categories, not by where the organization places the information in its Portfolio. For
example, if an institution discusses its faculty’s needs under Category 3 (rather than
Category 4, where the discussion should occur), the appraiser should read this and include
it in responding to Category 4.
• Use vocabulary and phrases from the Categories, Principles of High Performance
Organizations, and Category Summary Statement Guidelines in your feedback comments.
• Avoid jargon and acronyms, unless they are used by the organization.
• When referring to a figure, table, or chart, provide the figure number used in the Portfolio.
Grammatical Considerations
Focus on the substance of the comments you write, keeping the following considerations in
mind. This will save time and effort in editing and writing the final feedback report. Examples
are in italics type.
All students take a writing assessment

annually, and the university summarizes
these results each year, but the institution
When possible, use the present
collects no comparative results for
Tense tense of all verbs throughout the
achievement in student writing, making it
feedback.
difficult for faculty to assess program
effectiveness relative to that of competing
institutions.
Prefer active voice to passive. Active voice: The Director of Human
With active voice, the subject of Resources completes the survey process
the sentence names the actor. for assessing faculty satisfaction each
Active Voice With passive voice, the subject of May.
the sentence is the noun acted Passive voice: The survey process for
upon, and the actual actor is assessing faculty satisfaction is completed
often not explicitly named. each May.
26
Spell out all abbreviations and

acronyms the first time they are Human Resource (HR) managers are
used and indicate the responsible for leading the Dewey
Acronyms abbreviation in parentheses. If University Action Projects (DUAPS).
you use a term only once, do not
abbreviate or substitute an
acronym.
Spell out numbers from one There were five union workers in ten
through ten. If a number is used meetings. Eleven meetings are held
as the first word of a sentence, throughout the quarter. The new library
spell it out. Otherwise, use holds 200,000 more books than the facility
Numbers figures for numbers above ten. it replaced.
When numbers greater and less
than ten occur in the same The program has graduated only 15
sentence, use figures for the students in the last 8 years.
numbers.
Refer to an organization,
The institution rewards its faculty for
institution, board, or committee
attention to students. The Faculty Senate
as "it," not "their." Don’t let the
has not revised its operating principles in
Agreement fact that a singular body includes
15 years. Some faculty members are
many people confuse your
unhappy, but, as a whole, the Faculty is
selection of the right pronoun or
satisfied with the situation.
verb.
Follow the organization's own
conventions for capitalizing Director of Human Resources, Data
organizational divisions, Process Improvement Team, English and
departments, staff titles, and the Physics faculty, the new library, the
names of teams and internal President’s Office, etc.
processes.
Capitalization
Do not capitalize "the
organization" or "the institution."
The institution began two Action Projects
Capitalize references to the AQIP
relating to AQIP Category 1, “Helping
Categories, their names (e.g.,
Students Learn,” in 2002-03.
“Helping Students Learn”), and
AQIP processes.
Capitalize “Criteria” or “Criterion”
when referring to the The college meets the Commission’s first
Commission’s Criteria for Criterion, in part, because of its strong
Capitalization Accreditation, and capitalize processes for addressing AQIP Category
“Category” or “Categories” when 3, Understanding Students’ and Other
referring to the AQIP’s nine Stakeholders’ Needs.
Categories.
27
Strategy Forum, Action Projects, Annual

Capitalize references to specific Updates, Reaffirmation of Accreditation,
AQIP processes or activities. Organizational Overview, Systems
Portfolio
Awkward: Each staff member is
Avoid the awkwardness of he or responsible for his or her own professional
he/she & she, he/she, (s)he, his and her, development.
(s)he him and her by making nouns
plural. Better: All staff members are responsible
for their own professional development.
The chairs of departments make special
Sexist Avoid sexist terms like chairman,
efforts to include faculty spouses in social
language faculty wives, etc.
activities.
Put a comma before the
No federal, state, or local sanctions have
Items in conjunction (and, or, or nor) when
been imposed against the organization for
series three or more items are listed in a
the past five years.
series.
Figure In Figure references, spell out the
Write Figure 1.1-1, not Fig. 1.1-1.
References word "Figure."
Frequently Asked Questions for Systems Appraisers

I don’t understand what I should be doing next in the Systems Appraisal. Who can help
me?
Your team leader is the first person to call when you have questions, but the AQIP Systems
Appraisal Process Manager is also a source of information for Systems Appraisal Team
members and leaders. The name and contact information of the current process manager is
available on the AQIP website at www.AQIP.org.
Must every member of the appraisal team do an Independent Review of all nine of the
AQIP Categories in a Systems Portfolio? Wouldn’t it be more efficient to have two or
three team members review each Category?
Every team member, including the team leader, must review every Category. Although it might
be quicker and less burdensome to have each Category reviewed by only a subset of the team,
doing so would seriously weaken the quality of feedback the organization receives. It might limit
the number and quality of ideas the team generates in identifying strengths and opportunities for
improvement, and it might make it difficult for some team members to participate in the
consensus conversation (on Categories they did not independently analyze), thus making the
feedback report something other than the consensus of the team’s views.
Should our team limit the number of comments we provide to the organization under
each AQIP Category? Should we balance the number of comments pointing out
strengths (S, SS) with those pointing out opportunities for improvement (O, OO)?
Include in your feedback all comments that the team agrees on that respond to the Systems
Portfolio. Typically team members in their Independent Review will generate 10 - 15 comments
for Category 1 and about 5 - 10 comments for Categories 2 - 9. (Systems Portfolios usually
28
devote about twice as much attention to Category 1 as to any other Category.) After the
consensus review, about the same number of agreed-upon comments will remain for each
Category, but usually the consensus comments themselves will be fuller, particularly those that
represent opportunities for improvement (O, OO).
If you arbitrarily limit the number of your comments, two problems occur. First, the insights and
creativity of the team members is stifled in their Independent Reviews. Second, the organization
is prevented from getting the full range of consensus ideas that it deserves and needs to
maximize its effectiveness.
Don’t worry about the ratio of strengths and opportunities. Call ‘em as you see ‘em, honestly
and candidly. Typically, higher Category Summary Statement scores on a Category (5, 6, or 7)
go to organizations whose strengths greatly outnumber their opportunities for improvement, but
Category Summary Statements depend heavily on the significance of the particular strengths or
opportunities identified. One OO comment might outweigh several S comments.
AQIP Category 7 is about data and measurement, but there are also questions about
measures and results in each of the Categories. Where should we expect a college or
university to talk about these issues? Where should we comment about measurement,
information, or data?
AQIP expects that colleges and universities will specify, under each of the nine Categories, the
ways that they measure the processes and goals for that Category. Thus, the Systems Portfolio
section on Category 1 will explain how the organization measures student learning, and will
present performance data that these measures produce. The section on Category 5 will explain
how the organization measures and evaluates leadership and communication, and present the
results. There should be an array of measures and results for each of the nine Categories. The
appraisal team needs to point out any gaps — missing measures or missing results — and
explain why their absence is a problem, or how plugging the gaps can stimulate improvement.
Category 7 asks about the organization’s processes for selecting, gathering, storing, retrieving,
and using data and information — of any kind. Where the other Categories ask about specific
measures related to the processes that fall under each Category, Category 7 asks about the
processes for deciding what information to collect and what not to collect. Category 7 asks
about the security, currency, accuracy, and availability of stored data. It asks how the
organization improves its data and information collection and retrieval processes. Like the other
Categories, a college or university should have metrics for Category 7 (e.g., total cost of data
and information storage, satisfaction of employees with data they need, frequency and extent of
the use of information, and similar measures) and performance results data for these measures.
How many strategic issues should the appraisal team identify for a college or university?
Strategic issues are typically broad, and can generate a variety of actions (action projects,
initiatives, plans, tasks, or undertakings). If we expect an organization to concentrate its efforts
on a few broad strategic issues, addressing them through a variety of projects, then it makes
sense to limit the number of issues we identify as its highest priorities. Three or four is thus
probably a sensible number. The rare organization that is highly practiced at continuous
improvement, with a culture that can generate and attack new projects easily and complete
them quickly, may be able to deal with more than three or four strategic issues.
Similarly, too many OO items (or too few) may make it difficult for an institution to prioritize what
is most important in the feedback. Reserve OO for those things that need immediate attention,
typically those connected with a Strategic or Accreditation issue.
29
How should we relate strategic issues and accreditation issues?

Most colleges and universities are using AQIP to maintain their accreditation, so staying
accredited is of the utmost importance. Any opportunity or weakness that might endanger
accreditation, automatically becomes a major strategic issue. Moreover, accreditation issues are
almost invariably critical issues in their own right — issues about clarity of mission, educational
purpose and effectiveness, sufficiency and effective allocation of resources, and similar core
concerns.
An appraisal team that perceives a central issue of this sort needs to place it high on the
organization’s agenda. The team should explain how the issue might endanger the
organization’s accredited status, and why the substance of the issue matters — how ignoring it
would handicap the organization’s ability to accomplish its mission, serve its constituents, and,
perhaps, maintain its vitality and integrity.
In the feedback on strategic and accreditation issues, use letters (similar to using SS, S, O, and
OO) to distinguish issues. Put “A” before any issue that jeopardizes accreditation (even if it is
also of strategic importance), and “S” before an issue the team believes is strategic but not
endangering accreditation.
We’ve identified an issue we think is very serious for this organization. In order to
impress the leadership and faculty with the need to take action, should we identify it as
an “accreditation issue” instead of simply calling it “strategic?”
Accreditation is based on meeting the Commission’s five Criteria for Accreditation, and the
twenty-one Core Components under those Criteria. Identify as “accreditation” only issues that
truly endanger an organization’s ability to demonstrate it meets the Commission’s Criteria for
Accreditation. Remember: accreditation standards are a floor that must be met by all accredited
colleges and universities, and represent minimum requirements only. Organizations serious
about quality and continuous improvement should hold themselves to far higher and more
demanding expectations than those embodied in accreditation criteria, and so should an AQIP
Systems Appraisal team. Therefore, it is quite legitimate for your appraisal team to identify for
a college or university a strategic weakness that traditional accrediting processes would ignore.
To participate in AQIP, an organization must meet all of the regular accreditation standards. But
AQIP expects an organization serious about quality to do far more than just meet minimum
standards.
If your team identifies a serious accreditation concern that is a “sin of commission” (rather than
an instance where the institution has simply not yet provided enough evidence in its Portfolio to
substantiate that it meets every Core Component, your team leader should call the AQIP staff
immediately — before you submit the final Appraisal. Call Steve Spangehl at 800-621-7440,
x106, or, if you can’t reach Steve, call another staff member and ask them to have Steve call
your team leader directly.
What might be an example of a strategic issue that is not also an accrediting issue?
To take just one example, few accreditation agencies have standards that require a formal
process for leadership succession — a thoughtful, proactive approach to developing new
leadership talent, nurturing or mentoring future leaders. Yet quality organizations typically take
leadership succession and continuity very seriously, and work hard to address these areas with
effectives processes and plans. If your team sees an important concern like this, don’t hesitate
to identify it as a strategic issue for the organization. But resist associating it with accreditation
30
simply to “get the organization’s attention.” Instead, get the appropriate attention by explaining
in your comment why the issue is critical and what might happen if it continues to be ignored.
AQIP places demands on organizations that go far beyond accreditation requirements. AQIP
expects colleges and universities to gather comparative data and use it to set targets for
performance, and to identify best practices or benchmarks to study to learn how to improve
specific processes. A failing here would be a serious strategic issue, but unrelated to any
accrediting requirement contained in the Criteria for Accreditation.
Accrediting standards are focused on specific key areas. There are many, many issues of
serious concern to colleges and universities that accreditors do not address at all. AQIP wants
these issues “on the table” if they are serious, and if addressing them can help an organization
better meet and serve the needs of its stakeholders.
If your team identifies an accrediting issue, the team leader should speak with Steve Spangehl,
AQIP Director (at 800-621-7440, ext. 106, or sds@hlcommission.org), either before the
Consensus Conversation at which the issue will be discussed or (if the issue emerges as the
result of a Consensus Conversation) immediately afterwards. Issues touching accreditation are
serious, both for institutions and the Commission, and often require immediate action by the
AQIP staff.
As I was working on my Independent Review, I discovered there was critical information
missing from the Portfolio that I need to provide the organization helpful advice. Can I
call them to get clarification on how they currently do things?
No! It is critical that appraisers not contact the organization during or after the appraisal. AQIP
does not reveal the identity of specific appraisers to colleges and universities; by maintaining
confidentiality on the identity of reviewers, AQIP forces the organization to grapple with the
reviewers’ feedback, preventing the organization from engaging in the tempting practice of
dismissing advice or criticism from reviewers whom organizations might reject as “people not
like us” or “those who can’t appreciate our position”. Often it is just those reviewers with the
freshest perspectives who make the most insightful and helpful comments.
If you think critical information is missing, write a comment explaining what is missing, why its
absence is critical, and what the consequences of omitting (or including) it might be. If the
missing material makes it impossible to determine whether the organization meets accreditation
criteria, say so: the organization will have ample opportunity, before the Commission next
judges its accreditability, to fill the gap, but only if you tell it the gaps exists.
Finally, if you believe that a review is impossible without specific additional materials, contact
your team leader. Under certain conditions, team leaders can contact colleges and universities
to request specific additional information addressing specific questions. This is most likely to
happen when a team lacks the information it needs to agree on a Critical Characteristics
Analysis of the organization, a task which must be completed before the Independent Review of
Categories can begin. For example, if nothing in the Portfolio indicated the size, enrollment, or
total budget of the organization, the team might direct the team leader to call the organization
and get this vital information before proceeding to the rest of the Appraisal.
How does AQIP avoid conflict-of-interest situations in the System Appraisal process?
Before it makes assignments to Systems Appraisal teams, AQIP asks all appraisers to identify
those colleges and universities they could not fairly and objectively review. Organizations are
also given the roster of all AQIP Systems Appraisers, and are asked to identify any the
31
organization believes could not review it objectively. Thus, when teams are formed, no conflict
of interest situations should arise.
However, it sometimes happens that an appraiser does not become conscious of a conflict of
interest until he or she reads the organization’s Systems Portfolio. Because of this possibility, it
is critical that appraisers read (skim thoroughly) the entire Portfolio the moment it arrives, and
call AQIP immediately if a conflict-of-interest becomes apparent. This foresight can prevent the
disruption caused by replacing an appraiser midway through the process. However, any
appraiser who comes to recognize a conflict of interest, no matter when the realization occurs,
should contact AQIP immediately.
Whom should I call if I feel the Systems Appraisal is off-course: behind schedule,
improperly organized or conducted, or seriously flawed in some other way?
Contact the Systems Appraisal Process Manager (current manager’s name and contact
information available in the Contact AQIP Staff section of the AQIP website at www.AQIP.org);
Telephone AQIP at 800-621-7440, ask for Steve Spangehl, Director (x106) or Babatunde
Alokolaro, AQIP Program Facilitator (x109), and explain concisely and frankly whatever
concerns you. (Alternatively, email both sspangehl@hlcommission.org and
balokolaro@hlcommission.org.)
32

Systems Appraiser Guide: Academic Quality Improvement Program

Diunggah oleh

Informasi Dokumen

Deskripsi Asli:

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Systems Appraiser Guide: Academic Quality Improvement Program

Diunggah oleh

Hak Cipta:

Format Tersedia

Systems Appraiser Guide

Valuable Information for Systems Appraisers

Academic Quality Improvement Program

What’s New in the Systems Appraiser Guide:

• Using the institution's Self-Evaluation in preparing feedback, page 17.

The Systems Appraisal Process

Systems Appraisal Glossary

Identifying Accreditation Issues in the Systems Appraisal

Dealing with Systems Portfolio Gaps

In its Systems Portfolio, an institution addresses at length a subset of P, R, and I items

Where evidence is likely to appear in a Systems Portfolio that an institution is

1. Helping Students Learn x x x

Category Summary Statements

o identifying processes that need to be designed

Systems Appraisal Worksheets

Instructions for Completing the AQIP Systems Appraisal Worksheets

AI This is a sample of an accreditation issue as it might appear on worksheets 5, 7, or 9.

1P2 OO This is a sample of a comment on a single Category item. Most appraiser

1P3-1P6 SS This is a sample of a comment on a range of adjacent Category items

Chart showing flow among the nine worksheets

Steps in the Systems Appraisal Process

Systems Appraisal 2: Steps in the Consensus Review

Systems Appraisal 3: Steps in the Consensus Conversation

the discussion of each Category should

Systems Appraisal Timetable

Writing Feedback in Appraising an AQIP Systems Portfolio

Critical Characteristics Analysis (Draft, Proposed, and Feedback)

Independent Category Worksheets

Consensus Category Worksheets

Strategic and Accreditation Issues Analysis (Draft, Proposed, and Feedback)

accreditation if left unaddressed. Do not associate a strategic issue with accreditation

Writing Feedback Ready Comments

• mention specific examples from the organization’s Systems Portfolio

Writing Actionable Comments

• Focus on the organization's substantive strengths or improvement opportunities, not its

All students take a writing assessment

Spell out all abbreviations and

Strategy Forum, Action Projects, Annual

Frequently Asked Questions for Systems Appraisers

How should we relate strategic issues and accreditation issues?

Anda mungkin juga menyukai