Operations Guide - Support and Maintenance

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance
Operations Guide:
Support and Maintenance
Table of Contents
Introduction ............................................................................................................................... 1
Support....................................................................................................................................... 2
Structure ........................................................................................................................................................................... 2
Support Tools.................................................................................................................................................................. 5
Testing and Change Control ..................................................................................................... 8

Testing Phases ................................................................................................................................................................. 8
Change Implementation Process................................................................................................................................ 12
Ongoing Operations ................................................................................................................ 13

XenDesktop Operations.............................................................................................................................................. 13
XenApp Operations ..................................................................................................................................................... 17
PVS Operations ............................................................................................................................................................ 20
Appendix.................................................................................................................................. 25
Appendix A XenApp Delegated Rights.................................................................................................................. 25
Appendix B XenDesktop Delegated Rights ......................................................................................................... 27
Appendix C Provisioning Services Delegated Rights .......................................................................................... 27
Appendix D XenServer Delegated Rights ............................................................................................................ 27
Appendix E Sample Call Script/Questionnaire ................................................................................................... 28
ii
Introduction
When implementing Citrix environments, support and maintenance aspects for new farms often get
overlooked. Effectively maintaining a Citrix environment necessitates reliable systems be in place to
ensure smooth day to day operations. This document covers main duties involved in maintaining of
Citrix infrastructures.
This white paper covers the following 3 sections:
Support When problems arise, technical support often is the first point of contact
for issue resolution. This section addresses the proper staffing, organization,
training, and tools utilized in effective support organizations.
Testing and Change Control Regular upgrades are required to ensure a farm
environment is up to date. Change management processes are critical to ensure
improvements are properly approved, tested, and validated by appropriate parties.
This section covers the proper processes that ensure changes in production
environments are deliberate, proven, and accountable.
Ongoing Operations Maintenance, issue prevention and resolution are core

responsibilities in running a Citrix infrastructure. When the responsibilities and
assignments are structured properly, friction is kept to a minimum, reducing issues
and their resolution times. This section discusses routine operations that Citrix
environments require for optimal performance.
Support
Well trained support members with a clear organizational structure are critical for an effective
support team. Since technical support is the first place many users go, thought needs to be given to
its structure, training, and tools utilized by support members.
Structure
Multi-tier support levels have been found to be the most effective means in addressing user
support issues. Low criticality, low complexity or frequently occurring issues should be
managed and resolved at the lower tiers. High criticality and complex issues are escalated to
more experienced architects or infrastructure owners. The diagram below outlines a
common multi-tier support structure:
The first level should resolve 75% of all issues encountered. These issues include routine
problems requiring limited knowledge of the Citrix environment. Issues are quickly resolved
and may also be automated, like password resets. When non-routine problems exceeding the
abilities of level ones are encountered, they are then escalated to level two. Information on
2
the end users problem and attempted troubleshooting steps are documented at the first level
allowing level two technicians to immediately begin addressing the problem. Level two
technicians should handle only 20% of the support tickets and are highly knowledgeable on
the Citrix environments. Should a complex issue arise that exceeds level twos abilities it is
escalated to the third, and final active support tier level three. Level three issues are
complicated and often mission critical requiring expert knowledge of the Citrix environment.
Tier three support tickets should be uncommon, amount to no more than 5% of all support
issues, and be resolved as soon as possible.
The final tier, level four, is not involved in active support of a production environment. It is
focus solely on the strategic improvements for enterprise environment, testing new
technologies, planning migrations, and other high level changes.
Should support discover an issue is related to an application or underlying infrastructure the
ticket is handed to the appropriate team for troubleshooting. If a program bug is discovered
the issue is then re-escalated and a ticket is established with the programs vendor.
In parallel with the active support structure, self-service portals are available for end users to
assist themselves with basic troubleshooting tasks. This assists in resolving frequently
encountered issues and helps prevent the creation of new support tickets.
Table 1 below covers the levels and expectations in additional detail:

Support Level
Level 1
(Call Entry)
Description
Responsibilities
Provide first-line support of reported

issues. Initially, servicing support
messages and phone calls. This level
needs to perform initial issue analysis,
problem definition, ticket routing, and
simple issue resolution. Additionally
can handle requests for application
access or support with configuring
plugins.
Perform issue definition, initial analysis

and basic issue resolution
Perform initial troubleshooting to
determine the nature of the issue
Collect basic user and session
information
Create ticket and log all
troubleshooting steps performed
Resolve basic Citrix related issues,
connectivity problems and application
related issues using existing Wiki
articles
Escalate issue to Tier-2 if it requires
advanced skills or elevated permissions
If the issue is related to particular
applications or other technologies and
not the Citrix infrastructure, escalate
the issue to the appropriate application
or technology support units
If the issue is deemed important
enough, escalate directly to Tier-3
Generate requests for additional issue
resolution guides as necessary
Skill Set
Tier-1 support personnel should be
provided with basic training on Citrix
XenApp, Citrix XenDesktop and
supporting technologies. This can
include internal training from subject
matter experts or formal training. The
training provided should focus on the
following topics:
High-Level overview of the XenApp
and XenDesktop implementation
Using Citrix Desktop Director to
manage user sessions
Troubleshooting Citrix XenApp and
XenDesktop sessions
Managing AD group policies user
profiles
Troubleshooting methodology
In addition, regular training should be
provided to the Tier-1 team members
on the latest troubleshooting
recommendations from the Tier-2 and
Tier-3 teams as well as details on any
relevant changes to the environment.
This will help ensure a base knowledge
level amongst the team and consistent
customer service.

Support Level
Description
Responsibilities
Skill Set
Level 2
(Production
Support Engineer)
Primarily supporting day-to-day

operations of virtual desktops
environment, may include proactive
monitoring and management. In
addition, this role would also perform
advanced troubleshooting and utilize
available monitoring / advanced
troubleshooting tools. Assist with
resolving issues escalated by Level One
Support.
Perform intermediate issue analysis and

resolution
Identify root cause of issues
Respond to server alerts and system
outages
Review vendor knowledge base articles
Respond to out-of-hours helpdesk calls
Respond to critical monitoring alerts
Generate internal Wiki articles and
issue resolution scripts and maintain
Tier-1 troubleshooting workflows
Perform basic server maintenance and
operational procedures
Manage user profiles and data
Escalate ticket to Tier-3 or appropriate
technology owner if advanced skills or
elevated permissions are required
Generate requests for additional issue
resolution scripts and Wiki articles as
necessary
All members of the Tier-2 team should

achieve the Citrix Certified
Administrator (CCA) certification for
both XenApp and XenDesktop. In
addition, basic and advanced training on
Windows concepts will be essential for
team members who do not have
desktop or server support experience.
Finally, on-the-job training along with
close integration with Tier-3
administrators is essential as the Tier-2
roles are formalized and responsibilities
are handed over from Tier-3 to Tier-2.
In addition, regular training should be
delivered to the Tier-2 team to maintain
competencies on technology and
architectural updates. This will ensure a
baseline knowledge level across the
team.

Support Level
Level 3
(Build Engineer)
Description
Responsibilities
Skill Set
Central point for implementing,

administering and maintaining Citrix
desktop and application virtualization
infrastructure. This role focuses on
deploying new use cases and leading
lifecycle management initiatives.
Generally, one Build Engineer could
focus on one use-case at a time. For
example, three new concurrent use
cases would require three Build
Engineers. Escalates issues to software
vendor specific Technical Support and
notifies Level 4 about this issue.
Perform advanced issue analysis and

resolution
Perform maintenance and environment
upgrades
Addressing high severity issues and
service outages
Manage the Citrix environment
Oversee and lead administrative tasks
at the Tier-2 level
Manage network and storage
infrastructure as it relates to the Citrix
environment
Review periodic reports of server
health, resource usage, user experience,
and overall environment performance
Review vendor knowledge base articles
and newly released updates
Perform policy-level changes and make
Active Directory updates
Review change control requests that
impact the Citrix environment
Perform advanced server and
infrastructure maintenance
Review Wiki articles and issue
resolution scripts for accuracy,
compliance, and feasibility
Create Wiki articles and issue resolution
scripts to address Tier-2 requests
Tier-3 support team members with a

minimum of three years experience
implementing and supporting XenApp,
XenDesktop, Provisioning Services and
Windows operating systems.
The Tier-3 team should have enterprise
experience with industry recognized
certifications. The Tier-3 administrators
should complete the Citrix Certified
Enterprise Engineer certification track
as this will prepare them to proactively
manage the user community and the
Citrix solution. Citrix Certified
Integration Architect (CCIA) level
certifications would be optimal at this
level.

Support Level
Level 4
(Architect)
Vendor Support
Self Service
Description
The Level 4 team has minimal
exposure to administrative tasks but
focuses on translating business
requirements into technical
architectures, designing the
infrastructure or planning migrations.
Not involved in day-to-day support.
Responsibilities
Provide technical leadership for
upcoming projects
Lead design updates and architecture
revisions
Address high severity issues and service
outages
Oversee technology integration
workflows
Review periodic reports of server
health, resource usage, user experience,
and overall environment performance
to determine next steps and upgrade
paths
Review change control requests which
impact the Citrix environment
Review frequently recurring helpdesk
issues
Ensure technical specifications
continue to meet business needs
Update design documentation
Skill Set
A qualified Tier-4 resource should have

a minimum of five to six years of
experience implementing, supporting,
and serving in a technology architect
role for a XenApp and/or XenDesktop
environment as well as additional
administrative experience with
integrated technologies such as
application and profile management
solutions. The ideal candidate will have
served in such a capacity at two or more
environments for purposes of product
exposure and in at least one
environment of over 1,200 concurrent
users.
The Tier-4 resource should have
enterprise experience with industry
recognized certifications. A Citrix
Certified Integration Architect (CCIA)
certification or comparable training and
experience should be a prerequisite of
the role. Training or practical
knowledge on XenCenter, AD Group
Policies and the utilized hypervisor
should be considered essential.
Vendor assistance may be necessary should defects in a program be discovered. At this stage level 3 engineers need to establish
a support tickets with the appropriate vendor to assist with finding a solution.
A self-service portal is utilized for noncritical tasks such as application access, permissions, etc. The portal can range from a
simple FAQ page to a fully automated process requiring no human interaction. The purpose of the self-service portal is to add
an additional touch point for end users to address basic issues, preventing the creation of new support tickets.
Support Tools
Providing the appropriate tools for the aforementioned support levels is a key aspect in
order to achieve a high First-Time-Fix-Rate and a low Time-to-Resolution. The table below
highlights several tools available for supporting Citrix environments.
Tools\Processes
Tools
Details
Ticket Management
System
Call Scripts
Shadowing Tool
Knowledge Base
Citrix Administrative
Consoles
Used to document customer information and issues. Help Desk

should also document date and time of occurrence, the application
and module, steps to repeat error, obtains screenshots as needed.
The first contact help desk personnel should have documented scripts
or triage guides to ensure that all relevant data is captured while the
user is on the phone. This practice also assists in proper triage and
allows the next support tier to perform research off-line and review
logs prior to customer contact.
Shadowing users is a useful way for troubleshooting new issues. This
allows for support technicians and administrators to remotely observe
a users actions.
Documentation should be created and maintained in a knowledge
base or library of known issues. Articles should be searchable for
quick recovery. Knowledge bases help quickly resolve issues upon
reoccurrences and reduce the need to perform future research.
The incident management team will have access to the Citrix
Administrative Consoles for focused troubleshooting. Citrix
recommends that the Help Desk also have same level of access. The
initial support tier will primarily focus on resetting connections and
obtaining data for escalation.
The diagram below outlines the recommended tools for every support level:
Tool
Desktop
Director
Desktop
Studio
Product
Description
Desktop Director provides an
overview of XenDesktop hosted
desktops and XenApp sessions. It
enables support teams to perform
basic maintenance tasks and to
monitor and troubleshoot system
issues.
Desktop Studio enables
administrators to perform
configuration as well as
5
XD
XA
PVS
Support Level
XS
L1
L2
L3
Tool
AppCenter
HDX Monitor
Provisioning
Services
Console
XenCenter
License
Administratio
n Console
Product
Description
XD
maintenance tasks for a

XenDesktop site and associated
virtual desktops.
AppCenter provides support teams
the ability to monitor and
troubleshoot user sessions as well
as server systems within a XenApp
farm. In addition AppCenter
enables administrators to perform
configuration and maintenance
tasks for a XenApp farm.
HDX Monitor is a tool to validate
the operation of the Citrix
ICA/HDX stack of a user session.
Hereby HDX Monitor provides
information about client
capabilities, network performance
/ activity, session settings and
many more items.
The Provisioning Services Console
enables administrators to perform
configuration and maintenance
tasks for a Provisioning Services
farm.
XenCenter enables administrators
to perform configuration and
maintenance tasks for a XenServer
Resource Pool.
The License Administration
Console is the interface required to
manage license files and a license
server or appliance.
XA
PVS
Support Level
XS
L1
L2
L3
Please refer to the following appendices for details on how delegated rights could be
configured for the support staff
Appendix A (XenApp)
Appendix B (XenDesktop)
Appendix C (Provisioning Services)
In addition to the aforementioned Citrix support tools, every support organization should
leverage the following instruments:
Ticket Management System: It is vital for support organizations to implement a

Ticket Management System for tracking and escalating support calls (in between the
levels) as well as to document incident and customer related information.
Knowledge Base / Wiki: Ever support organization should document and

maintain a knowledge base or library of known issues. The articles should be
searchable for quick recovery. Knowledge bases help quickly resolve issues upon
reoccurrences and reduce the need to perform future research.
Call Script: The first contact help desk personnel should have documented scripts
or triage guides to ensure that all relevant data is captured while the user is on the
phone. This practice also assists in proper triage and allows the next support tier to
perform research off-line and review logs prior to customer contact. Please refer to
Appendix E Sample Call Script / Questionnaire for a sample call script.
Staffing Requirements
This section provides a guideline for the number of support staff required for several sizes
of Citrix XenApp or XenDesktop enterprises. This information is primarily based on Citrix
experience in deploying enterprise environments, taking into account the most typical design
requirements. Additional refinement may be necessary to optimize the support of the
environment based on specific business requirements.
XenApp centric organization
Farm or Load Managed
Groups
1
2-5
>6
#Servers
<100
<200
>200
#Users
<3000
<4000
>5000
Help Desk
1 CCA
3 CCA
3 CCEE
Admins
1-2 CCEE
3-4 CCEE
5+ CCEE
Architects
1 CCIA
1 CCIA
2 CCIA
XenDesktop centric organization

XenDesktop Sites
1
2-3
>3
#Desktops
<500
1000-5000
>5000
#vDisks
1-2
3-5
5+
Help Desk
1 CCA
3 CCA
3 CCEE
Admins
1-2 CCEE
3-4 CCEE
5+ CCEE
Architects
1 CCIA
1 CCIA
2 CCIA
Testing and Change Control

Regular updates and maintenance is an everyday part of Citrix environments. Standard processes
must be closely followed to ensure modifications do not inadvertently damage a production
environment. This includes maintaining dedicated infrastructure where modifications are tested prior
to production and processes that ensure changes are deliberate and accountable.
Testing Phases
Since changes to Citrix infrastructure can impact thousands of XenApp and XenDesktop
users, multi-phase testing is critical for the reliability of the environment. As such, Citrix
recommends implementing the following testing strategy:
Development. The development infrastructure exists outside of the production

network. Typically it consists of short-lived virtual machines, whose configuration
should match the configuration in production as close as possible. The purpose of
the development phase is to provide change requestors a non-production
environment to perform proof of concepts, determine integration requirements and
perform iterative testing as part of a discovery phase to making changes. Proposed
changes should be documented to they can be applied in the test phase.
Test. The test environment is a standalone 1:1 copy of the production

infrastructure and is used to confirm that the proposed changes can be easily
repeated prior to the pre-production staging environment. The changes made
should follow exactly from the documentation provided from discoveries made in
the development stages. If testing fails within the testing stage, the architect must
determine the severity of failure and determine whether minor updates to
documentation is sufficient or a full development cycle is needed.
Pre-Production. The staging environment can be co-located with the production

environment as the changes should be tightly managed. The goal of staging is to
implement the proposed changes with little risk or uncertainty. It is expected that any
changes made to the staging infrastructure have been tested and documented for
repeatability. There should not be any iterations or adjustments required within this
phase. For updates, the starting point for all systems in the staging environment
should be the current production status. However for new deployments, the
architect will determine the appropriate start point; whether an existing system or
newly created. Suring this phase and within this environment User Acceptance
Testing (UAT) should be performed.
Production. The production environment is the fully redundant and scalable

solution for normal usage of the XenDesktop and XenApp environments. There
should be minimal changes to the environment.
Infrastructure
There is a need for four dedicated levels of testing that ensures changes do not harm a
production farm. An isolated development environment should be used initially to try any
new changes, application installations, and updates. Conflicts resulting from these changes
have no impact in this environment and can be resolved before they affect any
infrastructure. Once proven, safe changes are then applied to a larger test environment. The
test environment should mimic production as closely as reasonably possible. This includes all
patches, applications, and policies be similar so that testing is as realistic as possible. Once
changes prove stable in the testing environment it is recommended the changes not be
initially rolled out to production all at once. Updates should be first applied to a targeted
pre-production subset of users to again validate and approve of changes made. This way if
any problems go undetected in the test environment the problems affect only a small subset
of users. When validated in pre-production changes can be applied to the production farm.
Change Management Processes
Standardized processes that manage application lifecycles through installation, updating, and
end of life are necessary to ensure consistent and accountable performance of a farm. The
change control process should be closely followed starting with a change request. For any
changes to be made to a production environment a change request form must first be filled
out detailing changes requested, reasons for the change, and intended timeframes for the
action. This is then reviewed and edited if required by a change manager and then advisory
board. When the change request has gone through the entire change approval process it is
then given to a change implementer who then stages the change for testing, and finally
conducts the implementation in production. The table and steps below further detail the
process:
10
Procedure
1. The Request for Change (RfC) form is completed by any person requesting a change
2. After appropriate manager approvals have been acquired, the RfC is forwarded to the
appropriate Change Manager(s).
3. The Change Manager validates the RfC for completeness and logs the RfC information
into the Change Control Log for tracking. Incomplete CRs are returned to the requestor
for update and re-submission.
4. The Change Manager assesses the impact of the change in conjunction with subject
matter experts and/or managers of the teams associated/affected by this change.
5. The Change Manager works with the associated/affected teams as well as the change
requestor in order to confirm the priority, category and type of the change as well as the
proposed back-out plan.
6. If the process succeeds the RfC is forwarded to the Change Advisory Board for
approval. If the process does not succeed the change control log is updated with the
current status as well as the reason of the rejection and the RfC is send back to the
requestor.
7. The Change Advisory Board reviews and validates the change in detail, and discusses
and evaluates purpose, reasons, impact, cost and benefits. Each Board member
represents their department and provides guidance on the change requests. The Change
Advisory Board also reviews multiple requests to coordinate implementations and
package requests into a single release schedule.
8. Upon approval the change is send back to the Change Manager to schedule the change
for implementation into the staging environment.
9. The change is implemented and test are conducted. The results are sent back to the
Change Manager.
10. If the staging implementation and testing was successful the change is scheduled for
production implementation. In case the staging phase was not successful another staging
iteration will be conducted.
11. The change is rolled out to production.
12. The Change Manager reviews the implementation and finally updates the Change
Control Log.
13. On a periodic basis, the Change Manager reviews the Change Control Log to analyze for
trends on type, frequency and size of changes and forwards the results to the Change
Advisory Board for review.
In the instance of an emergency the processes may be expedited. Should an issue be declared
an emergency a change request form is still filled out and delivered to the appropriate change
management representative. When approved the requested change is immediately
11
implemented and the advisory board notified. Due to the criticality of time in an emergency
timeframes are not required.
Change Implementation Process

The process to implement changes in Citrix environments need to be highly structured.
Standardization helps ensure reliable results while reducing risks of unexpected results to
production environments. Update requests undergo a centralized approval and then must go
through an engineering and tracking process as they are introduced into environment. The
process begins at a development environment and progresses through testing, preproduction, and finally full production.
Potential updates must first undergo an assessment to determine compatibility and
functionality in a development environment. This environment provides a rudimentary setup
where changes are initially attempted to observed potential conflicts or performance issues.
These results provide feedback on areas to be further tested while not risking more
significant environments.
Once proven and approved in the development environment further tests are conducted in a
testing environment. This is larger environment that models what is used in production.
Having an arrangement that closely matches production ensures testing results are accurate.
All changes to this environment are document for repeatability in the production
environment, future reference, and for change control.
After rigorous testing in the test environment changes must be re-approved and then reduplicated to a pre-production subset of users in the production environment. This selection
of users provide additional feedback on their experiences and ensure the changes perform as
expected. Should no issues be encountered by the pre-production users updates are then
written off by appropriate parties and applied to the rest of the farm. The below diagram
highlights the key sections in the change implementation process.
12
Ongoing Operations
It is best practice to keep operating systems, applications, and Citrix components all up to date with
applicable hotfixes and updates to ensure stability, performance, and security of a farm. In general
Citrix recommends that Hotfix Rollup Packs and Service Packs always be installed, general hotfixes
be installed on an as needed basis only, and that all changes follow standardized change
management processes for any changes (i.e. development, testing, pre-production, and production).
For more information regarding general hotfix maintenance refer to (CTX132799).
Monitoring of a new environment enables administrators to address issues proactively. By having an
in depth understanding of current and expected behavior of the various components, administrators
are better equipped to discover an issue before it impacts the user community.
Make sure to define the monitoring activities that administrators will perform on a periodic basis, as
well as the tools that will be used. In addition, determine which reports will be made available to
management.
A variety of built-in and customizable reports exist within the Citrix products so that administrators
can track trends, effectively plan capacity, and address auditing requirements. In addition,
troubleshooting is typically minimized when administrators fully utilize the tools integrated into the
Platinum edition, as well as other third party tools.
It is recommended that support staff and administrators involved in implementation review the
product documents, articles, and guides to support, manage and administer the environment. All of
these documents are available from the Citrix Product Documentation Library
(http://support.citrix.com/proddocs/index.jsp) or on the Citrix Knowledge Base
(http://support.citrix.com).
The remaining sections in this section identify the maintenance tasks (daily, weekly, yearly and
preventative) required to support each component of the XenDesktop Platinum edition.
XenDesktop Operations
This section covers the regular maintenance and upkeep involved in a XenDesktop
environment.
Overview
Several tasks need to be performed to ensure a stable, scalable Citrix XenDesktop platform.
The team responsible for supporting and managing the Citrix XenDesktop infrastructure
should perform regular tasks that should be incorporated into the existing standard
operating procedures. These are usually broken down into daily, weekly or yearly tasks.
13
Periodic Tasks
Daily Periodic Tasks
The following table outlines the necessary tasks that should be undertaken by the
appropriate Citrix XenDesktop administrators on a daily basis:
Daily Tasks Citrix XenDesktop
Checking of the event logs on all Citrix XenDesktop (Desktop Delivery
Event log checking
Controller) servers including Windows System, Application and Error logs.
Checking of the Citrix XenDesktop alerts in Desktop Studio needs to be
Citrix XenDesktop Alerts
performed at least once per day to ensure no issues have occurred with the
Citrix XenDesktop environment.
Checking of Citrix XenDesktop specific application logs including the Pool
Check Citrix XenDesktop
Management Logs. To assist with troubleshooting efforts, refer to
logs
Comprehensive List of Event Log Messages for XenDesktop on
http://support.citrix.com/article/CTX119978
Test Virtual Desktop
Simulate a connection to ensure access to Desktop Groups (Pooled and
Access
Assigned) is available to users.
Weekly Periodic Tasks

appropriate Citrix XenDesktop administrators on a weekly basis:
Weekly Tasks Citrix XenDesktop
Review Access
Check the status of the Citrix XenDesktop Desktop Groups, Idle Pool count
Management Console
and connectivity to backend hosting infrastructure.
Review the latest Citrix hotfixes and ascertain whether the XenDesktop
Hotfixes and Patches
Controller and virtual desktops require them.
Review the local installation of printer drivers on the virtual desktops, if
Printer Drivers
required for auto-creation.
Review Citrix Knowledge
Review incidents logged for any common issues related to Citrix XenDesktop
Base for latest incidents
on http://support.citrix.com/
Backup Citrix XenDesktop Perform full-data backup of the Citrix XenDesktop Datastore (could be a daily
Database
task due to size of the transaction log)
Backup User Data
Backup user-related file data.
Yearly Periodic Tasks

appropriate Citrix XenDesktop administrators on a yearly basis:
Yearly Tasks Citrix XenDesktop
Conduct backup tests to validate the actual restore process is functioning
Perform restore test
correctly.
Review Citrix Policies and determine whether new policies are required and
Citrix Policy Assessment
existing policies need to be updated.
14

Yearly Tasks Citrix XenDesktop
Review and asses the requirement for new Citrix XenDesktop software
Software Upgrade
releases/versions.
Capacity Assessment
Perform capacity assessment of the Citrix XenDesktop environment.
Preventative Maintenance
As a part of your preventative maintenance routine for your Citrix XenDesktop
environment, Citrix Xnapshot for XenDesktop should be run on regular three month
intervals. This tool collects a wealth of information about the underlying system and its
configuration, such as BIOS information, registry information, device drivers, Windows
services, installed hotfixes, Citrix binaries, and Citrix XenDesktop farm information.
Citrix recommends that the following key Microsoft Windows services be monitored on all
of the Citrix XenDesktop (Desktop Delivery Controller) servers to ensure reliability.
Citrix XenDesktop Services to be Monitored
Citrix Desktop Delivery Controller Service
Citrix Diagnostic Facility COM Server
Citrix Management Server
Citrix MFCOM Service
Citrix Pool Management Service
Citrix Services Manager
Citrix XTE Server
IPSEC Services
Remote Procedure Call (RPC)
Server
Windows Management Instrumentation Driver Extensions
Workstation
Updating MCS Images

MCS images require regular maintenance throughout the images lifecycle. It is standard
procedure to first take a snapshot of a master image prior to any updates. New changes are
made to this snapshot and it will become the new master image. The new image, following
standard change management policies, is fully tested in an isolated test environment. When
the new image successfully passes testing and is proven stable the image is rolled to preproduction subset in of production users. When a new image is shown to be reliable in preproduction it can then be rolled out to the rest of the farm. Below is a diagram and steps
further detailing this process.
15
1. When a change request is approved a snapshot is made of the master image.

2. The desired changes are made to the new snapshot.
3. Functionality and stability is of the new image is tested within an isolated test
environment.
4. When validated and written off as stable in the testing environment the image is
migrated to a pre-production catalog where IT friendly users use the new
image and ensure image stability.
5. Once proven, the image is added to the production catalog
16
Hotfixes
For hotfixes, backups of affected infrastructure (i.e. database, registry, etc) should be
made. Then half of the controllers should be updated and observed for several days. Should
no conflicts arise, the rest of environment may be updated.
XenApp Operations
This section covers the regular maintenance and upkeep involved in a XenApp environment
Overview
Several tasks need to be performed to ensure a stable, scalable Citrix XenApp platform.
The team responsible for supporting and managing the Citrix XenApp infrastructure should
perform regular tasks that should be incorporated into the existing standard operating
procedures. These are usually broken down into daily, weekly or yearly tasks.
Periodic Tasks
appropriate Citrix XenApp administrators on a daily basis:
Delete Contents
Event log checking
Reboot schedule checking
Citrix XenApp Alerts
Suggested daily Citrix
XenApp commands to be
run
Test Application Access
Checking of the event logs on all Citrix XenApp servers including Windows
System, Application and Error logs.
All Citrix XenApp servers are rebooted on a regular basis as part of an
automated process. Ensuring the reboot of the servers has been successful is
essential to maintain server health and performance.
Checking of the Citrix XenApp server alerts in the Access Management
Console needs to be performed at least once per day to ensure no issues have
occurred with the XenApp environment.
Qfarm /load - to check that the load of the Citrix XenApp servers are
reporting correctly
Qfarm /zone - to confirm that the correct Zone Data collector is assigned
Qfarm /online - to confirm all Citrix XenApp servers are online
Simulate an ICA connection to ensure key applications are available to users.
17

appropriate Citrix XenApp administrators on a weekly basis:
Weekly Tasks Citrix XenApp
Administration
Confirm that Citrix XenApp farm changes taken place during the previous
Configuration and Logging
week have been approved.
Reports
Citrix XenApp Commands
DSCheck.exe - verify that the contents of the datastore are still valid.
to be run
Review the latest Citrix hotfixes and ascertain whether the Citrix XenApp
farm requires them.
Review incidents logged for any common issues related to Citrix XenApp on
http://support.citrix.com/
Review the local installation of printer drivers on each Citrix XenApp server
Printer Drivers
to ensure no rogue drivers have been installed
Perform full-data backups of the following Citrix XenApp databases:
Citrix XenApp Datastore
Backup Citrix XenApp
related databases
Citrix SmartAuditor Datastore
Citrix XenApp Configuration Logging Database
Backup User Data
Backup user-related file data.
Backup Application
Perform full-data backup of the Citrix XenApp application profiles used for
Profiles
streaming.

appropriate Citrix XenApp administrators on a yearly basis:
Delete Contents
Citrix Policy Assessment
Perform BCP/DR test
Software Upgrade
Capacity Assessment

correctly.
Review Citrix Policies and determine whether new policies are required or
existing policies need to be updated.
Conduct functional Business Continuity Plan (BCP)/Disaster Recovery (DR)
test to confirm DR readiness.
Assess the validity of applications. Review and asses the requirement for new
Citrix Web Interface software releases/versions.
Perform capacity assessment of the Citrix XenApp environment to determine
environment utilization and any scalability requirements.
18
of the Citrix XenApp servers to ensure reliability.
Citrix XenApp Services to be Monitored
Citrix Client Network
Citrix Diagnostic Facility COM Server
Citrix Encryption Service
Citrix Health Monitoring and Recovery
Citrix Independent Management Architecture
Citrix MFCOM Service
Citrix Print Manager Service
Citrix Services Manager
Citrix SMA Service
Citrix Streaming Service
Citrix WMI Service
Citrix XTE Server
Print Spooler
Server
Remote Desktop (Terminal) Services
Citrix Systems Monitoring Agent
Application Updates\Installs
1. When an application change request is approved the application is tested in a

development environment to observe its performance and resource consumption
2. Once written off that the application is safe to install and use it is included in the test
environment for further testing
19
3. Upon satisfying testing requirements the changes are then duplicated in the
production farm but assigned only to a specific pre-production subset of users for
further validation testing.
4. When proven in pre-production changes then may be signed off on and rolled out to
the rest of the users.
For additional information on installing and publishing applications in XenApp please refer
to Citrixs eDocs - Publish a resource using the Publish Application wizard.
Hotfixes
The general hotfix recommendations still standing XenApp hotfix updates should be first
applied to the Zone Data Collector (ZDC), backups of the ZDC, the XML broker, and then
the rest of member servers in this order. This ensures against accidental elections or
undesired farm reactions from occurring. For more info reference CTX120842
Printing
It is recommended that environments leverage Citrixs Universal Print driver. In the instance
additional printer drivers are still required the driver should first be installed in an isolated
test environment. Be sure to ensure to test the printer when it is out of paper, offline, as well
as to stress test the printer leveraging the Citrix StressPrinters 1.3 application (CTX109374)
PVS Operations
This section covers basic maintenance involved in a Provisioning Server environment.
Overview
Several tasks need to be performed to ensure a stable, scalable Citrix Provisioning Services
platform.
The team responsible for supporting and managing the Citrix Provisioning Services
infrastructure should perform regular tasks that should be incorporated into the existing
standard operating procedures framework. These are usually broken down into daily, weekly
or yearly tasks.
Periodic Tasks
appropriate Citrix Provisioning Services administrators on a daily basis:
20

Daily Tasks Citrix Provisioning Services
Checking of the event logs on all Citrix Provisioning Servers including
Event log checking
Windows System, Application and Error logs.
Checking of Citrix Provisioning Server specific application logs including the
following:
Check Citrix Provisioning
Citrix Provisioning Server logs - %APPDATA%\Citrix\Provisioning
Server logs
Services\logs
Reference of PXE Error Codes http://support.citrix.com/article/CTX120956
Check Citrix Provisioning
Check the number of target devices connected to the Citrix Provisioning
Server utilization
Servers and balance the load across the servers, if required.

appropriate Citrix Provisioning Services administrators on a weekly basis:
Weekly Tasks Citrix Provisioning Services
Backup of Citrix Provisioning Server vDisk files (.VHD) on local and/or
Backup vDisk Image Files
shared storage.
Backup Citrix Provisioning Backup of Citrix Provisioning Server database hosted on SQL Server
Server Datastore
infrastructure.
Review the latest Citrix hotfixes and ascertain whether the Citrix Provisioning
Server farm requires them.
Review incidents logged for any common issues related to Citrix Provisioning
Server on http://support.citrix.com/
Update the golden vDisk image files and apply the following:
Windows software updates and patches
Perform vDisk Updates
Operating system and application changes
Anti-virus pattern and definitions updates
Deploy updated vDisk version to target devices post update process
Review the storage utilization used and free storage space for vDisks and
Check Storage Capacity
write cache files.
Check Auditing Reports
Review the Citrix Provisioning Server Auditing Logs.
Check Network and

Streaming Services
Confirm DHCP, TFTP (or PXE, if applicable) and Streaming services are
functional.

appropriate Citrix Provisioning Services administrators on a yearly basis:
Yearly Tasks Citrix Provisioning Services
Perform an archive of the Citrix Provisioning Server Audit Trail Information
Archive Audit Reports
for compliance requirements.
21

Yearly Tasks Citrix Provisioning Services
Update the vDisk desktop and/or server workloads (.VHD files) to reflect the
Update vDisk Workloads
current standard operating environment, if required.
correctly.
Conduct functional Business Continuity Plan (BCP)/Disaster Recovery (DR)
Perform BCP/DR test
test to confirm DR readiness.
Review and asses the requirement for new Citrix Provisioning Server software
Software Upgrade
releases/versions.
Perform capacity assessment of the Citrix Provisioning Server environment to
Capacity Assessment
determine environment utilization and scalability requirements including
server, storage and network components.
of the Citrix Provisioning Services servers to ensure reliability.
Citrix Provisioning Services to be Monitored
Citrix PVS Soap Server
Citrix PVS Stream Service
Citrix PVS TFTP Service
Citrix PVS Two-Stage Boot Service
HTTP SSL (This is a service dependency for Citrix PVS Soap Server service)
DHCP Service (Located on Microsoft DHCP Server/s)
22
vDisk Updates
Due to infrastructure requirements for Provisioning Services farms utilize a slightly modified
testing\updating environment arrangement. Instead of isolated environments testing and
production share several key infrastructure components. Each environment has its own
dedicated XenDesktop farm and collections but share networks and Active Directory
(utilizing differing organizational unit structures and group policies) with one another.
Testing, pre-production, and production devices are assigned separate collections but all
leverage the same vDisk. In this setup a master device gets maintained that vDisks are
created from. These vDisks are assigned to the various device collections as the testing flow
progresses. These differences aside general updating best practices still apply in the updating
of vDisks.
For high level updates the change process utilizes Provisioning Services versioning
capability. For incremental changes such as hotfixes and patches a new vDisk version is
created which the changes are applied to. These updates can also be automated utilizing
PVSs automated vDisk update process (Citrix eDocs). It is recommended that any changes
be applied to new versions are replicated on the master device for consistency.
More significant changes such as updating XenTools, NIC drivers, etc. require that a new
master vDisk be captured and tested. The steps below walk through the process:
1. Following standard procedures approved changes are initially attempted within an

isolated development environment.
2. Once approved these updates are replicated into a new vDisk within the test
environment for further validation.
Note: Changes should be manually replicated across environments and not copied.
23
3. After successful testing and approval, the vDisk is then assigned to a PreProduction subset of users for final validation testing prior to implementation.
4. When shown to be stable, this new vDisk is assigned to the rest of the production
end users.
24
Appendix
Appendix A XenApp Delegated Rights
Console Permission
Tier-1
Farm Node
Farm Management
Edit All Other Farm Settings
Edit Configuration Logging
Settings
Edit Zone Settings
View Farm Management
Administrators Node
Administrators
Edit centrally configured
Web Interface sites
Log on to Management
Console
View Administrators
Applications Node
Published Applications
Publish Applications and
Edit Properties
View Published Applications
and Content
Servers
Terminate Processes
Sessions
Connect Sessions
Disconnect Users
Log Off Users
Reset Sessions
Send Messages
View Session Management
Load Evaluators Node

Load Manager
Assign Load Evaluators
Edit Load Evaluators
View Load Evaluators
Load Balancing Policies Node
Load Balancing Policies
Edit Load Balancing Policies
View Load Balancing policies
Tier-2
Tier-3
Tier-4
25

Console Permission
Policies Node
Policies
Edit Policies
View Policies
Printer Management Node
Printers
Edit Printer Drivers
Replicate Printer Drivers
View Printers and Printer
Drivers
Servers Node
Published Applications
Assign Applications to
Servers
Servers
Edit Other Server Settings
Move and Remove Servers
Terminate Processes
View Server Information
Sessions
Connect Sessions
Disconnect Users
Log Off Users
Reset Sessions
Send Messages
View Session Management
Worker Groups
Assign Applications to
Worker Groups
View Worker Groups
Tier-1
Tier-2
Tier-3
Tier-4
For further information about delegated rights within a XenApp Farm, please refer to
eDocs - Delegating Tasks to Custom Administrators.
26
Appendix B XenDesktop Delegated Rights

Admin Role
Help Desk Administrator
Assignment Administrator
Full Administrator
Read-Only Administrator
Support Tier
Tier-1
Tier-2
Tier-3
Tier-4
For further information about delegated rights within a XenDesktop Site, please refer to
eDocs XenDesktop Delegated Administration.
Appendix C Provisioning Services Delegated Rights

Admin Role
Device Operators
Farm Administrator
Farm Administrator
Support Tier
Tier-1
Tier-2
Tier-3
Tier-4
For further information about delegated rights within a Provisioning Services Farm, please
refer to eDocs Provisioning Services Managing Administrative Roles.
Appendix D XenServer Delegated Rights

Admin Role
Virtual Machine Operator
Pool Administrator
Read-Only
Support Tier
Tier-1
Tier-2
Tier-3
Tier-4
For further information about delegated rights within a Provisioning Services Farm, please
refer to CTX130420 XenServer 6.0 Administrators Guide (see chapter Role Based Access
Control).
27
Appendix E Sample Call Script/Questionnaire

The following tables provide sample call scripts / questionnaires which should be used by
call entry employees in order to capture all information required to solve an incident. Items
marked grey should be captured for all incoming calls. All other information should be
captured in case the incident needs to be escalated to a higher support level.
Scripts
Steps
Details
1
2
3
Name of Customer, particular site or location if possible.

Name of User, department, Logon ID and callback number.
Do any other users at the site experience the same issue? Can they have a
colleague logon from same and/or different workstation? Helps to determine
whether this is a workstation issue vs. user account issue.
Identify workstation name or Inventory ID.
Name of Application attempting to access.
Review high level steps taken to authenticate. This validates that steps are
performed correctly.
Can user see the Web Interface or authentication page? Can users see other web
pages, such as Yahoo.com? This helps to identify and troubleshoot network issues.
Does user see the appropriate icon? Helps to troubleshoot user access and group
memberships.
Does application launch when icon is selected? Does the application logon screen
appear? Helps to determine if a connection is made into Citrix architecture.
Can user authenticate into the application? Does the issue occur after application
authentication? Helps determine if this is a Citrix infrastructure vs. Application
Issue.
What is the application error(s)? Provide a screenshot.
4
5
6
7
8
9
10
Incident
Personal
Information
11
Question
Name
Organization
Contact Number /Email
Contact Preference (Phone/Mail)
Resident Office
Current Location
Current Time Zone
Severity of the incident
# of affected users
Name of affected application
Description of the incident
Troubleshooting performed
Answer
<Provide Screenshot / Video when escalating>
28
Environment

Question
Client name
Client IP
Client hardware type
Client OS
Version Citrix Receiver
Name of XenApp Server / virtual
desktop
Name of PubApp / PubDesktop
Answer
29
Product Versions
Product
XenDesktop
XenApp
Provisioning Services
Version
5.0 / 5.5 / 5.6
5.0 / 6.0 / 6.5
6.x
Revision History
Revision
1.0
Change Description
Finalized Document
Updated By
Thomas Berger, Charles Wright
Date
May 30, 2012
About Citrix
Citrix Systems, Inc. (NASDAQ:CTXS) is a leading provider of virtual computing solutions that help
companies deliver IT as an on-demand service. Founded in 1989, Citrix combines virtualization,
networking, and cloud computing technologies into a full portfolio of products that enable virtual
work styles for users and virtual datacenters for IT. More than 230,000 organizations worldwide rely
on Citrix to help them build simpler and more cost-effective IT environments. Citrix partners with
over 10,000 companies in more than 100 countries. Annual revenue in 2011 was $2.20 billion.
2012 Citrix Systems, Inc. All rights reserved. Citrix, Access Gateway, Branch Repeater,
Citrix Repeater, HDX, XenServer, XenApp, XenDesktop and Citrix Delivery Center
are trademarks of Citrix Systems, Inc. and/or one or more of its subsidiaries, and may be registered
in the United States Patent and Trademark Office and in other countries. All other trademarks and
registered trademarks are property of their respective owners.
30

Operations Guide - Support and Maintenance

Diunggah oleh

Informasi Dokumen

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Operations Guide - Support and Maintenance

Diunggah oleh

Hak Cipta:

Format Tersedia

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Testing and Change Control ..................................................................................................... 8

Ongoing Operations ................................................................................................................ 13

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Ongoing Operations Maintenance, issue prevention and resolution are core

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Provide first-line support of reported

Perform issue definition, initial analysis

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Primarily supporting day-to-day

Perform intermediate issue analysis and

All members of the Tier-2 team should

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Central point for implementing,

Perform advanced issue analysis and

Tier-3 support team members with a

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

A qualified Tier-4 resource should have

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Used to document customer information and issues. Help Desk

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

maintenance tasks for a

Appendix C (Provisioning Services)

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Ticket Management System: It is vital for support organizations to implement a

Knowledge Base / Wiki: Ever support organization should document and

XenDesktop centric organization

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Testing and Change Control

Development. The development infrastructure exists outside of the production

Test. The test environment is a standalone 1:1 copy of the production

Pre-Production. The staging environment can be co-located with the production

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Production. The production environment is the fully redundant and scalable

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Change Implementation Process

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Weekly Periodic Tasks

Yearly Periodic Tasks

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Updating MCS Images

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

1. When a change request is approved a snapshot is made of the master image.

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Weekly Periodic Tasks

Yearly Periodic Tasks

Conduct backup tests to validate the actual restore process is functioning

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

1. When an application change request is approved the application is tested in a

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Weekly Periodic Tasks

Review the Citrix Provisioning Server Auditing Logs.

Check Network and

Yearly Periodic Tasks

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance

Worldwide Consulting Solutions | WHITE PAPER | Support and Maintenance