Network Performance Monitoring

Presenter(s):
Network Performance Monitoring
Topic
Technical
Level
Client
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Things our lawyers make us say...

Neither NextGen Healthcare nor any presenter at the Users Group Meeting is engaged in rendering legal or other professional advice and this presentation is not a substitute for the advice of your attorney, accountant and/or other professional advisor.
Session Guidelines
Silence all cell phones Questions will be addressed at the end of the presentation Refrain from personal discussion
BJC HealthCare Hospital Locations
13 Hospital Facilities BJC Medical Group Largest employer in state of Missouri Long-time relationship with Washington University School of Medicine
Broad Missouri/Illinois Geography (only 13% from St Louis City)
National Leader Local Brand Identity
400+ Providers 100+ Locations 1200+ Concurrent Users Growth Strategy Tier 1 Physicians employed by BJC using NextGen Tier 2 Private Physicians not employed by BJC and using NextGen Tier 3 Private Physicians not employed by BJC and not using NextGen. Leverage Community Health Connection (CHC)
How issues where handle in the past

Customer Calls
Challenges
Implement Fix Mobilize IS Teams
Communication between teams
Ability to capture data to access and implement fix quickly

Determine Root Cause
How to reduce customer impact

Capture Data
Access Data
Performance Monitoring Project

Customer Calls Or
Performance Monitoring Objectives Proactive monitoring Identify critical data Application Network Hardware Resolve problems before customer impact Keys to Success
ALERTS
Implement Fix
Mobilize IS Teams
Determine Root Cause
Capture Data
Access Data
Relationship with customers Relationship with vendors Relationship with internal teams
Things our lawyers make us say...

Neither NextGen Healthcare nor any presenter at the Users Group Meeting is engaged in rendering legal or other professional advice and this presentation is not a substitute for the advice of your attorney, accountant and/or other professional advisor.
Sample Encounters
Track encounter performance at different times of the day Identify system hangs, slowness before input from endusers Compare performance from different locations in the environment Can be as simple or as complicated as you want
Sample Encounter
Database Monitoring
SQL Counters SQL DMVs/Tables SQL Notifications SCOM Data
Page Life Expectancy

Page Life- The amount of time a page stays in memory before being replaced
Constant low values indicate memory pressure
Track values continuously to compare over time

1600 1400 1200 1000 800 600 400 200 0 10:00 10:05 10:10 10:20 10:25 10:30 10:35 10:40 10:45 10:55 Microsoft Recomended Value Microsoft Warning Value BJC Value
10:15
10:50

35000 30000 25000 20000 Microsoft Recomended Value 15000 10000 5000 Microsoft Warning Value BJC Value
Cache Hit Ratio

Buffer Cache Hit Ratio-Percent of pages found in cache without having to read from disk Recommended Value of >99%
Database Backup Completion

Backups that fail dont do you any good Backup during production hours can cause performance issues
Database Backup Completion

Real Life Example
14:24 12:00 9:36 7:12 4:48 2:24
0:00 Fri
Fri Fri
Tues
Thur
Thur
Mon
Thur
Thur
Tues
Tues
Tues
Thur
Mon
Mon
Wed
Wed
Wed
Wed
Mon
Wed
Fri
Indexes
Missing Indexes Impacts performance as SQL needs to read more data than necessary to return results to user Covering Indexes
Indexes
Unused Indexes Unnecessary writes to the database
Expensive Queries
Find procedures/queries that can be tuned to increase performance or scheduled to run during non-production hours
Find recent reports run that had large data reads
Daily/Weekly Maintenance
NextGen Weekly maintenance updates defragments indexes on a weekly basis
There are some indexes that fragment too quickly for once a week rebuilds Consider running weekly maintenance daily or daily defrag on a subset of tables
Database Blocking
SQL Agent Job that sends email to support staff to alert database blocking
Also writes blocking history (head blocker, blocked queries) to table for review Support response changes based on EHR, EPM, support application blocking
Database Deadlocks
Track Deadlocks every 15 minutes
Alert support team on excessive deadlocks Write Deadlock graphs to table for review Compare deadlocks during business hours vs after hours
100000
150000
200000
250000
50000
0 6/30/2013 7/1/2013 7/2/2013 7/3/2013 7/4/2013 7/5/2013 7/6/2013 7/7/2013 7/8/2013 7/9/2013 7/10/2013 7/11/2013 7/12/2013 7/13/2013 7/14/2013
7/15/2013
7/16/2013 7/17/2013 7/18/2013 7/19/2013 7/20/2013 7/21/2013 7/22/2013 7/23/2013 7/24/2013 7/25/2013 7/26/2013 7/27/2013 7/28/2013 7/29/2013
Database Storage Monitoring
LOB
Core
Index
100000
120000
20000
40000
60000
80000
Image File Storage
6/30/2013 7/1/2013 7/2/2013 7/3/2013 7/4/2013 7/5/2013 7/6/2013 7/7/2013 7/8/2013 7/9/2013 7/10/2013 7/11/2013 7/12/2013 7/13/2013 7/14/2013 7/15/2013 7/16/2013 7/17/2013 7/18/2013 7/19/2013 7/20/2013 7/21/2013 7/22/2013 7/23/2013 7/24/2013 7/25/2013 7/26/2013 7/27/2013 7/28/2013 7/29/2013
ICS Images
NextgenRoot
Server Storage
Application Server Monitoring

Citrix Server Utilization
3500 80 70 60 50 2000 40 1500 30 Min Free Memory Mem Warning Max CPU CPU Warning 20 10 0 3000
2500
1000
500
10
20
30
40
60
70
80
90
50
100

BJCNXXAP01 CPU Usage 10/17/2012
41199.35328 41199.36022 41199.36717 41199.37411 41199.38105 41199.388 41199.39494 41199.40189 41199.40883 41199.41578 41199.42272 41199.42984 41199.43696 41199.4439 41199.45085 41199.45779 41199.46474 41199.47168 41199.47862 41199.48557 41199.49251 41199.49946 41199.50658 41199.51352 41199.52046 41199.52741 41199.53435 41199.5413 41199.54824 41199.55519 41199.56213 41199.56908 41199.57602 41199.58297 41199.59008 41199.59703 41199.60397 41199.61092 41199.61786 41199.62481 41199.63175 41199.63869 41199.64564 41199.65258 41199.65953 41199.66647 41199.67376 41199.68088 41199.68783 41199.69477 41199.70172 41199.70866

Users on each server Load caused by new applications Environment Changes that will change the amount of users supported
Interface Server Monitoring

Intrf_Queue Table
Record Cleanup
Temp_Rec_Trfr
Network Monitoring
Alert Support on Node Down (Wan/Primary Switch) Track utilization on WAN links (inbound/outbound) Track network latency
BBP
Track completion times on time sensitive jobs Disable Job steps that require data when data is not present Track completion rates Notify users before job completes
Tools
SQL Server Management Studio Perfmon SCOM Orion Report Server Citrix Management Console In-house development
Monthly Review with Leadership

Summary Action Plan We will be adding indexes suggested by the SQL Server to improve performance of database reads after removing the indexes that were not used previously. In addition we discovered that the patient images on the templates add a significant amount of time to the template load time when using the fat client environment. The EHR team is working on removing the image where they can. after initially believing that only hiding the image was necessary. We have migrated the disk space monitoring from SCOM to the Orion data and resumed the display of all of the storage drives to the SQL and File servers. This was necessary due to a change made by Microsoft to the method that data is captured into SCOM.
Object
Notes
Impact
Workstation Monitoring
Workstation monitoring time was up slightly to 143s this month. We continue to have issues with the process completing on a consistent basis. The change in the workflow for the new KBM caused an increase in the amount of work inside the templates which is giving us more points of failure within the script. We may need to revisit the script to simplify the steps that it takes.
Users
We saw no change in the total user capacity in the environment which was expected as there were no major changes. The amount of users continue to trend upwards at a slight pace. The eight o'clock hour was up to 678 and the five o'clock hour was up to 307 users.
Blocking Deadlocks
We are working with the DBA team to remove the blocking emails that are sent out when only the BBP machines are impacted, that will eliminate some of the extraneous emails that are sent when users are not impacted. We had a major blocking issue on 11/6. We cleared the blocking by stopping the CHS service. In working with NextGen, they have offered us updates on triggers, that will lesson the possibility of blocking caused by CHS in the future. It is not known if CHS was the root cause of the block or another process, just that the interaction within CHS and the rest of the system caused blocking. We also found that information was being sent to CHS that was not necessary and removed that. Deadlocks were down slightly to 25/day during business days and 27.21 overall. Steelville clinic experienced a three hour outage caused by a router failure. The router was replaced by the network team. The overall down time was up due to overnight maintenance that took multiple sites down.
Network Outages
Jobs
The Report Server restore job failed multiple times at the end of the month due to human error. Programming has been put in place to prevent these types of failures in the future. All other job success rates were consistent with the previous month.
Citrix
The results from the sample of the five servers is complete. We did see periods of 100% usage on one of the servers caused by the DR. Watson program that generates the dump files when an application crashes. These periods were up to 3-5 minutes and would have impacted the users on that server. The servers with the higher number in the queue seemed to be utilized less than the servers with the lower numbers but it is unclear if this is random or actual behavior. We plan to request a followup sample of five servers. Overall usage of the Citrix servers was up slightly in CPU as well as Memory usage.
Any Questions?

Network Performance Monitoring

Diunggah oleh

Informasi Dokumen

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Network Performance Monitoring

Diunggah oleh

Hak Cipta:

Format Tersedia

Presenter(s):

Network Performance Monitoring

Things our lawyers make us say...

BJC HealthCare Hospital Locations

Broad Missouri/Illinois Geography (only 13% from St Louis City)

National Leader Local Brand Identity

How issues where handle in the past

Communication between teams

Ability to capture data to access and implement fix quickly

How to reduce customer impact

Performance Monitoring Project

Determine Root Cause

Things our lawyers make us say...

Page Life Expectancy

Track values continuously to compare over time

Page Life Expectancy

Page Life Expectancy

Cache Hit Ratio

Database Backup Completion

Database Backup Completion

14:24 12:00 9:36 7:12 4:48 2:24

Find recent reports run that had large data reads

Database Storage Monitoring

Image File Storage

Application Server Monitoring

Application Server Monitoring

Application Server Monitoring

Interface Server Monitoring

Monthly Review with Leadership

Anda mungkin juga menyukai