New Text Document

Daniel Burchett 8:34 AM
Good morning ..
Ok, current status ...
We were preparing to reboot the storagecell np1exda001cel02, since it was the on
e we were getting all the alerts on. I could not fin anything within the alert l
ogs or on the stoage cell.
So, I performed the eheck on the storage cell , everything was ready to go ..
Gudivada, Srikanth 8:36 AM
ok
Talked to Mark to convince him to let us go forward, when I re-did the check, it
started showing errors. The alert logs on all three ASM instances and the the n
odes also started showing disk errors at 11:48AM.
yes,we have gone throuh ur mail
St tht point, Andy Porras and I stopped out work and opened a Sev 1 Oracle SR. A
fter several hours, the deteremined that there was a single disk drive failed an
d sent and Oracle Engineer to replace the drive.
however, they replaced one of hte drive and realancing started in the ASM, the s
torage cell still showed the following :
oh
CellCLI> list griddisk
DATA_CD_00_np1exda001cel02 active
DATA_CD_02_np1exda001cel02 proactive failure
DATA_CD_03_np1exda001cel02 not present
DBFS_DG_CD_02_np1exda001cel02 proactive failure
DBFS_DG_CD_03_np1exda001cel02 not present
DBFS_DG_CD_04_np1exda001cel02 active
DBFS_DG_CD_05_np1exda001cel02 active
RECO_CD_00_np1exda001cel02 active
RECO_CD_02_np1exda001cel02 proactive failure
RECO_CD_03_np1exda001cel02 not present
I still have 6 3TB drives not usable.
I have asked Andy to add this ti the SR and see what Oracle says.
So, Will this resolve the performance issue today? I give it about 50/50
Yunus Ali, Shaik 8:41 AM
hi
ok
There is a bug on the firmware of the drives that when one fails.. It can impat
the other drives around it, but it can degrade performance..
ok
hi, shaik, can you see what I hav already typed ?
means, we have to wait again for oracle suggestion
yes.
ok
got it
However, it "appears" from my perspective that the queries are moveing faster th
an they were earlier today ..
Shaik Yunus Ali 8:44 AM
ok
So, here are the action plan.
Andy and his hardware team will keep the SR in their control on the storage cell
.
We will need to open an SR in case there is more than one issue here. Shaik or S
rikanth, can one of you open an SR on the poor query performance and upload teh
trace files and log s they may need. I believe they have a tool to load up every
thing (awrs. trace. etc..) and get it from the three nodes
ok
sure
If the ODS team startes to have issues, raise it from a sev 3 to sev 1, contacti
ng the duty manager immediately to get Oracle working on it ...
I believe Jon Burgess sent and email on the escalation method.
I look for it and forward it to you if you do not have it .
we dont have
On another note, did you see the email on Oracle response to the CLickSched/CSFT
disconnect issue?
:)
ys
yes
yes
I hope he settles down and look at the issue logically.
ok
mean these changes need to be applied at Target side but .not at exadata
yes. 11.2.0.3 has the fix
:)
is any other thing we have to fallow now on ODS database?
i am creating the SR# on this CSI
18972029
only
shal i go ahead
just watch the alert logs .. I need you to work with the hardware team off shore
if they need help. Shaik, They may need some help on checking ASM and rebalanci
ng
sure
Yes, go ahead and open an SR..
ok
ROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CO
DE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- -
---------- --------------------------------------------
1 REBAL RUN 2 2 118269 1523606 2111 665
3 REBAL WAIT 2
SQL>
we have a call at 10:30 CST
i am on top of it Danny
I know you would be ..:)
once done i will inform
Thank you to both of you for hanging around most of your evening . I really appr
eciate it ..
When I get on the tracks of troubleshooting an issue, I really do not like inter
ruptions from mgmt ..:D
:)
thanks,and welcome:)
Pradeep Reddy is the Hardware contact
ok Danny
David Clearwater is the mgr ..
ok
ok
https://meet.Halliburton.com/neeraj.chauhan/R0SER73Z (https://meet.halliburton.c
om/neeraj.chauhan/R0SER73Z)
Join by Phone
281 871 4999
88 250 4999
Find a local number (https://dialin.halliburton.com/)
Conference ID: 21560074
yes sir
create sr sir.
created*
nice
Thank you guys .. any questions ?
no questions
On the storage cell.. The following ommends can give you some information on the
disks .
cellcli {hit enter}
CELLCLI> Llist grid
CELLCLI> list physicaldisk detail
CELLCLI> list griddisk attributes name,asmmodestatus,asmdeactivationoutcome
CELLCLI> HELP
The last gives you the commands ..
ok
and it is list grid , not Llist
thank danny, with this Hit i am going to intitalize myself at Exadata:D
thanks for the opportunity
no problem... See whay I am concerned wit hthe HP dba team taking ove the exadat
a?
whay/why
yes...
They do not have a clue .. and if they set an event in the ASM by mistake, they
ca cause trouble with the storage cells.
yes
HALRT-02003: Data hard disk failure (Doc ID 1113013.1)
This is a guide of what may need to be done after a drive replacement;
https://support.oracle.com/epmos/faces/DocumentDisplay?id=1386147.1&parent=WIDGE
T_RECENTLY_VIEWED&sourceId=1113013.1
How to Replace a Hard Drive in an Exadata Storage Server (Hard Failure) (Doc ID
1386147.1)
Steps to shut down or reboot an Exadata storage cell without affecting ASM (Doc
ID 1188080.1)
I am going to call it done and go to sleep ...
ok
CAll me ONLY if it is something with the exadata servers:), If I'm not having a
nightmare about getting phone calls from India, I might answer .:)
after rebalance is done, is there any role for a DBA , from exadata perpective.
I now you guys can handle everything else .
any issuing of the command..
;)
thanks for the trust.
No, just monitor .. No, It is pretty much automatice ..
ok
ok, danny got it
The only thing I might do is increase the power from 2 to power 8
alter diskgroup XXXX rebalance power 8;
but you have to wath the load on the serves ...
wath/watch
yes got u
wat is the size of the disk
3TB
name: 20:10
deviceId: 18
diskType: HardDisk
enclosureDeviceId: 20
errMediaCount: 0
errOtherCount: 0
luns: 0_10
makeModel: "HITACHI H7230AS60SUN3.0T"
physicalFirmware: A310
physicalInsertTime: 2013-03-23T21:08:57-05:00
physicalInterface: sas
physicalSerial: RTDPDD
physicalSize: 2794.5199813842773G
slotNumber: 10
status: normal
increase .... as per part experience i use to give 8 or 11 at 10g and 11g
i will have a watch on cpu performance
yes, on the exadata you could go higher ...
if any issues , i will decrease it back to 2
maybe 4
k
Ok, Have a Great Day and hopefully I'll talk to you guys in 8, 9, may 10 hours f
rom now<:o)
ok
good night
ok Danny .
have great sleep

New Text Document

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

New Text Document

Diunggah oleh

Hak Cipta:

Format Tersedia

Daniel Burchett 8:34 AM

Anda mungkin juga menyukai