Formal CSR Answer

CSR creation 20.04.2015 14:13:57 (CAT)

Ericsson Local Service Organisation

Reference 2749876
Telephone
FAX
Email

Customer
Reference
Contact Person Orlando Mabuie
Telephone 258823203300
FAX
Email omabuie@mcel.co.mz

This is an automated message. You may reply to this e-mail without changes in the subject field. If you have any concerns or any further
Technical background
Network WCDMA
Service
Customized Service
Node EBS CUDB 13B
Node Customization/Adaptation
Site MTCUD02
Sub-Product
Severity Medium

CSR description:
Replacement GEP3 Board on PL_2_11

CSR Answer

1.CSR SLOGAN:
--------------------------------
Replacement GEP3 Board on PL_2_11

2.PROBLEM DESCRIPTION:
--------------------------------
Storage Engine (DS-group #3): node #3 down
Replace the affected blade PL_2_11. There is Hardware fault with Memory
in GEP3 blade of DSG3.

3.Node Info:
--------------------------------
Network : WCDMA
Node : EBS CUDB 13B

4i.Summary of Investigations:
--------------------------------
a)
MacAddress assigned to the new blade in cluster.conf was not correct
hence the PL was not accessible:

This is an automated message. You may reply to this e-mail without changes in the subject field. If you have any concerns or any further
enquiries about this CSR, please contact the Ericsson Support Office.
CUDB_21 SC_2_1# ssh PL_2_11
ssh: connect to host PL_2_11 port 1022: No route to host
CUDB_21 SC_2_1#

interface 11 eth3 ethernet 34:07:fb:f1:77:20

interface 11 eth4 ethernet 34:07:fb:f1:77:21
interface 11 eth2 ethernet 34:07:fb:f1:77:22
interface 11 eth0 ethernet 34:07:fb:f1:77:23
interface 11 eth1 ethernet 34:07:fb:f1:77:24

b)
SAF level in cudbHaState is not coming up as the rpm.conf was not having
the full .rpm information.

saAmfNodeOperState."safAmfNode=PL-11,safAmfCluster=myAmfCluster":
Disabled

SU States:
----------
safSu=PL-11,safSg=NWayActive,safApp=ERIC-CoreMW|AdminState=UNLOCKED(1)|O
perState=DISABLED(2)|PresenceState=UNINSTANTIATED(1)|ReadinessState=OUT-OF-SERVICE(1)
safSu=PL-11,safSg=NoRed,safApp=OpenSAF|AdminState=UNLOCKED(1)|OperState=
DISABLED(2)|PresenceState=UNINSTANTIATED(1)|ReadinessState=OUT-OF-SERVICE(1)
safSu=PL-11,safSg=NoRed,safApp=ERIC-CoreMW|AdminState=UNLOCKED(1)|OperSt
ate=DISABLED(2)|PresenceState=UNINSTANTIATED(1)|ReadinessState=OUT-OF-SERVICE(1)

c)
MySQL processes not running as a result of /dev/sdb1 and /dev/sdb2/ not
mounted on /local and /local2 directories respectively.

CUDB_21 SC_2_1# ./cudbPartTool check -n 11

This is an automated message. You may reply to this e-mail without changes in the subject field. If you have any concerns or any further
enquiries about this CSR, please contact the Ericsson Support Office.
CUDB partitioning tool for EBS
-= Cluster filesystem analysis =-

Payload PL_2_11 report:

ERROR: Invalid primary partitions for /local and /local2 storage.

CUDB_21 PL_2_11# df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 2.0G 1.4G 716M 66% /
root 2.0G 1.4G 716M 66% /
tmpfs 12G 696K 12G 1% /dev/shm
shm 12G 696K 12G 1% /dev/shm
192.168.0.100:/.cluster 62G 18G 41G 31% /cluster

d)
DSG 3 in Slave was unreachable from Master DSG:
Checking Replication Channels in the System:
Node | 11 | 21
====================
PLDB ___|__S2_|__M__
DSG 1 __|__M__|__S2_
DSG 2 __|__S2_|__M__
[-E-] DSG 3 __|__M__|__Xu_
DSG 4 __|__S2_|__M__

4i.Cause of problem:
--------------------------------
Blade replacement OPI not followed fully

5. Solution Description:

This is an automated message. You may reply to this e-mail without changes in the subject field. If you have any concerns or any further
enquiries about this CSR, please contact the Ericsson Support Office.
--------------------------------
a)
The MacAddress was changed in the cluster.conf using vi command and PL
blade rebooted.

interface 11 eth3 ethernet 34:07:fb:f1:77:20

interface 11 eth4 ethernet 34:07:fb:f1:77:21
interface 11 eth2 ethernet 34:07:fb:f1:77:22
interface 11 eth0 ethernet 34:07:fb:f1:77:24
interface 11 eth1 ethernet 34:07:fb:f1:77:25

After that above action, PL was accessible.

CUDB_21 SC_2_2# ssh PL_2_11
PL_2_11:~ #

b)
Copy the full .rpm in directory, /cluster/nodes/11/etc, from PL_2_12
onto PL_2_11.

After above action, SAF level was up:

saAmfNodeOperState."safAmfNode=PL-11,safAmfCluster=myAmfCluster":
Enabled

SU States:
----------
Status OK

c)
Manually mount /dev/sdb1 and /dev/sdb2/ on /local and /local2
directories respectively.

After above action all processes are up now.

This is an automated message. You may reply to this e-mail without changes in the subject field. If you have any concerns or any further
enquiries about this CSR, please contact the Ericsson Support Office.
CUDB_21 PL_2_11# dh f -h
Filesystem Size Used Avail Use% Mounted on
rootfs 2.0G 1.4G 711M 66% /
root 2.0G 1.4G 711M 66% /
tmpfs 12G 696K 12G 1% /dev/shm
shm 12G 696K 12G 1% /dev/shm
192.168.0.100:/.cluster 62G 18G 41G 31% /cluster
/dev/sdb1 138G 188M 131G 1% /local
/dev/sdb2 138G 188M 131G 1% /local2

Checking Process:
OAMs..................
Cluster Supervisor............................Running
System Monitor BC.............................Running
Reconciliation process........................Running in: OAM2
Smp-client....................................Running
Management Server Process (ndb_mgmd)..........Running
KeepAlive process.............................Running
ESA...........................................Running
LDAP counter..................................Running
Log Handler process...........................Running
PLs................................
Storage Engine process (ndbd).................Running
LDAP FE.......................................Running
KeepAlive process.............................Running
MySQL server process (Master).................Running
MySQL server process (Slave)..................Running
MySQL server process (Access).................Running
CudbNotifications process.....................Running
LDAP FE Monitor process.......................Running
DSs........................................................
Storage Engine process (ndbd).................Running

This is an automated message. You may reply to this e-mail without changes in the subject field. If you have any concerns or any further
enquiries about this CSR, please contact the Ericsson Support Office.
LDAP FE.......................................Running
KeepAlive process.............................Running
MySQL server process (Master).................Running
MySQL server process (Slave)..................Running
MySQL server process (Access).................Running
LDAP FE Monitor process.......................Running

d)
Perform a backup and restore.

This is an automated message. You may reply to this e-mail without changes in the subject field. If you have any concerns or any further
enquiries about this CSR, please contact the Ericsson Support Office.