CES Logging & Debug Level Logs User Guide

CallAnalyst Enterprise Server

Version 3.1

Logging, Debug Level Logs

User Guide

CES Logging & Debug Level Logs User Guide

Table of Contents

Prerequisite to handle data loss and data corruption ........................................................ 3

Essential .ini configuration to handle data loss ..................................................................... 4
MSRCDM - msrcdm_rawCDR<site-code>_<number>.txt file .............................................. 5
Import Raw CDR tool - msrcdm_rawCDR<site-code>_<number>.txt file ................. 7
MSDM - msdm_structuredCDR<number>.txt file................................................................... 8
Import structured CDR tool - msdm_structuredCDR<number>.txt file ...................... 9
MSDM - msdm_FailedToWriteToDB<date>.txt file. .............................................................. 10
Import structured CDR tool - msdm_FailedToWriteToDB<date>.txt file ................... 12
MSDM - msdm_baddata<date>.txt file. .............................................................................. 13

CES Logging & Debug Level Logs User Guide

Prerequisite to handle data loss and data corruption

In case of a shutdown of MSRCDM, either via MSPM or directly, the data in the buffer is lost. In
order to recover the lost calls that were contained in buffer, one must ensure that all the calls
initiated prior to MSRCDM shutdown are captured in MSRCDM log file.
In addition, operational shutdown of MSRCDM should be a rare and planned event. It must be
done during the off peak hour of the site (assumption is that the call volume is very low and calls
if missed are not important for the business). Note that if MSRCDM is shutdown, raw data is not
being collected and logged as mentioned earlier. This data loss is not recoverable.
Also note that stopping Multi Site Process Manager service will stop MSAdvPrs.exe, MSDM.exe,
MSPM.exe, msrcdm.exe and raw data is not being collected. When Multi Site Process Manager
service is in the stopped state, one cannot even start/stop CallAnalyst Enterprise Server Health
Monitor. So as a result starting/stopping msrcdm.exe via CallAnalyst Enterprise Server Health
Monitor requires Multi Site Process Manager service in the start state.

CES Logging & Debug Level Logs User Guide

Essential .ini configuration to handle data loss

In order to handle data loss and corruption of data in CES, one should ensure that the following
ini entries are configured appropriately.
Locate C:\Program Files\TriVium\cdm.ini file to look for ConnTimeout & MSDMConnTimeout
values.
The ConnTimeout entry is used to check the connection status between CallAnalyst Enterprise
Server application and the SQL Server database. If the value set is 0 then it checks the
connection status every second and tries to connect if there is a connection failure. If you set
value=600, then it means every 10min CallAnalyst Enterprise Server application will try to connect
to SQL Server database. Default value is set to 600 sec.
The MSDMConnTimeout entry is used to check the connection status between MSDM (Multi Site
Data Manager) and the SQL Server database. If the value set is 0 then it checks the
connection status every second and tries to connect if there is a connection failure. If you set
value=600, then it means every 10min MSDM (Multi Site Data Manager) will try to connect to SQL
Server database. Default value is set to 10 sec.

Locate C:\Program Files\TriVium\cdm.ini file to look for SQLEXPRESS restart delay value.

Locate C:\Program Files\TriVium\cdm.ini file to set value to 11 for low level logs to enable.
The different logs are MSPM, MSDM, MSRCDM, ASAdvPrs and by default these value are set to
3.

CES Logging & Debug Level Logs User Guide

Now all log related files are placed under one folder called Logs under TriVium folder. Locate
C:\Program Files\TriVium\Logs folder, which contains MSRCDM, MSDM, MsAdvPrs &
MSDM_FialedToWriteDB log files.

MSRCDM - msrcdm_rawCDR<site-code>_<number>.txt file

The basic principle for addressing data loss issues is that the solution must guarantee that once
data is received from the switch it MUST either get into the database or be available in a file to
be imported offline.
As soon as MSRCDM gets the raw data from the switch, before it does anything else with the
data or acknowledge receipt of data from the switch, we are writing the raw CDR into a
msrcdm_rawCDR<site-code>_<number>.txt (note site-code should be used not site id) log.
Also this log is a circular log but the number of files (defaulted to say 40, if it is set to 0 then the
raw CDR should not be logged with the assumption that the customer does not care about
recovery from loss of data) forming the circular log will be configurable, so will the size of each
file (defaulted to 5MB).
Locate C:\Program Files\TriVium\cdm.ini file to configure above values.

CES Logging & Debug Level Logs User Guide

The directory in which this log file is kept is also configurable so that an install can select a
location with sufficient space for the amount of data it wants to keep just in case it needs to
recover from data loss.
Once MSRCDM reads the raw CDR it goes to SMDR for parsing. In case of any failure in parsing,
the error is logged along with the raw CDR that failed. The error is clear indicative of the exact
failure and any failure of parsing should result in only that CDR being ignored and MSRCDM
continuing normally.
The MSRCDM then sends the parsed structured CDR to either a buffer (for purposes of collation)
or to the MSDM via a UDP socket connection. Also, any socket error, CES is handled in a way
that the MSRCDM continues normally and moves on to the next data.

Locate C:\Program Files\TriVium\BACKUP\Logs\msrcdm_rawCDR1_1.txt file to view the raw

CDR.

CES Logging & Debug Level Logs User Guide

Import Raw CDR tool - msrcdm_rawCDR<site-code>_<number>.txt

file
Since raw CDRs are available, the basic solution of recovery of data will depend on importing
from the raw CDR using the existing import tool. Since duplicate checking will be done in the
MSDM, the assumption is that one should import for the whole day or at least from some time
before data loss started (as per the MSRCDM log) and sometime after that.

Click Start->Program Files->TriVium->CES, select Tools->Import Raw CDR and Import Raw CDR
window will open.

Browse to the location where Raw CDRs are available, click Import button to proceed and
you will see a Finished Importing Call Records window to confirm raw CDRs are imported into
the CES.

CES Logging & Debug Level Logs User Guide

Note: Since Raw CDR files contain entire/complete data from the phone system, before
importing raw CDRs into CES, it is recommended that one should find out the missing data either
from the database or from the CES. Now select only missing data from the raw CDRs and import
CDRs in to CES, thus we can reduce duplicate check happen at MSDM level.

MSDM - msdm_structuredCDR<number>.txt file

Since importing data offline can result in duplicates, hence duplicate detection will be done at
the MSDM level. Duplicate detection at MSDM level is anyways the best place to avoid
duplicates in the database.
MSDM receives data sent by multiple MSRCDMs (one per site, i.e., PBX). If there is a failure of
receiving data or incomplete data is received, MSDM logs indicate the failure (and also the fact
that this could possibly mean data loss; if partial data was received also logged).
If the data received was complete, the first thing as in the MSRCDM that MSDM logs the
structured CDR in a set of log files. This log is different than the present MSDM log. Also this is a
circular log but the number of files forming the circular log is a configurable (defaulted to say 40,
if it is set to 0 then the structured CDR should not be logged with the assumption that the
customer does not care about recovery from loss of data), so will the size of each file (same as
for MSRCDM). The directory in which this log file is kept will also be configurable (same as for
MSRCDM) so that an install can select a location with sufficient space for the amount of data it
wants to keep just in case it needs to recover from data loss. By default the same directory as
used
by
the
normal
MSRCDM
log
will
be
used.
The
file
name
is
msdm_structuredCDR<number>.txt.
Locate C:\Program Files\TriVium\BACKUP\Logs\MSDM_StructuredData folder to view the logs.

Locate C:\Program Files\TriVium\BACKUP\Logs\MSDM_StructuredData_1.txt file to view the

data.
8

CES Logging & Debug Level Logs User Guide

Import structured CDR tool - msdm_structuredCDR<number>.txt file

Since structured CDRs are available, the basic solution of recovery of data will depend on
importing from the structured CDR using the existing Import structured CDR tool. Since
duplicate checking will be done in the MSDM, the assumption is that one should import for the
whole day or at least from some time before data loss started (as per the MSRCDM log) and
sometime after that.

Click Start->Program Files->TriVium->CES, select Tools->Import Structured CDR and Import

Structured Data window will open.

Browse to the location where Structured CDRs are available, click Import button to proceed
and you will see an Import Structured Data window to confirm Structured CDRs are imported
into the CES.

CES Logging & Debug Level Logs User Guide

MSDM - msdm_FailedToWriteToDB<date>.txt file.

MSDM Handling Duplicate Call:
MSDM puts these records in essentially an array. In addition, MSDM checks if the record is
duplicate or not. If duplicate it ignores the record and continues normally, else it keeps it in the
array. In case it is a duplicate, an appropriate log entry is made in the std. MSDM log file along
with the structured CDR. Since duplicate can be a fairly normal thing, the log entry is made at a
particular log level corresponding to warning (or informational but certainly not error which
should be the default level for log entries). Since this is a non-error type and not a data loss
situation, MSDM should not inform Call Alarm either.
In case the array is filled or a timer expires, the data will be flushed to the database. The records
are copied from the array into a record set and then flushed to the DB using bulk update. To do
this, MSDM has opened an appropriate database (DB) connection and a record set earlier. It is
possible that the database is unavailable, and hence this failed.
In such a case, MSDM however continues to receive data normally from the MSRCDM and keep
processing as usual, assuming that the database outage is temporary and it will keep trying to
connect depending on the value set in the ca.ini file (MSDMConnTimeout).

10

CES Logging & Debug Level Logs User Guide

Once the record set is populated, a DB bulk update is done. The bulk update is actually a nontransactional bulk insert. In case bulk update fails for any reason, typically either because data
has characters, which should normally be escaped or SQL has problems and database is
unavailable, we go back sanitize the data for use with SQL in the array, close the DB
connection, re-open the DB connection and retry the bulk.
If failure repeats, then the array is dumped (appended) into another log file,
msdm_FailedToWriteToDB<date>.txt, and MSDM then proceeds as normal to get more data
from MSRCDM.
Locate C:\Program Files\TriVium\Logs\MSDM_FailedToWriteToDB111809.txt file to view the logs.

11

CES Logging & Debug Level Logs User Guide

In case of failure to do bulk update to the DB, there is an entry in the std. MSDM log file to
indicate this along with the information that the records are available in the following <file>, i.e.,
the msdm_FailedToWriteToDB<date>.txt .

Import structured CDR tool - msdm_FailedToWriteToDB<date>.txt file

There is a tool similar to the present tool to import from raw CDR file, added to import from a file,
which has structured CDR data. This tool will also ensure duplicates are detected, and ignored.
The tool is also leave a log of which records got into the database and which were duplicates
and which has some other.
So most of the time, expectation of data recovery is when data loss is due to unavailability of
database, when the database becomes available, one would use the offline tool (available
from CA application) to load from the appropriate msdm_FailedToWriteToDB<date>.txt file.

12

CES Logging & Debug Level Logs User Guide

Click Start->Program Files->TriVium->CES, select Tools->Import Structured CDR and Import

Structured Data window will open.

Browse to the location where MSDM_FailedToWriteToDB log files are available, click Import
button to proceed and you will see an Import Structured Data window to confirm Structured
CDRs are imported into the CES.

MSDM - msdm_baddata<date>.txt file.

When the records are put from the array into the record set, it also detects if there is type
mismatch with the data type in the database, e.g., we have alphanumeric data and numeric is
what is defined in the database. If this error happens, the particular record is ignored and MSDM
continues. In this case we are making a log entry in the std. MSDM. In addition, we are logging
13

CES Logging & Debug Level Logs User Guide

the bad structured CDR in a separate log file msdm_baddata<date>.txt. This file is created in
the same directory as msdm_structuredCDR<number>.txt. Also, this file should be one per day.

Note:
1. Currently we are not able to import msdm_baddata<date>.txt file using import tool
available in the CES.

14