Mark V Voter Mismatch

Posted by Ron

on 10 September, 2013 - 12:46 pm

We have had Voter Mismatch alarms on our system for a few months now. Some of the voter mismatch alarm points are going to the QD2 core and some are going to the CD core (we have over 25 voter mismatch alarms and only a handful are going to the CD core). I checked the Pre Vote data and the T core does not agree with R and S. If all of the alarms were tied to the QD2 core it would make it easier to troubleshoot, but since they are on the C and QD2 core I am confused. Any advice? Reply to this post...

Posted by CSA

on 11 September, 2013 - 9:10 am

Ron, There are 100s of Voter Mismatch Diagnostic Alarms possible on any Mark V turbine control system. They can be related to logic signals or analog signals that are not directly related to inputs or outputs. They can be related to logic or analog signals directly related to inputs or outputs. From the information provided it would seem you are referring to Voter Mismatch Diagnostic Alarms related to discreet (contact) inputs to more than one Discreet Input core (in this case <QD2> and <CD>). When there are multiple Voter Mismatch Diagnostic Alarms related to discreet inputs connected to discreet input cores that usually suggests a high amount of AC ripple riding on top of the 125 VDC being used as the wetting/interrogation voltage for the discreet input circuits. You can usually take a quality digital VOM and set it to AC mode and connect one lead to a good ground and use the other to read the AC voltage present on the positive leg of the 125 VDC voltage and then the negative leg of the 125 VDC voltage. Usually when the AC voltage exceeds approximately 40 VAC with respect to ground on either DC leg then these kinds of intermittent Voter Mismatch Diagnostic Alarms. AC usually gets induced on contact input wiring in the field when it's incorrectly run in the same conduit as motor power wiring or in the same cable tray as motor power wiring. Somewhere, there is a high-current AC source run in close proximity to 125 VDC discreet input wiring and is inducing a high AC voltage on to DC wiring. Another possible cause is that the output filter capacitors of the 125 VDC battery charger are failing and need replacing. This is more difficult to diagnose, but does happen. Also, if the unit experiences frequent intermittent 125 VDC Battery Ground Alarms, this can make the problem worse. So, you have several possibilities. None are particularly easy to find. Lastly, there was a GE TIL (Technical Information Letter) out about a PROM change to help reduce the sensitivity of the TCDA PROMsets to AC ripple; sorry I don't recall the TIL number. But, if your site hasn't kept up with Mark V TILs this could be a possible problem/solution, though an expensive one that would require getting GE involved--and they are going to HARDsell a Mark V LE (Life Extension) or Mark V Migration controls retrofit. Unless you need to add I/O capacity and/or you require additional memory and/or computational horsepower it's not necessary to replace or upgrade your Mark V using either of the recommended GE solutions. Both of those "solutions" will leave you with the same I/O cores, with the same I/O card carriers (with cards mounted upside down on the backs of the card carriers), lots of ribbon cables, and the same high density I/O terminal boards and wiring mess. If you are going to upgrade your turbine control system, go for a full Mark VIe upgrade, removing and replacing the Mark V control panel altogether. That way you will get rid of all of the Mark V hardware and get a chance to clean up the field wiring terminations. A well-planned upgrade outage should take no more time than one of those Mark V LEs or -Migrations, and in the end you will have a much cleaner installation with easier to service cards and field wiring. You will also have I/O expansion capability and more memory and computational horsepower with a full Mark VIe, instead of Mark VIe components made to fit a Mark V form factor and using ribbon cables and twenty year-old input terminal boards in a very compact 54"x20" panel in the lovable Mark V I/O cores. At any rate, please write back to let us know how you fare with solving the problems. I would suspect that unless some wiring modifications have been made in the field that you will be battling several problems at the same time: battery charger output capacitors failing, poor construction wiring practices, and TCDA PROM version. Also, if the panel experiences frequent, intermittent 125 VDC battery ground alarms that can make other problems worse. Hope this helps! Reply to this post...

Posted by Ron

on 23 September, 2013 - 12:04 pm

Thank you for the reply!!! We are only getting 2 volts AC on the DC feed and our battery voltage is balanced. I mentioned in the first post that the Voter Mismatch was coming from the CD and T cores but apparently they are isolated to the T core on DTBA and DTBB boards. I also have a BATT REF alarm in and in Diag C I am showing 17 volts for P15 and N15 for the T core. I am not sure if this is related but it does seem suspect. Reply to this post...

Posted by CSA

on 23 September, 2013 - 5:37 pm

Ron, When you say "...DC feed..." are you referring to the two-conductor cable that feeds the DTBA, and is jumpered down to the DTBB? They come from the <PD> core; I think it's from the J12C (for <QD2>) connector/fuses in the <PD> core. I think there are LEDs on the fuses (at least one of them) for the power supply (125 VDC to the discrete I/O cores) on the printed circuit card in the <PD> core, next to the J12C connector, which if I recall correctly is in the lower left corner of the printed circuit card in the <PD> cor;. (It's been a long time since I've peered into a Mark V, and I may be wrong about the designation of the power connectors, too. It's either J12x or J8x, where 'x' is A, B, or C.) Probably once you solve the 125 VDC power supply problem, the BATREF alarms will go away. I'm afraid I have some bad news about the DIAGC information. There was only one person in the world who could tell you if the DIAGC.DAT file matched the card and PROMs in your Mark V panel, and he's retired from GE. Worse, a LOT of Mark V panels were shipped with the wrong DIAGC.DAT file, and so there are a LOT of Mark Vs in existence with the wrong DIAGC.DAT file. That means the data made available via DIAGC is questionable. Sorry; but that's the truth. Anyway, hope this information helps. Again, the power for the DTBA comes from the <PD> core, via a fused output. And, it's jumpered from the DTBA to the DTBB card. Please write back to let us know how you fare! Reply to this post...

Posted by smd123 on 27 September, 2013 - 4:25 pm

CSA, After reading some of the other topics on control.com, we have concluded that this is not an issue with the 125VDC power supply to the system. No AC ripple is present, no serious grounds are present, and the voter mismatches have been (and are) persistent. In addition to the information contained in the previous post, we would like to add the following history: 1. After a lightning strike, there was damage to the MkV system. We were compelled to replace the <PD> board and PTBA in <P> core. 2. After said lighting strike, a slew of voter mismatches have been in. These mismatches are what the original post was referring to. Today, we did the following tests, with the <T> core showing a voter mismatch: 1. Confirmed all fuses on <PD> are good. 2. Measured 125VDC across J12/JY on DTBA/DTBB 3. Confirmed that all 4 fuses on TCPS card in T core are good. 4. Confirmed readings on test points on TCPS in T core card 5. I cannot confirm, but it is believed that at this point the LED's on R and S were flashing in synch, and T was the "odd man out". No IOnet errors are present. All cables are firmly secured and no damage apparent. We do have a tube of the conductive grease and have been inspecting connectors carefully as we work with them. Once we were convinced that the issue wasn't a lack of power, we swapped TCDA boards between S and T. This had the following results, and is our current state: 1. A slew of voter mismatches came in on the <R> core. 2. The LED patterns on S and T now agree. <R> is a pattern on its own, but matches R, S, and T on our other MkV unit at the site. 3. The voltages on jumper JP match between S and T. R matches RST on our other unit. Swapping the cards back made no difference. We haven't been able to actually toggle an input on <R> yet and see the pre-vote data, to confirm that it is reading correctly. However, all of our other checks indicate that <R> is correct. For reference, the LED pattern on <R> is 11000000 to 11110000. S and T are both going from 11000000 to 11101101. Our conclusion at the moment is that the TCPS card has some kind of issue which is damaging the TCDA cards and that currently both <S> and <T> are dead. Does this sound at all reasonable? What else can we do to troubleshoot this? Thanks! Reply to this post...

Posted by bicycle mechanic on 27 September, 2013 - 8:52 pm

It sounds like an EEPROM on a board was affected by that lighting strike. I would do, a "down load all" to the cores and reboot.

If that doesn't work, start looking to replace EEPROM or proms to solve the problem. I would exhaust those possibility first. Also, the dual ported ram on the SDCC. Some of these were soldered and some socketed.

Reply to this post...

Posted by Ron

on 28 September, 2013 - 10:19 am

We have been instructed to download to "user" and NEVER download to "all" as this would erase the counters for fired hours, starts etc. We did perform a EEprom download to "user" which did not resolve our issue. If downloading to "all" is the solution, then we will have no choice but I would like to be sure before we erase the counters. Any thoughts? Thank you for the reply! Reply to this post...

Posted by bicycle mechanic on 28 September, 2013 - 8:35 pm

I am a bit fuzzy as to lost counter info by doing a "down load all". If the current running programs were never backed up somewhere, and you reverted to as shipped settings then lost of counter info possible among other things. (control constants) Try uploading first, save in a file and quarantined. IF you have A7 status on all cores and still voter mismatch, with newly downloaded files, it's indication of SDCC problem. Or better yet! before doing all that, use the terminal interface monitor (TIM) and see what cores have programs loaded and running, and status of memory segments. Start with TIM! Reply to this post...

Posted by CSA

on 29 September, 2013 - 12:30 pm

The ALL parameter includes FORMAT (which effectively erases the EEPROM) and TOTD (Totalizer Data). If the Totalizer Data ISN'T uploaded before it is downloaded then old, or even blank Totalizer Data will be downloaded with ALL. I've been ruminating on this, and while it seems like the power supply to the <T> TCDA seems to be causing the TCDA in Loc. 3 to fail (as indicated by the "failure" of the card swapped in to Loc. 3) it's not conclusive (at least to my thought process). It's difficult to conceive how the power supply cable could be damaged or be causing a card failure. What does the SLCC display say the I/O States of all of <T>'s complement of cards is? Is <T> (and <S>) at A7--but the TCDA card(s) is (are) at something other than A7? This would mean that at some point the TCDA was at A6/A7 then it went to some other level. I don't know what TIMN would tell us, and it would be necessary to be able to see the file to analyze it, and GE, in their consistently inconsistent way, doesn't document all of the information available in a TIMM session. I believe what Bicycle Mechanic is suggesting is to set up a computer using a serial cable connected to <T>'s TIMN port on the QTBA with the power to <T> off, then apply power to <T> and capture the boot-up sequence. The LED bar-graphs on the three TCDAs should be flashing in the same sequence; the fact that they aren't indicates there are Diagnostic Alarms on the card--which you have indicated. <Q> SDCCs communicate with TCDAs via the IONET, which connects to the respective TCEA in <P>. This revelation about a damaging lightning strike is causing me to have some questions. Actually I'm at loss because, for me, everything isn't adding up to a likely scenario. So, it seems the TCDA in Loc. 3 is "failing" for some reason. There IS 125 VDC at the DTBA/DTBB, but the TCDA in Loc. 3 didn't see it. And Voter Mismatch Diagnostic Alarms are toggling at a periodic rate. I would be looking at the power supply to the TCDA in Loc. 3 as well as the IONET from <T> to the TCEA in Loc. 5 of <P> to the TCDA in Loc. 3. When you swap TCDAs in a digital I/O core it's NOT necessary to download anything to either card--they have the same info (they are redundant). I wonder if some other cards (the TCEA Loc. 5?) weren't "disturbed" during the lightning troubleshooting. But that doesn't explain why TCDAs are "failing".... Again, something isn't adding up here. I'll keep thinking about what might be the problem, but I would suggest the IONET from <T> as well as the power supply cable from <T>'s TCPS be checked along their entire lengths to be sure something isn't amiss. Please write back to let us know how you progress. Reply to this post...

Posted by smd123

on 30 September, 2013 - 8:09 am

With laptop connected to <T> core via serial cable, we turned <T> on. No errors during bootup. In addition, TIMN shows that <T> and all of its IO cards are all at A7. No system errors present. Same for S, R. The only indicators we have found thus far are the persistent voter mismatch alarms and BATREF alarm in DiagC. If you are familiar with the DTBx boards, there is a small bank of 3 relays. Our understanding is that these are energized to provide the 125VDC wetting voltage to the TCDA cards. Currently, KS and KT are de-energized. So we are trying to figure out what would prevent these relays from being picked up. This issue must be at a very low level, since everything is at A7. It almost seems like a 24VDC power cable is broken somewhere for picking up the relays on DTBx boards. -----update----After wondering if some cap/inductive filter on the TCPS was bad, sending a startup surge to the TCDA, we pulled the TCPS board to inspect. We found that FU4 is actually blown. Previously, when we checked the fuses, we measured voltage on each side and continuity with the ohm meter (with core powered down). Due to the difficulty of seeing FU4, we didn't notice its condition until the board was out. However, with the fuse pulled, there is still continuity across the fuse clip (!). -----/update----Here is the log for <T> from TIMN. Note that we had <S> powered down for the moment. <<< SDCq GHD Version 1.3 186 restart >>> DCC/TCQC stall test started <<< SDCq GHD Version 1.3 186 restart >>> DCC/TCQC stall test passed DCC-RAM passed 1K DPRAM found at: 5000 : 0000 2K DPRAM found at: 5200 : 0000 2K DPRAM found at: 5400 : 0000 DPRAM unpopulated at: 5480 : 0000 DPRAM unpopulated at: 5500 : 0000 DPRAM unpopulated at: 5580 : 0000 <<< DCC-186 initialized >>> ----- SDCq GHD Version 1.3 Mark 5 Monitor <T> ----<Q> DCC 186: Oct 30 1997 at 15:18:57 BBL library: Oct 30 1997 at 15:17:58 Current time: Sep 30 2013 - 7:30:54 #S ----- Agent Status -----

Status> I obj card maj min I/O cfg qst cfg cfg eep rsp msg msg ID name rev rev stat stat ID flgs tics ofst type seq stat ----------------------------------------------------------------1 TCQA 02 06 A7 D0 22 01 0001 4200 05 0007 00 4 TCD1 03 09 A7 D0 21 01 0060 4280 05 0011 00 5 TCD2 03 09 A7 D0 21 01 0060 42AC 05 0013 00 8 LCCq 04 04 A7 D0 5 01 5320 42FB 05 0016 00 12 DCCq 01 03 A7 D0 27 01 0001 4322 05 0003 00 13 IOMA 04 05 A7 D0 21 01 0020 437E 05 000B 00 15 TCE1 05 03 A7 D0 21 01 0040 43C1 05 000D 00 Status> E ---- System error counters ---:- No system errors detected Status> V Agent: T I/O state: A7 Conforming: 03 Voter busy: 53%

I/O ready: yes Frame Rate: 16 Hz DCC voter: LCC voter: State: 0400 State: 0430

Agent rdy: yes Voting Mode: TMR Active: 0 BMS OK: 0

Voting Agents: CR-T

Status: 0 Status: 1

VDP: R TDP: C Present: R:2F S:00 Time stats: Status: 00 OK cntr: 1406 Vote stats: Status: 03 OK cntr: 1215 Voter buffs: Current: 5 Total: 6480 Misc counters: DCC OK: 1202 DCC aborted: 0 LCC degenerate votes: 1202 DCC stand alone passes: 0 LCC sequence error: 0 LCC lost time sync: 27 DCC frame_overrun: 0 Status>
Reply to this post...

T:0F - C:CF D:00 missed cntr: 2 missed cntr: 0 Bad type: 0

Posted by CSA

on 30 September, 2013 - 12:07 pm

sd123, Thanks for the information. I, too, have measured low resistance ("continuity") between fuse clips on TCPS cards when the fuse was not installed. Without having any elementary (as GE calls schematic drawings) for the cards I have to say there must be parallel paths which cause this. I do know that some circuits on the <P> core have leakage currents that make it appear as though circuits are energized (contacts closed) when they are not, which lead to a lot of head-scratching once. I don't have access to any TCPS drawings at this writing, but there is one small fuse (glass ferrule type) on the TCPS that is well hidden in the lower left corner, if I recall correctly, and until the card is removed it is very difficult to see if the fuse if blown or not. And, it's also very difficult to measure the presence or absence of voltage across the fuse clips when the card is installed in Loc. 5 of any of the processor cores. Unfortunately, with this particlar fuse, it's really necessary to remove the card and remove the fuse and measure the fuse resistance to see if it's intact or blown as the fuse element is almost micro-thin and the glass ferrule is very short and the open portion of he fuse can be hidden behind the metal cover on either end of the fuse (another painful lesson--"It doesn't LOOK like it's blown, so it must NOT be blown, and there is "continuity" [low resistance but not 0 ohms] between the fuse clips!"). Also, unfortunately when a lightning strike occurs all of the collateral damage can be difficult to find quickly. Since the DTBx cards are common to all three cores (for a <QDn> core), I don't think the wetting voltage can be segregated for particular cores--it's always applied to the contacts which are then "fanned out" to the individual TCDA cards. It might be that the wetting voltage isn't applied to the TCDA (or sensed by the TCDA) unless the respective relay on the DTBx card is energized, and if there isn't voltage from the TCPS card to do that then the TCDA can't see the wetting voltage and hence the BATREF low voltage alarm(s) get annunciated. However, it's also hard to imagine--and disconcerting, too--how the absence of power from one fuse can "fail" a TCDA and then substituting a good TCDA into that Loc. can cause another TCDA to fail, or how moving a TCDA from a Loc. which isn't getting power from a fuse on a TCPS into a Loc. which IS getting power from a fuse on a TCPS can "fail." I guess its possible the problem is the TCDA card which was in Loc. 3 caused the fuse in <T> to blow, and when swapped into Loc. 2 has also caused the same fuse on the TCPS in <S> to blow. And it's hard to believe this would allow a TCDA to get to A6/A7 if it can't see the wetting voltage. A6 is the I/O State that says the card is synchronized to the DCC/SDCC is ready to enable it's outputs. A7 means the outputs are enabled. A processor won't go to A7 until all of it's cards get to A6 and then they will all go to A7, and one or more cards can then "drop out" of A7 but the processor can still remain at A7.... Please let us know how the troubleshooting goes. It would certainly seem the TCPS(s), and or the fuse(s), are suspect--at a minimum. But, the TCPS cable from <T> to Loc. 3 of the <QDn> core, or the TCDA that was originally in Loc. 3 of that digital I/O core could also be suspect. Reply to this post...

Posted by Medley

on 30 September, 2013 - 3:55 am

Hi Ron, I recently came across a case which is similar to yours. We first started by measuring the panel power supply. For our case, the AC ripple were found on all voltage levels, this may be the root cause of problem but why was it only affecting a core? so we continued our investigation.

Our customers were just like you, swapped a card and had another core corrupted (i would say), both now failed to boot to A7. Long story cut short, our head engineer decided to reprogram the EPROMs of TCQA & TCDA (Data in EPROMs might have been corrupted) and that solved the former part, but not the latter part- voter mismatch. In the end,we found out the problem was the power supply from TCPS to <QD>TCDA. So we replaced TCPS card, and everything works like a charm. I would suggest checking TCPS card, and TCDA & TCQA EPROMS. Happy to share, thank you. Reply to this post...

Posted by bicyclemechanic on 30 September, 2013 - 11:34 am

I am looking at the checks thus far. A thought, have you disconnected 125V supply to the DTBA boards. What if, you have a condition that is bringing that entire 125V core circuit down upon boot up, which is not there when program is not running. I would disconnect the 125vdc plug on the terminal boards and see if all processors become stable. I recall those boards having a resistor bridge network(I could be wrong),but if main into that net work was lost expected signals would be different from one core to another. If system permits, disconnect across cores and compare. Reply to this post...

Posted by bicycle mechanic on 30 September, 2013 - 5:10 pm

BY the way, TCEA y&Z are out! Reply to this post...

Posted by G.Rajesh

on 2 October, 2013 - 1:08 am

Waiting for the thread owner feedback to know more.