Publish or Perish: An Evaluation of the Quality, Quantity, Ethics and Review Process of IEEE/PES Publications:

A Summary of the Power Globe Discussion

Paulo F. Ribeiro, Fellow, IEEE
un-reflected idea publishable and impressive, and thanks to advanced word processing, produce papers by the bundles. These developments are the background and source for much of the rubbish we produce in the name of scholarship and scientific / technical papers. This author does not exclude himself from contributing to this chaotic situation. But the problem becomes even more complex and disturbing when a soft form of plagiarism becomes part of the academic / research culture, when individuals are searching for any new ideas from wherever source they can get them and consciously or un-consciously neglecting or not acknowledging the possible sources or information. Soft plagiarists become academic alchemists who must turn (base metal into gold) other people's idea into their own. The sad thing, however, is that the system is not sensible enough to detect it and apparent success is possible. It is important to recognize that the number of papers published by an individual has very little relationship with their significance and contribution to the real academic, research and professional world. We must avoid replacing the true search for originality by a soft form of plagiarism, and act more consistently and ethically. A literature review [1-8] shows that the concern is well established. Recommendations for addressing different aspects have been made by several technical societies. II.
THE ISSUES FROM DIFFERENT PERCEPTIONS

Abstract- Every year, the number of papers submitted to the Power Engineering Society increases. This growth, which is in part the result of the pressure to publish, is straining the publication process, overloading the pool of reviewers, and creating the impression that the quality of publications is diminishing. This paper summarizes the informal discussions which have taken place on the Power Globe Listserve and shows the level of concern among members of the power engineering community. The paper covers aspects of the peer review process in PES, including present IEEE/PES publication policies, ethical aspects of the review process, technologies for aiding the review process, topics related to value of the technical content, graphics and peer review, and more. The several contributions make constructive suggestions which will hopefully improve the PES peer review process and the quality of the publications. Index Terms: Publications, peer review, technical papers, power engineering society.

I. INTRODUCTION The topic of the increasing number of papers (and the perception of decreasing quality) was discussed on the PowerGlobe Listserve. First, it was proposed that the "publish or perish" environment of the academic world tends to cause more papers to be submitted for publication. This tendency, plus the societal survivalist competition we all face have encouraged a culture of a so called soft plagiarism. We borrow from our own work, adding little new, and we may even build on the work of others without giving credit. These forms of plagiarism are not always easily detected due to the sophistication of the performers. These symptoms are added to our natural tendency to seek relevance and prominence among our peers and are usually disguised as the so called "search for originality." This type of originality we seek, encouraged by the university system, and which is regard as a sign of wealth, is in reality a confession of immaturity. One or two good paper resulting from a graduate degree used to make any advisor very happy. Nowadays this small number is unimaginable and students are forced to make any

The initial post summarized in the Introduction generated a number of excellent replies summarized below. (Due to space limitation only a fraction of the public replies are summarized.) A. The Academic Environment The first debater affirmed the seriousness of the issue and stated that The present situation is due to a number of factors. Proliferation of number of journals seeking papers to publish aggravates mediocrity and soft plagiarism. Insistence on the number of publications for getting promotion and research grants causes a rat race leading to the present situation. There was a time when the title of Senior lecturer was coveted and honored as a position as a full professor. In many instances there was only one professorship available. If

P. F. Ribeiro is with Calvin College, Engineering Department, Grand Rapids, MI 49546, A (e-mail: pfribeiro@ieee.org).

2008 IEEE.

financial benefits are decoupled from the title, some remedy is possible. Consideration for promotion and tenure is supposed to include factors such as teaching excellence, publications, funding, service, and collegiality. But the sad fact is that funding and publications decide the final outcome in many instances. Academic community experienced violence and murder in some instances. The subject has been a blot on the academic community for the last few decades and 'everyone talks about it but nothing is done about it' [9] B. A Historical Viewpoint Another contributor reflected on the historical developments leading to the present reality: . . . before we get too involved in complaining about it, let us look at how we got into this situation. I think there is a tendency to view the more remote past of our profession as a sort of golden age, and our intellectual ancestors as engineering superheroes: few in number, but capable of great accomplishments. Was it really like that, I wonder, or are our views clouded by the historical separation? Well, of course, there were some very great men back then, a century and a half ago. Their names are known to all of us: they are used for many of the units we use. A good many such luminaries, from several countries, have left their names to posterity this way. With perhaps a hint of envy, we acknowledge that few of the papers that we see published today have the potential to be as important as the product of those giants on whose shoulders we stand. Many of the people who made these early and great contributions were working in universities: Kelvin, for example, was at Glasgow University. Maxwell was at Kings College, London. It wasnt that the scientists of the time were well-educated, independently wealthy people working alone. The situation was, in fact, not enormously different than it is today. There are a couple of identifiable reasons for the changes. First, whether in academia or not, the researchers which were in natural philosophy (or physics) of 150 years ago faced a world where very little was known and understood about electricity. Thus, one reason for the importance of the contributions of the mid-nineteenth century was that, by definition, they laid the groundwork for all that has followed. Second, the competition was not numerous. Because there were so relatively few people involved, the papers seem special. But then times began to change. In 1878, William Ayrton, who had studied under Kelvin, returned to England from a spell in Japan, where he had established the first electrical engineering course in the world, at the Imperial College of Engineering in Tokyo. Back in London, at the City and Guilds Institute, he started to teach classes to electricians, or (as we would call them), electrical engineers. Along with a colleague, W.C. Haycraft, he wrote a paper showing how electrical calorimetry could accomplish in a matter of a few minutes what physicists had been occupied with for the previous few years, determining the value of what we nowadays call Joules

constant. The paper was A Students Simple Apparatus for Determining the Mechanical Equivalent of Heat, read before the Royal Society in 1894. This was shocking stuff, and the world of physics went ballistic. This paper marks the moment when electrical engineering split away from physics. To make matters worse, Ayrton was teaching electrical engineering at night-school, because the students had to earn a living during the day. In other words, by the 1890's, the business of electrical engineering was getting democratized. As the generations have passed since then, the number of electrical engineers has increased, and the amount of terra incognita remaining for exploration has diminished. Until we find ourselves in the present circumstance. . . It is my contention that the proliferation of papers that results was inevitable -- we must learn to live with it. So what do we do about it now? Well, about all we can do is to concentrate on the quality of the papers we publish. [10] C. Are We Just Moaning? A more optimistic view of the situation and was taken by another commentator that observed: Ancient Romans used to say O tempora! O mores! which is a moaning What bad times, what bad customs. Older generation has always had a tendency to moan about the current situation and hark back to good old times. While it is unquestionable that the "publish or perish pressure has caused an inflation of journals and papers of sometimes not very good quality, I also believe that current PhD students work much harder than my generation did. It used to be the case that PhD was treated as a goal in itself and publications were not strictly required. We should applaud the current generation of PhD students for striving to publish their research, not deride them. After all, a definition of a PhD work (at least my definition) is that the research must be of a publishable quality. If a PhD does not result in a journal publication (or at least a couple of good conference papers), it's either because it was so poor that it should not have been awarded in the first place, or because the PhD student was too lazy to do it (or more likely their supervisor too lazy to press for it). If a research is not published, who will know about it and therefore what good was it for? So well done, current PhD students; work hard and publish a lot! As for making sure that journal papers are indeed truly original, it's the task for journal editors and reviewers. Obviously, the standards must be maintained. And I believe that the standards are maintained, at least in the top 2-3 journals. Of course there seems to be a proliferation of second-rate journals but I don't have time to read them so I can't comment. [11] D. The Review Process and the Metrics Others looked into the dynamics of the review process and commented: . . . that the reviewers are to blame, but so too are the societies for not using technology to find papers of significant similarity for special review.

All reviews are accomplished by a small group of people, those willing to review. As industry leaves the process, as it is too busy keeping a job, it will get worse. Since no one wishes to pay for universities, especially the US, we are in for hard times. Hard times lead to insider groups, protectionism, and inexplicit favors in exchange. We all know what we will do to survive, if left to that low level of achievement. Given the exponential growth of information, I would suggest that the present paper publication process, based on the paper technology of the last century, should be replaced. [12] Another contributor said: . . . Who has the time anymore to spend a half a day or more per paper on a review? Thats what it takes to do a quality review. One way out for faculty is to ask graduate students to do them. Its a great educational opportunity if done right but again, it takes time. And time is in increasingly short supply. As a corollary and in my opinion, many people who review papers are not as competent or as conscientious as in the past and this is not always because of time constraints. If you are educated in a system where sloppy writing practices and poor understanding of engineering design practices (e.g., poor use of English, use of simulation as a substitute for mathematical proof, etc.) are tolerated, thats probably what you will bring to the table when reviewing the works of others. Editors have to take responsibility for selecting good reviewers not just the ones who will take on the job. And the educational system has to take responsibility for training good reviewers. But thats not my main concern. There is another problem that engineers in academia have. That is, engineers in academia are judged against metrics developed to judge scientists, not engineers. In his IEEE Proceedings article (6) William Wulf, President of the National Academy of Engineering said about engineers in academia, Unlike engineering, every other creative field on campus expects their faculty to practice/perform. Even if you do not buy the argument that engineering is creative in the same way as art or music, you can look to the professions of medicine and law, which also expect their faculties to practice the field they teach. Can you imagine a medical school where the faculty was prohibited from practicing medicine? Yet, this not so in engineering. Engineering faculty are, for the most part, judged by the same criteria as science faculty and the practice of engineering is not one of those criteria. The faculty reward system recognizes teaching, research, and service, but not delivering a marketable product or process, or designing an enduring piece of the nations infrastructure. The criteria for promotion and tenure make it hard to hire and reward people with such experience, even though it would be valuable for students. In many cases, even taking a sabbatical in industry may be risky for faculty because it does not contribute to the usual resume-building activities of an academic. I agree. Perhaps changing the performance metric for promotion in academia to something relevant to what

engineers do (or should be doing) would solve part of the publishing problem for engineering. As Charlie Gross [14] and others have said, faculty are smart people. Give them rules and they will play to win unfortunately sometimes ethics get stretched in the process. [13] E. The Pressure versus the Ethics The financial and professional pressure and the consequent stretching the ethics are the more difficult issues to resolve as one commentator noticed: I think the prime mover behind much of the pressure to publish may be money. At [some schools], to be a successful engineering professor, the target number at present is $300k in annual extramural funding. Untenured professors who fail to meet this goal are in real danger of being terminated which can end their academic career. Young professors entering the field for the most part, are brilliant, hard-working, ethical, and dedicated. But they are also realists and survivors. You define the game and they will figure out how to win at it. Acquiring funding requires obtaining contracts. Getting contracts requires publications. Unfortunately, quantity as well as quality counts. Faced with publish or perish alternatives, they respond rationally. They publish whatever and wherever they can. Can we really blame them? After all, they didn't create the system; they're just trying to survive in it. It has been said that 95% of the IEEE Transactions could be obliterated, and the profession wouldn't be affected at all. It's just that we're not sure which 95%. Perhaps the criticism is unfair; on the other hand, maybe the estimate is too low. We've all been disappointed to discover, that having dissected a journal paper, that there was essentially nothing there. On a positive note, virtually all of our colleagues individuals I've had the privilege of really getting know have been hard-working, ethical, dedicated, and trying to make the system better. The quality of journal publications can only be improved when we find some way to reduce the extreme pressure on faculty and graduate students to publish volumes of papers. Until then, given the ever-increasing number of graduate students worldwide, the ever-increasing expectations for promotion and tenure, an ever-increasing number of papers are inevitable. [14] F. Responsibilities and Recommendations Another commentator talked about responsibilities, listed common problems and made a number of detailed suggestions: The quality of our technical journals is the responsibility of the editors who have the last word in what gets published. They have a tough job though because in addition to maintaining the technical integrity of their journals, they must also be sensitive to the desires and needs of the technical community. If for example, the technical community isn't clear on what it considers to be a significant contribution, then the editors won't have a clear mandate for setting the bar for acceptance. As a reviewer, I am frequently disappointed by the papers I am asked to review. Here are common problems:

The paper is incomplete and a knowledgeable reader simply cannot use or reproduce the results 2. The paper is simply a report on an experiment or a calculation which was of interest to the author but is of no particular interest to anyone else 3. The paper is a rehash of existing, readily available knowledge 4. The paper is technically unsound 5. The paper is poorly written to the point that a reader cannot understand it In addition to being reasonably well written, here are examples of things that I like to see in a paper: 1. It presents a novel approach to a new or existing problem 2. It presents experimental results that can be of use to others in the technical community 3. It presents a new experimental or measurement technique 4. It presents a new application of existing technology 5. It presents new theoretical approaches or extends existing ones 6. It presents a totally new technology I would like to suggest that all of us can contribute to improving the quality of journals. Perhaps guidelines such as the following might be helpful: Senior faculty, managers, etc: I. Don't encourage publication simply for the sake of numbers. II. Evaluate publications on their technical merit. Certainly, the quality of publications reflects not only on the authors but also on the institutions which they represent. Authors: I. Don't submit a paper for publication unless you have something new to say that will be of significant interest to others in our technical community. II. Don't attempt to publish work that is incomplete. III. Make sure that what you do write is well written and clearly presented so that your reader will be able to fully follow what you have to say. Editors: I. Raise the bar! II. Clearly define the standards for acceptance of papers in your publication and then enforce them. This may require some interaction with reviewers to help them understand the criterion. Reviewers: I. Make sure that papers meet the minimum criterion for publication before you accept them. II. Ask yourself if there are readers who will be interested in and benefit from the paper in question and whether there the technical community will benefit if the paper is

1.

published and preserved for readers yet to come. III. Although I don't agree that there is any point in including reviewers' names at the time of publication, it would probably be a good idea if each reviewer asked him/her self if they would tell their colleagues that they recommended a specific paper for publication and that they really should read it as soon as it is published. Readers: I. Let the editors know if you are concerned that the quality of publications in any given journal is below your expectations. II. Participate in the process by volunteering to review papers. [15] G. The Transcendence of the Issue Finally, another commentator shows that the problem transcends the academic world and the need to be selective when using technical publications: As a non-academic working engineer with 45 years experience in the electricity industry I would like to heartily endorse [15] remarks, in particular his checklist of things to look for in a paper. In the course of my daily work I am on the continual lookout for ways of doing things better . . . and seize upon any new formula or methodology and create a spreadsheet model from it that will inform and improve our work practices. Over the last week for example I have downloaded 589 papers from the recent CIRED conference in Vienna, some of which are rubbish but others of which are real gems and I have circulated as widely as I can. Without the benefit of years of practical experience in the industry though it must be difficult for researchers and perhaps even reviewers to know what will be useful and what will be not. I must say I really appreciate this plethora of papers that creates a vibrant marketplace of ideas for me to shop in but there must be a downside in the wasted effort creating stuff that is condemned to sit on the shelf forever gathering dust. We used to be told at university that we would only use 10% of what we were taught; but no-one knew which 10%. Currently I think I'm batting in the low twenties and consider myself exceptionally privileged. [16] III. CONCLUSIONS This paper has summarized the informal discussions which took place on the power globe listserv during the month of September 2007. The debaters, in response to an initial posting, made many constructive comments and suggestions which will hopefully help the IEEE Power Engineering Society to cope with the increasing quantity of papers as it adjusts the review process and to maintain the high quality and ethics required for the proper functioning of the power engineering community. A panel session was organized for the PES General Meeting in Pittsburgh for July 2008 (Appendix 1).

As a final observation the author would like to refer to a previous posting on Power Globe (and transcribed in the Appendix 2) which encourages reviewers to be firm, but careful and civilized with their words. IV. APPENDIX 1 ADVICE TO REVIEWERS (*) 1 Avoid unqualified statements. It is not useful writing merely that the paper is of low quality or irrelevant or a case of bad engineering. Be specific and tell the failures and let the author(s) see by themselves where they should work to improve, correct or even accept the rejection. 2 Avoid the use of offending words. The vocabulary of endearment, complaint, and abuse, provides almost the only specimens of words that are purely emotional, words from which all imaginative or conceptual content has vanished, so that they have no function at all but to express or stimulate emotion, or both. 3 Be on guard when criticizing. If we honestly believe a paper to be very bad and we can hardly help but express our extreme dislike, remember that the function of the reviewer is to get out of the way and let logic speak; not to discharge hatred, but to expose the grounds for it; not to vilify faults but to diagnose and exhibit them. Unfortunately, to express our hatred and to revenge is easier. Hence there is a tendency to select pejorative epithets with a view not to their accuracy but to their power of hurting. The best protection against this is to remind ourselves again and again what the proper function of pejorative words is. The ultimate, simplest and most abstract, is the word bad itself. The only good purpose for ever departing from that monosyllable when we condemn anything is to be more specific, to answer the question, Bad in what way? Pejorative words are rightly used only when they do this. 4 Finally, be fair and kind. We must get it firmly fixed in our minds that the very occasions on which we should most like to write a slashing paper review are precisely those on which we had much better hold our tongues. The very desire is a danger signal. When a colleague whom we admire in general, writing about a topic we know, produces disappointing work, we may proceed with tolerable safety. We know what we had hoped for. We see, and would have relished, what he was trying to do. By that light we may possibly diagnose where the paper has gone wrong and make constructive suggestions. But when an author we do not like or whose knowledge of the subject we question is attempting (unsuccessfully - or worse still, successfully, according to the editorial rules) exactly the sort of thing we know will not be very useful (beautiful and consistent analysis, but without much physical or practical content) then, if we are wise, we shall be very careful with our words. The strength of our dislike is itself a probable symptom that all is not well within; that the raw place in our psychology has been touched, or else that some personal or partisan motive may be secretly at work. If we were simply exercising judgment we should be calmer; less anxious to speak. And if we do speak, we shall almost certainly make fools of ourselves. (*) Adapted from CS Lewis's writings on literary criticism [17].

V. APPENDIX 2 PANEL SESSION LINE UP FOR THE PESGM08 Publish or Perish: An Evaluation of the Quality, Quantity, Ethics and Review Process of IEEE/PES Publications Abstract: Every year, the number of papers submitted to the Power Engineering Society increases. This growth, largely the result of pressure to publish, is straining the publication process, overloading the pool of reviewers, and creating the impression that the quality of papers is diminishing. This panel plans to discuss many aspects of the peer review process in PES, including present IEEE/PES publication policies, ethical aspects of the review process, technologies for aiding the review process, topics related to value of the technical content, graphics and peer review, and more. Throughout, suggestions will be made for improving the PES peer review process. 1 A Short Summary of the PowerGlobe Discussion: Paulo Ribeiro 2 Publication Policies: Atef Morched 3 Dealing Ethically with the Publish or Perish Pressure: Charles Gross 4 Advanced Technology for Assisting the Review Process: Thomas Baldwin 5 Peer Review and Graphics: Harold Kirkham 6 Questioning the Metrics for Performance Evaluation: Mariesa Crow VI. REFERENCES
[1] Dimitris Kalles, Improving Professional Conduct in Publishing, IEEE Computer Society, October 2005 (Vol. 38, No. 10) pp. 116, 114-115. [2] Bruce Bower, Peer Review Under Fire, Science News, June 22, 1991, Vol 139 pp 394-395. [3] Jon Turney, End of the peer show: Who decides who should do what research? Most scientists agree that peer review is a deeply flawed method. But other solutions might be worse From New Scientist Print Edition, 22 September 1990. [5] James Hendler, Avoiding Rejection, A Letter from the Editor, IEEE Intelligent Systems, IEEE Computer Society, Sept.-Oct. 2005, Volume: 20, Issue: 5, pp. 2- 4 [6] William A. Wulf, How Shall We Satisfy the Long-Term Educational Needs of Engineers?, Proceedings of the IEEE, Volume 88, Number 4, April 2000, pp. 593-596. [7] Markel, M.; Sanders, S.P.; Turning a conference paper into a Transactions article, Professional Communication Conference, 1994. IPCC '94 Proceedings. 'Scaling New Heights in Technical Communication'., International, 28 September.-1 October. 1994 pp. 426 - 429 [8] Kashihara, Akihiro; Kamoshita, Yasuhiro; From Knowledge Publishing to Peer Review, Advanced Learning Technologies, 2007. ICALT 2007. Seventh IEEE International Conference on 18-20 July 2007 Page(s):459 463 [9] Arun Sekar, Power Globe Posting, Sept. 2007. [10] Harold Kirkham, Power Globe Posting, Sept. 2007. [11] Janusz Bialek, Power Globe Posting, Sept. 2007. [12] Gerald Sheble, Power Globe Posting, Sept. 2007. [13] Robert Thomas, Power Globe Posting, Sept. 2007. [14] Charles Gross, Power Globe Posting, Sept. 2007. [15] Steve Umans, Power Globe Posting, Sept. 2007. [16] Leith Elder, Power Globe Posting, Sept. 2007. [17] CS Lewis, Studies in Words, Cambridge University, 1960.

VII. BIOGRAPHY
Paulo F. Ribeiro (M1978, SM1988, F2003) received a BS in Electrical Engineering from the Federal University of Pernambuco, Brazil, completed the Electric Power Systems Engineering Course with Power Technologies, Inc. (PTI), and received his Ph.D. from the University of Manchester, Manchester

England. Presently, he is a Professor of Electrical Engineering at Calvin College, Grand Rapids, Michigan. Dr. Ribeiro is a Fellow of the IEEE.