JMSE 2017, 2 (1), 34-54 http://www.jmse.org.

cn/

doi: 10,3724 / SP.J.1383.201002 http://engine.scichina.com/publisher/CSPM/journal/JMSE

Artikel

Mengukur Kinerja Evolusi Jurnal Akademik dalam Ilmu Manajemen

dan Riset Operasi: Sebuah Pendekatan DEA-Malmquist

kun Chen 1, Guoliang Yang 2, * dan Mohammad Khoveyni 3

1
Pengembangan Pusat Penelitian Pemerintah Rakyat Provinsi Hunan, Changsha 410.011, Hunan, Cina; pk87516@163.com
2
Institut Kebijakan dan Manajemen, Chinese Academy of Sciences, Beijing 100.190, Cina
3
Departemen Matematika Terapan, Yadegar-e-Imam Khomeini (RAH) Shahre Rey Cabang, Universitas Islam Azad, Tehran, Iran;

mohammadkhoveyni@gmail.com

* Korespondensi: glyang@casipm.ac.cn , Tel .: + 86-10-5935-8816

Diterima: 30 Juli 2016; Diterima: 21 April 2017; Diterbitkan: 12 Mei 2017

Abstrak: Makalah ini bertujuan untuk mengukur produktivitas jurnal akademik di bidang ilmu manajemen dan riset operasi atas dasar
pendekatan -Malmquist analisis data envelopment (DEA). Menggunakan hasil empiris, kita membuat temuan sebagai berikut: (1)
Dibandingkan dengan DEA-CCR (Charnes, Cooper, Rhodes) model, DEA-BCC (Banker, Charnes, Cooper) Model dapat melebih-lebihkan
kinerja jurnal dengan lebih pendek dikutip paruh dan artikel yang lebih sedikit, dan mungkin meremehkan kinerja jurnal dengan
setengah-hidup lebih lama dikutip dan lebih artikel; (2) rata-rata kinerja jurnal dalam ilmu manajemen dan operasi lapangan penelitian
berkembang sebesar 1,8% selama 2007-2014, yang merupakan hasil dari efek perbatasan-shift; (3) Kesenjangan antara jurnal terbaik di
perbatasan DEA dan sisanya dari jurnal meningkat karena jurnal terbaik tumbuh relatif lebih cepat; (4) jurnal tingkat tinggi telah membuat
kemajuan lebih dan kinerja mereka lebih stabil, dan perbedaan antara jurnal tingkat tinggi juga lebih kecil selama 2007-2014; dan, (5)
Sebanyak nilai produktivitas faktor yang lebih tinggi sesuai dengan Asosiasi Bisnis Sekolah peringkat yang lebih tinggi untuk jurnal tertentu.

Kata kunci: jurnal akademik; Indeks produktivitas Malmquist; Analisis data meyampul

1. Pendahuluan

Evaluasi jurnal akademik telah menjadi isu penting dalam dunia akademis selama beberapa dekade. Hasil evaluasi jurnal di mana para
peneliti makalah yang diterbitkan memiliki dampak kritis pada peneliti seperti karir akademik. Pertama, premis khas yang satu memperoleh gelar
doktor dan posisi fakultas menerbitkan beberapa makalah dalam jurnal tingkat tinggi. Kedua, bahkan setelah berhasil direkrut, peneliti harus
mencari dana untuk melakukan penelitian mereka. Selain kualitas proposal penelitian ilmiah, dana yang sukses tergantung sampai batas
tertentu pada

34

Copyright © 2019 Cina Sains Publishing & Media Ltd Produksi dan Hosting oleh Elseiver BV Atas nama KeAi Komunikasi co. , Ltd Ini adalah akses artikel terbuka di bawah CC BY-NC-ND License ( http://creativecommons.o
.
35 JMSE 2017, 2 (1), 34 - 54

publikasi pemohon. Ketiga, salah satu kriteria yang paling penting untuk promosi peneliti adalah apakah ia (dia) telah menerbitkan makalah di
jurnal berkualitas tinggi. Untuk tujuan ini, semua orang di komunitas ilmiah sangat terlibat dalam evaluasi jurnal akademik. Sebagai contoh,
selama paruh pertama 2016, China Departemen Pendidikan (MoE) berusaha untuk mempromosikan klasifikasi baru jurnal akademik, yang
menyebabkan perdebatan luas di masyarakat akademik dan akhirnya ditunda. 1

Tidak diragukan lagi, metode evaluasi jurnal yang paling banyak digunakan adalah faktor dampak (IF) yang diusulkan oleh Garfield (1953, 2006)

dan yang diterbitkan dalam Journal Citation Reports tahunan Thomson Reuters. Hirsch (2005) mengusulkan h-indeks, yang baru-baru ini menjadi salah

satu metode yang paling berpengaruh untuk menilai para peneliti dan lembaga. Beberapa sarjana berpendapat bahwa h-index alamat sampai batas

tertentu kekurangan dari IF. Namun, baik IF dan h-indeks telah menerima kritik yang signifikan (misalnya, untuk IF, Seglen (1997), Blok dan Walter

(2001), dan Glänzel dan moed (2002), misalnya, untuk h-index, Vanclay (2006), Vinkler (2007), dan Adler et al. (2008)). Baru-baru ini, American Society

for Microbiology (ASM) menyatakan bahwa itu membebaskan IF delapan jurnal subordinasi.

Beberapa kelemahan untuk IF ada: ( 1) generalisasi yang berlebihan ( Falagas et al, 2010.; Kravitz, 2011; Seglen, 1997). JIKA mengukur rata-rata

keseluruhan dari semua makalah yang diterbitkan oleh jurnal. Namun, meskipun makalah mungkin telah diterbitkan dalam jurnal yang sama, mungkin ada

perbedaan yang signifikan dalam kualitas yang sebenarnya makalah ini. Dengan demikian, bias yang kuat mungkin ada ketika mengukur kualitas kertas tertentu

menggunakan jurnal IF. ( 2) Waktu lag. JIKA biasanya meliputi nomor kutipan dari artikel yang diterbitkan dalam dua atau lima tahun sebelumnya dan gagal untuk

memasukkan semua nomor kutipan dari waktu ke waktu. (3) Fluktuasi. IF kemungkinan perubahan sebagai jumlah artikel yang diterbitkan per tahun perubahan

(Garfield, 1953). Beberapa studi menemukan thatthe IF jurnal yang mempublikasikan kurang dari 35 artikel per tahun berfluktuasi sekitar 40% antara tahun

berdekatan. Meskipun beberapa jurnal mempublikasikan lebih dari 150 artikel per tahun, tingkat fluktuasi adalah sekitar 15% (moed, 1999; Seglen, 1997). ( 4) Bias

statistik. Karena kutipan berasal dari Web of Science (WOS) database dan terutama dalam bahasa Inggris, kutipan dari jurnal tidak termasuk dalam database Wos

dan ditulis dalam bahasa lain diabaikan. ( 5) Menyesatkan publik. Sebuah JIKA tinggi tidak sama dengan kualitas akademik tinggi atau perbatasan penelitian. Hal

ini dimungkinkan untuk mengabaikan pertanyaan penelitian atau tugas penelitian ketika membaca dengan teliti tinggi JIKA jurnal benar-benar penting. kelemahan

lainnya termasuk perbedaan lapangan (Dong, 2005), kutipan negatif (Thorne, 1977), dan sebagainya. Beberapa masalah tidak unik untuk IF tapi yang umum untuk

IF, h-index, dan indeks atau teknik lain berdasarkan analisis kutipan. Untuk diskusi tambahan pada kelemahan dari IF dan h-indeks, pembaca yang tertarik dapat

merujuk pada Leydesdorff (2012) dan Malesios dan Arabatzis (2012).

Beberapa peneliti memperhatikan bahwa cacat utama dari IF adalah bahwa hal itu disederhanakan. Dari perspektif sistem input-output, IF dapat

dilihat sebagai single-input dan sistem single-output. Input tunggal “artikel” dan output tunggal “kutipan”. Jelas, kutipan tidak terbatas pada artikel,

seperti dicatat oleh review dari proses penerbitan dan menghasilkan kutipan. Ketika penulis memandang siap, naskah tersebut diajukan ke kantor

redaksi sebuah jurnal. Selanjutnya, setelah pemeriksaan awal editor, naskah ditugaskan untuk beberapa pengulas anonim tergantung pada peraturan

khusus jurnal. Ketika komentar pengulas dikembalikan ke departemen editorial, keputusan mengambil salah satu dari tiga bentuk: (1) menerima, (2)

menolak, atau (3) merevisi. Jika penulis (s) memutuskan untuk merevisi, kemudian putaran review baru dimulai jika diperlukan. Setelah artikel ini

diterbitkan, peneliti lain yang merasa nilai referensi inspirasi atau memiliki akan mengutipnya dalam makalah mereka. Jelas, banyak faktor yang

mempengaruhi kutipan dan artikel. Misalnya, faktor yang berpengaruh tertentu (misalnya, IF sendiri dan jadwal penerbitan jurnal) dapat mempengaruhi

kutipan. Karena itu,

1 Tersedia online: http://news.sciencenet.cn/htmlnews/2016/5/345378.shtm (diakses pada 5 Mei 2017).

JMSE 2017, 2 (1), 34 - 54 36

kira evaluasi jurnal masalah dengan beberapa input dan output, sehingga penting untuk memilih metode yang tepat untuk mengatasi
masalah ini. Oleh karena itu, kinerja menganalisis jurnal akademik menggunakan beberapa input dan output sesuai.

Secara khusus, penelitian ini berfokus pada tren multi-tahun dalam perubahan dalam kinerja jurnal akademik. Menggunakan sebagai
contoh jurnal akademik internasional di bidang Manajemen Sains & Riset Operasi (MS & OR), makalah ini bertujuan untuk mengamati
perubahan kinerja dari jurnal akademik selama beberapa tahun dan faktor-faktor yang berkontribusi terhadap perubahan tersebut. Kami juga
menyelidiki perbedaan dalam kinerja dan perubahan yang antara tingkat tinggi dan jurnal tingkat rendah, dan faktor-faktor yang berpengaruh
yang sesuai. Untuk yang terbaik dari pengetahuan kita, tidak ada penelitian yang ada ada di masalah ini, terutama evolusi kinerja dari waktu ke
waktu. Dua makalah juga digunakan beberapa input dan output dan analisis data envelopment (DEA) untuk menilai jurnal akademik (Lee &
Shin, 2014; Petridis et al, 2013.). Namun,

Untuk meringkas, dalam bidang kajian jurnal akademis, kami belum menemukan penelitian yang menggunakan pendekatan DEA-Malmquist untuk

mengukur perubahan kinerja jurnal. Dua penelitian terkait mengukur efisiensi jurnal menggunakan metode DEA gagal melakukan analisis time-series.

Makalah ini menggunakan pendekatan DEA-Malmquist untuk melakukan penilaian jurnal akademis dan analisis kinerja evolusi. DEA adalah metode

non-parametrik yang mengukur efisiensi atau kinerja unit pengambilan keputusan (DMU) dengan beberapa input dan output, dan secara luas digunakan

dalam penilaian penelitian ilmiah (lihat, misalnya, Anderson & Peterson, 1993; Banker et al, 1984;. Charnes et al, 1978;. Cook & Seiford, 2009; Cooper

et al, 2000;. Doyle & Green, 1994; Emrouznejad & Yang, 2017; Yang et al, 2014;. Yang et al, 2016. ). DEA cocok untuk aplikasi untuk sistem generasi

kutipan yang produksinya fungsi tidak bisa diamati. Malmquist (1953) mengusulkan indeks Malmquist, yang dapat digunakan untuk menganalisis

perubahan produktivitas dalam beberapa periode. Tarif et al. (1994) menetapkan indeks produktivitas Malmquist dalam rangka DEA untuk memeriksa

produktivitas faktor total (TFP), yang digunakan untuk menganalisis evolusi produktivitas dari waktu ke waktu.

Sisa dari artikel ini disusun sebagai berikut. Bagian 2 merangkum penelitian sebelumnya yang menggunakan metode DEA-Malmquist untuk

mengukur kinerja jurnal akademis. Bagian 3 menjelaskan pendekatan DEA digunakan dalam makalah ini. Bagian 4 menyediakan sumber data,

indikator, dan langkah-langkah analisis. Bagian 5 menggambarkan hasil empiris mengukur jurnal akademik internasional di bidang Manajemen

Sains & Riset Operasi. Bagian 6 menyimpulkan tulisan ini.

2. Tinjauan Literatur

Beberapa penelitian yang ada diukur kinerja jurnal akademik dengan menggunakan metode DEA dari sudut pandang proses produksi
kutipan. Dari Wos, kami hanya menemukan dua makalah yang berusaha untuk mengukur efisiensi jurnal. Petridis et al. (2013) mengukur
efisiensi jurnal kehutanan dan terintegrasi skor IF, h-index, dan Eigenfactor tiga output yang berhubungan dengan keuntungan dari metode
DEA, yang efektif mengatasi beberapa input dan output indikator. Lee dan Shin (2014) berfokus pada cacat dari IF, yang tidak bisa kontras
antara kinerja jurnal akademik di bidang interdisipliner. Untuk mengatasi masalah ini, mereka memperkenalkan kepadatan kutipan dan
dikutip paruh sebagai masukan, dan menggunakan metode DEA dalam kasus jurnal akademis di bidang Teknik Industri.

Bila menggunakan DEA untuk menilai kinerja jurnal akademis, penting untuk memilih input yang tepat dan output. Kami meringkas indikator yang digunakan dalam

studi terkait, seperti yang ditunjukkan pada Tabel 1. Kami juga daftar IF dalam tabel (lihat Row 2) untuk memfasilitasi perbandingan.
37 JMSE 2017, 2 (1), 34 - 54

Tabel 1. Teknis Pengukuran Efisiensi

penulis input output

Garfield, Jumlah makalah yang diterbitkan di jurnal dalam dua tahun sebelumnya kutipan
(1955)
Petridis et al. Artikel diterbitkan per tahun; Eigenfactor mencetak; h-index;
(2013) Frekuensi publikasi jurnal dalam setahun faktor dampak lima tahun

Lee dan Shin Jumlah item citable diterbitkan dalam jurnal dalam dua tahun sebelumnya; Jumlah rata-rata Jumlah kutipan;
(2014) referensi per item dari kategori dengan yang jurnal berafiliasi dalam dua tahun sebelumnya kutipan eksternal;
(density Citation); Rata-rata dikutip paruh kategori dengan yang jurnal berafiliasi pada tahun Self-kutipan
tertentu (Citation dinamika)

Semua makalah pada Tabel 1 yang dipilih jumlah artikel sebagai masukan. Petridis et al. (2013) juga menggunakan frekuensi penerbitan jurnal sebagai masukan

lain untuk menghilangkan perbedaan yang disebabkan oleh jeda waktu yang terkait dengan publikasi. Lee dan Shin (2014) berpendapat bahwa IF lebih tinggi untuk

kepadatan kutipan yang lebih tinggi dan lebih pendek dikutip paruh. Dengan demikian, perbedaan antara disiplin bisa dihilangkan dengan kepadatan kutipan dan

dikutip paruh. Untuk output, Petridis et al. (2013) menggunakan skor Eigenfactor, h-index, dan faktor dampak lima tahun. Lee dan Shin (2014) diklasifikasikan kutipan

ke dalam diri kutipan dan kutipan eksternal dan menggunakan mereka output sebagai yang berbeda. Lihat Bagian 4 untuk diskusi rinci tentang pemilihan indikator

dalam makalah ini.

Perlu ditekankan bahwa, seperti yang dibahas sebelumnya, kami belum menemukan studi yang relevan yang menggunakan pendekatan

DEA-Malmquist untuk menganalisis perubahan berkala kinerja jurnal akademik. Oleh karena itu, makalah ini diharapkan dapat memberikan kontribusi untuk

literatur dalam arah ini.

3. DEA-Malmquist Pendekatan

3.1. DEA Model

DEA pertama kali diperkenalkan oleh Charnes et al. (1978), dan telah banyak digunakan dalam kinerja atau evaluasi produktivitas.
Selanjutnya, Banker et al. (1984) berpendapat bahwa skala hasil konstan (CRS) asumsi tersirat dalam bentuk aslinya (model DEA-CCR) DEA
dan mengusulkan model DEA-BCC atas dasar pengembalian variabel untuk skala asumsi (VRS). Konsep utama dari DEA adalah untuk
pertama mengidentifikasi perbatasan produksi yang DMU dianggap sebagai efisien. Kemudian, DMU tidak di perbatasan dibandingkan
dengan rekan-rekan mereka di perbatasan untuk memperkirakan nilai efisiensi mereka. Semua DMU di perbatasan dianggap memiliki tingkat
kinerja yang sama dan untuk mewakili praktek terbaik. Salah satu keuntungan utama dari DEA adalah bahwa hal itu memberikan kebebasan
penuh DMU untuk memilih bobot mereka,

Lee dan Shin (2014) menyatakan bahwa tidak ada bukti bahwa proses generasi kutipan memenuhi CRS; Oleh karena itu, model DEA-BCC lebih

tepat. Namun, penelitian ini mengadopsi sudut pandang yang berbeda. CRS menunjukkan bahwa peningkatan semua masukan, misalnya 1%, sesuai

dengan peningkatan output sebesar 1%. Dalam proses produksi tradisional, seperti pabrik sepatu, asumsi VRS biasanya lebih tepat diberikan efek skala

ekonomi. Namun, ketika kita membandingkan sebuah jurnal yang menerbitkan 1.000 artikel per tahun untuk satu yang menerbitkan hanya 10 artikel per

tahun, tidak ada skala ekonomi yang signifikan ada. Untuk penulis, pengulas, dan pembaca yang mengutip artikel ini, setiap kutipan mengikuti prosedur

yang sama dan independen dari kutipan lain, dan pembaca ini tidak dapat bekerja sama dengan masing-masing
JMSE 2017, 2 (1), 34 - 54 38

lainnya, pekerja pabrik tidak seperti. Singkatnya, kami percaya bahwa model DEA-CCR didasarkan pada asumsi CRS lebih tepat untuk mengevaluasi kinerja

jurnal. Selain itu, tujuan dari setiap jurnal adalah untuk meningkatkan jumlah kutipan daripada mengurangi artikel yang dipublikasikan dan masukan lainnya

untuk meningkatkan efisiensi; Oleh karena itu, penelitian ini terutama menggunakan model DEA-CCR output oriented.

Lebih formal, biarkan {( • •, • •) | • = 1, ..., •} (• • ∈ • • +, • • ∈ • • +) menjadi satu set diamati input dan output pasang untuk pengambilan keputusan n unit. Kemudian,

Set Produksi Kemungkinan (PPS) didefinisikan oleh

••• = {(•, •) | • ••• •• •••••••• •••• •}. (1)

Batas PPS disebut sebagai teknologi produksi atau frontier produksi. PPS dapat disimulasikan atas dasar pengamatan yang ada di
bawah asumsi yang berbeda dalam model DEA. Sebagai contoh, PPS di bawah asumsi skala hasil konstan (RTS) dalam model DEA-CCR (.
Charnes et al, 1978) didefinisikan masing-masing sebagai:

••• ••• = {( •, •) | • ≥ Σ =1 λ • • • ••, • ≤ Σ =1 λ • • • ••, λ • ≥ 0, • = 1, ..., •}. 2 (2)

Menggunakan ••• •••, kita memiliki CCR output oriented berikut sebagai berikut:

• 0 * = ••• {• 0 | ( • 0 - • 0+, • 0 • 0 + • 0-) ∈ ••• •••}, (3)

dimana S 0+ dan S 0- adalah vektor kendur.

3.2. Malmquist Indeks Produktivitas

Dalam studi ini, kami menggunakan pendekatan DEA-Malmquist dibangun oleh tarif et al. (1997) untuk melakukan analisis kinerja evolusi jurnal akademik dari

waktu ke waktu. DEA-Malmquist dapat mengatasi data panel dengan menggunakan beberapa input dan output selama beberapa periode untuk menyelidiki evolusi

produktivitas DMU yang dari waktu ke waktu.

Indeks DEA-Malmquist dapat mengukur perubahan TFP dari periode t dengan periode t + 1 dengan menggunakan teknologi periode t. Indeks Malmquist,

berdasarkan model DEA-CCR berbasis hasil, dapat dirumuskan sebagai berikut:

1)] 1 2 /,
•• (••, ••) •• + 1 (•• + 1, •• +
• (• • + 1, • • + 1, • •, • •) = [•• (•• + 1, •• + 1) (4)
•• (••, ••) ×

dimana • menunjukkan fungsi jarak dari rumus (3). Variabel • mengacu pada indeks produktivitas Malmquist, yang dapat diuraikan lebih lanjut menjadi

dua komponen (efek perbatasan-shift dan efek catch-up) sebagai berikut:

/,
• (• • + 1, • • + 1, • •, • •) = •• + 1 (•• + 1, •• + 1) [ •• (•• + 1, •• + 1) (5)
•• (••, ••) •• + 1 (••, ••) × •• (••, ••) •• + 1 (••, ••)] 1 2

di mana istilah pertama, •• + 1 (•• + 1, •• + 1) . menunjukkan perubahan efisiensi (Effch, efek catch-up) di DMU dari
•• (••, ••)
/
periode t dengan periode t + 1, dan istilah kedua, [ •• (•• + 1, •• + 1) . menunjukkan pergeseran perbatasan (Techch,
•• + 1 (••, ••) × •• (••, ••) •• + 1 (••, ••)] 1 2
perbatasan-shift efek) selama periode. Variabel Effch <1 (= 1, atau> 1) dan Techch <1 (= 1, atau> 1) menunjukkan efisiensi teknis dan peningkatan perbatasan yang

efisien (konstan, penurunan) dari waktu ke waktu, masing-masing.

Gambar 1 menggambarkan secara intuitif indeks produktivitas Malmquist dan komponen untuk input tunggal dan output. Asumsikan bahwa ada dua

periode 0 dan 1 dan tingkat output yang sama dalam dua periode ini. Kami juga menganggap bahwa

••=1
2 It should be noted that the DEA-BCC model incorporates the VRS assumption and has an extra convex constraint, ∑ ••= 1 , on its PPS.
39 JMSE 2017, 2(1), 34 – 54

the PPS is under the variable RTS assumption.

y EF at period 1

A
·
E1
B
· EF at period 0

C
·

D · E0

· O

Fig. 1 Output-Based Malmquist Productivity Index

The efficiency frontiers in period 0 and period 1 are also illustrated in this figure. Then, we consider an academic journal (DMU 0) that
operated at point E 0 in period 0 and moved to point E 1 in period 1. In this case, the Malmquist productivity index for this journal is as follows:

• 0( 0,1) = (•• •• ⁄ ) (•• ••⁄ ⁄ ) = (•• •• ⁄) . (6)

Its two components are as follows:

•••••••• − •ℎ••• 0( 0,1) = (•• •• ⁄ ) (•• ••⁄ ⁄ ) = (•• •• ⁄) , (7)

••••ℎ − •• 0( 0,1) = (•• •• ⁄ ) (•• •• ⁄)⁄ . (8)

Intuitively, the catch-up effect indicates the progress (no change or regress) in relative efficiency from period 0 to 1, and the frontier-shift
effect indicates progress (no change or regress) in the frontier technology around the university (DMU 0) from period 0 to 1. Ray and Desli (1997)
proposed the CRS decomposition of the traditional Malmquist productivity index.

4. Dataset and Indicators

4.1. Dataset

In this paper, given the availability of the selected indicators, we collected data on academic journals in the field of MS & OR from the Scimago

Journal rank database (Scimago, http://www.scimagojr.com/journalrank.php), which is supported by Scopus and the ISI WoS

(http://thomsonreuters.com/products services/scientific/Web of Science). All of the data on the indicators are from Scimago except for the cited

half-life, which is provided by the WoS. Given the differences in journal coverage, we first select the 84 academic journals from 2007 to 2014 in both

Scopus and the WoS, whereas the citations to these papers were collected up to December 2015. Among them, 16 journals could not be measured

because of missing data. We list the 68 and 16 academic journals with completed data in Appendix Tables A1–A2, respectively.
JMSE 2017, 2(1), 34 – 54 40

4.2. Input and Output Indicators

In terms of input, we primarily focus on the number of articles, citation density, cited half-life, and IF. First, we obviously need to use
articles as an input. Second, we use citation density and cited half-life as inputs on the basis of the consideration that MS & OR is a
multi-interdisciplinary subject involving various other subjects, such as economics, mathematics, and engineering. Because the size of the
journal field can be debated, we use the citation density and cited half-life of every journal and do not calculate these two indicators according
to the field to which the journal belongs.

However, through a regression analysis, we find that citation density is positively and significantly correlated with the IF (P < 0.01),
whereas cited half-life is not. Several journals that seem to belong to the same fields have very different citation density. For example, the
citation density of EUR J OPER RES is similar to that of MANAGE SCI but approximately double that of OPER RES LETT. This fact suggests
that citation density is a proxy for the quality of a journal regarding its positive and significant correlation with the IF, which may be attributable to
citation density being a symbol of the solidness of the previous literature. Better research normally leads to more citations. On the basis of this
consideration, we use cited half-life as an input and exclude citation density.

In this paper, impact factor is taken into account as an input to eliminate the “Matthew Effect” (Merton, 1968) of citation generation (Lariviere

& Gingras, 2010), which indicates that many authors tend to cite articles in higher impact journals even if articles in higher impact and lower impact

journals may be of the same quality.

Additionally, this study does not use the frequency of journal publication as an input. The reasoning is as follows. A journal that releases
twice in the second half of a year gives readers a shorter reading time than another journal that releases once in the first half of a year. In
contrast, a journal that releases twice in the first half of a year gives readers a longer reading time than another journal that releases once in the
second half of a year. From this point of view, journal publication frequency is not relevant to the IF. In particular, the adoption of cited half-life as
an input can compensate for journals with a longer cited half-life.

In terms of outputs, we only select total citations as the single output. We do not exclude self-citations because doing so has little impact
on the results, and several previous studies stated that excluding self-citations is still controversial (see, e.g., Archambault & Larivière, 2009;
Lee & Shin, 2014; Vanclay, 2013). In this paper, input and output indicators are listed in Table 2. We use the reciprocal of cited half-life and
take the cited half-life of journals as
10, which is indicated as “>10” in the WoS database in calculations.

Table 2. List of Input and Output Variables

Type Variables Descriptions

Input Articles Number of citable articles published in each journal in the past two years
Cited half-life Cited half-life of each journal in a certain year
Citation density Citation density of each journal IF
Impact factors during a two-year period
Output Total citations Total number of citations of citable articles in the past two years for a certain year

4.3. The Procedure of our Analysis

In this paper, we use the following procedure to unfold our analysis.

41 JMSE 2017, 2(1), 34 – 54

Step 1. We calculate the average efficiency values of selected academic journals from 2007 to 2014 in the field of MS & OR. 3 Using the three

models in Table 3 (i.e., M1, M2, and M3) to rank those journals, we observe the impact on journal ranking of different input indicators and the

difference in the rankings of those three models from the average IF. The average IF is calculated by transforming the IF in each year into the

number of citations and the number of articles, totaling them, and dividing them.

Step 2. We select the most suitable DEA model with proper inputs and outputs using the results of the first step to conduct the Malmquist

analysis and observe academic journal performance and its influential factors.
Step 3. We use the academic journal ranking system (Academic Journal Quality Guide) published by the
U.K.-based Association of Business Schools (ABS), which focuses on and categorizes academic journals in the field of business and management in

different levels (4-star, 3-star, 2-star, and 1-star) (see, e.g., Yang et al., 2016). In this paper, we categorize 4-star and 3-star journals as high level and

2-star and 1-star journals and others as low level, respectively. Our sample contains 29 high-level academic journals and 39 low-level ones. Using this

categorization, we conduct a performance analysis over time and detect its corresponding influential factors.

Table 3. List of Models Used in This Paper

Input Output
Model
Articles Cited Half-Life IF Total Citations

M1(CCR) √ √ √

M2(CCR) √ √ √

M3(BCC) √ √ √ √

M3(CCR) √ √ √ √

5. Results and Policy Implications

5.1. DEA Results

For each model used, the DEA score, rank, and rank difference between the IFs are shown in Table 4. From the results, we observe

significant differences between DEA ranking and IF ranking.

In model M1, using the IF as another input along with article numbers, journals with lower IFs (e.g., J OPTIMIZ THEORY APP, ANN OPER
RES, and J OPER RES SOC) are more highly ranked. However, journals with higher IFs (e.g., J QUAL TECHNOL, FUZZY OPTIM DECIS MA,
and TECHNOVATION) had relatively declining rankings.
In model M2, with cited half-life as another input along with article numbers, journals with a longer cited half-life (e.g., OPER RES LETT, ANN

OPER RES, EUR J OPER RES, and J OPER RES SOC) are more highly ranked. By using the IF, EUR J OPER RES rises in the ranks from 12 to 1

in model M2. In contrast, journals with a shorter cited half-life (e.g., TRANSPORT RES E-LOG, OPTIM ENG, and NETW SPAT ECON) had

declining rankings.

In model M3 (CCR), using impact factor and cited half-life along with article numbers as inputs, journals with a relatively lower impact factor
and longer cited half-life (e.g., ANN OPER RES, J OPTIMIZ THEORY APP, and J OPER RES SOC) are significantly highly ranked. Journals with
a relatively higher impact factor and shorter cited half-life

3 If there is a missing value for a certain journal in a certain year, we exclude this value when calculating the average.
JMSE 2017, 2(1), 34 – 54 42

(e.g., NETW SPAT ECON, FUZZY OPTIM DECIS MA, and TECHNOVATION) had declining rankings.
We also run the DEA-BCC model, denoted as model M3 (BCC), and find significant differences between it and model M3 (CCR). For
example, with respect to model M3 (CCR), the top three journals that are rising in the rankings are INFOR, DISCRETE EVENT DYN S, and J
MANUF SYST, which increased by 64, 55, and 36 in model M3 (BCC), relatively. In contrast, the top three journals that declined in the rankings
are J GLOBAL OPTIM, QUAL RELIAB ENG INT, and INT J SYST SCI, which decreased by 23, 16, and 16 in model M3 (BCC), respectively. The
data on the top three journals with increasing rankings indicate that the average citations and the average articles are much lower than the
corresponding mean and median, whereas the average citations and the average articles are higher than the corresponding median and
approximate the corresponding mean when using data on the top three declining journals by ranking.

The six journals previously mentioned (i.e., INFOR, DISCRETE EVENT DYN S, J MANUF SYST, J GLOBAL OPTIM, QUAL RELIAB ENG
INT, and INT J SYST SCI) are all 1-star or uncategorized. In particular, INFOR, DISCRETE EVENT DYN S, and J MANUF SYST are all ranked
first in model M3 (BCC). This fact suggests that model M3 (BCC) may overestimate the performance of journals with a shorter cited half-life and
fewer articles, and may underestimate the performance of journals with a longer cited half-life and more articles, thus confirming the analysis in
Section 3. We also find that the Spearman correlation coefficients among M1, M2, and M3 are relatively strong, indicating that the methods are
robust and the correlation coefficient between CCR and BCC is relatively low. See the details provided in Table 5.

Given the discussion in Section 4 and the empirical results of the four different DEA models, we use M3 (CCR) as our basic method of
analysis to develop the Malmquist productivity index. This index is used to analyze the performance evolution of the select academic journals in
the field of MS & OR for multiple years.

Table 4. Results of Different Models

M 1 (CCR ) M 2 (CCR ) M 3 (CCR ) M3 (BCC)

Journal Abbreviation Diff
Score Rank Diff
IF Score Rank Diff IF Score Rank Diff IF Score Rank Diff IF CCR
4OR-Q J OPER RES 0.256 51 •4 0.176 52 •5 0.256 51 •4 0.548 35 12 16
ANN OPER RES 0.353 34 10 0.319 28 16 0.353 34 10 0.463 46 •2 • 12
APPL STOCH MODEL BUS 0.215 59 •1 0.128 60 •2 0.215 59 •1 0.344 60 •2 •1

ASIA PAC J OPER RES 0.167 64 0 0.097 64 0 0.167 64 0 0.283 66 •2 •2

CENT EUR J OPER RES 0.249 53 1 0.153 57 •3 0.249 53 1 0.449 50 4 3

COMPUT OPER RES 0.854 7 3 0.814 8 2 0.854 8 2 0.886 16 •6 •8

COMPUT OPTIM APPL 0.367 32 1 0.252 34 •1 0.367 32 1 0.470 45 • 12 • 13

CONCURRENT 0.273 49 •6 0.202 48 •5 0.273 49 •6 0.603 30 13 19

ENG-RES A
DECIS SUPPORT SYST 0.931 4 •1 0.835 7 •4 0.931 5 •2 0.974 13 • 10 •8
DISCRETE EVENT DYN 0.242 56 • 20 0.218 44 •8 0.242 56 • 20 1.000 1 35 55
S
DISCRETE OPTIM 0.288 44 1 0.194 49 •4 0.288 44 1 0.517 40 5 4
ENG OPTIMIZ 0.377 29 2 0.253 33 •2 0.377 29 2 0.496 42 • 11 • 13
EUR J OPER RES 0.854 7 5 1.000 1 11 1.000 1 11 1.000 1 11 0
EXPERT SYST APPL 1.000 1 6 1.000 1 6 1.000 1 6 1.000 1 6 0
FUZZY OPTIM DECIS MA 0.422 23 •8 0.421 19 •4 0.422 23 •8 1.000 1 14 22
43 JMSE 2017, 2(1), 34 – 54

Table 4. Cont.
M1 (CCR) M2 (CC R) M 3 (CCR ) M 3 (BC C)
Journal Abbreviation Diff
Score Rank Diff Rank Diff Score Rank Diff
IF Score IF Score Rank Diff IF IF CCR
IIE TRANS 0.415 24 4 0.351 24 4 0.415 24 4 0.680 25 3 •1
INFOR 0.140 65 0 0.086 65 0 0.140 65 0 1.000 1 64 64
INFORMS J COMPUT 0.390 27 •2 0.268 32 •7 0.390 27 •2 0.521 39 • 14 • 12
INT J COMPUT 0.359 33 5 0.230 39 •1 0.359 33 5 0.462 48 • 10 • 15
INTEG M
INT J INF TECH 0.552 18 0 0.381 22 •4 0.552 18 0 0.788 21 •3 •3
DECIS
INT J PROD ECON 0.919 5 3 0.894 5 3 0.919 6 2 0.951 14 •6 •8
INT J PROD RES 0.512 20 4 0.529 15 9 0.529 20 4 0.640 27 •3 •7
INT J SYST SCI 0.448 21 5 0.390 21 5 0.448 21 5 0.545 37 • 11 • 16
INT J TECHNOL 0.246 55 2 0.178 51 6 0.246 55 2 0.312 63 •6 •8
MANAGE
INTERFACES 0.249 53 2 0.169 54 1 0.249 53 2 0.430 54 1 •1
J GLOBAL OPTIM 0.391 26 8 0.337 26 8 0.391 26 8 0.454 49 • 15 • 23
J IND MANAG 0.213 60 0 0.126 61 •1 0.213 60 0 0.305 64 •4 •4
OPTIM
J MANUF SYST 0.344 37 •2 0.221 43 •8 0.344 37 •2 1.000 1 34 36
J OPER MANAG 1.000 1 0 1.000 1 0 1.000 1 0 1.000 1 0 0
J OPER RES SOC 0.384 28 9 0.363 23 14 0.384 28 9 0.708 24 13 4
J OPER RES SOC JPN 0.082 67 1 0.048 68 0 0.082 67 1 0.445 51 17 16
J OPTIMIZ THEORY APP 0.377 29 10 0.351 24 15 0.377 29 10 1.000 1 38 28

J QUAL TECHNOL 0.351 35 • 13 0.291 29 •7 0.351 35 • 13 1.000 1 21 34

J SCHEDULING 0.401 25 •2 0.284 30 •7 0.401 25 •2 0.584 33 • 10 •8
J SYST ENG 0.101 66 0 0.065 66 0 0.101 66 0 0.123 68 •2 •2
ELECTRON
J SYST SCI SYST ENG 0.077 68 •1 0.051 67 0 0.077 68 •1 0.178 67 0 1
M&SOM-MANUF SERV 0.229 57 •8 0.174 53 •4 0.229 57 •8 0.298 65 • 16 •8
OP
MANAGE SCI 0.884 6 0 0.906 4 2 0.906 7 •1 1.000 1 5 6
MATH METHOD OPER 0.218 58 1 0.131 59 0 0.218 58 1 0.342 61 •2 •3
RES
MATH OPER RES 0.369 31 •2 0.282 31 •2 0.369 31 •2 0.561 34 •5 •3
MATH PROGRAM 0.577 16 3 0.532 14 5 0.577 16 3 0.717 23 •4 •7
NAV RES LOG 0.297 43 3 0.223 41 5 0.297 43 3 0.546 36 10 7
NETW SPAT ECON 0.336 39 • 12 0.250 35 •8 0.336 39 • 12 0.611 29 •2 10
NETWORKS 0.283 46 5 0.211 45 6 0.283 46 5 0.495 43 8 3
OMEGA-INT J 0.936 3 •1 0.876 6 •4 0.936 4 •2 0.996 12 • 10 •8
MANAGE S
OPER RES 0.640 14 2 0.599 13 3 0.640 14 2 0.786 22 •6 •8
OPER RES LETT 0.268 50 6 0.232 37 19 0.268 50 6 0.369 57 •1 •7
OPTIM CONTR APPL MET 0.307 42 •2 0.207 46 •6 0.307 42 •2 0.597 32 8 10

OPTIM ENG 0.309 41 •9 0.226 40 •8 0.309 41 •9 0.529 38 •6 3

OPTIM LETT 0.279 48 4 0.162 55 •3 0.279 48 4 0.328 62 • 10 • 14
OPTIM METHOD 0.287 45 5 0.181 50 0 0.287 45 5 0.414 55 •5 • 10
SOFTW
OPTIMIZATION 0.199 61 1 0.152 58 4 0.199 61 1 0.397 56 6 5
OR SPECTRUM 0.433 22 •1 0.329 27 •6 0.433 22 •1 0.602 31 • 10 •9
PAC J OPTIM 0.250 52 1 0.157 56 •3 0.250 52 1 0.489 44 9 8
PROBAB ENG 0.183 62 •1 0.114 62 •1 0.183 62 •1 0.364 58 3 4
INFORM SC
PROD OPER MANAG 0.534 19 •2 0.391 20 •3 0.534 19 •2 0.639 28 • 11 •9
JMSE 2017, 2(1), 34 – 54 44

Table 4. Cont.
M1 (CCR) M2 (CC R) M 3 (CCR ) M 3 (BC C)
Journal Abbreviation Diff
Score Rank Diff Rank Diff Score Rank Diff
IF Score IF Score Rank Diff IF IF CCR
PROD PLAN 0.340 38 4 0.231 38 4 0.340 38 4 0.463 46 •4 •8
CONTROL
QUAL RELIAB ENG INT 0.345 36 5 0.223 41 0 0.345 36 5 0.441 52 • 11 • 16

QUEUEING SYST 0.280 47 1 0.206 47 1 0.280 47 1 0.434 53 •5 •6

RELIAB ENG SYST SAFE 0.803 10 1 0.734 10 1 0.803 10 1 0.851 18 •7 •8

SAFETY SCI 0.593 15 5 0.498 16 4 0.593 15 5 0.662 26 •6 • 11

SYST CONTROL LETT 0.736 12 1 0.711 11 2 0.736 12 1 0.840 20 •7 •8
SYSTEMS ENG 0.316 40 • 10 0.236 36 •6 0.316 40 • 10 0.503 41 • 11 •1
TECHNOVATION 0.789 11 –6 0.602 12 •7 0.789 11 •6 0.861 17 • 12 •6
TOP 0.172 63 0 0.105 63 0 0.172 63 0 0.349 59 4 4
TRANSPORT RES 0.809 9 •5 0.755 9 •5 0.809 9 •5 0.948 15 – 11 •6
B–METH
TRANSPORT RES 0.720 13 •4 0.492 17 •8 0.720 13 •4 0.844 19 • 10 •6
E–LOG
TRANSPORT SCI 0.553 17 •3 0.444 18 •4 0.553 17 •3 1.000 1 13 16

Table 5. Spearman Correlation Coefficients

M1 (CCR) M2 (CCR) M3 (CCR) BCC

M1(CCR) 1.0000
M2(CCR) 0.9811 1.0000
M3(CCR) 0.9993 0.9817 1.0000
BCC 0.7383 0.7622 0.7393 1.0000

5.2. Malmquist Productivity Analysis

Using the calculation results, among 68 academic journals, we observe that 44 and 24 journals have TFP > 1 and TFP < 1, respectively, with

respect to the average value in the examined period. This fact indicates that, during 2007–2014, the performance of 44 journals progressed and

that of 24 journals regressed. In particular, J MANUF SYST had the maximum TFP score (1.335), whereas PAC J OPTIM had the minimum TFP

score (0.624). This result indicates that, during the examined period, the performance of J MANUF SYST progressed 33.5% and that of PAC J

OPTIM regressed 37.6%. See Table 6 for details.

Table 6. Results of the Malmquist Productivity Index

2007/ 2008/ 2009/ 2010/ 2011/ 2012/ 2013/

Journal Abbreviation 2007–2014
2008 2009 2010 2011 2012 2013 2014
4OR–Q J OPER RES N/A N/A N/A N/A N/A 1.303 0.74 0.982
ANN OPER RES 0.945 1.12 0.895 0.944 1.142 1.322 0.778 1.007
APPL STOCH MODEL BUS N/A N/A N/A N/A 0.833 1.04 1.151 0.999
ASIA PAC J OPER RES N/A N/A N/A N/A 1.35 0.931 1.113 1.118
CENT EUR J OPER RES N/A N/A N/A N/A N/A 1.408 0.876 1.111
COMPUT OPER RES 1.166 1.204 1.095 0.892 1.027 1.067 0.829 1.032
COMPUT OPTIM APPL 1.024 1.196 1.025 1.108 0.971 0.932 1.092 1.046
CONCURRENT ENG–RES A 0.822 1.108 1.142 0.585 0.742 1.18 1.078 0.924
DECIS SUPPORT SYST 1.5 1.238 1.024 0.796 1.089 0.901 1.141 1.078
DISCRETE EVENT DYN S 1.094 0.96 0.986 0.972 1.043 0.808 1.034 0.981
DISCRETE OPTIM N/A N/A N/A N/A N/A 0.986 1.017 1.001
ENG OPTIMIZ 1.179 1.135 0.838 1.135 1.11 1.071 0.981 1.058
EUR J OPER RES 1.316 1.349 1.053 0.876 0.881 0.969 0.87 1.028
45 JMSE 2017, 2(1), 34 – 54

T able 6. C ont.
2007/ 2008/ 2009/ 2010/ 2011/ 2012/ 2013/
Journal Abbreviation 2007–2014
2008 2009 2010 2011 2012 2013 2014
EXPERT SYST APPL 1.581 1.228 1.035 1.139 1.035 0.969 1.017 1.129
FUZZY OPTIM DECIS MA N/A N/A N/A N/A N/A 0.758 1.173 0.943
IIE TRANS 1.024 1.14 1.028 0.869 0.986 0.904 1.067 0.999
INFOR 1.807 0.834 0.968 0.945 1.066 1.027 1.146 1.081
INFORMS J COMPUT 1.036 1.082 1.241 0.968 1.082 0.825 0.913 1.013
INT J COMPUT INTEG M 1.184 1.314 0.902 1.271 1.01 0.967 1.124 1.101
INT J INF TECH DECIS N/A N/A 1.637 0.664 N/A N/A N/A 1.043
INT J PROD ECON N/A 1.124 0.961 1.152 0.893 1.141 N/A 1.049
INT J PROD RES 1.191 0.952 1.317 1.03 1.071 1.043 0.992 1.079
INT J SYST SCI 1.528 1.153 1.218 1.074 1.048 1.071 1.122 1.164
INT J TECHNOL MANAGE 0.86 1.021 0.994 1.102 0.818 1.05 0.909 0.960
INTERFACES 1.032 1.438 0.66 1.287 0.929 0.725 0.796 0.946
J GLOBAL OPTIM 1.187 1.171 1.057 0.821 1.263 0.907 0.857 1.024
J IND MANAG OPTIM N/A N/A N/A N/A 0.92 1.03 0.953 0.967
J MANUF SYST 1.139 1.514 1.71 1.126 1.615 1.388 1.015 1.335
J OPER MANAG 0.961 1.338 1.466 0.812 0.954 0.958 1.059 1.058
J OPER RES SOC 1.183 1.09 1.01 0.949 0.921 1.096 0.893 1.016
J OPER RES SOC JPN 0.708 0.986 0.695 1.917 N/A N/A N/A 0.982
J OPTIMIZ THEORY APP 1.161 1.008 1.122 1.103 1.169 0.932 0.878 1.048
J QUAL TECHNOL 1.113 1.16 0.822 1.194 0.851 0.904 0.778 0.961
J SCHEDULING 0.962 1.13 1.219 0.824 0.985 0.853 1.04 0.993
J SYST ENG ELECTRON N/A N/A N/A N/A N/A 1.261 0.883 1.055
J SYST SCI SYST ENG N/A N/A N/A N/A N/A 1.01 0.953 0.981
M&SOM–MANUF SERV OP N/A N/A N/A N/A 0.889 0.924 1.031 0.946
MANAGE SCI 1.057 1.066 0.932 0.854 0.976 1.134 0.929 0.988
MATH METHOD OPER RES N/A N/A 1.336 0.781 0.76 1.088 1.313 1.025
MATH OPER RES 0.964 1.124 0.922 1.099 0.925 1.044 1.011 1.010
MATH PROGRAM 1.327 1.079 0.866 0.865 1.325 0.953 0.914 1.031
NAV RES LOG 1.13 1.288 1.039 1.002 0.85 0.87 0.998 1.016
NETW SPAT ECON N/A N/A N/A 1.125 1.069 1.035 1.01 1.059
NETWORKS 0.987 1.302 0.951 1.043 0.707 1.448 0.793 1.005
OMEGA–INT J MANAGE S 1.38 1.329 0.984 0.923 0.775 1.079 1.227 1.080
OPER RES 0.903 1.067 1.269 0.905 1.104 0.946 0.92 1.009
OPER RES LETT 1.168 0.981 1.141 0.943 0.866 1.143 0.916 1.016
OPTIM CONTR APPL MET 1.01 1.111 0.927 0.947 1.93 1.028 0.938 1.090
OPTIM ENG N/A N/A 1.047 1.066 1.033 1.242 0.918 1.056
OPTIM LETT N/A N/A N/A N/A N/A N/A 0.859 0.859
OPTIM METHOD SOFTW 0.848 1.27 1.232 0.87 1.269 1.056 1.158 1.087
OPTIMIZATION N/A N/A 0.895 0.894 1.239 1.194 0.998 1.034
OR SPECTRUM 0.756 1.514 1.153 1.017 0.711 1.117 0.745 0.968
PAC J OPTIM N/A N/A N/A N/A N/A N/A 0.624 0.624
PROBAB ENG INFORM SC 1.004 0.985 1.33 0.744 0.84 0.728 0.849 0.908
PROD OPER MANAG 0.783 1.419 0.976 0.629 1.11 1.202 0.993 0.986
PROD PLAN CONTROL 0.87 1.025 0.993 0.945 1.211 1.054 1.38 1.057
QUAL RELIAB ENG INT 1.564 0.762 0.928 0.849 1.089 1.33 1.023 1.048
QUEUEING SYST 1.08 0.843 0.807 1.022 0.757 1.19 1.048 0.952
RELIAB ENG SYST SAFE 1.04 1.335 1.067 1.069 1.093 0.835 1.027 1.058
SAFETY SCI 1.303 1.237 1.355 0.994 0.961 1.054 1.139 1.140
SYST CONTROL LETT 1.08 1.003 1.015 0.928 1.025 1.269 0.8 1.009
SYSTEMS ENG N/A N/A N/A N/A N/A 1.031 0.852 0.937
TECHNOVATION 1.115 1.174 1.046 0.943 1.048 0.894 0.767 0.989
TOP N/A N/A N/A N/A N/A 1.315 0.821 1.039
TRANSPORT RES B–METH 1.088 1 1.141 1.285 0.923 1.292 0.865 1.074
TRANSPORT RES E–LOG 1.37 0.877 1.457 0.912 1.026 1.175 0.918 1.085
TRANSPORT SCI 1.071 0.703 1.085 1.102 1.114 1.061 0.873 0.990
JMSE 2017, 2(1), 34 – 54 46

In terms of all examined academic journals, the average TFP (Tfpch) is 1.018, indicating that the average performance of the journals in
the MS & OR field progressed by 1.8% during 2007–2014. However, the overall performance of academic journals did not always grow every
year and declined in 2010/2011 and 2013/2014. On the basis of the decompositions of the Malmquist productivity index, we find that the
progress of journals’ average performance comes from the frontier-shift effect (Techch, 2.3% in total), which indicates that journals on the
frontier continued to improve their performance during the examined period. However, the catch-up effect of technical efficiency change
(Techch) is lower than unity (only 0.995 < 1), illustrating that most journals are moving away from the reference set on the DEA frontier. This
phenomenon is understandable because the best journals grow faster than other journals, which probably results from the “Matthew Effect” in
science (Lariviere & Gingras, 2010; Lee & Shin,

2014). See Table 7 for details.

Table 7. Results of the Malmquist Productivity Index 4

Year Effch Techch Tfpch

2007/2008 0.990 1.106 1.095

2008/2009 0.905 1.235 1.118
2009/2010 0.963 1.097 1.057
2010/2011 1.044 0.928 0.969
2011/2012 1.110 0.907 1.007
2012/2013 1.010 1.031 1.041
2013/2014 0.976 0.981 0.958

2007–2014 0.995 1.023 1.018

Next, we conduct a more in-depth analysis and show the heterogeneity of the performance evolution of highand low-level academic
journals in the MS & OR field using the Malmquist productivity index.
The annual average (2007–2014) TFP value of 29 high-level journals is found in Table 7. Among these journals, 20 had TFP scores
greater than 1 and nine had TFP scores less than 1, indicating that the performance of 20 journals progressed and that of nine journals
regressed during 2007–2014. In particular, EXPERT SYST APPL has the maximum TFP score (1.129) and FUZZY OPTIM DECIS MA has the
minimum TFP score (0.943). This fact may indicate that, in the examined period, the performance of EXPERT SYST APPL progressed by
12.9% and that of FUZZY OPTIM DECIS MA regressed by 5.7%. See Table 8 for details.

Table 8. Malmquist Results of High-Level Journals

2007/ 2008/ 2009/ 2010/ 2011/ 2012/ 2013/ 2007–2014

Journal Abbreviation
2008 2009 2010 2011 2012 2013 2014 (Average)
ANN OPER RES 0.945 1.120 0.895 0.944 1.142 1.322 0.778 1.007
COMPUT OPER RES 1.166 1.204 1.095 0.892 1.027 1.067 0.829 1.032
COMPUT OPTIM APPL 1.024 1.196 1.025 1.108 0.971 0.932 1.092 1.046
DECIS SUPPORT SYST 1.500 1.238 1.024 0.796 1.089 0.901 1.141 1.078
EUR J OPER RES 1.316 1.349 1.053 0.876 0.881 0.969 0.870 1.028
EXPERT SYST APPL 1.581 1.228 1.035 1.139 1.035 0.969 1.017 1.129
FUZZY OPTIM DECIS MA N/A N/A N/A N/A N/A 0.758 1.173 0.943
IIE TRANS 1.024 1.140 1.028 0.869 0.986 0.904 1.067 0.999
INFORMS J COMPUT 1.036 1.082 1.241 0.968 1.082 0.825 0.913 1.013
INT J PROD ECON N/A 1.124 0.961 1.152 0.893 1.141 N/A 1.049

4 The 2007–2014 (average) calculation procedure is as follows. Step 1: Obtain the annual average Malmquist index of each journal; Step 2: Calculate the average Malmquist index using each

journal’s index (hereinafter).

47 JMSE 2017, 2(1), 34 – 54

Table 8. Cont.

2007/ 2008/ 2009/ 2010/ 2011/ 2012/ 2013/ 2007–2014

Journal Abbreviation
2008 2009 2010 2011 2012 2013 2014 (Average)
J OPER MANAG 0.961 1.338 1.466 0.812 0.954 0.958 1.059 1.058
J OPER RES SOC 1.183 1.090 1.010 0.949 0.921 1.096 0.893 1.016
J OPTIMIZ THEORY APP 1.161 1.008 1.122 1.103 1.169 0.932 0.878 1.048
J SCHEDULING 0.962 1.130 1.219 0.824 0.985 0.853 1.040 0.993
M&SOM–MANUF SERV OP N/A N/A N/A N/A 0.889 0.924 1.031 0.946
MANAGE SCI 1.057 1.066 0.932 0.854 0.976 1.134 0.929 0.988
MATH OPER RES 0.964 1.124 0.922 1.099 0.925 1.044 1.011 1.010
MATH PROGRAM 1.327 1.079 0.866 0.865 1.325 0.953 0.914 1.031
NAV RES LOG 1.130 1.288 1.039 1.002 0.850 0.870 0.998 1.016
OMEGA–INT J MANAGE S 1.380 1.329 0.984 0.923 0.775 1.079 1.227 1.080
OPER RES 0.903 1.067 1.269 0.905 1.104 0.946 0.920 1.009
OR SPECTRUM 0.756 1.514 1.153 1.017 0.711 1.117 0.745 0.968
PROD OPER MANAG 0.783 1.419 0.976 0.629 1.110 1.202 0.993 0.986
PROD PLAN CONTROL 0.870 1.025 0.993 0.945 1.211 1.054 1.380 1.057
RELIAB ENG SYST SAFE 1.040 1.335 1.067 1.069 1.093 0.835 1.027 1.058
TECHNOVATION 1.115 1.174 1.046 0.943 1.048 0.894 0.767 0.989
TRANSPORT RES B–METH 1.088 1.000 1.141 1.285 0.923 1.292 0.865 1.074
TRANSPORT RES E–LOG 1.370 0.877 1.457 0.912 1.026 1.175 0.918 1.085
TRANSPORT SCI 1.071 0.703 1.085 1.102 1.114 1.061 0.873 0.990

From the average TFP (Tfpch) changes for all high-level journals, we observe that the average value reaches 1.024 >

1, which indicates that these journals progressed by 2.4% on average per year during 2007–2014. In the examined period, some fluctuations
exist that do not affect the developmental trends of those high-level journals as a whole. Using the Malmquist decompositions, we find that the
overall progress comes primarily from the frontier-shift effect (Techch) instead of the catch-up effect (Effch). This fact indicates that only a few of
the best journals on the frontier can promote the frontier to shift in a better direction. We also observe that, to some extent, the variance of
high-level journals grows larger during the examined period. See Table 9.

Table 9. Average Malmquist Results of High-Level Journals

Year Effch Techch Tfpch

2007/2008 0.984 1.104 1.086

2008/2009 0.918 1.246 1.145
2009/2010 0.970 1.102 1.069
2010/2011 1.033 0.921 0.953
2011/2012 1.099 0.909 0.999
2012/2013 0.968 1.031 0.998
2013/2014 0.983 0.984 0.967

2007–2014 0.990 1.034 1.024

In contrast, regarding the average TFP (Tfpch) changes for the 43 low-level journals, 24 journals had an average TFP > 1 and 15 journals had

an average TFP < 1, respectively. It should be noted that there are four journals with TFP marked as N/A, which means that data are missing in the

examined period. In particular, the TFP of J MANUF SYST is the highest (1.335) and the TFP of PAC J OPTIM is the lowest (0.624), indicating that J

MANUF SYST progressed by

33.5%, and PAC J OPTIM regressed by 37.6% during the examined period. See Table 10 for details.
JMSE 2017, 2(1), 34 – 54 48

Table 10. Malmquist Results of Low-Level Journals

2007/ 2008/ 2009/ 2010/ 2011/ 2012/ 2013/

Journal Abbreviation 2007–2014
2008 2009 2010 2011 2012 2013 2014
4OR–Q J OPER RES N/A N/A N/A N/A N/A 1.303 0.740 0.982
APPL STOCH MODEL BUS N/A N/A N/A N/A 0.833 1.040 1.151 0.999
ASIA PAC J OPER RES N/A N/A N/A N/A 1.350 0.931 1.113 1.118
CENT EUR J OPER RES N/A N/A N/A N/A N/A 1.408 0.876 1.111
CONCURRENT ENG–RES A 0.822 1.108 1.142 0.585 0.742 1.180 1.078 0.924
DISCRETE EVENT DYN S 1.094 0.960 0.986 0.972 1.043 0.808 1.034 0.981
DISCRETE OPTIM N/A N/A N/A N/A N/A 0.986 1.017 1.001
ENG ECON 1.179 1.135 0.838 1.135 1.110 1.071 0.981 1.058
ENG OPTIMIZ 1.807 0.834 0.968 0.945 1.066 1.027 1.146 1.081
IEEE SYST J 1.184 1.314 0.902 1.271 1.010 0.967 1.124 1.101
INFOR N/A N/A 1.637 0.664 N/A N/A N/A 1.043
INT J COMPUT INTEG M 1.191 0.952 1.317 1.030 1.071 1.043 0.992 1.079
INT J INF TECH DECIS 1.528 1.153 1.218 1.074 1.048 1.071 1.122 1.164
INT J PROD RES 0.860 1.021 0.994 1.102 0.818 1.050 0.909 0.960
INT J SYST SCI 1.032 1.438 0.660 1.287 0.929 0.725 0.796 0.946
INT J TECHNOL MANAGE 1.187 1.171 1.057 0.821 1.263 0.907 0.857 1.024
INTERFACES N/A N/A N/A N/A 0.920 1.030 0.953 0.967
J GLOBAL OPTIM 1.139 1.514 1.710 1.126 1.615 1.388 1.015 1.335
J IND MANAG OPTIM 0.708 0.986 0.695 1.917 N/A N/A N/A 0.982
J MANUF SYST 1.113 1.160 0.822 1.194 0.851 0.904 0.778 0.961
J OPER RES SOC JPN N/A N/A N/A N/A N/A 1.261 0.883 1.055
J QUAL TECHNOL N/A N/A N/A N/A N/A 1.010 0.953 0.981
J SYST ENG ELECTRON N/A N/A 1.336 0.781 0.760 1.088 1.313 1.025
J SYST SCI SYST ENG N/A N/A N/A 1.125 1.069 1.035 1.010 1.059
MATH METHOD OPER RES 0.987 1.302 0.951 1.043 0.707 1.448 0.793 1.005
NETW SPAT ECON 1.168 0.981 1.141 0.943 0.866 1.143 0.916 1.016
NETWORKS 1.010 1.111 0.927 0.947 1.930 1.028 0.938 1.090
OPER RES LETT N/A N/A 1.047 1.066 1.033 1.242 0.918 1.056
OPTIM CONTR APPL MET N/A N/A N/A N/A N/A N/A 0.859 0.859
OPTIM ENG 0.848 1.270 1.232 0.870 1.269 1.056 1.158 1.087
OPTIM LETT N/A N/A 0.895 0.894 1.239 1.194 0.998 1.034
OPTIM METHOD SOFTW N/A N/A N/A N/A N/A N/A 0.624 0.624
OPTIMIZATION 1.004 0.985 1.330 0.744 0.840 0.728 0.849 0.908
PAC J OPTIM 1.564 0.762 0.928 0.849 1.089 1.330 1.023 1.048
PROBAB ENG INFORM SC 1.080 0.843 0.807 1.022 0.757 1.190 1.048 0.952
QUAL RELIAB ENG INT 1.303 1.237 1.355 0.994 0.961 1.054 1.139 1.140
QUEUEING SYST 1.080 1.003 1.015 0.928 1.025 1.269 0.800 1.009
RAIRO–OPER RES N/A N/A N/A N/A N/A 1.031 0.852 0.937
SAFETY SCI N/A N/A N/A N/A N/A 1.315 0.821 1.039
STUD INFORM CONTROL N/A N/A N/A N/A N/A 1.303 0.740 0.982
SYST CONTROL LETT N/A N/A N/A N/A 0.833 1.040 1.151 0.999
SYSTEMS ENG N/A N/A N/A N/A 1.350 0.931 1.113 1.118
TOP N/A N/A N/A N/A N/A 1.408 0.876 1.111

Using the average TFP (Tfpch) changes for all low-level journals, we find that the average value reaches 1.013 > 1, which indicates that the
progress rate of such journals is an average 1.3% per year during 2007–2014. Similar to high-level journals, the fluctuations in this period do not
affect the developmental trends of low-level journals as a whole. Using the Malmquist decomposition values, we find that overall progress also
comes primarily from the frontier-shift effect (Techch) at a rate of 1.4% instead of from the catch-up effect (Effch) at a rate of −3.3%, which is
lower than the average value of all examined journals. This fact shows us that, although low-level journals make slight progress, they
continuously move farther away from the frontier. Therefore, the performance of academic journals in the MS & OR field has increasingly larger
variances. In particular, those low-level journals are left continuously farther
49 JMSE 2017, 2(1), 34 – 54

when compared with high-level journals. See Table 11 for details.

Table 11. Average Malmquist Results of Low-Level Journals

Year Effch Techch Tfpch

2007/2008 0.997 1.109 1.106

2008/2009 0.889 1.221 1.086
2009/2010 0.957 1.091 1.045
2010/2011 1.054 0.936 0.986
2011/2012 1.122 0.905 1.015
2012/2013 1.046 1.032 1.079
2013/2014 0.971 0.979 0.951

2007–2014 0.999 1.014 1.013

5.3. Comparative Analysis on Heterogeneity

From the viewpoint of the TFP value range, the range length of high-level journals (0.943–1.129) is significantly shorter than that of low-level

journals (0.624–1.335), and the standard deviation of the TFP value of high-level journals is smaller than that of low-level journals (0.043 < 0.105).

This finding shows that the performance of high-level journals is more stable and the difference between high-level journals is also smaller, which is

observed in Figure 2.

Fig. 2 TFPs of Different Journal Groups 5

The TFP value of low-level journals fluctuations more significantly over the years. The average TFP value of high-level journals (1.024) is
larger than that of low-level journals (1.013), indicating that high-level journals made greater progress during 2007–2014. When comparing the
Effch as the main negative source of TFP, both types of journals have a TFP value less than 1. However, the average TFP of high-level journals
is smaller, which suggests that the performance of both types of journals gets farther away from the production frontier but also that high-level
journals get farther away than low-level journals, on the whole. The detailed data are provided in Table 12.

5 Note: Wtfp denotes the overall TFP value of all examined journals on average. Htfp and Ltfp denote the average TFP value of high-level and low-level journals, respectively.
JMSE 2017, 2(1), 34 – 54 50

Table 12. Heterogeneity of Malmquist Results of High-Level and Low-Level Journals

TFP–min TFP–max TFP–mean Techch Effch

High–level 0.943 1.129 1.024 1.034 0.990

Low–level 0.624 1.335 1.013 1.014 0.999

In total 0.624 1.335 1.018 1.023 0.995

We further investigate the decomposition of the TFPs of high-level and low-level academic journals, respectively. Using the average TFP
value during 2007–2014, we find that a higher TFP value corresponds to a higher ABS ranking for certain journals, which is consistent with our
previous results. The journal with a higher ABS ranking progressed faster than others, which promoted growth of the frontier and contributed
significantly to the entire progress of all journals. See details in Table 13.

Table 13. Malmquist Results of Journals with Different ABS Rankings

Level Effch Techch Tfpch

4 0.994 1.031 1.0244
HIHG
3 0.989 1.035 1.0240
2 0.995 1.026 1.021
LOW 1 1.013 1.000 1.013
Non 0.992 1.017 1.010

Note: Non denotes journals that are not included in the ABS list.

6. Conclusions and Discussions

In this paper, we propose the DEA-Malmquist approach to measure the changes in the performance of international academic journals. The
first step is to use the DEA-CCR model to assess the performance of academic journals and to discuss the applicability of input and output
variables. The second step is to estimate the performance evolution using the Malmquist productivity index and 2015 ABS journal guide. On the
basis of previous studies, we have the following new findings: (1) DEA-BCC is more suitable for the evaluation of academic journals’ performance
given that the scale economies effect in the production process of citations is minimal; and (2) the DEA-Malmquist approach performs well in
measuring the change in performance of academic journals, as is shown by the empirical results. The empirical results also show: (1) The
average performance of the journals in the MS & OR field progressed at 1.8% during 2007–2014, which results from the frontier–shift effect; (2)
High-level journals make greater progress and have more stable performance, and the difference between high-level journals is also smaller
during 2007–2014; and, (3) The higher TFP value corresponds to a higher ABS ranking for certain journals.

Thus, our proposed DEA-Malmquist approach could be a useful tool for measuring the performance evolution over time of academic
journals for policy makers, scholars, and editors. However, we do not intend to argue that the DEA-Malmquist approach is better than the IF in all
aspects. The widespread use of the IF may be attributable to its easy use. Intricate approaches may be rather difficult to use and for achieving
consensus.
Some remaining problems could be further explored. First, a need exists to investigate the production process of citations to determine the
production function. Second, during the production process of citations, factors affecting citation generation, the extent of the affect, and how to
reveal the “black box” of the citation process need to be determined. Third, using citation density to measure the difference in journal areas is
questionable, and more typical indicators may need to be found. Fourth, given the limitation of the DEA-CCR, not all journals whose results are
unity
51 JMSE 2017, 2(1), 34 – 54

have the same performance, and we can use other DEA techniques (e.g., for the cross-efficiency approach, see Doyle and Green (1994) and Yang

et al. (2013); for the super-efficiency approach; see Anderson and Petersen (1993)) to investigate the performance difference of these journals.

Acknowledgments: This work is financially supported by the National Natural Science Foundation of China (NSFC, Nos. 71201158,

71671181) and Hunan Province Social Science Achievement Evaluation Committee (2017, No. GLX195).

Conflicts of Interest: The authors declare no conflict of interest.

Appendix A

Table A1. List of Academic Journals Used in This Study in the MS & OR Field

Journal Abbreviation Full Title Star ISSN

4OR–Q J OPER RES 4OR–A QUARTERLY JOURNAL OF OPERATIONAL RESEARCH 2 1619–4500
ANN OPER RES ANNALS OF OPERATIONS RESEARCH 3 0254–5330
APPL STOCH MODEL BUS APPLIED STOCHASTIC MODELS IN BUSINESS AND INDUSTRY Non 1524–1904
ASIA PAC J OPER RES ASIA–PACIFIC JOURNAL OF OPERATIONAL RESEARCH 1 0217–5959
CENT EUR J OPER RES CENTRAL EUROPEAN JOURNAL OF OPERATIONS RESEARCH 1 1435–246X
COMPUT OPER RES COMPUTERS & OPERATIONS RESEARCH 3 0305–0548
COMPUT OPTIM APPL COMPUTATIONAL OPTIMIZATION AND APPLICATIONS 3 0926–6003
CONCURRENT ENG–RES A CONCURRENT ENGINEERING–RESEARCH AND APPLICATIONS 1 1063–293X
DECIS SUPPORT SYST DECISION SUPPORT SYSTEMS 3 0167–9236
DISCRETE EVENT DYN S DISCRETE EVENT DYNAMIC SYSTEMS–THEORY AND APPLICATIONS Non 0924–6703
DISCRETE OPTIM DISCRETE OPTIMIZATION 2 1572–5286
ENG OPTIMIZ ENGINEERING OPTIMIZATION 2 0305–215X
EUR J OPER RES EUROPEAN JOURNAL OF OPERATIONAL RESEARCH 4 0377–2217
EXPERT SYST APPL EXPERT SYSTEMS WITH APPLICATIONS 3 0957–4174
FUZZY OPTIM DECIS MA FUZZY OPTIMIZATION AND DECISION MAKING 3 1568–4539
IIE TRANS IIE TRANSACTIONS 3 0740–817X
INFOR INFOR 1 0315–5986
INFORMS J COMPUT INFORMS JOURNAL ON COMPUTING 3 1091–9856
INT J COMPUT INTEG M INTERNATIONAL JOURNAL OF COMPUTER INTEGRATED MANUFACTURING 2 0951–192X

INT J INF TECH DECIS INTERNATIONAL JOURNAL OF INFORMATION TECHNOLOGY & DECISION MAKING Non 0219–6220

INT J PROD ECON INTERNATIONAL JOURNAL OF PRODUCTION ECONOMICS 3 0925–5273

INT J PROD RES INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 2 0020–7543
INT J SYST SCI INTERNATIONAL JOURNAL OF SYSTEMS SCIENCE Non 0020–7721
INT J TECHNOL MANAGE INTERNATIONAL JOURNAL OF TECHNOLOGY MANAGEMENT 2 0267–5730
INTERFACES INTERFACES 2 0092–2102
J GLOBAL OPTIM JOURNAL OF GLOBAL OPTIMIZATION Non 0925–5001
J IND MANAG OPTIM JOURNAL OF INDUSTRIAL AND MANAGEMENT OPTIMIZATION 1 1547–5816
J MANUF SYST JOURNAL OF MANUFACTURING SYSTEMS 1 0278–6125
J OPER MANAG JOURNAL OF OPERATIONS MANAGEMENT 4 0272–6963
J OPER RES SOC JOURNAL OF THE OPERATIONAL RESEARCH SOCIETY 3 0160–5682
J OPER RES SOC JPN JOURNAL OF THE OPERATIONS RESEARCH SOCIETY OF JAPAN Non 0453–4514
J OPTIMIZ THEORY APP JOURNAL OF OPTIMIZATION THEORY AND APPLICATIONS 3 0022–3239
J QUAL TECHNOL JOURNAL OF QUALITY TECHNOLOGY Non 0022–4065
J SCHEDULING JOURNAL OF SCHEDULING 3 1094–6136
J SYST ENG ELECTRON JOURNAL OF SYSTEMS ENGINEERING AND ELECTRONICS Non 1004–4132
J SYST SCI SYST ENG JOURNAL OF SYSTEMS SCIENCE AND SYSTEMS ENGINEERING Non 1004–3756
M&SOM–MANUF SERV OP M&SOM–MANUFACTURING & SERVICE OPERATIONS MANAGEMENT 3 1523–4614
MANAGE SCI MANAGEMENT SCIENCE 4 0025–1909
MATH METHOD OPER RES MATHEMATICAL METHODS OF OPERATIONS RESEARCH 1 1432–2994
MATH OPER RES MATHEMATICS OF OPERATIONS RESEARCH 3 0364–765X
JMSE 2017, 2(1), 34 – 54 52

Table A1. Cont.

Journal Abbreviation Full Title Star ISSN
MATH PROGRAM MATHEMATICAL PROGRAMMING 4 0025–5610
NAV RES LOG NAVAL RESEARCH LOGISTICS 3 0894–069X
NETW SPAT ECON NETWORKS & SPATIAL ECONOMICS 2 1566–113X
NETWORKS NETWORKS Non 0028–3045
OMEGA–INT J MANAGE S OMEGA–INTERNATIONAL JOURNAL OF MANAGEMENT SCIENCE 3 0305–0483
OPER RES OPERATIONS RESEARCH 4 0030–364X
OPER RES LETT OPERATIONS RESEARCH LETTERS 2 0167–6377
OPTIM CONTR APPL MET OPTIMAL CONTROL APPLICATIONS & METHODS Non 0143–2087
OPTIM ENG OPTIMIZATION AND ENGINEERING Non 1389–4420
OPTIM LETT OPTIMIZATION LETTERS Non 1862–4472
OPTIM METHOD SOFTW OPTIMIZATION METHODS & SOFTWARE Non 1055–6788
OPTIMIZATION OPTIMIZATION 1 0233–1934
OR SPECTRUM OR SPECTRUM 3 0171–6468
PAC J OPTIM PACIFIC JOURNAL OF OPTIMIZATION 1 1348–9151
PROBAB ENG INFORM SC PROBABILITY IN THE ENGINEERING AND INFORMATIONAL SCIENCES Non 0269–9648
PROD OPER MANAG PRODUCTION AND OPERATIONS MANAGEMENT 4 1059–1478
PROD PLAN CONTROL PRODUCTION PLANNING & CONTROL 3 0953–7287
QUAL RELIAB ENG INT QUALITY AND RELIABILITY ENGINEERING INTERNATIONAL 1 0748–8017
QUEUEING SYST QUEUEING SYSTEMS Non 0257–0130
RELIAB ENG SYST SAFE RELIABILITY ENGINEERING & SYSTEM SAFETY 3 0951–8320
SAFETY SCI SAFETY SCIENCE Non 0925–7535
SYST CONTROL LETT SYSTEMS & CONTROL LETTERS Non 0167–6911
SYSTEMS ENG SYSTEMS ENGINEERING Non 1098–1241
TECHNOVATION TECHNOVATION 3 0166–4972
TOP TOP 1 1134–5764
TRANSPORT RES B–METH TRANSPORTATION RESEARCH PART B–METHODOLOGICAL 4 0191–2615
TRANSPORT RES E–LOG TRANSPORTATION RESEARCH PART E–LOGISTICS AND TRANSPORTATION 3 1366–5545
REVIEW
TRANSPORT SCI TRANSPORTATION SCIENCE 3 0041–1655

Note: Non denotes the journals that are not included in ABS list.

Table A2. List of Academic Journals Not Used in the MS & OR Field

Journal Abbreviation Full Title ISSN

APPL MATH MODEL APPLIED MATHEMATICAL MODELLING 0307–904X
ENG ECON ENGINEERING ECONOMIST 0013–791X
EUR J IND ENG EUROPEAN JOURNAL OF INDUSTRIAL ENGINEERING 1751–5254
FLEX SERV MANUF J FLEXIBLE SERVICES AND MANUFACTURING JOURNAL 1936–6582
IEEE SYST J IEEE SYSTEMS JOURNAL 1932–8184
IMA J MANAG MATH IMA JOURNAL OF MANAGEMENT MATHEMATICS 1471–678X
INT T OPER RES INTERNATIONAL TRANSACTIONS IN OPERATIONAL RESEARCH 0969–6016
J SIMUL JOURNAL OF SIMULATION 1747–7778
MEMET COMPUT MEMETIC COMPUTING 1865–9284
MIL OPER RES MILITARY OPERATIONS RESEARCH 0275–5823
OPER RES–GER OPERATIONAL RESEARCH 1109–2858
P I MECH ENG O–J RIS PROCEEDINGS OF THE INSTITUTION OF MECHANICAL ENGINEERS PART O–JOURNAL OF RISK AND 1748–006X
RELIABILITY
QUAL TECHNOL QUANT M QUALITY TECHNOLOGY AND QUANTITATIVE MANAGEMENT 1684–3703
RAIRO–OPER RES RAIRO–OPERATIONS RESEARCH 0399–0559
SORT–STAT OPER RES T SORT–STATISTICS AND OPERATIONS RESEARCH TRANSACTIONS 1696–2281
STUD INFORM CONTROL STUDIES IN INFORMATICS AND CONTROL 1220–1766
53 JMSE 2017, 2(1), 34 – 54

References

Adler, R., Ewing, J., Taylor, P., 2008. Joint committee on quantitative assessment of research: Citation statistics. The Australian Mathematical Society

Gazette 35, 166–188.

Andersen, P., Petersen, N.C., 1993. A procedure for ranking efficient units in data envelopment analysis. Management Science 39, 1261–1264.

Archambault, É., Larivière, V., 2009. History of the journal impact factor: Contingencies and consequences. Scientometrics 79, 635–649.

Banker, R.D., Charnes, A., Cooper, W., 1984. Some models for estimating technical and scale inefficiency in data envelopment analysis. Management

Science 30, 1078–1092.

Bergstrom, C., 2007. Eigenfactor: Measuring the value and prestige of scholarly journals. College & Research Libraries News 68, 314 –316.

Block, S., Walter, G., 2001. The impact factor: Time for a change. Australian and New Zealand Journal of Psychiatry 35, 563–568.

Braun, T., Glänzel, W., Schubert, A., 2006. A Hirsch-type index for journals. Scientometrics 69, 169–173.

Cameron, B.D., 2005. Trends in the usage of ISI bibliometric data: Uses, abuses, and implications. Portal: Libraries and the Academy 5, 105–125.

Chames, A., Cooper, W., Rhodes, E., 1978. Measuring the efficiency of decision-making units. European Journal of Operational Research 6, 429–444.

Cook, W.D., Seiford, L.M., 2009. Data envelopment analysis (DEA)–Thirty years on. European Journal of Operational Research 192, 1–17.

Cooper, W.W., Seiford, L.M., Tone, K., 2000. Data Envelopment Analysis: A Comprehensive Text with Models, Applications, References. Kluwer

Academic Publishers, Boston, MA, USA.

Dong, P., Loh, M., Mondry, A., 2005. The impact factor revisited. Biomedical Digital Libraries 2, 1–8.

Doyle, J., Green, R.H., 1994. Efficiency and cross efficiency in DEA: Derivations, meanings and uses. Journal of the Operational Research Society 45,

567–578.

Emrouznejad, A., Yang, G.L., 2017. A survey and analysis of the first 40 years of scholarly literature in DEA: 1978–2016. Socio–Economic Planning

Sciences, in press.

Falagas, M.E., Kouranos, V.D., Michalopoulos, A., Rodopoulou, S.P., Batsiou, M.A., Karageorgopoulos, D.E., 2010. Comparison of the distribution of

citations received by articles published in high, moderate, and low impact factor journals in clinical medicine. Internal Medicine Journal 40, 587–591.

Färe, R., Grosskopf, S., Norris, M., Zhang, Z., 1994. Productivity growth, technical progress, and efficiency change in industrialized countries. American

Economic Review 84, 66–83.

Färe, R., Grosskopf, S., Norris, M., 1997. Productivity growth, technical progress, and efficiency change in industrialized countries: Reply. American

Economic Review 87, 1040–1043.

Garfield, E., 1955. Citation indexes for science. A new dimension in documentation through association of ideas. Science 122, 108–111.

Garfield, E., 2006. The history and meaning of the journal impact factor. JAMA: The Journal of the American Medical Association 295, 90–93.

Glänzel, W., Moed, H.F., 2002. Journal impact measures in bibliometric research. Scientometrics 53, 171–193.

Moed, H.F., Leeuwen, T.N.V., Reedijk, J., 1999. Towards appropriate indicators of journal impact. Scientometrics 46, 575–589.

Hirsch, J.E., 2005. An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States

of America 102, 16569–16572.

Lee, H., Shin, J., 2014. Measuring journal performance for multidisciplinary research: An efficiency perspective. Journal of Informetrics 8, 77–88.

Kravitz, D.J., Baker, C.I., 2011. Toward a new model of scientific publishing: Discussion and a proposal. Frontiers in Computational Neuroscience 5, 1–12.

Lariviere, V., Gingras, Y., 2010. The impact factor’s Matthew Effect: A natural experiment in bibliometrics. JASIST 61, 424–427.

Leydesdorff, L., 2012. Alternatives to the journal impact factor: I3 and the Top-10% (or Top-25%?) of the most–highly cited papers. Scientometrics 92,

355–365.

Malesios, C., Arabatzis, G., 2012. An evaluation of forestry journals using bibliometric indices. Annals of Forest Research 55, 147–164.

Malmquist, S., 1953. Index numbers and indifference surfaces. Trabajos de Estadistica 4, 209–242.

Merton, R.K., 1968. The Matthew Effect in science. International Journal of Dermatology 159, 16.
JMSE 2017, 2(1), 34 – 54 54

Petridis, K., Malesios, C., Arabatzis, G., Thanassoulis, E., 2013. Efficiency analysis of forestry journals: Suggestions for improving journals’ quality.

Journal of Informetrics 7, 505–521.

Ray, S.C., Desli, E., 1997. Productivity growth, technical progress, and efficiency change in industrialized countries: Comment. American Economic

Association 87, 1033–1039.

Seglen, P.O., 1997. Why the impact factor of journals should not be used for evaluating research. British Medical Journal 314, 497–502.

Sexton, T.R., Silkman, R.H., Hogan, A.J., 1986. Data envelopment analysis: Critique and extensions, in: Silkman, R.H. (Ed.), Measuring Efficiency: An

Assessment of Data Envelopment Analysis. Jossey-Bass, San Francisco, CA, USA.

Thorne, F.C., 1977. The citation index: Author case of spurious validity. Journal of Clinical Psychology 33, 1157–1161.

Vanclay, J.K., 2006. Refining the h-index. The Scientist 20, 14–15.

Vanclay, J.K., 2013. Factors affecting citation rates in environmental science. Journal of Informetrics 7, 265–271.

Vinkler, P., 2007. Eminence of scientists in the light of the h-index and other scientometric indicators. Journal of Information Science 33, 481–491.

Yang, G.L., Ahlgren, P., Yang, L.Y., Rousseau, R., Ding, J.L., 2016. Using multi-level frontiers in DEA models to grade countries/territories. Journal of

Informetrics, 238–253.

Yang, G.L., Rousseau, R., Yang, L.Y., Liu, W.B., 2014. A study on directional returns to scale. Journal of Informetrics 8, 628–641.

Yang, G.L., Yang, J.B., Liu, W.B., Li, X.X., 2013. Cross-efficiency aggregation in DEA models using the evidential–reasoning approach. European Journal

of Operational Research 231, 393–404.

© 2017 by the authors; published by Science Press (CSPM Ltd.). This article is an open access article distributed under the terms and

conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).