Automated Approaches to Support Secondary Study Processes: a Systematic Review

Jefferson Seide Mollri

UNIVALI Universidade do Vale do Itaja Itaja, Brazil jefferson.molleri@univali.br
Abstract Context: There is the need to identify automated methodologies for supporting the systematic review process. Objectives: To conduct a systematic review of the literature searching for automated approaches used by researchers to support the systematic review process. Method: We undertook a systematic review following the guidelines set out in Kitchenham and Charters, analyzed the relevant studies, and compared our findings with previous studies. Results: 508 papers have been identified and reviewed according to inclusion and exclusion criteria, resulting in 31 relevant studies. Conclusions: A wide variation in the approaches were identified, in the concentrate context of selection on of primary studies, data extraction and monitoring and data synthesis from the conducting the review phase.

Fabiane Barreto Vavassori Benitti

UNIVALI Universidade do Vale do Itaja Itaja, Brazil fabiane.benitti@univali.br In the face of difficulties encountered during the execution of systematic reviews, there is the need to invest efforts in research methodologies for planning and carrying out systematic reviews [1]. The support of a automated tool is essential to provide higher quality and facilitate the systematic literature review process [6]. This short report outlines the results of a systematic review aiming to identify the current state of the art of automated approaches to support the systematic review process. II. METHODS

This research follows the guidelines set out in Kitchenham and Charters [3] guide for SLRs in software engineering. This research method provides a verifiable method of summarizing existing studies as well as identifying gaps in the current research. A. Research questions The review process begins with the construction of a protocol that contains the general scope of the study. The scope can contain a Population, Intervention, Comparison, Outcomes and Context criteria to frame research questions according to the PICOC method, referred per Kitchenham and Charters guide [3]. Population: researchers Intervention: automated approaches to support Comparison: phases and stages of the systematic review process Outcomes: productivity and reliability Context: secondary studies

I.

INTRODUCTION

The Systematic Literature Review (SLR) is defined as a specific methodology of research, developed in order to gather and evaluate the available evidence pertaining to a focused topic [1]. Systematic reviews are gaining popularity in software engineering, with reviews published on diverse topics, such as software engineering experiments. SLRs are a key tool for enabling evidence-based practice as they combine the findings from multiple studies. Such reviews are important, as the volume of research that needs to be considered by software engineering (SE) researchers is constantly expanding. [2] Despite its importance, the systematic literature process is not a easy task, as it uses specific concepts usually unknown to researchers familiar with traditional (unsystematic) literature reviews. Even when they are conducted according to their corresponding good practice rules, they suffer from lack of scientific rigor in performing its different steps. The development of a systematic approach of research review aims to establish a more formal and controlled process of conducting this type of investigation, avoiding the introduction of the biases of the unsystematic review. [1] Moreover, the systematic reviews require considerably more effort than traditional literature reviews, since provide additional information on variations in the primary studies in a wide variety of empirical methods [3]. The accomplishment of experimental studies in Software Engineering is time consuming, hard task, and produces great volume of information and knowledge with complex management [4]. Hence, some studies strongly depend on a computerized infrastructure to support its processes [5].

Every systematic review has at least one primary research question with the possibility of more secondary questions. The following are the primary (RQ1) and secondary research questions (RQ2, RQ3 and RQ4) for this study: RQ1: Which automated approaches to support the secondary studies processes are used by researchers? RQ2: Which the phases and stages are supported by the identified approaches? RQ3: What limitations, phases and stages are not covered by the identified approaches?

RQ4: What are the impacts of using an automated approach into productivity and reliability criteria the secondary studies process?

Non-English studies; Duplicated studies; and Was unable to access the full text.

B. Data sources For this SLR, the electronic databases in Table I were searched as these are the primary sources for software engineering research publications.
TABLE I.
Database

DATA SOURCES
URL

Although it was not possible to check every single study by a secondary reviewer, an inter-rater reliability test was performed in order to reduce the researcher's bias. The secondary reviewer selected 5 studies randomly from the list of relevant primary studies and performed the study selection process. The results between reviewers were compared and no differences were found. E. Data extraction The data extraction process was carried out by reading each of the 31 selected studies thoroughly and extracting relevant data, which were managed through an Microsoft Excel spreadsheet. In order to keep information consistent the data extraction for each selected study was driven by the following:
TABLE II. Extracted Data Bibliographic references Study classification Focus of the study Application Application context Application results DATA EXTRACTION FOR EACH STUDY Description Author, year of publication, title and source of publication Primary or secondary study Main topic area, and objective of the study Automated approach referenced Phase and stages of review process supported Productivity and reliability data Research questions RQ1 RQ1 RQ1 RQ1 RQ2, RQ3 RQ4

ACM Digital Library CiteSeer IEEE Explore IET The Institution of Engineering and Technology ScienceDirect SpringerLink

http://portal.acm.org http://citeseerx.ist.psu.edu http://ieeexplore.ieee.org http://www.theiet.org/ http://www.sciencedirect.com http://www.springerlink.com

C. Search string The PICOC method gives us a list of search terms to use in the electronic databases. The search string used the logical operator OR to include synonyms for each search term, and the logical operator AND to link together each set of synonyms. When concatenated using the appropriate boolean expressions the following generic search string was produced: ('secondary study' OR 'systematic review' OR 'literature review' OR 'mapping review') AND (software OR application) AND (planning OR conducting OR reporting OR 'research question' OR 'review protocol' OR 'study selection' OR 'quality assessment' OR 'data extraction' OR 'data synthesis' OR 'meta analysis' OR meta-analysis) AND (productivity OR reliability OR 'effort reduction') The generic search string was adapted to match the individual requirements of each of the electronic database on our data sources list above. D. Study selection In order to determine whether or not a study should be included, the methods section or similar section of the primary study was evaluated in search for citations of the automated approaches to support the review. Studies that were selected for inclusion in this systematic review were identified from online electronic databases within the following criteria: Primary studies that present automated approaches to support the conduct of systematic reviews; and Secondary studies (systematic reviews and mappings, etc.) that refer the automated support tools used.

F. Data synthesis For the data synthesis, we inspected the extracted data for similarities in order to define how results could be encapsulated. The results of the synthesis will be described in the subsequent sections. Extracted data were tabulated to demonstrate the basic information of each study. With respect to qualitative assessment of the studies, the following criteria were summarized: Bibliographic references and main topic area; Number of selected studies per database; Classification and studies by year of publication; Process phases and stages most widely covered; and Automated approaches most frequently cited. III. RESULTS

This review excluded studies based on the exclusion criteria, which have been built in order to exclude irrelevant publications while maintaining the studies of interest. Are related to non-automated approaches; Do not address the approaches used;

After the literature search, 508 potentially relevant primary studies have been obtained. The larger sets of results came from ScienceDirect (305 papers) and SpringerLink (95). Studies were then reviewed according to the predefined inclusion and exclusion criteria, focused on the methods section that references the review methodology and the

approaches to support the review process, reducing the number to 31 selected studies, as shown in Table III.
TABLE III. BIBLIOGRAPHIC REFERENCES FROM SELECTED STUDIES Main topic area Computer Science Computer Science Computer Science Computer Science Computer Science Computer Science Medicine Computer Science Computer Science Medicine Medicine Medicine Computer Science Computer Science Computer Science Computer Science Medicine Computer Science Medicine Computer Science Medicine Computer Science Medicine Computer Science Medicine Computer Science Computer Science Medicine Computer Science Medicine Envtl. Science Data source IEEE Xplore IEEE Xplore IEEE Xplore IEEE Xplore IEEE Xplore SpringerLink SpringerLink SpringerLink SpringerLink SpringerLink SpringerLink SpringerLink ScienceDirect ScienceDirect ScienceDirect ScienceDirect ScienceDirect ScienceDirect ScienceDirect ScienceDirect ScienceDirect ScienceDirect ScienceDirect ScienceDirect ScienceDirect ScienceDirect ScienceDirect ScienceDirect ScienceDirect ScienceDirect ScienceDirect

Authors Ali et al. (2010) [6] Bailey et al. (2007) [7] Dieste and Juristo, (2011) [8] Svensson et al. (2010) [9] Salleh et al. (2010) [10] mite et al. (2010) [11] Campbell et al. (2011) [12] Ivarsson et al. (2011) [13] Gu and Lago (2009) [14] Roberts et al. (2008) [15] Umoquit et al. (2011) [16] Schwappach et al. (2007) [17] Lane and Richardson (2010) [18] Beecham et al. (2007) [19] Dyb and Dingsyr (2008) [20] Breivold et al. (2011) [21] Eadie et al. (2011) [22] Hannay et al. (2009) [23] Bastani and Jaberzadeh (2011) [24] Mellado et al.(2010) [25] Liu et al. (2010) [26] Kitchenham (2009) [27] Klainin-Yobas et al. (2011) [28] Chen et al. (2009) [29] Wojtusiak et al. (2009) [30] Zhang et al. (2010) [31] Talaei-Khoei et al. (2011) [32] Watt et al. (2009) [33] Jahangirian et al. (2010) [34] Benchimol et al. (2010) [35] Belanger (1997) [36]

maturity of an area is the type of its publications. Journal articles are often more mature than conference papers [18]. Table V shows the number of each type of publication by year. Journal articles make up the majority (93,54%) of publications. These results likely to show that the more mature articles references best automated approaches used than studies in conferences and symposium papers.
TABLE V. Classification
1997

TYPES OF PUBLICATION BY YEAR Year of Publication

2007 2008 2009 2010 2011 2012

Primary Studies Journal Secondary Studies Conference Journal 1 Symposium 1 Total

1 1 1
2

3
3

2
3

1 7
8

9
9

5
5

C. Application Context A systematic literature review involves several tasks and activities. Kitchenham summarises the stages in a systematic review into three main phases: Planning the Review, Conducting the Review and Reporting the Review [3]. These three phases represent an overview of the systematic review process that could benefit form support of the automated approaches to grant productivity and reliability of execution. Throughout this SLR, all selected studies focused on the Conducting the Review stage, although they are distributed among some discrete activities covered by the five stages shown in Table VI. About 30,77% of the automated approaches address the selection of primary studies, 25,64% the data extraction and monitoring and 43,59% the data synthesis stage. These results show a lack of comprehensiveness of the approaches on the process stages and phases.
TABLE VI. Context (Phase / Stage) Conducting the Review Identification of research Selection of primary studies Data extraction and monitoring Study quality assessment Data synthesis Total APPLICATION CONTEXT Automated approaches* 0 12 (30,77%) 10 (25,64%) 0 17 (43,59%)
39

A. Selected Studies Among the electronic databases, IEEE Xplore provide most relevant studies: about 23,8% of the 21 searched studies were selected, as Table IV. ScienceDirect and SpringerLink databases had a higher amount of selected studies, but with a lower rate of selected per searched papers. One single study was obtained in ACM Digital database, however this was a duplicated paper already found in IEEE Xplore. Neither CiteSeer or EIT had selected studies among searched ones.
TABLE IV. Database IEEE Xplore SpringerLink ScienceDirect ACM Digital CiteSeer IET SELECTED STUDIES PER DATABASE Studies
Searched Selected

* In total 39 references to automated approaches were found in 31 studies selected

21 95 305 39 4 2
508

5 (23.8)* 7 (7.36%) 19 (6.22%) 1 (2.56%)* 31 (6.1%)

* IEEE Xplore and ACM Digital had a duplicated study

B. Studies Classification From the 31 papers selected there were 30 secondary studies and only one primary study, which suggest that the support of automated approaches in systematic review process is lacking in primary research. A good indication of the

D. Automated Approaches From the selected papers there was a wide variation in the approaches described, as well the context of their uses. Table VII shows the list of automated approaches referenced in selected studies. Microsoft Excel is the most common software identified in our study, i.e. 17,95%, but EndNote (15,38%), RevMan (12,82%) and Reference Manager (7,69%) were also well referenced. Two studies quote also a tool database that is not appointed in the article. Some of the identified automated approaches covers particularly the selection of primary studies data stage: Refman, Zotero, SCOPUS and WordStat. Others concentrated

in the extraction of data, such as Zotero, MS Access, and MIX. A number of other tools also support data synthesis: RevMan, Comprehensive Meta-Analysis, MySQL database, AQ21, Nvivo, PASW, PROCITE, SAS and STATA. Microsoft Excel was identified as a support tool to all the three stages listed, especially on data extraction and monitoring. EndNote was also referred as an approach capable to address both the selection of studies and data extraction stages of SLR process.
TABLE VII. AUTOMATED APPROACHES REFERENCED Selection of primary studies Data extraction and monitoring Data synthesis

tools were also well cited, such as EndNote, Reference Manager and RevMan. In order to answer RQ2 and RQ3, the application contexts of the identified approaches were extracted and summarized, showing that most approaches focus on Conducting the Review phase. This implies a lack of adequate automated procedures to researchers in Planning and Reporting phases, or ignorance of researchers about these approaches. These phases covers important activities of the SLR process, such as defining the research questions, producing a review protocol and writing up the results. Even the entire Conducting the Review stage was not supported by the tools identified in this study. It became clear the lack of automated approaches to support the Identification of research and Study quality assessment stages. It is important to recognize that many activities are initiated during a preliminary stage and refined when subsequent stages takes place, involving iteration methods [3] not showed by the identified approaches. With respect to the impacts of using an automated approach into productivity and reliability criteria, the review found no clear evidences in the studies that could be summarized. A single primary study [30] discussed application methods of the approach in detail, but lacked an evaluation process to provide evidences on the reduction of effort and quality assurance of the SLR process. A. Threats to Validity As with SLRs in general there is often a lot of subjectivity involved in the selection of primary studies as well as the data extraction stages. In order to minimize the subjectivity involved in this study the reviewers strictly adhered to the instructions set out in the review protocol. The majority of the SLR activities were conducted by the primary reviewer. In order to reduce the single reviewer bias, an inter-rater reliability test was performed by the secondary reviewer. There were no difference between the results of the two reviewers. Due to lack of the productivity and reliability data in results, the impacts of using an automated approach to support the systematic review process could not be observed. We notice that this issue may have two interpretations: the search string may be inaccurately constructed (although supported by PICOC method); or this topic of research has not yet reported results. We consider this issue a gap in the state of the art, and recommend this research question to be addressed by further studies. V. CONCLUSION

Total

Automated approach

Microsoft Excel [6], [11], [20], [21], [34], [35], [36] EndNote [14], [19], [20], [21], [25], [36] RevMan - Review Manager [12], [24], [26], [29], [33], Reference Manager (Refman) [15], [16], [20] Comprehensive Meta-Analysis [23], [28] MySQL database [7], [13] unnamed database tool [9], [17] Zotero [18], [32] AQ21 attributional rule learning program [30] Microsoft Access [18] MIX: Comprehensive Free Software for Meta-Analysis of Causal Research Data [10] Nvivo [20] PASW statistic [28] PROCITE [8] SAS [36] SCOPUS (search facilities) [27] STATA [17] WordStat [31] Total

1 4

4 2

7 6

5 3 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 12 10 17

5 3 2 2 2 2 1 1 1 1 1 1 1 1 1 1 39

* In total 39 references to automated approaches were found in 31 studies selected

E. Productivity and Reliability In order to answer RQ4 we aimed to obtain the results concerning productivity and reliability in application of the identified automated approaches. Productivity criteria should be scored by the reduction of effort in the SLR, and reliability would be given by the completion of discrete activities in accordance with process guidelines. However, no article had demonstrated productivity and reliability data that could be collected. IV. DISCUSSION

This systematic reviews overall goal, addressed by RQ1, is to identify existing automated approaches to support the systematic review process. Among the approaches identified, Microsoft Excel was most often cited, at three different stages of the process. This fact suggests that spreadsheets are the prime choice of researchers to support the process of conducting a systematic review. Other reference management

In order to enhance the comprehension of the state of the art, this paper reports the results of a secondary study on searching relevant automated approaches to support the systematic review process. A wide variation in the approaches were identified, supporting particularly some discrete activities of the Conducting the Review phase. Since no relevant study provided adequate results concerning productivity and reliability of the approaches, this paper serves well as a basis for further research efforts in the search for evidences of improvements in SLR process through the support of automated approaches.

REFERENCES
[1] J. Biolchini, P. G. Mian, A. C. Natali, and G. H. Travassos, Systematic Review in Software Engineering: Relevance and Utility, Rio de Janeiro, Brazil: Systems Engineering and Computer Science Department, UFRJ. T. Dyb, and T. Dingsyr, Strength of evidence in systematic reviews in software engineering, Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement, Kaiserslautern, Germany, pp. 178187, October 2008. B. Kitchenham and S. Charters. Guidelines for performing systematic literature reviews in software engineering (version 2.3), Technical report, Keele University and University of Durham, 2007. F. Shull, J. Carver, and G. H. Travassos, An empirical methodology for introducing software processes, SIGSOFT Softw. Eng. Notes}, vol. 26, no. 5, pp. 288296, September 2001. G. H. Travassos, P. S. M. dos Santos, P. G. M. Neto, and J. Biolchini, An Environment to Support Large Scale Experimentation in Software Engineering, Engineering of Complex Computer Systems, 2008. ICECCS 2008. 13th IEEE International Conference on, vol., no., pp.193 202, Mar.-Apr. 2008. A. Zamboni, A. Thommazo, E. C. M. Hernandes, and S. C. P. F. Fabbri, StArt Uma Ferramenta Computacional de Apoio Reviso Sistemtica, in CBSOFT Congresso Brasileiro de Software: Teoria e Prtica, 2010, Salvador, Anais do Congresso Brasileiro de Software, 2010, vol. 1, no., pp. 16, Setembro 2010. S. Ali, L. C. Briand, H. Hemmati, and R. K. Panesar-Walawege. A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation, Software Engineering, IEEE Transactions on , vol. 36, no. 6, pp. 742-762, Nov.-Dec. 2010. J. Bailey, D. Budgen, M. Turner, B. Kitchenham, P. Brereton, S. Linkman. Evidence relating to Object-Oriented software design: A survey, Empirical Software Engineering and Measurement, 2007. ESEM 2007. First International Symposium on , vol., no., pp. 482-484, 20-21 Sept. 2007. O. Dieste and N. Juristo. Systematic review and aggregation of empirical studies on elicitation techniques, Software Engineering, IEEE Transactions on , vol. 37, no. 2, pp. 283-304, March-April 2011. R. B. Svensson, M. Host and B. Regnell. Managing Quality Requirements: A Systematic Review, Software Engineering and Advanced Applications (SEAA), 2010 36th EUROMICRO Conference on , vol., no., pp. 261-268, 1-3 Sept. 2010. N. Salleh, E. Mendes and J. Grundy. Empirical Studies of Pair Programming for CS/SE Teaching in Higher Education: A Systematic Literature Review, Software Engineering, IEEE Transactions on , vol. 37, no. 4, pp. 509-525, July-Aug. 2011. D. mite, C. Wohlin, T. Gorschek, and R. Feldt, Empirical evidence in global software engineering: A systematic review, Empirical Software Engineering, vol. 15, no. 1, pp. 91-118, February 2010. F. Campbell, M. Johnson, J. Messina, L. Guillaume and E. Goyder, Behavioural interventions for weight management in pregnancy: a systematic review of quantitative and qualitative data, BMC Public Health, vol 11, no. 1, pp., December 2011. M. Ivarsson and T. Gorschek, A method for evaluating rigor and industrial relevance of technology evaluations, Empirical Software Engineering, vol. 16, no. 3, pp. 365-395, June 2011. Q. Gu and P. Lago, Exploring service-oriented system engineering challenges: A systematic literature review, Service Oriented Computing and Application, vol. 3, no. 3, pp. 171-188, September 2009. J. Roberts, A. Huissoon, J. Dretzke, D. Wang, and C. Hyde, A systematic review of the clinical effectiveness of acupuncture for allergic rhinitis, BMC Complement Altern Med, vol. 8, no. 1, pp., December 2008. M. J .Umoquit, P. Tso, H. E. D. Burchett, and M. J. Dobrow, A multidisciplinary systematic review of the use of diagrams as a means of collecting data from research subjects: application, benefits and recommendations, BMC Medical Research Methodology, vol. 11, no. 1, pp., December 2011.

[2]

[3] [4] [5]

[6]

[7]

[8]

[9] [10]

[11]

[12] [13]

[14] [15] [16]

[17]

[18] D. Schwappach, and T. Boluarte, HEE-GER: a systematic review of German economic evaluations of health care published 19902004, BMC Health Services Research, vol. 7, no. 1, December 2007. [19] S. Lane, and I. Richardson, Process models for service-based applications: A systematic literature review, Information and Software Technology, vol. 53, no. 5, pp. 424439, May 2011. [20] S. Beecham, N. Baddoo, T. Hall, H. Robinson, and H. Sharp, Motivation in Software Engineering: A systematic literature review, Information and Software Technology, vol. 50, no. 910, pp. 860878, August 2008. [21] T. Dyb, and T. Dingsyr, Empirical studies of agile software development: A systematic review, Information and Software Technology, vol. 50, no. 910, pp. 833859, August 2008. [22] H. P. Breivold, I. Crnkovic, and M. Larsson, A systematic review of software architecture evolution research, Information and Software Technology, vol. 54, no. 1, pp. 1640, January 2012. [23] L. H. Eadie, P. Taylor, and A. P. Gibson, A systematic review of computer-assisted diagnosis in diagnostic cancer imaging, European Journal of Radiology, vol. 81, no. 1, pp. e70e76, January 2012. [24] J. E. Hannay, T. Dyb, E. Arisholm, and D. I. K. Sjberg, The effectiveness of pair programming: A meta-analysis, Information and Software Technology, vol. 51, no. 7, pp. 11101122, July 2009. [25] A. Bastani, and S. Jaberzadeh, Does anodal transcranial direct current stimulation enhance excitability of the motor cortex and motor function in healthy individuals and subjects with stroke: A systematic review and meta-analysis, Clinical Neurophysiology, vol. 123, no. 4, pp. 644657, April 2012. [26] D. Mellado, C. Blanco, L. E. Snchez, and E. Fernndez-Medina, A systematic review of security requirements engineering, Computer Standards & Interfaces, vol. 32, no. 4, pp. 153165, June 2010. [27] F. Liu, M. Qiu, and S. Zhai, Tolerability and effectiveness of (S)amlodipine compared with racemic amlodipine in hypertension: A systematic review and meta-analysis, Current Therapeutic Research, vol. 71, no. 1, pp. 129, February 2010. [28] B. Kitchenham, Whats up with software metrics? A preliminary mapping study, Journal of Systems and Software, vol. 83, no. 1, pp. 3751, January 2010. [29] P. Klainin-Yobas, M. A. A. Cho, and D. Creedy, Efficacy of mindfulness-based interventions on depressive symptoms among people with mental disorders: A meta-analysis, International Journal of Nursing Studies, vol. 49, no. 1, pp. 109121, January 2012. [30] S. Chen, A. Flower, A. Ritchie, J. Liu, A. Molassiotis, H. Yu, and G. Lewith, Oral Chinese herbal medicine (CHM) as an adjuvant treatment during chemotherapy for non-small cell lung cancer: A systematic review, Lung Cancer, vol. 68, no. 2, pp. 137145, May 2010. [31] J. Wojtusiak, R. S. Michalski, T. Simanivanh, and A. V. Baranova, Towards application of rule learning to the meta-analysis of clinical data: An example of the metabolic syndrome, International Journal of Medical Informatics, vol. 78, no. 12, pp. e104e111, December 2009. [32] H. Zhang, M. A. Babar, and P. Tell, Identifying relevant studies in software engineering, Information and Software Technology, vol. 53, no. 6, pp. 625637, June 2011. [33] A. Talaei-Khoei, P. Ray, N. Parameshwaran, and L. Lewis, A framework for awareness maintenance, Journal of Network and Computer Applications, vol. 35, no. 1, pp. 199210, January 2012. [34] A. M. Watt, M. Patkin, M. J. Sinnott, R. J. Black, and G. J. Maddern, Scalpel safety in the operative setting: A systematic review, Surgery, vol. 147, no. 1, pp. 98106, January 2010. [35] M. Jahangirian, T. Eldabi, L. Garg, G. T. Jun, A. Naseer, B. Patel, L. Stergioulas, and T. Young, A rapid review method for extremely large corpora of literature: Applications to the domains of modelling, simulation, and management, International Journal of Information Management, vol. 31, no. 3, pp. 234243, June 2011. [36] E. I. Benchimol, D. G. Manuel, T. To, A. M. Griffiths, L. Rabeneck, and A. Guttmann, Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data, Journal of Clinical Epidemiology, vol. 64, no. 8, pp. 821829, August 2011. [37] S. E. Belanger, Literature Review and Analysis of Biological Complexity in Model Stream Ecosystems: Influence of Size and

Experimental Design, Ecotoxicology and Environmental Safety, vol. 36, no. 1, pp. 116, February 1997.