Applied Ergonomics 43 (2012) 632e636

Contents lists available at SciVerse ScienceDirect

Applied Ergonomics
journal homepage: www.elsevier.com/locate/apergo

An investigation of the reliability of Rapid Upper Limb Assessment (RULA) as a method of assessment of childrens computing posture
Sara Dockrell*, Eleanor OGrady, Kathleen Bennett, Clare Mullarkey, Rachel Mc Connell, Rachel Ruddy, Seamus Twomey, Colleen Flannery
Discipline of Physiotherapy, School of Medicine, Trinity Centre for Health Sciences, St. Jamess Hospital, Jamess St., Dublin 8, Ireland

a r t i c l e i n f o
Article history: Received 28 October 2010 Accepted 26 September 2011 Keywords: Reliability RULA Posture Children

a b s t r a c t
Rapid Upper Limb Assessment (RULA) is a quick observation method of posture analysis. RULA has been used to assess childrens computer-related posture, but the reliability of RULA on a paediatric population has not been established. The purpose of this study was to investigate the inter-rater and intra-rater reliability of the use of RULA with children. Video recordings of 24 school children were independently viewed by six trained raters who assessed their postures using RULA, on two separate occasions. RULA demonstrated higher intra-rater reliability than inter-rater reliability although both were moderate to good. RULA was more reliable when used for assessing the older children (8e12 years) than with the younger children (4e7 years). RULA may prove useful as part of an ergonomic assessment, but its level of reliability warrants caution for its sole use when assessing children, and in particular, younger children. 2011 Elsevier Ltd and The Ergonomics Society. All rights reserved.

1. Introduction Studies have shown that computer-related activities are risk factors for musculoskeletal pain among adolescents (Hakala et al., 2006), and more recently a U-shaped relationship between internet use and adolescent health has been established (Blanger et al., 2011). With increasing numbers of children using computers daily it is important to assess the impact of that use on their posture as they may be at risk of developing computer-related musculoskeletal disorders. Methods of assessment of childrens posture during computer use described in the literature include Rapid Upper Limb Assessment (Oates et al., 1998; Laeser et al., 1998; Breen et al., 2007; Kelly et al., 2009) and 3D Motion Analysis Systems (Straker et al., 2002; Briggs et al., 2004; Maslen and Straker, 2009). 1.1. Rapid Upper Limb Assessment Rapid Upper Limb Assessment (RULA) is a subjective observation method of posture analysis that focuses on the upper body, but includes the lower body. It uses a series of illustrations of different body postures, as shown in Fig. 1, and a numerical score is allocated to the most common observed posture (McAtamney and Corlett, 1992). Group A consists of the upper arm, lower arm and wrist,

* Corresponding author. Tel.: 353 1 8962126; fax: 353 1 4531915. E-mail address: sara.dockrell@tcd.ie (S. Dockrell).

and group B consists of the neck, trunk and legs. The score for group A postures and group B postures and the scores for static muscle work and force are added as appropriate to give a C score (upper limb) and a D score (neck, trunk and legs). The C and D scores are then combined in a table to give a Grand Score. The Grand Score is used to assign the observed posture into an Action Level that indicates the required intervention (Table 1). RULA was developed as a screening tool for exposure of adults to risk factors for workrelated upper limb disorders, and takes into account the repetitive movements and force that may be required for a task. It was designed to be carried out quickly and with minimal equipment or change to the working environment, and with minimal disruption to those under observation. It requires no previous skills in observation techniques and is easy to learn. RULA has previously been shown to be reliable with adults (McAtamney and Corlett, 1993). Statistical calculations were not published but the authors state that the scores indicated a high consistency among assessors. RULA has also been used to assess childrens posture, but there are no studies looking particularly at the reliability of RULA with children. Laeser et al. (1998) established the reliability of RULA prior to their study, which analysed the effects of computer workstation layout on childrens posture (11e15 years). An external group of observers rated four videotaped postures and compared the ratings with the lead investigators scores. RULA was found to be reliable (Pearsons r 0.96, p 0.039). Other studies that have also alluded to the reliability of RULA include Breen et al. (2007) who found r 0.946 when used to assess 9e10 year olds and

0003-6870/$ e see front matter 2011 Elsevier Ltd and The Ergonomics Society. All rights reserved. doi:10.1016/j.apergo.2011.09.009

S. Dockrell et al. / Applied Ergonomics 43 (2012) 632e636

633

usual working conditions. This was to improve external validity. The second phase of the study was laboratory-based with undergraduate physiotherapy students and experienced physiotherapists using RULA to assess the posture of the children in the video clips. 2.2. Sample size Calculations were made to establish the required sample size of three raters using a one-way Anova formula as suggested by Walter et al. (1998). The required number of video clips was twenty-four, in order to achieve a power of 80% and two-sided level of signicance (p < 0.05) to detect an ICC of 0.6 or greater, which is a moderate level of reliability. 2.3. Procedure The school principal was contacted by telephone to request a meeting to discuss the feasibility of doing the study in his school. Following the meeting the principal was given envelopes containing participant information leaets, consent and assent (for children who were 7 years and older) forms for distribution to potential participants. Parents/guardians of the children were requested to return completed consent and assent forms as appropriate to the class teacher in the self-addressed envelope provided. A period of seven days was allowed between distribution and collection of the consent and assent forms. The participants and their parents were aware that the study involved the assessment of childrens computing posture. Three children chosen by the principal from each of the eight classes in the school were included in the study. The only request from the researchers was that the principal should include at least one girl and one boy from each class to ensure that there was no gender bias in the sample.
Fig. 1. RULA posture scores for body part group A and B. McAtamney and Corlett (1992).

2.4. Measures Data collection commenced once consent and assent were given. Children were lmed in the computer room in their class groups i.e. in groups of three. The children sat at a computer workstation to work on an assignment given to them by the teacher who was also present in the computer room. Each child was lmed using a Sony Handycam DCR e SR52E for approximately 2 min once they were engrossed in a task. The videographer was seated on a chair with castors to facilitate ease of movement from the lateral to the posterior aspects within the same lm clip. It also allowed the camera to be level with the seated children rather than distorting the angle by lming from above. Filming started from the lateral aspect on the right hand side of the child, irrespective of the childs hand dominance, panned around to the posterior aspect of the child and then back to the lateral aspect again. Video les were converted to mpeg-4 for editing purposes, and the childrens faces were blurred. The video les were exported to digital video disc (DVD) studio where DVD menus were authored enabling individual clip play or playing of the entire sequence of clips. All of the individual clips were randomised over each of the three different DVDs with three different clip order sequences to reduce the learning or memory effects between sessions. Individual clips were named alphabetically. A training DVD with three clips, different from those used in the main study, was also authored. Four DVDs were burned in total. 2.5. Raters 2.5.1. Physiotherapy students An email was sent to all undergraduate physiotherapy students in Trinity College Dublin (TCD) at the time of the study inviting

Oates et al. (1998) who cited the main observers reliability using RULA as Ebel r 0.73 when assessing 8.5e11.5 year olds. Although these studies have sought to test inter-rater reliability as a preliminary part of a study, none had focussed specically on the reliability of RULA with children. The aim of this study was therefore to investigate the inter-rater and intra-rater reliability of RULA when used to assess the posture of children aged 4e12 years. The objectives were: 1) to establish the inter-rater reliability of RULA in children, 2) to establish the intra-rater reliability of RULA in children, and 3) to investigate the association, if any, between childs age and reliability of RULA. 2. Methods 2.1. Study design The rst phase of the study was school-based and involved the collection of video footage of primary school children as they worked at computers in school during normal school hours and in

Table 1 Rapid Upper Limb Assessment (RULA) levels and indications. Grand Score 1 3 5 7 or or or or 2 4 6 more Action Level 1 2 3 4 Indications Posture is acceptable if not maintained. Further investigation needed. May need changes. Further investigation and changes needed soon. Investigation and changes required immediately.

634

S. Dockrell et al. / Applied Ergonomics 43 (2012) 632e636

them to participate in the study and providing them with the Participation Information Leaet and a letter of consent. Any student who was less than eighteen years of age or had a visual impairment as dened by the Disabilities Services, TCD was excluded. Three of those who had agreed to participate were randomly selected by placing each of the names in identical sealed envelopes and drawing them from a hat. An email was sent to the physiotherapy students informing them of their selection, and an email was also sent to the remaining physiotherapy students thanking them for their goodwill and time. 2.5.2. Physiotherapists An email was sent to all physiotherapists working in the School of Medicine, Trinity College Dublin inviting them to participate in the study and providing them with the Participation Information Leaet. Any physiotherapist who had been involved with the undergraduate student leg of the project or had a known visual impairment was excluded. The format for selection of the participants and notifying them of the outcome was the same as previously described for the undergraduate students. 2.6. Rater training and testing Raters attended a 45-min training session on the use of RULA. The session included a lecture/demonstration using a PowerPoint presentation to introduce RULA and detailing the allocation of the scores. This was followed by a practical session where raters could observe and evaluate four video clips of primary school children working on computers on a training DVD. Results were compared and discussed until the raters felt comfortable with RULA. Interpretations of the allocation of RULA scores were claried. Following the training, raters were allocated an individual time for their testing session. Raters were randomly assigned DVD 1, 2 or 3 depending on the order they walked into the room for the training session. A set of simple instructions was given to each participant before they started the testing session and the same procedure and protocol was adhered to for all raters. All testing took place under similar conditions with raters seated at a desk 3 m from a screen onto which the video clips were projected. Raters were asked to assess the right hand side posture of each child. Each rater was allowed to take as much time as she required after each video clip to complete the RULA scoring sheet. A break was offered to participants after every eight video clips to prevent fatigue, as established during the pilot stage of the project. The testing protocol was repeated one week later. Each rater viewed the video clips on a different randomly selected DVD than that viewed during the earlier session. Rater bias was minimised by separating the two sessions by one week, by using a different DVD for each rater in session 1 and session 2 and by the fact that the raters were requested to complete the RULA assessment sheet, but the researchers calculated the scores at a later stage. On completion of testing the Grand Scores and the Action Levels were calculated and cross checked by two different researchers.

2.7. Ethical issues Ethical approval was sought and granted by the Faculty of Health Sciences Ethics Committee in Trinity College Dublin. To ensure anonymity each child and each rater was assigned a coded number that only the research team were aware of. The facial images of the children were blurred in the video clips to ensure that no individual child could be identied. All of the data were stored in password protected computer les. 2.8. Data analysis The analysis of the data was performed with Microsoft Excel 2007 and Statistical Package for the Social Sciences (SPSS, v16.0). A signicance level of p < 0.05 was set. Reliability was calculated with Intraclass Correlation Coefcients (ICC). Inter-rater reliability was calculated with ICC model (2, 1) and intra-rater reliability using ICC model (3, 1), with a 95% condence interval, as suggested by Shrout and Fleiss (1979). As a general guideline, an ICC value below 0.50 represents poor reliability and a value ranging from 0.50 to 0.75 indicates moderate reliability, while values above 0.75 represent good reliability (Portney and Watkins, 2008). 3. Results 3.1. Sample Six people (three physiotherapists and three undergraduate physiotherapy students) were randomly selected to participate in the study. All of the participants were female, but this reected the gender of those who volunteered to participate, and also the ratio of male: female physiotherapists and students who were eligible to participate (1:9; 1:3.5 respectively). The mean age of the students was 22.2 years (range 21e24) and the mean age of the physiotherapists was 37.3 years (range 31e45). 3.2. Inter-rater reliability Inter-rater reliability of Action Level (AL) and Grand Score (GS) was mostly moderate for both students and physiotherapists. The AL and GS scores were more reliable in session 2 than in session 1 for the students and for the physiotherapists in all cases to varying degrees. The inter-rater reliability of D Scores (trunk and legs) was better than C Scores (arms) with very little between-session difference for D Scores, as shown in Table 2. 3.3. Intra-rater reliability The ICCs for intra-rater reliability for Action Level and Grand Score were mostly moderate to good. The intra-rater reliability of D Scores was the most reliable of all ndings, with the exception of Student 1 (Table 3). There was an association between childs age and the reliability of RULA. The reliability was generally better for

Table 2 Inter-rater reliability (95% CI) by session, and by type of rater. Inter-rater reliability ICC (2,1) 95% CI Action Levels Session 1 Session 2 Grand Scores Session 1 Session 2 C Scores Session 1 Session 2 D Scores Session 1 Session 2

Rater 1 All 0.69 (0.49e0.81) 0.72 (0.54e0.83) 0.71 (0.53e0.83) 0.77 (0.62e0.86) 0.54 (0.25e0.73) 0.68 (0.48e0.81) 0.76 (0.62e0.86) 0.76 (0.61e0.86) vs. Rater 2 Physios 0.54 (0.09e0.79) 0.58 (0.18e0.81) 0.50 (0.01e0.77) 0.68 (0.36e0.85) 0.55 (0.12e0.79) 0.68 (0.38e0.85) 0.72 (0.46e0.87) 0.75 (0.50e0.88) vs. Rater 3 Students 0.58 (0.17e0.81) 0.67 (0.34e0.85) 0.62 (0.24e0.82) 0.66 (0.33e0.84) 0.34 (0.30e0.69) 0.59 (0.19e0.81) 0.64 (0.29e0.83) 0.68 (0.37e0.85)

S. Dockrell et al. / Applied Ergonomics 43 (2012) 632e636 Table 3 Intra-rater reliability (95% CI) by type of rater. Intra e rater reliability ICC (3,1) 95% CI Action Level Student 1 Student 2 Student 3 Physiotherapist 1 Physiotherapist 2 Physiotherapist 3 0.66 0.62 0.52 0.27 0.69 0.86 (0.23e0.86) (0.13e0.84) (0.12e0.79) (0.69e0.68) (0.29e0.87) (0.67e0.94) Grand Score 0.83 0.64 0.47 0.55 0.75 0.84 (0.62e0.93) (0.17e0.84) (0.23e0.77) (0.03e0.81) (0.41e0.89) (0.62e0.93) C Score 0.89 0.31 0.41 0.48 0.83 0.81 (0.76e0.95) (0.59e0.70) (0.36e0.74) (0.20e0.77) (0.62e0.93) (0.55e0.92) D Score 0.57 0.79 0.68 0.68 0.88 0.89

635

(0.00e0.81) (0.51e0.91) (0.26e0.86) (0.27e0.86) (0.72e0.95) (0.75e0.95)

the older children compared to the younger children, especially when RULA was used by the physiotherapists. (Table 4). 4. Discussion In any outcome measure each observed score is the composite of the true score and random error (Rankin and Stokes, 1998). These errors may be due to natural variation in the subject, variation in the measurement process, or both. For this study, possible areas of error were identied and controlled to gain more reliable results. A strict protocol was adhered to in an effort to decrease the amount of variable inuences on the results. All participants were given the same training. Sessions 1 and 2 were held in the same venue and wherever possible at the same time of day for each participant. Care was taken to ensure that the environment was the same for the two sessions. The same desk and chair were used and the distance of the desk from the screen and the room lighting were standardised. The same video clips were used in sessions 1 and 2, albeit in a different randomised order to counteract the effects of fatigue that might occur from beginning to end of the sessions, and also to prevent a learning effect. Further efforts to eliminate or decrease a learning effect included the provision of thorough training and the opportunity for participants to practice on training DVD clips prior to doing the assessments. However, there would appear to have been a modest learning effect as the reliability was always higher for session 2 than for session 1. A longer, more in-depth training session or a third session may be needed in future studies to estimate and control this effect more accurately. The results indicate that RULA is a moderately reliable tool for use on school children. The reliability demonstrated here is not as high as that reported in other studies (Laeser et al., 1998; Breen et al., 2007) on children. However reliability testing was not the main focus of study in previous research reports and therefore was not examined as closely or with the same rigour as in this study. RULA has been reported to have good reliability when used by physiotherapists, industrial and safety engineers to assess adults (McAtamney and Corlett, 1993), although the exact values are unknown. The participants of this study have likely had more posture analysis training than the engineers in McAtamney and Corletts study. It follows that a potential source for the poorer
Table 4 Comparison of Scores between junior (4e7 yrs) and senior (8e12 yrs). Intra-rater reliability ICC (3,1) 95% CI Action Level Junior (4e7 years) Student 1 Student 2 Student 3 Physiotherapist 1 Physiotherapist 2 Physiotherapist 3 0.66 (0.19e0.90) 0.73 (0.05e0.92) 0.00 (2.47e0.71) 0.20 (3.17e0.65) 0.00 (2.47e0.71) 0.64 (0.24e0.89)

reliability results is not the participants using RULA, but rather the application of RULA to children. Children universally and within this study tend to dget. Any movement can cause alterations in posture and a higher frequency of movement leads to an increased choice of postures. This could make the identication of the most common posture a more complex task in children than it is in adults. RULA remains more reliable in its application to adults than children. When used by the physiotherapists RULA had greater intra-rater reliability with the older children (8e12 years) than with the younger children (4e7 years). This was an expected nding as the older childrens stature is closer to that of an adult. It was also noted that the computing behaviour of the older children was more comparable to that of an adult than that of the younger children. This indicates that RULA may be suitable for use with older children, but may not be suitable for use with young children. This nding was not as clear cut for the physiotherapy students as only one out of the three students had greater reliability when using RULA with the older children. In the comparison between younger and older children some negative ICCs were noted. This could have occurred because the calculation of a negative ICC as a point estimate indicates that the observed variation within sessions is more than was expected by chance. This may be because of the small sample size or because the reliability is so poor. It was not possible to calculate the inter-rater reliability between RULA scores of the younger children (4e7 years) compared to the older children (8e12 years) because the variability was not large enough between the groups to calculate a valid measure of ICC. Several of the children assumed a supported slumped posture while sitting at the computer. This was also reported by Breen et al. (2007) and by Oates et al. (1998), who found 30e35% of fourth and fth graders to be in the slumped position. This posture is characterised by the child sliding forwards on the chair and weight bearing on the sacrum in a backward reclined position. In this position the spine is neither upright nor exed as shown on the RULA posture score sheet (Fig. 1). However, the thoracic area of the childs trunk is supported by the backrest of the chair, and therefore is often allocated a score of 1 for trunk. RULA does not differentiate the part of the spine that is supported by the backrest, although it does consider that the trunk should be well supported

Grand Score Senior (8e12 years) 0.53 0.38 0.69 0.18 0.58 0.89 (0.64e0.86) (1.16e0.82) (0.05e0.91) (1.86e0.76) (0.46e0.88) (0.64e0.97) Junior (4e7 years) 0.84 (0.46e0.95) 0.68 (0.12e0.91) 0.27 (3.41e0.63) 0.52 (0.66e0.86) 0.00 (2.47e0.71) 0.54 (0.58e0.87) Senior (8e12 years) 0.76 (0.17e0.93) 0.51 (0.72e0.86) 0.62 (0.33e0.89) 0.57 (0.49e0.88) 0.87 (0.55e0.96) 0.91 (0.68e0.97)

636

S. Dockrell et al. / Applied Ergonomics 43 (2012) 632e636

to be given a score of 1. Given the frequency of this posture among school children it is suggested that some additional guidance should be included in the training session on RULA prior to its use on school children and also that this section of RULA may need some modication when used with paediatric populations. Although the raters may have allocated a better score for trunk posture than was deserved, they were more consistent with the allocation of D Scores, which included neck, trunk and legs, than with any of the other scores. Hence D Score inter-rater and intrarater reliability was the highest. One explanation for this nding may be that the assessment of neck, trunk and legs posture is somehow simpler or more straightforward to assess than the upper limb posture. The assessment of wrist posture has been reported as being more difcult to assess than other joints (Dartt et al., 2009). A further explanation may be that there is less movement of the neck, trunk and legs compared to the upper limbs when a child is working at a computer, and static postures are easier to assess than dynamic postures with observational methods. None of the childrens Action Levels were considered acceptable based on the RULA assessments. This has also been the outcome in previous studies using RULA (Breen et al., 2007; Oates et al., 1998). The calculation of RULA Action Levels in practise may need to be modied, as it may be unreasonable to suggest that all of the children in this study and in other previous studies have unacceptable posture. One of the positive factors regarding children working at computers in school is that they do not maintain the same posture for long periods of time, and therefore may not currently be at risk of musculoskeletal disorders. However, the learned postures may be continued through to adulthood where there is a known association between computer use and musculoskeletal disorders (Ijmker et al., 2007). A limitation of the study was that the participants rated the posture of the children by viewing DVD clips. This is similar to the method used by McAtamney and Corlett (1993) to establish the reliability for use of RULA with adults. It may be preferable for RULA to be used in a real-life situation, and the reliability may have been different if the children were viewed live, but for practical reasons this was not possible. A further limitation is the subjective nature of the measure where the outcome may be dependent on an individual raters motivation. Each participants individual motivation during testing cannot be controlled, and this may have contributed to the low ICC values. The raters were permitted to request to see a DVD clip again or to pause for an amount of time to nish lling in the RULA form. Although RULA is not a timed method of postural assessment, Student 1 and Physiotherapist 3, who had the highest reliability within their own groups, asked for more frequent and longer pauses than the other raters, and therefore may have had a better opportunity to perform more thorough postural assessments. 5. Conclusion Given its moderate reliability, RULA may be useful as part of an ergonomic assessment of children working at computers. RULA does not have the sensitivity of a clinical measurement instrument, but it was never intended to be used as such. RULA is a screening tool used in conjunction with other tools of ergonomic assessment, such as anthropometric measurements, evaluation of computer

workstations and analysis of work practices. This study has shown that the reliability level is probably adequate for that purpose. However, the level of reliability of RULA found here warrants caution for its sole use when assessing school children, and in particular, younger children. A further consideration is that the levels of reliability found in this study were under controlled experimental conditions therefore there is a need for even more caution in situations where RULA is used on-site without the controls of a research protocol. RULA demonstrated higher intra-rater reliability than interrater reliability implying that serial assessments would be more consistent if carried out by the same person. There was also evidence of a learning effect with higher reliability in the second session compared to the rst session. This highlights the importance of adequate training and practice in its use before RULA is used. Acknowledgements Sincere thanks to the principal, children and teachers who took part or facilitated in the rst phase of the project, and to the raters and independent scorers who gave so generously of their time. References
Blanger, R., Akre, C., Berchtold, A., Michaud, P.-A., 2011. A U-shaped association between intensity of internet use and adolescent health. Pediatrics 127, 330e335. Breen, R., Pyper, S., Rusk, Y., Dockrell, S., 2007. An investigation of childrens posture and discomfort during computer use. Ergonomics 50, 1582e1592. Briggs, A., Straker, L., Greig, A., 2004. Upper quadrant postural changes of schoolchildren in response to interaction with different information technologies. Ergonomics 47, 79e819. Dartt, A., Rosecrance, J., Gerr, F., Chen, P., Anton, D., Merlino, L., 2009. Reliability of assessing upper limb postures among workers performing manufacturing tasks. Appl. Ergon. 40, 371e378. Hakala, P., Rimpel, A., Saarni, L., Salminen, J., 2006. Frequent computer-related activities increase the risk of neckeshoulder and low back pain in adolescents. Eur. J. Pub. Health 16, 536e541. Ijmker, S., Husamans, M., Blatter, B., van der Beek, A., van Mechelen, W., Bongers, P., 2007. Should ofce workers spend fewer hours at their computer? A systematic review of the literature. Occup. Environ. Med. 64, 211e222. Kelly, G., Dockrell, S., Galvin, R., 2009. Computer use in school: its effect on posture and discomfort in schoolchildren. Work 32, 321e328. Laeser, K., Maxwell, L., Hedge, A., 1998. The effect of computer workstation design on student posture. J. Res. Comput. Educ. 31, 173e188. Maslen, B., Straker, L., 2009. A comparison of posture and muscle activity means and variation amongst young children, older children and young adults whilst working with computers. Work 32, 311e320. McAtamney, L., Corlett, N., 1992. Reducing the Risks of Work Related Upper Limb Disorders. A Guide and Methods. The Institute of Occupational Ergonomics, Nottingham. McAtamney, L., Corlett, N., 1993. RULA: a survey method for the investigation of work-related upper limb disorders. Appl. Ergon. 24, 91e99. Oates, S., Evans, G., Hedge, A., 1998. An anthropometric and postural risk assessment of childrens school computer work environments. Comput. Schools 14, 55e63. Portney, L., Watkins, M., 2008. Foundations of Clinical Practice: Applications to Practice, third ed. Prentice Hall, London. Rankin, G., Stokes, M., 1998. Reliability of assessment tools in rehabilitation: an illustration on appropriate statistical analyses. Clin. Rehabil. 12, 187e199. Shrout, P., Fleiss, J., 1979. Intraclass correlations: uses in assessing rater reliability. Psychol. Bull. 86, 420e428. Straker, L., Briggs, A., Grieg, A., 2002. The effect of individually adjusted workstations on upper quadrant posture and muscle activity in schoolchildren. Work 18, 239e248. Walter, S., Eliazew, M., Donner, A., 1998. Sample size and optimal designs for reliability studies. Stat. Med. 17, 101e110.