Brain, Vol. 123, No. 9, 1964-1969,
September 2000
© 2000 Oxford University Press
Reproducibility of peer review in clinical neuroscience
Is agreement between reviewers any greater than would be expected by chance alone?
1 Department of Clinical Neurology, Radcliffe Infirmary, Oxford and 2 MRC Environmental Epidemiology Unit, Southampton General Hospital, Southampton, UK
Correspondence to:
Dr P. M. Rothwell, Department of Clinical Neurology, Radcliffe Infirmary, Woodstock Road, Oxford OX2 6HE, UK E-mail: peter.rothwell{at}clneuro.ox.ac.uk
We aimed to determine the reproducibility of assessments made by independent reviewers of papers submitted for publication to clinical neuroscience journals and abstracts submitted for presentation at clinical neuroscience conferences. We studied two journals in which manuscripts were routinely assessed by two reviewers, and two conferences in which abstracts were routinely scored by multiple reviewers. Agreement between the reviewers as to whether manuscripts should be accepted, revised or rejected was not significantly greater than that expected by chance [
= 0.08, 95% confidence interval (CI) –0.04 to –0.20] for 179 consecutive papers submitted to Journal A, and was poor ( = 0.28, 0.12 to 0.40) for 116 papers submitted to Journal B. However, editors were very much more likely to publish papers when both reviewers recommended acceptance than when they disagreed or recommended rejection (Journal A, odds ratio = 73, 95% CI = 27 to 200; Journal B, 51, 17 to 155). There was little or no agreement between the reviewers as to the priority (low, medium, or high) for publication (Journal A, = –0.12, 95% CI –0.30 to –0.11; Journal B, = 0.27, 0.01 to 0.53). Abstracts submitted for presentation at the conferences were given a score of 1 (poor) to 6 (excellent) by multiple independent reviewers. For each conference, analysis of variance of the scores given to abstracts revealed that differences between individual abstracts accounted for only 10–20% of the total variance of the scores. Thus, although recommendations made by reviewers have considerable influence on the fate of both papers submitted to journals and abstracts submitted to conferences, agreement between reviewers in clinical neuroscience was little greater than would be expected by chance alone.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  A. Casadevall and F. C. Fang Is Peer Review Censorship? Infect. Immun., April 1, 2009; 77(4): 1273 - 1274. [Full Text] [PDF]
|
|
|
|
 |
|
 B. Reider Kudos Am. J. Sports Med., December 1, 2008; 36(12): 2287 - 2288. [Full Text] [PDF]
|
|
|
|
 |
|
 I. T. Cohen and K. Patel Peer Review Interrater Concordance of Scientific Abstracts: A Study of Anesthesiology Subspecialty and Component Societies. Anesth. Analg., May 1, 2006; 102(5): 1501 - 1503. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. Wood, M. Roberts, and B. Howell The Reliability of Peer Reviews of Papers on Information Systems Journal of Information Science, February 1, 2004; 30(1): 2 - 11. [Abstract] [PDF]
|
|
|
|
 |
|
 W. H. Starbuck Turning Lemons into Lemonade: Where Is the Value in Peer Reviews? Journal of Management Inquiry, December 1, 2003; 12(4): 344 - 351. [Abstract] [PDF]
|
|
|
|
 |
|
 J.G. Ray Judging the judges: the role of journal editors QJM, December 1, 2002; 95(12): 769 - 774. [Full Text] [PDF]
|
|
|
|
 |
|
 R. Horton Postpublication Criticism and the Shaping of Clinical Knowledge JAMA, June 5, 2002; 287(21): 2843 - 2847. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 H. Marcovitch Atoms Arch. Dis. Child., February 1, 2002; 86(2): 71 - 71. [Full Text] [PDF]
|
|