Summary
The evaluation of medical expert systems, particularly those intended for decision support in the clinical domain, has not received sufficient emphasis. The techniques used for the evaluation of 14 medical systems are reviewed and contrasted with those used for the evaluation of new drugs. A three stage procedure is outlined. This is currently being used for the evaluation of ACORN, a decision support system to aid the admission of patients to cardiac care units.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Adams ID, Chan M, Clifford PC etalii, 1986: ‘Computer aided diagnosis of acute abdominal pain: a multicentre study’; BMJ vol. 293, pp. 800–804.
Aikins JS, Kunz JC, Shortliffe EH, 1983: ‘PUFF: an expert system for the interpretation of pulmonary function data’; Computers & Biomedical Res. vol. 16, pp. 199–208.
Aikins JS, 1983: ‘Prototypical knowledge for expert systems’; Artificial Intelligence vol. 20, pp. 163–210.
Alvey P, Greaves MF, 1986: ‘Observations on the development of a high performance system for leukaemia diagnosis’; in Proc. Expert Systems ’86 ( Brighton ), ed. Bramer M., pub. Cambridge University Press.
Armitage P, 1971: ‘Statistical methods in medical research’; pub. Blackwell Scientific, Oxford.
Catanzerite VA, Greenburg AG, Bremerman HJ, 1982: ‘Computer consultation in neurology: subjective and objective evaluations of the Neurologist system’; Comput. Biol. Med. vol. 12, pp. 342–355.
Diamond GA, Staniloff HM, Forester JS etalii, 1983: ‘Computer assisted diagnosis in the noninvasive evaluation of patients with suspected coronary artery disease’; J Am Coll Cardiol. vol. 1, pp. 444–455.
Ellam SV & Maisey MN, 1986: ‘A knowledge based system to assist in medical image interpretation: design and evaluation methodology’; in Proc. Expert Systems ’86 ( Brighton ), ed. Bramer M.,pub. CUP.
Fieschi M, Joubert M, Fieschi D etalii, 1983: ‘A program for expert diagnosis and therapeutic decision’; Med Inform, vol. 8, pp. 127–135.
Fox J, Barber D, Bardhan KD, 1980: ‘Alternatives to Bayes: a quantitative comparison with rule-based diagnostic inference’; Meth. Inform. Med. vol. 19, pp. 210–215.
Fox J, Myers CD, Greaves MF, Pegram S, 1985: ‘Knowledge acquisition for expert systems: experience in leukaemia diagnosis’; Meth. Inf. Med. vol. 24, pp. 65–72.
Fox J, Duncan TD, Frost D, Glowinski A, Hajnal S, O’Neill M, F, 1986: ‘Organising a large knowledge base: the Oxford System of Medicine’, paper presented at Expert Systems ’86 (Brighton).
Gaschnig J, Klahr P, Pople H, Shortliffe E, Terry A, 1983: ‘Evaluation of expert systems: issues and case studies’; in Hayes-Roth F, Waterman DA & Lenat D (eds), ‘Building expert systems’, Addison Wesley 1983.
Hudson DL, Cohen ME, Deedwania PC, 1984: ‘Emerge: a rule based expert system implemented on a microcomputer’; Microcomputer Applications vol. 3, pp. 79–83.
Mercer J, Talbot IC, 1985: ‘Clinical diagnosis: a post-mortem assessment of accuracy an the 1980s’; Postgrad. Med. J. vol. 61, pp. 713–716.
Miller PL, 1983: ‘Medical plan-analysis: the ATTENDING system’; proc. IJCAI-83, pp. 239–241.
Politakis P, Weiss SM, 1982: ‘A system for empirical experimentation with expert knowledge’; Proc. 15th Hawaii Int. Conf. on Systems Science, 2, pp. 649–657.
Schwartz WB, Patil RS, Szolovits P, 1987: ‘Artificial intelligence in medicine: where do we stand ?’, NEJMed vol. 316, pp. 685–688.
Shortliffe EH, Davis R, Axline SG etalii, 1975: ‘Computer based consultations in clinical therapeutics… the MYCIN system’; Computers & Biomed. Res. vol. 8, pp. 303–320
Shortliffe EH, Clancey WJ, 1984: ‘Anticipating the second decade’; in ‘Readings in medical AI’, Shortliffe & Clancey, pub. Addison Wesley 1984.
21.Spiegelhalter DJ 1983: ‘Evaluation of clinical decision aids, with an application to a system for dyspepsia’; Statistics in Med vol. 2, pp. 207–216.
22.Szolovits P, Long WJ, 1982: ‘The development of clinical expertise in the computer’; chap. 4 in ‘Artificial inteligence in medicine’, ed. Szolovits P, pub. Westview, for AAAS, Washington.
Wasson JH, Sox HC, Neff RK & Goldman L: 1985: ‘Clinical prediction rules: applications and methodological standards’; NEJMed vol. 313, pp. 793–799.
Weiss SM, Kulikowski CA, Galen RS, 1981: ‘Develo** microprocessor-based expert models for instrument interpretation’; in Proc. of seventh IJCAI, pp. 853–855, pub. Kaufmann Inc, Los Altos.
Weiss SM, Kulikowski CA, Amarel S, 1978: ‘A model-based method for computer-aided medical decision-making’; Artificial Intelligencevolil, pp. 145–172.
Wulff HR, 1976: ‘Rational diagnosis and treatment’; pub. Blackwell Scientific Publications, Oxford.
Wyatt JC, Emerson PA, Crichton NJ, 1986: ’Computer aided diagnosis of acute abdominal pain (letter); BMJ vol. 293, p. 1305.
Yu VL, Fagan LM, Wraith SM etalii, 1979: ‘Antimicrobial selection by computer: a blinded evaluation by infectious disease experts’; JAMA, vol. 242, pp. 1279–1282.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1987 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wyatt, J. (1987). The evaluation of clinical decision support systems: a discussion of the methodology used in the ACORN project. In: Fox, J., Fieschi, M., Engelbrecht, R. (eds) AIME 87. Lecture Notes in Medical Informatics, vol 33. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-95549-5_3
Download citation
DOI: https://doi.org/10.1007/978-3-642-95549-5_3
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-18402-7
Online ISBN: 978-3-642-95549-5
eBook Packages: Springer Book Archive