Abstract
This study addressed a need to more effectively assess progress in knowledge development for complex practice domains. The result of identifying and implementing such assessment methodologies may be useful in guiding the development of instructional interventions and support mechanisms to help novices progress more efficiently and effectively toward expertise. The first challenge was to determine whether a proposed annotated concept map** methodology could be used to identify patterns among experts in their problem conceptualizations of ill-structured problems. The second was to identify if there were differences between experts and novice problem conceptualizations. Highly experienced physicians (experts) and less experienced medical students (novices) were engaged in creating annotated problem representations for given medical diagnosis problems. The data generated were used to determine whether (1) patterns in medical diagnosis problem conceptualizations were identifiable for experts, (2) differences existed between expert and novice conceptualizations of the medical diagnosis problems, and (3) a strong argument for using annotated concept maps to assess learning in complex domains could be supported. Findings suggested that there were similarities in how experienced physicians thought about the representative ill-structured problems and these similarities were different than novices’ conceptions. These findings support the notion that this methodology is useful in generating data that can aid in distinguishing relative levels of expertise in conceptualization of ill-structured problems in a medical diagnosis contexts. Discussions of the implications of this line of research and for the further development of this methodology are provided.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Allison, D. J., Morfitt, G., & Demaerschalk, D. (1996). Cognitive complexity and expertise: Relationships between external and internal measures of cognitive complexity and abstraction, and responses to a case problem. (ERIC Document Reproduction Service No. ED 412604).
Anderson, J. R. (1982). Acquisition of cognitive skill. Psychological Review, 89, 369–406.
Andrews, G., & Halford, G. S. (2002). A cognitive complexity metric applied to cognitive development. Cognitive Psychology, 45(2), 153–219.
Baker, E. L., & Schacter, J. (1996). Expert benchmarks for student academic performance: The case for gifted children. Gifted Child Quarterly, 40, 61–65.
Chi, M. T. H., & Glaser, R. (1985). Problem solving ability. In R. J. Sternberg (Ed.), Human abilities: An information processing approach (pp. 227–257). San Francisco: W. H. Freeman & Co.
Coderre, S., Mandin, H., Harasym, P. H., & Fick, G. H. (2003). Diagnostic reasoning and diagnostic success. Medical Education, 37(8), 695–703.
Cunningham, A. C., & Stewart, L. M. (2002). Systems analysis of learning theory through causal influence diagrams. (ERIC Document Reproduction Service No. ED 475928).
Dabbagh, N. H., Jonassen, D. H., Yueh, H. P., & Samouilova, M. (2000). Assessing a problem-based learning approach to an introductory instructional design course: A case study. Performance Improvement Quarterly, 13(3), 60–83.
Dörner, D. (1987). On the difficulties people have in dealing with complexity. In J. Rasmussen, K. Duncker, & J. Leplat (Eds.), New technology and human error (pp. 97–109). Chichester, NY: John Wiley & Sons, Inc.
Dörner, D. (1996). The logic of failure: Why things go wrong and what we can do to make them right (R. Kimber & R. Kimber, Trans.). New York: Metropolitan Books (Original work published in 1989).
Epstein, R. (2007). Assessment in medical education. The New England Journal of Medicine, 356(4) 387–396.
Ericsson, K. A. (2001). Expertise in interpreting: An expert-performance perspective. Interpreting, 5(2), 187–220.
Ericsson, K. A., & Simon. H. A. (1993). Protocol analysis: Verbal reports as data (Rev. ed.). Cambridge, MA: MIT Press.
Ericsson, K. A., & Smith, J. (1991). Prospects and limits in the empirical study of expertise: An introduction. In K. A. Ericsson & J. Smith (Eds.), Toward a general theory of expertise: Prospects and limits (pp. 1–38). Cambridge: Cambridge University Press.
Frensch, P. A., & Funke, J. (1995). Definitions, traditions, and a general framework for understanding complex problem solving. In P. A. Frensch & J. Funke (Eds.), Complex problem solving: The European perspective (pp. 3–26). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Funke, J. (2001). Dynamic systems as tools for analysing human judgment. Thinking and Reasoning, 7, 69–89.
Funke, J., & Frensch, P. A. (2007). Complex problem solving: The European perspective In D. H. Jonassen (Ed.), Learning to solve complex scientific problems (pp. 25–47). New York: Lawrence Erlbaum.
Gagné, R. M. (1985). The conditions of learning and theory of instruction (4th ed.). New York: Holt, Rinehart, and Winston, Inc.
Ganesan, R., & Spector, J. M. (2000). Causal influence diagrams as pedagogical and assessment tools. Presentation at CILT 2000, Tyson’s Corner, VA, 28 October 2000.
Gogus, A., Koszalka, T. A., & Spector, J. M. (2009). Assessing conceptual representations of ill- structured problems. Technology Instruction, Cognition and Learning (TICL), 7(1), 1–20.
Goldsmith, T. E., Johnson, P. J., & Acton, W. H. (1991). Assessing structural knowledge. Journal of Educational Psychology, 83(1), 88–96.
Grotzer, T. A., & Perkins, D. N. (2000). A taxonomy of causal models: The conceptual leaps between models and students’ reflections on them. Paper presented at the National Association of Research in Science Teaching (NARST), New Orleans, LA.
Herl, H. E., O’Neil, H. F., Jr., Chung, G. L., & Schacter, J. (1999). Reliability and validity of a computer-based knowledge map** system to measure content understanding. Computers in Human Behavior, 15(3–4), 315–333.
Huber, O. (1995). Complex problem solving as multi stage decision making. In P. A. Frensch & J. Funke (Eds.), Complex problem solving: The European Perspective (pp. 151–173). Hillsdale, NJ: Lawrence Erlbaum Associates.
Ifenthaler, D., & Seel, N. M. (2005). The measurement of change: Learning-dependent progression of mental models. Technology, Instruction, Cognition, and Learning, 2(4), 317–336.
Jacobson, M. J. (2000). Problem solving about complex systems: Difference between experts and novices. In B. Fishman & S. O’Connor-Divelbiss (Eds.), Fourth International Conference of the learning sciences (pp. 14–21). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
Jonassen, D. H. (1997). Instructional design models for well-structured and ill-defined problem-solving learning outcomes. Educational Technology: Research and Development, 45(1), 65–95.
Jonassen, D. H. (2000). Toward a design theory of problem solving. Educational Technology Research & Development, 48(4), 63–85.
Kitchner, K. S. (1983). Cognition, metacognition, and epistemic cognition: A three-level model of cognitive processing. Human Development, 26, 222–232.
Klein, G. A. (1998). Sources of power: How people make decisions. Cambridge, MA: MIT Press.
Liu, X., & Hichey, M. (1996). The internal consistency of a concept map** scoring scheme and its effect on prediction validity. International Journal of Science Education, 18(8), 921–937.
Markham, K. M., Mintzes, J. J., & Jones, M. G. (1994). The concept map as a research and evaluation tool: Further evidence of validity. Journal of Research in Science Teaching, 31(1) 91–101.
Mayer, R. E., & Wittrock, M. C. (1996). Problem-solving transfer. In D. C. Berlinert & R. C. Calfee (Eds.), Handbook of educational psychology (pp. 47–62). New York: Macmillan.
McClure, J., Sonak, B., & Suen, H. (1999). Concept map assessment of classroom learning: Reliability, validity, and logistical practicality. Journal of Research in Science Teaching, 36, 475–492.
Norman, G. R. (1994). Cognitive differences in clinical reasoning. Teaching and Learning in Medicine, 6, 114–120.
Pirnay-Dummer, P. (2007). Model inspection trace of concepts and relations. A heuristic approach to language-oriented model assessment. Paper presentation at the Annual Meeting of the American Education Research Association, April, 2007, Chicago, IL.
Pirnay-Dummer, P., Ifenthaler, D., & Spector, J. M. (2009). Highly integrated model assessment technology and tools. Educational Technology Research and Development, doi: 10.1007/s11423-009-9119-8.
Ruiz-Primo, M. A., & Shavelson, R. J. (1997). Concept-map based assessment: On possible sources of sampling variability. (ERIC Document Reproduction Service No. ED422 403).
Seel, N. M. (2003). Model centered learning and instruction. Technology, Instruction, Cognition and Learning, 1(1), 59–85.
Shute, V. J., Jeong, A. C., Spector, J. M., Seel, N. M., & Johnson, T. E. (2009). Model based methods for assessment, learning and instruction. In M. Orey (Ed.), Educational media and technology yearbook 2009. Westport, CT: Greenwood Publishing.
Spector, J. M., Christensen, D. L., Siotine, A. V., & McCormack, D. (2001). Models and simulations for learning in complex domains: Using causal loop diagrams for assessment and evaluation. Computers in Human Behavior, 17(5–6), 517–545.
Spector, J. M., Dennen, V. P., & Koszalka, T. A. (2005). Causal maps, mental models and assessing acquisition of expertise. Technology, Instruction, Cognition and Learning, 3(1–2).
Spector, J. M., & Koszalka, T. A. (2004). The DEEP methodology for assessing learning in complex domains (Technical Report No. NSF-03-542). Syracuse, NY: Syracuse University, Instructional Design Development, and Evaluation (IDD&E).
Spiro, R. J. (1987). Knowledge acquisition for application: Cognitive flexibility and transfer in complex content domains. (ERIC Document Reproduction Service No. ED 287155).
Sterman, J. D. (1994). Learning in and about complex systems. System Dynamics Review, 10(2–3), 291–330.
Sternberg, R. J. (1985). Beyond IQ: A triarchic theory of human intelligence. New York: Cambridge University Press.
Sutherland, D. C. (2002). Improving medical diagnoses by understanding how they are made. Internal Medicine Journal, 32(5–6), 277–280.
Suzuki, K., & Harnisch, D. L. (1995). Measuring cognitive complexity: An analysis of performance-based assessment in mathematics. Paper presented at the 1995 Annual Meeting of the American Educational Research Association, San Francisco, April 18–22. (ERIC Document Reproduction Service No. ED 390924)
Taricani, E. M., & Clariana, R. B. (2006). A technique for automatically scoring open ended concept maps. Educational Technology, Research, & Development, 54(1), 61–78.
Voss, J. F., & Post, T. A. (1988). On the solving of ill-defined problems. In M. T. H. Chi, R. Glaser & M. J. Farr (Eds.), The nature of expertise (pp. 261–285). Hillsdale, NJ: Lawrence Erlbaum Associates.
Wiener, S. L. (1996). Clinical reasoning in the health professions. Annals of Internal Medicine, 124(5), 537.
Wood, P. K. (1983). Inquiring systems and problem structures: Implications for cognitive development. Human Development, 26, 249–265.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer Science+Business Media, LLC
About this chapter
Cite this chapter
Koszalka, T.A., Epling, J. (2010). A Methodology for Assessing Elicitation of Knowledge in Complex Domains: Identifying Conceptual Representations of Ill-Structured Problems in Medical Diagnosis. In: Ifenthaler, D., Pirnay-Dummer, P., Seel, N. (eds) Computer-Based Diagnostics and Systematic Analysis of Knowledge. Springer, Boston, MA. https://doi.org/10.1007/978-1-4419-5662-0_16
Download citation
DOI: https://doi.org/10.1007/978-1-4419-5662-0_16
Published:
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4419-5661-3
Online ISBN: 978-1-4419-5662-0
eBook Packages: Humanities, Social Sciences and LawEducation (R0)