Abstract
As the scale and complexity of ontologies increases, so too do errors and engineering challenges. It is frequently unclear, however, to what degree extralogical ontology errors negatively affect the application that the ontology underpins. For example, “Shoe SubClassOf Foot” may be correct logically, but not in a human interpretation. Indeed, such errors, not caught by reasoning, are likely to be domain-specific, and thus identifying salient ontology errors requires consideration of the domain. There are both automated and manual methods that provide ontology quality assurance. Nevertheless, these methods do not readily scale as ontology size increases, and do not necessarily identify the most salient extralogical errors. Recently, crowdsourcing has enabled solutions to complex problems that computers alone cannot solve. For instance, human workers can quickly and more accurately identify objects in images at scale. Crowdsourcing presents an opportunity to develop methods for ontology quality assurance that overcome the current limitations of scalability and applicability. In this work, I aim (1) to determine the effect of extralogical ontology errors in an example domain, (2) to develop a scalable framework for crowdsourcing ontology verification that overcomes current ontology Q/A method limitations, and (3) to apply this framework to ontologies in use. I will then evaluate the method itself and also its effect in the context of a specific domain. As an example domain, I will use biomedicine, which applies many large-scale ontologies. Thus, this work will enable scalable quality assurance for extralogical errors in biomedical ontologies.
Terminology
-
Error: Extralogical ontology error (i.e., non-logical error than can only be detected by human interpretation)
-
Application: A system, method, or application that uses an ontology (e.g., decision support system)
-
Salient error: An error that negatively affects an application
-
Verification: The process of finding errors
Chapter PDF
Similar content being viewed by others
Keywords
- Biomedical Ontology
- Response Aggregation
- Ontology Engineering
- Human Interpretation
- Amazon Mechanical Turk
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Blake, J.A., Bult, C.J.: Beyond the data deluge: data integration and bio-ontologies. J. Biomed. Inform. 39(3), 314–320 (2006)
Blumenthal, D., Tavenner, M.: The “meaningful use” regulation for electronic health records. New England Journal of Medicine 363(6), 501–504 (2010)
Bodenreider, O., Stevens, R.: Bio-ontologies: current trends and future directions. Briefings in Bioinformatics 7(3), 256–274 (2006)
Ceusters, W., Smith, B., Goldberg, L.: A terminological and ontological analysis of the NCI Thesaurus. Methods of Information in Medicine 44(4), 498 (2005)
Demartini, G., Difallah, D.E., Cudré-Mauroux, P.: ZenCrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In: 21st World Wide Web Conference WWW 2012, Lyon, France, pp. 469–478 (2012)
Gangemi, A., Catenacci, C., Ciaramita, M., Lehmann, J.: Modelling ontology evaluation and validation. In: Sure, Y., Domingue, J. (eds.) ESWC 2006. LNCS, vol. 4011, pp. 140–154. Springer, Heidelberg (2006)
Geller, J., Perl, Y., Halper, M., Cornet, R.: Special issue on auditing of terminologies. J. Biomed. Inform. 42(3), 407–411 (2009)
Gruber, T.R.: Toward Principles for the Design of Ontologies Used for Knowledge Sharing. Tech. Rep. 5–6, Knowledge Systems Laboratory, Stanford University (1993)
Guarino, N., Welty, C.: Evaluating Ontological Decisions with OntoClean. Communications of the ACM 45(2), 61–65 (2002)
Horridge, M., Parsia, B., Sattler, U.: Laconic and precise justifications in OWL. In: Sheth, A.P., Staab, S., Dean, M., Paolucci, M., Maynard, D., Finin, T., Thirunarayan, K. (eds.) ISWC 2008. LNCS, vol. 5318, pp. 323–338. Springer, Heidelberg (2008)
Howe, J.: Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business. Crown Business (2009)
Ipeirotis, P.G., Provost, F., Wang, J.: Quality management on amazon mechanical turk. In: Proc. of the ACM SIGKDD Workshop on Human Computation, pp. 64–67. ACM (2010)
Kalyanpur, A., Parsia, B., Horridge, M., Sirin, E.: Finding all justifications of OWL DL entailments. In: Aberer, K., et al. (eds.) ISWC/ASWC 2007. LNCS, vol. 4825, pp. 267–280. Springer, Heidelberg (2007)
Kalyanpur, A., Parsia, B., Sirin, E., Cuenca-Grau, B.: Repairing unsatisfiable concepts in OWL ontologies. In: Sure, Y., Domingue, J. (eds.) ESWC 2006. LNCS, vol. 4011, pp. 170–184. Springer, Heidelberg (2006)
Kalyanpur, A., Parsia, B., Sirin, E., Hendler, J.: Debugging unsatisfiable classes in owl ontologies. Web Semantics: Science, Services and Agents on the World Wide Web 3(4), 268–293 (2005)
Lependu, P., et al.: Pharmacovigilance using clinical notes. Clin. Pharmacol. Ther. 93(6), 547–555 (2013)
Lozano-Tello, A., Gómez-Pérez, A.: Ontometric: A method to choose the appropriate ontology. Journal of Database Management 2(15), 1–18 (2004)
McGuinness, D.L., Borgida, A., Alex, D.D., Borgida, E.: Explaining reasoning in description logics. Tech. rep (1996)
Mortensen, J.M., Alexander, P.R., Musen, M.A., Noy, N.F.: Crowdsourcing Ontology Verification. In: International Conference on Biomedical Ontologies (accepted, 2013)
Mortensen, J.M., Musen, M.A., Noy, N.F.: Crowdsourcing the Verification of Relationships in Biomedical Ontologies. In: AMIA Annual Symposium (submitted, 2013)
Musen, M.A., Noy, N.F., Shah, N.H., Whetzel, P.L., Chute, C.G., Storey, M.A., Smith, B., Team, T.N.: The National Center for Biomedical Ontology. JAMIA 19, 190–195 (2012)
Noy, N.F., et al.: Mechanical Turk as an Ontology Engineer? Using Microtasks as a Component of an Ontology Engineering Workflow. In: Web Science (2013)
Parsia, B., Sirin, E., Kalyanpur, A.: Debugging OWL ontologies. In: Proceedings of the 14th International Conference on World Wide Web, pp. 633–640. ACM (2005)
Quinn, A.J., Bederson, B.B.: Human computation: a survey and taxonomy of a growing field. In: Annual Conference on Human Factors in Computing Systems (CHI 2011), Vancouver, BC, pp. 1403–1412. ACM (2011)
Rector, A.L., Brandt, S., Schneider, T.: Getting the foot out of the pelvis: modeling problems affecting use of SNOMED CT hierarchies in practical applications. Journal of the American Medical Informatics Association 18(4), 432–440 (2011)
Ruvolo, P., Whitehill, J., Movellan, J.: Exploiting structure in crowdsourcing tasks via latent factor models. Tech. rep (2010)
Sarasua, C., Simperl, E., Noy, N.F.: CrowdMAP: Crowdsourcing Ontology Alignment with Microtasks. In: Cudré-Mauroux, P., et al. (eds.) ISWC 2012, Part I. LNCS, vol. 7649, pp. 525–541. Springer, Heidelberg (2012)
Schlobach, S.: Non-standard reasoning services for the debugging of description logic terminologies, pp. 355–362. Morgan Kaufmann (2003)
Schlobach, S., Huang, Z., Cornet, R., Van Harmelen, F.: Debugging incoherent terminologies. Journal of Automated Reasoning 39(3), 317–349 (2007)
Siorpaes, K., Hepp, M.: Games with a Purpose for the Semantic Web. IEEE Intelligent Systems 23(3), 50–60 (2008)
Surowiecki, J.: The wisdom of crowds. Anchor (2005)
Tartir, S., Arpinar, I.B., Moore, M., Sheth, A.P., Aleman-meza, B.: OntoQA: Metric-based ontology quality analysis. In: IEEE Workshop on Knowledge Acquisition from Distributed, Autonomous, Semantically Heterogeneous Data and Knowledge Sources, vol. 9 (2005)
Völker, J., Vrandečić, D., Sure, Y., Hotho, A.: AEON–An approach to the automatic evaluation of ontologies. Applied Ontology 3(1), 41–62 (2008)
Von Ahn, L., Dabbish, L.: Designing games with a purpose. Communications of the ACM 51(8), 58–67 (2008)
Yu, J., Thom, J.A., Tam, A.: Requirements-oriented methodology for evaluating ontologies. Information Systems 34(8), 766–791 (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Mortensen, J.M. (2013). Crowdsourcing Ontology Verification. In: Alani, H., et al. The Semantic Web – ISWC 2013. ISWC 2013. Lecture Notes in Computer Science, vol 8219. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-41338-4_30
Download citation
DOI: https://doi.org/10.1007/978-3-642-41338-4_30
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-41337-7
Online ISBN: 978-3-642-41338-4
eBook Packages: Computer ScienceComputer Science (R0)