SpringerLink
Log in

Mechanisms for provenance collection in scientific workflow systems

  • Published:
Computing Aims and scope Submit manuscript

Abstract

Scientific workflow management systems run scientific experiments. They manage sequences of complex process transformations and collect provenance information at various levels of abstraction. Collected provenance information from scientific experiments documents how experimental results are derived from input values along with experimental parameters and workflow configurations. Provenance greatly enhances usability and acceptance of workflow systems among scientists, because provenance allows workflow systems to capture process configuration and behaviour at different levels of detail. On this basis, a sufficient level of collected provenance information enables scientists to validate their hypotheses and make a workflow reproducible. Currently SWfMS’s do not use a standard or portable provenance model for either capturing, storing, querying or representing model. There are a variety of design issues in provenance models and mechanisms in workflow system, owing to the variation of design dimensions in workflow architectures. Given this variety, it seems desirable to classify provenance mechanisms in workflow systems. We aim to survey provenance collection mechanisms, that are either a part of scientific workflow system, or of a software infrastructure that supports collection mechanisms in a scientific workflow system. In this paper, firstly, we identify and define a set of design dimensions and conventions for provenance collection mechanisms in the context of working on scientific workflow systems. After this, we survey a set of scientific workflow projects based on our design dimensions with an emphasis on provenance collection mechanisms. Then, those conventions are used in order to evaluate a number of existing provenance collection mechanisms, presented at the end of this paper. This survey provides an understanding of primary design issues for provenance collection mechanisms along with a set of desirable design dimensions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Lin C, Lu S, Fei X, Chebotko A, Pai D, Lai Z, Fotouhi F, Hua J (2009) A reference architecture for scientific workflow management systems and the VIEW SOA solution. IEEE Trans Serv Comput 2(1):79–92. https://doi.org/10.1109/TSC.2009.4

    Article  Google Scholar 

  2. Ranno F, Shrivastava S (1999) A review of distributed workflow management systems. In: The international joint conference on Work activities coordination and collaboration (WACC99), San Francisco, California

  3. Wu Q, Zhu M, Gu Y, Brown P, Lu X, Lin W, Liu Y (2012) A distributed workflow management system with case study of real-life scientific applications on Grids. J Grid Comput 10(3):367–393. https://doi.org/10.1007/s10723-012-9222-7

    Article  Google Scholar 

  4. Miller JA, Sheth AP, Kochut KJ, Wang X (1996) CORBA-based run-time architectures for workflow management systems. J Database Manag (JDM) 7(1):16–27. https://doi.org/10.4018/jdm.1996010102

    Article  Google Scholar 

  5. Li H, Yang Y, Shi M (2003) Key issues and experiences in development of distributed workflow management systems. In: Zhou X, Orlowska M, Zhang Y (eds) Web technologies and applications, vol 2642. Lecture notes in computer science. Springer, Berlin, pp 507–512. https://doi.org/10.1007/3-540-36901-5_51

  6. Görlach K, Sonntag M, Karastoyanova D, Leymann F, Reiter M (2011) Conventional workflow technology for scientific simulation. In: Yang X, Wang L, Jie W (eds) Guide to e-science. Computer communications and networks. Springer, London, pp 323–352. https://doi.org/10.1007/978-0-85729-439-5_12

  7. Hahn C, Horn S, Jablonski S, Lay R, Neeb J, Schamburger R, Schlundt M Taxonomy of distribution concepts for workflow management. University Erlangen-Nürnberg

  8. Simmhan YL, Plale B, Gannon D (2005) A survey of data provenance techniques, vol 47405. Indiana University, Bloomington

    Google Scholar 

  9. Freire J, Koop D, Santos E, Silva CT (2008) Provenance for computational tasks: a survey. Comput Sci Eng 10(3):11–21. https://doi.org/10.1109/MCSE.2008.79

    Article  Google Scholar 

  10. Lee B, Awad A, Awad M (2015) Towards secure provenance in the cloud: a survey. In: 2015 IEEE/ACM 8th international conference on utility and cloud computing (UCC), 7–10 Dec 2015, pp 577-582. https://doi.org/10.1109/UCC.2015.102

  11. Tao L, Ling L, **aolong Z, Kai X, Chao Y (2014) ProvenanceLens: service provenance management in the cloud. In: 2014 international conference on collaborative computing: networking, applications and worksharing (CollaborateCom), 22–25 Oct 2014, pp 275-284

  12. Chen P, Plale BA (2015) Big data provenance analysis and visualization. In: 15th IEEE/ACM international symposium on cluster, cloud and grid computing (CCGrid), 4–7 May 2015, pp 797-800. https://doi.org/10.1109/CCGrid.2015.85

  13. Agrawal R, Imran A, Seay C, Walker J (2014) A layer based architecture for provenance in big data. In: IEEE international conference on big data (Big Data), 27–30 Oct. 2014, pp 1-7. https://doi.org/10.1109/BigData.2014.7004468

  14. Tan YS, Ko RKL, Holmes G (2013) Security and data accountability in distributed systems: a provenance survey. In: 2013 IEEE 10th international conference on high performance computing and communications & 2013 IEEE international conference on embedded and ubiquitous computing (HPCC_EUC), 13–15 Nov 2013, pp 1571–1578. https://doi.org/10.1109/HPCC.and.EUC.2013.221

  15. Moreau L, Kwasnikowska N, Van den Bussche J (2009) The foundations of the open provenance model. http://eprints.soton.ac.uk/id/eprint/267282

  16. Davidson SB, Boulakia SC, Eyal A, Ludäscher B, McPhillips TM, Bowers S, Anand MK, Freire J (2007) Provenance in scientific workflow systems. IEEE Data Eng Bull 30(4):44–50

    Google Scholar 

  17. Amsterdamer Y, Davidson SB, Deutch D, Milo T, Stoyanovich J, Tannen V (2011) Putting lipstick on pig: enabling database-style workflow provenance. Very Large Data Base (VLDB) Endow 5(4):346–357

    Google Scholar 

  18. Bowers S (2012) Scientific workflow, provenance, and data modeling challenges and approaches. J Data Sem 1(1):19–30. https://doi.org/10.1007/s13740-012-0004-y

    Article  Google Scholar 

  19. Stamatogiannakis M, Groth P, Bos H (2015) Looking inside the black-box: capturing data provenance using dynamic instrumentation. In: Ludäscher B, Plale B (eds) Provenance and annotation of data and processes, vol 8628. Lecture notes in computer science. Springer, Switzerland, pp 155–167. https://doi.org/10.1007/978-3-319-16462-5_12

  20. Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. EBSE Technical Report Ver. 2:3

  21. Keele S (2007) Guidelines for performing systematic literature reviews in software engineering. In: Technical report, Ver. 2.3 EBSE Technical Report. EBSE

  22. Daneva M, Damian D, Marchetto A, Pastor O (2014) Empirical research methodologies and studies in requirements engineering: how far did we come? J Syst Softw 95:1–9. https://doi.org/10.1016/j.jss.2014.06.035

    Article  Google Scholar 

  23. Kitchenham B, Pretorius R, Budgen D, Pearl Brereton O, Turner M, Niazi M, Linkman S (2010) Systematic literature reviews in software engineering: a tertiary study. Inf Softw Technol 52(8):792–805. https://doi.org/10.1016/j.infsof.2010.03.006

    Article  Google Scholar 

  24. Burnham JF (2006) Scopus database: a review. Biomed Digit Libr 3(1):1

    Article  Google Scholar 

  25. Zahedi M, Shahin M, Babar MA (2015) A systematic review of knowledge sharing challenges and practices in global software development. Int J Inf Manag (submiited to)

  26. Shahin M, Liang P, Babar MA (2014) A systematic review of software architecture visualization techniques. J Syst Softw 94:161–185. https://doi.org/10.1016/j.jss.2014.03.071

    Article  Google Scholar 

  27. Zhang H, Babar MA, Tell P (2011) Identifying relevant studies in software engineering. Inf Softw Technol 53(6):625–637. https://doi.org/10.1016/j.infsof.2010.12.010

    Article  Google Scholar 

  28. Chen L, Babar MA, Zhang H (2010) Towards an evidence-based understanding of electronic data sources. Paper presented at the. Proceedings of the 14th international conference on evaluation and assessment in software engineering, UK

  29. Scheidegger C, Koop D, Santos E, Vo H, Callahan S, Freire J, Silva C (2008) Tackling the provenance challenge one layer at a time. Concurr Comput Pract Exp 20(5):473–483. https://doi.org/10.1002/cpe.1237

    Article  Google Scholar 

  30. Moreau L, Clifford B, Freire J, Futrelle J, Gil Y, Groth P, Kwasnikowska N, Miles S, Missier P, Myers J, Plale B, Simmhan Y, Stephan E, den Bussche JV (2011) The open provenance model core specification (v1.1). Future Gener Comput Syst 27(6):743–756. https://doi.org/10.1016/J.Future.2010.07.005

    Article  Google Scholar 

  31. Groth P, Miles S, Missier P, Moreau L (2009) A proposal for handling collections in the open provenance model. http://mailman.ecs.soton.ac.uk/pipermail/provenance-challenge-ipaw-info/attachments/20090605/85b3e182/attachment-0001.pdf

  32. Groth P, Moreau L (2013) PROV-Overview. W3C. http://www.w3.org/TR/prov-overview/. Accessed 30 April 2013

  33. Garijo D, Gil Y (2012) The OPMW ontology. http://www.opmw.org/model/OPMW_20121009/

  34. Simmhan Y, Groth P, Moreau L (2011) Special section: the third provenance challenge on using the open provenance model for interoperability. Future Gener Comput Syst 27(6):737–742

    Article  Google Scholar 

  35. Crawl D, Wang J, Altintas I (2011) Provenance for MapReduce-based data-intensive workflows. In: 6th workshop on workflows in support of large-scale science, Seattle, Washington, ACM, pp 21–30

  36. Cruz SMS, Paulino CE, Oliveira Dd, Campos MLM, Mattoso M (2011) Capturing distributed provenance metadata from cloud-based scientific workflows. J Inf Data Manag 2(1):43–50

    Google Scholar 

  37. Muniswamy-Reddy K-K, Macko P, Seltzer MI (2010) Provenance for the cloud. In: the 8th USENIX conference on file and storage technologies, San Jose, California, USENIX Association, 1855526, pp 15–14

  38. Muniswamy-Reddy K-K, Macko P, Seltzer MI (2009) Making a cloud provenance-aware. In: Workshop on the theory and practice of provenance, San Francisco, California, USENIX Association

  39. Marinho A, Murta L, Werner C, Braganholo V, Cruz SMS, Ogasawara E, Mattoso M (2012) ProvManager: a provenance management system for scientific workflows. Concurr Comput Pract Exp 24(13):1513–1530. https://doi.org/10.1002/cpe.1870

    Article  Google Scholar 

  40. Chapman A, Blaustein BT, Seligman L, Allen MD (2011) PLUS: a provenance manager for integrated information. In: IEEE international conference on information reuse and integration (IRI), 3–5 August 2011, pp 269–275. https://doi.org/10.1109/IRI.2011.6009558

  41. Buchert T, Nussbaum L, Gustedt J (2015) Towards complete tracking of provenance in experimental distributed systems research. In: Hunold S, Costan A, Giménez D et al (eds) Euro-Par 2015: parallel processing workshops: Euro-Par 2015 international workshops, Vienna, Austria, 24–25 August 2015, Revised Selected Papers. Springer, Cham, pp 604-616. https://doi.org/10.1007/978-3-319-27308-2_49

  42. Carata L, Akoush S, Balakrishnan N, Bytheway T, Sohan R, Seltzer M, Hopper A (2014) A primer on provenance. Commun ACM 57(5):52–60. https://doi.org/10.1145/2596628

    Article  Google Scholar 

  43. Cruz SMS, Campos MLM, Mattoso M (2009) Towards a taxonomy of provenance in scientific workflow management systems. In: IEEE congress on services, Los Angeles, California, 6–10 July 2009. IEEE, pp 259–266. https://doi.org/10.1109/SERVICES-I.2009.18

  44. Glavic B, Dittrich KR (2007) Data provenance: a categorization of existing approaches. In: Conference on Datenbanksysteme in Buisness, Technologie und Web (BTW), Aachen, Germany, vol 12, pp 227–241

  45. Simmhan YL, Plale B, Gannon D (2005) A survey of data provenance in e-science. ACM Sigmod Rec 34(3):31–36. https://doi.org/10.1145/1084805.1084812

    Article  Google Scholar 

  46. Sarikhani M (2015) An adaptive provenance collection architecture in scientific workflow systems. Ph.D. Thesis, The University of Adelaide, Adelaide, Australia

  47. Lee EA, Parks TM (1995) Dataflow process networks. Proc IEEE 83(5):773–801. https://doi.org/10.1109/5.381846

    Article  Google Scholar 

  48. Brooks C, Lee EA, Liu X, Neuendorffer S, Zhao Y, Zheng H (2008) Heterogeneous concurrent modeling and design in Java (volume 3: Ptolemy ii domains). EECS Department, University of California, Berkley, California

  49. Muniswamy-Reddy K-K (2010) Foundations for provenance-aware systems. Harvard University, Cambridge

    Google Scholar 

  50. Anand MK (2010) Managing scientific workflow provenance. Univeristy of California Davis, Davis

    Google Scholar 

  51. Bowers S, McPhillips TM, Ludäscher B (2008) Provenance in collection-oriented scientific workflows. Concurr Comput Pract Exp 20(5):519–529. https://doi.org/10.1002/cpe.1226

    Article  Google Scholar 

  52. Moreau L, Freire J, Futrelle J, McGrath R, Myers J, Paulson P (2008) The open provenance model: an overview. In: Freire J, Koop D, Moreau L (eds) Provenance and annotation of data and processes, vol 5272. Lecture notes in computer science. Springer, Berlin, Germany, pp 323–326. https://doi.org/10.1007/978-3-540-89965-5_31

  53. Sonntag M, Karastoyanova D, Deelman E (2010) Bridging the gap between business and scientific workflows: humans in the loop of scientific workflows. In: Sixth international conference on e-science (e-science 2010), Brisbane, Queensland, Australia, pp 206–213. IEEE. https://doi.org/10.1109/eScience.2010.12

  54. Ludäscher B, Weske M, McPhillips T, Bowers S (2009) Scientific workflows: business as usual? In: Dayal U, Eder J, Koehler J, Reijers H (eds) Business process management, vol 5701. Lecture notes in computer science. Springer, Berlin, Germany, pp 31–47. https://doi.org/10.1007/978-3-642-03848-8_4

  55. Barga R, Gannon D (2007) Scientific versus business workflows. In: Taylor I, Deelman E, Gannon D, Shields M (eds) Workflows for e-science. Springer, London, pp 9–16. https://doi.org/10.1007/978-1-84628-757-2_2

  56. Andrews T, Curbera F, Dholakia H, Goland Y, Klein J, Leymann F, Liu K, Roller D, Smith D, Thatte S (2003) Business process execution language for web services. version

  57. Juric MB, Mathew B, Sarang PG (2006) Business process execution language for web services: an architect and developer’s guide to orchestrating web services using BPEL4WS. Packt Publishing Ltd, Birmingham

    Google Scholar 

  58. Altintas I, Barney O, Jaeger-Frank E (2006) Provenance collection support in the Kepler scientific workflow system. In: Moreau L, Foster I (eds) Provenance and annotation of data, vol 4145. Lecture notes in computer science. Springer, Berlin, Germany, pp 118–132. https://doi.org/10.1007/11890850_14

  59. Deelman E, Gannon D, Shields M, Taylor I (2009) Workflows and e-science: an overview of workflow system features and capabilities. Future Gener Comput Syst 25(5):528–540. https://doi.org/10.1016/j.future.2008.06.012

    Article  Google Scholar 

  60. Mattoso M, Werner C, Travassos GH, Braganholo V, Ogasawara E, Oliveira D, Cruz SMS, Martinho W, Murta L (2010) Towards supporting the life cycle of large scale scientific experiments. Int J Bus Process Integr Manag 5(1):79–92

    Article  Google Scholar 

  61. Stevens R, Zhao J, Goble C (2007) Using provenance to manage knowledge of in silico experiments. Briefings Bioinform 8(3):183–194. https://doi.org/10.1093/bib/bbm015

    Article  Google Scholar 

  62. Cruz SMS, Barros PM, Bisch PM, Campos MLM, Mattoso M (2008) Provenance services for distributed workflows. In: 8th IEEE international symposium on cluster computing and the grid (CCGRID), Lyon, France, 19–22 May 2008. IEEE, pp 526–533. https://doi.org/10.1109/CCGRID.2008.73

  63. Belhajjame K, Wolstencroft K, Corcho O, Oinn T, Tanoh F, William A, Goble C (2008) Metadata management in the Taverna workflow system. In: 8th IEEE international symposium on cluster computing and the grid, CCGRID’08. IEEE, pp 651–656

  64. Davidson SB, Freire J (2008) Provenance and scientific workflows: challenges and opportunities. In: ACM SIGMOD international conference on management of data, Vancouver, Canada. ACM, 1376772, pp 1345–1350. https://doi.org/10.1145/1376616.1376772

  65. Lim C, Lu S, Chebotko A, Fotouhi F (2010) Prospective and retrospective provenance collection in scientific workflow environments. In: International Conference on Services Computing (SCC), Miami, Florida. IEEE, pp 449–456. https://doi.org/10.1109/SCC.2010.18

  66. McPhillips T, Bowers S, Belhajjame K, Ludascher B (2015) Retrospective provenance without a runtime provenance recorder. Paper presented at the proceedings of the 7th USENIX conference on theory and practice of provenance, Edinburgh, Scotland

  67. Marinho A, Werner C, Cruz S, Mattoso M, Braganholo V, Murta L (2009) A strategy for provenance gathering in distributed scientific workflows. In: International conference on services computing (SCC), Bangalore, India. IEEE, pp 344–347. https://doi.org/10.1109/SERVICES-I.2009.53

  68. Groth PT (2005) On the record: provenance in large scale, open distributed systems. A mini-thesis for transfer from M.Phil. to Ph.D., University of Southampton, Southampton, England

  69. Spillane RP, Sears R, Yalamanchili C, Gaikwad S, Chinni M, Zadok E (2009) Story book: an efficient extensible provenance framework. In: Theory and practice of provenance (TaPP’09), San Francisco, California

  70. Vahdat A, Anderson TE (1998) Transparent result caching. In: USENIX annual technical conference, New Orleans, Louisiana

  71. Malik T, Gehani A, Tariq D, Zaffar F (2013) Sketching distributed data provenance. In: Liu Q, Bai Q, Giugni S, Williamson D, Taylor J (eds) Data provenance and data management in eScience. Studies in computational intelligence. Springer, Berlin, Germany, pp 85–107. https://doi.org/10.1007/978-3-642-29931-5_4

  72. Malik T, Nistor L, Gehani A (2010) Tracking and sketching distributed data provenance. In: Sixth international conference on e-science (e-Science 2010), Brisbane, Queensland, Australia. IEEE, pp 190–197. https://doi.org/10.1109/eScience.2010.51

  73. Gehani A, Tariq D (2012) SPADE: support for provenance auditing in distributed environments. In: Narasimhan P, Triantafillou P (eds) Middleware 2012, vol 7662. Lecture notes in computer science. Springer, Berlin, Germany, pp 101–120. https://doi.org/10.1007/978-3-642-35170-9_6

  74. Widom J (2005) Trio: a system for integrated management of data, accuracy, and lineage. In: Conference on innovative data systems research (CIDR), Asilomar, California

  75. Ikeda R, Widom J (2010) Panda: a system for provenance and data. IEEE Data Eng Bull 33(3):42–49

    Google Scholar 

  76. Foster IT, Vöckler J-S, Wilde M, Zhao Y (2003) The virtual data grid: a new model and architecture for data-intensive collaboration. In: Conference on innovative data systems research (CIDR), Asilomar, California, pp 18–29

  77. Cao B, Plale B, Subramanian G, Robertson E, Simmhan Y (2009) Provenance information model of karma version 3. In: International conference on services computing (SCC), Bangalore, India. IEEE, pp 348–351. https://doi.org/10.1109/SERVICES-I.2009.54

  78. Simmhan YL, Plale B, Gannon D (2008) Karma2: provenance management for data-driven workflows. Int J Web Serv Res 5(2):1–22

    Article  Google Scholar 

  79. Lanter DP (1990) Lineage in gis: The problem and a solution. In: National center for geographic information and analysis (NCGIA), Santa Barbara, California

  80. Hasan R, Sion R, Winslett M (2009) The case of the fake Picasso: preventing history forgery with secure provenance. Paper presented at the proccedings of the 7th conference on file and storage technologies, San Francisco, California

  81. Asghar MR, Ion M, Russello G, Crispo B (2012) Securing data provenance in the cloud. In: Camenisch J, Kesdogan D (eds) Open problems in network security: IFIP WG 11.4 international workshop, iNetSec 2011, Lucerne, Switzerland, June 9, 2011, Revised Selected Papers. Springer, Berlin, pp 145–160. https://doi.org/10.1007/978-3-642-27585-2_12

  82. Murta L, Braganholo V, Chirigati F, Koop D, Freire J (2015) noWorkflow: capturing and analyzing provenance of scripts. In: Ludäscher B, Plale B (eds) Provenance and annotation of data and processes: 5th international provenance and annotation workshop, IPAW 2014, Cologne, Germany, 9–13 June 2014. Revised selected papers. Springer International Publishing, Cham, pp 71–83. https://doi.org/10.1007/978-3-319-16462-5_6

  83. McPhillips T, Song T, Kolisnik T, Aulenbach S, Belhajjame K, Bocinsky K, Cao Y, Chirigati F, Dey S, Freire J (2015) YesWorkflow: a user-oriented, language-independent tool for recovering workflow information from scripts. ar**v preprint ar**v:1502.02403

  84. Reynolds P, Killian C, Wiener JL, Mogul JC, Shah MA, Vahdat A (2006) Pip: detecting the unexpected in distributed systems. Paper presented at the proceedings of the 3rd conference on networked systems design & implementation—volume 3, San Jose, CA

  85. Singh A, Maniatis P, Roscoe T, Druschel P (2006) Using queries for distributed monitoring and forensics. Paper presented at the Proceedings of the 1st ACM SIGOPS/EuroSys European conference on computer systems 2006, Leuven, Belgium

  86. Ruth P, Xu D, Bhargava B, Regnier F (2004) E-notebook middleware for accountability and reputation based trust in distributed data sharing communities. In: Jensen C, Poslad S, Dimitrakos T (eds) Trust management: second international conference, iTrust 2004, Oxford, UK, March 29–April 1, 2004. Proceedings. Springer, Berlin, pp 161–175. https://doi.org/10.1007/978-3-540-24747-0_13

  87. Marathe AP (2001) Tracing lineage of array data. J Intell Inf Syst 17(2–3):193–214. https://doi.org/10.1023/A:1012857830230

    Article  MATH  Google Scholar 

  88. Otero C (2012) Software engineering design: theory and practice, 1st edn. CRC Press, Boca Raton

    Google Scholar 

  89. Aktas MS, Plale B, Leake D, Mukhi NK (2013) Unmanaged workflows: their provenance and use. In: Liu Q, Bai Q, Giugni S, Williamson D, Taylor J (eds) Data provenance and data management in eScience. Studies in computational intelligence. Springer, Berlin, Germany, pp 59–81. https://doi.org/10.1007/978-3-642-29931-5_3

  90. De Nies T, Coppens S, Van Deursen D, Mannens E, Van de Walle R (2012) Automatic discovery of high-level provenance using semantic similarity. In: Groth P, Frew J (eds) Provenance and annotation of data and processes. Lecture notes in computer science. Springer, Berlin, pp 97–110. https://doi.org/10.1007/978-3-642-34222-6_8

  91. Magliacane S (2012) Reconstructing provenance. In: Cudré-Mauroux P, Heflin J, Sirin E et al (eds) The semantic web (ISWC), vol 7650. Lecture notes in computer science. Springer, Berlin, pp 399–406. https://doi.org/10.1007/978-3-642-35173-0_29

  92. Tariq D, Ali M, Gehani A (2012) Towards automated collection of application-level data provenance. In: Theory and practice of provenance (TaPP’12), Boston, Massachusetts. USENIX Association

  93. Web-Page (2013) getting started with kepler provenance 2.4. https://code.kepler-project.org/code/kepler/trunk/modules/provenance/docs/provenance.pdf

  94. Simmhan Y, Barga R, Van Ingen C, Lazowska E, Szalay A (2009) Building the trident scientific workflow workbench for data management in the cloud. In: 3rd international conference on advanced engineering computing and applications in sciences (ADVCOMP), pp 41–50. https://doi.org/10.1109/ADVCOMP.2009.14

  95. Barga R, Simmhan Y, Withana EC, Sahoo S, Jackson J, Araujo N (2010) Provenance for scientific workflows towards reproducible research. IEEE Data Eng Bull 33:50–59

    Google Scholar 

  96. Barga R, Jackson J, Araujo N, Guo D, Gautam N, Simmhan Y (2008) The trident scientific workflow workbench. In: IEEE fourth international conference on eScience (eScience ’08), Indianapolis, Indiana, pp 317–318. https://doi.org/10.1109/eScience.2008.126

  97. Freeman E, Robson E, Bates B, Sierra K (2004) Head first design patterns. O’Reilly Media, Inc., Sebastopol, CA, USA

  98. Forman IR, Forman N (2004) Java reflection in action. Manning Publications Co., Greenwich, CT, USA

  99. Maes P (1987) Concepts and experiments in computational reflection. In: Meyrowitz N (ed) Object-oriented programming systems, languages and applications (OOPSLA), Orlando, Florida, USA. ACM, 38821, pp 147–155. https://doi.org/10.1145/38765.38821

  100. Oliva A, Garcia IC, Buzato LE (1998) The reflective architecture of Guaraná. State University of Campinas, Sao Paulo

    Google Scholar 

  101. Corradi A, Lodolo E, Monti S, Pasini S (2009) Dynamic reconfiguration of middleware for ubiquitous computing. In: the 3rd international workshop on Adaptive and dependable mobile ubiquitous systems, London, UK. ACM, pp 7–12

  102. Smith BC (1984) Reflection and semantics in Lisp. In: The 11th ACM SIGACT-SIGPLAN symposium on principles of programming languages (POPL84), Salt Lake City, Utah, USA. ACM, 800513, pp 23–35. https://doi.org/10.1145/800017.800513

  103. Coulson G (2001) What is reflective middleware. IEEE Distrib Syst Online 2(8):165–169

    Google Scholar 

  104. Barbosa R, Pinho LM (2004) Monitoring of real time systems: a case for reflection. Polytechnic Institute of Porto, Porto

    Google Scholar 

  105. McKinley PK, Sadjadi SM, Kasten EP, Cheng BHC (2004) Composing adaptive software. Computer 37(7):56–64. https://doi.org/10.1109/MC.2004.48

    Article  Google Scholar 

  106. Aksit M, Choukair Z (2003) Dynamic, adaptive and reconfigurable systems overview and prospective vision. In: the 23rd international conference on distributed computing systems workshops, 19–22 May 2003. IEEE, pp 84–89. https://doi.org/10.1109/ICDCSW.2003.1203537

  107. Webb D, Wendelborn A (2003) The PAGIS grid application environment. In: Sloot PA, Abramson D, Bogdanov A, Gorbachev Y, Dongarra J, Zomaya A (eds) Computational science—ICCS 2003, vol 2659. Lecture notes in computer science. Springer, Berlin, pp 1113–1122. https://doi.org/10.1007/3-540-44863-2_110

  108. Lopes CV (2002) Aspect-oriented programming: an historical perspective (what’s in a name?). University of California, Irvine

    Google Scholar 

  109. Pawlak R, Seinturier L, Retaillé J-P, Younessi H (2005) Foundations of AOP for J2EE Development. Apress. https://doi.org/10.1007/978-1-4302-0063-5

    Google Scholar 

  110. Elrad T, Aksit M, Kiczales G, Lieberherr KJ, Ossher H (2001) Discussing aspects of AOP. Commun ACM 44(10):33–38

    Article  Google Scholar 

  111. Web-Page Provenance Aware Service Oriented Architecture (PASOA) (2014). http://twiki.pasoa.ecs.soton.ac.uk/bin/view/PASOA/WebHome. Accessed 10 July 2014

  112. Ding L, Michaelis J, McCusker J, McGuinness DL (2011) Linked provenance data: a semantic web-based approach to interoperable workflow traces. Future Gener Comput Syst 27(6):797–805. https://doi.org/10.1016/j.future.2010.011

    Article  Google Scholar 

  113. Moreau L, Ludäscher B, Altintas I, Barga RS, Bowers S, Callahan S, Chin G, Clifford B, Cohen S, Cohen-Boulakia S, Davidson S, Deelman E, Digiampietri L, Foster I, Freire J, Frew J, Futrelle J, Gibson T, Gil Y, Goble C, Golbeck J, Groth P, Holland DA, Jiang S, Kim J, Koop D, Krenek A, McPhillips T, Mehta G, Miles S, Metzger D, Munroe S, Myers J, Plale B, Podhorszki N, Ratnakar V, Santos E, Scheidegger C, Schuchardt K, Seltzer M, Simmhan YL, Silva C, Slaughter P, Stephan E, Stevens R, Turi D, Vo H, Wilde M, Zhao J, Zhao Y (2008) Special issue: the first provenance challenge. Concurr Comput Pract Exp 20(5):409–418. https://doi.org/10.1002/cpe.1233

    Article  Google Scholar 

  114. Web-Page (2010) The fourth and last provenance challenge. http://twiki.ipaw.info/bin/view/Challenge/FourthProvenanceChallenge

  115. Web-Page Provenance Challenge Wik. http://twiki.ipaw.info/bin/view/Challenge/WebHome. Accessed November 2008

  116. Garijo D, Gil Y (2012) The OPMW ontology. http://www.opmw.org/model/OPMW_20121009/

  117. Garijo D, Gil Y (2012) Towards open publication of reusable scientific workflows: abstractions, standards and linked data. http://www.isi.edu/~gil/papers/garijo-gil-opmw12.pdf

  118. Altintas I, Berkley C, Jaeger E, Jones M, Ludascher B, Mock S (2004) Kepler: an extensible system for design and execution of scientific workflows. In: 16th international conference on scientific and statistical database management, 21–23 June 2004. IEEE, pp 423–424. https://doi.org/10.1109/SSDM.2004.1311241

  119. Kim J, Deelman E, Gil Y, Mehta G, Ratnakar V (2008) Provenance trails in the wings/pegasus system. Concurr Comput Pract Exp 20(5):587–597. https://doi.org/10.1002/cpe.1228

    Article  Google Scholar 

  120. Web-Page Pegasus workflow management system. http://pegasus.isi.edu/. Accessed 10 March 2015

  121. Lin C, Lu S, Lai Z, Chebotko A, Fei X, Hua J, Fotouhi F (2008) Service-oriented architecture for VIEW: a visual scientific workflow management system. In: IEEE international conference on services computing (SCC’08), Honolulu, Hawaii. IEEE, pp 335–342

  122. Simmhan Y, Plale B, Gannon D, Marru S (2006) Performance evaluation of the karma provenance framework for scientific workflows. In: Moreau L, Foster I (eds) Provenance and annotation of data, vol 4145. Lecture notes in computer science. Springer, Berlin, pp 222–236. https://doi.org/10.1007/11890850_23

  123. Wolstencroft K, Haines R, Fellows D, Williams A, Withers D, Owen S, Soiland-Reyes S, Dunlop I, Nenadic A, Fisher P (2013) The Taverna workflow suite: designing and executing workflows of Web Services on the desktop, web or in the cloud. Nucl Acids Res 41(W1):W557–W561

    Article  Google Scholar 

  124. Oinn T, Greenwood M, Addis M, Alpdemir MN, Ferris J, Glover K, Goble C, Goderis A, Hull D, Marvin D (2006) Taverna: lessons in creating a workflow environment for the life sciences. Concurr Comput Pract Exp 18(10):1067–1100

    Article  Google Scholar 

  125. Marinho A, Murta L, Werner C, Braganholo V, Ogasawara E, Cruz SMS, Mattoso M (2010) Integrating provenance data from distributed workflow systems with ProvManager. In: Provenance and annotation of data and processes. Springer, pp 286–288

  126. Marinho A, Murta L, Werner C, Braganholo V, Cruz SMS, Ogasawara E, Mattoso M (2010) Managing provenance in scientific workflows with ProvManager. In: International workshop on challenges in e-Science (CIS2010), Petrópolis, Rio de Janeiro, Brazil, pp 17–24

  127. Olston C, Reed B, Srivastava U, Kumar R, Tomkins A (2008) Pig latin: a not-so-foreign language for data processing. Paper presented at the proceedings of the 2008 ACM SIGMOD international conference on management of data, Vancouver, Canada

  128. Green TJ, Karvounarakis G, Tannen V (2007) Provenance semirings. Paper presented at the proceedings of the twenty-sixth ACM SIGMOD-SIGACT-SIGART symposium on principles of database systems, Bei**g, China

  129. Dey S, Belhajjame K, Koop D, Raul M, Ludascher B (2015) Linking prospective and retrospective provenance in scripts. Paper presented at the proceedings of the 7th USENIX conference on theory and practice of provenance, Edinburgh, Scotland

  130. Kim J, Deelman E, Gil Y, Mehta G, Ratnakar V (2008) Provenance trails in the wings/pegasus system. Concurr Comput Pract Exper 20(5):587–597. https://doi.org/10.1002/cpe.1228

    Article  Google Scholar 

  131. Williams DN, Bremer T, Doutriaux C, Patchett J, Williams S, Shipman G, Miller R, Pugmire DR, Smith B, Steed C, Bethel EW, Childs H, Krishnan H, Prabhat P, Wehner M, Silva CT, Santos E, Koop D, Ellqvist T, Poco J, Geveci B, Chaudhary A, Bauer A, Pletzer A, Kindig D, Potter GL, Maxwell TP (2013) Ultrascale visualization of climate data. Computer 46(9):68–76. https://doi.org/10.1109/MC.2013.119

    Article  Google Scholar 

  132. Web-Page vistrails. http://www.vistrails.org/index.php/Main_Page. Accessed 20 March 2015

  133. Silva CT, Freire J, Callahan SP (2007) Provenance for visualizations: reproducibility and beyond. Comput Sci Eng 9(5):82–89. https://doi.org/10.1109/MCSE.2007.106

    Article  Google Scholar 

  134. Hey AJG, Tansley S, Tolle KM (2009) The fourth paradigm: data-intensive scientific discovery, 1st edn. Microsoft Research Redmond, Washangton

    Google Scholar 

  135. Delaney J, Heath G, Chave A, Howe B, Kirkham H (2000) NEPTUNE: real-time ocean and earth sciences at the scale of a tectonic plate. Oceanography 13(2):71–79

    Article  Google Scholar 

Download references

Author information

Author notes

  1. Andrew Wendelborn

    Present address: School of Computer Science, Level 4 Ingkarni Wardli building, University of Adelaide, Adelaide, SA, 5005, Australia

Authors and Affiliations

  1. Department of Computer Engineering and Information Technology, Safashahr Branch, Islamic Azad University, Safashahr, Iran

    Mehdi Sarikhani & Andrew Wendelborn

Authors
  1. Mehdi Sarikhani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrew Wendelborn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mehdi Sarikhani.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sarikhani, M., Wendelborn, A. Mechanisms for provenance collection in scientific workflow systems. Computing 100, 439–472 (2018). https://doi.org/10.1007/s00607-017-0578-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-017-0578-1

Keywords

Mathematics Subject Classification

Search

Navigation