Provenance challenge
Second provenance challenge design overview
This page describes the implementation of how to answer the queries of the second provenance challenge.
The goal of this project is to create an api capable of querying different kinds of databases containing provenance data. The main focus will be on provenance generated by scientific workflows.
data model overview
This is a description of the data model that i am trying to implement.
Module definition is a description of a processor that takes inputs and generates outputs.
Workflow definition is a description of a workflow that contains modules and connections between them through ports. In the case of VisTrails, it also contains the evolution of the workflow through a parent relation.
Execution log is the information about a workflow execution. It contains information about the processors that were executed and the data items that were created.
primitives
The api will deal with the basic primitives describing workflow executions.
node types:
| name | description |
|---|---|
| dataitem | a dataitem that is input/output to a module execution |
| module | the module/service that is to be executed |
| moduleInstance | the module as represented in a workflow |
| moduleExecution | the execution of a module |
| workflow | a description of a process containing modules and connections |
| workflowExecution | the representation of a workflow execution |
| inputPort | represents a specific port thas can be assigned an input to a module execution |
| outputPort | represents a specific port thas can contain a product of a module execution |
| connection | represents a connection between module Instances |
Relations
The basic relations between the primitives is shown in the diagram above. However few of the current provenance systems contains all of this information. Some may lack information about produced data items or miss the definition of a workflow. The goal must thus be to extract as much information as possible from each source.
The most common use of provenance data will be to perform the transitive closure of some connected executions or data items i.e. to track data dependencies back and forward in time. We call this upstream for tracking back in time and downstream for tracking forward in time.
We have identified 4 primitives to which upstream/downstream tracking is relevant:
| primitive | description |
|---|---|
| dataitem | tracking data dependencies |
| moduleExecution | tracking execution dependencies |
| moduleInstances | tracking module dependencies within a workflow |
| workflow | tracking workflow design history e.g. different workflow versions in the VisTrails action tree |
transitive relations: datatype relation
upstreams:
dataitem derivedFrom - .outputOf()[forall].hasInput() moduleInstance prevModuleInstance - .hasInputPort()[forall].connectedTo().outputPortOf() moduleExecution prevModuleExecution - .hasInput()[forall].OutputOf()
downstreams:
dataitem derivedData - .inputOf()[forall].hasOutput() moduleInstance nextModuleInstance - .hasOutputPort()[forall].connectedTo().inputPortOf() moduleExecution nextModuleExecution - .hasOutput()[forall].inputOf()
--Tommy 03:40, 13 April 2007 (MDT)--