Provenance-Enabled Data Exploration and Visualization with VisTrails

Level

Elementary to advanced.

Presenters

Cláudio Silva, University of Utah
Juliana Freire, University of Utah
Emanuele Santos, University of Utah
Erik Anderson, University of Utah

Abstract

Scientists are now faced with an incredible volume of data to analyze. To explore and understand the data, they need to assemble complex workflows (pipelines) to manipulate the data and create insightful visual representations. Provenance is essential in this process. The provenance of a digital artifact contains information about the process and data used to derive the artifact. This information is essential for preserving the data, for determining the data's quality and authorship, for both reproducing and validating results -- all important elements of the scientific process. Provenance has shown to be particularly useful for enabling comparative visualization and data analysis. This tutorial will inform computational and visualization scientists, users and developers about different approaches to provenance and the trade-offs among them. Using the VisTrails project as a basis, we will cover different approaches to acquiring and reusing provenance, including techniques that attendees can use for provenance-enabling their own tools. The tutorial will also discuss uses of provenance that go beyond the ability to reproduce and share results.

Using the VisTrails system (http://www.vistrails.org) and real application scenarios, we will cover different approaches to capture and reuse provenance. We will show how the availability of provenance supports reflective reasoning as users perform exploratory tasks, including the ability to navigate through versions of analysis and visualization pipelines in an intuitive way, to undo changes but not lose any results, to visually compare different pipelines and their results, and to examine the actions that led to a result. We will also present a series operations and user interfaces that leverage provenance simplify pipeline design, use and sharing, as well as the publication of documented, reproducible results. Last, but not least, we will discuss techniques and present an API that attendees can use to provenance-enable their own tools.

Download VisTrails

Downloads for all platforms available here.

Working Outline

Introduction to Provenance and VisTrails (1h 30m)

Slides used at Vis2009: vis2009_tut_intro.pdf

Brief view of plugins and released features of VisTrails
Define provenance, explain why it is important for data analysis and visualization
Introduce VisTrails, explain the differences and similarities between VisTrails and other vis and scientific workflow systems
- Say it is open source
- Multi-platform: written in Python+QT
- Downloaded 1000's of times
Explain the version tree and the action-based provenance model
Show features that are enabled by the action-based provenance, and how they can help in the process of analyzing and visualizing data
- Going back and forth on the tree
- Automatic capture of changes
- Compare different workflows and their results
- Exploring parameter spaces
- Querying workflows by example
- Refining workflows by analogy
Discuss how the action-based provenance model can be combined with interactive tools
- Give a brief overview of the plugins
- Show videos for a couple of them and point to the detailed tutorial of how to build a plugin later.

Basic hands-on usage of VisTrails (1h 30m)

Slides used at Vis2009: vis2009_tut_basic_handson.pdf

Building Simple Pipelines (check spx.vt example included in the examples folder of VisTrails installation)
- VisTrails Builder overview
- Adding and connecting modules
- Setting parameters
- Setting module labels
- Accessing module documentation
- Using modules for displaying results in the spreadsheet
Using the Spreadsheet
- Spreadsheet overview
- Syncing cells
- Moving cells
- Saving camera
- Using Edit mode functionality
Using PythonSource: Co2maunaloa.vt
- Defining and using input ports and output ports
Connecting to databases
- Defining custom database connections
- Loading a vistrail from the database
- Saving a vistrail to the database
VisTrails as a Server
- Environment Setup
- Running VisTrails in Server Mode
- Showing Examples to illustrate the types of requests that can be done
  - Querying
  - Executing
  - Getting the result images
Custom widgets Tf_terminator.vt (for custom widgets)

Advanced hands-on usage of VisTrails (1h 30m)

This will all be in the context of a very specific example - We have recieved permission from Yvette Spitz to use the ROMS simulation for this. This allows us to give concrete examples that participants can follow along on their own as the tutorial progresses.

Slides: vis2009_tut_adv_handson.pdf

Organizational principles of the History Tree for Visualization creation and exploration
- Multiple, independent workflows contained in the same version tree. This method of organization makes creating more complex visualization easier.
QBE, Analogies, Parameter Exploration -> New Version
- First Query for visualizations supporting picking so that we can add this features by analogy to another version created from a parameter exploration over color map ranges. This will highlight data exploration and its interplay with the existing technologies inside of VisTrails
vtkInteractionHandlers - Slicing and dicing data
- We will investigate, in detail, the specific requirements for adding advanced interaction capabilities to a VTK-based visualization. Specifically, we will create workflows in which we can perform arbitrary culling as well as slicing along all 3 principle axes.
PythonSource as callback definition - Picking specific elements
- Callbacks allow many systems to fire unique events that can drive interactivity with visualizations. Here we will explore the details of implementing VTK-style picking in the ROMS data to change the scalar field being displayed in visualization.
The workflow debugger
- The workflow debugger allows stateful inspection of Module input/outputs. These advanced features can be used to provide interactive debugging and exploration of individual Module processes and interim data products.
Transforming VTK pipelines into VisTrails workflows --- modified version of Dave's script to get all VTK example into a .vt, and show how to query/search that. As a summary, this will be shown and posted allowing people to search for a single VTK module to see how it must be connected and parameterized in order to be properly used.

Developing a plug-in (30-40m)

Slides: vis2009_tut_plugins.pdf

Plug-in architecture overview
Different implementation strategies
- Attaching the host plugin to the main application
  - Callback mechanism
  - Undo/Redo Stack
- Communication between Host Plugin and Capture API
  - callbacks
  - Sockets
ParaView Plug-in Implementation
- Explaining required changes to the source
- Showing the limitations
- Demo

New Extensions to VisTrails (50-60m)

Display Wall
Provenance Rich Publications
Mashups
Repository
Web-based execution (?)

Related Papers

Managing Rapidly-Evolving Scientific Workflows (by Juliana Freire, Claudio T. Silva, Steven P. Callahan, Emanuele Santos, Carlos E. Scheidegger and Huy T. Vo) Invited paper, in the proceedings of the International Provenance and Annotation Workshop (IPAW), 2006.

Using Provenance to Streamline Data Exploration through Visualization (by Steven P. Callahan, Juliana Freire, Emanuele Santos, Carlos E. Scheidegger, Claudio T. Silva and Huy T. Vo) SCI Institute Technical Report, No. UUSCI-2006-016, University of Utah, 2006.

Querying and Creating Visualizations by Analogy (by Carlos E. Scheidegger, Huy T. Vo, David Koop, Juliana Freire and Claudio T. Silva). IEEE Trans. Vis. Comp. Graph (Proceedings of IEEE Vis 2007) 13(6):1560-1567, 2007. IEEE Vis 2007.

Software Infrastructure for Exploratory Visualization and Data Analysis: Past, Present, and Future (by C.Silva and J. Freire) In Journal of Physics: Conference Series, SciDAC 2008 Conference, 2008.

VisComplete: Automating Suggestions for Visualization Pipelines (by David Koop, Carlos E. Scheidegger, Steven P. Callahan, Huy T. Vo, Juliana Freire and Claudio T. Silva). In IEEE Transactions on Visualization and Computer Graphics, 14(6), pp. 1691-1698, 2008.

VisMashup: Streamlining the Creation of Custom Visualization Applications (by Emanuele Santos, Lauro Lins, James Ahrens, Juliana Freire and Claudio Silva). IEEE Trans. Vis. Comp. Graph (Proceedings of IEEE Vis 2009), 15(6), pp. 1539-1546, 2009.

Tutorials/Sibgrapi2010

Contents