History Tree Evaluation

Project Description

Creating scientific visualizations is an exploratory process that involves managing large amount of data and workflows. Keeping track of the development of a visualization workflow is not common in practice. This project will focus on the VisTrails software, a new dataflow visualization system, being developed at the University of Utah for creating and maintaining large and complex visualizations. VisTrails maintains detailed provenance of the visualization workflow, tracking changes made by the user to the visualization during workflow development. This allows users to return to earlier versions of the workflow and also to compare different versions of a single visualization. The benefit to the user is simplifying the process of creating and maintaining visualization workflows.

For this project, we are interested in conducting a user performance study of the history management feature of VisTrails. Preliminary tests have shown that there is a user performance increase when using the history management feature of VisTrails for developing visualizations versus other methods or systems that do not involve a history component. One of the overall goals of the project is to study if VisTrails can be used in classrooms and collaborative environments to help improve performance.

Project Goal

The goal of this project is to quantitatively measure user performance in developing visualizations using the history management feature of VisTrails. The analysis will help to understand the effectiveness of VisTrails as a tool for developing and maintaining large sets of scientific visualizations as part of the data exploration process.

Plan of Work

To measure performance, study participants will be asked to perform a specific set of tasks of various difficulty that involve creating visualizations using VisTrails. For each task, the participant will be timed. In the end, the time information will be used as measurement for the user performance analysis.

Subjects Users will need to have some background in visualization, and more specifically have used VTK. Two groups will be identified for the study beginners and experts based on their level of experience in developing visualizations using VTK. Beginners would have taken the Scientific Visualization course taught at the University of Utah (or some equivalent course or related experience) but have very little additional experience, and experts who have more extensive knowledge and practice in developing visualizations.

Procedure

Develop tasks to be performed by participants of the study. Tasks will involve using VisTrails to either create a new or modify an existing visualization and the tasks will be divided according to difficulty level. The more difficult tasks will require longer times to complete. The tasks will be based on the homework projects assigned in the Scientific Visualization class. A subset or modified subset of those projects will be used to create tasks for the study.

For each task, determine the average time it takes to complete without using VisTrails. This will be done by performing the task using basic script writing tools such as Notepad. Time will be measured using a timer and will include all time spent in completing the task. The measured time will be an average and will serve as the baseline for comparing performance results. A minimum length of time will be set and all tasks developed for the study must not take less than the minimum time. If the task is too short then the performance advantage in using VisTrails will not be obvious. This will be preliminary work that will be completed by the Master’s student before running the user experiments.

For the user experiments, each user will be given a tutorial to introduce the software before moving on to performing the actual study tasks. Each user will be asked to complete a basic task to demonstrate competency. Users who require some type of assistance during the completion of the task will be trained further until they can successfully complete the task without assistance. At this point the user will be ready to carryout the study tasks. The tasks will be performed on the user’s personal machine or any machine that the user is comfortable with. There will be no lab setup for the study.

Evaluation

Completion of the project will be based on the completion of the user experiments and analysis. The following must be met:

• Identifying users for the study.
• Creating tasks for the study. At least 4 different tasks of different difficulty levels.
• Determining average time to complete each task without using VisTrails.
• Conducting a trial study using 2-3 subjects. Use results to make changes to the tasks if necessary.
• Carrying out additional experiments using larger number of subjects.
• Collecting time information from all the experiments and performing the necessary analysis to output performance results.
• Documenting the project (exact form will be determined later).