Development/2011

Dec 13, 2011

• Change time to 11am

• Derek's suggestion for "Writing VisTrails Packages"
  • Split old/new way of writing modules?
  • Do we still support having a package as a .py file?
  • Should we document adding custom configuration screens?

Dec 6, 2011

• Test suite
  • our current test suite is not comprehensive
  • add tests for the core that was separated from the GUI, e.g., calling update functions, ...
  • focus on accuracy and reliability of the provenance that is saved
  • will not create scripts for the actual GUI right now
  • Action items: iterate through the directories and add the missing tests--start from controller (should document functions as the tests are created) --- start on this after UVCDAT is done

• UVCDAT

Nov 29, 2011

• UVCDAT

Nov 8, 2011

• Command-line flag fixes:
  • detached history view done
  • check multiheads for spreadsheet---spreadsheet not shown on startup now

• Merge core_no_gui branch with master
  • Check unit tests here
  • Can we merge this into master, or keep as a separate branch?

• UV-CDAT
  • Call last Friday to talk about Variables
  • Dave looking at Variables
  • Tommy looking at workspace for UV-CDAT application
  • Emanuele managing integration and gui development
  • Keep track of things on new UV-CDAT Development page
  • Examples: Tom's examples
  • Avoid adding complications right now
  • Persistence for variables: simplfy and have a checkbox to allow system to save those things
  • Scenario from Jerry Potter? Juliana working on
  • Plot: graphics method, template, plot, variable (list)
  • Templates for subworkflows? Define all the subworkflows that must respect this interface
  • Dave: write a Variable module
    • Emanuele: send Dave CDAT python examples, one vt in paraview package

Oct 18, 2011

• We will focus on UV-CDAT and on completing the alpha version of the system. Please read the documents Emanuele emailed.

• Using different servers with CrowdLabs

• Closing tickets (Tommy)

Oct 11, 2011

• Bug reports:

• • execute_cmdline issue: CLTools supports both

• 2.0 bug fixes: push new release soon (for change_selected_version bug fix)

• Fix migrateTags issue and any other minor bugs, then release new version of 2.0

• CLTools:
  • latest is in repository
  • README in package directory
  • adding to online user's/developer's guide

Oct 4, 2011

• We will discuss issues related to remote execution of VisTrails modules as well as multi-threaded and parallel execution of workflows
• We will be joined by Matthias Troyer and Troels Frimodt Rønnow (ETH Zurich), Joel Daniels (NYU Poly), Marta Mattoso, Jonas Dias (UFRJ, Brazil)
• Terence Van Zyl and Derek Hohls (CSIR, South Africa) will not be able to make it today

We will develop a package that supports the ability to spawn a 'batch' job either on a local machine, a remote server, or a specialized cluster/supercomputer (with its own batch submission system). The basic API will provide functions to:

• start jobs and obtain id (which will be saved in VisTrails and in UFRJ's database)
• poll the status of a job (if it is still running of it is completed)
• copy files to and from the remote server

Once a job is submitted by the 'remote module', the workflow stops. The user can poll the status through the UFRJ DB, or by running the workflow again on VisTrails. When the workflow is re-run, VisTrails will check the status of the job; if the job is still running, the workflow stops; if the job is complete, the files (if any) will be transferred and the execution of the workflow resumes.

Next steps:

• Troels will work with Joel on extend Joel's package to implement the basic API; and with Dave on implementing the idea of 'persistent jobs'
• Once the basic API is ready, we will work with Jonas to integrate his DB interface and the ability to access the provenance information about the remote jobs from multiple devices

• Tommy gave an update on CLTools. He already has a GUI to create the wrappers. He will make some additional changes and we will get some people to try it out.

September 27, 2011

• Stéphane Descorps-Declere and Loic Couderc from Institut Pasteur will join us
  • Show demo of their package
  • Discuss their specific needs
  • How to deal with tools that have a large number of parameters and help the users not get lost setting these parameters, some ideas:
    • Port documentation in tooltips so mouse-over shows the documentation---need to support the specification of port-level documentation in the wrapper
    • Defining in the wrapper required and dependent ports. For the latter, define the dependency graph among the different parameters, and then use this information in the GUI. For example, given parameters A, B, C and dependencies C->B, B->A, and if A is defined, the GUI should require that B and C be set
    • Define in the wrapper port cardinality, i.e., the number of connections supported by the port
    • Support HTML-style documentation, which would allow the inclusion of images---the problem here is that it isn't good for command-line based use of VisTrails, so we also need to support plain text
  • Stephane and his group will take a stab at implementing these ideas in two steps: first, making the required changes to the registry; and then, updating the GUI to use the new information that will be available in the wrapper.

• Discuss CLTools features/design
  • Tommy has an initial function that parses man pages
  • He will develop a GUI to help users wrap command-line tools---this will be a tool "separate" from VisTrails
  • How to handle pre- and post-conditions? Could create scripts and wrap them using CLTools; or in VisTrails, use Python source
  • reuse code from existing PythonSource GUI? Maybe for pre/post-conditions, but the port list probably isn't a great interface for this.

• Issue with change_selected_version
  • the graph is probably being corrupted by some version not being added to the version tree, could be a single action type
  • Asked Tom if he knows steps to reproduce

September 20, 2011

• Tommy will give a demo of CLTools package for wrapping command line tools
  • Each tool creates a module, but modules cannot be versioned?
  • Do you have specific tools/scripts I can try?
  • Next step is to create more automatic wizards for tools and scripts.

September 13, 2011

• Making it easier to wrap command line tools, motivated by the need to simplify the creation of executable papers
  • Goal: build a wizard that guides a user on the creation of a wrapper.
  • Assumption: the user has code that derived results for paper, and needs to package it in a way that reviewers can both reproduce and validate the results (e.g., try different parameter values)
  • Initially, require the user to manually provide the information---parameter names and optionally, types.
  • The output of the wizard should be a machine-readable specification which could be loaded as a module in VisTrails
  • Some desirable functionality:
    • ability to 'test' the wrapper; after the specification is input, show a vismashup-like interface for the user to set values, and allow him/her to execute the module
  • provide 'smart' parsing to help in the wrapper construction, for example, extract the information from a python script

• We will create a separate branch for external developers, so that they can contribute patches to VisTrails

• David has cleaned up most of the GUI dependencies, the only remaining ones are:
  • Notes associated with versions (they need Qt)
  • Generation of PDF representation for workflows and trees
  • Thumbnails

• Web services (EMBOSS example)
  • All now works except Prophet example (<choice> bug in SUDS). Should we remove it?

• Invalid modules (those in red) will not show parameters - Fixed

September 7, 2011

• Query mode
  • need to finish this implementation

• Analogy parent() issue?
  • check what this was (form August 16)

• Web services (EMBOSS example)
  • changes with SUDS (should be easier)

• VTK Examples:
  • check old examples (don't seem to work with new version)
  • these were auto-generated from VTK python examples a while ago and may have issues with newer VTK versions

• Auto-updating tags: migrateTags option in preferences
  • Document this
  • MIgrate notes as well so that the notes and tags are migrated

August 31, 2011

• Diff tab has other view modes disabled.
  • Opening a version or mashup from workspace, open in new tab instead of overwriting diff

• Binaries
  • Matthias reported his tutorials were working with the current version
  • Emanuele will start rebuilding the binaries more frequently again

• Mashup saving
  • keep does not save
    • change the name keep to tag, to avoid confusion
  • changes in workflows and mashups are reflected in the workspace now
    • still working on the executions

• Provenance browser
  • instead of using version numbers for untagged versions, use the same convention used in pipeline view (eg., parent_tag + 1)

• Documentation
  • VTK?
  • At least a section in the user's guide that points the users to the right examples

• Examples
  • problem with examples that depend on 3rd-party packages
    • move them to the package folder?

August 16, 2011

• Diff tab should have other view modes disabled.
  • Tab doesn't make sense to switch back to history
  • Could either try to make mode buttons work or just disable the modes
  • Easier to disable the other modes
  • If open a version or mashup from workspace, open in new tab instead of overwriting diff
  • this way, a diff view is only closed by a user explicitly closing it

• Binaries for Mac
  • Can build 64-bit binary and not 32-bit
  • Emanuele will try to build 32-bit again
  • ALPS has separate 32-bit and 64-bit versions
  • Have two DLLs and registry for 2008 and 2010 Visual Studio
  • Problem is that VisTrails may not run if the registry does not have the updates from Visual Studio
  • 64-bit support allows more than 4GB of memory

• Should Mashups or Workflows be children if the vistrail in the workspace?
  • Expand them automatically
  • Mashups are meant to be shortcuts

• Mashup saving
  • keep does not save (workspace is not updated)
  • can we emit signal to workspace so it is updated
  • can we put things in memory for the workspace?
  • set_changed is called so we should update the workspace when a vistrail to be changed

• Subworkflows
  • new changes to infrastructure to address distributed version control
  • issue where multiple versions of the same subworkflow are listed
  • multiple edits still an issue

• Analogies
  • issue with parent() bug, Dave needs to file this

• Databases
  • no direct database access currently allowed at NYU
  • either create localhost database or use via crowdlabs

August 9, 2011

• Ticket #459
  • for an empty pipeline, the exploration is not resetting correctly

• Dave looking into query issues

• Issue with views not being set up correctly and handling callbacks (#469) amy also play a role in #473.

• Ticket #472. Setting values in the user-defined list or any other type of window
  • Going to any other window causes everything to reset, after executing, focus out event causes things to reset

• Subworkflows: edit bugs, Claurissa

• Groups inside of groups: currently cannot view them, Claurissa was looking at this

• Cmd-Q when the spreadsheet is on top seems to cause problems, need to force quit on the Mac.

• Enhancement: spreadsheet camera reset button?

• Cmd-E for configuring ports, Ticket #443 fix the old configuration windows or do away with, maybe keep this

• Parameter exploration: link spreadsheet to the parameter explorations, or display the parameters used with each cell, way to write

• Missing example for vtk.vt for cell locations and sheet references.

• Diff when upgraded doesn't work

• Upgrade entire version tree? How do we preserve tree structure?

• Mashups: display with pipelines? having both expanded to solve the extra click
  • add this as a preference, other ways to view, add a button

• Opening a lot of vistrail files, closing them without saving, have "Yes to All", "No to All" buttons.

August 2, 2011

• Ticket #459 (Tommy)
  • Pipeline inspector may not get updated correctly when pipeline is empty

July 26, 2011

• Zoom level in the first vistrail bug
  • file on trac
  • start vistrails and go to version view, history tree is not there, Cmd+R shows root node
  • all vistrails after work fine

• Subworkflows
  • issue with adding input ports and naming
  • separate issue with editing a subworkflow more than once

• Analogies
  • how to update the algorithm so the terminator with HTTPFile works correctly

• LaTeX Assistant
  • Dave's new changes (read tex file and find \vistrails tags) moved to 2.1, remove from 2.0 for now
  • Add back as a new feature for 2.1

• ungrabMouse in setModuleLabel
  • probably happens on any module

• query issue: want to get back to original query from the search results and modify the query, currently just removes the query and resets everything

• keep triangle menu initially in 2.0, may remove in later versions if we can maintain access to all features

July 19, 2011

• Features added
  • Display mashups in the workspace
  • Detach configuration window, rename, in a read-only, non-changing state (allow copy)
    • Improve button layout?
    • move read-only to left side and other buttons to be right side
• Status of 2.0
  • subworkflows bug, claurissa
• [Dave] Need to fIx bug with retagging

July 12 13, 2011

• Features added
  • Display mashups in the workspace
    • Tommy added the support for the workspace, items need to be added
  • Detach configuration window, rename, in a read-only, non-changing state (allow copy)
• Menu/Key shortcuts issues
  • QBaseView window with menubar, add parts of menu
  • QVistrailView inherits so we can reuse part of menus
  • Need to show menus on Windows and Linux for other windows
  • PythonSource has no menubar by default, can create its own menu
• Show Pipeline for a group
  • delegate the creation of this window to QVistrailView?
  • windows are set on show_group_function
  • don't create a separate window
  • eventually we should make groups logical not physical (so everything doesn't change each time to modify a group)
  • just capture new pipeline

• spreadsheet editing and analogies
  • add static methods to controller to allow general access to create methods
  • need to factor code so virtual_cell can work with this code, too

• look into the issue when seeing a workflow for the first time; might be linked to upgrades and indexing
  • Tommy should check that the exceptions are propograted

July 5, 2011

• Server migration is almost completed
  • GIT/Wiki/Trac working fine. Machine will be relocated to get better bandwidth
  • Still need to update vistrails.org to point to new location

• Some bugs from last week were fixed, still a few more to go (see trac tickets)
• Minor issues raised at the meeting
  • Add shortcut key for executing a pipeline
    • [ES] Ctrl+Return (Command+Return on a Mac) should work. The problem is that the shortcut does not show up on the Menu on a Mac. I've tested on QtDesigner and it doesn't show up there either. So it seems an issue with Macs.
  • Being able to execute the selected node in Version View (having to open a pipeline just to execute it involve too many steps)
• Features to be implemented, then VisTrails will be feature frozen
  • Display mashups in the workspace
  • Detach configuration window, rename, in a read-only, non-changing state (allow copy)

June 28, 2011

• svn offline this weekend to migrate servers (Huy)
  • send an email to vistrails-dev
• fix bugs from demo
  • fix zoom on open, change view
  • detach configuration window, rename, in a read-only, non-changing state (allow copy)
  • read-only controller for diff view, or use Emanuele's read-only view flag
  • check for pruned versions in "(latest)" version
  • failed to index error on open (possibly old version of vistrail--with filenames)

• releases?
  • waiting for ALPS build for Mac OS X 10.5
  • allow 2.0 vt files to open in 1.7 (e.g. those with mashups)

June 21, 2011

• Querying
  • need ability to better navigate results
  • show query results in side panel?
  • search all open vistrails (change label on the search)

• Updated workspace window
  • Separate open and recent files
  • Delete closes an open vistrail or removes a closed one from recent file list
  • Also has an "open in new open window" option for right-click, suggested "open in new window", "open in new tab" for workflows
  • Want to be able to show this palette at all times? This allows better navigation, but problem is the screen space
  • Try showing the workspace window in the upper left and "normal" left palettes in the lower-left.

• Module Palette
  • Searching should expand things automatically, especially for searches that have only a few search results
  • What about searching a particular package? Could auto-expand on begin search, user collapses and then expands the package they are intereseted in

• Global variables
  • one bug on disconnecting variables

• Spreadsheet and analogy issue on re-executing pipelines? Dave will look at

• Diff
  • Diff properties not showing up automatically, Emanuele will check
  • Implement a diff in workspace window so you can do cross-vistrail diff
  • Diff uses version tree for efficiency, but we can use analogy match or heuristic even when no actions are shared

• Inspector window: would be nice to keep this window smaller (certain tabs keep it large)

June 14, 2011

• Detach windows
  • notifications?
  • creating secondary windows
  • don't duplicate the entire vistrails window, just individual views
  • detach icon not displaying correctly in Windows

• Global Variables
  • How to add to modules?
  • Drag constant modules to the vistrails variables panel
  • Set the value by editing the constant in the vistrals variables panel
  • Connect the value by dragging the constant from the vistrails variables panel to a port in the pipeline
  • Update default text to "Drag constants from Modules panel to create variables"
  • Fix bugs with connecting globals and deleting them (versionTab issue, etc.)

• Query tab in vistrail
  • Can we use normal version/pipeline views as result views?
    • Which view to start with? history/pipeline
    • Show the workspace panel instead of history view
    • Show results in the query panel (as pipeline or workspace as appropriate)
  • Should we always combine "keyword" and "visual" queries?
    • try to combine queries when possible, but allow user to search just text if there is no pipeline or just pipeline if there is no text
    • get back to the search window after seeing results (need a button here)
  • What interactivity do we want? Open pipeline when matching version is double-clicked?
    • yes, single-click shows in results view, double-click opens as normal vistrail
  • Dave will look at query view

• Add read-only PythonSource

• Workspace: keep closed vistrails here too
  • try to order so that open vistrails are on top (highlighted in bold?)
  • also show different views here (full tree view)
  • show current workflow (ie latest version in the panel), too

June 7, 2011

• Status of 2.0

May 31, 2011

• Go over bugs and new features for 2.0

May 17, 2011

• Add comments, bugs, critiques, etc. about the 2.0 changes here

• Global variables
  • Check issue on MacOS
  • Add documentation

• Mashups [Emanuele]
  • on track, separate feedback
  • switching vistrails issue
  • execute button move --- move to right side (and make it possible to disable)
  • how does execute work
  • close builder window doesn't trigger quit?

• Provenance view [Tommy]
  • read all provenance right now
  • create an index file in the vt to load pieces of provenance on demand
  • enhancement idea: highlight errors in the list view of Log Details
  • tooltips in the browser view show large icons
  • search for provenance browser
  • fix issues with the left box
  • number of search boxes...

• Update splash screen [Juliana will talk to Nathan]

• Go over Jan's feedback on the PythonSource editor

A slight UI glitch is that I couldn't access the undo/redo functionality with the usual Cmd+[Shift+]Z shortcuts (neither via the menu or toolbars) but only via the right mouse button. One additional feature that might be nice in the future would be some minimal indentation assistance, like when pressing [Return] the current indentation is kept for the new line.

May 2, 2011

• Demos of new 2.0 functionality:
  • Mashup
    • log issue: break the link between vistrail.log and controller.log...
    • experimental version so we can play with it and see if there are issues, what improvements
  • Provenance browser
    • waiting to merge
  • PythonSource editor
    • compile for windows, check how it works on Mac
    • issue with multiple windows?
  • Global variables
    • expressions with variable, wait on this
    • set parameters on vistrail
    • click on ports to associate with globals

• Bug fixes
  • Web services
    • upgrades are working
    • Upgrades are in a separate package
    • hooks in vistrails to allow detection when a package is not loaded

Apr 26, 2011

• Updates:
  • Mashup
    • easier ways to generate package wrappers
    • working on merging code
  • Provenance browser
    • fixed thumbnails to show larger thumbnails
    • via tooltip?
    • keeping separate branch right now, detailed commit message on merge
  • Persistence Package
    • stand-alone browser for writing and deleting files
    • writing annotations for execution provenance
    • find where version is not being written as an execution annotation
  • bug: upgrade and Web services
    • last modified header not in all http transfers (e.g. for wsdl)
    • how to track versions of web services
    • one package per wsdl? see if this will work
    • seems to be an issue with network and that server (have to lookup twice)
  • Global variables
    • next week demo
  • new GUI

• • v2.0 branch
    • keep mashups off master
    • merging is pretty good using git
    • changing to a single-window interface, adding menus back

• Decide:
  • PythonSource editor
    • try Eric or some QScintilla-based approach

Apr 18, 2011

• 2.0 Release: new features to be included and bug fixes
  • Mashup
    • if already have alias, carry these over
    • have ability to choose GUI elements
    • storing provenance
    • allow users to import aliases, but don't write back to vistrail unless users wants to
    • provide user
    • create new palette for showing information about subworkflows and mashups (and tagged workflows) that shows on the left side [Dave]
  • Provenance browser
    • Index for provenance information
    • Showing this in a separate tab, linking to a existing workflow
  • Global variables
    • create a separate window
    • have underlying code for this
  • control-flow assistant
    • Dan send documentation to Juliana and Claurissa
  • Updated documentation
    • especially for executable paper
    • mashups
  • Feature 'executable publications' more prominently
  • FIX: upgrade and Web services
  • Python editor
    • indentation and coloring
    • modeless window

• Updates
  • Workflow search [Tommy]
    • can now save multiple objects to the database in two round trips (modulo number of groups)
    • create bigger thumbnails using tooltip in Ubuntu
    • view pipeline execution with groups
    • qbe. indexing high-level overview to Tommy
  • Global variables [Daniel]
  • Mashup integration [Emanuele]
  • Python source editor [Emanuele]
  • Web services update [Dan]
    • don't have upgrades
    • version numbers for web services modules, detect via some type of hashing
    • use last modified date + hash

Apr 12, 2011

• Workflow search [Tommy]
  • Thumbnails in search
  • Workflow search for individual workflows
  • Materialize workflows: user-controlled for now, button in search interface to materialize workflows
  • 2.0: log search for sure, possibly workflow/execution search
  • Show groups in the execution view, add navigation bar at top of view (like Illustrator group navigation)
  • Dave look at code for QBE & Indexing
  • Also try to work with data

• Groups/Subworkflows
  • need more documentation
  • better ways to view and navigate still needed

• Documentation [Claurissa]
  • Persistence
  • Could use ListToDirectory to take a list of files and convert to Directory (use for example)
  • Delete/modify persistent data
  • Persistent data provenance [Dave]

• Mashups in VisTrails [Emanuele, In Two Weeks]

• Interface changes [Dave]

• PythonSource updates [Emanuele]
  • DeVide, Eric4

• HTTPDirectory
  • create a manifest file that can be used for HTTP directory
  • check with Matthias for how this works, not supported by HTTP

Apr 1, 2011

• Meeting time change suggestion to 10AM (Tommy) -- sorry Tommy, only now I saw this item...
  • Next week: meet if need be
  • Move future meetings to 10am if room is possible (Juliana checking with Deb)
• Workflow Execution Explorer demo (Tommy)
  • search interface for workflow executions
  • thumbnails? problem is size
  • can open a result and view the execution provenance
  • querying vistrails
  • materialize tagged versions in the database (so we can query workflows directly)
  • extensions to allow query-by-example to work?
• 1.6.2 release (mainly with bug fixes) (Emanuele)
  • integrate changes to module configuration? on master, wait for 2.0 to put in release
  • investigate different options for Python editors
  • users guide? change to use sphinx for this release, add new features for 2.0
• VisMashup support in VisTrails discussion (Emanuele, Wendel)
• Wendel will give a demo of his work on supporting interactive visualizations in Web browsers; and of his simplified version of VisMashup
  • generate 3D view and parameter interaction just from aliases
  • suggested parameters: can associate parameters from workflows from other vistrails to labels, also order according to prevalence
  • data: allow user to select other datasets from crowdlabs
  • how to identify parameters that occur in different modules
  • identify aliases in the web interface
  • creating workflow view
  • how to store the provenance of such views
  • save dictionary of values from the mashup with the executions
  • workflow to mashup bridge
  • also allow a mashup mode to start from
  • assistant interface
• Executable Pubs
  • shell-escape not as well supported
  • change the latex extension to be a preprocessing step (and make a vt_pdflatex cmd or something like that)
  • Microsoft and MacOS X for publishing plugins--no Active X, known issue
  • add better annotations to pdfs after the fact so that users can annotate papers
  • meta-presenting
• Manual
  • packages: difference between wiki page and users guide
  • split users and developers guide, may have to do this for chapters like ControlFlow
  • keep list of questions, write emails, talk to people

Mar 22, 2011

• support a subset of VisMashup within VisTrails: re-use subworkflow infrastructure, and add the functions which create the application. Similar to subworkflows, mashups could live inside a vistrail
  • [DK] Note that mashups are higher-level objects than workflows. It would seem that mashups should include vistrails.
  • [JF] My idea is really simple: when we create a subworkflow we are essentially creating a 'view' that hides away the details of the underlying modules and only exposes a subset of the parameters. What I would like to have is the component of VisMashup that takes a subworkflow specification and generates the app. This would greatly simplify the publication of *interactive* results. And I believe, it would be simple to implement since it would re-use the existing infrastructure we have for subworkflows.

• support for global variables: maybe, an initial approach could be to have such variables associated with a vistrail. This would greatly simplify packaging of workflows, in particular, to deal with file locations in different systems
  • [DK] We have discussed this before as having workflow-global parameters and have proposed a method for accomplishing this---namely modify the interface to hide global parameters. It may be possible to have them for entire vistrails, but note that this poses problems. Suppose a user that creates a workflow identifying 'x' (a Float) as a global variable, and then changes the workflow and identifies a new parameter as 'x' but its type is now a String. A per-workflow definition of the global 'x' is fine, but a per-vistrail definition is not. I agree that we should try to support some style of alias-like behavior so that a user can set parameters more globally, but it is important that these settings are saved as provenance as many of these might be per-session...
  • [JF] I want global variables to simplify packaging of results. Having them on a per workflow basis would not solve the problem I had: In the WikiQuery case study I have a series of related workflows that all rely on the same set of input files; so in this case, I would need global variables at the VisTrails level.
  • [JF] As for the issue DK raised, I think we could have a very simple solution: we check for conflicts with global variables and prevent the used from making mistakes

• SubWorkflows in the database (Tommy)
  • abstractions==subworkflows? Some error messages still refer to abstractions
  • Logic for merging subworkflows from different sources? (Upgrading issues?)
• Control Flow Assistant feedback (Tommy)
• From Matthias:
  • Remote execution: make it easier to move executions to clusters, starting them from VisTrails, waiting for the results. Can we make Joel's code for the wildfire simulations more flexible?
  • Using VisTrails for validation: make it possible to take results of a workflow, and compare them with a previous result to validate that changes in code or VisTrails have not affected the results.
  • Make it easier to select a persistent repository (test, release, etc.) so that the same workflow can be run against different repositories
  • Better support for querying hundreds of data files (ie create a database of metadata that a user can use to filter the set of files to locate specific files quickly)
  • Along with this, allow this to work over existing (non-persistent) repositories.
  • A better python editor for PythonSource

Mar 15, 2011

• database to file: works for log now
• database: save subworkflows to the database
• add uuids for vistrails, subworkflows, merges
• merging: same uuid and data changed
• look into distributed algorithm
• merge should possibly include logs
• allow additional metadata to be controlled, possible to have a 1-1 correspondence

• speedup on database queries to speed up joins over entities, combine different attributes
• loops on module executions
• traversing module executions
• build transitive closure
• build table to link the workflow_exec with module_exec

• fix for users-list problem with Embed GUI issue
• how to handle users that want to contribute code or examples

• [Dan] committing control flow assistant soon, issue with database version (see trac commit from today)

Mar 8, 2011

• DB speedup finished? [Tommy]
  • Speedup done.
  • Looking into issues with saving log
  • Execution Provenance API

• Control flow [Dan]
  • Fixed issues
  • Input List Override working
  • Cartesian product now working
  • Error message with version tree
  • Weird issue with the group executing before the map starts

• Documentation [Claurissa]
  • check in guide to git

Mar 1, 2011

• Documentation Log

• Updating the relational DB (Tommy)
  • Import vistrail from DB is now very fast.
  • Mostly done with export vistrail to DB. Still a few issues.

Feb 22, 2011

• Python Source widget (Emanuele)

• News: Claurissa will help us with documentation. She will start by translating the existing user guide into Sphynx, and then move on to updating the content as well as documenting the many features that are currently not described in the manual.
  • Please add any undocumented features as well any (informal/incomplete) descriptions you might have to the following page: Undocumented Features

• • focus issue (will ask Huy if he knows a good solution for this)

• Browser updates (Tommy)
  • project browser, workspace

• Updating the relational DB (Tommy)
  • insert size: Tommy tested with 60,000 inserts
  • loading: multiple selects and retrieve them all at once
  • incremental updates, is_dirty, is_new
  • no need to bump the schema version
  • start thinking about better ways to do queries
  • can prepare statements via low level mysql commands

• Loop Assistant (Dan)
  • add tutorial for this loop

• mailing list emails
  • Dan: keep track of which emails and update the FAQ with answers
  • Could think about putting this into a system eventually, but for now it's low-traffic enough

Feb 8, 2011

• LaTex extension demo (Emanuele)

• • New embed dialog

• Non-modal PythonSource windows (link to discussion)

• • make it non-modal, single window
  • update selection on click

• Where to keep User's guide and other documentation

• • Sphinx, commit to doc directory

• Log Explorer demo (Tommy)

• Log_API RFC (Tommy)

• Control Flow Assistant (Daniel)

Feb 1, 2011

• Execution Provenance View (Tommy)

• Control flow assistant (Daniel)

• Script Interaction (Emanuele)
  • CDAT feature that might be useful in other settings

• Port/parameter management (Dave)
  • moved close button
  • what about NULL parameters?
  • global parameters

Jan 18, 2011

• VisTrails browser (Tommy)
  • demo
  • can import files and directories to the list of files
  • can see and open vistrails, workflows
  • need to think about how to show execution information

• Control flow assistant (Daniel)
  • integration with parameter exploration
  • can re-use infrastructure with parameter exploration
  • can keep provenance for parameter explorations for free if we use the looping constructs

• New port/parameter management (Dave)
  • demo
  • ordering?
  • ability to filter, sort parameters
  • documentation
  • required ports
  • update modules to show when parameters are set on ports
  • talk to Matthias

• 1.6.1 Release
  • tomorrow, bug fixes + executable papers
  • create tickets

Jan 11, 2011

• Release (Emanuele)
  • 1.6 released 1/7/2011
  • there is a branch on git
  • nightly builds
  • bundling ALPS python support into releases

• Control flow assistant (Dan)
  • focus on supporting the Map module

• VisTrails browser/repository (Tommy)
  • open a vistrail get included in an index
  • index includes vistrails, workflows, and executions
  • use thumbnails for showing the workflows
  • in the future, sort by projects

• Port and parameter updates (Dave)
  • Colin Talbert at USGS-Fort Collins requests the same global parameters feature that Matthias has already mentioned. We should start working on this functionality.
  • Colin also asked about highlighting required ports
  • Changes to schema: add required flag, cardinality flag, and global parameters
  • Changes to execution: way to add global parameters and connections to workflow before execution begins (not display visually)
  • Changes to interface: method to select ports and tie them to global parameters, global parameter display, highlight for ports to show required (red?), do we need to show cardinality?

• Executable paper tasks