Development

Weekly Meetings

Nov 16, 2010

• Data and workflow browser (Nivan)

• Upgrade issues -- a wrap? (Daniel)

• Assistant for control package (Daniel)

• Web services on users' guide (Tommy)

• Improve error handling/reporting (Tommy)

• Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)

• Data Publishing with crowdLabs. (Phillip)
  • Do we want data publishing capabilities in VisTrails and crowdLabs? YES!!!
  • If so...
    • Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.
    • Persistent Data works for experimental data. Can we use it for publishing data?
      • Publishing data or storing data on the server and pushing back and forth?
      • What is the current scheme for identifying data on the server?
      • Do we need to do better than the ids, versions, hashes that the peristence package provides?
    • Do we want to utilize data publishing software like The Dataverse Network or maybe Fedora Commons?
  • [ES] Feature request: Can RepoSync support a directory of files?
    • [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.

I dont know if DisplayWall package would be a topic for VisTrails meeting, but.. anyway.

• DisplayWall (Wendel)
  • What examples should be in the SVN? If we decide to keep some examples in the svn, some of them use data that should be along.
  • DisplayWall Client on the Apple Store? I dont like the idea of sharing the client source code. And putting the app without login/password will allow anyone to access our displaywall.

Nov 9, 2010

• FAQ
  • As we reply to users queries, let's add the question and answer to the FAQ!
  • Just a reminder

• Caching of File module
  • There was a message from a user who got confused because this module is cached by default (and silently). And in his application, since the file actually changed in between runs, he did not see the 'expected' result.
  • Should we keep a hash for files and check whether they changed?
  • Should we no cache File by default?
  • [DK] File is supposed to have a special signature computation that detects changes in the file contents so this may be a bug instead. It does work, it seems to be an issue with the create_file
  • [ES] In this particular case, the file was empty. He deleted the file outside VisTrails and executed it again. As he set create_file to True, he was expecting the file to be created again.
  • [DK] Add random salt to hash signature when the file doesn't exist and create_file is True
  • [JF: DONE] Juliana will edit the FAQ to note that the signatures are dependent on modification times

• Claurissa will demo different ways to visualize the version tree
  • weighting on various criteria for displaying nodes (importance from session, user, tags, etc.)
  • other modes for viewing the versions (lists or timelines)
  • often see only linear trees, teaching or thinking other than tree?

• Upgrade issues (Dan)
  • Vanilla upgrades should be in today's commit
  • Added code so when doing upgrade to check whether the latest code is upgraded
  • Two types of upgrades: (1) User changes the subworkflow, (2) Modules are out-of-date and need to be upgraded.
  • Add higher-level features like "Upgrade All" for (1), "Import Changes to My Subworkflows", etc.
  • Want a 1-1 mapping between unique subworkflow id and a subworkflow file (even if we copy the same subworkflow to different vistrails)

• Offscreen bug (update from Huy)
  • Mac issue when switching to Cocoa
  • Huy added a QWidget for each of the render windows that is created, so should be fixed.

• Gesture support
  • Someone on Windows check the pinch gesture support; it doesn't work for Matthias under Parallels
  • VTKCell is working now
  • Cell dependent, panning now maps to middle button, pinch on the base widget
  • What are Matthias's requirements here? Does touch gestures in the cells matter or only in the vt/wf views

• Thumbnails: [JF: this needs to go to trac, for 2.0?]
  • Resolution? -- [ES] problem is the size of the vistrail, but this can be configured
  • Can we export to PDF instead of PNG? [ES] cannot display the thumbnail as a PDF in Qt?
  • Can we have the option to save higher-res or PDF versions of the thumbnails
  • Can we have preferences that allow users to save a higher-res or PDF version, maybe a checkbox
  • Maybe associate a high-res version with a spreadsheet cell so that we can save a high-res version on demand
  • high-res version associated with a version but allow user to initiate action from spreadsheet
  • can we have thumbnails dependent on camera position [ES] doesn't like, user won't see exactly that image upon re-executing
  • compact vistrail option by removing thumbnails

• Tommy has rewritten the Web services module as well as updated all of the examples
  • Server for one of the examples seems to be down
  • Need to update the manual

• Improve error handling/reporting (Tommy)
  • Maybe display the intialization messages with the splash screen?
  • Debugging levels have no way to access debug level
  • Tommy takes first cut at trying to determine how to map print statements, notes those he is unsure of
  • Developer guidelines for debug usage, use them instead of print statements in the futur

• Merge functionality (Tommy) [JF: Cool!]
  • Can now merge an existing vistrail into your own vistrail using menu item

• Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)
  • [PM] I haven't had a chance to implement anything yet.

• Data Publishing with crowdLabs. (Phillip)
  • Do we want data publishing capabilities in VisTrails and crowdLabs? YES!!!
  • If so...
    • Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.
    • Persistent Data works for experimental data. Can we use it for publishing data?
      • Publishing data or storing data on the server and pushing back and forth?
      • What is the current scheme for identifying data on the server?
      • Do we need to do better than the ids, versions, hashes that the peristence package provides?
    • Do we want to utilize data publishing software like The Dataverse Network or maybe Fedora Commons?
  • [ES] Feature request: Can RepoSync support a directory of files?
    • [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.

Nov 3, 2010

• Upgrading subworkflows (Dan)
  • add updated versions to subworkflow vistrail and push to registry
  • fixed bug with version/descriptor redirects
  • key remaining task is to replace the box representing the subworkflow in the top-level workflow to reflect the change to the upgraded version of the subworkflow
  • many corner cases, but we should get the vanilla upgrade (working) to git
  • need to ensure that each subworkflow file has its own uuid; if a subworkflow exists in two different vistrails, it should have a different uuid (namespace).
  • otherwise, we can get crosstalk where one vt can update the subworkflow from another vt (we can propgate changes to other vts via merges, but this should be a user choice...)
  • enhancements:
    • allow users to merge subworkflows from a file into their own subworkflow (to incorporate outside changes)
    • latest version is the most recent non-upgraded version; if that version has an upgrade, use the upgrade
    • Next step: work on usability for controlflow package; explore Dave's idea of an assistant

• Offscreen bug
  • Huy is looking into this

• Crash due to issues in loaded user packages (Tommy has fixed this)

• Web services (Tommy)
  • Package is completed, and all but one of our examples are working
  • Tommy will test if the Web services will work when there is a proxy

• Improve error handling/reporting (Tommy)
  • we should have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages.
  • the goal is to implement a function that will be the single point for error messages, and that work without emitting GUI signals in core. As Huy suggested, we should have a GUI-wrapper which will be a no-op when the GUI is not instantiated---this will get rid of some of the pyqt dependencies (at least for the error messages).

• Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)
  • Currently the VisTrails servers consists of: a single-threaded instance with GUI; multiple GUI-less threads
  • We will extend the server API to allow clients to check the server status. The server will provide a separate socket and the client will be able to 'ping' the server; if the server is working properly, it will respond; if it does not respond, the client will have a time out
  • The API will also support the ability to kill both the GUI-less threads and the single-threaded GUI instance. We can then restart the server components using Emanuele's new script.
  • We should also allow users to set a per-workflow time-out; if the workflow execution takes longer than the pre-defined threshold, VisTrails will abort the execution

• Update on fixing VTK Package (Wendel)

• • It seems that the ParaView package has the same issues.
  • Wendel has already checked in the new wrapper into the trunk; Emanuele will test it

• Synchronizing VisTrails and ALPS releases (Emanuele)
  • VisTrails latex package was extended so that python is no longer required. If python is not present in the system, instead of issuing a request to execute the workflow and retrieve the image from the Web, a local, previously saved image will be used. It is now also possible to embed the images of workflows into the latex file. Documentation about these and other features are currently in the README file and examples (example.tex) provided with the latex package.
  • Windows and Mac beta binaries already include ALPS
  • Are we distributing ALPS as part of VisTrails then? This seems backwards. ALPS should VisTrails as part of their stuff, no?
    • We will have separate binary distributions: one with and one without ALPS

• Is a multi-touch interface to VisTrails possible now that it's supported by Qt? (Phillip)
  • Yes, Qt 4.6+ supports multi-tocuh --- Huy will look into this; the goal is to make this work both for the workflow builder/tree view and the spreadsheet
  • We still need to figure out what makes sense for multi-touch, however, do we have requirements from ETH-Zurich?
    • We want to map the zoom, pan, click, etc to multi-touch, so that VisTrails can be intractable on a multi-touch screen without a mouse

• What is the status of the Vismashup i{Phone|Pad|Touch} app? (Phillip)
  • If you mean mac binary, there's an alpha version here: http://www.sci.utah.edu/~emanuele/files/vismashup
  • Do you mean iphone app? [Phillip: yes]
  • Wendel will look into this
    • One of the issues is how to effectively handle images that are larger than the memory on the iPods and iPads---we need to process these on the server or use a model that allows the image to be manipulated on the client
    • We also need to connect the app with the crowdlabs server, so that the app can get the list of mashups
    • Phillip will investigate the feasibility of working directly with JavaScript and bypass flash

Oct 19, 2010

• Update on Trac and Roadmap (Emanuele and David)
  • Trac is now linked to git repository
  • Roadmap on Trac has been cleaned up so we can hide completed milestones
  • Tickets are being reassigned/revisited. Many tickets (24) are not associated with a milestone yet https://vistrails.sci.utah.edu/report/3
• Synchronizing VisTrails and ALPS releases
• Web services support (Tommy)
  • discuss the interaction between the Web Services package and upgrades
• Subworkflows update (Daniel)
• Testing of new module drawing (Erik)
• Aliases
  • Currently, an alias is stored on a parameter. We need aliases to be stored at a higher level so that changing an alias is not a change to a parameter. The one issue is that there is a link between parmaeters and aliases in that an alias can only exist for versions that have the specified parameter. We might also have two versions where the alias points to different parameters. We could just store aliases as a root-level workflow element so that the set of aliases is versioned corrected.
  • In the current implementation, the aliases parsed from the parameters are stored in a dictionary in the workflow, so storing aliases as a root-level workflows element might be the way to go --Emanuele 19:50, 15 October 2010 (MDT)
• Error logging
  • Suggest that we have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages. This should improve our error handling significantly
• PyQt dependencies in core
  • Can we get rid of PyQt dependencies in core (and db)? This would probably require moving the signals/slots that we currently use to a similar Python implementation which shouldn't be too difficult. This would also require the specification of configuration widgets not as classes but rather some text that can be used to import the GUI elements only when using the GUI code. However, what would happen with workflows that run in command-line mode but require some graphical output?
• Ports
  • One annoying part of the current VisTrails model is that users are not given any visual indication that a port value is already set by a function or vice versa by a connection. In addition, we have no way to specify or enforce cardinality on ports at design time. It seems like we could allow developers to specify when a port should not be connected to more than one value (via a function or a connection), and give some visual feedback when a port has already been specified in one manner.

Meeting Notes

• Bug regarding userpackages at initial vistrails startup causing startup to fail.
  • Still an issue, ticket still open as a 1.6 milestone
• Copy-paste bug
  • Shortcut not working on Mac. Focus makes shortcut try to copy pipeline.
  • Current fix appends to clipboard as users type - this is not a good fix.
• Synchronizing ALPS and VisTrails releases
  • 1.6 scheduled for Dec. 1
• Webservices
  • New library being used, but it's very low level.
  • Simple types may cause problems with some web-services using complex or XML-based types.
  • Need to make sure that this library can handle at least MOST of the web-services out there.
• Sub-workflows
  • Daniel's changes seem functional and very slick.
  • A little more work on it seems necessary.
  • Need some easy GUI mechanism to delete a subworkflow.
  • Is the exclamation point in the module draw the best way to handle this? Make a tool-tip to explain it.
• VTK Wrapping
  • Changes to VTK Python wrapping is causing headaches - particularly in backwards compatibility.
  • Method typing has changed to expect things like "List" or "Vector" - makes things hard on the user when defining these inputs.
  • Removing List and Vector types and replacing them with dynamically generated versions from Tuples.
• Aliases - Agenda pushed until after Vis.
• Error Logging - Unify some error handling to improve how exceptions are dealt with. More on this after Vis.
• Refactoring out PyQt dependencies from Core/Db/etc - Need to move all the signals/slots into GUI elements. Questions come when a workflow uses GUI elements as inputs. - More after vis.
• Ports - We need a way of showing how a port is specified - function vs. input port. Cardinality must be established in these cases. Ordering on multiple connections should be handled in some way - whether it's just a documentation issue or a more fundamental one.
  • Should we have a tutorial mode that bring up tooltips when someone does something new as if it were a live tutorial.

Oct 12, 2010

• Trac and Roadmap (Emanuele): Emanuele suggested we should create a development roadmap and make it a permanent item on the agenda. The idea is to go through the Trac tickets and use them to build the roadmap that would be made public.
  • go through trac and sort by priority
  • add other suggestions to roadmap as well
  • admins can add milestones via the admin tab
  • David will assign tasks on list; those assigned need to follow up and check and fix them

• Update on PythonSource error reporting and logging (Tommy)
  • Need to make sure that all messages are printed through debug.X, so that the debug level (verboseness in preferences) is correctly used
  • Can cut/copy/paste now
  • Can we prevent the delete from happening?
  • Phillip noted the spyder project has an interactive python console: http://packages.python.org/spyder/
  • Can press enter to get to the end
  • Need to check if this works or if we need to change
  • Now have ability to see stack trace from the GUI (triangle menu)
  • Suggest using a dialog to display trace when this item is selected instead of printing to console
  • Also saving stack trace to the execution log
  • Also printing debug information to vistrails log
  • Add trac item to fix prints to use debug
  • Try to fix core.debug to have gui.debug to eliminate Qt dependency

• Update on subworkflow (Daniel)
  • Notification when subworkflow is outdated triggered
  • new_abstraction signal from registry in addition to new_module signal so that we can check if we need subworkflows to update
  • How to display the possible upgrades (can be upgraded, can be upgraded but may break, etc.)

• New utility added by Carlos (currently at scripts/module_appearance) to draw fringes visually, which generates output that can be pasted directly into the add_module call.
  • Erik will test this on Mac and Windows.

• Preparing a "Get Started" tutorial for SIGMOD repeatability
  • Do we have instructions on how to use the latex package without crowdlabs?
    • links to actual workflows in latex
    • have README and example for latex in the source
  • Metadata associated with VisTrails: should we have the ability to add vistrail-level metadata? e.g., who created the vistrail, its purpose, etc.
    • have pointers to paper to workflow and vistrails
    • add pointers to papers
    • allow access to add/edit annotations at the vistrail level
    • add GUI element to allow people to edit/add annotations

• Maintaining VTK package
  • have four classes that don't wrap, have error even in python level
  • works well with VisTrails otherwise
  • haven't checked the new changes from the wrapping
  • Wendel will check on the new version

• Web services package (Tommy)

Oct 5, 2010

• Welcome Tommy!

• Update on subworkflow (Daniel)

• Issues raised by Matthias:

- need to automatically upgrade subworkflows. Currently it is necessary to manually upgrade a subworkflow whenever the version of one of the modules inside them changes.

- explorations do not detect an MplFigureCell embedded in a subworkflow.