Difference between revisions of "Development"

From VistrailsWiki
Jump to navigation Jump to search
 
(957 intermediate revisions by 5 users not shown)
Line 1: Line 1:
<div class="nonumtoc">
<div class="nonumtoc toclimit-1">
== 2013 ==
__TOC__
 
</div>
=== Nov 20, 2013 ===
==== Updates ====
* [RR] Branches to be merged in: tuple-fixes ([https://www.vistrails.org/ticket/224 #224]), for-module ([https://www.vistrails.org/ticket/282 #282]), richtextcell-rtf-support ([https://www.vistrails.org/ticket/789 #789], needs pyth lib), ''remove-lockmethod ([https://www.vistrails.org/ticket/791 #791])''
* [TE] Fixing bugs for 2.1
 
=== Nov 13, 2013 ===
==== Updates ====
* [RR] Branches to be merged in: improves-module-doc ([https://www.vistrails.org/ticket/426 #426]), remove-persistence_exp ([https://www.vistrails.org/ticket/703 #703])
* [DK] Scripting support (parsing python source to guess ports), translate python functions to VT modules
 
==== Items to Discuss ====
* [RR] Delete old branches? [[Development Branches#To be deleted?|Development Branches#To be deleted]]
** Yes, this is fine, commits are preserved (just not named any longer)
* [RR] matplotlib figure/plot modules problem ([https://www.vistrails.org/ticket/396 #396])
** We are using matplotlib backwards
** try to defer code execution until we hit MplFigure (just save code in a buffer)
* [BB] Add uvcdat branches to github
** Question about whether to make vistrails a submodule, move the repository to github?
** Do we create a separate repository here?
** [BB] to talk to Claudio about this
* [TE] What to include in 2.1
** Fixing all tickets would delay the release. Some of these are present in 2.0 so it would still be a good release.
** Suggest making feature releases (2.2, 2.3, ...) for the working branches that are not ready.
** (updated with (status) as of 11/20 -- RR)
** '''[https://www.vistrails.org/ticket/224 224](tbm)''' tuple objects - move to 2.x?
** '''[https://www.vistrails.org/ticket/248 248]''' multiple 1D transfer function - priority?
** '''[https://www.vistrails.org/ticket/277 277](fixed)''' & '''[https://www.vistrails.org/ticket/722 722](fixed)''' pipeline validation bug - priority?
** '''[https://www.vistrails.org/ticket/282 282](tbm)''' fold flexibility
** '''[https://www.vistrails.org/ticket/490 490]''' add label via api
** '''[https://www.vistrails.org/ticket/581 581](reprod?)''' latex: clicking figure in pdf - reproducible?
** '''[https://www.vistrails.org/ticket/652 652]''' helper method for function upgrades - move to 2.x?
** '''[https://www.vistrails.org/ticket/670 670]("easy")''' Traceback in configuration window - easy fix?
** '''[https://www.vistrails.org/ticket/699 699](mainly fixed)''' parameter-configuration-api ready for 2.1?
** '''[https://www.vistrails.org/ticket/709 709]''' & '''[https://www.vistrails.org/ticket/712 712]''' abstraction bugs - move milestone to 2.2? (use uuids) - no, these are more simple ones
** '''[https://www.vistrails.org/ticket/710 710]''' Module configuration widgets should be default-aware - fix for 2.1?
** '''[https://www.vistrails.org/ticket/736 736]''' update documentation
** '''[https://www.vistrails.org/ticket/754 754]''' usersguide controlflow - fix Conditional module
** '''[https://www.vistrails.org/ticket/755 755]''' new persistence package
** '''[https://www.vistrails.org/ticket/762 762](tbm)''' import sys package
** '''[https://www.vistrails.org/ticket/771 771](fixed)''' create schema if missing?
** Also see [[Development_Branches]]
 
=== Nov 6, 2013 ===
==== Updates ====
* [RR] Branches to be merged in: disable-thumbnail-test-old-vtk ([https://www.vistrails.org/ticket/764 #764]), cacheable-controlflow ([https://www.vistrails.org/ticket/778 #778])
* [RR] Added a level to the LoopExec structure (LoopExec/LoopIteration), <strike>doesn't work, need help ([https://www.vistrails.org/ticket/774, #774])</strike>
* [TE] job-info done - ready for integration with other branches
* [RR] to be closed: [https://www.vistrails.org/ticket/757 #757] [https://www.vistrails.org/ticket/683 #683]
 
==== Items to Discuss ====
* [RR] Data movement with multithreaded-interpreter
** Right now, single module (opt. Group) sent with inputs, outputs & provenance sent back
** Input/output serialized with pickle
** Doesn't work with e.g. File (should allow modules to customize?), might be very slow (send multiple modules to minimize data movement?)
** [JF] ReproZip does some rewriting to change filenames
** Implement the simplest solution.
** For now, look at fixing this for simple File modules
* [TE] Update ticket milestones and sort [http://vistrails.org/index.php/Development_Branches branches] by milestone
* [TE] 2.1 release?  Specify what we want to put in the release
** Windows 8 build machine crash
** [TE] Go over open questions, distill what has been fixed
* [BB] UV-CDAT Animation issues
* [BB] ClimatePipes UI work, minor fixes and features
* Try to decide features, release data on 2.1 for next week.  Look at tickets, try to determine multi-threaded and command-line/documentation progress.
 
=== Oct 30, 2013 ===
==== Updates ====
* [RR] Rewrite of the logger ([https://www.vistrails.org/ticket/774 #774]) nearly completed (loops haven't been tested extensively).
** It looks like the test suite doesn't really cover that
** Then multithreaded-interpreter can be stabilized
** job-info?
* [DK] Further work on configuration/preferences
** Make it easy to add preference settings to GUI
* [TE] job-info
** Fixed issues with job identifiers
*** Can not use module signatures
** Working on examples
 
==== Items to Discuss ====
* [RR] Some tests [http://128.238.102.101:9050/builders/fedora17-64-v2.1/builds/24/steps/TestSuite/logs/stdio] [http://128.238.102.101:9050/builders/fedora17-64-master/builds/180/steps/TestSuite/logs/stdio] fail on Fedora (both v2.1 and master) but pass everywhere else, anybody knows why? (added [https://www.vistrails.org/ticket/775 issue 775])
* [TE] Signatures are not always unique---can have two different groups, do not depend on upstream?
** Groups that are duplicated in the same pipeline have the same signature, execute only once? [https://www.vistrails.org/ticket/765 issue 765]
** Does looping corrupt the cache? (e.g. with Map) (it does, added [https://www.vistrails.org/ticket/777 issue 777])
** Extra ModuleSuspended?
** Rémi showed some bizarre beahvior with the same subworkflow outside a group and one in a group
** Two options to make this uniform:
*** Always compute multiple copies when they appear in the same pipeline
*** Issue here is that the cache only contains one entry despite there being multiple copies of the same subworkflow
*** Always use the cache (build it incrementally)
*** Problem here is that if we have something which isn't quite stateless (e.g. vtk), having no copy might cause only one copy of the output
* [DK] Cache replacement policy?
** Ben may have done something with this for UV-CDAT
** Issue was determining the size of objects in Python
* [TE] Different brightness of images produced for VTK examples on Debian?
** Different versions of VTK? 5.4.2 (bundled with Debian 6)
** Added [https://www.vistrails.org/ticket/764 issue 764]
 
=== Oct 24, 2013 ===
==== Updates ====
* [TE] job-info
** Core functionality complete
** TODO: Testing, console mode, integration with batchq/hadoop/pbs
* [RR] Branches to be merged in: install_package_requirements, non-english-locale
** Yes
** non-english-locale - strange files in db
 
==== Items to Discuss ====
* [TE] OSX 10.9 does not work with VisTrails beta
** Has anybody upgraded yet?
* [TE] Does control flow modules get unique signatures?
** Log have extra "completed" instances of modules used in a Map.
* [TE] Debian thumbnail difference
 
=== Oct 16, 2013 ===
==== Updates ====
* [RR] Branches to be merged in: install_packages_requirements ([https://www.vistrails.org/ticket/694 issue 694])
* [DK] Rewriting startup and configuration
** unify command-line options that match configuration options
** reduce number of single-character flags (e.g. "-U")
** push startup into db to enable translation for future versions
* [TE] New [http://128.238.102.101:9050/buildslaves/debian6-64 test system] for Minimum Requirements running Debian 6 (Squeeze)
* [TE] job-info: Work in progress, core functionality implemented
 
==== Items to Discuss ====
* [RR] Support for spawned VisTrails instances (multithreaded branches, parallelflow)
** used to use local configuration which is problematic
** just use configuration that always add packages
** Sometimes need to reneable the spreadsheet so that certain modules (e.g. VTKCell) are available.
** [TE] Perhaps this could support the VisTrails server too?
*** Server needs to be looked at because it seems to be out of date
* [TE] Update crowdlabs for 2.1? At least start testing
* [TE] New [http://128.238.102.101:9050/buildslaves/debian6-64 test system] for Minimum Requirements running Debian 6 (Squeeze)
** [http://www.vistrails.org/index.php/Building_From_Source Minimum requirements] have been updated and are based on this system
** A few bugs fixed, still an issue with [https://www.vistrails.org/ticket/759 setPlaceholderText does not exist in Qt 4.6.3]
*** Should be able to replace this with setText in cases where Qt is older.
* {DK] Command-line stlye
** subcommands? e.g. "vistrails batch", "vistrails paramexp" [TE] "vistrails job"
*** [TE] makes sense to have different commands
** support multiple vistrails? multiple versions? (e.g. vistrails /path/to/example.vt -v tag -v 12, /path/to/example.vt:12)
** package options (move spreadsheetDumpPDF, fixedSpreadsheetSize but allow flags (e.g. -P spreadsheet.outputType=pdf)
** GUI Configuration also needs to change-improve "Always, Never, "For this session"?
*** Always show the configuration options, possibly flag the ones changed with an icon
** Other changes? [[Rewrite Startup Usage]]
 
=== Oct 9, 2013 ===
==== Updates ====
* [DK] Rewrite startup code
** use db to better structure startup.xml
** default, persisted, current configuration
* [TE] job-info
** Persist jobs to disk
** Cache finished jobs to disk
** Run from command-line
** Integrate with BatchQ/PBS/Hadoop/parallelflow
 
==== Items to Discuss ====
* [BB] VTK Startup Times
** can we run this beforehand?
* [TE] Feedback on [https://www.vistrails.org/wiki/WritingCommitMessages WritingCommitMessages]?
** Approved.
* [TE] Minimum Requirements
** Add build machine with minimum requirements?
** What Requirements?
*** Python 2.6
*** Qt 4.6
*** 5 years for others?
* [TE] Status of PostgreSQL support (Ricardo@vistrails-users)
* [RR] Matthias wants the archive to have choosable filenames (can be done but makes collisions possible)
** two separate features: persistent caching and archival of final results
** key part is probably the transformation between the two: want to make this easy to go from an exploratory workflow to one that works with the archive
 
=== Oct 2, 2013 ===
==== Updates ====
* [DK] rename-api branch updated with documentation
** includes name changes to match underscore style
* [DK] module configuration widgets
* [TE] Examples in usersguide are now clickable
* [TE] Added registration of file types on Linux
 
==== Items to Discuss ====
* Parallel Design for UV-CDAT
* New persistence package
** add UUIDs as identifiers for vistrails/actions so that each workflow could have a unique id
* [DK] When/if to merge rename-api?
** Should be ok to merge into 2.1?
* [TE] Show pipeline or history view when opening a vistrail?
** Either use configuration setting, or default to history view if vistrail contains tagged workflows.
** Default to the history view for saved files but make a configuration setting
* [TE] New commit message syntax
** #ticket/bugfix/feature 777 (on third line)?
** why not Ticket: Bugfix: Feature: on last line, like Signed-off-by lines? or could be on any line really
** Go with last line and don't start with hash as this may conflict with comments (which start with a hash, too)
 
=== Sep 25, 2013 ===
==== Updates ====
* [RR] I forgot to mention [[Development Branches]]
** [TE] Thanks, lets continue to use this and keep it up to date.
* [DK] Rework Developer API and Documentation
** Documentation: [http://vgc.poly.edu/~dakoop/tmp_vistrails_doc/api_documentation.html API] and [http://vgc.poly.edu/~dakoop/tmp_vistrails_doc/packages.html Writing VisTrails Packages]
*** Reorganized the "Writing VisTrails Packages" chapter; I could never find the text I wanted: [http://vgc.poly.edu/~dakoop/tmp_vistrails_doc/packages.html Updated Chapter]
*** Still work to be done
*** Use sphinx autodoc to generate real API documentation: [http://vgc.poly.edu/~dakoop/tmp_vistrails_doc/api_documentation.html API Docs]
** Rework the whole Module API but maintain old calls for backward compatibility:
*** setResult -> set_output
*** getInputFromPort -> get_input
*** drop "FromPort" on other similar calls
** Add vistrails.core.modules.config classes ModuleSettings, InputPort, OutputPort, ConfigWidgets to give kwarg capabilities
* [TE] VisTrails v2.1beta2 Released
* [TE] Linking to examples from usersguide
** Uses vtl files on server to point to local examples
** Works with both html/pdf
** Cannot use crowdlabs because it cannot link to mashups and parameter explorations
** .vtl file extension need to be set up for this to work
 
==== Items to Discuss ====
* [BB] Aashish and Utkarsh from Kitware may be interested in using VisTrails in another project of theirs, and they had a few questions
** How hard would it be to export a workflow as a python script to run independently of vistrails?
*** easiest if there is a 1-1 mapping from lines of source code to module creation/parameter steps
*** would be nice to know what the goal of the work is (is this VTK or ParaView related)?
** Is it possible to extract just the workflow part of Vistrails as a lightweight api with or without provenance?
*** probably can be done, but currently there is no "core.workflow" project so action and module objects are both currently in "core.vistrail".
*** would be nice to separate the workflow pieces but this is currently not done, if it happens, would like to include in VisTrails
** Allow folks from Kitware to push code to the VisTrails repo (which after code review ends up in master).
*** Sure.
* [RR] core.system rewrite: merge it in? ([https://www.vistrails.org/ticket/743 #743])
** executable_is_in_path() returns True or False, only returns True if executable is on the PATH (previously: returned executable name if on the PATH or in top directory)
** get_executable_path() looks for executable in PATH and top directory and returns full path (previously: didn't look in top directory, didn't return file with extension)
** Removes executable_is_in_pythonpath() (purpose unknown) and example()
** Trying to remove temporary_directory(): remaining calls look like bugs ([https://www.vistrails.org/ticket/744 #744])
** Can test branches on buildbot for other branches using the web API
* [RR] Meeting with Matthias: rework the persistence package
** [[Archive|Summary of meeting]]
** [DK] Dropping git is fine, but one of the reasons for the choice was the ability to pull/push files to different machines.
** [DK] Do the key-value indexes work for searching for time?
** [DK] Are there existing flat object stores (with hases) that we can use?
 
=== Sep 17, 2013 ===
==== Updates ====
* [TE] build notifications
** If you break the build you get notified
** New mailing list vistrails-build@vistrails.org
** Will send digests to vistrails-dev once tested
 
==== Items to Discuss ====
* [RR] Issues to be closed: [https://www.vistrails.org/ticket/413 #413 HTTPDirectory]
 
* [TE] New bugs found by Matthias
** FileSink overwrite port FIXED
** Git username missing FIXED
** Persistent directories
*** The main issue was fixed, empty directories possibly still an issue
** Release new beta once these fixed?
*** Yes.
 
* [RR] Logger problem with multithreading: where to store parent executions?
** link between Group module in main workflow and modules inside the group
** create a structure to track this information
** make the modules themselves dictate the change, tell the logger when they have sub-executions
 
* [TE] adding/improving examples
** KEGG web service has moved to REST
*** Should we look into adding a REST package?
** Add more controlflow examples for new modules and add to usersguide
** Subworkflow, Group, Mashup, Parameter Exploration, aliases, vistrail variables - in usersguide, should we add examples?
** Packages without examples:
*** ImageMagick
*** qgis
*** rpy
*** sql
*** tabledata
*** vtlcreator
 
=== Sep 11, 2013 ===
==== Updates ====
* [RR] Reworking logger because of issues and incompatibility with multithreaded-interpreter
* [TE] Updating Job Monitor to support serialized jobs
* [DK] Module/PortSpec APIs.
 
==== Items to Discuss ====
* [RR] Issues to be closed (no further work on them?): [https://vistrails.org/ticket/131 #131], [https://vistrails.org/ticket/413 #413], [https://vistrails.org/ticket/626 #626], [https://vistrails.org/ticket/684 #684], [https://vistrails.org/ticket/703 #703], [https://vistrails.org/ticket/708 #708], [https://vistrails.org/ticket/720 #720]
 
* [TE] Issues raised by Mathias
** IPython console does not receive console output FIXED
** Map module broken FIXED
** Persistent directories broken FIXED
** Variant type checking FIXED by making it optional
** Subworkflow upgrades FIXED?
** Dulwich not included in Windows binaries FIXED in next release
** Removing files in persistence fails:
*** We're rewriting history which causes issues with identifying versions by commit hashses
*** Delete is disabled
** It seems to be working for him now...
 
* [DK] Improve testing / code management
** Are the VMs still running the test suite?  Are we seeing errors? Can we get notifications?
** Should we adopt a master/next/branches strategy like UV-CDAT uses?
** Strategies for ensuring all files are added to commits?
 
* [RR] [[VisTrailsWiki:About]] has an (outdated?) description of VisTrails but there is nothing on the main page
* [TE] New binaries for beta being completed
 
=== Sep 5, 2013 ===
==== Updates ====
* [TE] Hadoop
** Asynchronous hadoop streaming now works
*** Hadoop jobs detach using ModuleSuspended and persist when restarting vistrails
*** Uses a custom updateUpstream to execute Machine while skipping other modules if job is already running
*** Need to add widget for monitoring job output
*** Fetching job result files is not cached so it happens everytime
*** Demo?
 
==== Items to Discuss ====
* Multiple instances of VisTrails
** There were some errors in the way messages were received and the error code propogated, should be fixed
** One place where we exit() instead of returning code, want to make sure the correct shutdown code
** Try to take the same path as the "error path" which shuts things down
* [RR] Requirement installation from packages ([https://vistrails.org/ticket/694 issue 694]): init.py is imported before the package_requirements() check
** agree that we should change this, new packages should do py_import in package_requirements call
** update release notes and manual to highlight this ("as of verison 2.1...")
* [RR] Remove "creating new figure" code from MplFigureCellWidget for [https://vistrails.org/ticket/672 issue 672]
** [DK] Should be able to remove this, left over from trials to move
* [RR] Remove persistence_exp package? (or at least, code?) ([https://vistrails.org/ticket/703 issue 703])
** [DK] Should be able to get rid of, should be all in the current "persistence" package
** Update GUI view for persistence to allow more meaningful searches
* [TE] Change app logic so that exit codes can be propagated correctly?
* [DK] Fix defaults to be more universally known in the VisTrails code (see tickets [https://vistrails.org/ticket/708 708] and [https://vistrails.org/ticket/710 710])
* [DK] Defaulted ports in Module.inputPorts versus ModuleRegistry.module_destination_ports
** Module.inputPorts stores only connected or set ports (e.g. the user has done something to that port)
** Iterating over Module.inputPorts will thus not include defaulted ports
** This is good for packages like VTK where we do not want to call all of the methods that default these values since they were already defaulted by the constructor
** This is not ideal for package developers who expect to see all of the ports that have values set in self.inputPorts
** Compromise might be to have a new method in Module like getInputPortNamesWithValues()?
* [DK] TODO: history of groups/subworkflows and start conversation on design
** [RR]: really want to be able to edit groups in place
 
=== Aug 28, 2013 ===
==== Updates ====
* [TE] Hadoop: Making hadoop package use batchq for communication over ssh
** first version of package needed to connect across firewall
** need to make hadoop work, then reorganize
* [BB] UV-CDAT: Get past the last few bugs
** Release target is Sept. 1, or 7-10
* [RR] Finishing up the threading stuff
** looking to unite with the JobSubmission/PBS work
** main issue preventing merge in: [https://vistrails.org/ticket/685 matplotlib incompatibility #685]
* [DK] Parameter widgets
 
==== Items to Discuss ====
* [TE] 2.1beta binaries
** Upgrading identifiers was broken. Dave, is this fixed? Should we update the binaries?
*** Think tickets 698, 699 could be fixed for this
** Keep lung.vt although we cannot run it?
*** nice example of a branching tree
*** could try to use other data for this?
*** should we contintue to distribute this?
** Other issues?
* Remi: nested modules
** tie each module with a configuration
** putting group information into the pipeline_view widget
* Parameter config widgets
* While module not in 2.0 yet, have a branch but need other bug fixes to make it work
** incorporate this into 2.0 branch, but don't make separate release for this
 
=== Aug 21, 2013 ===
==== Updates ====
* [TE] Hadoop: cannot run client from outside poly
* [TE] PE and mashup module id upgrades now fixed
* [TE] Currently building 2.1 beta binaries
* [RR] Execution target selection UI done for the multithreaded branch; needs to port IPython code.
 
==== Items to Discuss ====
 
=== Aug 14, 2013 ===
==== Updates ====
* [RR] New log schema on multithreaded-interpreter (see [[Multithreaded-interpreter#Changes]])
* [TE] Upgraded hadoop package to latest version, will try to integrate with the poly cluster
* [TE] Need to fix PE and mashup module id upgrades
* [DK] Output modules and PE upgrades
 
==== Items to Discuss ====
* Branches to be merged: dont_stop_on_first_error, optimize-module (While module) (v2.0? see [[New features|all changes to the pkg]])
* OutputModules questions: [[OutputModules]]
* Package configurations with multiple instances of VisTrails (e.g. with iPython worker instances)
** read-only mode that doesn't change configuration?
* Module remap upgrades: how to enable them immediately
** what type of remaps do mashups need? functions/parameters?
 
=== Aug 7, 2013 ===
==== Updates ====
* [TE] VisTrails 2.0.3 released
* [DK] Much improved db tests
* [DK] Many bugfixes to translations
 
==== Items to Discuss ====
* [DK] More general parameter exploration?
** Unify some of the mashup constructions with parameter exploration elements?
* [DK] More general module outputs
 
=== Jul 31, 2013 ===
==== Updates ====
 
==== Items to Discuss ====
* [DK] Where to put code to store bundles; should this be versioned?
** [TE] Yes, this makes sense, can address issues like mashups which didn't appear in earlier versions
* [DK] Caching: vtDV3D has its own persistent module store so that underlying VTK modules do not need to be reconstructed each time
** this means that the underlying VTK pipeline need not be reconstructed when a parameter on one of the files is changed
** does this actually speed things up? if so, can we generalize this style of caching where structure is the same and updating parameters on the structure can be done more quickly than reconstructing a new (sub-)pipeline
* [DK] UV-CDAT Developer Documentation
** document how to create a new package in UV-CDAT (simplest example to begin, pull in pieces of DV3D to showcase more interesting pieces)
** document pipeline_helper
* [DK] Coordinated spreadsheet cells (e.g. selection in one cell updates another cell)
** Jorge had a Coordinator module, VTK has some other coordination, DV3D may have some?
* [BB] UV-CDAT
** still looking at memory issues, using pysizer didn't seem to be picking up everything
Ω¸* [TE] Matthias has fixes for ALPS so we can build binaries
 
=== Jul 24, 2013 ===
==== Updates ====
* [DK] sqlalchemy changes working, need more tests
 
==== Items to Discuss ====
* [EV] NASA work
** familiar representation, convert between workflows specified already (e.g. using pyxb like java xml binding)
** NED Client: everything is Java, can run on local machine or remotely
** has workflow editor, tasks specifiied in hierarchical lists
** each task can have and executable and task dependencies, specified in tabular view
** different types of tasks (including loops, system, java)
** scientists have shied away from using workflow tool directly
** some scientists comfortable working at low level, running qsub directly
** [JF] Look at UV-CDAT-style layer?
** [EV] still need a way to control copying at the file level
** workflow front-and-center versus configuration front-and-center
** [JF] VisTrails treats changes to parameters the same way changes to the structure of the workflow
** configuration tree with lots of parameters that can be changed
** only four workflow variables? (user, cwd, ...)
** current scripts could be broken into smaller components
** global variables on a per-workflow basis
** creates running instance view showing completed/running tasks
** can examine logs for each task
** can suspend, resume, or abort a workflow
** build coarser modules?
* [BB] UV-CDAT bugs
* [DK] versioning bundles? what is the common interface for all versions?
** common locator and bundle -> specific version serializes/deserializes?
** if so, bundle must be extensible
** translation between versions happens at bundle granularity?
* [RR] referencing modules in Groups
 
=== Jul 18, 2013 ===
==== Updates ====
* [TE] Win32 version of ALPS still does not work
* [DK]  sql-alchemy, use-uuid branches
 
==== Items to Discuss ====
* [RR] Branches to merge? dont_stop_on_first_error, cltools
** [TE] to test the cltools changes?
* [TE] Update on NASA work
** has script that he wants to send to PBS through VisTrails
** issue comes up with PBS Pro (different versions of PBS)
** restore missing or failed jobs
* [JF] Microsoft using workflows in cloud
** data movement is a first-class operation, moving data and computation around
** also helper with resiliency
** vision seems to take things beyond crowdLabs-style
* [RR] Recording provenance in multiple machines (machine-wf-exec)
** Duplicate entries (part of module runs on local machine, part on remote), this is ok, but want to know these are linked
** Need more information in log about how execution occurred
** Push ip info into machine?
** have other information (name and/or MAC address)? privacy issues?
** allow users to configure machine name as a preference?
* [JF] have a wiki page for new features and bulleted list of items, things that changed, etc. so that we make sure to document these later
** [[New features]]
* [RR] parallel execution: choose where to execute each module, happens at runtime, want to allow user to control this
** where to store this information?
** store separately, but write each configuration to the log upon execution
* [BB] UVCDAT bug status
** 5 or 6 bugs left
** still looking at memory issues in UV-CDAT, seems to be a Python issue
*** read more on this? PySizer, Dowser
** grower bug needs significant redesign (need two outputs)
* [DK] uuid/bundle status
 
=== Jul 11, 2013 ===
 
==== Updates ====
* [TE] 2.0.3 release - Still waiting for reply from Matthias
* [RR] tabledata merged in
* [RR] Variant type checking #600
* [DK] UUID
 
==== Items to Discuss ====
* Demos of parallel flow and job submission
* UVCDAT bugs [https://github.com/UV-CDAT/uvcdat/issues?state=open]
** #221 Memory leak [https://github.com/UV-CDAT/uvcdat/issues/221]
*** could try to just delete the memory hogging piece internal in the VisTrails variable
*** caching in VisTrails: need to check size of python objects:
*** size of python object: http://code.activestate.com/recipes/546530-size-of-python-objects-revised/
** Make a .uvcdat dir (don't just use the .vistrails dir)
** [https://github.com/UV-CDAT/uvcdat/issues/230 #230]: Need to debug this
* Unique identifiers and configuration management
** looking at use-uuid branch again, updating to master
* [RR] Continuing on errors (<tt>dont_stop_on_first_error</tt>)
** Need to make it clear that there was an error (both in GUI and via API)
** able to get to results from each module via objects returned from execute
* Pausing workflows
** Interact using the progress dialog to cancel/pause
** multithreading doesn't fix everything because of UI drawing from packages (e.g. matplotlib)
* v2.1 release
** put together binaries, test
** anything left to integrate?
** hold off on UUID changes, mulithreading
** waiting on Matthias's changes for 2.0.3 release
 
=== Jul 5, 2013 ===
==== Updates ====
*[TE] Implemented mashup animations
*[TE] Eduardo at NASA is testing BatchQ-PBS.
*[TE] Waiting to hear from Mathias about fixing Alps for v2.0.3 release.
*[TE] Testing BatchQ-PBS on the NYU HPC cluster.
 
==== Items to Discuss ====
* Eduardo's work
** Installing VisTrails (can download binary and then replace source with latest source code)
** Workflow that allows people to create sandboxes or tap into existing runs in supercomputing environment
** Scientists have specific models that use various parallel techniques that use different codes and libraries, may have to grab source code and compile
** Important to be able to execute things from scratch (compile, set up environment, and kick off run), all of this has depdnency graph
** Running means everything is staged and just using qsub
** Monitoring using qstats to see where you are
** Have to know if/when it makes it to the queue, which files were produced, model output
** machines outside security boundaries still have two-factor authentication
** HPC environment won't allow connections over 12 or 24 hours
** Running scripts needs to be supported
** VisTrails needs to be able to track the scripts and progress in workflow based on outcome of scripts
** Not every failure means the next module cannot execute, could continue in other directions
** [JF] Suggest looking at ModuleSuspended state and CLTools & contrflowflow packages
* Terrence's work
** Distributed computing, running processes of different machines
** VT lacked multithreading at module level, adds layer so don't have to patch
** implications of order
** [TvZ] intention to use multithreading in the core of VisTrails?
*** [JF] Yes
** can run modules on different machines, VisTrails just orchestrates remote processes
** [JF] Have availalble at sub- workflow level, would have to have something running that understands the subworkflow scheme
** [TvZ] machines have "crippled" versions of VT that permite this sort of execution, crippled meaning that only enough to run the module, makes calls back to the remote machines
** Not a solution for everyone (manage the distributed computing environment)
** Use tagging for different workflow versions
** Modues will store configuration
* Provenance in multithreaded execution?
** Fernando wrote for parallelflow package
** Have some support for sending python code to remote machines
 
=== June 26, 2013 ===
==== Updates ====
* [DK] SQLAlchemy changes added, basic write/read works
** need to check things like updates
** mysql is one of few DBMS that supports multiple selects at the same time, need to special case *_many_* calls
* [DK] Fixed bug in PortSpecItem serialization (new expandAction flag)
* [TE] Working on animation support for mashups, similar to that in crowdlabs.
* [BB] UV-CDAT: working to keep variables in cache (load variable now creates a pipeline)
 
==== Items to Discuss ====
* [DK] Output types:
** Discussed before, but want to allow modules that produce output like SpreadsheetCell or StandardOutput to support a get_output() call that takes a mime-type and returns None or an output that matches the type.
** Could support, for example, inserts of textual values in LaTeX/wiki documents
** Also allow InputModule to ensure that we can do data lineage-style queries?
* Package flag on load_configuration should be removed
* Parallel execution: write an interface so that you could choose on a per-module basis which type of execution it can support
** parallelflow versus multiprocessing (xml serialization of pipeline versus pickling objects)
** advantage of xml is that large intermediate objects don't need to be pickled
 
=== June 19, 2013 ===
==== Updates ====
* [DK] Bundles work
** basic architecture: bundle contains a set of VisTrails entities, storage-specific serializers provide support for reading/writing the bundle
** serializers can register sub-serializers for each entity, sub-serializers could be added at runtime (e.g. for packages that wish to save to a vt bundle)
* [TE] VisTrails 1.0.3 binaries almost ready
** Troyer is looking into a problem with ALPS on win64
 
==== Items to Discuss ====
* [DK] Move DB support to something like [http://www.sqlalchemy.org SQLAlchemy]?
** can just use core SQL expression work for now
** makes cross-platform DB access much easier
* [DK] DB Updates would allow move of temporary vistrails to sqlite3 database
** should make updates much quicker for auto-save
** could consider updating database during runs == improved provenance
* [JF] Why not just use XML fragments?
** Actions are monotonic
** [DK] Survey of the "Web" seems to show that people don't like fragments
** [JF] Shouldn't be an issue
* Tagging
** Could we have a save/save as for tags (the tag moves down unless someone "tags as"
* [DK] UUID identifiers?
** Switch identifiers at same time?
** Allows easy merges
* [TE] Status of import_rewrite branch
** replace import statement to add "vistrails." prefix
** kind of broken in master right now
** new branch that has fixes for the
* [RR] Parallel execution
** controlflow issues
** change in update_upstream
 
=== June 12, 2013 ===
==== Updates ====
* [TE] Initial wrapping for Madagascar
* [RR] Fix imports for old-style packages
* [DK] Bundles
** Can load old vt files using the new interface, saving with manifest works
* [RR] Python library installation
** Can use pip (defaults to distrib package manager)
** Works on Windows
 
==== Items to Discuss ====
*[TE] old-style imports
** Did you test NumSciPy?
** Any ideas for PythonSource upgrades? Terminator "isosurface script" uses "core." imports.
** A few ideas:
** save the temporary package created in the look at call
** still an isue with python dependencies (better to have a graph than a list), otherwise when packages have same python dependencies, one package "owns" them, and the other doesn't have
** How to deal with PythonSource (__import__ comes from string exec)
* [RR, DK] Package states
** [RR] noticed that the code the preferences dialog uses to look at a package's dependencies, etc. uses a temporary object
** Means that any imports in __init__.py are not captured since the package was imported as temporary
** [DK] suggests that we have three states for packages: available, loaded, and enabled so that the look_at dependencies are captured but we can differentiate between packages that are loaded and those that are not
 
* [DK] New package development (dynamic or generator-based?)
** Noticed when looking at Madagascar wrapper
** Difference between VTK and new matplotlib, VTK builds the wrapper from underlying code each time VT starts while new matplotlib builds the wrapper from underlying code only at build time
** Likely to have better stability and consistency with generator-based
** Easier to incorporate new code in the dynamic environment
** Generator-based is more indirect
 
* [DK] Port types
** Consistent style for referring to port types?
** Have old class-type method which doesn't work well for userpackages
** String-based method may be harder to track errors?
 
* [RR] Multithreading attempt
** what are the issues with multithreading
** uses a task system to track things
** for backgrounding thread, using future library
** since Python has GIL, can call setResult from different threads without worrying about memory
** old modules still run the same way
** can we get workflows running on a separate thread so the interface doesn't hang
* [TE] Survey
 
=== June 5, 2013 ===
 
==== Updates ====
* [TE] Added IPython Console
** Requires installing IPython from source
** Uses an updated PyQt interface
* [DK] Bundle/locator updates, bug fixes
 
==== Items to Discuss ====
* Workshop discussion
** IPython console: required some pyqt changes
** Packaging
** Componentizing VisTrails
** Wrapping Madagascar (estimate the amount of effort)
* Pending feature branches:
** import-rewrite
*** Importing NumSciPy now works, but not unloading
*** probably because global __import__ override
*** potential issue with __init__ and temporary package?
** automatic-conversion
*** e.g. Float -> Int
** <strike>multiline-strings</strike>
*** Already merged but there was a textedit parameter view issue (It works on Linux) [TE]
** tabledata-pkg
*** merge the old tabdata package in here
** apt
*** a lot of (bad) things can happen in apt, but under normal operation, things work
*** use pip or easy_install for python packages when possible?
** new-pyqt-api
*** May break third-party packages
*** deals with QString, QVariant
*** Requires latest version from git
*** Connect to running python (without separate kernel)
* Scripting support
** Python script <-> workflow
* [TE] v2.0.3 Release
** MySQLdb was missing
** Autoremove vistrails-single-instance file
** Fixed latex examples [ES]
*** crowdLabs needs to be updated for full support
** Fixes for JobView, save_vistrail_to_db, scipy.weave
** More?
* User survey
** Claudio suggested talking with power-users like Tom Maxwell
 
=== May 28, 2013 ===
 
==== Updates ====
* [RR] 'tabledata' package: numpy arrays, CSV, data conversion
* [DK] GMapCell, TableCell, bundles
 
==== Items to Discuss ====
* [TE] Testing for db backend
** Currently no automatic tests
** Use sqlite, local/remote mysql
* Data conversion modules
** IBM WebSphere to convert image types
** useful in scientific applications
** Converter base module class
* New user survey?
** Look at old PDF, add new/old questions to a Google Doc
* [RR] new __import__ override
* [RR] Python 3: tricky (circular imports)
** Probably easier to keep
 
=== May 22, 2013 ===
==== Updates ====
* [TE] branches are merged, but there is some work to be done
* [TE] Images in thumbnails are now sorted by cell location
 
==== Items to Discuss ====
* [TE] Newline support for Strings
** We can use the Configuration Button to open a multi-line editor
** Do we want an optional small multi-line editor in the ports window as well?
* [TE] Unicode support
** Derek@CSIR is interested in using unicode symbols in labels, module names, and code.
** In a matplotlib source this works: xlabel(u'\u2211')
** We have a branch where saving to unicode works
** Full unicode support would require converting all strings to unicode u""-strings and making sure everything still works
** Supporting Python 3 is another option, problem is many packages don't support Python 3 yet.
*** The 'six' pakcage might be useful here http://pythonhosted.org/six/
* [TE] API tests fail on Linux, see [http://128.238.102.101:9050/builders/fedora17-64-master/builds/80/steps/TestSuite/logs/stdio here].
 
=== May 15, 2013 ===
==== Updates ====
* [DK,RR] Locator changes (Rémi also added unicode support)
* [DK] Persistence fix
* [TE] Demoed CrowdLabs on UV-CDAT meeting
** They want to create a web interface for UV-CDAT and may be interested in using CrowdLabs
 
==== Items to Discuss ====
* Feature branches to be merged in master?
** new-matplotlib-pkg (includes pkg-id-changes)
** locator-filenames
** spreadsheet-resizing, tempfiles, tests
* [DK] Mac binary with scipy.weave causes doesn't have nice Mac binary update, mac_site.py embedded in site-packages.zip
** look into easy_install issues, would help solve some of the missing package issues
* Persistence bugfixes:
** how to tell different versions (add a prompt to save a comment when a file changes)
* [DK] Bundle updates:
** Create DirectoryBundle, ZipBundle, DBBundle
** Add manifest file to Directory/Zip bundles?
** Serializer base class, register serializer with bundle based on serialization type and object(s) serialized
*** For example, a LogDBSerializer loads/saves the log to the database and would be called for DBBundle
*** Packages subclass from Serializer and define how they read/write data (given base properties)
** XML Fragments (appending data)?
** Support different types of bundles (e.g. register a Local/PersistentFileSerializer to add files used in a workflow to make a self-contained bundle).
** crowdLabs support?
** checkboxes in the UI so that things can scale (allow users to select what is exported)
* [TE] NumSciPy and "vistrails." import issues
** Update NumSciPy?
** create run.py at top level to get rid of old import path?
** How good is the support for old packages?
** what about compute for PythonSource?
*** could wrap __import__
*** use upgrades
*** add prompt and warning before doing these upgrades
* matplotlib date plotting not working, shows number
 
=== May 9, 2013 ===
 
==== Updates ====
* [TE] Mac binary: added support for scipy.weave
* [DK] Added better support for untitled locators
* [DK] More API Changes
* [DK] controller's current_pipeline_view is actually the pipeline view now (not the scene)
 
==== Items to Discuss ====
* [TE] Canceling VisTrails execution [DEMO]
** Implemented progress dialog similar to parameter exploration
** Shows progress by counting the modules
** Shows the executing module name
** Cancel works by catching pipeline scene update signals
** Users can enable module to be canceled by implementing progress reporting
* [DK] Threading for pipeline execution?
** Issue with Qt windows on different threads?
** In the middle solution where users could choose to run non-GUI workflows on a separate thread?
* [TE] UVCDAT Web meeting Monday
** Present crowdlabs/webgl
* [RR] Spreadsheet ideas from discussions with Dean
** Operations on multiple cells
** Select multiple cells and then perform same operation on all of them
** Selecting L-shaped set of cells (textual entry A1:A3, D1:D2)
** Formula-type bar in spreadsheet (like Excel)
** Provenance issues here
** Synchronization between cells (look at common properties)
 
=== May 2, 2013 ===
==== Updates ====
* [DK] Identifier switch
* [DK] matplotlib migration
* [TE] VisTrails 2.0.2 released - 100 downloads already
* [RR] has new utilities from constructing tests for testing packages
==== Items to Discuss ====
* [TE] VisTrails 2.1 beta ready to go
** would like to include the changes on pkg-id-changes and new-matplotlib-pkg
** should put the 'old_identifiers' in the schema
** add tests for new features
** what about reading 2.1 vistrails in 2.0?
** what happens when you load a version created using a future version of the package in a older version of VisTrails?
* [DK] core/gui coding: want to keep most logic in core
** want to be able to use packages like job monitor, etc. from core
* [DK] api/application logic
** currently have been moving logic from vistrails_window into api/application, does this make sense?
** should application call api or vice versa?
* [FC] Trying to gt Win8 installed
** Sent an email to Anup about this
** Also OSX 10.8?
 
=== April 24, 2013 ===
==== Updates ====
* [RR] New version of 'parallelflow' module (IPython)
** Now supports remote engines through ipython profiles
** No ssh support
** We could look into PBS support
* [TE] 2.0.2 binaries ready
* [DK] API
==== Items to Discuss ====
*[TE] Support more build machines?
** OSX 10.6? Emanuele might be able to build using her machine
** OSX 10.7? The build machine version (The current build is falsely named 10.6)
** OSX 10.8? Need a new build machine
*** Need to ask Emanuele how other versions can be built on the build machine.
** Windows 8? May not need a separate binary but should be good to test on
*** Ask Fernando to set up a virtual machine on the build machine
* [TE] Changes in 2.1
** Renamed vistrails.py to run.py to avoid conflicts
*** Need to make sure documentation is updated
** Renamed the "packages" directory containing contributions to "contrib" to avoid confusion
* [DK] API/App Changes
** Dave will commit changes that makes the api clearer
* [TE] I suggested creating a new devel branch and a new workflow for committing changes. I will look in to this some more.
 
=== April 17, 2013 ===
==== Updates ====
* [TE] Done with checking schema conversion and creating tests
* [TE] Started generating new v2.0 binaries
* [RR] DAT: data provenance viewer implemented
 
==== Items to Discuss ====
* [JF] The LaTeX extension issues
** uses the weather example
** [ES] related to CLTools package?
* [TE] No known v2.0 bugs that need to be fixed
* [TE] What is left to do for v2.1beta release?
** are some bugs
* [DK] need to look at merging the new matplotlib package
** need to add MplScript module
* cannot run parallelflow on multiple machines?
** we should set features before releasing the v2.0 version
* have root-level API?
** move the logic to core/gui as needed, and have the root-level import depending on mode
** need to separate concerns as necessary
* try to make the switch to the org.vistrails prefix
** keep the old identifiers and have both point to the same package?
* three types of subworkflow-like pieces: mashups, subworkflows, DAT
** maybe more implementation-level differences
** can we comine the infrastructure for these?
** should we try to use a similar API?
** for visual compression, more of a conceptual change rather than create new group/ungroup each time
* synchronizing aliases and vistrail variables
** really want to have some unifying solution here
* including GDAL, etc. in binaries?
 
=== April 10, 2013 ===
==== Updates ====
 
* [DK] Fixed the matplotlib cell resizing issue in current and new mpl packages.
 
==== Items to Discuss ====
* v2.0 Release:
** [DK] matplotlib cell size issue should be fixed
** Any reported bugs that are not fixed?
** add v1.0.3 -> v1.0.2 translation in the v2.0 release
* v2.1 Beta:
** What are the new features we should include?
*** auto-connection
*** visual port feedback (i.e. parameter set)
*** different port shapes
*** vistrails.-prefix
*** new matplotlib package?
*** mashup integration?
*** parameter exploration serialization
*** new schema for port spec
*** parallel flow, multiprocessing work?
*** what else?
*** add matplotlib examples inside the examples directory
*** add tests for the new features
** What needs to be checked before a beta is released?
*** want to make sure users that create vistrails in the beta can use them later
*** need to freeze the schema---are we happy with what is there?
*** need to check the schema v1.0.3 <-> v1.0.2 translations: are they robust?
**** [TE] I am checking by using the vistrail files on vis-7, but the update_db script needs to be updated.
**** v2.0 should support importing the v1.0.3 schema
*** add tests for things like schema
** matplotlib package
*** have Sunitha add some documentation about how to use the mpl package
*** still need legend updates, patch support, colorbars, script support
** DAT VTK issue with flickering
 
=== April 3, 2013 ===
==== Updates ====
 
* [BB] Working on UV-CDAT bugs, release mid-April
* [RR] Working on the VTKCell Overlay
 
==== Items to Discuss ====
 
* [DK] Better DOT_VISTRAILS management.  Why not have "modes" which correspond to certain package sets?
** USGS users have some requirements here (sometimes run with SAHM, other times without)
** It's really annoying to have to update the enabled packages each time
** That way, if I am interested in running without the spreadsheet, etc., I can just run in a given mode and when I want the spreadsheet and VTK, etc. back, I just switch back
** basically, just segment out the packages part of startup.xml and make multiples of these
** also add command-line flag
** [TE} maybe ony specify only whether some packages are enabled or disabled?
** [TE} maybe just point to a different user-packages directory
** what about just making command-line flags to enable/disable packages
 
* [DK] Latest VTK version
** Has anyone tested how the latest version of VTK runs with VisTrails?
** Huy ran into an error about "no concrete implementation..."
** [TE] May just need to ignore the class causing this issue
** [ES] Seems like the 5.10 master also had this issue before the release but perhaps they removed the features
 
* [DK] Installing packages/python dependencies?
** [ES] can install distutils into the Mac binary?
** WIndows has a full python install so this isn't as much of a problem.
** Just have a menu-item in VisTrails that sets the right configuration variables for the installer (point to the setup.py that the user wants to install)
** Want to install to a VisTrails library directory that lives outside of the binary.
** How to deal with python dependencies for compiled pieces?
 
* [TE] Ryan Danks on vistrails-users: I've been trying to get Vistrails to work in parallel both by using the server instructions found in the documentation as well as by using the standard Python multiprocessing.Pool objects within my module. Neither of these approaches seem to work. Has anybody successfully run their Vistrails code in parallel? If so, how?
** Tommy to check with Fernando on whether ParallelFlow satisfies these requires
** Is ParallelFlow ready for release?
 
=== March 27, 2013 ===
==== Updates ====
*[BB]Online/Incremental layout
** Added option to prevent any gaps in layers between connected modules. This in addition to preserving horizontal order seems to produce nice layouts from version to version.
* [RR] working on DAT: multiple variables per port working
** Also added variable number of inputs on List module
** Also did some changes on the controlflow package
** Also made some changes to the VT GUI as suggested by Juliana last week to the PythonSource and List modules
** Also updated DAT to use matplotlib plots and add conversion typecasts if variable types don't match
 
==== Items to Discuss ====
* [JF] Update to package API documentation
** Need to write documentation with this in mind
* [DK] Colin (USGS) asked about installing packages into VisTrails's python distribution.  What is the status of the easy_install work; are there other solutions?
** need to talk to Emanuele/Tommy about this
* [DK] UV-CDAT Scripting.  Is there anyone who can take this on?  Charles is planning a meeting for early April to show what he would like to see for this.
** Part of this is just designing meaningful modules (the wrapping side of things)
** The other side is what can be done to make the scripting interface more like a normal python script
** Currently, the scripting interface just builds a workflow.  This is somewhat unintuitive if you expect that things act like a normal python shell...
** Do we support partial execution as modules are added?
** What to do if a call is not wrapped?
** Perhaps a good starting point is the VTK example scripts.  I converted these to workflows by tracking all the calls.  Why don't we see if the scripting interface can convert these on-the-fly?
** Also, I want to keep in mind numerical ops and numpy.  Typing expressions is often much easier but we don't have a great way to represent this in a workflow.  What should be done?
** python has a built-in abstract syntax tree (ast) for itself that is user-facing.  Could we use this to better translate scripts to workflows?
* [BB] Add auto layout to api calls?
** Possble flag for this so that the user can choose whether things are laid out or not
** Also try to use this for the auto-snap feature in the GUI, BUT need to be careful to animate this as users will not be able to follow what is going on otherwise.
** the layout code now runs without any GUI dependencies but can still use QFontMetrics (like for the Cmd+L call)
* VirtualBox issues on the testing machine?  Emanuele is trying to fix, may need to contact Fernando
 
=== March 20, 2013 ===
==== Updates ====
* [TE] Save Camera (and similar) now works for unsaved and "saved as" vistrails using the new ensureController method
* [TE] DB Mashups merged into master
** working to upgrade the older mashups
* [TE] Added JobSubmission entry to FAQ
* [ES] Planning to have the binary ready for testing on Friday
* [DK] Work on matplotlib
* [DK] Help answer Charles's UV-CDAT questions
 
==== Items to Discuss ====
* [RR] Control flow modules:
** Lists vs multiple connections
** Using 'self' ports on ungrouped modules
** along each port to take a list or a single change a la Taverna
** try to write a new module to dynamically create a set of connections from a list
** do this in the interpreter? (need to modify the executing pipeline, not the specification
* [RR] Issue with "self" port connected (emailed Fernando)
* [DK] Similarly, should we try to add some method for ordering connections?
** Could display the order using a number overlay when a user moves the mouse over the port
** Then, selecting connection, could move up/down like illustrator...
** Probably would require an additional attribute to be stored in the workflow
** Rémi suggests mirroring Tuple here, (set number and type)
* [DK] Dynamic ports behind the scenes?
** Problem: Would like to have both stronger container types (e.g. List<String>) as well as flexible types (e.g. Float|List)
** Idea: Don't change anything about the current registry, but allow this at the interface layer and dynamically add the port to the module/registry when needed?
** For flexible types, could create myportFloat or myportList ports depending on the connection and then map these back to the same port before compute call...
** For container types, could only add a port type to the registry when it is actually used so we don't have to materialize everything beforehand.
* [JF] PythonSource/Tuple port types
** cannot type to search for port types -- have to type fast
** use a completer in the text field for this?
** drag and drop from module list?
* [BB] Online layout and gui-free layout
** Am close to a working solution that just uses the old layout but optionally preserves order of specified modules
** Having an issue with saving the positions, and strange issue with newly dragged modules vs modules loaded from file
 
=== March 13, 2013 ===
==== Updates ====
* Juliana's visit to Brookhaven National Labs
** interested in climate data analysis on observational data (also light source data)
** need to have executino on parallel machines, not able to find batchq package
** batchQ seems to be pure python and should not be hard to install, need to specify which version to install, add to FAQ
** UV-CDAT Sandy demo: [http://www.youtube.com/watch?v=BFN0RzN1hSE&feature=em-share_video_user Sandy YouTube video]
* [TE] Saving mashups to DB works, but there is still a bug with tagging a mashup, the interface is redrawn while tagging and a new mashup version is created.
* [TE] Working on getting spreadsheet "to version" method work with unsaved vistrails
** Added an "ensureController" method for loading a view from a controller object
** Getting the spreadsheet to work with controllers is difficult because it uses the locator everywhere and still needs the locator for things like saving the spreadsheet.
* [DK] More progress on matplotlib updates
* [RR] Fixed the bugs on DAT, trying to bring in more examples from VTK
 
==== Items to Discuss ====
* It seems the ALPS package is not included in the Linux binary distribution. Is this a bug or is it omitted by design?
** [DK] Is it included in other distributions?  I thought ALPS provided its own installers that update VisTrails with the necessary support for ALPS.
** Ask Emanuele and/or Matthias about this to see what the current setup is
* We used to have a scripting window in VisTrails and also in an early version of UV-CDAT. What is the status of the support for shell-like commands in VisTrails?
** [DK] core.api has some support for commands like this.  See the [http://www.vistrails.org/index.php/FAQ#How_do_I_create_and_execute_workflows_in_a_standard_python_shell.3F FAQ]
* [DK] How to best deal with importing Modules from other packages.  For example, if I want to subclass a Module in matplotlib, I can use the registry (ugly, see [http://www.vistrails.org/index.php/FAQ#How_do_I_access_modules_from_other_packages.3F FAQ], or I could try to directly import from vistrails.packages (e.g. import MplFigureManager from vistrails.packages.pylab).  While the second method is nicer, it is also not very robust because
# it does not enforce package dependencies.  I can import from pylab even if that package has not been imported.
# It does not work great for a package that might be in packages or $DOTVISTRAILS/userpackages depending on how it was installed (e.g. ALPS).  A developer would have to check both places.
** One option is to make any import of vistrails.packages use the registry method but hide it from the user.  Not sure how this works for non-modules, however.
** Another option is to have vistrails.packages import all of $DOTVISTRAILS/userpackages which means a developer has to use the "from vistrails.packages import" syntax to access items.
** Other ideas?
** Just limit the way an import occurs (only the "from vistrails.packages import <package_name>" syntax)
* [TE] Should we add BatchQ to VisTrails?
** Users are having problems because only the stable branch of BatchQ works with JobSubmission
** Could we use git submodule?
 
=== March 06, 2013 ===
==== Updates ====
 
* [RR] DAT can now execute operations on variables
* [ES] Build Machine: Windows 32-bit setup is ready, but the 3D Texture Mapping doesn't work on the VM. I will test if the problem will be present in the binary.
* [TE] Saving mashups to DB now works
** Will add more tests and then merge into master
** After that update crowdlabs to use it
 
==== Items to Discuss ====


* [ES] Build machine:  
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]
** Do we have to include the patched version of SUDS in the binaries? (I noticed the example that required it was removed from the repository)
*** It was decided to keep it
** Do we have to support old packages (for example, NumSciPy)?
*** It was decided to use the prebuilt binaries for now
* [TE] Adding locators to unsaved vistrails
** This will fix view switching issues with spreadsheet and vtk
** We should be able to add a MemoryLocator class to unsaved vistrails
** It was decided to store a translation from controller/view to locator


=== Feb 27, 2013 ===
== 2017 ==
==== Updates ====


* [TE] Working on mashup bug fixes and DB support
=== October 30, 2017 ===


==== Items to Discuss ====
'''Updates'''


* [TE] Animation support for Matplotlib cells?
* [DK] PyQt5
* [TE] Will look at adding locator for unsaved vistrails
* [RR] Running Python3 code


=== Feb 20, 2013 ===
'''Items to Discuss'''
==== Updates ====


* [TE] Added PBS support to BatchQ master branch
* Installation:
** Troels working on a few performance items
** Use conda? (macOS hack to have .app bundle load conda-installed vistrails)
* [TE] Also working on adding mashp
** Update conda recipe (pyqt4 restriction for now)
* [BB] Looked into some existing graph libraries for layout (NetworkX, igraph, boost graph tools, ...)
* Interpreter
* [DK] Checked in WIP on the new matplotlib package (include README about how things are generated)
* Release
** [DK] Need to clean up the specs for this package
* Integration
** [DK] Figure out what the new student can help with
** hard to build same type of GUI in other apps that integrate core VisTrails (tend to use just the API)
* [RR] Determining how to type data
** independent pieces of the GUI could be reused if they were more independent.


==== Items to Discuss ====
=== September 6, 2017 ===


* need some type of dataset standard for use with the DAT
'''Updates'''
* what about list of (floats, strings, any type of VisTrails object)
* require some additional wrapping for the DAT right now
* also think about numpy/scipy operations


=== Feb 13, 2013 ===
* [RR] Fixes for scikit-learn, tables, usagestats
==== Updates ====


* [TE] Job Monitor Improvements (New DEMO?)
'''Items to Discuss'''
** need some documentation
** Troels is working on some features for multiple jobs (100+) because each job creates a new connection
** also going to change a bunch in the implementation
** some projects need PBS submission capabilities to be using VisTrails
* [DK] matplotlib work improving (a few things still missing but working, mostly automated)


==== Items to Discuss ====
* Stripped-down version of VisTrails to showcase new features?
** include meta-vistrail, bundles
** hide mashups, extra widgets, latex mode, etc.
* Interpreter


* [DK] vistrails prefix and numpy issue that Remi found
=== August 16, 2017 ===
* [DK] online pipeline layout code
* [JF] shape for File module
* [JF] Sunitha to work on examples for matplotlib


=== Feb 06, 2013 ===
'''Updates'''
==== Updates ====
* [DK] meta-vistrail
** Working actions: normal action in vistrail, pruning version tree, tagging operations, and change
** Interface


* [FC] Working on the package for SciCumulus
'''Items to Discuss'''
** Widget to query run-time provenance from the cloud (almost done)
* Notebook
** Use ModuleSuspended in the execution (to be done)
* Interpreter
* Web visualization packages (e.g. bokeh)
** [https://github.com/VisTrails/VisTrails/pull/969 PR 969] has more details
* Pipeline templates


* [TE] BatchQ-PBS
=== August 2, 2017 ===
** Finished a Job Monitor Widget for VisTrails (DEMO?)
** Need to add provenance gathering


* [DK] Fixed minor issue with upgrades (USGS)
'''Updates'''
* [DK] Working on matplotlib bindings
* [DK] meta-vistrail work
** Extend Vistrail to MetaVistrail to support meta-actions
** Vistrail becomes less tied to workflows so it can also supporting versioning vistrails


==== Items to Discuss ====
'''Items to Discuss'''
* DAT project
** Basic functionality is there, some small example packages
** How do we handle multiple projects?
** How do we support multiple 'layered' plots per cell? (UV-CDAT needs this)
** [BB] Configuration widget
*** show ports pane for each module in plot subworkflow
** [CS] Automatic labeling of the pipelines
** [CS] Also try to auto-generate some of the code that is used to specify ports in the DAT
** Talk to Jerry and Tom about what they like about UV-CDAT, what can be improved as feedback for the DAT (talk about refactor)


=== Jan 30, 2013 ===
* Interpreter
==== Updates ====
** Using dask?
** Streaming an issue
* Notebooks
** What is an output of a module? (need to expose a specific output port or the entire module)
** Unifies some of the work from the script-workflow code


* [DK] Fix for older packages to run using new vistrails. prefix
=== June 28, 2017 ===
* [DK] Finished dulwich translation of persistence package
* [BB] working on DAT)
* [FC] working on integration between VisTrails and SciCumulous
* [TE] BatchQ and PBS, module suspended state
* [TE] way of mointorining jobs
* [TE] mashups on crowdlabs in database, Emanuele will help on this


==== Items to Discuss ====
'''Updates'''
* [DK] Testing UUID/SQLAlchemy branch
* [DK] Working on meta-versions
** would be nice to have pipeline and vistrail separate
** [RR] Then do we only have meta-actions, any change to the pipeline is a meta-action that adds a version that does the change?
** [DK] Yes?
* [RR] Notebook interface (Vizier) making good progress, figuring out cell outputs


* [DK] Need to update documentation for vistrails. prefix
'''Items to Discuss'''
** check vtl files here
* [DK] [https://vistrails.org/ticket/649 Ticket #649]


=== Jan 23, 2013 ===
=== June 7, 2017 ===
==== Updates ====
* DataONE Updates
** obseleted tags
* [TE] First version of BatchQ-PBS ready. Waiting for feedback.
** forked on github, no changes to the VisTrails package
* [DK} Fixed Python dependency determination
* [DK] Added a branch with UUIDs as identifiers
* [DK] Working on using dulwich (git implmentation in python) for peristence package
* [FC] Updates in PROV exporter
* [FC] Start integration between [http://www.computer.org/csdl/proceedings/cloud/2010/4130/00/4130a378-abs.html SciCumulus] and VisTrails
* [RR] Multiple projects in DAT?
** Some way to tie cells back to the project is needed
** Global/local mismatch is an issue here (if I drag a variable from one project and combine it with a variable from another project, which project does the new vis/data below to?)


==== Items to Discuss ====
'''Updates'''
* [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs


* [JF] Update on package David worked on for USGS (control/suspend execution, support execution on different computational environments) -- msg data 12/18/2012
'''Items to Discuss'''
** methods to fork off processes and to condor
* [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)
** need to track the settings for condor/asynchronous runs in provenance
** Can designate in module description an output port
** more general modules for this?
** Value has to be serializable
** abstraction mechanism for subworkflows to split up workflows (mixed execution models)
** Should use the OutputModes work to handle multiple modes, configuration, selection, ...
** better indication of when a module has been started but is not completed.
* [RR] Is VisTrails with SQL usable concurrently?
** BatchQ shows "suspended" state for this, stops executed this branch.
** [DK] Probably not, demo from [TE] exists


* [TE] Specifying python dependencies and py_import at the same time is confusing. We could probably call py_import from the package loader if the needed information was specified in the dependencies.
=== May 10, 2017 ===
** merge these two things for packages (otherwise have issues with specifying new dependencies and trying to import them)
* [TE] Will new import structure and UUID:s go into 2.1?
** Synchronization issue
* [DK] Nix ideas?
** we should at least record some hash of the wrappers/packages in the provenance


=== Jan 16, 2013 ===
'''Items to Discuss'''


==== Updates ====
* Identifiers
* Bundles & DB Code
* Interpreter
* Merge / Versioning Version Trees
* Python3
* Workings of List
* Packages: Wrappers + new Versions (VTK7)
* Script Conversion
* Intermediate Outputs: reuse output module work?
** would be useful for both workflows (run to a particular checkpoint)
** and useful for notebook interfaces
* SQL backend: does it allow for concurrent access?
** Yes, real-time collaboration work for concurrent single vistrail access
** Should be able to access different vistrails at once without issues


* [TE] WebGL port issue: Wendel added fallback mode (use pre-rendered data), is now helping me getting it to work using port 80 through apache.
=== April 19, 2017 ===
* [DK] DataONE package
* Rémi's progress on the DAT
** working on VisTrails code to modify the spreadsheet
** GUI is usable except that the interface doesn't execute VisTrails pipelines
** Has a document available in Google Docs documenting progress
* [BB] All critical open bugs for UVCDAT are complete, one minor bug and some enhancements left


==== Items to Discuss ====
'''Updates'''


* WebGL on crowdLabs
'''Items to Discuss'''
** try to send cached images through port 80 if necessary
** need to inform users why an issue comes up (e.g. port is closed, using cached image)
* Remove the sci mailing lists finally?
* Building VisTrails on a NASA supercomputer?
* Web interface aka ClimatePipes


=== Jan 9, 2013 ===
=== April 5, 2017 ===


==== Updates ====
'''Updates'''
* Emanuele resumed setting up the build machine for Windows.
* [TE] crowdlabs now uses rendered image in vistrail tab if available
* [TE] Discussing WebGL port issue with Wendel
** One possibility is to use port 80 on another server and use a proxy
* [TE] Working with Troels on adding PBS support to BatchQ
* [BB] 5 Bugs remaining with UVCDAT
* [DK] DataONE work


==== Items to Discuss ====
'''Items to Discuss'''
* Uploading Mashups to crowdLabs from VisTrails
** For this to work need to complete relational support for Mashups (see ticket [https://www.vistrails.org/ticket/611 #611]). The code is in mashups branch and needs to be updated with the current master (it was branched from old schema-v1_0_3 branch and not updated since then).
** Need someone to work on this for Matthias
** Not just relational support here, want to improve the schema
** Could just dump xml as a blob into the database, but were trying to make things work better
* Cleaning up older files (see last week)
* UV-CDAT Bug #69:
** The variables are global; cdat has a global dictionary
** thus closing a project has no effect on the dictionary
** if you have two projects that reference the same variable name, you run into issues
** discuss this with Charles
* UV-CDAT undo button:
** should be ok, just remember to update cell version


=== Jan 2, 2013 ===
=== March 8, 2017 ===


==== Updates ====
'''Updates'''
* [RR] Added a looping indicator on modules [https://github.com/VisTrails/VisTrails/pull/1217 #1217]
** This makes it easier to understand what's looping, no need to follow edges and count
* [RR] Refactoring CLTools for integration in Docker package
* [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)


==== Items to Discuss ====
'''Items to Discuss'''
* [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers [https://github.com/VisTrails/VisTrails/issues/1216 #1216]
** Can we move this up to the renderer modules?
** Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)


* Rémi went through the code to find modules that are no longer used
=== February 24, 2017 ===
** core.modules.module_configure and related have been moved to gui, can we remove the core redirects?  Problem is that this could break older packages
*** need to update documentation and add to FAQ
** should we continue to maintain third-party packages (those in /packages not /vistrails/packages) or should these be moved?
*** do we distribute the /packages directory with the source tarball?
*** we don't want to be responsible for keeping these packages up-to-date, but at the same time, we shouldn't distribute packages that do not work with our releases...


* crowdLabs issues?
'''Updates'''
** Trying to make ALPS server work with VisTrails
* [RR] Rich-text module documentation ([https://github.com/VisTrails/VisTrails/pull/1210 #1210])
** Thumbnails: make these higher resolution
** Went over packages and fixed format; maybe we should have a test to validate RST?
** Issue with ports and WebGL?
* [RR] Single-instance shutdown with <tt>--remote-shutdown</tt> ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])
*** need an error message if this is a port issue
* [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment
*** should be solutions/workarounds to use "standard" ports with the WebGL data


== 2012 ==
'''Items to Discuss'''
* [RR] There are issues with list-depth
** Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)
** Issue with looping groups


=== Dec 19, 2012 ===
=== January 25, 2017 ===


==== Updates ====
'''Updates'''


* [DK] USGS Work:
** Modules that use package settings
** Modules that do not update when input ports are not set


* [DK] Reproducibility Workshop
'''Items to Discuss'''


* [TE] Using a vistrail to test VisTrails (the approach adopted by UV-CDAT)
* Scaling VisTrails
* sql-alchemy, uuid, bundle branches/merges


* [TE] Parallel VisTrails (Remote PBS)
=== January 11, 2017 ===
** Trying to add PBS support to BatchQ
** Have a working PBS server for testing


*[BB] UV-CDAT Bug status
'''Updates'''
** 2 critical bugs and 5 minor bugs/enhancements remaining
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]


==== Items to Discuss ====
'''Items to Discuss'''
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails
** Have homemade workflow system based on R scripts and symlinks
*** Each steps has multiple sets of parameters
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code>
** VisTrails improves provenance capture and makes editing pipelines easier
*** Need better support for aggregation (groupby operation might be good enough; pandas?)


* iPython-style faceted output:
== 2016 ==
** iPython has a very nice feature where output can have multiple facets meaning the output in a command-line environment is different from that in a GUI environment
=== December 14, 2016 ===
** Basically, an output is actually a dictionary where keys are mime-types and values is the corresponding output for that type
** A similar approach would be very useful for cases where we wish to run a VTK workflow that on the GUI appears in the spreadsheet cell but on the command-line generates a file, and in crowdLabs uses Wendel's WebGL solution
** That way, a user writes one workflow that runs everywhere
** I think the output dictionary should be mime-type -> function, however, as we don't want to produce all the outputs if we only need one at a time
** Could also have modules that lookup the output types and could be used for configuring output types.
** Should use the same cell names for output


* Merge refactor-add-vistrails-prefix branch?
'''Updates'''
** Want to discuss with Rémi
* [RR] Single-instance code: over-engineered, but will work
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)


* Python on menubar for UV-CDAT
'''Items to Discuss'''
** [ES] Only way to change this is to change this in a plist directory of the python being run
* Email from Matt Dirks: combobox should work, ask for more code?
** [ES] Sent this to Charles at some point, will forward to Ben
* Email from marin.nl:
** looks like they would benefit from new interpreter
** how to store their consolidated provenance when they exchange .vt files around?


* Global variables in UV-CDAT
=== November 30, 2016 ===
** Can have multilpe projects open at once, each should have its own set of variables
** Possibility of copying variable from one project to another?


* NASA work
'''Updates'''
** use batchq and pbs, batchq uses lsf instead of pbs
** batchq uses suspended state
** execute workflow and run in the background


* Thumbnails and mime type issue
'''Items to Discuss'''
** python2.7 issue?
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])
** Still improving it
** Want to return something more structured to the client (success/fail code, stdout, stderr)
** Should be usable without Qt
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason
* [DK] Working on batch execution using console_mode
* [DK] Settings aliased parameters when using core.api?


=== Dec 12, 2012 ===
=== November 16, 2016 ===


==== Updates ====
'''Updates'''


[TE] Finished Vistrail Variable support for Parameter explorations
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)


[TE] Finished Improving interface for uploading to crowdlabs
'''Items to Discuss'''


[TE] Talking with Eduardo and Troyer about adding support for PBS to BatchQ
* Email from Pieter: executing pipelines with parameters from command-line using single-instance
** Seems to work, main instance gets full command-line and alters configuration temporarily
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful
** Actions:
*** Can add switch to kill main instance
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?


[BB] UVCDAT Bugs
=== November 2, 2016 ===
*format pipeline automatically
*multi-variable drag'n'drop
*plot and variable drag'n'drop behavior


==== Items to Discuss ====
'''Updates'''
[BB] UVCDAT 'Python' in menu bar on mac


=== Dec 05, 2012 ===
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)


==== Updates ====
'''Items to Discuss'''


[FC] New version of PROV exporter, now generating a PROV-XML from a VisTrails workflow
=== October 5, 2016 ===


[TE] Working on Vistrail Variable support for Parameter explorations
'''Items to Discuss'''


[TE] Working on improving interface for uploading to crowdlabs
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]
** unique ids
** bundles
** GUI fixes
** wrapping updates
** python3
** interpreter?


==== Items to Discuss ====
* command-line input parameters to provenance
** [DK] we don't create a persisted workflow from those input params, i think
** [DK] these are already defined as aliases so we treat them differently form a normal input param
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases
** Do the vistrails variables values get stored in provenance? (what if values change?)


* VisTrails: parallel and remote execution (Tommy will lead the discussion)
=== September 14, 2016 ===
** A way of running multiple workflows in parallel in vistrails
** Add support for cluster frameworks: BatchQ/Hadoop/MapReduce/Remote PBS
* Idea for parallel execution: use/extend the persistent sub-system so that "make-like dependencies" can be tracked. The spreadsheet should probably be avoided in the middle layers of such parallel executions.
* VisTrails and usability: http://www.wolfram.com/mathematica/new-in-9 (Claudio will lead the discussion)


=== Nov 28, 2012 ===
'''Updates'''


==== Updates ====
'''Items to Discuss'''
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)
** Did we know/contribute to this?
*** [DK] Yes. We wrote the text for the VisTrails chapter
** Drafted response
*** [DK] Looks good, will forward.


* [BB] UV-CDAT bug fixes
=== August 31, 2016 ===
** Redirected python output to a session log, and added command line flag to change the default log file or redirect back to std.out
** Initial work on setting up vistrails as a git submodule of a new uvcdat-vistrails git repo
* [ES] Package Installation
** easy-install branch should work on all platforms (not tested on Linux).
** currently working on making it platform aware (running easy_install on Windows x on Mac, for example)
* [TE] Working on VisTrails bug fixes
** Which ones are most urgent?
* [DK] DataONE package, a bit of UV-CDAT


==== Items to Discuss ====
'''Updates'''
* [RR] Went over vgc.poly.edu/vistrails.org config:
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
** Fixed usersguide scripts (won't disappear again)
** Moved some websites over to HTTPS
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that


* [BB] ParaView releasing new version 3.98
'''Items to Discuss'''
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
** Get rid of more services? (svn, trac, ...)
* [RR] Notebook execution works. How do we support more languages (than Python)?
** Need: detection, metadata reader, code to input values
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?


* [BB] Ideas for VisTrails "DAT" - generalization of UV-CDAT
=== August 17, 2016 ===


=== Nov 21, 2012 ===
'''Updates'''
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.
* [TE] Matplotlib and sklearn are exportable using module-to-script api
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.


==== Updates ====
'''Items to Discuss'''
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
* [TE] Wrap-up notes


* [FC] ParallelFlow -- working on creating SSH engines through VisTrails
=== August 10, 2016 ===
** I'm using vgchead for the tests, because there is no zmq (a required module) available for webdb1 (as it is an outdated version Suse)


* [TE] SubWorkflows now work on crowdlabs
'''Updates'''


* [ES] Package Installation
'''Items to Discuss'''
** branch easy-install works on a Mac 10.7 and above using a binary built with python 2.7.2 (it's in tests area in sourceforge)
* [TE] Python Wrapper and scripting
*** the problem is that py2app does not support setuptools and we have to use easy_install available on the system. We require python 2.7 and it is included by default on Mac only starting on version 10.7. If you install python 2.7 on 10.6 it will also work on that system.
** All changes merged into python-wrapper branch
** after I commit my changes it will also work on Windows (it will use easy_install available on the binary)
*** Tests pass
*** Still need to update sub-branches
** Minor fixes
*** vtkPlot3DReader upgrade bug
*** Added tuple exporting
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
**** Could be a bug in the old vistrail, but not sure
* [DK] Subworkflows and uuids


* [DK] spreadsheet analogy bug, group caching issue, switch to uuid
=== August 3, 2016 ===


* [BB] UV-CDAT bug fixes
'''Updates'''
** Fixed calculator command line related bug
** Added a simple cdms file cache to prevent files from loading/downloading twice, along with a GUI widget to selectively delete cached cdms files


==== Items to Discuss ====


* [JF] Upgrades
'''Items to Discuss'''
** Developer usability---make it easier to find and define the upgrades
** Improve error messages
** Also, want to get to the point where we detect changes in packages and alert developers & users to changes that could be important for provenance


* [FC] ParallelFlow -- IPython engines must be started remotely, and we also need to copy a configuration file from localhost (controller) to the machine where the ssh engines are located, before the execution starts
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]
** [http://www.lag.net/paramiko/ paramiko] is a nice Python module that connects to a remote machine, being able to execute commands and transfer files
** Connection variables now lowercase
** Is it ok to use this module, or a solution that does not use an external module is preferred?
** Can we use code as output variables?
** Talk about this next meeting, also see Joel Daniel's work on pushing data over ssh?
*** E.g. "vtk_dataset_mapper.GetOutput(0)"
*** Could mutate output when output method is only supposed to be called once
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)


* [ES] Package Installation
=== July 27, 2016 ===
** Just have Snow Leopard users install python 2.7
** Dealing with scripts and/or executables installed by easy_install
*** Where to put them? The location needs to be added to PATH
*** Put this in the user manual, do not add to PATH by default
** [DK] what about frameworks/libraries?
*** how would you deal with links to dynamic libraries?
** still need to work on package repository on vistrails.org
*** support indirection so that we don't have to host the packages
*** also put architectures supported in this repository


* [JF] Do we have configuration for environment variables in VisTrails?
'''Updates'''


* [BB] Moving uvcdat branch to it's own github repo
* [DK] DB bundle serialization implemented


=== Nov 14, 2012 ===
'''Items to Discuss'''
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]
** locale?
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches]
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]
** Still need to do something nicer for the patch formatting
*** patch keys currently contain non-python-variable characters but this can be fixed
*** Need to replace string template with python variables that is replacable using RedBaron
**** Patches could then look like:


==== Updates ====
    def vtkInstance():
        self.vtkInstance = self


* [DK] Refactored to be vistrails. imports
=== July 20, 2016 ===
** rope (Python refactoring tool) is awesome


* [DK] Package installation
'''Updates'''
** met with Tommy and Emanuele on Thursday
** Emanuele is working on detailing the process for installing to ~/.vistrails/Python directory
** [ES] I made some progress but also found some issues that I would like to discuss
*** When running VisTrails with the same interpreter used by easy_install, when enabling a package that supports easy_install, VisTrails will install it and enable it at runtime. This is using the same infrastructure to install python packages on linux systems but should work on any platform:
  rpy_dict = {'easy_install': 'rpy2',
                'linux-ubuntu': 'python-rpy2'}
  rpy2 = py_import('rpy2', rpy_dict)


* [DK] DataONE Package
'''Items to Discuss'''
**  [http://github.com/dakoop/vistrails-dataone GitHub]
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]
** Now supports uploading data to DataONE
** A patch is a string template with self/input/output variables for patching a class method
** Patches are stored in xml specification
** Patches are assigned to methods in ClassSpec's
** Used both for execution in vistrails and when exporting as script
*** Executed with "exec" when run in vistrails
*** Turns patches into natural calls in script
** Wrapping VTK library no longer needed
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)
** TODO's
*** Add to export_as_script
*** Patch Modules that represent Python functions as well


* [TE] JobSubmission
* [vistrails-users] Read data from SQL example?
** Eduardo would like to have synchronous job submission (wait until finished).


* [TE] DB SubWorkflows is working. Will test on vis-7 then add to crowdlabs.
=== July 13, 2016 ===


* [BB] Added two new command line flags to uvcdat
'''Updates'''
** -P --noDebugPopups prevents dialog popups from happening when debug/error messages occur
** -T TIME --time=TIME runs the gui for the specified TIME in seconds and then quits, useful for basic testing
** Should these be added to the vistrails master as well?


* [BB] Calculator commands related bug in uvcdat
* [DK] bundles on sql-alchemy branch
** Likely needs to be largely refactored to prevent further bugs with these
** Bundle, BundleMapping, and BundleSerializer
** New BaseSerializers to deal with versions of directory, zip, db serializers
** Can register the BundleSerializers with the base serializers
** Translation between bundle versions
** Bundle.add_object "just works" if an appropriate mapping was registered


==== Items to Discuss ====
'''Items to Discuss'''
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]
** API Module can be executed using '''compute''' method
*** Takes list of outputs to compute + inputs as keyword arguments
* Status of general cell synchronization (vistrails-users)


* UV-CDAT Repo?
=== July 6, 2016 ===
** Charles is moving the LLNL repo to GitHub
** Wants to know if we want to move our UV-CDAT code too
** We should probably not refactor this now, but we could move the branches?
** Any reason not to move the branches as is?


* Issues with calculated variables in UV-CDAT
'''Updates'''
** open file from calculator example
** Ben will ask Charles about information to CDMS variable


* VisTrails Package Installation
* [DK] Working on bundles
** It seems that we can't start an interpreter different from the one we are running. Does anybody know about this?
*** Because of the issue above, I couldn't execute system's easy_install from the VisTrails bundle
*** My solution would be to ship easy_install in the VisTrails bundle and use it when installing extra packages
** Need to test from the binary and frmo Windows
** Interrupted system call error
*** R_HOME = tmp.readlines() IOError: [Errno 4] Interrupted system call
*** ok when you enable from preferences


* Refactoring
'''Items to Discuss'''
** branch where everything uses vistrails. imports
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]
** e.g. "import core.vistrail.module" becomes "import vistrails.core.vistrail.module"
** Call module in script using vistrails API
** Makes installing into a system python much more straightforward
** Can be used when no export method is defined.
*** doesn't require PYTHONPATH tweaks
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?
*** deoesn't overlap with other libraries as easily
**[RR] No, unreleased
* [DK] New bundle organization
** BundleMapping contains individual mappings from raw objects to BundleObj
** Must be separate in order to have in-memory, separate from serializer settings


* [vistrails-users]
=== June 29, 2016 ===
** I would like to ask if it is possible to build stand alone applications based on vistrails?
*** Emanuele will answer, we can, just not sure what example is best here
** To try and avoid reinventing the wheel, I am asking if anyone has developed, or knows of a development, of a "text" module.  By this I mean, a module that is somewhat equivalent to a String module, but capable of allowing the input of multi-line text in a very simple editor.  At this time, plain text is more than adequate (i.e. for handling configuration files, or multi-line test data), but the module should have the capacity to write out the data strings to file.
*** Tommy will answer, String type is not the problem, just want a subclass of String that can have the multi-line entry as its widget.  Also suggest having a WriteStringToFile module that writes to temp file (FileSink persists as normal).  Could submit a pull request to github.


=== Nov 7, 2012 ===
'''Updates'''


==== Updates ====
'''Items to Discuss'''
* [TE] Export VTK as python script
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]
** Preludes (imports) still need improvements
** Works with vtkInteractionHandler (clipping plane)
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)
** vtkRendererOutput Uses VTKRenderWindow as output
*** Should support different output modes?
*** Does not work with multiple outputs (Next cell is displayed when current is closed)
*** Uses wrapped vtkInstance (as vvtk)
**** Create new patch system with code that can be used both in vistrails and in a python script?


* [FC] ParallelFlow
=== June 22, 2016 ===
** ParallelFlow can execute a module (or a subworkflow) for different parameters, in different IPython engines (embarrassingly parallelism).
** It is assumed that, in the machines where the engines are started, we have both VisTrails and IPython. Also, it is assumed that VisTrails is in PYTHONPATH.
** If IPython engines are local, this means that different threads are created for each engine - these threads are automatically created by IPython. This is particularly useful for multicore machines.
** IPython engines can also be created in a remote machine using SSH - this functionality still needs to be tested though.
** Modules can now define serialize and deserialize methods in case they need to be retrieved from the engines (e.g.: 'self' output port). In this case, each module needs to define its own methods. For instance, VTK modules need to implement these methods using VTK's serialization routines.


* [FC] PROV exporter
'''Updates'''
** First version of the [http://www.w3.org/TR/prov-dm/ PROV] exporter in VisTrails - this exporter will be used by the DataONE ProvWG.


* [TE] JobSubmission
'''Items to Discuss'''
** Eduardo working with Troels to add features to BatchQ
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])
** Uses module location to find module remap (The best we can do right now?)
** Also fixes remapped port names for connections to existing modules
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name


* [TE] Add SubWorkflow support to DB
* [TE] Working on exporting wrapped libraries (VTK)
** Almost done, need to fix some issues with upgrades.


* [BB] More UV-CDAT bug fixing
=== June 15, 2016 ===


* [DK] UV-CDAT bugs, DataONE additions, Inc report/updates
'''Updates'''
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])
** Fully implements connection merging and list looping


==== Items to Discuss ====
'''Items to Discuss'''


* [DK] VisTrails packages
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])
** Emanuele recently asked the py2app developers about making it easier to install other packages (and their dependencies into VisTrails)
** Analogy only works We need to use the non-upgraded action chain
** Reply is here: http://mail.python.org/pipermail/pythonmac-sig/2012-October/023744.html
** But pipelines need to be valid (matcher uses module portspecs)
** I like the idea of having system-wide and local user vistrails directories for added packages
** Tried to get upgrade remap from upgrade action, but this seems impossible
** This needs to be integrated with a much easier installation process.  Ideally, this looks something like:
** Use module location instead?
*** Users goes to package manager and clicks "Install Package".
*** Location should be identical for one-to-one module upgrades
*** We have a vistrails.org-based list of available packages that appears as a list.
*** User selects a package and VisTrails determines any VisTrails, python, or other dependencies.
*** We use setuptools or something like it to install the package and its dependencies into the local directories
** Ideally, this also works for system-based python installs as well, this probably means separate python 2.x directories for VisTrails packages.
** Also, for "import vistrails" style use of VisTrails, it would be nice to be able to specify which packages will be used instead of relying on the ~/.vistrails/startup.xml settings
** Potential issues include packages that use external libraries.  Can these also be installed into the package directories?
** Developer support: how does a developer build an installer for a more involved package (e.g. SAHM, ALPS)?
** [JF] VTK and such will be difficult.
** [DK] user-hosted packages
** how does this work for packages?
** Dave, Tommy, and Emanuele will meet on this


=== Oct 31, 2012 ===
=== June 1, 2016 ===


* [DK] scripting support (UV-CDAT)
'''Updates'''
** dependencies?
* VisTrails 2.2.4 announced (finally)
** allow scripting to automatically load dependencies
** 5 survey replies
** GUI dependencies here:
** 47 downloads last week
** .vistrails clashes between scripting and GUI
* [JF] organizing workshop on reproducibility
** need to understand what is out there (actually how they work and fit into big picture)
** others will need to help with this
** proposals are out
* [TE] Added git access as: git://vistrails.org/git/vistrails.git
** this mirrors the http://vistrails.org/git/vistrails.git
** UV-CDAT wants access via http
** http access adds a /git subdirectoy
* [TE] Subworkflows on crowdlabs not working
** they are not stored in the database
** need to have a unique identifier
** could also copy to the .vistrails directory, possible name clashes
** in the long-term, nice to have vistrails access to database
* [BB] Added new uvcdat-next and uvcdat-master branches
** mirror the LLNL uvcdat development structure
** allows us to test new features in next before they go into master
* [BB] packages were failing on the UV-CDAT side
** shouldn't try and import package when package requirements are not met
** .uvcdat directory or a sub-directory of .vistrails (as a project-specific directory)
** [CS} don't do anything right now, need a stable release
** [CS] need a UV-CDAT workshop
* [BB] most of the remaining bugs are not crashing things, just annoyances
* [CS] get Fabio started on DAT
* [FC] ParallelFlow package
** execution in the engines not always working
** starting engines using ssh, could also use webdb machines


=== Oct 24, 2012 ===
'''Items to Discuss'''
* [ES] Update: Package distribution
* [TE] Export VTK as script (python-wrapper)
** Installing extra python packages to be used in the binary
** to_python_script
** easy_install configuration that installs in a custom folder in user's home directory
*** needs the pipeline (or at least the connections)
** worked for R, rpy
**** Should not  generate code for any possible VTK method (exporter detects which ports is used in the source)
** possible issues with compiled things
**** VTK needs to know connection order
** should work for all platforms
**** Adding functions directly to the script results in nicer code
* [ES] New problem with rpy package (other than RVector)
** Add support for multiple input connections
** globalenv variable has changed--capitalization
** Add support for list looping
* [ES] SourceForge VisTrails Project upgrade
** new features: https://sourceforge.net/p/allura/wiki/Features/
** do we just move everything to github?
* [FC] Parallel Flow
** Local controller and local engines are now started using VisTrails GUI (under Packages menu)
** Trying now SSH engines (IPython also supports MPI and PBS)
** Local controller needs to be stopped when VisTrails closes (in case user forgets)
*** if controller is not stopped, will have issues upon restarting because controllers are still there
*** cannot stop controller outside of vistrails
** Outer subworkflows are transformed into groups -- for inner subworkflows, needs to be recursive
*** problem is that the xml files that define the subworkflows cannot be found on remote machines
*** [DK} look for code that finds the subworkflows
** have an example, can test with Emanuele's new package idea
** have test workflow where module takes 10 seconds, executing 4 times with ipython also takes 10 seconds!
** execution logs from engines are being insereted into provenance
** machines are also being added for all engines
* [TE] posted issue on github with temporary directories, added configuration option
* [TE] crowdlabs executable workflows, external ALPS server is not accessible
** script to check external error
** string object not callable error
** one Titan workflow made non-executable
* [TE} check email addresses for github for the fixes/closes shorthand
* [BB] working on UV-CDAT issues
** checking on mountain lion build
** expects variables to be cdms variables
** how to handle all variables (python-based)
* [BB] Visit: no callbacks on the c++ side coming through, issue with linking to the correct global variables


=== Oct 17, 2012 ===
=== May 25, 2016 ===


* [FC] paraflow
'''Updates'''
** New commit
* Mailing-list fixed
** No more startup code
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])
** Add start/stop menu options
* [FC] Remoting package
** Our solution cover this
*[BB] uvcdat calculator commands
** Need to talk to Emanuele about making this more robust, and what capabilities it should have.
*[BB] visit plugin
** Need to find how to access to methods used to implement the "isWorking" function originally created specially for the python VisTrails plugin
** Need to see if threading is still needed for the C++ version


=== Oct 10, 2012 ===
'''Items to Discuss'''
* Update the People page: http://vistrails.org/index.php/People
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)
** Also publications are likely out of date
* [TE] Union ports
* Mailing List
** Show union ports in pipeline (DEMO)
** Huy will check for anyone who signed up on the old vistrails-users list at SCI after we moved the list
** Then we can write Nick to shutdown the SCI list
* Remoting package:
** Terrence's solution: http://code.google.com/p/rpyc4vistrails/
** NASA and weather forecasting
* Parallel Execution
** Startup code in engines is possible (it executes a script before the initialization of the engines), but still not working. There was an [https://github.com/ipython/ipython/issues/2213 issue] before with it, that was apparently solved, but some other people still complained - waiting answer in mailing list
** Currently changing schema to add the notion of parallel execution
** Provenance capture:
*** make a new parallel execution entry, or
*** annotate that module was executed in parallel, can annotate each module execution with machine information
*** what about trying to determine which modules were tied to the same parallel execution
** Still going to look into serialization of some output values (e.g.: self)
** IPython shipped with VisTrails?
*** How big is iPython? 10-20 MBs
* Package distribution
** Ideally, users could download supplemental installers to add packages into binaries
** Hard to know which pieces belong to each package
** py2app selects only modules that we are using, hard to separate out particular packages
** could we tell py2app to install a full python distribution?
* [DK] bugs, UV-CDAT, NASA meeting, helping Fernando
* [BB] working on UV-CDAT
* [ES] working on builds
* [HV] finishing parallel execution in parallel, not locking GUI when doing execution
* [TE[ looking into making crowdlabs examples work, gridfields and ALPS


=== Oct 3, 2012 ===
=== May 18, 2016 ===
* UV-CDAT development
** status of bug fixing (Ben)
*** to do: msg with summary
** release date (do we know?)
** code re-org plans (David)
*** git submodule for vistrails core code
*** refactoring some code where we have added features (e.g. in CDMSPipelineHelper)
*** list issues
*** planning for what Fabio, Jorge, and Feng
** proposal for interface improvements (Claudio)
*** can we bring in other analysis here?
*** have a start on improving matplotlib support (could better integrate this into VisTrails)
** discussion of interface improvements for supporting Jorge's workflows
* VisTrails and IPython
** Singleton pattern in VisTrails: should it be really changed? Is there a way to destroy the global state?
** If output port is an object (e.g.: self), serialize it and send it back?
** How to include the execution (performed in the IPython engines) in the log?
** How engines should be started?
* website (www.vistrails.org)
** The header was updated and the source files added to the master branch
** Need to decide about the sponsors (currently they have a section on the main page).
* 2.0.1 release
** Matthias fixed the ALPS package and the release should be out today
* Including packages with dependencies
** how to best support this for developers and users
** ideally, users have automated or double-click installers
** developers need to be able to match and compile against the existing libraries inside the binaries


=== Sep 26, 2012 ===
'''Items to Discuss'''
* github
* [TE] mailing list still down - Julio working on it
** add README to github
* [TE] Improved union port
** fixes #NN for issues?  [DK] didn't seem to close the issue when I used this
** New schema 1.0.5 adds portitem.union string attribute
* website (www.vistrails.org)
** Works with pipeline view, parameter exploration view, and mashup view
** do we have access to the original illustrator? header
** Should work well for matplotlib
** documentation link should go to the current release sphinx documentation now? (even if not, the documentation page should point to the web version)
** Use with VTK ports?
* [DK] MissingPackage errors: made to ignore module_id so we do not get duplicates
*** It uses [item_1,item_2] style ports usually with different tuple length.
* [FC] Parallel execution
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]
** Using iPython and serializing workflow pieces and sending the xml to the engines
** Execution happens in workflow batch mode
** vtl is a wrapper for a vt file or a workflow, documentation
* [ES] 2.0.1 release
** almost done, checking ALPS package on Windows
** need tp test Mac version
** also should test SAHM package
** documenting the build and installation instructions
* Still need to have machines for other Mac OS-es (10.6, 10.8)
** signing the application: need to create certificates on developer
* [TE] BuildBot bugs when execcuting multiple instances at the same time
** disable some of the timing tests (e.g. for memoization)
** could configure branches to use different .vistrails directories
* [TE] Fixed the single-instance checks
* [TE] testing for crowdlabs workflows
** Titan package not working right now -- check with Wendel?, is this in VTK now?
** fixes for the crowdlabs web page (description overflowing, workflow names truncated)
* [BB]: working on the UV-CDAT colormap code: adding the module in code...


=== Sep 19, 2012 ===
=== May 11, 2016 ===
* No known items?


=== Sept 12, 2012 ===
'''Items to Discuss'''
* Improvements to connection with CrowdLabs (see Dave's message)
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]
** clean up interface
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?
** currently using 2.0 on crowdLabs
*** Shows single port for all ports with same sort_key
** check into whether master can be translated back to 2.0
*** Select specific type when adding/showing
* Update on CrowdLabs + WebGL
*** Only one type should be visible on the module at any time
** need some better documentation here
*** Making tuple parameters union types is probably too complicated
** examples in the user's manual would also be useful
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly)  
** ALPS image resolution is not good, might be using the uploaded thumbnail
** seems not to be working right now
* Update on iPython and multi-threaded VisTrails
** each workflow would run on a different thread
** trying to serialize Module object
** example is executing Map concurrently
** pickling errors...
** can we push port values across only---do not need to pickle module
* Feedback from Jorge on his experience with VisTrails (and the remote job submission package)
** need some documentation on using the package
** how are the UV-CDAT instances going to be running on the cluster?
** want to push a workflow and run on a cluster
** sync up with Fernando and Huy's work
** would be good to have linked views support
** add initial support from Jorge's UV-CDAT work
* FIXED! mac_update_bin.sh is not working -- it does not delete the existing directories from lib/python_XY
* CLTools:
** Add reload button in the wizard window
*** does the new module reload? or do I have to delete the module and add a new one?
*** try to use the upload logic here
** Add a debug mode that shows how the command will be invoked -- this should help the users understand how CLTools work and to get a wrapper working more quickly
*** want to see what the command line would look like with the settings
** We should have more examples that cover additional features of command line tools, e.g.,  the use additional parameters such as a flag to specify an integer value. It would also be useful to show what the module will look like in VisTrails --- this will help users understand, e.g., what the "Visible" flag means.
** It looks like the ENV var is set for the CLTools package. But what if different modules need different environments?
** In the documentation, there is mention to file "suffix". Is this related to the exchange with Remi on June 14, 2012 4:14:32 PM EDT?
* Open Tickets:
** [https://vistrails.org/ticket/577 577]: grayed out when publishing until focus change
** [https://vistrails.org/ticket/578 578]: two instances of VisTrails
* Testing
** BuildBot running at http://128.238.102.101:9050
** Testing section added to: https://www.vistrails.org/wiki
** People who break the build are notified by email
** To add more Virtual Machines we need more memory
* [ES]: Mac builds are now working
** pytables support has been removed (not working on Lion)
** already added to the tests
* Colin asked for new binaries to fix conflict with osgeo
** wants to distribute for his users


=== Aug 29, 2012 ===
* [TE] Feature suggestion: Let connections connect to functions/parameters
* Mailing lists?
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters
** should we keep the sci.utah.edu addresses?
** Use separate views for ports and functions (like before)
* Unicode issues [Rémi]
*** Allow rearranging of functions
** sqlite filename
** Pros:
** configuration files
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
** and filenames in vistails
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.
* WebGL and crowdLabs [Wendel]
**** Makes List module simpler
** Lis was able to add visualization
*** Connect to parameter in tuple
** Google bot tries to access some pages and gets sent an error
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds.
** fix the transformation
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.
** version issues, Wendel not getting the errors when opening versions from newer versions
** Cons:
** make work for any workflows
*** Interface more complicated and crowded
* Reproducibility [Fernando]
*** More difficult for users to grasp?
* iPython [Fernando and Huy]
*** No immediate use case
* crowdLabs bugs [Tommy]
*** Makes no union type for tuple parameters a bigger problem.
* test machine access [Tommy]
** working on
* Bugtracker on github
** how to specify branches in bugtracker
** want to add bugs for java branch
** cannot fork own repository
* which version of VTK does VisTrails currently support?
** we should make it clear on the website whch versions work (or what limitations exist in which versions)?
** tracking this for provenance


=== Aug 22, 2012 ===
* [TE] Publish user survey?
** No response from vistrails release yet
** Not announced on vistrails-users yet.


* Automated workflow layout -- done by Lauro -- Dave integrated
=== May 4, 2016 ===
** https://bitbucket.org/laurolins/wflayout/wiki/Home
'''Updates'''
** piecewise layout
* VisTrails 2.2.4 released
* Serialization (pickling) Issues with IPython
** take advantage of the the multiple engines in iPython
** schedule modules to run in parallel
** Dependencies in the Module class
** Objects that stays at library level
** VTK is problematic because data is in C++ (cannot send data)
* fix UV-CDAT bugs
** separate meeting for this?
** merge in latest v2.0
* [RR] Java Updates
** add README file on git with link to documentation
** README is shown on main github page so this is a better link back to the documentation
** Should we maintain both the Java and PyQt interfaces?
*** problem of maintaining both
*** jpype limitations (reflection, introspection), maintenance
* [DK] DataONE Package
** able to load DataONE data in a VisTrails workflow
* Add Jorge's work to news page
* [ES] Build machine is almost setup
** gridfields support not there
** move to Windows build
* [JP] Feedback from Oak Ridge:
** Mashups do not work in UV-CDAT, vcs or DV3D
** More next week and demo
** using scikit-learn
* Website maintenance:
** Update header
** Add page for contributors, contributing organizations
* [WS] WebGL and crowdLabs
** update next week


=== Aug 15, 2012 ===
'''Items to Discuss'''
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]


* Big Data (Huy will join the call)
=== April 27, 2016 ===
** [https://docs.google.com/document/d/1ph0vEdaV5Di4VNVWKlzN9_EXTj_j9wMXreRir2yxg-8/edit Big Data and VisTrails] (let me know if you cannot access the file)
'''Updates'''
** [https://docs.google.com/drawings/d/1X56DD-NkbuS8mq08v0t1rfRiJlMdgdeDhXy6iRT4NUg/edit Big Data Pipeline] (let me know if you cannot access the file)
** [http://ipython.org/ IPython] seems to be promising
** streaming and parallel execution
** simple example: parameter exploration
** ipython multithreading (python interpreter pool)
** threading helps on single machine
** for cluster, want to have "execution engine" component
** how do packages work here?
* Burrito + VisTrails
** Unpacking experiment: data files inside or outside .vt? (currently, only creating xml)
** Libs and hardcoded input files: how the user can specify that? A configuration file?
* Java code in our repository [Rémi]
** add java files and a Makefile to compile the jar
** need to figure out how to make sure this happens for the release (build script)
* Java version of spreadsheet
** working modulo a few issues
* Testing framework?
** Tommy prefers BuildBot
** Question is what mileage we get from adopting CTest? (UV-CDAT?)
* Move bug tracking to github?
** for now, keep separate from our trac and see the utility


=== Aug 8, 2012 ===
'''Items to Discuss'''
* [TE] Wrapping Bokeh
* Java [Rémi]
** New method type "operation"
** working on spreadsheet--can be used with real packages
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output
** writing exclusively in jython, would not work via jpype
**** Glyph sets function name and arguments as output
** keep GUI in java, cell widgets in Java, too
***** Not serializable, but should always be computed together with the subject of the operation.
** network visualization using perfuse, could port to regular version of vistrails
**** Figure applies Glyph operation on itself
* Possible analogies issue in spreadsheet
*** Useful for matplotlib?
* Show in spreadsheet which version a cell correponds to
**** matplotlib currently find actors by searching upstream?
** show a label in upper-right corner (tag or "tag + N")
**** But this is more difficult since matplotlib is not function-based.
** parameter values for parameter exploration
** New "union" port type
* Burrito [Fernando]
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value
** directory names
**** Uses only first value in tuple
** packaging into vistrails
**** Bokeh have many properties that support (list, string, Float)
* Caching issue
*** Does not work with depth=1 types (e.g. Float of depth 1)
** Dave will write an email summarizing the issue here
**** Turn Depth 1 types with known type into lists?
* Big data
** Wait for Huy
* crowdLabs integration
** Tommy is working with Wendel on this
* Testing machine
* setting up github
** Tommy is going to do this


=== Aug 1, 2012 ===
* [TE] New bugfix release
** Finalize survey


* testing [Tommy]
=== April 20, 2016 ===
** Switched from CTest to Buildbot
'''Updates'''
** The buildbot is up and running with a few slaves:
* Merged upgrade fixes
*** http://vis-7.poly.edu:8010
** Now has infinite loop test
*** currently linux distributions, also supports mac and windows
** loop test now correctly loops 5 times
** can set up schedulers to run tests upon git checkins
*** remap_module used wrong (current) package version when creating ModuleDescriptor
** need windows license: both 32-bit and 64-bit?
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])
** virtual machine with snow leopard
** virtualbox for virtual machines
** expand test coverage
** looking to test gui (mouse clicks, etc.)
** Qt has a testing framework?
* VisTrails on Mac OS X 10.8 (Mountain Lion)
** possible issue with X11 dependencies
** also, how long should we continue to build for 10.5 (Leopard)
* Bug #615: https://www.vistrails.org/ticket/615
** need new version of PyQt?
* [Rémi] working on Java spreadsheet
** have to write JTable that supports functionality like in Qt
** have basic UI in Jython, possible to use jpype
* [Fernando] burrtio and dtrace
** translate burrito script from systemtap to dtrace
** dtrace's dscript doesn't have control flow structures (only if-then-else, no for loops)
* Paris report [Emanuele]
** feature request: which parameters values are used in a cell
** overlay these parameters onto
** almost like vtk to build package based on xml files from galaxy


=== July 25, 2012 ===
'''Items to Discuss'''
* testing [Tommy]
* New bugfix release
** CTest
** Merge usagestats
*** only reports if test passes or fails
** Need to finalize survey
** CDash
*** http://vis-7.poly.edu/CDash/index.php?project=VisTrails&date=2012-07-20
*** set up cron jobs for the different virtual machines
*** Rémi: BuildBot -- coordinate virtual machines: http://trac.buildbot.net/wiki/ScreenShots
** coverage
*** http://vis-7.poly.edu/CDash/htmlcov_vistrails/
*** gives us coverage view of which lines are covered by the test suite
* abstraction bug: possibly triggered by upgrades
* [Fernando] burrito
** using DTrace on Mac, translating calls from SystemTap
** reading file, python programs read 64 files for a simple program
** need all files that are dependent on the program
** scripting virtual machines?
* setting up virtual machines
** Fernando and Tommy coordinate
* Rémi
** still working on spreadsheet -- need to talk to Huy
** issue with parameters updating: sometimes need to click on the version again
* Big Data and VisTrails
** meeting next week?


=== July 18, 2012 ===
=== April 13, 2016 ===
* Big data & vistrails
'''Updates'''
* Publishing bug
** hangs right after the spreadsheet comes up
** Ctrl-C on command-line causes process to finish and
* Java [Rémi]
** working on spreadsheet with jpype, cannot subclass java classes in python
* burrito: lower barrier to adoption for VisTrails
** can build workflow that uses information from burrito
** system tap: gets all process information, dtrace for mac os
* new machine for testing and releases
** virtual machines for linux distributions are setup
** create windows or mac?
* vis-7 machine
** upgraded to latest suse linux
** problem updating the crowdlabs machine, poly sysadmins working on
* github
** Tommy sent email on this
** goal is to have a public repository
** function to notify main developers if there is something to add
* WebGL [Wendel]
** fixed issue with upgrades
** workflow does not execute, still looking into this


=== July 11, 2012 ===
'''Items to Discuss'''
* Big data [Fernando, Huy]
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])
** change execution engine to allow streaming of data
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?
** probably need to change language for execution engine in C++
* Burrito [Fernando]
** if you have a program running and have process id, stores in db information about files being written, input
** mongoDB and python
** linux-based (possibly only fedora?)


* Jython and Jpype [Rémi]
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])
** Some Java functions (reflection) are not accessible and needed
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]
** Walks jar file to see which modules can be created; writes these to a cache file
** Cherry-pick test to preserve source?
** Package needs to load this file and auto-generate VisTrails modules
** Need maxPipelineFixAttempts option?
** Cannot build cache file in python right now, only jython
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?
*** Would need to be moved into handle_invalid_pipeline


* Publishing bug [Tommy]
=== April 6, 2016 ===
** Get two versions when running
'''Updates'''
** Two cases: when running VisTrails and not running VisTrails?
* [TE] Query View improvements
** Either executes or it hangs
** Done except for displaying "Version Info" and #1164
* [RR] Working on finishing new interpreter (finally!)


* WebGL [Wendel, Tommy]
'''Items to Discuss'''
** vis-7 issue: VTK 5.10
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])
** after uploading, get id already used message?
** Try latest upgrade first, then previous.
** queue from xmlrpc gets full and don't receive any more commands
** If all fail, select latest upgrade?
** Possible issue with upgrades (call change_selected_version instead of getPipeline)
** operating system on crowdlabs and vis-7 is getting old, need to upgrade these?


* Workflow cache replacement policies? [UV-CDAT, Dave]
=== March 30, 2016 ===
** measuring memory usage of python usage
'''Updates'''
** pympler: http://pypi.python.org/pypi/Pympler/


* Installing binaries or into standard python
'''Items to Discuss'''


* Testing
* [TE] Query View improvements
** build machine has arrived
** Unified do_version_switch and get_upgrades as validate_version()
** how to run multiple tests
** Problems with query view using its own controller
** need to have some type of protocol
*** Fixed module info not being displayed for query results
** create tests for new functionality
*** Still no version info for query results
** also establish governance
*** Still an issue with version query view not being updated


=== July 5, 2012 ===
=== March 23, 2016 ===
'''Updates'''


* Java dvelopment [Rémi]
'''Items to Discuss'''
** Weka package can be run in both jython and python via jpype
* [RR] Usage reporting almost ready to go
** Jython error when changing parameters (changes don't seem to happen, and trying this a second time causes db_id exception)
** Need to record whether specific features are used: paramexpore, mashup, query
** Instructions to run Jython version?
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation
** Is it possible to write a java spreadhseet that works in the python version?
** How to record size of vistrail?
* Parameter Exploration Changes [Tommy]
*** [DK] IdScope already walks over vistrail, use that
** Added apply to other versions
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines
** Drag and drop to apply exploration to another version
** Similar methods to apply analogies and mashups? [Add to trac]
** Also save analogies
** Apply to other versions works by module id so it won't
** Reference by tag or by unique id (resilient with unique ids)
** Visibility controlled by when there are tags
* Move to github and governance
** github: can create organization
** others can fork repository, we can decide what to include or not
* Survey users -- what are people using VisTrails for?
** look for old survey?
** what information are people willing to share? how to encourage?
* VisTrails users' and developers' day
* Status on crowd labs (Tommy will contact Wendel)
* Big data: Huy, Fernando, Tommy, Juliana need to discuss


=== June 27, 2012 ===
=== March 16, 2016 ===
'''Updates'''


* Annotations for general data identification in provenance?
'''Items to Discuss'''
* IPAW and DataONE Report
* [TE] Upgrades in Query Mode
* Parameter Exploration Interface [Tommy]:
** Now using code similar to do_version_switch
** save all parameter explorations for each version
*** Uses handle_invalid_pipeline
** adding more metadata to the panel (user, date, exploration name)
*** Differences: Not using current_pipeline, flushes changes
* LaTeX Issue
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)
** Emanuele cannot reproduce the issue Juliana encountered
*** Integrate to avoid repeated code?
** Used CLTools with multiple configurations
*** Use this for query mode?
** Tommy fixed some process blocking bug, perhaps this fixed it?
* [TE] VisTrails 2.2.4 ready for release?
* Port Documentation update from Pasteur folks
** tej package updated. Others?
** Always in center
* [JF] upgrades and the version tree
* Rémi working on Weka package (try to use jpype here?)
* [JF] provenance in Jupyter
* Possible issue with core code in java branch --- validating pipeline and/or adding functions?
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]
* exporting VisTrails trace for DataONE working group


=== June 20, 2012 ===
=== March 9, 2016 ===
'''Updates'''


* Move to github and governance
'''Items to Discuss'''
* Survey users -- what are people using VisTrails for?
* [TE] Fixing search mode
* VisTrails users' and developers' day
** String query now matches all by default
* Status on crowd labs (Tommy will contact Wendel)
*** This caused pipeline queries to never match
* Big data: Huy, Fernando, Tommy, Juliana need to discuss
** String query can search: User/Notes/Tagname/Time/Module name
* Parameter exploration: apply to different versions
*** Should we search module names by default? (Disabled by default because it is expensive)
*** Fixed notes search
** '>' parameter queries in documentation but has been removed
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)
*** Upgrades still a problem
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)
*** [JF] Allow search on past tags


=== June 13, 2012 ===
* [TE] Create new bugfix release
** Finish querying
** Path issue on windows


* Tommy:
* [DK] Look into sqlalchemy and new bundle branches
** Add support for parameter exploration on the command line---provide the user the ability to invoke a named parameter exploration
** Add support for parameter exploration from the API: this will be for users that need to customize the exploration
* Remi:
** Ironing out some remaining bugs on the Weka wrapping
* Emanuele:
** Working on UVCDAT improvements: ability to replace a variable
** Need to look into the issue of guiding users on what variables are needed to run a pipeline
** Will look into the latex package issue, where the workflow is not executed (maybe an issue with CLTools or Persistence Package)
* David:
** global variables: can be used in the same workflow in multiple places; similar to having multiple aliases that are synched
** aliases: only set for one parameter
** Should we combine aliases and global vars?
** Currently we cannot do parameter exploration over a global variable


=== June 6, 2012 ===
=== March 2, 2016 ===
'''Updates'''
* [TE] Fixed cache bug with port specs
* [TE] VisTrails Next
** Most packages now work
** No qgis for Python 3


* Tommy: Almost finished with the parameter exploration updates
'''Items to Discuss'''
** Parameter explorations show up in project list now
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]
** Can use functions that have not been set in parameter explorations
** Also working on serialization of parameter explorations to their own schema
* Wendel
** Changed combo box style for HTML5 medleys
** Using XMLRPC
** Linux python 2.7.1 issue with cElementTree?
* Emanuele: API 1.7 vs. 2.0 returning execution results?
* Fernando: git-annex demo
** looks to satisfy the main requirements from Matthias
** Windows haskell issues...
* Rémi
** automatically wrapping Weka package
** building Weka workflows
** use Fernando's simple workflows to test this package
** jars: configure button not available before loading a package?  check this?
* Dave
** updates for USGS, others
** basemap example
** can we package basemap without full resolution maps?


=== May 23, 2012 ===
=== February 23, 2016 ===
* Review remaining bugs
'''Updates'''
* Issue with multiple instances
* [TE] Fixed PythonSource 100% CPU bug
* Wendel: almost done with mashups in HTML5, trying to make work on different browsers
* [TE] VisTrails Next
** Emanuele will send Wendel examples for web and desktop
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples
* Juliana's issues:
*** Now uses REST service, tabledata, and Google Maps
** analogies are only displayed if we change the focus out of VisTrails to another application and back
*** Web Service replaced with DownloadFile
** in publish window, the snippet appears grayed even if the vistrail is saved -- but if the focus is changed to the main window and back, it works
** it seems there are some issues with focus in general...
** can't run 2 instances of vistrails: Juliana managed to break Fernando's setup too
** need better viewer for persistence package: as is, it is not possible to look at values---tedious to change the length of the fields
** persistence: should allow labels in module config--can be hard to identify a file in the persistence manager
** latex extension: clicking on a figure on the PDF file (on acrobat) gives an error (is this related to the change to relative paths?)
** CLTools: should support other types, e.g., integer
** CLTools: sometimes, in the module list, under CLTools, a module called CLTools is displayed


=== May 16, 2012 ===
'''Items to Discuss'''
* Review remaining bugs
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.
** Default values for parameters not showing
** New 2.2 release soon?
* CLTools: allow it to be invoked from within VisTrails (Tommy will look into this)
* [RR] Anonymous usage reports
** Emanuele created a command that makes it easier to invoke it on MacOS
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]
* Remi will demo the Java version of VisTrails and lead a discussion on design issues
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?
* Investigate connection to iRods
* noWorkflow - provenance for python scripts
* Mashups on iPad (ask Wendel to look into this)
** Recent work on tracking dependencies
** Need to translate the XML spec for the mashups into HTML5 (should be similar to what Wendel did for the automatically generated mashups he had)
* Also integrates ReproZip with Jupyter
** [http://mybinder.org mybinder]
*** Use with ReproZip to automatically build Jupyter notebook interfaces
* Daves comments on Jupyter integration
** It is possible to identify cell order on the server side
** Not trivial to track order of cells on client side
* How does Galaxy workflows relate to VisTrails?
** Packaging in Galaxy is hard
** Galaxy for biomedical workflows, difficulty making it general
* Action items
** How difficult to track cell dependencies in Jupyter notebooks?
** Use noWorkflow to track cell dependencies?
** How difficult to modify mybinder to use ReproZip packages?


=== February 17, 2016 ===
'''Updates'''


=== May 11, 2012 ===
'''Items to Discuss'''
* Fix for issue with caching group modules
* [TE] Fixed Group memory usage
** looks to be working ok
** Modules in groups are supposed to be freed when done executing
* Workspace fixes
*** Prevented by logger that stored references to the modules (fixed, now using id(module))
** seem to be working ok
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)
* QPixmap:scaled error?
* [TE] VisTrails Next
* detaching panels issue? Cannot reproduce
** Removed deprecated code
* Ticket #551: Make Preferences non-modal?
*** Core imports that should be gui imports
* Ticket #539
*** GetInputFromPort etc.
* Ticket #540: Tommy will check
**** PythonSources will be upgraded
* Ticket #541
*** Fixes for Python < 3
* Ticket #517
*** fixes for VTK < 6
* Ticket #523: Dave with check
** TODOs
* Ticket #532
*** Remove RemoteQ
* Ticket #533: Emanuele
**** Port Hadoop to tej (Already have pbs)
*** Merge package wrapper
*** Finish SQLAlchemy branch
*** Finish new bundle branch
** [JF] Will contact Jeff and Matthias about moving to Python 3
* [JF] Query by Example is broken
* [JF] VisTrails Data Cleaning Project
** Use VisTrails to process and explore task-specific data while keeping provenance.
** Remi's Idea
*** Use VisTrails as the backend in a Jupyter notebook
*** Overcome problems with executing cells in random order
**** Interesting research topic, but little interest has yet been noted
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)
*** Impose restrictions to notebook cell boundaries
*** It will only work for functional operations
*** Can we fix, or detect, non-functional (mutating) operations?
*** Analyse code and data flow and see if it mutates data in unexpected ways?
*** Mutable operations can be created in many ways in Python
* Next meeting Tuesday 11:30


=== May 2, 2012 ===
=== February 10, 2016 ===
* Performance issues with workspace reconstruction
'''Updates'''
** should be fixed, need to close ticket
* Streamlining build process
** Set up virtual machines?
** Do we need to buy a dedicated server for this?
*** [ES] It could be a desktop machine, with windows and mac installed. I don't think it needs to be a dedicated machine, but we would like it to be available when a binary is built. I think it could be a mac with windows installed as a virtual machine. A 64-bit Windows has to be installed. I believe Lion allows multiple users logged in the graphical system at the same time, so if it is a powerful machine, it can be used as the development machine for another user. Also, we need to make sure the vnc port is open to the outside so a remote user can connect to it.
** Running multiple Mac versions?
*** [ES] I am not sure if a binary built on Lion will run on Snow Leopard. The contrary is true, but then if we could run snow leopard as a virtual machine would be fine. It seems that Parallels only allows running the server version of Snow Leopard as a virtual machine. If we want to create a virtual machine of Snow Leopard, we need to do some hacking (some users were able to do this).


* File management
'''Items to Discuss'''
** Does git-annex work with cygwin?
* VisTrails Next
** Fernando contacted author of this code, he pointed to the web page on windows requirements, thinks symbolic links are the issue
** Test suite passes
* Preferences Dialog is modal and does not allow switch to error messages.
*** Python 3 less deterministic, module upgrade order random.
** Make preferences non-modal?
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...
** May already be working non-modal...
** All of our packages should run fine :)
** Move "Module Packages" from Preferences to "Tools" window?
** VTK thumbnail comparison works
* Package identifier checks
*** Rendering bug on some systems
** move the specific code for SUDS packages checks to the package itself
**** QVTKCell now inherits QGLWidget
** add hooks to allow packages to identify which identifiers they can load
*** Old thumbnails had wrong height due to spreadsheetwindow being too small
* Enumerations
**** Fixed by regenerating thumbnails
** mashups already has a widget to select from a list of existing constant values (general, not just a combo box)
*** Silenced deprecation messages when generating modules
* Auto connect
*** Many new classes in VTK 7
** highlight possible ports to give the user the idea which ports are possible
** Remove deprecated code
** Check names of ports to try to make these connections
*** Core imports that should be gui imports
** Issue: when there are multiple inputs that match multiple outputs (which connection to create)
*** GetInputFromPort etc.
**** Will break existing PythonSources
**** Try to upgrade?
*** Old-style packages
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
***** Read info.py without importing package?
*** Fixes for Python < 3
*** Fixes for Qt < 5
*** fixes for VTK < 7
*** Remove self port?


=== April 25, 2012 ===
=== February 3, 2016 ===
* Persistent Package
'''Updates'''
** File management: discuss git-annex
*** Issue with Windows support
*** May work through cygwin, but written in Haskell which may lead to issues of integration in the Windows environment
** Make inputs of persistent modules read-only, to maintain consistency (only configure them through the configuration widget)?
** Reproducibility of workflows that have persistent inputs: get the version id from the log, put it in PersistentRef and execute the workflow?
* Parameter exploration
** Do we want to add named parameter explorations and multiple explorations per version? (Tommy)


=== April 20, 2012 ===
'''Items to Discuss'''
* File management:
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]
** USGS suggestion: use just files in the file system; more concerned about performance
** Git Annex may provide a good solution: http://git-annex.branchable.com/ "Fernando will look into days"
* Wendel: update on Elsevier/Sciverse/Crowdlabs
** crowdlabs now uses the proper identification for workflows
** Assumption: author has already uploaded the vt to crowdlabs, and the paper refers to the workflow id, then our app would load the associated mashup and allow the user to manipulate the mashup on the browser
** Bug: random message from VisTrails--interrupt system call "Wendel will ask Emanuele for help"
** Wendel will add all vtk_examples to crowdlabs
* Dave: update on additional improvements (e.g., global variables)
** Moved code from pipeline to controller
** The interaction needs to be improved so that if a var is removed, when the user clicks on a workflow, the var is automatically resolved (deleted from the workflow)
** Update Huong's example to use global variables
* Discuss changes to persistence package (see email exchange)
** Also more from USGS on persistence and large files
* Parallel execution
**python paralellism: for modules that serialize their output, they  can be executed in different cores
** allow modules to say they are detachable--they have  implement methods for reading the input and serializing output
** ask Matthias for Troels' code--USGS could test it!
* Possible issue in Windows version where you need to press space bar in the console before workflows execute? (USGS)
** We need to have a Windows tester (Tommy can test this)
* Defaults are not showing in the new ports panel?
* Change parameter exploration to allow all parameters to be shown
** Make unset parameters accessible from a Parameter Exploration view (Tommy)
* Enumerations (Pasteur group)
** Expand portspec to have subitems to better support this?
* Annoying bug where selecting a version selects the text box instead of the version ellipse


=== April 11, 2012 ===
* VisTrails Next
* Wendel gave update on Crowdlabs/WebGL
** All tests but thumbnail comparison passes
* Update on persistence package and the need to support a global repository
** Still issues with gui and more packages needs to be tested
** Juliana followed up with Matthias (see email exchange)
** PythonSource and MplSource upgraded with lib2to3
* Schema updates going smoothly
** Keep contrib packages? Update?
* Dave made several improvements to module attributes to handle different 'symbols' for the ports, ability to mark ports as optional, visible, etc
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]
** discussed usability issues, including the recording of which ports are marked as visible
*** Supports Python 3
*** Will hopefully get into conda and others soon
*** Deprecates a bunch of modules


=== April 4, 2012 ===
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next
* Fernando will look into the persistent file management
** Runs!
** [[Persistence Package]] notes and improvements
** Uses conda
** We need to investigate GIT's facilities for managing metadata: http://schacon.github.com/git/git-notes.html
*** Many levels of requirements
** To handle the metadata, one approach is to keep the local SQLite, and pull the data from GIT
**** apt packages
* Updates to schema
**** conda packages
** Add mashups to schema (to better support crowdlabs)
**** python packages in requirements.txt
** Other changes? best to make all needed changes at the same time
**** python packages in setup.py
* Bug-fix release
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!
* Crowdlabs
*** But useful if we want to use conda in our own build process.
** Get rid of flash
** Lots of errors I do not get locally?


=== Mar 21, 2012 ===
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]
* Need to remind people at Poly about the meeting
** Have you used any alternatives to vistrails?
* Tommy will send an email the day before our meetings
*** Relevant? More options?
* Integer slider for Matthias (need to translate)
** How often do you use any of these particular features in VisTrails?
* Schema changes
*** Add/Remove alternatives?
** How important would you rank the following potential improvements to VisTrails?
*** Add alternatives?


=== Mar 14, 2012 ===
=== January 27, 2016 ===
'''Updates'''
* New interpreter
** Working prototype soon
** Looping not completely done (and port combination mode: pairwise/cartesian...)
** Need to integrate with VisTrails
*** Compatibility with existing modules
*** Use log controller
*** Want to cache at multiple levels (whole stream/single element of stream)
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)


* Mashups in the database
'''Items to Discuss'''
** Matthias's error: is there an upgrade here?
* User Survey
** Mashups are not currently being saved in the database, Emanuele is fixing this


* Schema changes:
* ZMQ failing on windows (vistrails-users)
** Mashup changes
** Ticket on trac about schema changes
** Global variables: promote this to specified schema, not just annotations
** Port cardinality: single input or multiple input guidance
** What else should be updated in the schema?


* ClimatePipes
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]
** Demo
** Testsuite almost passes
 
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.
=== Mar 7, 2012 ===
 
* Proposal to change the interpreter to support suspended execution (Tommy will look into this and then coordinate with Troels)
** any module should be allowed to have its execution suspended --- we will have a new execution state
** modules that are declared by the developer as suspendable will have a different shape (oval?)
** during execution, suspended modules are shown in a different color (orange?)
** VisTrails should execute all other modules that do not depend on the suspended modules
** the suspension of a module should generate a log entry that states the module is suspended
** Question: how often should VisTrails polls the suspended process? Should we have a 'resume' button in the interface to allow the user to also control this manually?
 
* Bug with updates (Dave will look into this)
* export to stable menu  does not work, and we need a export to XML (Emanuele will add these to trac)
* Crowdlabs and upgrades: Tommy and Wendel will work on this
* Emanuele is implementing support to export a single workflow. Caveat: the workflow will be detached from its original, and it will not be possible to 'update' it on Crowdlabs (this will be documented in the manual)
* Oracle interface: add dialog error msg when 'reproduce' is not possible; differentiate between execution and 'reproduce' (use italics for reproduce, and keep the existing colors)
 
 
 
 
* Matthias says: Troels has now found the time to finish his Python background job submission package, which can run calculations in the background locally, on remote Unix machines, and on remote clusters and supercomputers. We are integrating it into ALPS now and next want to do a VisTrails package for it. That should not be too hard, but we'll need some changes in VisTrails to be able to implement it well. If a background job starts we want to stop the execution of the downstream part of the workflow and can do that by raising an exception - but this exception should be treated differently to others. In particular we don't want to have the execution log record it as a failed execution, but maybe as "suspended" or "incomplete" or similar. One might also use another color than red to mark the status of the module and the downstream modules. Finally there might still be other branches of the workflow that could be executed and one does not have to stop all execution.
 
=== Feb 21, 2012 ===
 
* Wendel: 3D crowdlabs -- found some issues that have been fixed
** TODO: create a function to add a package and reload a package
** TODO: add a function to automatically modify a workflow to produce WebGL results
 
* Tommy: Crowdlabs
** allow users to upload packages -- similar to safe workflows
 
* Fernando: VisTrails/Oracle
** TODO: save db state ID in the log, associated with a workflow execution
** Oracle package supports all functionality of the MySQL package
** already supports 'reproduce'
 
* Emanuele:
** Updated VisTrails doc to include info about server setup
** working on the release
** hard to use VisTrails as a server because of the dependence on X server; when one does not need the GUI, it would be nice to transform the workflow so that it could run in batch, for example, use VTK offscreen support
** caveat: requires VTK to be compiled with offscreen support
** it is not clear if this would work for matplotlib, which also requires X server
** http://stackoverflow.com/questions/4931376/generating-matplotlib-graphs-without-a-running-x-server
 
=== Feb 14, 2012 ===
 
To Do:
* new branch in crowdlabs for Wendel
* update doc
* Oracle package
* release 2.0 beta
* Future tasks: packaging, testing
 
* Update on crowdlabs (Wendel and Tommy)
** Wendel had problems with black images being returned by the VisTrails server, we suspect the issue is related to the virtual frame buffer---he will investigate
** We need a script to migrate old crowdlabs information into the new format
** There is a new GIT repo for crowdlabs, currently Emanuele, Tommy, Juliana and Wendel are on the notification list
** Tommy is almost done with the update, need to clean up documentation
 
* VisTrails + Oracle Total Recall (Fernando)
** Overview of total recall
** Having the ability to execute and reproduce
** To reproduce go through the log; only works for read-only queries
 
* UVCDAT
** Done, except for icons
 
* Tighter integration with scripting
** require the modules to support serialization into Python
** not clear if can use compute method as it; python serialization is simpler than compute
** need to better investigate this -- ask Dave about his implementation for the VTK scripts
** in UVCDAT there is translation only from pipeline to script, not the other way around
 
* Tasks for new guys (Tommy):
** https://www.vistrails.org/ticket/525
** https://www.vistrails.org/ticket/526
 
=== Jan 31, 2012 ===
 
* Climate Tools and Provenance
* Server:
** Save workflows to the server database
** Keep the parameter changes
** Add annotation that says when a version is WebGL enabled
** Just send actions that are the changes that are being made in WebGL version
 
 
=== Jan 24, 2012 ===
 
* TE: crowdLabs is running, able to upload a file from VisTrails
** still debugging, found transaction error when writing to database maybe order
* defaults:
** problem is that we currently are not showing default values for modules in the port panel
** don't know if we should clutter the interface with all defaults, should we save them as provenance?
** set all defaults when a user drags in a module, this way we have provenance
** hide defaults in the interface--if user clicks on a port, show it, but don't automatically expand it
* error handling:returns values and exceptions
** need to track down bug from Colin's workflow
 
=== Jan 20, 2012 ===
 
* Updating crowdlabs install
* calling VisTrails from command-line using "python vistrails.py"
* want to change this to use the VisTrails sever
* create a new page, option to view visualization in 3D
* send GET request to Wendel's 3D stuff
* running on git version of VisTrails (so the code is up-to-date)
* server currently has an issue with the change to core_no_gui code
* crowdlabs, one change to have "View Vistrail in 3D" link
* one new parameter to settings (ip address)
* Tommy has crowdlabs code running with new versions of Django
* Wendel's code is on the vgc git repo
* Branch a new version for crowdlabs
 
=== January 10, 2012 ===
 
* need to support mandatory vs. optional attributes
* find the status of remote execution
* Dave will reply to question from QGIS developer
* Emanuele has sent information to LLNL
* To do for VisTrails
** test server version for CrowdLabs
** revisit CrowdLabs--issue: Django has evolved, many libraries no longer exist
 
== 2011 ==
 
=== Dec 13, 2011 ===
 
* Change time to 11am
 
* Derek's suggestion for "Writing VisTrails Packages"
** Split old/new way of writing modules?
** Do we still support having a package as a .py file?
** Should we document adding custom configuration screens?
 
=== Dec 6, 2011 ===
 
* Test suite
** our current test suite is not comprehensive
** add tests for the core that was separated from the GUI, e.g., calling update functions, ...
** focus on accuracy and reliability of the provenance that is saved
** will not create scripts for the actual GUI right now
** Action items: iterate through the directories and add the missing tests--start from controller (should document functions as the tests are created) --- start on this after UVCDAT is done
 
* UVCDAT
 
=== Nov 29, 2011 ===
 
* UVCDAT
 
=== Nov 8, 2011 ===
 
* Command-line flag fixes:
** detached history view done
** check multiheads for spreadsheet---spreadsheet not shown on startup now
 
* Merge core_no_gui branch with master
** Check unit tests here
** Can we merge this into master, or keep as a separate branch?
 
* UV-CDAT
** Call last Friday to talk about Variables
** Dave looking at Variables
** Tommy looking at workspace for UV-CDAT application
** Emanuele managing integration and gui development
** Keep track of things on new [[UV-CDAT Development]] page
** Examples: Tom's examples
** Avoid adding complications right now
** Persistence for variables: simplfy and have a checkbox to allow system to save those things
** Scenario from Jerry Potter?  Juliana working on
** Plot: graphics method, template, plot, variable (list)
** Templates for subworkflows?  Define all the subworkflows that must respect this interface
** Dave: write a Variable module
*** Emanuele: send Dave CDAT python examples, one vt in paraview package
 
=== Oct 18, 2011 ===
* We will focus on UV-CDAT and on completing the alpha version of the system. Please read the documents Emanuele emailed.
 
* Using different servers with CrowdLabs
 
* Closing tickets (Tommy)
 
=== Oct 11, 2011 ===
 
* Bug reports:
** execute_cmdline issue: CLTools supports both
 
* 2.0 bug fixes: push new release soon (for change_selected_version bug fix)
 
* Fix migrateTags issue and any other minor bugs, then release new version of 2.0
 
* CLTools:
** latest is in repository
** README in package directory
** adding to online user's/developer's guide
 
=== Oct 4, 2011 ===
 
* We will discuss issues related to remote execution of VisTrails modules as well as multi-threaded and parallel execution of workflows
* We will be joined by Matthias Troyer and Troels Frimodt Rønnow (ETH Zurich), Joel Daniels (NYU Poly), Marta Mattoso, Jonas Dias (UFRJ, Brazil)
* Terence Van Zyl and Derek Hohls (CSIR, South Africa) will not be able to make it today
 
We will develop a package that supports the ability to spawn a 'batch' job either on a local machine, a remote server, or a specialized cluster/supercomputer (with its own  batch submission system). The basic API will provide functions to:
* start jobs and obtain id (which will be saved in VisTrails and in UFRJ's database)
* poll the status of a job (if it is still running of it is completed)
* copy files to and from the remote server
 
Once a job is submitted by the 'remote module', the workflow stops. The user can poll the status through the UFRJ DB, or by running the workflow again on VisTrails. When the workflow is re-run, VisTrails will check the status of the job; if the job is still running, the workflow stops; if the job is complete, the files (if any) will be transferred and the execution of the workflow resumes.
 
Next steps:
* Troels will work with Joel on extend Joel's package to implement the basic API; and with Dave on implementing the idea of  'persistent jobs'
* Once the basic API is ready, we will work with Jonas to integrate his DB interface and the ability to access the provenance information about the remote jobs from multiple devices
 
 
* Tommy gave an update on CLTools. He already has a GUI to create the wrappers. He will make some additional changes and we will get some people to try it out.
 
=== September 27, 2011 ===
* Stéphane Descorps-Declere and Loic Couderc from [http://www.pasteur.fr/ip/easysite/pasteur/en Institut Pasteur] will join us
** Show demo of their package
** Discuss their specific needs
** How to deal with tools that have a large number of parameters and help the users not get lost setting these parameters, some ideas:
*** Port documentation in tooltips so mouse-over shows the documentation---need to support the specification of port-level documentation in the wrapper
*** Defining in the wrapper required and dependent ports. For the latter, define the dependency graph among the different parameters, and then use this information in the GUI. For example, given parameters A, B, C and dependencies C->B, B->A, and if A is defined, the GUI should require that B and C be set
*** Define in the wrapper port cardinality, i.e., the number of connections supported by the port
*** Support HTML-style documentation, which would allow the inclusion of images---the problem here is that it isn't good for command-line based use of VisTrails, so we also need to support plain text
** Stephane and his group will take a stab at implementing these ideas in two steps: first, making the required changes to the registry; and then, updating the GUI to use the new information that will be available in the wrapper.
 
* Discuss CLTools features/design
** Tommy has an initial function that parses man pages
** He will develop a GUI to help users wrap command-line tools---this will be a tool "separate" from VisTrails
** How to handle pre- and post-conditions? Could create scripts and wrap them using CLTools; or in VisTrails, use Python source
** reuse code from existing PythonSource GUI?  Maybe for pre/post-conditions, but the port list probably isn't a great interface for this.
 
* Issue with change_selected_version
** the graph is probably being corrupted by some version not being added to the version tree, could be a single action type
** Asked Tom if he knows steps to reproduce
 
=== September 20, 2011 ===
 
* Tommy will give a demo of CLTools package for wrapping command line tools
** Each tool creates a module, but modules cannot be versioned?
** Do you have specific tools/scripts I can try?
** Next step is to create more automatic wizards for tools and scripts.
 
=== September 13, 2011 ===
 
* Making it easier to wrap command line tools, motivated by the need to simplify the creation of executable papers
** Goal: build a wizard that guides a user on the creation of a wrapper.
** Assumption: the user has code that derived results for paper, and needs to package it in a way that reviewers can both reproduce and validate the results (e.g., try different parameter values)
** Initially, require the user to manually provide the information---parameter names and optionally, types.
** The output of the wizard should be a machine-readable specification which could be loaded as a module in VisTrails
** Some desirable functionality:
***ability to 'test' the wrapper; after the specification is input, show a vismashup-like interface for the user to set values, and allow him/her to execute the module
** provide 'smart' parsing to help in the wrapper construction, for example, extract the information from a python script
 
* We will create a separate branch for external developers, so that they can contribute patches to VisTrails
 
* David has cleaned up most of the GUI dependencies, the only remaining ones are:
** Notes associated with versions  (they need Qt)
** Generation of PDF representation for workflows and trees
** Thumbnails
 
 
* Web services (EMBOSS example)
** All now works except Prophet example (<choice> bug in SUDS). Should we remove it?
 
* Invalid modules (those in red) will not show parameters - Fixed
 
=== September 7, 2011 ===
 
* Query mode
** need to finish this implementation
 
* Analogy parent() issue?
** check what this was (form August 16)
 
* Web services (EMBOSS example)
** changes with SUDS (should be easier)
 
* VTK Examples:
** check old examples (don't seem to work with new version)
** these were auto-generated from VTK python examples a while ago and may have issues with newer VTK versions
 
* Auto-updating tags: migrateTags option in preferences
** Document this
** MIgrate notes as well so that the notes and tags are migrated
 
*
 
=== August 31, 2011 ===
* Diff tab has other view modes disabled.
** Opening a version or mashup from workspace, open in new tab instead of overwriting diff
 
* Binaries
** Matthias reported his tutorials were working with the current version
** Emanuele will start rebuilding the binaries more frequently again
 
* Mashup saving
** keep does not save
*** change the name keep to tag, to avoid confusion
** changes in workflows and mashups are reflected in the workspace now
*** still working on the executions
 
* Provenance browser
** instead of using version numbers for untagged versions, use the same convention used in pipeline view (eg., parent_tag + 1)
 
* Documentation
** VTK?
** At least a section in the user's guide  that points the users to the right examples
 
* Examples
** problem with examples that depend on 3rd-party packages
*** move them to the package folder?
 
=== August 16, 2011 ===
 
* Diff tab should have other view modes disabled.
** Tab doesn't make sense to switch back to history
** Could either try to make mode buttons work or just disable the modes
** Easier to disable the other modes
** If open a version or mashup from workspace, open in new tab instead of overwriting diff
** this way, a diff view is only closed by a user explicitly closing it
 
* Binaries for Mac
** Can build 64-bit binary and not 32-bit
** Emanuele will try to build 32-bit again
** ALPS has separate 32-bit and 64-bit versions
** Have two DLLs and registry for 2008 and 2010 Visual Studio
** Problem is that VisTrails may not run if the registry does not have the updates from Visual Studio
** 64-bit support allows more than 4GB of memory
 
* Should Mashups or Workflows be children if the vistrail in the workspace?
** Expand them automatically
** Mashups are meant to be shortcuts
 
* Mashup saving
** keep does not save (workspace is not updated)
** can we emit signal to workspace so it is updated
** can we put things in memory for the workspace?
** set_changed is called so we should update the workspace when a vistrail to be changed
 
* Subworkflows
** new changes to infrastructure to address distributed version control
** issue where multiple versions of the same subworkflow are listed
** multiple edits still an issue
 
* Analogies
** issue with parent() bug, Dave needs to file this
 
* Databases
** no direct database access currently allowed at NYU
** either create localhost database or use via crowdlabs
 
=== August 9, 2011 ===
 
* Ticket #459
** for an empty pipeline, the exploration is not resetting correctly
 
* Dave looking into query issues
 
* Issue with views not being set up correctly and handling callbacks (#469) amy also play a role in #473.
 
* Ticket #472.  Setting values in the user-defined list or any other type of window
** Going to any other window causes everything to reset, after executing, focus out event causes things to reset
 
* Subworkflows: edit bugs, Claurissa
 
* Groups inside of groups: currently cannot view them, Claurissa was looking at this
 
* Cmd-Q when the spreadsheet is on top seems to cause problems, need to force quit on the Mac.
 
* Enhancement: spreadsheet camera reset button?
 
* Cmd-E for configuring ports, Ticket #443 fix the old configuration windows or do away with, maybe keep this
 
* Parameter exploration: link spreadsheet to the parameter explorations, or display the parameters used with each cell, way to write
 
* Missing example for vtk.vt for cell locations and sheet references.
 
* Diff when upgraded doesn't work
 
* Upgrade entire version tree? How do we preserve tree structure?
 
* Mashups: display with pipelines? having both expanded to solve the extra click
** add this as a preference, other ways to view, add a button
 
* Opening a lot of vistrail files, closing them without saving, have "Yes to All", "No to All" buttons.
 
=== August 2, 2011 ===
* Ticket [https://www.vistrails.org/ticket/459 #459] (Tommy)
** Pipeline inspector may not get updated correctly when pipeline is empty
 
=== July 26, 2011 ===
 
* Zoom level in the first vistrail bug
** file on trac
** start vistrails and go to version view, history tree is not there, Cmd+R shows root node
** all vistrails after work fine
 
* Subworkflows
** issue with adding input ports and naming
** separate issue with editing a subworkflow more than once
 
* Analogies
** how to update the algorithm so the terminator with HTTPFile works correctly
 
* LaTeX Assistant
** Dave's new changes (read tex file and find \vistrails tags) moved to 2.1, remove from 2.0 for now
** Add back as a new feature for 2.1
 
* ungrabMouse in setModuleLabel
** probably happens on any module
 
* query issue: want to get back to original query from the search results and modify the query, currently just removes the query and resets everything
 
* keep triangle menu initially in 2.0, may remove in later versions if we can maintain access to all features
 
=== July 19, 2011 ===
* Features added
** Display mashups in the workspace
** Detach configuration window, rename, in a read-only, non-changing state (allow copy)
*** Improve button layout?
*** move read-only to left side and other buttons to be right side
* Status of 2.0
** subworkflows bug, claurissa
* [Dave] Need to fIx bug with retagging
 
=== July <del>12</del> 13, 2011 ===
 
* Features added
** Display mashups in the workspace
*** Tommy added the support for the workspace, items need to be added
** Detach configuration window, rename, in a read-only, non-changing state (allow copy)
* Menu/Key shortcuts issues
** QBaseView window with menubar, add parts of menu
** QVistrailView inherits so we can reuse part of menus
** Need to show menus on Windows and Linux for other windows
** PythonSource has no menubar by default, can create its own menu
* Show Pipeline for a group
** delegate the creation of this window to QVistrailView?
** windows are set on show_group_function
** don't create a separate window
** eventually we should make groups logical not physical (so everything doesn't change each time to modify a group)
** just capture new pipeline
 
* spreadsheet editing and analogies
** add static methods to controller to allow general access to create methods
** need to factor code so virtual_cell can work with this code, too
 
* look into the issue when seeing a workflow for the first time; might be linked to upgrades and indexing
** Tommy should check that the exceptions are propograted
 
=== July 5, 2011 ===
* Server migration is almost completed
** GIT/Wiki/Trac working fine. Machine will be relocated to get better bandwidth
** <del>Still need to update vistrails.org to point to new location</del>
 
* Some bugs from last week were fixed, still a few more to go (see trac tickets)
* Minor issues raised at the meeting
** Add shortcut key for executing a pipeline
*** [ES] Ctrl+Return (Command+Return on a Mac) should work. The problem is that the shortcut does not show up on the Menu on a Mac. I've tested on QtDesigner and it doesn't show up there either. So it seems an issue with Macs.
** Being able to execute the selected node in Version View (having to open a pipeline just to execute it involve too many steps)
* Features to be implemented, then VisTrails will be feature frozen
** Display mashups in the workspace
** Detach configuration window, rename, in a read-only, non-changing state (allow copy)
 
=== June 28, 2011 ===
 
* svn offline this weekend to migrate servers (Huy)
** send an email to vistrails-dev
* fix bugs from demo
** fix zoom on open, change view
** detach configuration window, rename, in a read-only, non-changing state (allow copy)
** read-only controller for diff view, or use Emanuele's read-only view flag
** check for pruned versions in "(latest)" version
** failed to index error on open (possibly old version of vistrail--with filenames)
 
* releases?
** waiting for ALPS build for Mac OS X 10.5
** allow 2.0 vt files to open in 1.7 (e.g. those with mashups)
 
=== June 21, 2011 ===
 
* Querying
** need ability to better navigate results
** show query results in side panel?
** search all open vistrails (change label on the search)
 
* Updated workspace window
** Separate open and recent files
** Delete closes an open vistrail or removes a closed one from recent file list
** Also has an "open in new open window" option for right-click, suggested "open in new window", "open in new tab" for workflows
** Want to be able to show this palette at all times?  This allows better navigation, but problem is the screen space
** Try showing the workspace window in the upper left and "normal" left palettes in the lower-left.
 
* Module Palette
** Searching should expand things automatically, especially for searches that have only a few search results
** What about searching a particular package? Could auto-expand on begin search, user collapses and then expands the package they are intereseted in
 
* Global variables
** one bug on disconnecting variables
 
* Spreadsheet and analogy issue on re-executing pipelines? Dave will look at
 
* Diff
** Diff properties not showing up automatically, Emanuele will check
** Implement a diff in workspace window so you can do cross-vistrail diff
** Diff uses version tree for efficiency, but we can use analogy match or heuristic even when no actions are shared
 
* Inspector window: would be nice to keep this window smaller (certain tabs keep it large)
 
=== June 14, 2011 ===
 
* Detach windows
** notifications?
** creating secondary windows
** don't duplicate the entire vistrails window, just individual views
** detach icon not displaying correctly in Windows
 
* Global Variables
** How to add to modules?
** Drag constant modules to the vistrails variables panel
** Set the value by editing the constant in the vistrals variables panel
** Connect the value by dragging the constant from the vistrails variables panel to a port in the pipeline
** Update default text to "Drag constants from Modules panel to create variables"
** Fix bugs with connecting globals and deleting them (versionTab issue, etc.)
 
* Query tab in vistrail
** Can we use normal version/pipeline views as result views?
*** Which view to start with? history/pipeline
*** Show the workspace panel instead of history view
*** Show results in the query panel (as pipeline or workspace as appropriate)
** Should we always combine "keyword" and "visual" queries?
*** try to combine queries when possible, but allow user to search just text if there is no pipeline or just pipeline if there is no text
*** get back to the search window after seeing results (need a button here)
** What interactivity do we want? Open pipeline when matching version is double-clicked?
*** yes, single-click shows in results view, double-click opens as normal vistrail
** Dave will look at query view
 
* Add read-only PythonSource
 
* Workspace: keep closed vistrails here too
** try to order so that open vistrails are on top (highlighted in bold?)
** also show different views here (full tree view)
** show current workflow (ie latest version in the panel), too
 
=== June 7, 2011 ===
 
* Status of 2.0
 
=== May 31, 2011 ===
 
* Go over bugs and new features for 2.0
 
=== May 17, 2011 ===
 
* Add comments, bugs, critiques, etc. about the 2.0 changes [[Version 2 | here]]
 
* Global variables
** Check issue on MacOS
** Add documentation
 
* Mashups [Emanuele]
** on track, separate feedback
** switching vistrails issue
** execute button move --- move to right side (and make it possible to disable)
** how does execute work
** close builder window doesn't trigger quit?
 
* Provenance view [Tommy]
** read all provenance right now
** create an index file in the vt to load pieces of provenance on demand
** enhancement idea: highlight errors in the list view of Log Details
** tooltips in the browser view show large icons
** search for provenance browser
** fix issues with the left box
** number of search boxes...
 
* Update splash screen [Juliana will talk to Nathan]
 
* Go over Jan's feedback on the PythonSource editor
 
''A slight UI glitch is that I couldn't access the undo/redo functionality with the usual Cmd+[Shift+]Z shortcuts (neither via the menu or toolbars) but only via the right mouse button. One additional feature that might be nice in the future would be some minimal indentation assistance, like when pressing [Return] the current indentation is kept for the new line.''
 
=== May 2, 2011 ===
 
* Demos of new 2.0 functionality:
** Mashup
*** log issue: break the link between vistrail.log and controller.log...
*** experimental version so we can play with it and see if there are issues, what improvements
** Provenance browser
*** waiting to merge
** PythonSource editor
*** compile for windows, check how it works on Mac
*** issue with multiple windows?
** Global variables
*** expressions with variable, wait on this
*** set parameters on vistrail
*** click on ports to associate with globals
 
* Bug fixes
** Web services
*** upgrades are working
*** Upgrades are in a separate package
*** hooks in vistrails to allow detection when a package is not loaded
 
=== Apr 26, 2011 ===
 
* Updates:
** Mashup
*** easier ways to generate package wrappers
*** working on merging code
** Provenance browser
*** fixed thumbnails to show larger thumbnails
*** via tooltip?
*** keeping separate branch right now, detailed commit message on merge
** Persistence Package
*** stand-alone browser for writing and deleting files
*** writing annotations for execution provenance
*** find where version is not being written as an execution annotation
** bug: upgrade and Web services
*** last modified header not in all http transfers (e.g. for wsdl)
*** how to track versions of web services
*** one package per wsdl? see if this will work
*** seems to be an issue with network and that server (have to lookup twice)
** Global variables
*** next week demo
** new GUI
 
** v2.0 branch
*** keep mashups off master
*** merging is pretty good using git
*** changing to a single-window interface, adding menus back
 
* Decide:
** PythonSource editor
*** try Eric or some QScintilla-based approach
 
=== Apr 18, 2011 ===
 
* 2.0 Release: new features to be included and bug fixes
** Mashup
*** if already have alias, carry these over
*** have ability to choose GUI elements
*** storing provenance
*** allow users to import aliases, but don't write back to vistrail unless users wants to
*** provide user
*** create new palette for showing information about subworkflows and mashups (and tagged workflows) that shows on the left side [Dave]
** Provenance browser
*** Index for provenance information
*** Showing this in a separate tab, linking to a existing workflow
** Global variables
*** create a separate window
*** have underlying code for this
** control-flow assistant
*** Dan send documentation to Juliana and Claurissa
** Updated documentation
*** especially for executable paper
*** mashups
** Feature 'executable publications' more prominently
** FIX: upgrade and Web services
** Python editor
*** indentation and coloring
*** modeless window
 
* Updates
** Workflow search [Tommy]
*** can now save multiple objects to the database in two round trips (modulo number of groups)
*** create bigger thumbnails using tooltip in Ubuntu
*** view pipeline execution with groups
*** qbe. indexing high-level overview to Tommy
** Global variables [Daniel]
** Mashup integration [Emanuele]
** Python source editor [Emanuele]
** Web services update [Dan]
*** don't have upgrades
*** version numbers for web services modules, detect via some type of hashing
*** use last modified date + hash
 
=== Apr 12, 2011 ===
 
* Workflow search [Tommy]
** Thumbnails in search
** Workflow search for individual workflows
** Materialize workflows: user-controlled for now, button in search interface to materialize workflows
** 2.0: log search for sure, possibly workflow/execution search
** Show groups in the execution view, add navigation bar at top of view (like Illustrator group navigation)
** Dave look at code for QBE & Indexing
** Also try to work with data
 
* Groups/Subworkflows
** need more documentation
** better ways to view and navigate still needed
 
* Documentation [Claurissa]
** Persistence
** Could use ListToDirectory to take a list of files and convert to Directory (use for example)
** Delete/modify persistent data
** Persistent data provenance [Dave]
 
* Mashups in VisTrails [Emanuele, In Two Weeks]
 
* Interface changes [Dave]
 
* PythonSource updates [Emanuele]
** DeVide, Eric4
 
* HTTPDirectory
** create a manifest file that can be used for HTTP directory
** check with Matthias for how this works, not supported by HTTP
 
===Apr 1, 2011===
 
* Meeting time change suggestion to 10AM (Tommy) -- sorry Tommy, only now I saw this item...
** Next week: meet if need be
** Move future meetings to 10am if room is possible (Juliana checking with Deb)
* Workflow Execution Explorer demo (Tommy)
** search interface for workflow executions
** thumbnails?  problem is size
** can open a result and view the execution provenance
** querying vistrails
** materialize tagged versions in the database (so we can query workflows directly)
** extensions to allow query-by-example to work?
* 1.6.2 release (mainly with bug fixes) (Emanuele)
** integrate changes to module configuration?  on master, wait for 2.0 to put in release
** investigate different options for Python editors
** users guide? change to use sphinx for this release, add new features for 2.0
* VisMashup support in VisTrails discussion (Emanuele, Wendel)
* Wendel will give a demo of his work on supporting interactive visualizations in Web browsers; and of his simplified version of VisMashup
** generate 3D view and parameter interaction just from aliases
** suggested parameters: can associate parameters from workflows from other vistrails to labels, also order according to prevalence
** data: allow user to select other datasets from crowdlabs
** how to identify parameters that occur in different modules
** identify aliases in the web interface
** creating workflow view
** how to store the provenance of such views
** save dictionary of values from the mashup with the executions
** workflow to mashup bridge
** also allow a mashup mode to start from
** assistant interface
* Executable Pubs
** shell-escape not as well supported
** change the latex extension to be a preprocessing step (and make a vt_pdflatex cmd or something like that)
** Microsoft and MacOS X for publishing plugins--no Active X, known issue
** add better annotations to pdfs after the fact so that users can annotate papers
** meta-presenting
* Manual
** packages: difference between wiki page and users guide
** split users and developers guide, may have to do this for chapters like ControlFlow
** keep list of questions, write emails, talk to people
 
===Mar 22, 2011===
 
* support a subset of VisMashup within VisTrails: re-use subworkflow infrastructure, and add the functions which create the application. Similar to subworkflows, mashups could live inside a vistrail
** [DK] Note that mashups are higher-level objects than workflows.  It would seem that mashups should include vistrails.
** [JF] My idea is really simple: when we create a subworkflow we are essentially creating a 'view' that hides away the details of the underlying modules and only exposes a subset of the parameters. What I would like to have is the component of VisMashup that takes a subworkflow specification and generates the app.  This would greatly simplify the publication of *interactive* results. And I believe, it would be simple to implement since it would re-use the existing infrastructure we have for subworkflows.
 
* support for global variables: maybe, an initial approach could be to have such variables associated with a vistrail. This would greatly simplify packaging of workflows, in particular, to deal with file locations in different systems
** [DK] We have discussed this before as having '''workflow'''-global parameters and have proposed a method for accomplishing this---namely modify the interface to hide global parameters. It may be possible to have them for entire vistrails, but note that this poses problems. Suppose a user that creates a workflow identifying 'x' (a Float) as a global variable, and then changes the workflow and identifies a new parameter as 'x' but its type is now a String.  A per-workflow definition of the global 'x' is fine, but a per-vistrail definition is not.  I agree that we should try to support some style of alias-like behavior so that a user can set parameters more globally, but it is important that these settings are saved as provenance as many of these might be per-session...
** [JF] I want global variables to simplify packaging of results. Having them on a per workflow basis would not solve the problem I had: In the WikiQuery case study I have a series of related workflows that all rely on the same set of input files; so in this case, I would need global variables at the VisTrails level.
** [JF] As for the issue DK raised, I think we could have a very simple solution: we check for conflicts with global variables and prevent the used from making mistakes
 
* SubWorkflows in the database (Tommy)
** abstractions==subworkflows? Some error messages still refer to abstractions
** Logic for merging subworkflows from different sources? (Upgrading issues?)
* Control Flow Assistant feedback (Tommy)
* From Matthias:
** Remote execution: make it easier to move executions to clusters, starting them from VisTrails, waiting for the results.  Can we make Joel's code for the wildfire simulations more flexible?
** Using VisTrails for validation: make it possible to take results of a workflow, and compare them with a previous result to validate that changes in code or VisTrails have not affected the results.
** Make it easier to select a persistent repository (test, release, etc.) so that the same workflow can be run against different repositories
** Better support for querying hundreds of data files (ie create a database of metadata that a user can use to filter the set of files to locate specific files quickly)
** Along with this, allow this to work over existing (non-persistent) repositories.
** A better python editor for PythonSource
 
===Mar 15, 2011===
 
* database to file: works for log now
* database: save subworkflows to the database
* add uuids for vistrails, subworkflows, merges
* merging: same uuid and data changed
* look into distributed algorithm
* merge should possibly include logs
* allow additional metadata to be controlled, possible to have a 1-1 correspondence
 
* speedup on database queries to speed up joins over entities, combine different attributes
* loops on module executions
* traversing module executions
* build transitive closure
* build table to link the workflow_exec with module_exec
 
* fix for users-list problem with Embed GUI issue
* how to handle users that want to contribute code or examples
 
* [Dan] committing control flow assistant soon, issue with database version (see trac commit from today)
 
===Mar 8, 2011===
 
* DB speedup finished? [Tommy]
** Speedup done.
** Looking into issues with saving log
** Execution Provenance API
 
* Control flow [Dan]
** Fixed issues
** Fixed issues
** Input List Override working
*** Signatures are now bytes (convert to string to show in widget)
** Cartesian product now working
*** Cannot compare different types, e.g.,  (None < 1)
** Error message with version tree
**** PortSpec.depth and min/maxConn could be None. (Backport!)
** Weird issue with the group executing before the map starts
**** cell.row could be None
 
*** Lots of unicode/bytes fixes to serializations
* Documentation [Claurissa]
**** We have encryption implementations in the code! (keychain)
** check in guide to git
*** Lots of small fixes to package method differences
 
** Current issues
===Mar 1, 2011===
*** logging widget empty?
 
*** version tree signals messed up
* [[Documentation Log]]
*** provenance view legend icons black
 
*** Text do not fit in some widgets
* Updating the relational DB (Tommy)
*** VTK thumbnail comparison fails
** Import vistrail from DB is now very fast.
**** VTK Cells have some faces in wrong direction?
** Mostly done with export vistrail to DB. Still a few issues.
****  Because running in VM?
 
** Installing on Ubuntu LTS/Latest Mint
=== Feb 22, 2011 ===
*** Install latest matplotlib from pip
 
*** Install VTK 7.0.0rc2 from source
* Python Source widget (Emanuele)
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)
 
* News: Claurissa will help us with documentation. She will start by translating the existing user guide into Sphynx, and then move on to updating the content as well as documenting the many features that are currently not described in the manual.
** Please add any undocumented features as well any (informal/incomplete) descriptions you might have to the following page: [[Undocumented Features]]
 
** focus issue (will ask Huy if he knows a good solution for this)
 
* Browser updates (Tommy)
** project browser, workspace
 
* Updating the relational DB (Tommy)
** insert size: Tommy tested with 60,000 inserts
** loading: multiple selects and retrieve them all at once
** incremental updates, is_dirty, is_new
** no need to bump the schema version
** start thinking about better ways to do queries
** can prepare statements via low level mysql commands
 
* Loop Assistant (Dan)
** add tutorial for this loop
 
* mailing list emails
** Dan: keep track of which emails and update the FAQ with answers
** Could think about putting this into a system eventually, but for now it's low-traffic enough
 
=== Feb 8, 2011 ===
 
* LaTex extension demo (Emanuele)
 
** New embed dialog
 
* Non-modal PythonSource windows ([https://vistrails.sci.utah.edu/ticket/408 link to discussion])
 
** make it non-modal, single window
** update selection on click
 
* Where to keep User's guide and other documentation
 
** Sphinx, commit to doc directory
 
* Log Explorer demo (Tommy)
 
* [[Log_API]] RFC (Tommy)
 
* Control Flow Assistant (Daniel)
 
=== Feb 1, 2011 ===
 
* Execution Provenance View (Tommy)
 
* Control flow assistant (Daniel)
 
* Script Interaction (Emanuele)
** CDAT feature that might be useful in other settings
 
* Port/parameter management (Dave)
** moved close button
** what about NULL parameters?
** global parameters
 
=== Jan 18, 2011 ===
 
* VisTrails browser (Tommy)
** demo
** can import files and directories to the list of files
** can see and open vistrails, workflows
** need to think about how to show execution information
 
* Control flow assistant (Daniel)
** integration with parameter exploration
** can re-use infrastructure with parameter exploration
** can keep provenance for parameter explorations for free if we use the looping constructs
 
* New port/parameter management (Dave)
** demo
** ordering?
** ability to filter, sort parameters
** documentation
** required ports
** update modules to show when parameters are set on ports
** talk to Matthias
 
* 1.6.1 Release
** tomorrow, bug fixes + executable papers
** create tickets
 
*
 
=== Jan 11, 2011 ===
 
* Release (Emanuele)
** 1.6 released 1/7/2011
** there is a branch on git
** nightly builds
** bundling ALPS python support into releases
 
* Control flow assistant (Dan)
** focus on supporting the Map module
 
* VisTrails browser/repository (Tommy)
** open a vistrail get included in an index
** index includes vistrails, workflows, and executions
** use thumbnails for showing the workflows
** in the future, sort by projects
 
* Port and parameter updates (Dave)
** Colin Talbert at USGS-Fort Collins requests the same global parameters feature that Matthias has already mentioned. We should start working on this functionality.
** Colin also asked about highlighting required ports
** Changes to schema: add required flag, cardinality flag, and global parameters
** Changes to execution: way to add global parameters and connections to workflow before execution begins (not display visually)
** Changes to interface: method to select ports and tie them to global parameters, global parameter display, highlight for ports to show required (red?), do we need to show cardinality?
 
* Executable paper tasks
 
== 2010 ==
 
=== Dec 21, 2010 ===
 
* saving parameters in a module (email from Tom Maxwell, from NASA)
* explorations still do not detect MplFigureCell modules inside groups or subworkflows
 
=== Dec 14, 2010 ===
 
* [ES] Unit tests
** Ran unit tests last week
** Errors with old versions of pipelines, need to fix pipelines
** console_mode_test can't be enabled
 
* Data publishing with CrowdLabs
** using persistent files and directories?
** couldn't think about how to synch user directories with server directories
** look into how to push data files across with git
** how to organize data (ie persistent files) from a specific workflow or all workflows with specific parameters
 
* [DR] Subworkflow upgrades: update--is it a wrap?
** functionality are completed
** one cosmetic issue -- disabling a package
** subworkflows move from top to bottom of the list
** [DR] Add documentation to wiki on subworkflow upgrade paths
 
* [DR] control flow assistant
** started looking into this
 
* [TE] [[Data/provenance browser]]: update
** looking at the old code
** some problems with the logs
** cannot find parent workflow
** adding project annotations to sort VisTrails into different projects, different vistrails in different project, need to have gui to do this
** synchronize when vistrail changes, especially with execution logs, create index schema for these files (vistrail and workflow exec)
** persistent data index, need to integrate eventually, scan through actions to generate list of files used
 
* PythonSource editor
** seeing which PythonSource editor is open
** need to be able to have "save now" button or automatically save when someone presses "execute"
** generate id tag automatically, close the other window immediately
** highlight module being edited/configured, keep window open (like console, messages)
 
* Lack of global variables and bookmarks
** don't need to reconfigure executions to do this
** just don't display the "global" modules and connections
 
* [CS] look into 3 items for [MT]'s group: ports display, python source, global vars and bookmarks (and persistent files dump)
 
* [PM] Merging master and crowdlabs git branches. How should we merge the crowdlabs branch into the master branch?
** problem is that there are changes in crowdlabs that can't go in the master branch (i.e. HTTPFile)
** [ES] just merge both so we don't have the crowdlabs branch
** [PM] change RepoSync to have a server config preference that is set to True for server, False for client, and RepoSync has an if in the compute that does one thing if server is true, another if server is false
** have a branch for workflow indexing so that patch lives in another place
 
* Box input ports should be colored (e.g. black) if they are already set by an internal method. Another color (e.g. red) could be used to indicate mandatory input ports.
** Currently, non-optional ports are square, shown. Optionals are circles and must be enabled
** Add a third category for mandatory, problem is backward compat.
** [ES] Colors are already overloaded, not for ports, use shading instead?
** Also needs to be with global settings, etc.
 
* 1.6 release: [ES] already has 1.6 candidate
** don't include the new gui features yet
** freeze 1.6, just bug fixes
** merge crowdlabs branch
 
* [DK] persistent files on the server
** [MT] has problems with files on server, issue with /tmp/, running older python (2.5)
** stress test, test on [DR]'s machine
 
=== Dec 7, 2010 ===
 
* [DR] Subworkflow upgrades
** Made sure that we don't rewrite actions but create new actions that show where subworkflows are upgraded
** Document what is implemented and how it works (at high level), also what isn't yet implemented
 
* [TE] Errors and web services
** allow user to decide to only display errors to messages window instead of also showing popup windows
** Allow add/delete of web services from GUI without going to preferences
** Qt dependencies for error messages gone
 
* [PM] Some specific questions
** Question: Should v1_0_1 MySql schemas be upgraded to v1_0_2?
** db_connection.begin() being removed, stale results
** memory size is too big for the field size
** [DK] check on schema changes, see what is important for changes
 
* Question: do we display 'warnings' or only errors?
** Need to write ModuleErrors (maybe make ModuleWarnings) to the messages (debug) as well as adding as tool tips
 
* Some suggestions/requests by Matthias:
** Support the grouping of vistrails and files into projects
*** Annotation here?
** Ability to set global parameters/bookmarks=workflows (e.g., a filename or path)
** Archive results -- create a big tar/zip file with all vt and results
** Ability to add a vt: if a node is deleted, all data associated with that should also be deleted
** Create a test suite: run a workflow, save and archive results (user could specify which results should be saved)
 
* [[Data/provenance browser]]
** We need a better mechanism that allows users to search/query for workflows and provenance, for example search by project, parameter names and values
** One way to view this is to allow the creation of mappings between the vistrails/projects into different structures---like "smart folders", e.g., /projectN/vt1/wf3/exec23/O1....
** Support editing and synchronization in both ways (if edit smart folder, should update vt)
** Should be able to script the use of the browser, e.g., to test a new version of a workflow against a saved test suite
** Scientists like a directory structure and can go browse there like they browse files with finder/explorer
** Keep mappings from our datatypes to file structures
** Need an exportable archive (to send with a paper, allow browsing outside of VisTrails)
 
* On merging master and crowdlabs git branches
** some updates on the crowdlabs branch need to be merged/patched into the master branch
** how should this be done?
 
* PythonSource editor
** modal or multiple window editors?
 
* Lack of global variables and bookmarks
** Problem is that aliases don't change workflow specification
** Need to store these values in the execution log
** Keep alias dictionary for everything
** [PM] Not a need for global variables above the single workflow level
** This already exists at the conceptual level---one module with connections to other modules, this just is cluttered
 
* Unit tests
** [ES]: I ran the test suite on current master branch and 207 tests are executed of which 11 fail. Some of the tests are failing because console_mode_test package can be imported anymore (this is a package created specific to run console_mode tests).
** Most of the other problems are issues with
 
  InvalidPipeline: Pipeline cannot be instantiated:
  Missing version 0 of package <package>
 
* Data publishing with CrowdLabs
** Using git to manage data between local and server data stores might be too messy.
*** Does one create a new git repository for each vistrail on crowdlabs?
*** The local persistent git repo holds all persistent data used on that machine. How does one push/publish just one data product using git?
*** Can anybody think of a plausible git workflow to do this?
** Perhaps we should just look into adding directory support for RepoSync module along with a better data management UI.
** Are there OS independent file transfer protocols better than HTTP? Something like rsync might work nicely, but windows support is limited.
 
=== Nov 30, 2010 ===
 
* Subworkflow Upgrades (Daniel)
** Should be completed finished tonight
** Local subworkflow upgrades should be working now
** Need to test these after the checkin
** Using a dummy controller
 
* Assistant for control package (Daniel)
** Look at examples
** Try to make an assistant to modify workflow using currently selected modules as looping group
 
* [[Improve error handling/reporting]]
** Tons of message boxes pop up when a workflow has a bunch of errors, UI response is bad after this...
** Update existing error messages
** Show Details rendering on Mac
** What to do when multiple errors occur in a single event?
** New dialog that shows "next message" but allows user to dismiss messages or view all in the messages window
** Color coding for messages: use gui/theme.py to configure: maybe try black for warning, gray for log level messages
 
* Data and workflow browser (Nivan will give a status update)
 
* Saving Explorations (Feature Request from ALPS)
** We already save parameter explorations with the workflow in the version tree. As is, we save only the last exploration that was done for a given workflow. If you ship the vistrail, the saved parameter explorations will be there.  Is this sufficient?
** [MT] Yes, that takes us a long way. Could one add a button in the explorations to save a certain set of explorations as a new version, like you do with the camera angle in the spreadsheet? That way one could store more than one set of exploration parameters without forcing a new version to be created with every change?
** Eventually we should provide a selector in parameter exploration
** Fastest to just allow a null action new version (a bit clunky) but wouldn't require too many changes
 
* Clean up/prune persistent directory
** What are the concrete use cases for this?
** [MT] Here are two concrete use cases that appeared over the last week:
*** a project is done, the important results copied to an archive and I want to get rid of all its intermediate files, etc., to save space - but not those files associated with other projects. Or, similarly, I'm done testing some workflows, want to remove all (partially buggy or irrelevant) files, and then start with production runs.
*** there was a problem in a workflow(e.g. a bug in HTTPFile module with binary files or a bug in some user code). I want to force a certain persisted directory to be "forgotten", so that I can repair the wrong files. Bumping the version number of the ALPS package would be one way, but that is overkill since it would immediately invalidate all persisted files. Removing the whole persistence directory is just as radical.
** just testing simple tutorials with small files leaves me with 2 GB of persistent data after 10 days, in 142 persistent directories or files. I need some way to clean this up that is less radical then removing all persistent directories and files.  This could be done, by e.g. removing all those which do not occur in tagged or leaf workflows in a certain set of vistrails. Or marking those which occur in certain vistrails with certain labels, and then being able to prune all those with or without some label/tag. Another option might be having the ability to choose from various repositories.
** [DK] follow up on this
 
* Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)
** work in progress


* Data Publishing with crowdLabs. (Phillip) 
=== January 20, 2016 ===
** Do we want data publishing capabilities in VisTrails and crowdLabs?  '''YES!!!'''
'''Updates'''
** If so...
*** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.
*** Persistent Data works for experimental data. Can we use it for publishing data?
**** Publishing data or storing data on the server and pushing back and forth?
**** What is the current scheme for identifying data on the server?
**** Do we need to do better than the ids, versions, hashes that the peristence package provides?
*** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?
** [ES] Feature request: Can RepoSync support a directory of files?
*** [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.
** persistence used more for exploratory
** want to focus on more archive
** move from exploratory data to published
** Dataverse: essentially a site to curate data, archive data, host their own site, each university has their own server, linked together
** persistent store integration: maybe use git to do this, can add to local repositories to move to git
** use some of the dataverse: standard hash schemes, etc.
** privacy: can users push workflows without exposing data
** permissions for who can check things out: download permissions, upload permissions
** look into setting up persistent store on


* DisplayWall (Wendel)
'''Items to Discuss'''
** new version of vistrails receives rotate/scale messages much more slowly
* [TE] Backporting features from Python 3 + PyQt5 + Qt5
** using the vtk version that is already there
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])
** comparing VisTrails 1.2 with nightly src
*** One unicode test failing
** test when the messages are received on the clients
*** Problem saving to xml
** [ES] Changed something in the VTKCell, using the Qt interactor from vtk now, just uncomment this (vtkcell.py file)
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])
** some commands are missed; check why this is happening
*** Updates print to functions 'from __future__ import print_function'
** preferences switch? 2x2 not 2x3
*** Updates exception calls to use 'as'
** check what modifications exist in the client code that's not exactly 1.2?
*** removes most usages of 'long' (mostly used unnecessarily)
** new version of vtk package changed some things
**** Only needed for 32-bit support
** update vtk upgrade: resetcamera and addrgbpoint => _# versions
*** Fixes relative imports (from __future__ import absolute_import)
** Demos on displaywall: Wendel has to keep both VisTrails running and stop work when visitors stop by unannounced
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])
*** Updated all signals to new-style
* If we go 2 to 3, this could break packages
** means we should schedule any package format changes for the same time
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)
** putting package identifier as xml?
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)
** version the APIs?
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?
* file formats on 2 to 3:
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2


* Relative paths in workflows, how to deal with them? (Emanuele)
=== January 13, 2016 ===
** [ES] Updated all examples to use HTTPFile for these
'''Updates'''
** some of these had to change completely (color widget), kept tree the same format


=== Nov 16, 2010 ===
'''Items to Discuss'''
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]
** Basic GUI and execution works
** Minimal work to support 2to3
** Hard to support both PyQt4 and PyQt5


* Data and workflow browser (Nivan)
=== January 6, 2016 ===
** Check into git branch (Dave)
'''Updates'''
** Add data support to the vistrails, workflows, and execs already exposed in the browser
'''Items to Discuss'''
** Add thumbnails to the browser
* Updating VisTrails dependencies
** Bittorrent support?
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.
**** Next version of VTK 6 is rumored to support Python 3
**** There is a development branch of VTK 6 supporting Python 3
*** Can travis build this?
**** We can skip tests for WebEngine and VTK if needed
** Python 3 [https://python3wos.appspot.com/ showstoppers]
*** No MySQL-python (sqlalchemy supports mysql through other drivers)
*** No suds (there are active forks but we could also drop support?)
* Options for python2/3 support:
** Interpreter could be python2/python3 based on packages
** six layer works but not the easiest to maintain (e.g. in reprozip)
* Drop support for win-64
* Update copyright headers to 2016


* Upgrade issues -- a wrap? (Daniel)
== Older meetings ==
** Fixing some issues with namespaces
* [[Development/2015]]
** Upgrades for subworkflows that are included with packages, need to save these upgrades with the vt file as a "local abstraction" to ensure provenance.  May be able to tie these back to the package versions once the package is upgraded.
* [[Development/2014]]
 
* [[Development/2013]]
* Test Suite
* [[Development/2012]]
** Matthias has his own test suite now
* [[Development/2011]]
** We should make sure that our tests are up-to-date and run them before releases
* [[Development/2010]]
 
* Web services on users' guide (Tommy)
** Users' guide needs only a few changes.  One of the changes should be a note making it explicit that we are using a new library and the old one was deprecated.
** Do we want to update the 39-page guide as well?  For now, just update the users' guide.  Don't distribute this guide.  Just include it in a zip file that includes the old version.
 
* [[Improve error handling/reporting]] (Tommy)
** print statements, has a new window
** add a menu item to view message window and also keep all messages from the session
** look into writing QMessageBox errors and warning to message window
** write all messages to a single window, write critical messages in red?
** have a utility method that encapsulates the gui error display and the core.debug display (if not running gui, only write to core.debug)
** don't display stack trace in a qmessagebox?  -- try to emulate the invalid pipeline with "Show Details" button.
** make it easier for users to report errors by having a "copy" or "send to developers" button to report the details of an error.
 
* DisplayWall (Wendel)
** What examples should be in the SVN? If we decide to keep some examples in the svn, some of them use data that should be along.
*** Indicate progress of HTTPFile
*** Had to upgrade examples to work with new examples of VisTrails
*** VisTrails can send progress information with another thread like crowdlabs can do
*** Each machine has a .vistrails and needs to download own version of data, make a single .vistrails for all machines?
*** All slave machines can use the server as a cache?
** DisplayWall Client on the Apple Store? I dont like the idea of sharing the client source code. And putting the app without login/password will allow anyone to access our displaywall.
*** Add a password for this
*** Do user management?
 
* VTK package:
** Have two ResetCamera modules now
** Need to write upgrade code
** Only with vtk 5.7 right now
** Encode library version in the package version?
** For reproducibility, we probably want to stick to specific versions
** We still need some ability to run new versions of vtk for testing, new features, etc., too...
 
* Relative paths in workflows, how to deal with them? (Emanuele)
** Should we use dataDirectory?
** Change examples to not use hard-coded files?
** Problem with workflows, not data files
** On Windows, use ../examples
** On Mac, use ../../../examples
** dataDirectory not used, I think
** create "$VT_EXAMPLES"?
** problem is using new versions of VisTrails, data may change so referencing old location will fail
** don't allow users to change this variable
** convert to use HTTPFile
 
* Data files and the GIT Repository (Emanuele)
** make a separate repository or use the submodule git functionality
** go with submodule
 
* Next release: What still needs to be done?
** high priority from ALPS list
** Dan's fixes to the upgrade workflow
** Date for release?  Official release in beginning of December for ALPS
** All developers should run test suite on push
** Could make this somewhat automatic, run with nightly release script and email -dev list
** Could have different classes of tests, make sure that critical ones pass
** Tests have a bunch of errors currently
 
* Saving explorations:
** Keeps track of the latest exploration for each version
** Check with Matthias what the requirements for this are
** Have way to send just specification to a server (run this workflow with this parameter exploration)?
** Can we have an xml spec that specifies the parameter exploration (export parameter exploration)
** Currently persisting via xml and then converting to string so we already have an xml serialization
** Dan wrote API that takes xml string and populates the param_exp gui with the appropriate values
** Need to back the GUI state with core state to enable short-circuiting the gui
 
 
* Box input ports should be colored (e.g. black) if they are already set by an internal method. Another color (e.g. red) could be used to indicate mandatory input ports.
** Currently, non-optional ports are square, shown.  Optionals are circles and must be enabled
** Add a third category for mandatory, problem is backward compat.
** [ES] Colors are already overloaded, not for ports, use shading instead?
 
=== Nov 9, 2010 ===
 
* FAQ
** As we reply to users queries, let's add the question and answer to the FAQ!
** Just a reminder
 
* Caching of File module
** There was a message from a user who got confused because this module is cached by default (and silently). And in his application, since the file actually changed in between runs, he did not see the 'expected' result.
** Should we keep a hash for files and check whether they changed?
** Should we no cache File by default?
** [DK] File is supposed to have a special signature computation that detects changes in the file contents so this may be a bug instead.  It does work, it seems to be an issue with the create_file
** [ES] In this particular case, the file was empty. He deleted the file outside VisTrails and executed it again. As he set create_file to True, he was expecting the file to be created again.
** [DK] Add random salt to hash signature when the file doesn't exist and create_file is True
** [JF: DONE] Juliana will edit the FAQ to note that the signatures are dependent on modification times
 
* Claurissa will demo different ways to visualize the version tree
** weighting on various criteria for displaying nodes (importance from session, user, tags, etc.)
** other modes for viewing the versions (lists or timelines)
** often see only linear trees, teaching or thinking other than tree?
 
 
 
* Upgrade issues (Dan)
** Vanilla upgrades should be in today's commit
** Added code so when doing upgrade to check whether the latest code is upgraded
** Two types of upgrades: (1) User changes the subworkflow, (2) Modules are out-of-date and need to be upgraded.
** Add higher-level features like "Upgrade All" for (1), "Import Changes to My Subworkflows", etc.
** Want a 1-1 mapping between unique subworkflow id and a subworkflow file (even if we copy the same subworkflow to different vistrails)
 
* Offscreen bug  (update from Huy)
** Mac issue when switching to Cocoa
** Huy added a QWidget for each of the render windows that is created, so should be fixed.
 
* Gesture support
** Someone on Windows check the pinch gesture support; it doesn't work for Matthias under Parallels
** VTKCell is working now
** Cell dependent, panning now maps to middle button, pinch on the base widget
** What are Matthias's requirements here?  Does touch gestures in the cells matter or only in the vt/wf views
 
* Thumbnails: [JF: this needs to go to trac, for 2.0?] [ES: 2 tickets were created for 2.0]
** Resolution? -- [ES] problem is the size of the vistrail, but this can be configured
** Can we export to PDF instead of PNG? [ES] cannot display the thumbnail as a PDF in Qt?
** Can we have the option to save higher-res or PDF versions of the thumbnails
** Can we have preferences that allow users to save a higher-res or PDF version, maybe a checkbox
** Maybe associate a high-res version with a spreadsheet cell so that we can save a high-res version on demand
** high-res version associated with a version but allow user to initiate action from spreadsheet
** can we have thumbnails dependent on camera position [ES] doesn't like, user won't see exactly that image upon re-executing
** compact vistrail option by removing thumbnails
 
* Tommy has rewritten the Web services module as well as updated all of the examples
** Server for one of the examples seems to be down
** Need to update the manual
 
* [[Improve error handling/reporting]] (Tommy)
** Maybe display the intialization messages with the splash screen?
** Debugging levels have no way to access debug level
** Tommy takes first cut at trying to determine how to map print statements, notes those he is unsure of
** Developer guidelines for debug usage, use them instead of print statements in the futur
 
* Merge functionality (Tommy) [JF: Cool!]
** Can now merge an existing vistrail into your own vistrail using menu item
 
* Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)
** [PM] I haven't had a chance to implement anything yet.
 
* Data Publishing with crowdLabs. (Phillip) 
** Do we want data publishing capabilities in VisTrails and crowdLabs?  '''YES!!!'''
** If so...
*** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.
*** Persistent Data works for experimental data. Can we use it for publishing data?
**** Publishing data or storing data on the server and pushing back and forth?
**** What is the current scheme for identifying data on the server?
**** Do we need to do better than the ids, versions, hashes that the peristence package provides?
*** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?
** [ES] Feature request: Can RepoSync support a directory of files?
*** [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.
 
=== Nov 3, 2010 ===
 
* Upgrading subworkflows (Dan)
** add updated versions to subworkflow vistrail and push to registry
** fixed bug with version/descriptor redirects
** key remaining task is to replace the box representing the subworkflow in the top-level workflow to reflect the change to the upgraded version of the subworkflow
** many corner cases, but we should get the vanilla upgrade (working) to git
** need to ensure that each subworkflow file has its own uuid; if a subworkflow exists in two different vistrails, it should have a different uuid (namespace).
** otherwise, we can get crosstalk where one vt can update the subworkflow from another vt (we can propgate changes to other vts via merges, but this should be a user choice...)
** enhancements:
*** allow users to merge subworkflows from a file into their own subworkflow (to incorporate outside changes)
*** latest version is the most recent non-upgraded version; if that version has an upgrade, use the upgrade
***'''Next step: work on usability for controlflow package; explore Dave's idea of an assistant'''
 
* Offscreen bug
** Huy is looking into this
 
* Crash due to issues in loaded user packages (Tommy has fixed this)
 
* Web services (Tommy)
** Package is completed, and all but one of our examples are working
** '''Tommy will test if the Web services will work when there is a proxy'''
 
* Improve error handling/reporting (Tommy)
** we should have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages.
** the goal is to implement a function that will be the single point for error messages, and that work without emitting GUI signals in core. As Huy suggested, we should have a GUI-wrapper which will be a no-op when the GUI is not instantiated---this will get rid of some of the pyqt dependencies (at least for the error messages).
 
* Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)
** Currently the VisTrails servers consists of: a single-threaded instance with GUI; multiple GUI-less threads
** We will extend the server API to allow clients to check the server status. The server will provide a separate socket and the client will be able to 'ping' the server; if the server is working properly, it will respond; if it does not respond, the client will have a time out
** The API will also support the ability to kill both the GUI-less threads and the single-threaded GUI instance. We can then restart the server components using Emanuele's new script.
** We should also allow users to set a per-workflow time-out; if the workflow execution takes longer than the pre-defined threshold, VisTrails will abort the execution
 
 
* Update on fixing VTK Package (Wendel)
 
** It seems that the ParaView package has the same issues.
** '''Wendel has already checked in the new wrapper into the trunk; Emanuele will test it'''
 
* Synchronizing VisTrails and ALPS releases (Emanuele)
** VisTrails latex package was extended so that python is no longer required. If python is not present in the system, instead of issuing a request to execute the workflow and retrieve the image from the Web, a local, previously saved image will be used.  It is now also possible to embed the images of workflows into the latex file. Documentation about these and other features are currently in the README file and examples (example.tex) provided with the latex package.
** Windows and Mac beta binaries already include ALPS
** Are we distributing ALPS as part of VisTrails then?  This seems backwards.  ALPS should VisTrails as part of their stuff, no?
*** '''We will have separate binary distributions: one with and one without ALPS'''
 
* Is a multi-touch interface to VisTrails possible now that it's supported by Qt? (Phillip)
** Yes, Qt 4.6+ supports multi-tocuh --- '''Huy will look into this; the goal is to make this work both for the workflow builder/tree view and the spreadsheet'''
** We still need to figure out what makes sense for multi-touch, however, do we have requirements from ETH-Zurich?
*** We want to map the zoom, pan, click, etc to multi-touch, so that VisTrails can be intractable on a multi-touch screen without a mouse
 
* What is the status of the Vismashup i{Phone|Pad|Touch} app? (Phillip)
** If you mean mac binary, there's an alpha version here: http://www.sci.utah.edu/~emanuele/files/vismashup
** Do you mean iphone app? [Phillip: yes]
** '''Wendel will look into this'''
*** One of the issues is how to effectively handle images that are larger than the memory on the iPods and iPads---we need to process these on the server or use a model that allows the image to be manipulated on the client
*** We also need to connect the app with the crowdlabs server, so that the app can get the list of mashups
*** Phillip will investigate the feasibility of working directly with JavaScript and bypass flash
 
 
 
=== Oct 19, 2010 ===
 
* Update on Trac and Roadmap (Emanuele and David)
** Trac is now linked to git repository
** Roadmap on Trac has been cleaned up so we can hide completed milestones
** Tickets are being reassigned/revisited. Many tickets (24) are not associated with a milestone yet https://vistrails.sci.utah.edu/report/3
* Synchronizing VisTrails and ALPS releases
* Web services support (Tommy)
** discuss the interaction between the Web Services package and upgrades
* Subworkflows update (Daniel)
* Testing of new module drawing (Erik)
* Aliases
** Currently, an alias is stored on a parameter.  We need aliases to be stored at a higher level so that changing an alias is not a change to a parameter.  The one issue is that there is a link between parmaeters and aliases in that an alias can only exist for versions that have the specified parameter.  We might also have two versions where the alias points to different parameters.  We could just store aliases as a root-level workflow element so that the set of aliases is versioned corrected.
** In the current implementation, the aliases parsed from the parameters are stored in a dictionary in the workflow, so storing aliases as a root-level workflows element might be the way to go --[[User:Emanuele|Emanuele]] 19:50, 15 October 2010 (MDT)
* Error logging
** Suggest that we have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages.  This should improve our error handling significantly
* PyQt dependencies in core
** Can we get rid of PyQt dependencies in core (and db)?  This would probably require moving the signals/slots that we currently use to a similar Python implementation which shouldn't be too difficult.  This would also require the specification of configuration widgets not as classes but rather some text that can be used to import the GUI elements only when using the GUI code.  However, what would happen with workflows that run in command-line mode but require some graphical output?
* Ports
** One annoying part of the current VisTrails model is that users are not given any visual indication that a port value is already set by a function or vice versa by a connection.  In addition, we have no way to specify or enforce cardinality on ports at design time.  It seems like we could allow developers to specify when a port should not be connected to more than one value (via a function or a connection), and give some visual feedback when a port has already been specified in one manner.
 
==== Meeting Notes ====
 
* Bug regarding userpackages at initial vistrails startup causing startup to fail.
** Still an issue, ticket still open as a 1.6 milestone
* Copy-paste bug
** Shortcut not working on Mac.  Focus makes shortcut try to copy pipeline.
** Current fix appends to clipboard as users type - this is not a good fix.
* Synchronizing ALPS and VisTrails releases
** 1.6 scheduled for Dec. 1
* Webservices
** New library being used, but it's very low level.
** Simple types may cause problems with some web-services using complex or XML-based types.
** Need to make sure that this library can handle at least MOST of the web-services out there.
* Sub-workflows
** Daniel's changes seem functional and very slick.
** A little more work on it seems necessary.
** Need some easy GUI mechanism to delete a subworkflow.
** Is the exclamation point in the module draw the best way to handle this?  Make a tool-tip to explain it.
* VTK Wrapping
** Changes to VTK Python wrapping is causing headaches - particularly in backwards compatibility.
** Method typing has changed to expect things like "List" or "Vector" - makes things hard on the user when defining these inputs.
** Removing List and Vector types and replacing them with dynamically generated versions from Tuples.
* Aliases - Agenda pushed until after Vis.
* Error Logging - Unify some error handling to improve how exceptions are dealt with.  More on this after Vis.
* Refactoring out PyQt dependencies from Core/Db/etc - Need to move all the signals/slots into GUI elements.  Questions come when a workflow uses GUI elements as inputs.  - More after vis.
* Ports - We need a way of showing how a port is specified - function vs. input port.  Cardinality must be established in these cases.  Ordering on multiple connections should be handled in some way - whether it's just a documentation issue or a more fundamental one.
** Should we have a tutorial mode that bring up tooltips when someone does something new as if it were a live tutorial.
 
 
=== Oct 12, 2010 ===
 
* Trac and Roadmap (Emanuele): Emanuele suggested we should create a development roadmap and make it a permanent item on the agenda. The idea is to go through the Trac tickets and use them to build the roadmap that would be made ''public''.
** go through trac and sort by priority
** add other suggestions to roadmap as well
** admins can add milestones via the admin tab
** ''David will assign tasks on list; those assigned need to follow up and check and fix them''
 
* Update on [[PythonSource error reporting and logging]] (Tommy)
** ''Need to make sure that all messages are printed through debug.X, so that the debug level (verboseness in preferences) is correctly used''
** Can cut/copy/paste now
** Can we prevent the delete from happening?
** Phillip noted the spyder project has an interactive python console: http://packages.python.org/spyder/
** Can press enter to get to the end
** Need to check if this works or if we need to change
** Now have ability to see stack trace from the GUI (triangle menu)
** ''Suggest using a dialog to display trace when this item is selected instead of printing to console''
** Also saving stack trace to the execution log
** Also printing debug information to vistrails log
** ''Add trac item to fix prints to use debug''
** ''Try to fix core.debug to have gui.debug to eliminate Qt dependency''
 
* Update on subworkflow (Daniel)
** Notification when subworkflow is outdated triggered
** new_abstraction signal from registry in addition to new_module signal so that we can check if we need subworkflows to update
** How to display the possible upgrades (can be upgraded, can be upgraded but may break, etc.)
 
* New utility added by Carlos (currently at scripts/module_appearance) to draw fringes visually, which generates output that can be pasted directly into the add_module call.
** ''Erik will test this on Mac and Windows.''
 
* Preparing a "Get Started" tutorial for SIGMOD repeatability
** Do we have instructions on how to use the latex package without crowdlabs?
*** links to actual workflows in latex
*** have README and example for latex in the source
** Metadata associated with VisTrails: should we have the ability to add vistrail-level metadata? e.g., who created the vistrail, its purpose, etc.
*** have pointers to paper to workflow and vistrails
*** add pointers to papers
*** allow access to add/edit annotations at the vistrail level
*** ''add GUI element to allow people to edit/add annotations''
 
* Maintaining VTK package
** have four classes that don't wrap, have error even in python level
** works well with VisTrails otherwise
** haven't checked the new changes from the wrapping
** ''Wendel will check on the new version''
 
* [[Web services package]] (Tommy)
 
=== Oct 5, 2010 ===
 
* Welcome Tommy!
 
* Update on subworkflow (Daniel)
 
* Issues raised by Matthias:
 
- need to automatically upgrade subworkflows. Currently it is necessary to manually upgrade a subworkflow whenever the version of one of the modules inside them changes.
 
- explorations do not detect an MplFigureCell embedded in a subworkflow.
</div>

Latest revision as of 15:34, 30 October 2017

See also the Github wiki

2017

October 30, 2017

Updates

  • [DK] PyQt5
  • [RR] Running Python3 code

Items to Discuss

  • Installation:
    • Use conda? (macOS hack to have .app bundle load conda-installed vistrails)
    • Update conda recipe (pyqt4 restriction for now)
  • Interpreter
  • Release
  • Integration
    • hard to build same type of GUI in other apps that integrate core VisTrails (tend to use just the API)
    • independent pieces of the GUI could be reused if they were more independent.

September 6, 2017

Updates

  • [RR] Fixes for scikit-learn, tables, usagestats

Items to Discuss

  • Stripped-down version of VisTrails to showcase new features?
    • include meta-vistrail, bundles
    • hide mashups, extra widgets, latex mode, etc.
  • Interpreter

August 16, 2017

Updates

  • [DK] meta-vistrail
    • Working actions: normal action in vistrail, pruning version tree, tagging operations, and change
    • Interface

Items to Discuss

  • Notebook
  • Interpreter
  • Web visualization packages (e.g. bokeh)
  • Pipeline templates

August 2, 2017

Updates

  • [DK] meta-vistrail work
    • Extend Vistrail to MetaVistrail to support meta-actions
    • Vistrail becomes less tied to workflows so it can also supporting versioning vistrails

Items to Discuss

  • Interpreter
    • Using dask?
    • Streaming an issue
  • Notebooks
    • What is an output of a module? (need to expose a specific output port or the entire module)
    • Unifies some of the work from the script-workflow code

June 28, 2017

Updates

  • [DK] Testing UUID/SQLAlchemy branch
  • [DK] Working on meta-versions
    • would be nice to have pipeline and vistrail separate
    • [RR] Then do we only have meta-actions, any change to the pipeline is a meta-action that adds a version that does the change?
    • [DK] Yes?
  • [RR] Notebook interface (Vizier) making good progress, figuring out cell outputs

Items to Discuss

June 7, 2017

Updates

  • [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs

Items to Discuss

  • [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)
    • Can designate in module description an output port
    • Value has to be serializable
    • Should use the OutputModes work to handle multiple modes, configuration, selection, ...
  • [RR] Is VisTrails with SQL usable concurrently?
    • [DK] Probably not, demo from [TE] exists

May 10, 2017

Items to Discuss

  • Identifiers
  • Bundles & DB Code
  • Interpreter
  • Merge / Versioning Version Trees
  • Python3
  • Workings of List
  • Packages: Wrappers + new Versions (VTK7)
  • Script Conversion
  • Intermediate Outputs: reuse output module work?
    • would be useful for both workflows (run to a particular checkpoint)
    • and useful for notebook interfaces
  • SQL backend: does it allow for concurrent access?
    • Yes, real-time collaboration work for concurrent single vistrail access
    • Should be able to access different vistrails at once without issues

April 19, 2017

Updates

Items to Discuss

April 5, 2017

Updates

Items to Discuss

March 8, 2017

Updates

  • [RR] Added a looping indicator on modules #1217
    • This makes it easier to understand what's looping, no need to follow edges and count
  • [RR] Refactoring CLTools for integration in Docker package
  • [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)

Items to Discuss

  • [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers #1216
    • Can we move this up to the renderer modules?
    • Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)

February 24, 2017

Updates

  • [RR] Rich-text module documentation (#1210)
    • Went over packages and fixed format; maybe we should have a test to validate RST?
  • [RR] Single-instance shutdown with --remote-shutdown (#1204)
  • [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment

Items to Discuss

  • [RR] There are issues with list-depth
    • Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)
    • Issue with looping groups

January 25, 2017

Updates


Items to Discuss

  • Scaling VisTrails
  • sql-alchemy, uuid, bundle branches/merges

January 11, 2017

Updates

  • [RR] New VisTrails version in Debian [1]

Items to Discuss

  • [RR] Collaboration with school of medicine to port HiC-bench to VisTrails
    • Have homemade workflow system based on R scripts and symlinks
      • Each steps has multiple sets of parameters
      • Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step
      • Gets all results in directory structure step3-parameters/step2-parameters/step1-parameters/files
    • VisTrails improves provenance capture and makes editing pipelines easier
      • Need better support for aggregation (groupby operation might be good enough; pandas?)

2016

December 14, 2016

Updates

  • [RR] Single-instance code: over-engineered, but will work
    • (Have to avoid race conditions on startup, and OSes offer very different locking primitives)

Items to Discuss

  • Email from Matt Dirks: combobox should work, ask for more code?
  • Email from marin.nl:
    • looks like they would benefit from new interpreter
    • how to store their consolidated provenance when they exchange .vt files around?

November 30, 2016

Updates

Items to Discuss

  • [RR] Working on single instance code for MARIN (#1205)
    • Remote shutdown ready (#1204)
    • Still improving it
    • Want to return something more structured to the client (success/fail code, stdout, stderr)
    • Should be usable without Qt
  • [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason
  • [DK] Working on batch execution using console_mode
  • [DK] Settings aliased parameters when using core.api?

November 16, 2016

Updates

  • [RR] Issue #1200 from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)

Items to Discuss

  • Email from Pieter: executing pipelines with parameters from command-line using single-instance
    • Seems to work, main instance gets full command-line and alters configuration temporarily
    • Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful
    • Actions:
      • Can add switch to kill main instance
      • Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails
      • Add client/server code to core that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like gui.application.VistrailsApplicationSingleton#noninteractiveMode())?

November 2, 2016

Updates

  • New version 2.2-2 of Debian package, still wrong copyright, and broken bundle installation
    • Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch
  • Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)

Items to Discuss

October 5, 2016

Items to Discuss

  • working toward a 3.0 release; project board
    • unique ids
    • bundles
    • GUI fixes
    • wrapping updates
    • python3
    • interpreter?
  • command-line input parameters to provenance
    • [DK] we don't create a persisted workflow from those input params, i think
    • [DK] these are already defined as aliases so we treat them differently form a normal input param
    • [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases
    • Do the vistrails variables values get stored in provenance? (what if values change?)

September 14, 2016

Updates

Items to Discuss

August 31, 2016

Updates

  • [RR] Went over vgc.poly.edu/vistrails.org config:
    • Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
    • Fixed usersguide scripts (won't disappear again)
    • Moved some websites over to HTTPS
    • Websites
  • [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs (currently there) and users' guide under the same RTD "project"
  • [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that

Items to Discuss

  • [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
    • Get rid of more services? (svn, trac, ...)
  • [RR] Notebook execution works. How do we support more languages (than Python)?
    • Need: detection, metadata reader, code to input values
  • [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?

August 17, 2016

Updates

  • [TE] Fixed parameter order upgrade bug that breaks VTK examples.
  • [TE] Matplotlib and sklearn are exportable using module-to-script api
  • [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.

Items to Discuss

  • Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
  • Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
  • [TE] Wrap-up notes

August 10, 2016

Updates

Items to Discuss

  • [TE] Python Wrapper and scripting
    • All changes merged into python-wrapper branch
      • Tests pass
      • Still need to update sub-branches
    • Minor fixes
      • vtkPlot3DReader upgrade bug
      • Added tuple exporting
    • Most of vtk examples can be exported example
      • Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
        • Could be a bug in the old vistrail, but not sure
  • [DK] Subworkflows and uuids

August 3, 2016

Updates


Items to Discuss

  • [TE] Wrapper patches now stored as python code
    • Connection variables now lowercase
    • Can we use code as output variables?
      • E.g. "vtk_dataset_mapper.GetOutput(0)"
      • Could mutate output when output method is only supposed to be called once
      • Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)

July 27, 2016

Updates

  • [DK] DB bundle serialization implemented

Items to Discuss 

   def vtkInstance():
       self.vtkInstance = self

July 20, 2016

Updates

Items to Discuss

  • [TE] New patch method for python-wrapper example
    • A patch is a string template with self/input/output variables for patching a class method
    • Patches are stored in xml specification
    • Patches are assigned to methods in ClassSpec's
    • Used both for execution in vistrails and when exporting as script
      • Executed with "exec" when run in vistrails
      • Turns patches into natural calls in script
    • Wrapping VTK library no longer needed
      • Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)
    • Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)
    • Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)
    • TODO's
      • Add to export_as_script
      • Patch Modules that represent Python functions as well
  • [vistrails-users] Read data from SQL example?

July 13, 2016

Updates

  • [DK] bundles on sql-alchemy branch
    • Bundle, BundleMapping, and BundleSerializer
    • New BaseSerializers to deal with versions of directory, zip, db serializers
    • Can register the BundleSerializers with the base serializers
    • Translation between bundle versions
    • Bundle.add_object "just works" if an appropriate mapping was registered

Items to Discuss

  • [TE] Simplified running module using API for scripting example
    • API Module can be executed using compute method
      • Takes list of outputs to compute + inputs as keyword arguments
  • Status of general cell synchronization (vistrails-users)

July 6, 2016

Updates

  • [DK] Working on bundles

Items to Discuss

  • [TE] Using vistrails API in scripting example 1 (API with VTK) example 2 (DownloadFile only)
    • Call module in script using vistrails API
    • Can be used when no export method is defined.
  • [TE] Server news bug Critical?
    • [RR] No, unreleased
  • [DK] New bundle organization
    • BundleMapping contains individual mappings from raw objects to BundleObj
    • Must be separate in order to have in-memory, separate from serializer settings

June 29, 2016

Updates

Items to Discuss

  • [TE] Export VTK as python script
    • Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) example
    • Preludes (imports) still need improvements
    • Works with vtkInteractionHandler (clipping plane)
    • Resulting clipping plane is incorrect? (supposed to hide one side of the plane)
    • vtkRendererOutput Uses VTKRenderWindow as output
      • Should support different output modes?
      • Does not work with multiple outputs (Next cell is displayed when current is closed)
      • Uses wrapped vtkInstance (as vvtk)
        • Create new patch system with code that can be used both in vistrails and in a python script?

June 22, 2016

Updates

Items to Discuss

  • [TE] Fix for upgraded analogies (PR #1177)
    • Uses module location to find module remap (The best we can do right now?)
    • Also fixes remapped port names for connections to existing modules
    • May still fail on complex upgrades like adding a function on an upgraded module that has a different port name
  • [TE] Working on exporting wrapped libraries (VTK)

June 15, 2016

Updates

  • [TE] Added depth logic to python script exporter (full commit)
    • Fully implements connection merging and list looping

Items to Discuss

  • [TE] Making analogy work on upgraded workflows (#1175)
    • Analogy only works We need to use the non-upgraded action chain
    • But pipelines need to be valid (matcher uses module portspecs)
    • Tried to get upgrade remap from upgrade action, but this seems impossible
    • Use module location instead?
      • Location should be identical for one-to-one module upgrades

June 1, 2016

Updates

  • VisTrails 2.2.4 announced (finally)
    • 5 survey replies
    • 47 downloads last week

Items to Discuss

  • [TE] Export VTK as script (python-wrapper)
    • to_python_script
      • needs the pipeline (or at least the connections)
        • Should not generate code for any possible VTK method (exporter detects which ports is used in the source)
        • VTK needs to know connection order
        • Adding functions directly to the script results in nicer code
    • Add support for multiple input connections
    • Add support for list looping

May 25, 2016

Updates

  • Mailing-list fixed
  • Getting stats from the "server news" mechanism (via parser)

Items to Discuss

  • [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)
  • [TE] Union ports
    • Show union ports in pipeline (DEMO)

May 18, 2016

Items to Discuss

  • [TE] mailing list still down - Julio working on it
  • [TE] Improved union port
    • New schema 1.0.5 adds portitem.union string attribute
    • Works with pipeline view, parameter exploration view, and mashup view
    • Should work well for matplotlib
    • Use with VTK ports?
      • It uses [item_1,item_2] style ports usually with different tuple length.
  • [TE] Remove emacs dependency from db generate #1173

May 11, 2016

Items to Discuss

  • [TE] Union type #1172
    • GUI-based solution Branch union-port Demo?
      • Shows single port for all ports with same sort_key
      • Select specific type when adding/showing
      • Only one type should be visible on the module at any time
      • Making tuple parameters union types is probably too complicated
      • Should work with alternate specs like matplotlib (if sort_key is defined correctly)
  • [TE] Feature suggestion: Let connections connect to functions/parameters
    • Show functions and parameters as ports and toggle port visibility directly for functions and parameters
    • Use separate views for ports and functions (like before)
      • Allow rearranging of functions
    • Pros:
      • Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
        • We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.
        • Makes List module simpler
      • Connect to parameter in tuple
      • Functions that are visible on the module can connect to an actual function, and not just to the first one it finds.
      • Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.
    • Cons:
      • Interface more complicated and crowded
      • More difficult for users to grasp?
      • No immediate use case
      • Makes no union type for tuple parameters a bigger problem.
  • [TE] Publish user survey?
    • No response from vistrails release yet
    • Not announced on vistrails-users yet.

May 4, 2016

Updates

  • VisTrails 2.2.4 released

Items to Discuss

April 27, 2016

Updates

Items to Discuss

  • [TE] Wrapping Bokeh
    • New method type "operation"
      • Transforms Figure->Glyph->Output into Glyph->Figure->Output
        • Glyph sets function name and arguments as output
          • Not serializable, but should always be computed together with the subject of the operation.
        • Figure applies Glyph operation on itself
      • Useful for matplotlib?
        • matplotlib currently find actors by searching upstream?
        • But this is more difficult since matplotlib is not function-based.
    • New "union" port type
      • Turns "x(float), x(list)" into "x (float, list)" and uses first set value
        • Uses only first value in tuple
        • Bokeh have many properties that support (list, string, Float)
      • Does not work with depth=1 types (e.g. Float of depth 1)
        • Turn Depth 1 types with known type into lists?
  • [TE] New bugfix release
    • Finalize survey

April 20, 2016

Updates

  • Merged upgrade fixes
    • Now has infinite loop test
    • loop test now correctly loops 5 times
      • remap_module used wrong (current) package version when creating ModuleDescriptor
  • Fixed multi-step group upgrades (PR: #1168)

Items to Discuss

  • New bugfix release
    • Merge usagestats
    • Need to finalize survey

April 13, 2016

Updates

Items to Discuss

  • [TE] Chaining upgrades done (#1164)
    • Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?
  • [TE] Iterating handle_invalid_pipeline (#1165)
    • Not based on chain-upgrades-loop
    • Cherry-pick test to preserve source?
    • Need maxPipelineFixAttempts option?
      • The code is now smarter about recursion, so max attempts will "probably" never be reached.
    • Need debug messages?
      • Would need to be moved into handle_invalid_pipeline

April 6, 2016

Updates

  • [TE] Query View improvements
    • Done except for displaying "Version Info" and #1164
  • [RR] Working on finishing new interpreter (finally!)

Items to Discuss

  • [TE] Chaining upgrades (#1164)
    • Try latest upgrade first, then previous.
    • If all fail, select latest upgrade?

March 30, 2016

Updates

Items to Discuss

  • [TE] Query View improvements
    • Unified do_version_switch and get_upgrades as validate_version()
    • Problems with query view using its own controller
      • Fixed module info not being displayed for query results
      • Still no version info for query results
      • Still an issue with version query view not being updated

March 23, 2016

Updates

Items to Discuss

  • [RR] Usage reporting almost ready to go
    • Need to record whether specific features are used: paramexpore, mashup, query
      • [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation
    • How to record size of vistrail?
      • [DK] IdScope already walks over vistrail, use that
      • but total number of modules and versions doesn't give an idea of average size of individual pipelines

March 16, 2016

Updates

Items to Discuss

  • [TE] Upgrades in Query Mode
    • Now using code similar to do_version_switch
      • Uses handle_invalid_pipeline
      • Differences: Not using current_pipeline, flushes changes
      • New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)
      • Integrate to avoid repeated code?
      • Use this for query mode?
  • [TE] VisTrails 2.2.4 ready for release?
    • tej package updated. Others?
  • [JF] upgrades and the version tree
  • [JF] provenance in Jupyter
  • [DK] bundles: #1092

March 9, 2016

Updates

Items to Discuss

  • [TE] Fixing search mode
    • String query now matches all by default
      • This caused pipeline queries to never match
    • String query can search: User/Notes/Tagname/Time/Module name
      • Should we search module names by default? (Disabled by default because it is expensive)
      • Fixed notes search
    • '>' parameter queries in documentation but has been removed
    • Fixed module matches for upgraded pipelines (By not upgrading pipeline)
      • Upgrades still a problem
      • Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)
      • [JF] Allow search on past tags
  • [TE] Create new bugfix release
    • Finish querying
    • Path issue on windows
  • [DK] Look into sqlalchemy and new bundle branches

March 2, 2016

Updates

  • [TE] Fixed cache bug with port specs
  • [TE] VisTrails Next
    • Most packages now work
    • No qgis for Python 3

Items to Discuss

    • Always dereference symlinks? #1146

February 23, 2016

Updates

  • [TE] Fixed PythonSource 100% CPU bug
  • [TE] VisTrails Next
    • Updated NOAA Web Service examples
      • Now uses REST service, tabledata, and Google Maps
      • Web Service replaced with DownloadFile

Items to Discuss

  • [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.
    • New 2.2 release soon?
  • [RR] Anonymous usage reports
    • Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features #1154
  • [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?
  • noWorkflow - provenance for python scripts
    • Recent work on tracking dependencies
  • Also integrates ReproZip with Jupyter
    • mybinder
      • Use with ReproZip to automatically build Jupyter notebook interfaces
  • Daves comments on Jupyter integration
    • It is possible to identify cell order on the server side
    • Not trivial to track order of cells on client side
  • How does Galaxy workflows relate to VisTrails?
    • Packaging in Galaxy is hard
    • Galaxy for biomedical workflows, difficulty making it general
  • Action items
    • How difficult to track cell dependencies in Jupyter notebooks?
    • Use noWorkflow to track cell dependencies?
    • How difficult to modify mybinder to use ReproZip packages?

February 17, 2016

Updates

Items to Discuss

  • [TE] Fixed Group memory usage
    • Modules in groups are supposed to be freed when done executing
      • Prevented by logger that stored references to the modules (fixed, now using id(module))
      • Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)
  • [TE] VisTrails Next
    • Removed deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • PythonSources will be upgraded
      • Fixes for Python < 3
      • fixes for VTK < 6
    • TODOs
      • Remove RemoteQ
        • Port Hadoop to tej (Already have pbs)
      • Merge package wrapper
      • Finish SQLAlchemy branch
      • Finish new bundle branch
    • [JF] Will contact Jeff and Matthias about moving to Python 3
  • [JF] Query by Example is broken
  • [JF] VisTrails Data Cleaning Project
    • Use VisTrails to process and explore task-specific data while keeping provenance.
    • Remi's Idea
      • Use VisTrails as the backend in a Jupyter notebook
      • Overcome problems with executing cells in random order
        • Interesting research topic, but little interest has yet been noted
      • Interesting to study how such a notebook would be used (Is there notebook usage data available?)
      • Impose restrictions to notebook cell boundaries
      • It will only work for functional operations
      • Can we fix, or detect, non-functional (mutating) operations?
      • Analyse code and data flow and see if it mutates data in unexpected ways?
      • Mutable operations can be created in many ways in Python
  • Next meeting Tuesday 11:30

February 10, 2016

Updates

Items to Discuss

  • VisTrails Next
    • Test suite passes
      • Python 3 less deterministic, module upgrade order random.
    • [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...
    • All of our packages should run fine :)
    • VTK thumbnail comparison works
      • Rendering bug on some systems
        • QVTKCell now inherits QGLWidget
      • Old thumbnails had wrong height due to spreadsheetwindow being too small
        • Fixed by regenerating thumbnails
      • Silenced deprecation messages when generating modules
      • Many new classes in VTK 7
    • Remove deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • Will break existing PythonSources
        • Try to upgrade?
      • Old-style packages
        • Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
          • Read info.py without importing package?
      • Fixes for Python < 3
      • Fixes for Qt < 5
      • fixes for VTK < 7
      • Remove self port?

February 3, 2016

Updates

Items to Discuss

  • VisTrails Next
    • All tests but thumbnail comparison passes
    • Still issues with gui and more packages needs to be tested
    • PythonSource and MplSource upgraded with lib2to3
    • Keep contrib packages? Update?
    • VTK 7 is released
      • Supports Python 3
      • Will hopefully get into conda and others soon
      • Deprecates a bunch of modules
  • Travis builder for VisTrails Next
    • Runs!
    • Uses conda
      • Many levels of requirements
        • apt packages
        • conda packages
        • python packages in requirements.txt
        • python packages in setup.py
      • Can probably drop conda. Default builder does not have Qt5, but beta builder do!
      • But useful if we want to use conda in our own build process.
    • Lots of errors I do not get locally?
  • User Survey
    • Have you used any alternatives to vistrails?
      • Relevant? More options?
    • How often do you use any of these particular features in VisTrails?
      • Add/Remove alternatives?
    • How important would you rank the following potential improvements to VisTrails?
      • Add alternatives?

January 27, 2016

Updates

  • New interpreter
    • Working prototype soon
    • Looping not completely done (and port combination mode: pairwise/cartesian...)
    • Need to integrate with VisTrails
      • Compatibility with existing modules
      • Use log controller
      • Want to cache at multiple levels (whole stream/single element of stream)
      • Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)

Items to Discuss

  • User Survey
  • ZMQ failing on windows (vistrails-users)
  • [TE] Python 3 branch
    • Testsuite almost passes
      • Failing are unicode filenames, some users guide examples, and thumbnail comparisons.
    • Fixed issues
      • Signatures are now bytes (convert to string to show in widget)
      • Cannot compare different types, e.g., (None < 1)
        • PortSpec.depth and min/maxConn could be None. (Backport!)
        • cell.row could be None
      • Lots of unicode/bytes fixes to serializations
        • We have encryption implementations in the code! (keychain)
      • Lots of small fixes to package method differences
    • Current issues
      • logging widget empty?
      • version tree signals messed up
      • provenance view legend icons black
      • Text do not fit in some widgets
      • VTK thumbnail comparison fails
        • VTK Cells have some faces in wrong direction?
        • Because running in VM?
    • Installing on Ubuntu LTS/Latest Mint
      • Install latest matplotlib from pip
      • Install VTK 7.0.0rc2 from source
      • No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)

January 20, 2016

Updates

Items to Discuss

  • [TE] Backporting features from Python 3 + PyQt5 + Qt5
    • Merged unicode with master (unicode)
      • One unicode test failing
      • Problem saving to xml
    • Used python-future to run futurize stage 1 to update legacy (pre 2.6) code (unicode-futurized)
      • Updates print to functions 'from __future__ import print_function'
      • Updates exception calls to use 'as'
      • removes most usages of 'long' (mostly used unnecessarily)
        • Only needed for 32-bit support
      • Fixes relative imports (from __future__ import absolute_import)
    • Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 (unicode-futurized-qt5)
      • Updated all signals to new-style
  • If we go 2 to 3, this could break packages
    • means we should schedule any package format changes for the same time
    • e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* #1002)
    • any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)
    • putting package identifier as xml?
      • Python not so bad, plus some stuff in there is Python code (checking dependencies)
    • version the APIs?
      • RR planning on versioning vistrails_module.Module, no point versioning the whole package?
  • file formats on 2 to 3:
    • no real translation/upgrade, but non-ascii values that work on 3 won't load in 2

January 13, 2016

Updates

Items to Discuss

January 6, 2016

Updates Items to Discuss

  • Updating VisTrails dependencies
    • Successfully built Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7
      • VTK 7 is the first supporting Python 3 but is only in release candidate phase.
        • Next version of VTK 6 is rumored to support Python 3
        • There is a development branch of VTK 6 supporting Python 3
      • Can travis build this?
        • We can skip tests for WebEngine and VTK if needed
    • Python 3 showstoppers
      • No MySQL-python (sqlalchemy supports mysql through other drivers)
      • No suds (there are active forks but we could also drop support?)
  • Options for python2/3 support:
    • Interpreter could be python2/python3 based on packages
    • six layer works but not the easiest to maintain (e.g. in reprozip)
  • Drop support for win-64
  • Update copyright headers to 2016

Older meetings

Retrieved from "https://www.vistrails.org//index.php?title=Development&oldid=14736"