Difference between revisions of "Development"

From VistrailsWiki
Jump to navigation Jump to search
 
(763 intermediate revisions by 4 users not shown)
Line 3: Line 3:
</div>
</div>


== 2014 ==
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]


=== Mar 26, 2014 ===
== 2017 ==
 
=== October 30, 2017 ===


'''Updates'''
'''Updates'''
* Branches to be merged in: application-no-default-argv, export-cells, testsuite-module-imports, reset-cell-sizes-button (icon?), export-versiontree-dot, custom-colors
 
* [TE] Added branch control-parameters (used by list handling)
* [DK] PyQt5
** For setting hashed control options on modules (How to combine lists, while looping, caching)
* [RR] Running Python3 code
** Missing: Include in diff/merges/analogies?


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Documentation buildbot
** [TE] currently [https://www.vistrails.org/#Testing here]
* [RR] Last(?) migration issue: [https://www.vistrails.org/ticket/837 #837]
* VTK 6 support [https://www.vistrails.org/ticket/739 #739]
* spreadsheet export button in UV-CDAT
* Develop menu?


=== Mar 19, 2014 ===
* Installation:
** Use conda? (macOS hack to have .app bundle load conda-installed vistrails)
** Update conda recipe (pyqt4 restriction for now)
* Interpreter
* Release
* Integration
** hard to build same type of GUI in other apps that integrate core VisTrails (tend to use just the API)
** independent pieces of the GUI could be reused if they were more independent.
 
=== September 6, 2017 ===


'''Updates'''
'''Updates'''
* [DK] UV-CDAT #407 fix, version tree rewrites
 
* [RR] Fixes for scikit-learn, tables, usagestats


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Cleaning up test suite log
** Deprecated core.modules.constant_configuration imports
*** check whether the deprecations for vistrails.core.vistrails_module.VistrailsModule camel-case are similar
** KeyError (Ticket [https://www.vistrails.org/ticket/837 #837]), one of [DK]'s commits
*** [DK] to look at
** Skip modules not to be imported (or fix/remove)
** Python invalid drawable (Probably from mashups)
** Missing version 0.8 of package org.vistrails.vistrails.tests.upgrade (bug?)
* [TE] Add annotations to signature? (hash-annotations)
** for the streaming/automatic looping for new values on the modules
** could be any module
** label doesn't affect caching
** prefix, flag involved, add fields to module about looping
** schema change? problem is that don't want to have to recreate module each time
** backward compatibility
** new type of "control parameter"
* VTK 6?
** See Ticket [https://www.vistrails.org/ticket/739 #739]
* UV-CDAT #408: [RR] has worked on this
** support adding more formats to export
** decided to have single menu item, and put all available formats in the dialog
* Spreadsheet resizing bug
** if you resized the sheet, would reset to equal-size cells
** spreadsheet-resizing branch changed this to allow non-equal-size cells on resize
** has some math issues because doing this dynamically difficult
** button to resize to equal-size on branch
** updating the documentation
** [TE] can we store the locations more precisely? (problem might be if a header column is also resized)


=== Mar 13, 2014 ===
* Stripped-down version of VisTrails to showcase new features?
** include meta-vistrail, bundles
** hide mashups, extra widgets, latex mode, etc.
* Interpreter
 
=== August 16, 2017 ===


'''Updates'''
'''Updates'''
* [RR] Package dependency ordering issue ([https://www.vistrails.org/ticket/829 #829]); fixed(?), merge?
* [DK] meta-vistrail
** [DK] Yes, merge
** Working actions: normal action in vistrail, pruning version tree, tagging operations, and change
* [RR] Will merge in tabledata additions
** Interface
* [RR] New SQL package, using tabledata and sqlalchemy. Should be fully backward compatible(?)
** Check if Fernando has anything here from the Oracle workflow and database provenance
* [RR] Searching in vistrail and regex ([https://www.vistrails.org/ticket/373#comment:7 #373], [https://www.vistrails.org/ticket/779 779]); fixed?
** Error was hidden (entering '?' searched for '.*?.*' which was a valid regex but still didn't do what was expected)
* [TE] IPython console now shows <STANDARD OUTPUT> before printing from stdout/stderr
* [TE] Vistrail variables fixed
** Deleting a module with a variable did not remove the variable from the workflow
** Copying modules with variables now works


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Do not run modules upstream of submitted job
* Notebook
** jobs-use-signatures: Uses the subpipeline signature to identify jobs
* Interpreter
** works: if job is submitted, check it without running upstream (we don't need the parameters this time(?))
* Web visualization packages (e.g. bokeh)
** doesn't work in Map
** [https://github.com/VisTrails/VisTrails/pull/969 PR 969] has more details
** jobs-use-signatures_/connect-folded-module
* Pipeline templates
*** create a connection from folded module to InputList upstream
*** looped module can run upstream if it needs, else we don't
*** problem: the Map still needs the list, for length and type-checking
** Could we rearchitect the looping behavior?  Basically, just copy the pipelines and run them with each entry from the loop
*** Provenance could expand
* [RR] module-upgrade-recursion status? (not yet merged)
** [DK] to look at and merge (should be done)
* [DK] UV-CDAT Issues: [https://github.com/UV-CDAT/uvcdat/issues/407 #407] and [https://github.com/UV-CDAT/uvcdat/issues/408 #408]
* [RR] Thumbnail comparison bug: different references are selected on master and dont-use-modules-as-data! (pipeline results are identical) Problem with upgrade code?
** using different reference images
** [TE] to look at
** look at the console_mode, how it gets back the upgraded version id
** filed as [https://www.vistrails.org/ticket/831 #831]
* [TE] Vistrail variables
** Normally only one hidden module exist for each vistrail variable, but pasting a module with a vistrail variable will create a new hidden module. This does not break anything, but should be taken into account when modifying the code.
** When a vistrail variable module is created, an input named "value" is created. A bug with default values gave it the value "None". This is fixed. To make validation of existing vistrail variables work we need to check if the value is "None", but this would not allow "None" as a string.
** [DK] to look into this, we cannot have "None" there.
* [JF] NSF Big Data proposals (due in June)
* uuid status
** need to check tests and features outside of normal execution


=== Mar 5, 2014 ===
=== August 2, 2017 ===


'''Updates'''
'''Updates'''
* [DK] meta-vistrail work
** Extend Vistrail to MetaVistrail to support meta-actions
** Vistrail becomes less tied to workflows so it can also supporting versioning vistrails


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] New SQL package, using tabledata and sqlalchemy
** No upgrade possible? (output changed from List (of rows) to Table)
*** Should we make it a new package?
*** Will try to make backwards compatible
* [TE] Change pinning to always show pinned panels? ([https://www.vistrails.org/ticket/828 #828])
** Yes, it is a better behavior
* [RR] Package dependency ordering issue ([https://www.vistrails.org/ticket/829 #829])
** Will try to fix
* [RR] Merging multithreaded-interpreter and JobManager
** Need examples of JobSubmission
* [TE] Merging automatic-loops-streaming
** Try to merge with multithreaded-interpreter
* Wiki menu looks strange in Chrome
** [TE] Will try to fix


=== Feb 26, 2014 ===
* Interpreter
** Using dask?
** Streaming an issue
* Notebooks
** What is an output of a module? (need to expose a specific output port or the entire module)
** Unifies some of the work from the script-workflow code
 
=== June 28, 2017 ===


'''Updates'''
'''Updates'''
* Branch to be merged in: hidden-port-icon (v2.1)
* [DK] Testing UUID/SQLAlchemy branch
* [RR] tabledata-merge-tabdata: added tests, fixed issues (todo: json)
* [DK] Working on meta-versions
* [TE] Added package RemoteQ (BatchQ/PBS/Hadoop)
** would be nice to have pipeline and vistrail separate
** new Mac binary that includes everything here
** [RR] Then do we only have meta-actions, any change to the pipeline is a meta-action that adds a version that does the change?
* [DK] gmaps, climate proposal
** [DK] Yes?
* [RR] Notebook interface (Vizier) making good progress, figuring out cell outputs


'''Items to Discuss'''
'''Items to Discuss'''
* Climate proposal
** recommendation and best practices
** automatically segmenting the version tree, better modes of interaction
* [RR] tabledata
** selections, projections, joins
** writing out table as csv?
** much improved bugs
** [DK] should merge aggregate module int
** [RR] working on different json representations
* [TE] Hadoop configuration
** has package configuration now
** still have machine module to hardcode a configuration for a workflow
** http://www.vistrails.org/index.php/Hadoop_Package
* Data cleaning/repo:
** always need the vistrails engine to run things
** scripting interface


=== Feb 19, 2014 ===
=== June 7, 2017 ===


'''Updates'''
'''Updates'''
* [TE] automatic-loops-streaming works
* [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs
* [TE] Hadoop package:
** Merged BatchQ-PBS and hadoop into RemoteQ
* [DK] improvements to tabledata, gmaps
* [RR] java-pkg is done


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] automatic-loops-streaming
* [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)
** Add documentation
** Can designate in module description an output port
** Perhaps a matplotlib example where the plot updates as more data streams
** Value has to be serializable
** Do we want to add additional controlflow features?  
** Should use the OutputModes work to handle multiple modes, configuration, selection, ...
*** Implemented while module, but this needs to be improved because of annotations
* [RR] Is VisTrails with SQL usable concurrently?
*** annotations don't invalidate cache
** [DK] Probably not, demo from [TE] exists
*** could include annotations in hasher for cache updates
*** how does if module work?
** [RR] pairwise for three ports?
*** the list modules do this, but we may need an interface for this
** http://dev.mygrid.org.uk/wiki/display/taverna/List+handlings
* [TE] Hadoop package:
** Best way to separate task and configuration parameters?
** URI is different for different clusters
** use package configuration variables
** move from checkbox to allow more general configuration?
** use configuration file for this?
* [RR] working to merge multithreading code with job-monitor, look at streaming
* [RR] java-pkg
** works for some classes (call constructors, setters, getters)
** cannot call any methods from VisTrails, don't want to mutate
** can call these methods from your own modules, also insert code in Python into these packages
** may add in an interface for loading jar files and wrapping them


=== Feb 11, 2014 ===
=== May 10, 2017 ===


'''Updates'''
'''Items to Discuss'''
* [TE] automatic-loops-streaming - partially works
** Using a python generator for each streaming module
** TODO: Streaming Sinks
*** last module collecting values
** Logging/Feedback performance
** Demo?
* [DK] vtk-new-package
* [RR] java package: conflicts with dlls and jpype, going to work


'''Items to Discuss'''
* Identifiers
* [TE] Hadoop package:
* Bundles & DB Code
** a few installation issues, vistrails-hadoop pkg on git but was not accessible to public
* Interpreter
** same package for local cluster and aws? Yes, but need to change two lines, path to configuration files on server
* Merge / Versioning Version Trees
** could config files be set in a module instead or in the preferences?
* Python3
** directories are hardcoded in hadoop package, need to be parameters
* Workings of List
** anything that can change in different installations should be parameters
* Packages: Wrappers + new Versions (VTK7)
** can be hard to figure out what does change
* Script Conversion
* [DK] merge tabledata-merge-tabdata, matplotlib-add-helpers?
* Intermediate Outputs: reuse output module work?
** examples are not very useful, no example with multiple plotline modules?
** would be useful for both workflows (run to a particular checkpoint)
** Update matplotlib examples to better show off capabilities
** and useful for notebook interfaces
* [DK] package loading speed, kwargs
* SQL backend: does it allow for concurrent access?
** kwargs can be significantly slower
** Yes, real-time collaboration work for concurrent single vistrail access
* java issues:
** Should be able to access different vistrails at once without issues
** [RR] should work, just a few installation issues
** machine learning in python: scikit-learn (http://scikit-learn.org/stable/)


=== Feb 5, 2014 ===
=== April 19, 2017 ===


'''Updates'''
'''Updates'''
* [TE] automatic-loops
** PortSpec-based version now works
*** Connections show how modules will be iterated
** Streaming is tricky
* [DK] Examples, merging tabledata


'''Items to Discuss'''
'''Items to Discuss'''
* mailing list question
** no resopnse yet
* UV-CDAT: warnings when downsizing spreadsheet
** problem: warnings will pop up a lot
** keep offscreen cache of (x,y) locations in spreadsheet?
** cache pipeline instead of full rendered pipeline
* [TE] streaming with branches becomes interesting
** only want to update upstream branches once
** Taverna has streaming-style (pipelining)
* PythonSource deleting port?
* [RR] Java: possibly considering the matplotlib-style port creation versus java-style ports


=== Jan 29, 2014 ===
=== April 5, 2017 ===


'''Updates'''
'''Updates'''
* [TE] automatic-loops - Groups and Multi-level looping now works
* [DK] matplotlib-add-helpers: Rework the semi-automatic generators
* [RR] java-pkg: bringing in old code, spreadsheet still works, probably going for dynamic generation of packages
** looking at jpype versus pyjnius


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] str-format-module: in master. Merge in v2.1? Having to open the configuration window and click on a button is awkward
** [DK] watch out for dynamically created format strings, could cause inconsistency at compute time if the format string doesn't match the ports
** Go ahead to merge since similar problem with PythonSource
** What about using number of connections to enforce this? (set to 0 for config-style ports?)
* Streaming in VisTrails
** Q: how is parallelism working in VisTrails? A: both theading and multiprocessing, but separate branch
** for streaming, order of input and outut must be preserved
** module can generate a number of pieces, a module that collects needs to know when the stream ends
** need to preserve order of map operations and
** goal is to reduce the storage costs, want to make sure that we don't keep entire data in memory, don't generate the entire list at beginning
** hyperflow needs to know when a stream starts and when it ends: start and end token, hyperflow wraps data in a token
** Huy advises getting rid of the list and using tokens instead?
** interpreter can call compute-upstream and just tell interpreter to go upstream
** VTK streaming interpreter: might be in svn
** Remi: just be able to pass an object on the conection that knows the shape of the data and how to get next element (basically an iterator)
*** where do we call get-next? module in the collector calls
** main advantage of streaming is being able to do aggregation during the compute (otherwise, we're doing map-reduce)
** the iterator encapsulates the calls to the module's compute
** issue with logging? already rewritten for parallel, should work here
** Examples: triangle_area, image histograms
** Slowdown caused by lists? colors? logging?
** Schema changes: need to record depth, what type of semantics for combining port values
** Also probably means that we don't pass just raw values, need to wrap with depth and iterator
** Issues Huy raised:
*** issue with multithreading and streaming both enabled
*** triangle with modules: need to be careful about order with items coming from two different modules (e.g. BuildTable in Rémi's example)
* UV-CDAT


=== Jan 22, 2014 ===
=== March 8, 2017 ===


'''Updates'''
'''Updates'''
* [RR] Branches to be merged in: reload-disabled-package ([https://www.vistrails.org/ticket/714 #714]), ungroup-keep-disconnected-ports_/hybrid
* [RR] Added a looping indicator on modules [https://github.com/VisTrails/VisTrails/pull/1217 #1217]
* [TE] VisTrails 2.1.1 released
** This makes it easier to understand what's looping, no need to follow edges and count
* [DK] matplotlib, uuid work
* [RR] Refactoring CLTools for integration in Docker package
* [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Automatic Looping
* [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers [https://github.com/VisTrails/VisTrails/issues/1216 #1216]
** Using ListOf module as input will trigger the looping
** Can we move this up to the renderer modules?
*** Runtime type checking because we don't have a ListOfT class for each type.
** Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)
*** [DK] USGS interested in working with this
*** Add a merge module to make it clear where the iteration stops and ends
*** Have different type of ports
*** Remi to talk to Huy about streaming ideas
*** can we detect the iteration and draw those modules differently
*** decided that some fold/merge module would be useful, even if we don't always need to use it explicitly
* [RR] Ticking in matplotlib
** formatters, locators, tick scale
* [RR] Sort tickets? There are [https://www.vistrails.org/report/18 issues for the 2.0 milestone] and [https://www.vistrails.org/report/17 issues that affect released versions]. Tickets assigned to 2.1 are [https://www.vistrails.org/report/19 here].
* [DK] To contact USGS about the new persistence package and get feedback
* [DK] Merge weather and subway examples to use new tabledata package
* Look into pandas http://pandas.pydata.org/
** could be helpful to support some of the table operations
** in the future, we may want to support it for data analysis and stats (but for this, additional dependencies are required)


=== Jan 15, 2014 ===
=== February 24, 2017 ===


'''Updates'''
'''Updates'''
* [RR] improves-logging:
* [RR] Rich-text module documentation ([https://github.com/VisTrails/VisTrails/pull/1210 #1210])
** Pass exceptions to it directly for single line, or traceback.format_exc() for traceback
** Went over packages and fixed format; maybe we should have a test to validate RST?
** WARNING level now printed in the console by default, -V 1: INFO (log() calls), -V 2: DEBUG (debug() calls)
* [RR] Single-instance shutdown with <tt>--remote-shutdown</tt> ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])
** Messages view now shows whatever is selected, no matter the console log level
* [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment
** Python warnings get captured
** Use warnings.warn(..., category=VistrailsWarning|VistrailsDeprecation) to warn once
** Backport to v2.1? (so deprecation warnings actually show up)
*** -> Yes, include in v2.1.1
* [TE] Hadoop package documented here: http://www.vistrails.org/index.php/Hadoop_Package
* [DK] UV-CDAT, LSU


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] input-module-no-subclass: input ports of type 'Module' shouldn't accept any output? These are used for 'self'->controlflow module connections...
* [RR] There are issues with list-depth
** controlflow modules have a function port that accepts a module
** Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)
** check for modules that might use Module for something else
** Issue with looping groups
** suggestion is to use Variant for any type being supported (StandardOutput, List, others)
** warns for now (doesn't actually disallow the connection until next version)
** -> David wants to look at this more
* [RR] ungroup-keep-disconnected-ports: alternative ungrouping, keeps pipeline disconnected and preserves InputPort/OutputPort modules
** ideally, want to edit in place, could try to emulate the edit subworkflow code here, just have to save things back to the original vistrail
** -> hybrid approach, materialize the unconnected ports
* Bugfix release? Would solve issues with server and parameter exploration. (also, logging improvement?)
** [TE] Also fixes to workspace
* [DK] font issue for OS X 10.9
* Look at Java support in Java?
** can we leverage Rémi's past work here? on Weka
** pyjnius: cannot subclass a Java class in Python, should be able to create and call out to java classes
** issue with Jython is the interface
** also had java-based spreadsheet
* also look at moving VTK to a matplotlib style generation


=== Jan 8, 2014 ===
=== January 25, 2017 ===


'''Updates'''
'''Updates'''
* [TE] Hadoop package: Added -combine option for [http://www.vistrails.org/index.php/Cs9223_Mapreduce_Assignment Cs9223_Mapreduce_Assignment]
** tricky to set up the package still, need documentation to understand how to assemble workflows from scratch
* [RR] Logging improvements underway (branch 'improves-logging')
** Fixes some logging gotchas
** Better exception printing (pass them directly to critical|warning|log)
** WARNING is now the default level for the console (only CRITICAL displays a popup)
* [RR] Alternative Mac .app ([https://docs.google.com/uc?id=0B3ucPz7GSthBZG5TS2pXUVdUWEk&export=download example dmg])
** Standard build of Python (currently as a Mac .framework)
** Have to relink libraries with install_name_tool
** Seems to work
** Advantages: standard, full build of Python; pip will work
** Last problem: if Qt is installed, it gets loaded and conflicts with bundled Qt (I missed a dylib link)
* [DK] Work with USGS, refactoring version trees
'''Items to Discuss'''
* Colin's feedback on loops is that it has taken some time to wrap his head around what is going on
** what does Taverna do? automatically determines when a collection is being input and does iteration over the input and determines what output looks like
** look at what this would take, figure out what the best way to attack this ([[Automatic loops]])
* Colin also asked about use of persistence in another module


=== Jan 3, 2014 ===


'''Updates'''
* [TE] VisTrails 2.1 released
* [DK] UUID branch, USGS work, upgrade recursion
** van Wijk on tree vis, and Buneman on XML Updates
'''Items to Discuss'''
'''Items to Discuss'''


== 2013 ==
* Scaling VisTrails
 
* sql-alchemy, uuid, bundle branches/merges
=== Dec 25, 2013 ===
 
* No meeting, holiday break


=== Dec 18, 2013 ===
=== January 11, 2017 ===


'''Updates'''
'''Updates'''
* [DK] USGS module-suspended branch, still concern about usability for loops
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]
* [TE] Added mac binary testing to buildbot
** Working on windows script
* [BB] Started work on separating uvcdat gui from vistrails (use git submodule)


'''Items to Discuss'''
'''Items to Discuss'''
* 2.1 release date (2013)
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails
** ALPS integration
** Have homemade workflow system based on R scripts and symlinks
** Would like to release in 2013
*** Each steps has multiple sets of parameters
* [DK] Updating signatures during execution? (e.g. in loops)
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step
* [DK] merge rename-api?
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code>
* [DK] log schema in master
** VisTrails improves provenance capture and makes editing pipelines easier
** 1.0.4 schema on master
*** Need better support for aggregation (groupby operation might be good enough; pandas?)
* [DK] upgrade logic (where to recurse)
** remap method:
* [BB] uvcdat packages
** starting to separate uvcdat from vistrails.git, uv-cdat packages
** try to generalize the package list to support more package locations
* [BB] Disabling provenance
** maybe focus more on scripting (more a new to have scripting work
* [BB] VTK 6
** see migration guide


=== Dec 11, 2013 ===
== 2016 ==
=== December 14, 2016 ===


'''Updates'''
'''Updates'''
* [TE] Merged job-info into master
* [RR] Single-instance code: over-engineered, but will work
** Updated PBS and hadoop to use job-info
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)
* [RR] dont-use-modules-as-data: works, awkward in a limited number of cases (two classes instead of one), brought up some issues (listed on [https://www.vistrails.org/ticket/804 #804])
** Also, [https://www.vistrails.org/ticket/805 #805]


'''Items to Discuss'''
'''Items to Discuss'''
* [DK] Job submission in 2.1
* Email from Matt Dirks: combobox should work, ask for more code?
** two modules are executing before the workflow stops (in "Stop on Error" mode) [https://www.vistrails.org/ticket/806 #806]
* Email from marin.nl:
** job info in 2.1.
** looks like they would benefit from new interpreter
* [RR] remove-socket: change is there, improves single-instance handling, but still race condition
** how to store their consolidated provenance when they exchange .vt files around?
** creates a file (a socket) on which the new instances communicate with new one
** if VT crashes, the file stays behind even though no one is connected so next startup will fail
** check process id?
* interaction provenance: need API, how to best capture this
** concern about the interaction handlers
* module upgrades: [DK] to work on
* mashup upgrades: [TE] to look at to determine next step
* webServices is deprecated: we should get rid of this


=== Dec 6, 2013 ===
=== November 30, 2016 ===


'''Updates'''
'''Updates'''
* [TE] VisTrails 2.1 ready - waiting for updated ALPS version


'''Items to Discuss'''
'''Items to Discuss'''
* [DK] UV-CDAT Meeting Update
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])
* [RR] Pending examples on new-matplotlib-pkg branch
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])
* [RR] Module types (after discussion on [https://www.vistrails.org/ticket/798 #798])
** Still improving it
** We use Modules to represent both computations and types
** Want to return something more structured to the client (success/fail code, stdout, stderr)
** Sometimes, objects of the type are instances; sometimes not (Integer and List vs File and vtkActor)
** Should be usable without Qt
** Passing down Module objects is dangerous (keeps references to moduleexecution, upstream modules, logger, even interpreter)
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason
** Keeping references that we don't need, references to pipeline, etc., memory concerns
* [DK] Working on batch execution using console_mode
** some modules (Map, If) may need executing infrastructure
* [DK] Settings aliased parameters when using core.api?
** note that anything accessed downstream should be a "data field" not a core Module field
** Remi to try to make change ([https://googledrive.com/host/0B3ucPz7GSthBVDV5Z3dTRklwclE/ test here])
* [TE] What to include in next vistrails version (the multiprocessing update?)
** multiprocessing?
** unicode support?
** uuid identifiers?
** Improvements to startup, configuration, parameters?
** start working on merges, may use other branches first, but we need to try to start integrating
* Last few changes for 2.1?
** XSL module? ([https://www.vistrails.org/ticket/292 #292])
** Better handling for disconnected single-instance socket (WIP, [https://www.vistrails.org/ticket/793 #793])
* py2app?
** create app bundle using virtualenv?
** disk space is about the same
** try on build machine (10.7)


=== Nov 27, 2013 ===
=== November 16, 2016 ===


'''Updates'''
'''Updates'''
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Fixing bugs for 2.1
** Synch release with ALPS 2.2


=== Nov 20, 2013 ===
* Email from Pieter: executing pipelines with parameters from command-line using single-instance
** Seems to work, main instance gets full command-line and alters configuration temporarily
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful
** Actions:
*** Can add switch to kill main instance
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?
 
=== November 2, 2016 ===


'''Updates'''
'''Updates'''
* [RR] Branches to be merged in: tuple-fixes ([https://www.vistrails.org/ticket/224 #224]), for-module ([https://www.vistrails.org/ticket/282 #282]), richtextcell-rtf-support ([https://www.vistrails.org/ticket/789 #789], needs pyth lib), ''remove-lockmethod ([https://www.vistrails.org/ticket/791 #791])''


'''Items to Discuss'''
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]
* [TE] Fixing bugs for 2.1
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch
* [DK] Ordered connections, scripting ideas
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)
* [DK] Enforce connection cardinality?
* [RR] Documentation/API effort? (improve-api, parameter-configuration-api, rename-api)
** [DK] look into merging master
* [BB] Speeding up vtk init
** Store auto-generated results in a file in the vistrails user directory?
** [DK] No, run the generated spec once and store it until VTK version is updated
*** Auto-generate python files for the vtk package, which then loads like any other package
** [BB] Profile vtk load and unload
* [RR] Parallel multithreading
** base functionality is there
** working to limit data movement, persistent file tie-in
** the JobSubmission package works to run later jobs
** partition workflow to take into account dependencies, right now, just push modules out when there are extra engines
 
=== Nov 13, 2013 ===
 
'''Updates'''
* [RR] Branches to be merged in: improves-module-doc ([https://www.vistrails.org/ticket/426 #426]), remove-persistence_exp ([https://www.vistrails.org/ticket/703 #703])
* [DK] Scripting support (parsing python source to guess ports), translate python functions to VT modules


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Delete old branches? [[Development Branches#To be deleted?|Development Branches#To be deleted]]
** Yes, this is fine, commits are preserved (just not named any longer)
* [RR] matplotlib figure/plot modules problem ([https://www.vistrails.org/ticket/396 #396])
** We are using matplotlib backwards
** try to defer code execution until we hit MplFigure (just save code in a buffer)
* [BB] Add uvcdat branches to github
** Question about whether to make vistrails a submodule, move the repository to github?
** Do we create a separate repository here?
** [BB] to talk to Claudio about this
* [TE] What to include in 2.1
** Fixing all tickets would delay the release. Some of these are present in 2.0 so it would still be a good release.
** Suggest making feature releases (2.2, 2.3, ...) for the working branches that are not ready.
** (updated with (status) as of 11/20 -- RR) (11/26 -- TE)
** '''[https://www.vistrails.org/ticket/224 224](fixed)''' tuple objects - move to 2.x?
** '''[https://www.vistrails.org/ticket/248 248](postpone)''' multiple 1D transfer function - priority?
** '''[https://www.vistrails.org/ticket/277 277](fixed)''' & '''[https://www.vistrails.org/ticket/722 722](fixed)''' pipeline validation bug - priority?
** '''[https://www.vistrails.org/ticket/282 282](fixed)''' fold flexibility
** '''[https://www.vistrails.org/ticket/396 396]''' matplotlib figures (see above)
** '''[https://www.vistrails.org/ticket/490 490](postponed)''' add label via api
** '''[https://www.vistrails.org/ticket/581 581](reprod?)''' latex: clicking figure in pdf - reproducible?
** '''[https://www.vistrails.org/ticket/652 652](postponed)''' helper method for function upgrades - move to 2.x?
** '''[https://www.vistrails.org/ticket/670 670](fixed)''' Traceback in configuration window - easy fix?
** '''[https://www.vistrails.org/ticket/699 699](postponed enh.)''' parameter-configuration-api ready for 2.1?
** '''[https://www.vistrails.org/ticket/709 709](fixed)''' & '''[https://www.vistrails.org/ticket/712 712](postponed)''' abstraction bugs - move milestone to 2.2? (use uuids) - no, these are more simple ones
** '''[https://www.vistrails.org/ticket/710 710](postponed)''' Module configuration widgets should be default-aware - fix for 2.1?
** '''[https://www.vistrails.org/ticket/736 736]''' update documentation
** '''[https://www.vistrails.org/ticket/754 754](question)''' usersguide controlflow - fix Conditional module
** '''[https://www.vistrails.org/ticket/755 755](postponed)''' new persistence package
** '''[https://www.vistrails.org/ticket/762 762](tbm)''' import sys package
** '''[https://www.vistrails.org/ticket/771 771](fixed)''' create schema if missing?
** Also see [[Development_Branches]]


=== Nov 6, 2013 ===
=== October 5, 2016 ===
 
'''Updates'''
* [RR] Branches to be merged in: disable-thumbnail-test-old-vtk ([https://www.vistrails.org/ticket/764 #764]), cacheable-controlflow ([https://www.vistrails.org/ticket/778 #778])
* [RR] Added a level to the LoopExec structure (LoopExec/LoopIteration), <strike>doesn't work, need help ([https://www.vistrails.org/ticket/774, #774])</strike>
* [TE] job-info done - ready for integration with other branches
* [RR] to be closed: [https://www.vistrails.org/ticket/757 #757] [https://www.vistrails.org/ticket/683 #683]


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Data movement with multithreaded-interpreter
** Right now, single module (opt. Group) sent with inputs, outputs & provenance sent back
** Input/output serialized with pickle
** Doesn't work with e.g. File (should allow modules to customize?), might be very slow (send multiple modules to minimize data movement?)
** [JF] ReproZip does some rewriting to change filenames
** Implement the simplest solution.
** For now, look at fixing this for simple File modules
* [TE] Update ticket milestones and sort [http://vistrails.org/index.php/Development_Branches branches] by milestone
* [TE] 2.1 release?  Specify what we want to put in the release
** Windows 8 build machine crash
** [TE] Go over open questions, distill what has been fixed
* [BB] UV-CDAT Animation issues
* [BB] ClimatePipes UI work, minor fixes and features
* Try to decide features, release data on 2.1 for next week.  Look at tickets, try to determine multi-threaded and command-line/documentation progress.
=== Oct 30, 2013 ===


'''Updates'''
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]
* [RR] Rewrite of the logger ([https://www.vistrails.org/ticket/774 #774]) nearly completed (loops haven't been tested extensively).
** unique ids
** It looks like the test suite doesn't really cover that
** bundles
** Then multithreaded-interpreter can be stabilized
** GUI fixes
** job-info?
** wrapping updates
* [DK] Further work on configuration/preferences
** python3
** Make it easy to add preference settings to GUI
** interpreter?
* [TE] job-info
** Fixed issues with job identifiers
*** Can not use module signatures
** Working on examples


'''Items to Discuss'''
* command-line input parameters to provenance
* [RR] Some tests [http://128.238.102.101:9050/builders/fedora17-64-v2.1/builds/24/steps/TestSuite/logs/stdio] [http://128.238.102.101:9050/builders/fedora17-64-master/builds/180/steps/TestSuite/logs/stdio] fail on Fedora (both v2.1 and master) but pass everywhere else, anybody knows why? (added [https://www.vistrails.org/ticket/775 issue 775])
** [DK] we don't create a persisted workflow from those input params, i think
* [TE] Signatures are not always unique---can have two different groups, do not depend on upstream?
** [DK] these are already defined as aliases so we treat them differently form a normal input param
** Groups that are duplicated in the same pipeline have the same signature, execute only once? [https://www.vistrails.org/ticket/765 issue 765]
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases
** Does looping corrupt the cache? (e.g. with Map) (it does, added [https://www.vistrails.org/ticket/777 issue 777])
** Do the vistrails variables values get stored in provenance? (what if values change?)
** Extra ModuleSuspended?
** Rémi showed some bizarre beahvior with the same subworkflow outside a group and one in a group
** Two options to make this uniform:
*** Always compute multiple copies when they appear in the same pipeline
*** Issue here is that the cache only contains one entry despite there being multiple copies of the same subworkflow
*** Always use the cache (build it incrementally)
*** Problem here is that if we have something which isn't quite stateless (e.g. vtk), having no copy might cause only one copy of the output
* [DK] Cache replacement policy?
** Ben may have done something with this for UV-CDAT
** Issue was determining the size of objects in Python
* [TE] Different brightness of images produced for VTK examples on Debian?
** Different versions of VTK? 5.4.2 (bundled with Debian 6)
** Added [https://www.vistrails.org/ticket/764 issue 764]


=== Oct 24, 2013 ===
=== September 14, 2016 ===


'''Updates'''
'''Updates'''
* [TE] job-info
** Core functionality complete
** TODO: Testing, console mode, integration with batchq/hadoop/pbs
* [RR] Branches to be merged in: install_package_requirements, non-english-locale
** Yes
** non-english-locale - strange files in db


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] OSX 10.9 does not work with VisTrails beta
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)
** Has anybody upgraded yet?
** Did we know/contribute to this?
* [TE] Does control flow modules get unique signatures?
*** [DK] Yes. We wrote the text for the VisTrails chapter
** Log have extra "completed" instances of modules used in a Map.
** Drafted response
* [TE] Debian thumbnail difference
*** [DK] Looks good, will forward.


=== Oct 16, 2013 ===
=== August 31, 2016 ===


'''Updates'''
'''Updates'''
* [RR] Branches to be merged in: install_packages_requirements ([https://www.vistrails.org/ticket/694 issue 694])
* [RR] Went over vgc.poly.edu/vistrails.org config:
* [DK] Rewriting startup and configuration
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
** unify command-line options that match configuration options
** Fixed usersguide scripts (won't disappear again)
** reduce number of single-character flags (e.g. "-U")
** Moved some websites over to HTTPS
** push startup into db to enable translation for future versions
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]
* [TE] New [http://128.238.102.101:9050/buildslaves/debian6-64 test system] for Minimum Requirements running Debian 6 (Squeeze)
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"
* [TE] job-info: Work in progress, core functionality implemented
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Support for spawned VisTrails instances (multithreaded branches, parallelflow)
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
** used to use local configuration which is problematic
** Get rid of more services? (svn, trac, ...)
** just use configuration that always add packages
* [RR] Notebook execution works. How do we support more languages (than Python)?
** Sometimes need to reneable the spreadsheet so that certain modules (e.g. VTKCell) are available.
** Need: detection, metadata reader, code to input values
** [TE] Perhaps this could support the VisTrails server too?
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?
*** Server needs to be looked at because it seems to be out of date
* [TE] Update crowdlabs for 2.1? At least start testing
* [TE] New [http://128.238.102.101:9050/buildslaves/debian6-64 test system] for Minimum Requirements running Debian 6 (Squeeze)
** [http://www.vistrails.org/index.php/Building_From_Source Minimum requirements] have been updated and are based on this system
** A few bugs fixed, still an issue with [https://www.vistrails.org/ticket/759 setPlaceholderText does not exist in Qt 4.6.3]
*** Should be able to replace this with setText in cases where Qt is older.
* {DK] Command-line stlye
** subcommands? e.g. "vistrails batch", "vistrails paramexp" [TE] "vistrails job"
*** [TE] makes sense to have different commands
** support multiple vistrails? multiple versions? (e.g. vistrails /path/to/example.vt -v tag -v 12, /path/to/example.vt:12)
** package options (move spreadsheetDumpPDF, fixedSpreadsheetSize but allow flags (e.g. -P spreadsheet.outputType=pdf)
** GUI Configuration also needs to change-improve "Always, Never, "For this session"?
*** Always show the configuration options, possibly flag the ones changed with an icon
** Other changes? [[Rewrite Startup Usage]]


=== Oct 9, 2013 ===
=== August 17, 2016 ===


'''Updates'''
'''Updates'''
* [DK] Rewrite startup code
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.
** use db to better structure startup.xml
* [TE] Matplotlib and sklearn are exportable using module-to-script api
** default, persisted, current configuration
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.
* [TE] job-info
** Persist jobs to disk
** Cache finished jobs to disk
** Run from command-line
** Integrate with BatchQ/PBS/Hadoop/parallelflow


'''Items to Discuss'''
'''Items to Discuss'''
* [BB] VTK Startup Times
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
** can we run this beforehand?
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
* [TE] Feedback on [https://www.vistrails.org/wiki/WritingCommitMessages WritingCommitMessages]?
* [TE] Wrap-up notes
** Approved.
* [TE] Minimum Requirements
** Add build machine with minimum requirements?
** What Requirements?
*** Python 2.6
*** Qt 4.6
*** 5 years for others?
* [TE] Status of PostgreSQL support (Ricardo@vistrails-users)
* [RR] Matthias wants the archive to have choosable filenames (can be done but makes collisions possible)
** two separate features: persistent caching and archival of final results
** key part is probably the transformation between the two: want to make this easy to go from an exploratory workflow to one that works with the archive


=== Oct 2, 2013 ===
=== August 10, 2016 ===


'''Updates'''
'''Updates'''
* [DK] rename-api branch updated with documentation
** includes name changes to match underscore style
* [DK] module configuration widgets
* [TE] Examples in usersguide are now clickable
* [TE] Added registration of file types on Linux


'''Items to Discuss'''
'''Items to Discuss'''
* Parallel Design for UV-CDAT
* [TE] Python Wrapper and scripting
* New persistence package
** All changes merged into python-wrapper branch
** add UUIDs as identifiers for vistrails/actions so that each workflow could have a unique id
*** Tests pass
* [DK] When/if to merge rename-api?
*** Still need to update sub-branches
** Should be ok to merge into 2.1?
** Minor fixes
* [TE] Show pipeline or history view when opening a vistrail?
*** vtkPlot3DReader upgrade bug
** Either use configuration setting, or default to history view if vistrail contains tagged workflows.
*** Added tuple exporting
** Default to the history view for saved files but make a configuration setting
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]
* [TE] New commit message syntax
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
** #ticket/bugfix/feature 777 (on third line)?
**** Could be a bug in the old vistrail, but not sure
** why not Ticket: Bugfix: Feature: on last line, like Signed-off-by lines? or could be on any line really
* [DK] Subworkflows and uuids
** Go with last line and don't start with hash as this may conflict with comments (which start with a hash, too)


=== Sep 25, 2013 ===
=== August 3, 2016 ===


'''Updates'''
'''Updates'''
* [RR] I forgot to mention [[Development Branches]]
** [TE] Thanks, lets continue to use this and keep it up to date.
* [DK] Rework Developer API and Documentation
** Documentation: [http://vgc.poly.edu/~dakoop/tmp_vistrails_doc/api_documentation.html API] and [http://vgc.poly.edu/~dakoop/tmp_vistrails_doc/packages.html Writing VisTrails Packages]
*** Reorganized the "Writing VisTrails Packages" chapter; I could never find the text I wanted: [http://vgc.poly.edu/~dakoop/tmp_vistrails_doc/packages.html Updated Chapter]
*** Still work to be done
*** Use sphinx autodoc to generate real API documentation: [http://vgc.poly.edu/~dakoop/tmp_vistrails_doc/api_documentation.html API Docs]
** Rework the whole Module API but maintain old calls for backward compatibility:
*** setResult -> set_output
*** getInputFromPort -> get_input
*** drop "FromPort" on other similar calls
** Add vistrails.core.modules.config classes ModuleSettings, InputPort, OutputPort, ConfigWidgets to give kwarg capabilities
* [TE] VisTrails v2.1beta2 Released
* [TE] Linking to examples from usersguide
** Uses vtl files on server to point to local examples
** Works with both html/pdf
** Cannot use crowdlabs because it cannot link to mashups and parameter explorations
** .vtl file extension need to be set up for this to work
'''Items to Discuss'''
* [BB] Aashish and Utkarsh from Kitware may be interested in using VisTrails in another project of theirs, and they had a few questions
** How hard would it be to export a workflow as a python script to run independently of vistrails?
*** easiest if there is a 1-1 mapping from lines of source code to module creation/parameter steps
*** would be nice to know what the goal of the work is (is this VTK or ParaView related)?
** Is it possible to extract just the workflow part of Vistrails as a lightweight api with or without provenance?
*** probably can be done, but currently there is no "core.workflow" project so action and module objects are both currently in "core.vistrail".
*** would be nice to separate the workflow pieces but this is currently not done, if it happens, would like to include in VisTrails
** Allow folks from Kitware to push code to the VisTrails repo (which after code review ends up in master).
*** Sure.
* [RR] core.system rewrite: merge it in? ([https://www.vistrails.org/ticket/743 #743])
** executable_is_in_path() returns True or False, only returns True if executable is on the PATH (previously: returned executable name if on the PATH or in top directory)
** get_executable_path() looks for executable in PATH and top directory and returns full path (previously: didn't look in top directory, didn't return file with extension)
** Removes executable_is_in_pythonpath() (purpose unknown) and example()
** Trying to remove temporary_directory(): remaining calls look like bugs ([https://www.vistrails.org/ticket/744 #744])
** Can test branches on buildbot for other branches using the web API
* [RR] Meeting with Matthias: rework the persistence package
** [[Archive|Summary of meeting]]
** [DK] Dropping git is fine, but one of the reasons for the choice was the ability to pull/push files to different machines.
** [DK] Do the key-value indexes work for searching for time?
** [DK] Are there existing flat object stores (with hases) that we can use?
=== Sep 17, 2013 ===


'''Updates'''
* [TE] build notifications
** If you break the build you get notified
** New mailing list vistrails-build@vistrails.org
** Will send digests to vistrails-dev once tested


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Issues to be closed: [https://www.vistrails.org/ticket/413 #413 HTTPDirectory]


* [TE] New bugs found by Matthias
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]
** FileSink overwrite port FIXED
** Connection variables now lowercase
** Git username missing FIXED
** Can we use code as output variables?
** Persistent directories
*** E.g. "vtk_dataset_mapper.GetOutput(0)"
*** The main issue was fixed, empty directories possibly still an issue
*** Could mutate output when output method is only supposed to be called once
** Release new beta once these fixed?
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)
*** Yes.


* [RR] Logger problem with multithreading: where to store parent executions?
=== July 27, 2016 ===
** link between Group module in main workflow and modules inside the group
** create a structure to track this information
** make the modules themselves dictate the change, tell the logger when they have sub-executions


* [TE] adding/improving examples
'''Updates'''
** KEGG web service has moved to REST
*** Should we look into adding a REST package?
** Add more controlflow examples for new modules and add to usersguide
** Subworkflow, Group, Mashup, Parameter Exploration, aliases, vistrail variables - in usersguide, should we add examples?
** Packages without examples:
*** ImageMagick
*** qgis
*** rpy
*** sql
*** tabledata
*** vtlcreator
 
=== Sep 11, 2013 ===


'''Updates'''
* [DK] DB bundle serialization implemented
* [RR] Reworking logger because of issues and incompatibility with multithreaded-interpreter
* [TE] Updating Job Monitor to support serialized jobs
* [DK] Module/PortSpec APIs.


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Issues to be closed (no further work on them?): [https://vistrails.org/ticket/131 #131], [https://vistrails.org/ticket/413 #413], [https://vistrails.org/ticket/626 #626], [https://vistrails.org/ticket/684 #684], [https://vistrails.org/ticket/703 #703], [https://vistrails.org/ticket/708 #708], [https://vistrails.org/ticket/720 #720]
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]
** locale?
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches]  
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]
** Still need to do something nicer for the patch formatting
*** patch keys currently contain non-python-variable characters but this can be fixed
*** Need to replace string template with python variables that is replacable using RedBaron
**** Patches could then look like:


* [TE] Issues raised by Mathias
    def vtkInstance():
** IPython console does not receive console output FIXED
        self.vtkInstance = self
** Map module broken FIXED
** Persistent directories broken FIXED
** Variant type checking FIXED by making it optional
** Subworkflow upgrades FIXED?
** Dulwich not included in Windows binaries FIXED in next release
** Removing files in persistence fails:
*** We're rewriting history which causes issues with identifying versions by commit hashses
*** Delete is disabled
** It seems to be working for him now...


* [DK] Improve testing / code management
=== July 20, 2016 ===
** Are the VMs still running the test suite?  Are we seeing errors? Can we get notifications?
** Should we adopt a master/next/branches strategy like UV-CDAT uses?
** Strategies for ensuring all files are added to commits?
 
* [RR] [[VisTrailsWiki:About]] has an (outdated?) description of VisTrails but there is nothing on the main page
* [TE] New binaries for beta being completed
 
=== Sep 5, 2013 ===


'''Updates'''
'''Updates'''
* [TE] Hadoop
** Asynchronous hadoop streaming now works
*** Hadoop jobs detach using ModuleSuspended and persist when restarting vistrails
*** Uses a custom updateUpstream to execute Machine while skipping other modules if job is already running
*** Need to add widget for monitoring job output
*** Fetching job result files is not cached so it happens everytime
*** Demo?


'''Items to Discuss'''
'''Items to Discuss'''
* Multiple instances of VisTrails
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]
** There were some errors in the way messages were received and the error code propogated, should be fixed
** A patch is a string template with self/input/output variables for patching a class method
** One place where we exit() instead of returning code, want to make sure the correct shutdown code
** Patches are stored in xml specification
** Try to take the same path as the "error path" which shuts things down
** Patches are assigned to methods in ClassSpec's
* [RR] Requirement installation from packages ([https://vistrails.org/ticket/694 issue 694]): init.py is imported before the package_requirements() check
** Used both for execution in vistrails and when exporting as script
** agree that we should change this, new packages should do py_import in package_requirements call
*** Executed with "exec" when run in vistrails
** update release notes and manual to highlight this ("as of verison 2.1...")
*** Turns patches into natural calls in script
* [RR] Remove "creating new figure" code from MplFigureCellWidget for [https://vistrails.org/ticket/672 issue 672]
** Wrapping VTK library no longer needed
** [DK] Should be able to remove this, left over from trials to move
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)
* [RR] Remove persistence_exp package? (or at least, code?) ([https://vistrails.org/ticket/703 issue 703])
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)
** [DK] Should be able to get rid of, should be all in the current "persistence" package
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)
** Update GUI view for persistence to allow more meaningful searches
** TODO's
* [TE] Change app logic so that exit codes can be propagated correctly?
*** Add to export_as_script
* [DK] Fix defaults to be more universally known in the VisTrails code (see tickets [https://vistrails.org/ticket/708 708] and [https://vistrails.org/ticket/710 710])
*** Patch Modules that represent Python functions as well
* [DK] Defaulted ports in Module.inputPorts versus ModuleRegistry.module_destination_ports
** Module.inputPorts stores only connected or set ports (e.g. the user has done something to that port)
** Iterating over Module.inputPorts will thus not include defaulted ports
** This is good for packages like VTK where we do not want to call all of the methods that default these values since they were already defaulted by the constructor
** This is not ideal for package developers who expect to see all of the ports that have values set in self.inputPorts
** Compromise might be to have a new method in Module like getInputPortNamesWithValues()?
* [DK] TODO: history of groups/subworkflows and start conversation on design
** [RR]: really want to be able to edit groups in place


=== Aug 28, 2013 ===
* [vistrails-users] Read data from SQL example?


'''Updates'''
=== July 13, 2016 ===
* [TE] Hadoop: Making hadoop package use batchq for communication over ssh
** first version of package needed to connect across firewall
** need to make hadoop work, then reorganize
* [BB] UV-CDAT: Get past the last few bugs
** Release target is Sept. 1, or 7-10
* [RR] Finishing up the threading stuff
** looking to unite with the JobSubmission/PBS work
** main issue preventing merge in: [https://vistrails.org/ticket/685 matplotlib incompatibility #685]
* [DK] Parameter widgets
 
'''Items to Discuss'''
* [TE] 2.1beta binaries
** Upgrading identifiers was broken. Dave, is this fixed? Should we update the binaries?
*** Think tickets 698, 699 could be fixed for this
** Keep lung.vt although we cannot run it?
*** nice example of a branching tree
*** could try to use other data for this?
*** should we contintue to distribute this?
** Other issues?
* Remi: nested modules
** tie each module with a configuration
** putting group information into the pipeline_view widget
* Parameter config widgets
* While module not in 2.0 yet, have a branch but need other bug fixes to make it work
** incorporate this into 2.0 branch, but don't make separate release for this
 
=== Aug 21, 2013 ===


'''Updates'''
'''Updates'''
* [TE] Hadoop: cannot run client from outside poly
* [TE] PE and mashup module id upgrades now fixed
* [TE] Currently building 2.1 beta binaries
* [RR] Execution target selection UI done for the multithreaded branch; needs to port IPython code.


'''Items to Discuss'''
* [DK] bundles on sql-alchemy branch
 
** Bundle, BundleMapping, and BundleSerializer
=== Aug 14, 2013 ===
** New BaseSerializers to deal with versions of directory, zip, db serializers
 
** Can register the BundleSerializers with the base serializers
'''Updates'''
** Translation between bundle versions
* [RR] New log schema on multithreaded-interpreter (see [[Multithreaded-interpreter#Changes]])
** Bundle.add_object "just works" if an appropriate mapping was registered
* [TE] Upgraded hadoop package to latest version, will try to integrate with the poly cluster
* [TE] Need to fix PE and mashup module id upgrades
* [DK] Output modules and PE upgrades


'''Items to Discuss'''
'''Items to Discuss'''
* Branches to be merged: dont_stop_on_first_error, optimize-module (While module) (v2.0? see [[New features|all changes to the pkg]])
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]
* OutputModules questions: [[OutputModules]]
** API Module can be executed using '''compute''' method
* Package configurations with multiple instances of VisTrails (e.g. with iPython worker instances)
*** Takes list of outputs to compute + inputs as keyword arguments
** read-only mode that doesn't change configuration?
* Status of general cell synchronization (vistrails-users)
* Module remap upgrades: how to enable them immediately
** what type of remaps do mashups need? functions/parameters?


=== Aug 7, 2013 ===
=== July 6, 2016 ===


'''Updates'''
'''Updates'''
* [TE] VisTrails 2.0.3 released
* [DK] Much improved db tests
* [DK] Many bugfixes to translations


'''Items to Discuss'''
* [DK] Working on bundles
* [DK] More general parameter exploration?
** Unify some of the mashup constructions with parameter exploration elements?
* [DK] More general module outputs
 
=== Jul 31, 2013 ===
 
'''Updates'''


'''Items to Discuss'''
'''Items to Discuss'''
* [DK] Where to put code to store bundles; should this be versioned?
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]
** [TE] Yes, this makes sense, can address issues like mashups which didn't appear in earlier versions
** Call module in script using vistrails API
* [DK] Caching: vtDV3D has its own persistent module store so that underlying VTK modules do not need to be reconstructed each time
** Can be used when no export method is defined.
** this means that the underlying VTK pipeline need not be reconstructed when a parameter on one of the files is changed
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?
** does this actually speed things up? if so, can we generalize this style of caching where structure is the same and updating parameters on the structure can be done more quickly than reconstructing a new (sub-)pipeline
**[RR] No, unreleased
* [DK] UV-CDAT Developer Documentation
* [DK] New bundle organization
** document how to create a new package in UV-CDAT (simplest example to begin, pull in pieces of DV3D to showcase more interesting pieces)
** BundleMapping contains individual mappings from raw objects to BundleObj
** document pipeline_helper
** Must be separate in order to have in-memory, separate from serializer settings
* [DK] Coordinated spreadsheet cells (e.g. selection in one cell updates another cell)
** Jorge had a Coordinator module, VTK has some other coordination, DV3D may have some?
* [BB] UV-CDAT
** still looking at memory issues, using pysizer didn't seem to be picking up everything
Ω¸* [TE] Matthias has fixes for ALPS so we can build binaries


=== Jul 24, 2013 ===
=== June 29, 2016 ===


'''Updates'''
'''Updates'''
* [DK] sqlalchemy changes working, need more tests


'''Items to Discuss'''
'''Items to Discuss'''
* [EV] NASA work
* [TE] Export VTK as python script
** familiar representation, convert between workflows specified already (e.g. using pyxb like java xml binding)
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]
** NED Client: everything is Java, can run on local machine or remotely
** Preludes (imports) still need improvements
** has workflow editor, tasks specifiied in hierarchical lists
** Works with vtkInteractionHandler (clipping plane)
** each task can have and executable and task dependencies, specified in tabular view
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)
** different types of tasks (including loops, system, java)
** vtkRendererOutput Uses VTKRenderWindow as output
** scientists have shied away from using workflow tool directly
*** Should support different output modes?
** some scientists comfortable working at low level, running qsub directly
*** Does not work with multiple outputs (Next cell is displayed when current is closed)
** [JF] Look at UV-CDAT-style layer?
*** Uses wrapped vtkInstance (as vvtk)
** [EV] still need a way to control copying at the file level
**** Create new patch system with code that can be used both in vistrails and in a python script?
** workflow front-and-center versus configuration front-and-center
** [JF] VisTrails treats changes to parameters the same way changes to the structure of the workflow
** configuration tree with lots of parameters that can be changed
** only four workflow variables? (user, cwd, ...)
** current scripts could be broken into smaller components
** global variables on a per-workflow basis
** creates running instance view showing completed/running tasks
** can examine logs for each task
** can suspend, resume, or abort a workflow
** build coarser modules?
* [BB] UV-CDAT bugs
* [DK] versioning bundles? what is the common interface for all versions?
** common locator and bundle -> specific version serializes/deserializes?
** if so, bundle must be extensible
** translation between versions happens at bundle granularity?
* [RR] referencing modules in Groups


=== Jul 18, 2013 ===
=== June 22, 2016 ===


'''Updates'''
'''Updates'''
* [TE] Win32 version of ALPS still does not work
* [DK]  sql-alchemy, use-uuid branches


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Branches to merge? dont_stop_on_first_error, cltools
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])
** [TE] to test the cltools changes?
** Uses module location to find module remap (The best we can do right now?)
* [TE] Update on NASA work
** Also fixes remapped port names for connections to existing modules
** has script that he wants to send to PBS through VisTrails
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name
** issue comes up with PBS Pro (different versions of PBS)
** restore missing or failed jobs
* [JF] Microsoft using workflows in cloud
** data movement is a first-class operation, moving data and computation around
** also helper with resiliency
** vision seems to take things beyond crowdLabs-style
* [RR] Recording provenance in multiple machines (machine-wf-exec)
** Duplicate entries (part of module runs on local machine, part on remote), this is ok, but want to know these are linked
** Need more information in log about how execution occurred
** Push ip info into machine?
** have other information (name and/or MAC address)? privacy issues?
** allow users to configure machine name as a preference?
* [JF] have a wiki page for new features and bulleted list of items, things that changed, etc. so that we make sure to document these later
** [[New features]]
* [RR] parallel execution: choose where to execute each module, happens at runtime, want to allow user to control this
** where to store this information?
** store separately, but write each configuration to the log upon execution
* [BB] UVCDAT bug status
** 5 or 6 bugs left
** still looking at memory issues in UV-CDAT, seems to be a Python issue
*** read more on this? PySizer, Dowser
** grower bug needs significant redesign (need two outputs)
* [DK] uuid/bundle status
 
=== Jul 11, 2013 ===
 
'''Updates'''
* [TE] 2.0.3 release - Still waiting for reply from Matthias
* [RR] tabledata merged in
* [RR] Variant type checking #600
* [DK] UUID


'''Items to Discuss'''
* [TE] Working on exporting wrapped libraries (VTK)
* Demos of parallel flow and job submission
* UVCDAT bugs [https://github.com/UV-CDAT/uvcdat/issues?state=open]
** #221 Memory leak [https://github.com/UV-CDAT/uvcdat/issues/221]
*** could try to just delete the memory hogging piece internal in the VisTrails variable
*** caching in VisTrails: need to check size of python objects:
*** size of python object: http://code.activestate.com/recipes/546530-size-of-python-objects-revised/
** Make a .uvcdat dir (don't just use the .vistrails dir)
** [https://github.com/UV-CDAT/uvcdat/issues/230 #230]: Need to debug this
* Unique identifiers and configuration management
** looking at use-uuid branch again, updating to master
* [RR] Continuing on errors (<tt>dont_stop_on_first_error</tt>)
** Need to make it clear that there was an error (both in GUI and via API)
** able to get to results from each module via objects returned from execute
* Pausing workflows
** Interact using the progress dialog to cancel/pause
** multithreading doesn't fix everything because of UI drawing from packages (e.g. matplotlib)
* v2.1 release
** put together binaries, test
** anything left to integrate?
** hold off on UUID changes, mulithreading
** waiting on Matthias's changes for 2.0.3 release


=== Jul 5, 2013 ===
=== June 15, 2016 ===


'''Updates'''
'''Updates'''
*[TE] Implemented mashup animations
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])
*[TE] Eduardo at NASA is testing BatchQ-PBS.
** Fully implements connection merging and list looping
*[TE] Waiting to hear from Mathias about fixing Alps for v2.0.3 release.
*[TE] Testing BatchQ-PBS on the NYU HPC cluster.


'''Items to Discuss'''
'''Items to Discuss'''
* Eduardo's work
** Installing VisTrails (can download binary and then replace source with latest source code)
** Workflow that allows people to create sandboxes or tap into existing runs in supercomputing environment
** Scientists have specific models that use various parallel techniques that use different codes and libraries, may have to grab source code and compile
** Important to be able to execute things from scratch (compile, set up environment, and kick off run), all of this has depdnency graph
** Running means everything is staged and just using qsub
** Monitoring using qstats to see where you are
** Have to know if/when it makes it to the queue, which files were produced, model output
** machines outside security boundaries still have two-factor authentication
** HPC environment won't allow connections over 12 or 24 hours
** Running scripts needs to be supported
** VisTrails needs to be able to track the scripts and progress in workflow based on outcome of scripts
** Not every failure means the next module cannot execute, could continue in other directions
** [JF] Suggest looking at ModuleSuspended state and CLTools & contrflowflow packages
* Terrence's work
** Distributed computing, running processes of different machines
** VT lacked multithreading at module level, adds layer so don't have to patch
** implications of order
** [TvZ] intention to use multithreading in the core of VisTrails?
*** [JF] Yes
** can run modules on different machines, VisTrails just orchestrates remote processes
** [JF] Have availalble at sub- workflow level, would have to have something running that understands the subworkflow scheme
** [TvZ] machines have "crippled" versions of VT that permite this sort of execution, crippled meaning that only enough to run the module, makes calls back to the remote machines
** Not a solution for everyone (manage the distributed computing environment)
** Use tagging for different workflow versions
** Modues will store configuration
* Provenance in multithreaded execution?
** Fernando wrote for parallelflow package
** Have some support for sending python code to remote machines
=== June 26, 2013 ===
'''Updates'''
* [DK] SQLAlchemy changes added, basic write/read works
** need to check things like updates
** mysql is one of few DBMS that supports multiple selects at the same time, need to special case *_many_* calls
* [DK] Fixed bug in PortSpecItem serialization (new expandAction flag)
* [TE] Working on animation support for mashups, similar to that in crowdlabs.
* [BB] UV-CDAT: working to keep variables in cache (load variable now creates a pipeline)


'''Items to Discuss'''
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])
* [DK] Output types:
** Analogy only works We need to use the non-upgraded action chain
** Discussed before, but want to allow modules that produce output like SpreadsheetCell or StandardOutput to support a get_output() call that takes a mime-type and returns None or an output that matches the type.
** But pipelines need to be valid (matcher uses module portspecs)
** Could support, for example, inserts of textual values in LaTeX/wiki documents
** Tried to get upgrade remap from upgrade action, but this seems impossible
** Also allow InputModule to ensure that we can do data lineage-style queries?
** Use module location instead?
* Package flag on load_configuration should be removed
*** Location should be identical for one-to-one module upgrades
* Parallel execution: write an interface so that you could choose on a per-module basis which type of execution it can support
** parallelflow versus multiprocessing (xml serialization of pipeline versus pickling objects)
** advantage of xml is that large intermediate objects don't need to be pickled


=== June 19, 2013 ===
=== June 1, 2016 ===


'''Updates'''
'''Updates'''
* [DK] Bundles work
* VisTrails 2.2.4 announced (finally)
** basic architecture: bundle contains a set of VisTrails entities, storage-specific serializers provide support for reading/writing the bundle
** 5 survey replies
** serializers can register sub-serializers for each entity, sub-serializers could be added at runtime (e.g. for packages that wish to save to a vt bundle)
** 47 downloads last week
* [TE] VisTrails 1.0.3 binaries almost ready
** Troyer is looking into a problem with ALPS on win64


'''Items to Discuss'''
'''Items to Discuss'''
* [DK] Move DB support to something like [http://www.sqlalchemy.org SQLAlchemy]?
* [TE] Export VTK as script (python-wrapper)
** can just use core SQL expression work for now
** to_python_script
** makes cross-platform DB access much easier
*** needs the pipeline (or at least the connections)
* [DK] DB Updates would allow move of temporary vistrails to sqlite3 database
**** Should not  generate code for any possible VTK method (exporter detects which ports is used in the source)
** should make updates much quicker for auto-save
**** VTK needs to know connection order
** could consider updating database during runs == improved provenance
**** Adding functions directly to the script results in nicer code
* [JF] Why not just use XML fragments?
** Add support for multiple input connections
** Actions are monotonic
** Add support for list looping
** [DK] Survey of the "Web" seems to show that people don't like fragments
** [JF] Shouldn't be an issue
* Tagging
** Could we have a save/save as for tags (the tag moves down unless someone "tags as"
* [DK] UUID identifiers?
** Switch identifiers at same time?
** Allows easy merges
* [TE] Status of import_rewrite branch
** replace import statement to add "vistrails." prefix
** kind of broken in master right now
** new branch that has fixes for the
* [RR] Parallel execution
** controlflow issues
** change in update_upstream


=== June 12, 2013 ===
=== May 25, 2016 ===


'''Updates'''
'''Updates'''
* [TE] Initial wrapping for Madagascar
* Mailing-list fixed
* [RR] Fix imports for old-style packages
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])
* [DK] Bundles
** Can load old vt files using the new interface, saving with manifest works
* [RR] Python library installation
** Can use pip (defaults to distrib package manager)
** Works on Windows


'''Items to Discuss'''
'''Items to Discuss'''
*[TE] old-style imports
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)
** Did you test NumSciPy?
* [TE] Union ports
** Any ideas for PythonSource upgrades? Terminator "isosurface script" uses "core." imports.
** Show union ports in pipeline (DEMO)
** A few ideas:
** save the temporary package created in the look at call
** still an isue with python dependencies (better to have a graph than a list), otherwise when packages have same python dependencies, one package "owns" them, and the other doesn't have
** How to deal with PythonSource (__import__ comes from string exec)
* [RR, DK] Package states
** [RR] noticed that the code the preferences dialog uses to look at a package's dependencies, etc. uses a temporary object
** Means that any imports in __init__.py are not captured since the package was imported as temporary
** [DK] suggests that we have three states for packages: available, loaded, and enabled so that the look_at dependencies are captured but we can differentiate between packages that are loaded and those that are not
 
* [DK] New package development (dynamic or generator-based?)
** Noticed when looking at Madagascar wrapper
** Difference between VTK and new matplotlib, VTK builds the wrapper from underlying code each time VT starts while new matplotlib builds the wrapper from underlying code only at build time
** Likely to have better stability and consistency with generator-based
** Easier to incorporate new code in the dynamic environment
** Generator-based is more indirect


* [DK] Port types
=== May 18, 2016 ===
** Consistent style for referring to port types?
** Have old class-type method which doesn't work well for userpackages
** String-based method may be harder to track errors?
 
* [RR] Multithreading attempt
** what are the issues with multithreading
** uses a task system to track things
** for backgrounding thread, using future library
** since Python has GIL, can call setResult from different threads without worrying about memory
** old modules still run the same way
** can we get workflows running on a separate thread so the interface doesn't hang
* [TE] Survey
 
=== June 5, 2013 ===
 
'''Updates'''
* [TE] Added IPython Console
** Requires installing IPython from source
** Uses an updated PyQt interface
* [DK] Bundle/locator updates, bug fixes


'''Items to Discuss'''
'''Items to Discuss'''
* Workshop discussion
* [TE] mailing list still down - Julio working on it
** IPython console: required some pyqt changes
* [TE] Improved union port
** Packaging
** New schema 1.0.5 adds portitem.union string attribute
** Componentizing VisTrails
** Works with pipeline view, parameter exploration view, and mashup view
** Wrapping Madagascar (estimate the amount of effort)
** Should work well for matplotlib
* Pending feature branches:
** Use with VTK ports?
** import-rewrite
*** It uses [item_1,item_2] style ports usually with different tuple length.
*** Importing NumSciPy now works, but not unloading
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]
*** probably because global __import__ override
*** potential issue with __init__ and temporary package?
** automatic-conversion
*** e.g. Float -> Int
** <strike>multiline-strings</strike>
*** Already merged but there was a textedit parameter view issue (It works on Linux) [TE]
** tabledata-pkg
*** merge the old tabdata package in here
** apt
*** a lot of (bad) things can happen in apt, but under normal operation, things work
*** use pip or easy_install for python packages when possible?
** new-pyqt-api
*** May break third-party packages
*** deals with QString, QVariant
*** Requires latest version from git
*** Connect to running python (without separate kernel)
* Scripting support
** Python script <-> workflow
* [TE] v2.0.3 Release
** MySQLdb was missing
** Autoremove vistrails-single-instance file
** Fixed latex examples [ES]
*** crowdLabs needs to be updated for full support
** Fixes for JobView, save_vistrail_to_db, scipy.weave
** More?
* User survey
** Claudio suggested talking with power-users like Tom Maxwell
 
=== May 28, 2013 ===


'''Updates'''
=== May 11, 2016 ===
* [RR] 'tabledata' package: numpy arrays, CSV, data conversion
* [DK] GMapCell, TableCell, bundles


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Testing for db backend
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]
** Currently no automatic tests
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?
** Use sqlite, local/remote mysql
*** Shows single port for all ports with same sort_key
* Data conversion modules
*** Select specific type when adding/showing
** IBM WebSphere to convert image types
*** Only one type should be visible on the module at any time
** useful in scientific applications
*** Making tuple parameters union types is probably too complicated
** Converter base module class
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly)  
* New user survey?
** Look at old PDF, add new/old questions to a Google Doc
* [RR] new __import__ override
* [RR] Python 3: tricky (circular imports)
** Probably easier to keep


=== May 22, 2013 ===
* [TE] Feature suggestion: Let connections connect to functions/parameters
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters
** Use separate views for ports and functions (like before)
*** Allow rearranging of functions
** Pros:
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.
**** Makes List module simpler
*** Connect to parameter in tuple
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds.
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.
** Cons:
*** Interface more complicated and crowded
*** More difficult for users to grasp?
*** No immediate use case
*** Makes no union type for tuple parameters a bigger problem.


'''Updates'''
* [TE] Publish user survey?
* [TE] branches are merged, but there is some work to be done
** No response from vistrails release yet
* [TE] Images in thumbnails are now sorted by cell location
** Not announced on vistrails-users yet.
 
'''Items to Discuss'''
* [TE] Newline support for Strings
** We can use the Configuration Button to open a multi-line editor
** Do we want an optional small multi-line editor in the ports window as well?
* [TE] Unicode support
** Derek@CSIR is interested in using unicode symbols in labels, module names, and code.
** In a matplotlib source this works: xlabel(u'\u2211')
** We have a branch where saving to unicode works
** Full unicode support would require converting all strings to unicode u""-strings and making sure everything still works
** Supporting Python 3 is another option, problem is many packages don't support Python 3 yet.
*** The 'six' pakcage might be useful here http://pythonhosted.org/six/
* [TE] API tests fail on Linux, see [http://128.238.102.101:9050/builders/fedora17-64-master/builds/80/steps/TestSuite/logs/stdio here].
 
=== May 15, 2013 ===


=== May 4, 2016 ===
'''Updates'''
'''Updates'''
* [DK,RR] Locator changes (Rémi also added unicode support)
* VisTrails 2.2.4 released
* [DK] Persistence fix
* [TE] Demoed CrowdLabs on UV-CDAT meeting
** They want to create a web interface for UV-CDAT and may be interested in using CrowdLabs


'''Items to Discuss'''
'''Items to Discuss'''
* Feature branches to be merged in master?
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]
** new-matplotlib-pkg (includes pkg-id-changes)
** locator-filenames
** spreadsheet-resizing, tempfiles, tests
* [DK] Mac binary with scipy.weave causes doesn't have nice Mac binary update, mac_site.py embedded in site-packages.zip
** look into easy_install issues, would help solve some of the missing package issues
* Persistence bugfixes:
** how to tell different versions (add a prompt to save a comment when a file changes)
* [DK] Bundle updates:
** Create DirectoryBundle, ZipBundle, DBBundle
** Add manifest file to Directory/Zip bundles?
** Serializer base class, register serializer with bundle based on serialization type and object(s) serialized
*** For example, a LogDBSerializer loads/saves the log to the database and would be called for DBBundle
*** Packages subclass from Serializer and define how they read/write data (given base properties)
** XML Fragments (appending data)?
** Support different types of bundles (e.g. register a Local/PersistentFileSerializer to add files used in a workflow to make a self-contained bundle).
** crowdLabs support?
** checkboxes in the UI so that things can scale (allow users to select what is exported)
* [TE] NumSciPy and "vistrails." import issues
** Update NumSciPy?
** create run.py at top level to get rid of old import path?
** How good is the support for old packages?
** what about compute for PythonSource?
*** could wrap __import__
*** use upgrades
*** add prompt and warning before doing these upgrades
* matplotlib date plotting not working, shows number
 
=== May 9, 2013 ===


=== April 27, 2016 ===
'''Updates'''
'''Updates'''
* [TE] Mac binary: added support for scipy.weave
* [DK] Added better support for untitled locators
* [DK] More API Changes
* [DK] controller's current_pipeline_view is actually the pipeline view now (not the scene)


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Canceling VisTrails execution [DEMO]
* [TE] Wrapping Bokeh
** Implemented progress dialog similar to parameter exploration
** New method type "operation"
** Shows progress by counting the modules
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output
** Shows the executing module name
**** Glyph sets function name and arguments as output
** Cancel works by catching pipeline scene update signals
***** Not serializable, but should always be computed together with the subject of the operation.
** Users can enable module to be canceled by implementing progress reporting
**** Figure applies Glyph operation on itself
* [DK] Threading for pipeline execution?
*** Useful for matplotlib?
** Issue with Qt windows on different threads?
**** matplotlib currently find actors by searching upstream?
** In the middle solution where users could choose to run non-GUI workflows on a separate thread?
**** But this is more difficult since matplotlib is not function-based.
* [TE] UVCDAT Web meeting Monday
** New "union" port type
** Present crowdlabs/webgl
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value
* [RR] Spreadsheet ideas from discussions with Dean
**** Uses only first value in tuple
** Operations on multiple cells
**** Bokeh have many properties that support (list, string, Float)
** Select multiple cells and then perform same operation on all of them
*** Does not work with depth=1 types (e.g. Float of depth 1)
** Selecting L-shaped set of cells (textual entry A1:A3, D1:D2)
**** Turn Depth 1 types with known type into lists?
** Formula-type bar in spreadsheet (like Excel)
** Provenance issues here
** Synchronization between cells (look at common properties)


=== May 2, 2013 ===
* [TE] New bugfix release
** Finalize survey


=== April 20, 2016 ===
'''Updates'''
'''Updates'''
* [DK] Identifier switch
* Merged upgrade fixes
* [DK] matplotlib migration
** Now has infinite loop test
* [TE] VisTrails 2.0.2 released - 100 downloads already
** loop test now correctly loops 5 times
* [RR] has new utilities from constructing tests for testing packages
*** remap_module used wrong (current) package version when creating ModuleDescriptor
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] VisTrails 2.1 beta ready to go
* New bugfix release
** would like to include the changes on pkg-id-changes and new-matplotlib-pkg
** Merge usagestats
** should put the 'old_identifiers' in the schema
** Need to finalize survey
** add tests for new features
** what about reading 2.1 vistrails in 2.0?
** what happens when you load a version created using a future version of the package in a older version of VisTrails?
* [DK] core/gui coding: want to keep most logic in core
** want to be able to use packages like job monitor, etc. from core
* [DK] api/application logic
** currently have been moving logic from vistrails_window into api/application, does this make sense?
** should application call api or vice versa?
* [FC] Trying to gt Win8 installed
** Sent an email to Anup about this
** Also OSX 10.8?
 
=== April 24, 2013 ===


=== April 13, 2016 ===
'''Updates'''
'''Updates'''
* [RR] New version of 'parallelflow' module (IPython)
** Now supports remote engines through ipython profiles
** No ssh support
** We could look into PBS support
* [TE] 2.0.2 binaries ready
* [DK] API


'''Items to Discuss'''
'''Items to Discuss'''
*[TE] Support more build machines?
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])
** OSX 10.6? Emanuele might be able to build using her machine
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?
** OSX 10.7? The build machine version (The current build is falsely named 10.6)
** OSX 10.8? Need a new build machine
*** Need to ask Emanuele how other versions can be built on the build machine.
** Windows 8? May not need a separate binary but should be good to test on
*** Ask Fernando to set up a virtual machine on the build machine
* [TE] Changes in 2.1
** Renamed vistrails.py to run.py to avoid conflicts
*** Need to make sure documentation is updated
** Renamed the "packages" directory containing contributions to "contrib" to avoid confusion
* [DK] API/App Changes
** Dave will commit changes that makes the api clearer
* [TE] I suggested creating a new devel branch and a new workflow for committing changes. I will look in to this some more.


=== April 17, 2013 ===
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]
** Cherry-pick test to preserve source?
** Need maxPipelineFixAttempts option?
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?
*** Would need to be moved into handle_invalid_pipeline


=== April 6, 2016 ===
'''Updates'''
'''Updates'''
* [TE] Done with checking schema conversion and creating tests
* [TE] Query View improvements
* [TE] Started generating new v2.0 binaries
** Done except for displaying "Version Info" and #1164
* [RR] DAT: data provenance viewer implemented
* [RR] Working on finishing new interpreter (finally!)


'''Items to Discuss'''
'''Items to Discuss'''
* [JF] The LaTeX extension issues
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])
** uses the weather example
** Try latest upgrade first, then previous.
** [ES] related to CLTools package?
** If all fail, select latest upgrade?
* [TE] No known v2.0 bugs that need to be fixed
* [TE] What is left to do for v2.1beta release?
** are some bugs
* [DK] need to look at merging the new matplotlib package
** need to add MplScript module
* cannot run parallelflow on multiple machines?
** we should set features before releasing the v2.0 version
* have root-level API?
** move the logic to core/gui as needed, and have the root-level import depending on mode
** need to separate concerns as necessary
* try to make the switch to the org.vistrails prefix
** keep the old identifiers and have both point to the same package?
* three types of subworkflow-like pieces: mashups, subworkflows, DAT
** maybe more implementation-level differences
** can we comine the infrastructure for these?
** should we try to use a similar API?
** for visual compression, more of a conceptual change rather than create new group/ungroup each time
* synchronizing aliases and vistrail variables
** really want to have some unifying solution here
* including GDAL, etc. in binaries?
 
=== April 10, 2013 ===


=== March 30, 2016 ===
'''Updates'''
'''Updates'''
* [DK] Fixed the matplotlib cell resizing issue in current and new mpl packages.


'''Items to Discuss'''
'''Items to Discuss'''
* v2.0 Release:
** [DK] matplotlib cell size issue should be fixed
** Any reported bugs that are not fixed?
** add v1.0.3 -> v1.0.2 translation in the v2.0 release
* v2.1 Beta:
** What are the new features we should include?
*** auto-connection
*** visual port feedback (i.e. parameter set)
*** different port shapes
*** vistrails.-prefix
*** new matplotlib package?
*** mashup integration?
*** parameter exploration serialization
*** new schema for port spec
*** parallel flow, multiprocessing work?
*** what else?
*** add matplotlib examples inside the examples directory
*** add tests for the new features
** What needs to be checked before a beta is released?
*** want to make sure users that create vistrails in the beta can use them later
*** need to freeze the schema---are we happy with what is there?
*** need to check the schema v1.0.3 <-> v1.0.2 translations: are they robust?
**** [TE] I am checking by using the vistrail files on vis-7, but the update_db script needs to be updated.
**** v2.0 should support importing the v1.0.3 schema
*** add tests for things like schema
** matplotlib package
*** have Sunitha add some documentation about how to use the mpl package
*** still need legend updates, patch support, colorbars, script support
** DAT VTK issue with flickering


=== April 3, 2013 ===
* [TE] Query View improvements
** Unified do_version_switch and get_upgrades as validate_version()
** Problems with query view using its own controller
*** Fixed module info not being displayed for query results
*** Still no version info for query results
*** Still an issue with version query view not being updated


=== March 23, 2016 ===
'''Updates'''
'''Updates'''
* [BB] Working on UV-CDAT bugs, release mid-April
* [RR] Working on the VTKCell Overlay


'''Items to Discuss'''
'''Items to Discuss'''
* [DK] Better DOT_VISTRAILS management.  Why not have "modes" which correspond to certain package sets?
* [RR] Usage reporting almost ready to go
** USGS users have some requirements here (sometimes run with SAHM, other times without)
** Need to record whether specific features are used: paramexpore, mashup, query
** It's really annoying to have to update the enabled packages each time
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation
** That way, if I am interested in running without the spreadsheet, etc., I can just run in a given mode and when I want the spreadsheet and VTK, etc. back, I just switch back
** How to record size of vistrail?
** basically, just segment out the packages part of startup.xml and make multiples of these
*** [DK] IdScope already walks over vistrail, use that
** also add command-line flag
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines
** [TE} maybe ony specify only whether some packages are enabled or disabled?
** [TE} maybe just point to a different user-packages directory
** what about just making command-line flags to enable/disable packages
 
* [DK] Latest VTK version
** Has anyone tested how the latest version of VTK runs with VisTrails?
** Huy ran into an error about "no concrete implementation..."
** [TE] May just need to ignore the class causing this issue
** [ES] Seems like the 5.10 master also had this issue before the release but perhaps they removed the features
 
* [DK] Installing packages/python dependencies?
** [ES] can install distutils into the Mac binary?
** WIndows has a full python install so this isn't as much of a problem.
** Just have a menu-item in VisTrails that sets the right configuration variables for the installer (point to the setup.py that the user wants to install)
** Want to install to a VisTrails library directory that lives outside of the binary.
** How to deal with python dependencies for compiled pieces?
 
* [TE] Ryan Danks on vistrails-users: I've been trying to get Vistrails to work in parallel both by using the server instructions found in the documentation as well as by using the standard Python multiprocessing.Pool objects within my module. Neither of these approaches seem to work. Has anybody successfully run their Vistrails code in parallel? If so, how?
** Tommy to check with Fernando on whether ParallelFlow satisfies these requires
** Is ParallelFlow ready for release?
 
=== March 27, 2013 ===


=== March 16, 2016 ===
'''Updates'''
'''Updates'''
*[BB]Online/Incremental layout
** Added option to prevent any gaps in layers between connected modules. This in addition to preserving horizontal order seems to produce nice layouts from version to version.
* [RR] working on DAT: multiple variables per port working
** Also added variable number of inputs on List module
** Also did some changes on the controlflow package
** Also made some changes to the VT GUI as suggested by Juliana last week to the PythonSource and List modules
** Also updated DAT to use matplotlib plots and add conversion typecasts if variable types don't match


'''Items to Discuss'''
'''Items to Discuss'''
* [JF] Update to package API documentation
* [TE] Upgrades in Query Mode
** Need to write documentation with this in mind
** Now using code similar to do_version_switch
* [DK] Colin (USGS) asked about installing packages into VisTrails's python distribution.  What is the status of the easy_install work; are there other solutions?
*** Uses handle_invalid_pipeline
** need to talk to Emanuele/Tommy about this
*** Differences: Not using current_pipeline, flushes changes
* [DK] UV-CDAT Scripting.  Is there anyone who can take this on?  Charles is planning a meeting for early April to show what he would like to see for this.
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)
** Part of this is just designing meaningful modules (the wrapping side of things)
*** Integrate to avoid repeated code?
** The other side is what can be done to make the scripting interface more like a normal python script
*** Use this for query mode?
** Currently, the scripting interface just builds a workflow.  This is somewhat unintuitive if you expect that things act like a normal python shell...
* [TE] VisTrails 2.2.4 ready for release?
** Do we support partial execution as modules are added?
** tej package updated. Others?
** What to do if a call is not wrapped?
* [JF] upgrades and the version tree
** Perhaps a good starting point is the VTK example scripts.  I converted these to workflows by tracking all the calls.  Why don't we see if the scripting interface can convert these on-the-fly?
* [JF] provenance in Jupyter
** Also, I want to keep in mind numerical ops and numpy. Typing expressions is often much easier but we don't have a great way to represent this in a workflow. What should be done?
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]
** python has a built-in abstract syntax tree (ast) for itself that is user-facing. Could we use this to better translate scripts to workflows?
* [BB] Add auto layout to api calls?
** Possble flag for this so that the user can choose whether things are laid out or not
** Also try to use this for the auto-snap feature in the GUI, BUT need to be careful to animate this as users will not be able to follow what is going on otherwise.
** the layout code now runs without any GUI dependencies but can still use QFontMetrics (like for the Cmd+L call)
* VirtualBox issues on the testing machine?  Emanuele is trying to fix, may need to contact Fernando
 
=== March 20, 2013 ===


=== March 9, 2016 ===
'''Updates'''
'''Updates'''
* [TE] Save Camera (and similar) now works for unsaved and "saved as" vistrails using the new ensureController method
* [TE] DB Mashups merged into master
** working to upgrade the older mashups
* [TE] Added JobSubmission entry to FAQ
* [ES] Planning to have the binary ready for testing on Friday
* [DK] Work on matplotlib
* [DK] Help answer Charles's UV-CDAT questions


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Control flow modules:
* [TE] Fixing search mode
** Lists vs multiple connections
** String query now matches all by default
** Using 'self' ports on ungrouped modules
*** This caused pipeline queries to never match
** along each port to take a list or a single change a la Taverna
** String query can search: User/Notes/Tagname/Time/Module name
** try to write a new module to dynamically create a set of connections from a list
*** Should we search module names by default? (Disabled by default because it is expensive)
** do this in the interpreter? (need to modify the executing pipeline, not the specification
*** Fixed notes search
* [RR] Issue with "self" port connected (emailed Fernando)
** '>' parameter queries in documentation but has been removed
* [DK] Similarly, should we try to add some method for ordering connections?
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)
** Could display the order using a number overlay when a user moves the mouse over the port
*** Upgrades still a problem
** Then, selecting connection, could move up/down like illustrator...
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)
** Probably would require an additional attribute to be stored in the workflow
*** [JF] Allow search on past tags
** Rémi suggests mirroring Tuple here, (set number and type)
* [DK] Dynamic ports behind the scenes?
** Problem: Would like to have both stronger container types (e.g. List<String>) as well as flexible types (e.g. Float|List)
** Idea: Don't change anything about the current registry, but allow this at the interface layer and dynamically add the port to the module/registry when needed?
** For flexible types, could create myportFloat or myportList ports depending on the connection and then map these back to the same port before compute call...
** For container types, could only add a port type to the registry when it is actually used so we don't have to materialize everything beforehand.
* [JF] PythonSource/Tuple port types
** cannot type to search for port types -- have to type fast
** use a completer in the text field for this?
** drag and drop from module list?
* [BB] Online layout and gui-free layout
** Am close to a working solution that just uses the old layout but optionally preserves order of specified modules
** Having an issue with saving the positions, and strange issue with newly dragged modules vs modules loaded from file


=== March 13, 2013 ===
* [TE] Create new bugfix release
** Finish querying
** Path issue on windows


'''Updates'''
* [DK] Look into sqlalchemy and new bundle branches
* Juliana's visit to Brookhaven National Labs
** interested in climate data analysis on observational data (also light source data)
** need to have executino on parallel machines, not able to find batchq package
** batchQ seems to be pure python and should not be hard to install, need to specify which version to install, add to FAQ
** UV-CDAT Sandy demo: [http://www.youtube.com/watch?v=BFN0RzN1hSE&feature=em-share_video_user Sandy YouTube video]
* [TE] Saving mashups to DB works, but there is still a bug with tagging a mashup, the interface is redrawn while tagging and a new mashup version is created.
* [TE] Working on getting spreadsheet "to version" method work with unsaved vistrails
** Added an "ensureController" method for loading a view from a controller object
** Getting the spreadsheet to work with controllers is difficult because it uses the locator everywhere and still needs the locator for things like saving the spreadsheet.
* [DK] More progress on matplotlib updates
* [RR] Fixed the bugs on DAT, trying to bring in more examples from VTK
 
'''Items to Discuss'''
* It seems the ALPS package is not included in the Linux binary distribution. Is this a bug or is it omitted by design?
** [DK] Is it included in other distributions?  I thought ALPS provided its own installers that update VisTrails with the necessary support for ALPS.
** Ask Emanuele and/or Matthias about this to see what the current setup is
* We used to have a scripting window in VisTrails and also in an early version of UV-CDAT. What is the status of the support for shell-like commands in VisTrails?
** [DK] core.api has some support for commands like this.  See the [http://www.vistrails.org/index.php/FAQ#How_do_I_create_and_execute_workflows_in_a_standard_python_shell.3F FAQ]
* [DK] How to best deal with importing Modules from other packages.  For example, if I want to subclass a Module in matplotlib, I can use the registry (ugly, see [http://www.vistrails.org/index.php/FAQ#How_do_I_access_modules_from_other_packages.3F FAQ], or I could try to directly import from vistrails.packages (e.g. import MplFigureManager from vistrails.packages.pylab).  While the second method is nicer, it is also not very robust because
# it does not enforce package dependencies.  I can import from pylab even if that package has not been imported.
# It does not work great for a package that might be in packages or $DOTVISTRAILS/userpackages depending on how it was installed (e.g. ALPS).  A developer would have to check both places.
** One option is to make any import of vistrails.packages use the registry method but hide it from the user.  Not sure how this works for non-modules, however.
** Another option is to have vistrails.packages import all of $DOTVISTRAILS/userpackages which means a developer has to use the "from vistrails.packages import" syntax to access items.
** Other ideas?
** Just limit the way an import occurs (only the "from vistrails.packages import <package_name>" syntax)
* [TE] Should we add BatchQ to VisTrails?
** Users are having problems because only the stable branch of BatchQ works with JobSubmission
** Could we use git submodule?
 
=== March 06, 2013 ===


=== March 2, 2016 ===
'''Updates'''
'''Updates'''
* [RR] DAT can now execute operations on variables
* [TE] Fixed cache bug with port specs
* [ES] Build Machine: Windows 32-bit setup is ready, but the 3D Texture Mapping doesn't work on the VM. I will test if the problem will be present in the binary.
* [TE] VisTrails Next
* [TE] Saving mashups to DB now works
** Most packages now work
** Will add more tests and then merge into master
** No qgis for Python 3
** After that update crowdlabs to use it


'''Items to Discuss'''
'''Items to Discuss'''
* [ES] Build machine:
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]
** Do we have to include the patched version of SUDS in the binaries? (I noticed the example that required it was removed from the repository)
*** It was decided to keep it
** Do we have to support old packages (for example, NumSciPy)?
*** It was decided to use the prebuilt binaries for now
* [TE] Adding locators to unsaved vistrails
** This will fix view switching issues with spreadsheet and vtk
** We should be able to add a MemoryLocator class to unsaved vistrails
** It was decided to store a translation from controller/view to locator
 
=== Feb 27, 2013 ===


=== February 23, 2016 ===
'''Updates'''
'''Updates'''
* [TE] Working on mashup bug fixes and DB support
* [TE] Fixed PythonSource 100% CPU bug
* [TE] VisTrails Next
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples
*** Now uses REST service, tabledata, and Google Maps
*** Web Service replaced with DownloadFile


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Animation support for Matplotlib cells?
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.
* [TE] Will look at adding locator for unsaved vistrails
** New 2.2 release soon?
 
* [RR] Anonymous usage reports
=== Feb 20, 2013 ===
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?
* noWorkflow - provenance for python scripts
** Recent work on tracking dependencies
* Also integrates ReproZip with Jupyter
** [http://mybinder.org mybinder]
*** Use with ReproZip to automatically build Jupyter notebook interfaces
* Daves comments on Jupyter integration
** It is possible to identify cell order on the server side
** Not trivial to track order of cells on client side
* How does Galaxy workflows relate to VisTrails?
** Packaging in Galaxy is hard
** Galaxy for biomedical workflows, difficulty making it general
* Action items
** How difficult to track cell dependencies in Jupyter notebooks?
** Use noWorkflow to track cell dependencies?
** How difficult to modify mybinder to use ReproZip packages?


=== February 17, 2016 ===
'''Updates'''
'''Updates'''
* [TE] Added PBS support to BatchQ master branch
** Troels working on a few performance items
* [TE] Also working on adding mashp
* [BB] Looked into some existing graph libraries for layout (NetworkX, igraph, boost graph tools, ...)
* [DK] Checked in WIP on the new matplotlib package (include README about how things are generated)
** [DK] Need to clean up the specs for this package
** [DK] Figure out what the new student can help with
* [RR] Determining how to type data


'''Items to Discuss'''
'''Items to Discuss'''
* need some type of dataset standard for use with the DAT
* [TE] Fixed Group memory usage
* what about list of (floats, strings, any type of VisTrails object)
** Modules in groups are supposed to be freed when done executing
* require some additional wrapping for the DAT right now
*** Prevented by logger that stored references to the modules (fixed, now using id(module))
* also think about numpy/scipy operations
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)
 
* [TE] VisTrails Next
=== Feb 13, 2013 ===
** Removed deprecated code
*** Core imports that should be gui imports
*** GetInputFromPort etc.
**** PythonSources will be upgraded
*** Fixes for Python < 3
*** fixes for VTK < 6
** TODOs
*** Remove RemoteQ
**** Port Hadoop to tej (Already have pbs)
*** Merge package wrapper
*** Finish SQLAlchemy branch
*** Finish new bundle branch
** [JF] Will contact Jeff and Matthias about moving to Python 3
* [JF] Query by Example is broken
* [JF] VisTrails Data Cleaning Project
** Use VisTrails to process and explore task-specific data while keeping provenance.
** Remi's Idea
*** Use VisTrails as the backend in a Jupyter notebook
*** Overcome problems with executing cells in random order
**** Interesting research topic, but little interest has yet been noted
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)
*** Impose restrictions to notebook cell boundaries
*** It will only work for functional operations
*** Can we fix, or detect, non-functional (mutating) operations?
*** Analyse code and data flow and see if it mutates data in unexpected ways?
*** Mutable operations can be created in many ways in Python
* Next meeting Tuesday 11:30


=== February 10, 2016 ===
'''Updates'''
'''Updates'''
* [TE] Job Monitor Improvements (New DEMO?)
** need some documentation
** Troels is working on some features for multiple jobs (100+) because each job creates a new connection
** also going to change a bunch in the implementation
** some projects need PBS submission capabilities to be using VisTrails
* [DK] matplotlib work improving (a few things still missing but working, mostly automated)


'''Items to Discuss'''
'''Items to Discuss'''
* [DK] vistrails prefix and numpy issue that Remi found
* VisTrails Next
* [DK] online pipeline layout code
** Test suite passes
* [JF] shape for File module
*** Python 3 less deterministic, module upgrade order random.
* [JF] Sunitha to work on examples for matplotlib
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...
 
** All of our packages should run fine :)
=== Feb 06, 2013 ===
** VTK thumbnail comparison works
*** Rendering bug on some systems
**** QVTKCell now inherits QGLWidget
*** Old thumbnails had wrong height due to spreadsheetwindow being too small
**** Fixed by regenerating thumbnails
*** Silenced deprecation messages when generating modules
*** Many new classes in VTK 7
** Remove deprecated code
*** Core imports that should be gui imports
*** GetInputFromPort etc.
**** Will break existing PythonSources
**** Try to upgrade?
*** Old-style packages
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
***** Read info.py without importing package?
*** Fixes for Python < 3
*** Fixes for Qt < 5
*** fixes for VTK < 7
*** Remove self port?


=== February 3, 2016 ===
'''Updates'''
'''Updates'''
* [FC] Working on the package for SciCumulus
** Widget to query run-time provenance from the cloud (almost done)
** Use ModuleSuspended in the execution (to be done)
* [TE] BatchQ-PBS
** Finished a Job Monitor Widget for VisTrails (DEMO?)
** Need to add provenance gathering
* [DK] Fixed minor issue with upgrades (USGS)
* [DK] Working on matplotlib bindings


'''Items to Discuss'''
'''Items to Discuss'''
* DAT project
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]
** Basic functionality is there, some small example packages
** How do we handle multiple projects?
** How do we support multiple 'layered' plots per cell? (UV-CDAT needs this)
** [BB] Configuration widget
*** show ports pane for each module in plot subworkflow
** [CS] Automatic labeling of the pipelines
** [CS] Also try to auto-generate some of the code that is used to specify ports in the DAT
** Talk to Jerry and Tom about what they like about UV-CDAT, what can be improved as feedback for the DAT (talk about refactor)


=== Jan 30, 2013 ===
* VisTrails Next
** All tests but thumbnail comparison passes
** Still issues with gui and more packages needs to be tested
** PythonSource and MplSource upgraded with lib2to3
** Keep contrib packages? Update?
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]
*** Supports Python 3
*** Will hopefully get into conda and others soon
*** Deprecates a bunch of modules


'''Updates'''
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next
* [DK] Fix for older packages to run using new vistrails. prefix
** Runs!
* [DK] Finished dulwich translation of persistence package
** Uses conda
* [BB] working on DAT)
*** Many levels of requirements
* [FC] working on integration between VisTrails and SciCumulous
**** apt packages
* [TE] BatchQ and PBS, module suspended state
**** conda packages
* [TE] way of mointorining jobs
**** python packages in requirements.txt
* [TE] mashups on crowdlabs in database, Emanuele will help on this
**** python packages in setup.py
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!
*** But useful if we want to use conda in our own build process.
** Lots of errors I do not get locally?


'''Items to Discuss'''
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]
* [DK] Need to update documentation for vistrails. prefix
** Have you used any alternatives to vistrails?
** check vtl files here
*** Relevant? More options?
* [DK] [https://vistrails.org/ticket/649 Ticket #649]
** How often do you use any of these particular features in VisTrails?
 
*** Add/Remove alternatives?
=== Jan 23, 2013 ===
** How important would you rank the following potential improvements to VisTrails?
*** Add alternatives?


=== January 27, 2016 ===
'''Updates'''
'''Updates'''
* DataONE Updates
* New interpreter
** obseleted tags
** Working prototype soon
* [TE] First version of BatchQ-PBS ready. Waiting for feedback.
** Looping not completely done (and port combination mode: pairwise/cartesian...)
** forked on github, no changes to the VisTrails package
** Need to integrate with VisTrails
* [DK} Fixed Python dependency determination
*** Compatibility with existing modules
* [DK] Added a branch with UUIDs as identifiers
*** Use log controller
* [DK] Working on using dulwich (git implmentation in python) for peristence package
*** Want to cache at multiple levels (whole stream/single element of stream)
* [FC] Updates in PROV exporter
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)
* [FC] Start integration between [http://www.computer.org/csdl/proceedings/cloud/2010/4130/00/4130a378-abs.html SciCumulus] and VisTrails
* [RR] Multiple projects in DAT?
** Some way to tie cells back to the project is needed
** Global/local mismatch is an issue here (if I drag a variable from one project and combine it with a variable from another project, which project does the new vis/data below to?)


'''Items to Discuss'''
'''Items to Discuss'''
* [JF] Update on package David worked on for USGS (control/suspend execution, support execution on different computational environments) -- msg data 12/18/2012
* User Survey
** methods to fork off processes and to condor
** need to track the settings for condor/asynchronous runs in provenance
** more general modules for this?
** abstraction mechanism for subworkflows to split up workflows (mixed execution models)
** better indication of when a module has been started but is not completed.
** BatchQ shows "suspended" state for this, stops executed this branch.


* [TE] Specifying python dependencies and py_import at the same time is confusing. We could probably call py_import from the package loader if the needed information was specified in the dependencies.
* ZMQ failing on windows (vistrails-users)
** merge these two things for packages (otherwise have issues with specifying new dependencies and trying to import them)
* [TE] Will new import structure and UUID:s go into 2.1?
** Synchronization issue
* [DK] Nix ideas?
** we should at least record some hash of the wrappers/packages in the provenance


=== Jan 16, 2013 ===
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]
** Testsuite almost passes
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.
** Fixed issues
*** Signatures are now bytes (convert to string to show in widget)
*** Cannot compare different types, e.g.,  (None < 1)
**** PortSpec.depth and min/maxConn could be None. (Backport!)
**** cell.row could be None
*** Lots of unicode/bytes fixes to serializations
**** We have encryption implementations in the code! (keychain)
*** Lots of small fixes to package method differences
** Current issues
*** logging widget empty?
*** version tree signals messed up
*** provenance view legend icons black
*** Text do not fit in some widgets
*** VTK thumbnail comparison fails
**** VTK Cells have some faces in wrong direction?
****  Because running in VM?
** Installing on Ubuntu LTS/Latest Mint
*** Install latest matplotlib from pip
*** Install VTK 7.0.0rc2 from source
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)


=== January 20, 2016 ===
'''Updates'''
'''Updates'''
* [TE] WebGL port issue: Wendel added fallback mode (use pre-rendered data), is now helping me getting it to work using port 80 through apache.
* [DK] DataONE package
* Rémi's progress on the DAT
** working on VisTrails code to modify the spreadsheet
** GUI is usable except that the interface doesn't execute VisTrails pipelines
** Has a document available in Google Docs documenting progress
* [BB] All critical open bugs for UVCDAT are complete, one minor bug and some enhancements left


'''Items to Discuss'''
'''Items to Discuss'''
* WebGL on crowdLabs
* [TE] Backporting features from Python 3 + PyQt5 + Qt5
** try to send cached images through port 80 if necessary
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])
** need to inform users why an issue comes up (e.g. port is closed, using cached image)
*** One unicode test failing
* Remove the sci mailing lists finally?
*** Problem saving to xml
* Building VisTrails on a NASA supercomputer?
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])
* Web interface aka ClimatePipes
*** Updates print to functions 'from __future__ import print_function'
 
*** Updates exception calls to use 'as'
=== Jan 9, 2013 ===
*** removes most usages of 'long' (mostly used unnecessarily)
**** Only needed for 32-bit support
*** Fixes relative imports (from __future__ import absolute_import)
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])
*** Updated all signals to new-style
* If we go 2 to 3, this could break packages
** means we should schedule any package format changes for the same time
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)
** putting package identifier as xml?
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)
** version the APIs?
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?
* file formats on 2 to 3:
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2


=== January 13, 2016 ===
'''Updates'''
'''Updates'''
* Emanuele resumed setting up the build machine for Windows.
* [TE] crowdlabs now uses rendered image in vistrail tab if available
* [TE] Discussing WebGL port issue with Wendel
** One possibility is to use port 80 on another server and use a proxy
* [TE] Working with Troels on adding PBS support to BatchQ
* [BB] 5 Bugs remaining with UVCDAT
* [DK] DataONE work


'''Items to Discuss'''
'''Items to Discuss'''
* Uploading Mashups to crowdLabs from VisTrails
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]
** For this to work need to complete relational support for Mashups (see ticket [https://www.vistrails.org/ticket/611 #611]). The code is in mashups branch and needs to be updated with the current master (it was branched from old schema-v1_0_3 branch and not updated since then).
** Basic GUI and execution works
** Need someone to work on this for Matthias
** Minimal work to support 2to3
** Not just relational support here, want to improve the schema
** Hard to support both PyQt4 and PyQt5
** Could just dump xml as a blob into the database, but were trying to make things work better
* Cleaning up older files (see last week)
* UV-CDAT Bug #69:
** The variables are global; cdat has a global dictionary
** thus closing a project has no effect on the dictionary
** if you have two projects that reference the same variable name, you run into issues
** discuss this with Charles
* UV-CDAT undo button:
** should be ok, just remember to update cell version
 
=== Jan 2, 2013 ===


=== January 6, 2016 ===
'''Updates'''
'''Updates'''
'''Items to Discuss'''
'''Items to Discuss'''
* Rémi went through the code to find modules that are no longer used
* Updating VisTrails dependencies
** core.modules.module_configure and related have been moved to gui, can we remove the core redirects?  Problem is that this could break older packages
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]
*** need to update documentation and add to FAQ
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.
** should we continue to maintain third-party packages (those in /packages not /vistrails/packages) or should these be moved?
**** Next version of VTK 6 is rumored to support Python 3
*** do we distribute the /packages directory with the source tarball?
**** There is a development branch of VTK 6 supporting Python 3
*** we don't want to be responsible for keeping these packages up-to-date, but at the same time, we shouldn't distribute packages that do not work with our releases...
*** Can travis build this?
 
**** We can skip tests for WebEngine and VTK if needed
* crowdLabs issues?
** Python 3 [https://python3wos.appspot.com/ showstoppers]
** Trying to make ALPS server work with VisTrails
*** No MySQL-python (sqlalchemy supports mysql through other drivers)
** Thumbnails: make these higher resolution
*** No suds (there are active forks but we could also drop support?)
** Issue with ports and WebGL?
* Options for python2/3 support:
*** need an error message if this is a port issue
** Interpreter could be python2/python3 based on packages
*** should be solutions/workarounds to use "standard" ports with the WebGL data
** six layer works but not the easiest to maintain (e.g. in reprozip)
* Drop support for win-64
* Update copyright headers to 2016


== Older meetings ==
== Older meetings ==
* [[Development/2015]]
* [[Development/2014]]
* [[Development/2013]]
* [[Development/2012]]
* [[Development/2012]]
* [[Development/2011]]
* [[Development/2011]]
* [[Development/2010]]
* [[Development/2010]]

Latest revision as of 15:34, 30 October 2017

See also the Github wiki

2017

October 30, 2017

Updates

  • [DK] PyQt5
  • [RR] Running Python3 code

Items to Discuss

  • Installation:
    • Use conda? (macOS hack to have .app bundle load conda-installed vistrails)
    • Update conda recipe (pyqt4 restriction for now)
  • Interpreter
  • Release
  • Integration
    • hard to build same type of GUI in other apps that integrate core VisTrails (tend to use just the API)
    • independent pieces of the GUI could be reused if they were more independent.

September 6, 2017

Updates

  • [RR] Fixes for scikit-learn, tables, usagestats

Items to Discuss

  • Stripped-down version of VisTrails to showcase new features?
    • include meta-vistrail, bundles
    • hide mashups, extra widgets, latex mode, etc.
  • Interpreter

August 16, 2017

Updates

  • [DK] meta-vistrail
    • Working actions: normal action in vistrail, pruning version tree, tagging operations, and change
    • Interface

Items to Discuss

  • Notebook
  • Interpreter
  • Web visualization packages (e.g. bokeh)
  • Pipeline templates

August 2, 2017

Updates

  • [DK] meta-vistrail work
    • Extend Vistrail to MetaVistrail to support meta-actions
    • Vistrail becomes less tied to workflows so it can also supporting versioning vistrails

Items to Discuss

  • Interpreter
    • Using dask?
    • Streaming an issue
  • Notebooks
    • What is an output of a module? (need to expose a specific output port or the entire module)
    • Unifies some of the work from the script-workflow code

June 28, 2017

Updates

  • [DK] Testing UUID/SQLAlchemy branch
  • [DK] Working on meta-versions
    • would be nice to have pipeline and vistrail separate
    • [RR] Then do we only have meta-actions, any change to the pipeline is a meta-action that adds a version that does the change?
    • [DK] Yes?
  • [RR] Notebook interface (Vizier) making good progress, figuring out cell outputs

Items to Discuss

June 7, 2017

Updates

  • [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs

Items to Discuss

  • [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)
    • Can designate in module description an output port
    • Value has to be serializable
    • Should use the OutputModes work to handle multiple modes, configuration, selection, ...
  • [RR] Is VisTrails with SQL usable concurrently?
    • [DK] Probably not, demo from [TE] exists

May 10, 2017

Items to Discuss

  • Identifiers
  • Bundles & DB Code
  • Interpreter
  • Merge / Versioning Version Trees
  • Python3
  • Workings of List
  • Packages: Wrappers + new Versions (VTK7)
  • Script Conversion
  • Intermediate Outputs: reuse output module work?
    • would be useful for both workflows (run to a particular checkpoint)
    • and useful for notebook interfaces
  • SQL backend: does it allow for concurrent access?
    • Yes, real-time collaboration work for concurrent single vistrail access
    • Should be able to access different vistrails at once without issues

April 19, 2017

Updates

Items to Discuss

April 5, 2017

Updates

Items to Discuss

March 8, 2017

Updates

  • [RR] Added a looping indicator on modules #1217
    • This makes it easier to understand what's looping, no need to follow edges and count
  • [RR] Refactoring CLTools for integration in Docker package
  • [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)

Items to Discuss

  • [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers #1216
    • Can we move this up to the renderer modules?
    • Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)

February 24, 2017

Updates

  • [RR] Rich-text module documentation (#1210)
    • Went over packages and fixed format; maybe we should have a test to validate RST?
  • [RR] Single-instance shutdown with --remote-shutdown (#1204)
  • [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment

Items to Discuss

  • [RR] There are issues with list-depth
    • Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)
    • Issue with looping groups

January 25, 2017

Updates


Items to Discuss

  • Scaling VisTrails
  • sql-alchemy, uuid, bundle branches/merges

January 11, 2017

Updates

  • [RR] New VisTrails version in Debian [1]

Items to Discuss

  • [RR] Collaboration with school of medicine to port HiC-bench to VisTrails
    • Have homemade workflow system based on R scripts and symlinks
      • Each steps has multiple sets of parameters
      • Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step
      • Gets all results in directory structure step3-parameters/step2-parameters/step1-parameters/files
    • VisTrails improves provenance capture and makes editing pipelines easier
      • Need better support for aggregation (groupby operation might be good enough; pandas?)

2016

December 14, 2016

Updates

  • [RR] Single-instance code: over-engineered, but will work
    • (Have to avoid race conditions on startup, and OSes offer very different locking primitives)

Items to Discuss

  • Email from Matt Dirks: combobox should work, ask for more code?
  • Email from marin.nl:
    • looks like they would benefit from new interpreter
    • how to store their consolidated provenance when they exchange .vt files around?

November 30, 2016

Updates

Items to Discuss

  • [RR] Working on single instance code for MARIN (#1205)
    • Remote shutdown ready (#1204)
    • Still improving it
    • Want to return something more structured to the client (success/fail code, stdout, stderr)
    • Should be usable without Qt
  • [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason
  • [DK] Working on batch execution using console_mode
  • [DK] Settings aliased parameters when using core.api?

November 16, 2016

Updates

  • [RR] Issue #1200 from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)

Items to Discuss

  • Email from Pieter: executing pipelines with parameters from command-line using single-instance
    • Seems to work, main instance gets full command-line and alters configuration temporarily
    • Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful
    • Actions:
      • Can add switch to kill main instance
      • Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails
      • Add client/server code to core that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like gui.application.VistrailsApplicationSingleton#noninteractiveMode())?

November 2, 2016

Updates

  • New version 2.2-2 of Debian package, still wrong copyright, and broken bundle installation
    • Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch
  • Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)

Items to Discuss

October 5, 2016

Items to Discuss

  • working toward a 3.0 release; project board
    • unique ids
    • bundles
    • GUI fixes
    • wrapping updates
    • python3
    • interpreter?
  • command-line input parameters to provenance
    • [DK] we don't create a persisted workflow from those input params, i think
    • [DK] these are already defined as aliases so we treat them differently form a normal input param
    • [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases
    • Do the vistrails variables values get stored in provenance? (what if values change?)

September 14, 2016

Updates

Items to Discuss

August 31, 2016

Updates

  • [RR] Went over vgc.poly.edu/vistrails.org config:
    • Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
    • Fixed usersguide scripts (won't disappear again)
    • Moved some websites over to HTTPS
    • Websites
  • [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs (currently there) and users' guide under the same RTD "project"
  • [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that

Items to Discuss

  • [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
    • Get rid of more services? (svn, trac, ...)
  • [RR] Notebook execution works. How do we support more languages (than Python)?
    • Need: detection, metadata reader, code to input values
  • [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?

August 17, 2016

Updates

  • [TE] Fixed parameter order upgrade bug that breaks VTK examples.
  • [TE] Matplotlib and sklearn are exportable using module-to-script api
  • [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.

Items to Discuss

  • Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
  • Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
  • [TE] Wrap-up notes

August 10, 2016

Updates

Items to Discuss

  • [TE] Python Wrapper and scripting
    • All changes merged into python-wrapper branch
      • Tests pass
      • Still need to update sub-branches
    • Minor fixes
      • vtkPlot3DReader upgrade bug
      • Added tuple exporting
    • Most of vtk examples can be exported example
      • Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
        • Could be a bug in the old vistrail, but not sure
  • [DK] Subworkflows and uuids

August 3, 2016

Updates


Items to Discuss

  • [TE] Wrapper patches now stored as python code
    • Connection variables now lowercase
    • Can we use code as output variables?
      • E.g. "vtk_dataset_mapper.GetOutput(0)"
      • Could mutate output when output method is only supposed to be called once
      • Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)

July 27, 2016

Updates

  • [DK] DB bundle serialization implemented

Items to Discuss 

   def vtkInstance():
       self.vtkInstance = self

July 20, 2016

Updates

Items to Discuss

  • [TE] New patch method for python-wrapper example
    • A patch is a string template with self/input/output variables for patching a class method
    • Patches are stored in xml specification
    • Patches are assigned to methods in ClassSpec's
    • Used both for execution in vistrails and when exporting as script
      • Executed with "exec" when run in vistrails
      • Turns patches into natural calls in script
    • Wrapping VTK library no longer needed
      • Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)
    • Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)
    • Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)
    • TODO's
      • Add to export_as_script
      • Patch Modules that represent Python functions as well
  • [vistrails-users] Read data from SQL example?

July 13, 2016

Updates

  • [DK] bundles on sql-alchemy branch
    • Bundle, BundleMapping, and BundleSerializer
    • New BaseSerializers to deal with versions of directory, zip, db serializers
    • Can register the BundleSerializers with the base serializers
    • Translation between bundle versions
    • Bundle.add_object "just works" if an appropriate mapping was registered

Items to Discuss

  • [TE] Simplified running module using API for scripting example
    • API Module can be executed using compute method
      • Takes list of outputs to compute + inputs as keyword arguments
  • Status of general cell synchronization (vistrails-users)

July 6, 2016

Updates

  • [DK] Working on bundles

Items to Discuss

  • [TE] Using vistrails API in scripting example 1 (API with VTK) example 2 (DownloadFile only)
    • Call module in script using vistrails API
    • Can be used when no export method is defined.
  • [TE] Server news bug Critical?
    • [RR] No, unreleased
  • [DK] New bundle organization
    • BundleMapping contains individual mappings from raw objects to BundleObj
    • Must be separate in order to have in-memory, separate from serializer settings

June 29, 2016

Updates

Items to Discuss

  • [TE] Export VTK as python script
    • Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) example
    • Preludes (imports) still need improvements
    • Works with vtkInteractionHandler (clipping plane)
    • Resulting clipping plane is incorrect? (supposed to hide one side of the plane)
    • vtkRendererOutput Uses VTKRenderWindow as output
      • Should support different output modes?
      • Does not work with multiple outputs (Next cell is displayed when current is closed)
      • Uses wrapped vtkInstance (as vvtk)
        • Create new patch system with code that can be used both in vistrails and in a python script?

June 22, 2016

Updates

Items to Discuss

  • [TE] Fix for upgraded analogies (PR #1177)
    • Uses module location to find module remap (The best we can do right now?)
    • Also fixes remapped port names for connections to existing modules
    • May still fail on complex upgrades like adding a function on an upgraded module that has a different port name
  • [TE] Working on exporting wrapped libraries (VTK)

June 15, 2016

Updates

  • [TE] Added depth logic to python script exporter (full commit)
    • Fully implements connection merging and list looping

Items to Discuss

  • [TE] Making analogy work on upgraded workflows (#1175)
    • Analogy only works We need to use the non-upgraded action chain
    • But pipelines need to be valid (matcher uses module portspecs)
    • Tried to get upgrade remap from upgrade action, but this seems impossible
    • Use module location instead?
      • Location should be identical for one-to-one module upgrades

June 1, 2016

Updates

  • VisTrails 2.2.4 announced (finally)
    • 5 survey replies
    • 47 downloads last week

Items to Discuss

  • [TE] Export VTK as script (python-wrapper)
    • to_python_script
      • needs the pipeline (or at least the connections)
        • Should not generate code for any possible VTK method (exporter detects which ports is used in the source)
        • VTK needs to know connection order
        • Adding functions directly to the script results in nicer code
    • Add support for multiple input connections
    • Add support for list looping

May 25, 2016

Updates

  • Mailing-list fixed
  • Getting stats from the "server news" mechanism (via parser)

Items to Discuss

  • [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)
  • [TE] Union ports
    • Show union ports in pipeline (DEMO)

May 18, 2016

Items to Discuss

  • [TE] mailing list still down - Julio working on it
  • [TE] Improved union port
    • New schema 1.0.5 adds portitem.union string attribute
    • Works with pipeline view, parameter exploration view, and mashup view
    • Should work well for matplotlib
    • Use with VTK ports?
      • It uses [item_1,item_2] style ports usually with different tuple length.
  • [TE] Remove emacs dependency from db generate #1173

May 11, 2016

Items to Discuss

  • [TE] Union type #1172
    • GUI-based solution Branch union-port Demo?
      • Shows single port for all ports with same sort_key
      • Select specific type when adding/showing
      • Only one type should be visible on the module at any time
      • Making tuple parameters union types is probably too complicated
      • Should work with alternate specs like matplotlib (if sort_key is defined correctly)
  • [TE] Feature suggestion: Let connections connect to functions/parameters
    • Show functions and parameters as ports and toggle port visibility directly for functions and parameters
    • Use separate views for ports and functions (like before)
      • Allow rearranging of functions
    • Pros:
      • Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
        • We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.
        • Makes List module simpler
      • Connect to parameter in tuple
      • Functions that are visible on the module can connect to an actual function, and not just to the first one it finds.
      • Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.
    • Cons:
      • Interface more complicated and crowded
      • More difficult for users to grasp?
      • No immediate use case
      • Makes no union type for tuple parameters a bigger problem.
  • [TE] Publish user survey?
    • No response from vistrails release yet
    • Not announced on vistrails-users yet.

May 4, 2016

Updates

  • VisTrails 2.2.4 released

Items to Discuss

April 27, 2016

Updates

Items to Discuss

  • [TE] Wrapping Bokeh
    • New method type "operation"
      • Transforms Figure->Glyph->Output into Glyph->Figure->Output
        • Glyph sets function name and arguments as output
          • Not serializable, but should always be computed together with the subject of the operation.
        • Figure applies Glyph operation on itself
      • Useful for matplotlib?
        • matplotlib currently find actors by searching upstream?
        • But this is more difficult since matplotlib is not function-based.
    • New "union" port type
      • Turns "x(float), x(list)" into "x (float, list)" and uses first set value
        • Uses only first value in tuple
        • Bokeh have many properties that support (list, string, Float)
      • Does not work with depth=1 types (e.g. Float of depth 1)
        • Turn Depth 1 types with known type into lists?
  • [TE] New bugfix release
    • Finalize survey

April 20, 2016

Updates

  • Merged upgrade fixes
    • Now has infinite loop test
    • loop test now correctly loops 5 times
      • remap_module used wrong (current) package version when creating ModuleDescriptor
  • Fixed multi-step group upgrades (PR: #1168)

Items to Discuss

  • New bugfix release
    • Merge usagestats
    • Need to finalize survey

April 13, 2016

Updates

Items to Discuss

  • [TE] Chaining upgrades done (#1164)
    • Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?
  • [TE] Iterating handle_invalid_pipeline (#1165)
    • Not based on chain-upgrades-loop
    • Cherry-pick test to preserve source?
    • Need maxPipelineFixAttempts option?
      • The code is now smarter about recursion, so max attempts will "probably" never be reached.
    • Need debug messages?
      • Would need to be moved into handle_invalid_pipeline

April 6, 2016

Updates

  • [TE] Query View improvements
    • Done except for displaying "Version Info" and #1164
  • [RR] Working on finishing new interpreter (finally!)

Items to Discuss

  • [TE] Chaining upgrades (#1164)
    • Try latest upgrade first, then previous.
    • If all fail, select latest upgrade?

March 30, 2016

Updates

Items to Discuss

  • [TE] Query View improvements
    • Unified do_version_switch and get_upgrades as validate_version()
    • Problems with query view using its own controller
      • Fixed module info not being displayed for query results
      • Still no version info for query results
      • Still an issue with version query view not being updated

March 23, 2016

Updates

Items to Discuss

  • [RR] Usage reporting almost ready to go
    • Need to record whether specific features are used: paramexpore, mashup, query
      • [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation
    • How to record size of vistrail?
      • [DK] IdScope already walks over vistrail, use that
      • but total number of modules and versions doesn't give an idea of average size of individual pipelines

March 16, 2016

Updates

Items to Discuss

  • [TE] Upgrades in Query Mode
    • Now using code similar to do_version_switch
      • Uses handle_invalid_pipeline
      • Differences: Not using current_pipeline, flushes changes
      • New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)
      • Integrate to avoid repeated code?
      • Use this for query mode?
  • [TE] VisTrails 2.2.4 ready for release?
    • tej package updated. Others?
  • [JF] upgrades and the version tree
  • [JF] provenance in Jupyter
  • [DK] bundles: #1092

March 9, 2016

Updates

Items to Discuss

  • [TE] Fixing search mode
    • String query now matches all by default
      • This caused pipeline queries to never match
    • String query can search: User/Notes/Tagname/Time/Module name
      • Should we search module names by default? (Disabled by default because it is expensive)
      • Fixed notes search
    • '>' parameter queries in documentation but has been removed
    • Fixed module matches for upgraded pipelines (By not upgrading pipeline)
      • Upgrades still a problem
      • Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)
      • [JF] Allow search on past tags
  • [TE] Create new bugfix release
    • Finish querying
    • Path issue on windows
  • [DK] Look into sqlalchemy and new bundle branches

March 2, 2016

Updates

  • [TE] Fixed cache bug with port specs
  • [TE] VisTrails Next
    • Most packages now work
    • No qgis for Python 3

Items to Discuss

    • Always dereference symlinks? #1146

February 23, 2016

Updates

  • [TE] Fixed PythonSource 100% CPU bug
  • [TE] VisTrails Next
    • Updated NOAA Web Service examples
      • Now uses REST service, tabledata, and Google Maps
      • Web Service replaced with DownloadFile

Items to Discuss

  • [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.
    • New 2.2 release soon?
  • [RR] Anonymous usage reports
    • Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features #1154
  • [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?
  • noWorkflow - provenance for python scripts
    • Recent work on tracking dependencies
  • Also integrates ReproZip with Jupyter
    • mybinder
      • Use with ReproZip to automatically build Jupyter notebook interfaces
  • Daves comments on Jupyter integration
    • It is possible to identify cell order on the server side
    • Not trivial to track order of cells on client side
  • How does Galaxy workflows relate to VisTrails?
    • Packaging in Galaxy is hard
    • Galaxy for biomedical workflows, difficulty making it general
  • Action items
    • How difficult to track cell dependencies in Jupyter notebooks?
    • Use noWorkflow to track cell dependencies?
    • How difficult to modify mybinder to use ReproZip packages?

February 17, 2016

Updates

Items to Discuss

  • [TE] Fixed Group memory usage
    • Modules in groups are supposed to be freed when done executing
      • Prevented by logger that stored references to the modules (fixed, now using id(module))
      • Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)
  • [TE] VisTrails Next
    • Removed deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • PythonSources will be upgraded
      • Fixes for Python < 3
      • fixes for VTK < 6
    • TODOs
      • Remove RemoteQ
        • Port Hadoop to tej (Already have pbs)
      • Merge package wrapper
      • Finish SQLAlchemy branch
      • Finish new bundle branch
    • [JF] Will contact Jeff and Matthias about moving to Python 3
  • [JF] Query by Example is broken
  • [JF] VisTrails Data Cleaning Project
    • Use VisTrails to process and explore task-specific data while keeping provenance.
    • Remi's Idea
      • Use VisTrails as the backend in a Jupyter notebook
      • Overcome problems with executing cells in random order
        • Interesting research topic, but little interest has yet been noted
      • Interesting to study how such a notebook would be used (Is there notebook usage data available?)
      • Impose restrictions to notebook cell boundaries
      • It will only work for functional operations
      • Can we fix, or detect, non-functional (mutating) operations?
      • Analyse code and data flow and see if it mutates data in unexpected ways?
      • Mutable operations can be created in many ways in Python
  • Next meeting Tuesday 11:30

February 10, 2016

Updates

Items to Discuss

  • VisTrails Next
    • Test suite passes
      • Python 3 less deterministic, module upgrade order random.
    • [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...
    • All of our packages should run fine :)
    • VTK thumbnail comparison works
      • Rendering bug on some systems
        • QVTKCell now inherits QGLWidget
      • Old thumbnails had wrong height due to spreadsheetwindow being too small
        • Fixed by regenerating thumbnails
      • Silenced deprecation messages when generating modules
      • Many new classes in VTK 7
    • Remove deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • Will break existing PythonSources
        • Try to upgrade?
      • Old-style packages
        • Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
          • Read info.py without importing package?
      • Fixes for Python < 3
      • Fixes for Qt < 5
      • fixes for VTK < 7
      • Remove self port?

February 3, 2016

Updates

Items to Discuss

  • VisTrails Next
    • All tests but thumbnail comparison passes
    • Still issues with gui and more packages needs to be tested
    • PythonSource and MplSource upgraded with lib2to3
    • Keep contrib packages? Update?
    • VTK 7 is released
      • Supports Python 3
      • Will hopefully get into conda and others soon
      • Deprecates a bunch of modules
  • Travis builder for VisTrails Next
    • Runs!
    • Uses conda
      • Many levels of requirements
        • apt packages
        • conda packages
        • python packages in requirements.txt
        • python packages in setup.py
      • Can probably drop conda. Default builder does not have Qt5, but beta builder do!
      • But useful if we want to use conda in our own build process.
    • Lots of errors I do not get locally?
  • User Survey
    • Have you used any alternatives to vistrails?
      • Relevant? More options?
    • How often do you use any of these particular features in VisTrails?
      • Add/Remove alternatives?
    • How important would you rank the following potential improvements to VisTrails?
      • Add alternatives?

January 27, 2016

Updates

  • New interpreter
    • Working prototype soon
    • Looping not completely done (and port combination mode: pairwise/cartesian...)
    • Need to integrate with VisTrails
      • Compatibility with existing modules
      • Use log controller
      • Want to cache at multiple levels (whole stream/single element of stream)
      • Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)

Items to Discuss

  • User Survey
  • ZMQ failing on windows (vistrails-users)
  • [TE] Python 3 branch
    • Testsuite almost passes
      • Failing are unicode filenames, some users guide examples, and thumbnail comparisons.
    • Fixed issues
      • Signatures are now bytes (convert to string to show in widget)
      • Cannot compare different types, e.g., (None < 1)
        • PortSpec.depth and min/maxConn could be None. (Backport!)
        • cell.row could be None
      • Lots of unicode/bytes fixes to serializations
        • We have encryption implementations in the code! (keychain)
      • Lots of small fixes to package method differences
    • Current issues
      • logging widget empty?
      • version tree signals messed up
      • provenance view legend icons black
      • Text do not fit in some widgets
      • VTK thumbnail comparison fails
        • VTK Cells have some faces in wrong direction?
        • Because running in VM?
    • Installing on Ubuntu LTS/Latest Mint
      • Install latest matplotlib from pip
      • Install VTK 7.0.0rc2 from source
      • No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)

January 20, 2016

Updates

Items to Discuss

  • [TE] Backporting features from Python 3 + PyQt5 + Qt5
    • Merged unicode with master (unicode)
      • One unicode test failing
      • Problem saving to xml
    • Used python-future to run futurize stage 1 to update legacy (pre 2.6) code (unicode-futurized)
      • Updates print to functions 'from __future__ import print_function'
      • Updates exception calls to use 'as'
      • removes most usages of 'long' (mostly used unnecessarily)
        • Only needed for 32-bit support
      • Fixes relative imports (from __future__ import absolute_import)
    • Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 (unicode-futurized-qt5)
      • Updated all signals to new-style
  • If we go 2 to 3, this could break packages
    • means we should schedule any package format changes for the same time
    • e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* #1002)
    • any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)
    • putting package identifier as xml?
      • Python not so bad, plus some stuff in there is Python code (checking dependencies)
    • version the APIs?
      • RR planning on versioning vistrails_module.Module, no point versioning the whole package?
  • file formats on 2 to 3:
    • no real translation/upgrade, but non-ascii values that work on 3 won't load in 2

January 13, 2016

Updates

Items to Discuss

January 6, 2016

Updates Items to Discuss

  • Updating VisTrails dependencies
    • Successfully built Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7
      • VTK 7 is the first supporting Python 3 but is only in release candidate phase.
        • Next version of VTK 6 is rumored to support Python 3
        • There is a development branch of VTK 6 supporting Python 3
      • Can travis build this?
        • We can skip tests for WebEngine and VTK if needed
    • Python 3 showstoppers
      • No MySQL-python (sqlalchemy supports mysql through other drivers)
      • No suds (there are active forks but we could also drop support?)
  • Options for python2/3 support:
    • Interpreter could be python2/python3 based on packages
    • six layer works but not the easiest to maintain (e.g. in reprozip)
  • Drop support for win-64
  • Update copyright headers to 2016

Older meetings

Retrieved from "https://www.vistrails.org//index.php?title=Development&oldid=14736"