Difference between revisions of "Development"

From VistrailsWiki
Jump to navigation Jump to search
 
(543 intermediate revisions by 4 users not shown)
Line 3: Line 3:
</div>
</div>


== 2014 ==
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]
 
== 2017 ==
 
=== October 30, 2017 ===


=== September 17, 2014 ===
'''Updates'''
'''Updates'''
* [TE] Fixed database from command-line, server mode, command line flags
 
* [TE] Fixed slow view switching
* [DK] PyQt5
* [RR] Running Python3 code


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] unicode and python3 compatibility
* [TE] Parameter Widgets (function-tooltip) ready to merge


=== September 10, 2014 ===
* Installation:
** Use conda? (macOS hack to have .app bundle load conda-installed vistrails)
** Update conda recipe (pyqt4 restriction for now)
* Interpreter
* Release
* Integration
** hard to build same type of GUI in other apps that integrate core VisTrails (tend to use just the API)
** independent pieces of the GUI could be reused if they were more independent.
 
=== September 6, 2017 ===
 
'''Updates'''
'''Updates'''
* [RR] Backported tabledata package: https://github.com/remram44/tabledata-backport
 
* [TE] GUI Improvements
* [RR] Fixes for scikit-learn, tables, usagestats
** Added type checking in parameter view
** Switch to workflow info when no module selected
** Improved fitToView (used a square setting?)
* [TE] Edit widgets
** Hide when zooming out
** Improved proxied widgets
** Improved selection border
** Profiling
*** Nothing major for edit widgets
*** Fixed mark_list_depth calls when switching versions


'''Items to Discuss'''
'''Items to Discuss'''
* Workflow longevity: mta example no longer works
** Fernando suggested offering an option that saves the package (or a description of the package) together with the workflow
** [DK] I agree, this had been talked about with the generic bundle support
** [TE] Everything stored in a zipped vt file, could also store to a directory
*** [DK] some support for this
** [DK] to look over old code and point out what needs to be done for bundle support
* [RR] maybe work to distribute packages independent of VisTrails (or at least make moving between versions of packages easier)
* [RR] VisTrails integration with ReproUnzip: https://github.com/ViDA-NYU/reprozip/issues/69
** [DK] possible to do python install using smarter package installers?


=== September 3, 2014 ===
* Stripped-down version of VisTrails to showcase new features?
** include meta-vistrail, bundles
** hide mashups, extra widgets, latex mode, etc.
* Interpreter
 
=== August 16, 2017 ===
 
'''Updates'''
'''Updates'''
* [TE] Fixed 1.0.3 schema and log upgrades
* [DK] meta-vistrail
** [TE] Make sure that 1.0.3 has the right items across versions
** Working actions: normal action in vistrail, pruning version tree, tagging operations, and change
** Keep startup only in master since it is only needed for upgrades
** Interface


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Edit widgets demo
* Notebook
** don't show name of port for constants
* Interpreter
** boolean widget doesn't look great (still native widget)
* Web visualization packages (e.g. bokeh)
** different layout on Linux than Mac/Windows due to font?
** [https://github.com/VisTrails/VisTrails/pull/969 PR 969] has more details
* [DK] UV-CDAT build:
* Pipeline templates
** Linux build and check for scipy issue (#590)
 
* python3?
=== August 2, 2017 ===
* [TC] Good way to wrap up userpackages?
** using conda package manager (environment has directory and hard-linked into, need to be able to hard link into environment)
** [RR] is working to do a standard python package to recognize these automatically
** For userpackages, can configure userPackageDirectory to point to a hard-linked environment
** Preferences -> Paths & URLs -> Local Packages Directory
** everything to be in a "bubble" at BNL
* [ED] Parameter exploration:
** one variable that I care about but plot as 5x3 not 15x1 or 1x15, enhancement request
* [ED] Series of Qt widgets wrapped into VisTrails modules
** subclass SpreadsheetCell, Qt widgets use matplotlib
** [ED] to provide a demo on what is going on trac


=== August 27, 2014 ===
'''Updates'''
'''Updates'''
* [TE] VisTrails 2.1.4 released
* [DK] meta-vistrail work
** [RR] Will update [https://pypi.python.org/pypi/vistrails PyPI] and [https://binstar.org/remram/vistrails Conda]. Note that this currently uses its own branch for 2.1.x (<tt>deployment/v2.1</tt>), will be easier in 2.2
** Extend Vistrail to MetaVistrail to support meta-actions
** Vistrail becomes less tied to workflows so it can also supporting versioning vistrails


'''Items to Discuss'''
'''Items to Discuss'''
* [https://github.com/UV-CDAT/uvcdat/issues/520 UV-CDAT Bug #520] and [https://github.com/UV-CDAT/uvcdat/issues/557 #557]: make sure we don't show errors for packages that don't load (put these at info level)
** [RR] already created a branch with this name
* matplotlib and UV-CDAT: should be able to include freetype in binaries
* [https://www.vistrails.org/ticket/886 Bug 886]:
** merge in this branch, perhaps not 100% ideal but still better than what we had
** problem is that you sort of want to be able to do a bunch of actions and then raise InvalidPipeline so you don't have to do multiple try-catch
* [https://www.vistrails.org/ticket/892#comment:2 Bug 892]:
** not pythonic to forceably squelch propagation
** should be a configuration option to do this
** add a filter to the root logger so that VisTrails messages can be controlled
** will think about this more
* [TC] blocker on python3 compatibility?
** [RR] unicode support should probably be done first: unicode from all the files, weird casts everywhere in the code because we used Qt before api v2
** from future unicode literals and str type makes the right thing happen on 2 and 3?
** BNL really wants 3 support, willing to work on
** Issues are [https://www.vistrails.org/ticket/673 #673 (unicode)] and [https://www.vistrails.org/ticket/674 #674 (python3)]
* Relayout programmatically?
** [http://git.io/MR0hZw layout_modules()] in controller


=== August 20, 2014 ===
* Interpreter
** Using dask?
** Streaming an issue
* Notebooks
** What is an output of a module? (need to expose a specific output port or the entire module)
** Unifies some of the work from the script-workflow code
 
=== June 28, 2017 ===
 
'''Updates'''
'''Updates'''
* [TE] Parameter Exploration bugfixes
* [DK] Testing UUID/SQLAlchemy branch
** "Create Version" spreadsheet button fixed
* [DK] Working on meta-versions
** Missing date attribute in xml caused errors
** would be nice to have pipeline and vistrail separate
** Issue opening vtl links
** [RR] Then do we only have meta-actions, any change to the pipeline is a meta-action that adds a version that does the change?
** [DK] Yes?
* [RR] Notebook interface (Vizier) making good progress, figuring out cell outputs


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Pending branches:
** pipeline-validation ([https://www.vistrails.org/ticket/886 #886])
*** (dep for fix-open-workflow (loading a workflow keeps the locator, so saving will overwrite))
*** Ok, will merge
** logger-propagation ([https://www.vistrails.org/ticket/892 #892])
*** Might be a better way? Discuss this more?
** hide-missing-dependencies ([https://www.vistrails.org/ticket/895 #895])
*** Ok, will merge
* [TE] New bugfix release? - Parameter Explorations and wrong usersguide in Mac binary
* [TE] Module edit widgets demo
** start with tooltips for type names rather than showing them right now
** give warnings on input validation -- add to trac ([https://www.vistrails.org/ticket/898 #898])
* [Thomas] matplotlib is meant to be used the other way
** [RR] problem: this is not the way our other plotting packages work (including VTK: data -> data -> renderer -> cell)
** [RR] we've already released a matplotlib package (data -> plots -> figure -> cell) and can't really turn it around
** [RR] we all agree that current code is hacky (but works), see [https://www.vistrails.org/ticket/685 #685] then [https://www.vistrails.org/ticket/396 #396]


=== August 13, 2014 ===
=== June 7, 2017 ===
 
'''Updates'''
* [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs
 
'''Items to Discuss'''
* [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)
** Can designate in module description an output port
** Value has to be serializable
** Should use the OutputModes work to handle multiple modes, configuration, selection, ...
* [RR] Is VisTrails with SQL usable concurrently?
** [DK] Probably not, demo from [TE] exists
 
=== May 10, 2017 ===
 
'''Items to Discuss'''
'''Items to Discuss'''
* [TE] FileOpen event on Mac ([https://www.vistrails.org/ticket/889 #889])
** We can pass argv to QApplication
* [TE] 1.0.3 schema differences ([https://www.vistrails.org/ticket/894 #894])
** We can restore schema 1.0.3 on master from branch v1.2


=== August 6, 2014 ===
* Identifiers
* Bundles & DB Code
* Interpreter
* Merge / Versioning Version Trees
* Python3
* Workings of List
* Packages: Wrappers + new Versions (VTK7)
* Script Conversion
* Intermediate Outputs: reuse output module work?
** would be useful for both workflows (run to a particular checkpoint)
** and useful for notebook interfaces
* SQL backend: does it allow for concurrent access?
** Yes, real-time collaboration work for concurrent single vistrail access
** Should be able to access different vistrails at once without issues
 
=== April 19, 2017 ===
 
'''Updates'''
'''Updates'''
* [DK] bug fixes, improve-vt-bundle, uvcdat


'''Items to Discuss'''
'''Items to Discuss'''
* [LB] LSU, instrument building, tomography, interferometry system
* [TC] image registration/processing, infrastructure at BNL
* [ED] working on data analysis at BNL, hoping to leverage VT for data provenance, user activity
* [LB] tomopy and VT integration
* [TC] how to integrate tomography software
* [LB] goal to communicate with others, data security for crowdLabs?
* [LB] improve parameter exploration
** [JF] different ways: whole workflows, loops inside workflows, multiple threads, implict looping
** [TE] add new examples of saved parameter explorations, different versions
* [RR] Bug [https://www.vistrails.org/ticket/886 #886] ("from_root" parameter for <tt>change_selected_version()</tt>)
* [RR] [[API Design|API redesign]], example: [http://nbviewer.ipython.org/gist/remram44/2a67e6269ba2f052a19d notebook], target: [http://nbviewer.ipython.org/gist/remram44/a59a6dbf2026a0433ede notebook]
** [DK] looks good, nicer string rep, improve access to and use of results is important
* Jeffrey Guenther - list handling


=== July 30, 2014 ===
=== April 5, 2017 ===
 
'''Updates'''
'''Updates'''
* [DK] UV-CDAT animation, sql-alchemy branch update
 
'''Items to Discuss'''
'''Items to Discuss'''
* [TE] parameters in workflow - demo
** show functions on mouse_over
** move set functions to the top
** inline parameter widgets
*** show/hide all
*** toggle in ports list to show/hide these widgets
* API design [[API_Design|wiki page]]
** vistrails.api versus vistrails.core.api
** API is currently stateful, would like to change
** API calls using module names, parameter names instead of ids
** tighter integration with iPython (what were ideas from meeting last year with iPython devs)?
* argparse mutually_exclusive_group bug
* provenance viewer: make panel wider
* parameter views: combo box with different parameter viewing types
* change parameter not updating GUI
* installing python packages into binary
* modules that dynamically configure their ports


=== July 23, 2014 ===
=== March 8, 2017 ===
 
'''Updates'''
'''Updates'''
* [DK] output-modules merge, uvcdat, improve-vt-bundle
* [RR] Added a looping indicator on modules [https://github.com/VisTrails/VisTrails/pull/1217 #1217]
** This makes it easier to understand what's looping, no need to follow edges and count
* [RR] Refactoring CLTools for integration in Docker package
* [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)


'''Items to Discuss'''
'''Items to Discuss'''
* API design [[API_Design|wiki page]]
* [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers [https://github.com/VisTrails/VisTrails/issues/1216 #1216]
* [TE] output-modules feedback
** Can we move this up to the renderer modules?
** Can we have 'Output' module that uses input value to determine output?
** Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)
*** [DK] Unsure what this means, we could have more than one input port, seems like the mode of output is independent of what the value is
 
*** [RR] Infer from python types would take a lot of work, explicit easier to understand
=== February 24, 2017 ===
*** Usability vs. issues for developers
** Hide all but value port for outputmodules
*** Unsure here
** Make output mode be an enumeration rather than text entry (actually need to allow own entry)
**  Should vtkRendererOutput replace VTKCell?
*** [DK] Need to see if display views are involved, interaction handler, etc.
** Use module name as base name?
*** Also naming for parameters on command-line here
*** if you name module "flower", default base name is "flower"
** Also consider possibly only generating one output based on this type of naming
** Global imageFile format setting not working?
*** Put on trac
** Created WebGL output for vtkRenderer (As empty package)
** Matplotlib WebAgg and MBAgg backends better for web
** Clash between mode identifiers
*** Register this on trac
** [RR] how to load a specific mode if it isn't loaded?  How do we find it?
* [TE] other ideas
** See other workflow systems: http://imgur.com/a/xY37B
** Add icons to modules (graphical representation)
** Add module by double-clicking in the pipeline view and writing module name
** Edit parameters directly on the module
*** First step: move set parameters to top of list (or make more visible)
*** Second step: mine to find commonly used parameters
** Better use of hovering to provide module information
*** Show parameters on right
** [JF] Inspect workflows in the same window
* [DK] matplotlib feedback from Thomas Caswell
** figure top-down?
** see Tom's github fork of matplotlib of pyplot changes
** declarative syntax (MEP 25), goal for 1.5
** create artists in modules
* [RR] export-as-script
** It's mainly about renaming variables, to connect module and avoid collisions
** Currently gets code for script using package-provided to_python_script methods
** Using the [https://github.com/Psycojoker/redbaron redbaron] library for this (under early development)
** Need to handle the different types of input corresponding to different calls (get_input, get_input_list, force_get_input, ...)
* [BNL] Adding new palette tool
** search box with many fields to format query, send back dictionaries
* [BNL] Use numpy docstring format? yes
* [BNL] PEP whitespace? yes
* [BNL] python3 compatibility? no a big priority right now
* [BNL] PyQt new style connections: yes
* [BNL] Access to vistrails.org repository? yes


=== July 16, 2014 ===
'''Updates'''
'''Updates'''
* [TE] VisTrails 2.1.3 released
* [RR] Rich-text module documentation ([https://github.com/VisTrails/VisTrails/pull/1210 #1210])
* [DK] UV-CDAT
** Went over packages and fixed format; maybe we should have a test to validate RST?
* [RR] Single-instance shutdown with <tt>--remote-shutdown</tt> ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])
* [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment


'''Items to Discuss'''
'''Items to Discuss'''
* Integrate patches from github
* [RR] There are issues with list-depth
** we push from vistrails.org to github
** Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)
# git fetch git@github.com:VisTrails/VisTrails.git +refs/pull/17/head:refs/heads/pull17
** Issue with looping groups
# git checkout master
 
# git merge pull17
=== January 25, 2017 ===
# git push origin master
** Synchrotron needs tools for computational pipelines
* Discussion on scripting support
** we are going to have to support a set of restricted operations
** readability versus parsing
** blocks vs. functions
* layered API:
** scripting api is base, interactive api has convenience methods that use state
** design document on API, [[API_Design|wiki page]]?
* Other requirements for scripting
** higher-level operations to find results linked to certain provenance
* vt bundles
** should we look into improve-vt-bundle and sql-alchemy branches?
** design this (configurable bundles)
** have some hooks to add files to the vt bundle already
* move-dist-directory branch
** conflicts with python build directories
** does this affect build machines?


=== July 11, 2014 ===
'''Updates'''
'''Updates'''


'''Items to Discuss'''
'''Items to Discuss'''
* Claudio wants to make it easy to edit PythonSource without VisTrails
** [RR] Dropping url-encoding would allow you to edit them from XML? But they are also quoted because used in <parameter val="..."/>
* Scripting support
** Export workflow as standalone script:
*** Modules could provide their own logic, or we can make something up from compute() ast in some cases
*** Also, add annotations to script to make it easy to reconstruct (and module IDs, so we can update)
** Read back script as workflow
*** General case is hard
*** Building PythonSources from blocks might be doable
*** First step: simply reading some parameters from some modules in annotated script (i.e. has module ids)
* IPython notebook integration
** Wait on output-modules branch; then we can show visualizations in notebook
* API needs work
** API calls need to return things (to put them in notebook)
** Make initialization easier (shouldn't need to create application, etc)
** Currently stateful! (<code>app.new_vistrail(), ..., api.save_vistrails(f)</code> instead of <code>vt=Vistrail(), ..., vt.save(f)</code>)
** Should probably be in vistrails package directly


=== July 9, 2014 ===
* Scaling VisTrails
* sql-alchemy, uuid, bundle branches/merges
 
=== January 11, 2017 ===
 
'''Updates'''
'''Updates'''
* [TE] Added logging of parameter explorations
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Crash when adding String module on Mac [https://www.vistrails.org/ticket/760 #760]
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails
** Updated binary to Qt 4.8.6
** Have homemade workflow system based on R scripts and symlinks
** Release 2.1 with Qt 4.8.6 to fix String module crash
*** Each steps has multiple sets of parameters
* [TE] Table cell - Use HTML cell and D3?
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step
** Interactivity for free
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code>
** Can it work over multiple cells?
** VisTrails improves provenance capture and makes editing pipelines easier
** Will work directly on web and crowdlabs
*** Need better support for aggregation (groupby operation might be good enough; pandas?)
* [Claudio] Scripting support
 
** VisTrails is general but does not interact well with other systems
== 2016 ==
** What is not working, how can we make it easier?
=== December 14, 2016 ===
** Need ability to work without the GUI
*** Construct workflows using utility functions that does not use the VisTrails GUI
** Convert python script to/from vistrails workflow
*** Show PythonSource contents in script
*** Start with workflow that do not show stuff at spreadsheet
** TODO:
*** Design a better mechanism to integrate VisTrails with other tools, starting with Python
**** Provide the ability to save a workflow in a format that can be edited using a text editor -- for example, save a workflow and use an editor to change the contents of the Python Source boxes in the workflow
*** Provide functionality similar to what was implemented in UV-CDAT, where workflows can be used as black-boxes from Python. See slide 41 in http://uvcdat.llnl.gov/media/pdf/VisTrails_UV-CDAT_course.pdf
*** For which applications would scripting support be useful?
*** Look at [http://cycling74.com/products/max max] and [http://grasshopper3d.com grasshopper3d]
*** Claudio wants to test how parallellization works in VisTrails
*** How would scripting support work?
*** Meet on Friday


=== July 2, 2014 ===
'''Updates'''
'''Updates'''
* [TE] List handling improvements done
* [RR] Single-instance code: over-engineered, but will work
** Now List ports can connect to anything
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)
** Only exception is List<->Variant


'''Items to Discuss'''
'''Items to Discuss'''
* [DK] List/Variant bug: [https://www.vistrails.org/ticket/871 #871]
* Email from Matt Dirks: combobox should work, ask for more code?
** [TE] Fixed
* Email from marin.nl:
** [TE] Why is Map output a Variant?
** looks like they would benefit from new interpreter
** looks like this was done because Fold produces a Variant (makes sense since fold can be a sum), but Map should always be a list)
** how to store their consolidated provenance when they exchange .vt files around?
** this will allow depth 1 input ports to connect
 
* [RR] added List.head as depth 1 port
=== November 30, 2016 ===
** add documentation for this new functionality
* [DK] QPipelineScene.current_pipeline, controller refs [https://www.vistrails.org/ticket/869 #869]
* [DK] PythonSource crash? [https://www.vistrails.org/ticket/760 #760]
** update Qt for newer binaries?
** pyside support?
** Also, see [https://www.vistrails.org/ticket/737 #737]
* [TE] Log Parameter Exploration? [https://www.vistrails.org/ticket/870 #870]
** Very simple to enable, but we should annotate cell info
* [DK] tabledata pandas support?
** investigate on best way to make this available (data types, scripts, modules)?
* [JF] SIGMOD: python is becoming lingua franca for data analysis, munging


=== June 25, 2014 ===
'''Updates'''
'''Updates'''
* [DK] master merged into output-modules


'''Items to Discuss'''
'''Items to Discuss'''
* [DK] Global configuration settings for output-modules when using inheritance:
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])
** Example: Allow different file.suffix='.dat', imageFile.suffix='.png'?
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])
** For an individual output module, we only have one setting since imageFile derives from file, but how should this work if we a global setting for moduleA=vtkRendererOutput and moduleB=FileOutput?
** Still improving it
** decided that overriding for specific subclases makes sense
** Want to return something more structured to the client (success/fail code, stdout, stderr)
* [TE] automatic-looping
** Should be usable without Qt
** get_input now merges all connections when depth>0
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason
** Typechecking now respects configuration flags - fix for float/int reverted
* [DK] Working on batch execution using console_mode
** Issues with allowing List<->"Type of depth 1" conversions
* [DK] Settings aliased parameters when using core.api?
*** Treat list as Variant of depth 1
*** List->Variant need to be treated differently otherwise it would loop over the variant
** wouldn't be able to do implicit looping over List/Variant because we don't know what the list depth of incoming list objects?
* [RR] matplotlib issue: log doesn't reflect the actual execution
** can we update logging provenance here?
** other packages (VTK) have similar issues since the renderer actually causes some of the upstream internal modules to do computations.


=== June 18, 2014 ===
=== November 16, 2016 ===


'''Updates'''
'''Updates'''
* [TE] Chris is using BatchQ
 
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] automatic-loops-changes
** Mixing 0 and 1 depth types now works
** Removed use of moduleInfo
** transfer_attrs copies specs and control parameters
** Use get_input_list to support multiple connections
* [RR] matplotlib issues
** pushed branch to create a function that does this
** [http://mpld3.github.io mpld3] -- do they use same matplotlib model where figure must exist?
* [DK] list of depth 1, can we pass a list here
** exception of List input port type?
** will this work? allows only
* [TE] numpy types in matplotlib?
** e.g. different float types
** workaround for numpy for List types already
** what about adding other types later?  how would a package developer do this?
** Converter modules could work here but add more modules?  hide the extra modules?
** Unify type checking that occurs over connections (controlled by settings) and Loop/depth type checking method (now in vistrails.core.modules.vistrails_module)
* [RR] depth 1 port going to List head port
* [TE]  variant for (float, integer) tuple
* [RR] multiple connections into a depth-1 port only use one of the values
** should we have a self.get_input_list(flatten=True) option?
* [DK] output-modules: should work on so we can merge


=== June 11, 2014 ===
* Email from Pieter: executing pipelines with parameters from command-line using single-instance
** Seems to work, main instance gets full command-line and alters configuration temporarily
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful
** Actions:
*** Can add switch to kill main instance
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?
 
=== November 2, 2016 ===


'''Updates'''
'''Updates'''
* [TE] RemoteQ jobs can now skip upstream
 
* [TE] Fixed zipfile bug corrupting vt files
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]
* [TE] Fixed ungrouping caused by
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch
* [TE] Google maps binary available
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)
** Updated for zipfile bug but not for ungrouping bug yet


'''Items to Discuss'''
'''Items to Discuss'''


* [TE] File as parameter is confusing
=== October 5, 2016 ===
** signature changes with mtime
 
** Make File non-constant and use Path instead
'''Items to Discuss'''
* [TE] Talked with Chris at LSU
** PBS Jobs 20 min to a few hours
** VisTrails does most of what they need
*** Suggest we focus on bug-fixing and documentation
** Want interactivity
*** Fetch and visualize files while PBS Job is running
** May use Hadoop in the future


=== June 4, 2014 ===
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]
** unique ids
** bundles
** GUI fixes
** wrapping updates
** python3
** interpreter?
 
* command-line input parameters to provenance
** [DK] we don't create a persisted workflow from those input params, i think
** [DK] these are already defined as aliases so we treat them differently form a normal input param
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases
** Do the vistrails variables values get stored in provenance? (what if values change?)
 
=== September 14, 2016 ===


'''Updates'''
'''Updates'''
* [TE] Added skip upstream for RemoteQ jobs
* [DK] tabledata/gmaps updates


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Test run.py [https://www.vistrails.org/ticket/863 #863]
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)
** Changes in run.py are currently not detected by test suite
** Did we know/contribute to this?
** Other changes in configuration also don't get tested
*** [DK] Yes. We wrote the text for the VisTrails chapter
** What about importing unittest2 in 2.6?
** Drafted response
* [DK] userpackages/packages shadowing [https://www.vistrails.org/ticket/864 #864]
*** [DK] Looks good, will forward.
** add the directory
** being able to see both the package that exists in userpackages and packages
** checking loading/unloading
* [DK] need to improve upgrade utility methods
* [RR] Removing zip/unzip executables [https://www.vistrails.org/ticket/862 #862]
** check in the old commit logs and/or trac to figure out what the old problem was
* [RR] Rolling out conda packages: anyone has experience with Anaconda?


=== May 28, 2014 ===
=== August 31, 2016 ===


'''Updates'''
'''Updates'''
* [DK] rewrite-startup close to stable, fixed issue with going back and forth from v2.1 (trying to resolve package config settings issue), means output-modules is close, too
* [RR] Went over vgc.poly.edu/vistrails.org config:
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
** Fixed usersguide scripts (won't disappear again)
** Moved some websites over to HTTPS
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that


'''Items to Discuss'''
'''Items to Discuss'''
* [DK] extra_info on modules: is it really module-specific info?
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
* [DK] programmatic package configuration settings without direct imports?
** Get rid of more services? (svn, trac, ...)
* [TE] job-cache
* [RR] Notebook execution works. How do we support more languages (than Python)?
** New control parameter for caching module outputs to disk
** Need: detection, metadata reader, code to input values
** checks for Constant modules, so any Constant module can be cached
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?
** used for any module to save state of, not necessarily jobs
** is_serializable flag?
** will this be package-specific?
** have an is_pickeable decorator and allow more complex is_serializable interface code
* [TE] LSU feedback (Jinghua and Chris)
** Have used PBS with VisTrails (by running commands over SSH)
** Want to use distributed workflows with vistrails (more than one machine) Could use IPython?
** Using MOAB (Meta scheduler) Module would be useful
** Interested in moving files back and forth
** Have experience with SAGA (Could likely make use of it)


=== May 21, 2014 ===
=== August 17, 2016 ===


'''Updates'''
'''Updates'''
* [DK] output-modules, MplFigureToFile
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.
* [TE] VisTrails 2.1.2 Released (100 downloads so far)
* [TE] Matplotlib and sklearn are exportable using module-to-script api
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.
 
'''Items to Discuss'''
'''Items to Discuss'''
* [DK] persistence across machines (USGS use case)
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
* [TE] Merge automatic-loops-streaming into master?
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
** Implicit looping
* [TE] Wrap-up notes
** Streaming
** ControlParameters
** New schema version 1.0.4
** Possible conflict with multithreaded-interpreter?
* control parameters can also be used for other things like caching?
* [TE] job-cache
** Make module jobs independent of workflow
** Ability to cache jobs for all modules with constant outputs. Set in module/PythonSource/ControlParameter
** Needs branches:  controlflow-fake-signatures, jobs-use-signature_/connect-folded-module
* [TE] [vistrails-users] Michele: VisTrails on remote machine
* [TE] [http://saga-project.github.io/saga-python/ SAGA-Python]: A Light-Weight Access Layer for Distributed Computing Infrastructure
** Alternative to support distributed computing in VisTrails?
** Support for most job schedulers
** Could replace BatchQ/SSH/PBS/LSF/HTTP modules
** No Hadoop support
** Open Grid standard
** Compatibility with VisTrails/ModuleSuspended?
* [RR] Debian vistrails package


=== May 14, 2014 ===
=== August 10, 2016 ===


'''Updates'''
'''Updates'''
* CrowdLabs mashups done
 
* v2.1.2 release ready
* [DK] work on rewrite-startup, make sure preferences work in new style; output-modules
'''Items to Discuss'''
'''Items to Discuss'''
* vagrant for reproducibility
* [TE] Python Wrapper and scripting
** simpler interface for exploring the experiments
** All changes merged into python-wrapper branch
** [RR] generating a VisTrails workflow from reprozip was planned, but not integration of the tracer in VisTrails, to trace VisTrails workflows -- it probably makes sense. See https://github.com/remram44/reprozip-ptrace/issues/8
*** Tests pass
*** Still need to update sub-branches
** Minor fixes
*** vtkPlot3DReader upgrade bug
*** Added tuple exporting
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
**** Could be a bug in the old vistrail, but not sure
* [DK] Subworkflows and uuids


=== May 7, 2014 ===
=== August 3, 2016 ===


'''Updates'''
'''Updates'''
* CrowdLabs mashups
 
** Uploading works
 
** HTML looping works
'''Items to Discuss'''
** Flash removed (actually hidden)
 
* [DK] output-modules
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]
** Running in batch mode switches to file mode
** Connection variables now lowercase
** Allows parameters to be specified on command-line
** Can we use code as output variables?
** Work on GUI for parameters
*** E.g. "vtk_dataset_mapper.GetOutput(0)"
*** Could mutate output when output method is only supposed to be called once
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)
 
=== July 27, 2016 ===
 
'''Updates'''
 
* [DK] DB bundle serialization implemented
 
'''Items to Discuss'''
'''Items to Discuss'''
* VTK 6
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]
* [DK] output-modules: target single module parameters?
** locale?
** [RR] use labels to name specific modules, useful in command-line
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]
* scripting support
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches]  
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]
** Still need to do something nicer for the patch formatting
*** patch keys currently contain non-python-variable characters but this can be fixed
*** Need to replace string template with python variables that is replacable using RedBaron
**** Patches could then look like:


=== Apr 30, 2014 ===
    def vtkInstance():
        self.vtkInstance = self
 
=== July 20, 2016 ===


'''Updates'''
'''Updates'''
* Does spreadsheet resizing work correctly? Is it too slow? ([https://www.vistrails.org/ticket/833 #833])
** looks like on downsize the image is smaller than it should be and then flickers to flll space (upsize is ok)
* [TE] Mashups on Crowdlabs
** HTML mashups is now rendered in the crowdlabs template
** Added embedding html mashups (using jsonp)
** Fixed uploading mashups with a vistrail
** Investigating issue with saving mashups to DB
* [DK] rewrite-startup, output-modules
** working on command-line and global parameters


'''Items to Discuss'''
'''Items to Discuss'''
* 2.1.2 release
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]
** some major bugs a
** A patch is a string template with self/input/output variables for patching a class method
* VTK 6
** Patches are stored in xml specification
* iPython integration
** Patches are assigned to methods in ClassSpec's
** how to migrate from scripts to workflows
** Used both for execution in vistrails and when exporting as script
* Introducing vistrails as a normal python package
*** Executed with "exec" when run in vistrails
** check on anaconda support for Qt, VTK support?
*** Turns patches into natural calls in script
* output-modules
** Wrapping VTK library no longer needed
** make sure that filenames can be set
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)
** output directory
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)
** interpreter should return a dictionary with python objects produced (what is input to the output modules?)
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)
** unify OutputPort and OutputModule? Right now, act differently, but function in similar ways
** TODO's
* directions: scripting support, large-scale data analysis, reproducibility
*** Add to export_as_script
*** Patch Modules that represent Python functions as well
 
* [vistrails-users] Read data from SQL example?


=== Apr 23, 2014 ===
=== July 13, 2016 ===


'''Updates'''
'''Updates'''
* debugger
 
** have to enable the configuration option, starts python debugger in console
* [DK] bundles on sql-alchemy branch
** have similar flag for test suite
** Bundle, BundleMapping, and BundleSerializer
* Branches to be merged in: new-url-package
** New BaseSerializers to deal with versions of directory, zip, db serializers
** API calls don't go through upgrades currently, can we support this so we don't break scripts?
** Can register the BundleSerializers with the base serializers
** added HTTPS certificate verification
** Translation between bundle versions
** new dependencies for binaries?
** Bundle.add_object "just works" if an appropriate mapping was registered
* dont-use-modules-as-data?
** branch is ready?
** changes a lot of packages,


'''Items to Discuss'''
'''Items to Discuss'''
* output-specific modules (e.g. vtkInteractionHandler)
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]
** only need to be enabled when spreadsheet is active, but if we create a workflow with such a module, it would be nice to still be able to run that workflow in batch mode
** API Module can be executed using '''compute''' method
** not verify entire workflow?
*** Takes list of outputs to compute + inputs as keyword arguments
** load module logic on-demand so we don't need dependencies?
* Status of general cell synchronization (vistrails-users)
* reload packages
** interactions with spreadsheet, stale controller objects
** spreadsheet may need to get information about pipeline when copying cells (get from controller?)
* [TE] server mode
** Changed to non-interactive
** Pass debug messages back?
** Auto-load packages security?
*** startup.xml file gets corrupted?
*** using same xml file for multiple instances
** Locale errors
*** crowdlabs defaults to english, but vistrails server does not.
*** set LANGUAGE will fix it
*** should we set this environment variable in VisTrails
*** decided to chose English for now, can reevaluate when unicode branch is ready
** Default to HTML5 interface?
*** Flash interface as backup
*** look at animations for mashups in HTML5
* module descriptor weakrefs
* VTK6
* continue-dialog branch
** changes SpreadsheetCell modules
** merge into 2.1 for the LSU folks
* output modules: where should the output module live? only one package?


=== Apr 16, 2014 ===
=== July 6, 2016 ===


'''Updates'''
'''Updates'''
* Branches to be merged in: readfile-module, save-module-moves ([https://www.vistrails.org/ticket/853 #853])
 
* CrowdLabs needs update to VisTrails 2.1 with schema 1.0.3
* [DK] Working on bundles
* [DK] OutputModules (FileOutput)


'''Items to Discuss'''
'''Items to Discuss'''
* LSU meetings
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]
** Pausing workflow/confirmation module ([https://www.vistrails.org/ticket/854 #854])
** Call module in script using vistrails API
*** Checking an intermediate result before carrying on with the workflow
** Can be used when no export method is defined.
*** Their problems: Qt event loop, sinks ordering
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?
** CLTools problem
**[RR] No, unreleased
** crowdLabs usage? (they have 100+GB files)
* [DK] New bundle organization
*** Parameter exploration
** BundleMapping contains individual mappings from raw objects to BundleObj
** Running mathematica on a cluster (JobSubmission, RemoteQ?)
** Must be separate in order to have in-memory, separate from serializer settings
** Exporting VTK visualization (vti/vtp?) for Kitware's [http://www.kiwiviewer.org/ KiwiViewer]
** Wrapping SNARK09 (C++ code) as VisTrails modules
* eo4vistrails (Derek, Terence)
** Add FTP module?


=== Apr 9, 2014 ===
=== June 29, 2016 ===


'''Updates'''
'''Updates'''
* Branches to be merged in: export-cells, reset-cell-sizes-button (-> v2.1; icon?), export-versiontree-dot & custom-colors, input-module-no-subclass
* [DK] UV-CDAT release, OutputModules (see below)


'''Items to Discuss'''
'''Items to Discuss'''
* In-place spreadsheet updates: [https://www.vistrails.org/ticket/847 #847]
* [TE] Export VTK as python script
** need to be able to replacement mode toggle on and off
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]
** Colin has a working solution, is there a need to do this globally?
** Preludes (imports) still need improvements
* Autosave for existing vistrails not working <strike>[https://www.vistrails.org/ticket/849 #849]</strike>
** Works with vtkInteractionHandler (clipping plane)
* Other tickets from USGS
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)
** Reason for pipeline scroll area being 100 times greater than pipeline?
** vtkRendererOutput Uses VTKRenderWindow as output
** File Selection Dialog <strike>[https://www.vistrails.org/ticket/846 #846]</strike>
*** Should support different output modes?
** Copy/paste with retained connections [https://www.vistrails.org/ticket/851 #851]
*** Does not work with multiple outputs (Next cell is displayed when current is closed)
** Auto-connect ports when moving modules [https://www.vistrails.org/ticket/852 #852]
*** Uses wrapped vtkInstance (as vvtk)
*** Also think about fixing some of the port-snapping for selecting a port on the same module when there are multiples of the same type
**** Create new patch system with code that can be used both in vistrails and in a python script?
* Unicode support
** question on mailing list
** [RR} dates is something else, should be able to put on v2.1
** fix for LC on master, port to v2.1?
** should try to persist dates in numeric formats, but GUI level can be language-specific.
* VTK 6 support [https://www.vistrails.org/ticket/739 #739]
* [TE] Analogies does not delete annotations [https://www.vistrails.org/ticket/848 #848]
** Code never worked as intended
* [TE] Add eo4vistrails features to vistrails? https://github.com/ict4eo/eo4vistrails
** Look at these and try to determine what may be appropriate
** data types are probably most important, doesn't look like anything there
* [DK] [[OutputModules]]


=== Apr 2, 2014 ===
=== June 22, 2016 ===


'''Updates'''
'''Updates'''
* [TE] Added annotations and control parameters to visual diff
** three tabs
** TODO: analogies
*** just work on making sure the actions execute, don't worry about the matching
* [TE] Build documentation using [http://128.238.102.101:9050/builders/usersguide-master buildbot]
** makefile is run directly so no need to patch (no merges)
** the nightly build still generates the pages, and the cron job now pulls the built pages
** cron job is set to pull built pages after nightly build
* [TE] Updated [http://128.238.102.101:9050/builders/coverage code coverage tester] (currently 33%)
** coverage is done manually, it is not done nightly
** takes 15-minutes to run, on vis-7.
* [DK] Output modules


'''Items to Discuss'''
'''Items to Discuss'''
* Unicode support
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])
** question on mailing list
** Uses module location to find module remap (The best we can do right now?)
** fix for LC on master, port to v2.1?
** Also fixes remapped port names for connections to existing modules  
* Output modules
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name
** outputs, modes, mode configurations
 
** spreadsheet as a package
* [TE] Working on exporting wrapped libraries (VTK)
** output mode configuration dependencies
* VTK 6 support [https://www.vistrails.org/ticket/739 #739]
** trying to complete output modules before new VTK package


=== Mar 26, 2014 ===
=== June 15, 2016 ===


'''Updates'''
'''Updates'''
* Branches to be merged in: application-no-default-argv, export-cells, testsuite-module-imports, reset-cell-sizes-button (icon?), export-versiontree-dot, custom-colors
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])
* [TE] Added branch control-parameters (used by list handling)
** Fully implements connection merging and list looping
** For setting hashed control options on modules (How to combine lists, while looping, caching)
** Missing: Include in diff/merges/analogies?


'''Items to Discuss'''
'''Items to Discuss'''
* iPython
** script to workflow bridge
** access to VisTrails commands in iPython
** changes to the API
* Multi-faceted output
* [RR] Documentation buildbot
** [TE] currently [https://www.vistrails.org/#Testing here]
** problem is that we don't get a message when the documentation failed to build
** what about copying the output from a build machine to vistrails.org?
* [RR] Last(?) migration issue: [https://www.vistrails.org/ticket/837 #837]
** [DK] to look at
* VTK 6 support [https://www.vistrails.org/ticket/739 #739]
* spreadsheet export button in UV-CDAT
* Develop menu?


=== Mar 19, 2014 ===
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])
** Analogy only works We need to use the non-upgraded action chain
** But pipelines need to be valid (matcher uses module portspecs)
** Tried to get upgrade remap from upgrade action, but this seems impossible
** Use module location instead?
*** Location should be identical for one-to-one module upgrades
 
=== June 1, 2016 ===


'''Updates'''
'''Updates'''
* [DK] UV-CDAT #407 fix, version tree rewrites
* VisTrails 2.2.4 announced (finally)
** 5 survey replies
** 47 downloads last week


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Cleaning up test suite log
* [TE] Export VTK as script (python-wrapper)
** Deprecated core.modules.constant_configuration imports
** to_python_script
*** check whether the deprecations for vistrails.core.vistrails_module.VistrailsModule camel-case are similar
*** needs the pipeline (or at least the connections)
** KeyError (Ticket [https://www.vistrails.org/ticket/837 #837]), one of [DK]'s commits
**** Should not generate code for any possible VTK method (exporter detects which ports is used in the source)
*** [DK] to look at
**** VTK needs to know connection order
** Skip modules not to be imported (or fix/remove)
**** Adding functions directly to the script results in nicer code
** Python invalid drawable (Probably from mashups)
** Add support for multiple input connections
** Missing version 0.8 of package org.vistrails.vistrails.tests.upgrade (bug?)
** Add support for list looping
* [TE] Add annotations to signature? (hash-annotations)
** for the streaming/automatic looping for new values on the modules
** could be any module
** label doesn't affect caching
** prefix, flag involved, add fields to module about looping
** schema change? problem is that don't want to have to recreate module each time
** backward compatibility
** new type of "control parameter"
* VTK 6?
** See Ticket [https://www.vistrails.org/ticket/739 #739]
* UV-CDAT #408: [RR] has worked on this
** support adding more formats to export
** decided to have single menu item, and put all available formats in the dialog
* Spreadsheet resizing bug
** if you resized the sheet, would reset to equal-size cells
** spreadsheet-resizing branch changed this to allow non-equal-size cells on resize
** has some math issues because doing this dynamically difficult
** button to resize to equal-size on branch
** updating the documentation
** [TE] can we store the locations more precisely? (problem might be if a header column is also resized)


=== Mar 13, 2014 ===
=== May 25, 2016 ===


'''Updates'''
'''Updates'''
* [RR] Package dependency ordering issue ([https://www.vistrails.org/ticket/829 #829]); fixed(?), merge?
* Mailing-list fixed
** [DK] Yes, merge
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])
* [RR] Will merge in tabledata additions
 
* [RR] New SQL package, using tabledata and sqlalchemy. Should be fully backward compatible(?)
'''Items to Discuss'''
** Check if Fernando has anything here from the Oracle workflow and database provenance
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)
* [RR] Searching in vistrail and regex ([https://www.vistrails.org/ticket/373#comment:7 #373], [https://www.vistrails.org/ticket/779 779]); fixed?
* [TE] Union ports
** Error was hidden (entering '?' searched for '.*?.*' which was a valid regex but still didn't do what was expected)
** Show union ports in pipeline (DEMO)
* [TE] IPython console now shows <STANDARD OUTPUT> before printing from stdout/stderr
 
* [TE] Vistrail variables fixed
=== May 18, 2016 ===
** Deleting a module with a variable did not remove the variable from the workflow
** Copying modules with variables now works


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Do not run modules upstream of submitted job
* [TE] mailing list still down - Julio working on it
** jobs-use-signatures: Uses the subpipeline signature to identify jobs
* [TE] Improved union port
** works: if job is submitted, check it without running upstream (we don't need the parameters this time(?))
** New schema 1.0.5 adds portitem.union string attribute
** doesn't work in Map
** Works with pipeline view, parameter exploration view, and mashup view
** jobs-use-signatures_/connect-folded-module
** Should work well for matplotlib
*** create a connection from folded module to InputList upstream
** Use with VTK ports?
*** looped module can run upstream if it needs, else we don't
*** It uses [item_1,item_2] style ports usually with different tuple length.
*** problem: the Map still needs the list, for length and type-checking
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]
** Could we rearchitect the looping behavior? Basically, just copy the pipelines and run them with each entry from the loop
*** Provenance could expand
* [RR] module-upgrade-recursion status? (not yet merged)
** [DK] to look at and merge (should be done)
* [DK] UV-CDAT Issues: [https://github.com/UV-CDAT/uvcdat/issues/407 #407] and [https://github.com/UV-CDAT/uvcdat/issues/408 #408]
* [RR] Thumbnail comparison bug: different references are selected on master and dont-use-modules-as-data! (pipeline results are identical) Problem with upgrade code?
** using different reference images
** [TE] to look at
** look at the console_mode, how it gets back the upgraded version id
** filed as [https://www.vistrails.org/ticket/831 #831]
* [TE] Vistrail variables
** Normally only one hidden module exist for each vistrail variable, but pasting a module with a vistrail variable will create a new hidden module. This does not break anything, but should be taken into account when modifying the code.
** When a vistrail variable module is created, an input named "value" is created. A bug with default values gave it the value "None". This is fixed. To make validation of existing vistrail variables work we need to check if the value is "None", but this would not allow "None" as a string.
** [DK] to look into this, we cannot have "None" there.
* [JF] NSF Big Data proposals (due in June)
* uuid status
** need to check tests and features outside of normal execution


=== Mar 5, 2014 ===
=== May 11, 2016 ===


'''Items to Discuss'''
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?
*** Shows single port for all ports with same sort_key
*** Select specific type when adding/showing
*** Only one type should be visible on the module at any time
*** Making tuple parameters union types is probably too complicated
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly)
* [TE] Feature suggestion: Let connections connect to functions/parameters
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters
** Use separate views for ports and functions (like before)
*** Allow rearranging of functions
** Pros:
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.
**** Makes List module simpler
*** Connect to parameter in tuple
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds.
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.
** Cons:
*** Interface more complicated and crowded
*** More difficult for users to grasp?
*** No immediate use case
*** Makes no union type for tuple parameters a bigger problem.
* [TE] Publish user survey?
** No response from vistrails release yet
** Not announced on vistrails-users yet.
=== May 4, 2016 ===
'''Updates'''
* VisTrails 2.2.4 released
'''Items to Discuss'''
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]
=== April 27, 2016 ===
'''Updates'''
'''Updates'''


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] New SQL package, using tabledata and sqlalchemy
* [TE] Wrapping Bokeh
** No upgrade possible? (output changed from List (of rows) to Table)
** New method type "operation"
*** Should we make it a new package?
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output
*** Will try to make backwards compatible
**** Glyph sets function name and arguments as output
* [TE] Change pinning to always show pinned panels? ([https://www.vistrails.org/ticket/828 #828])
***** Not serializable, but should always be computed together with the subject of the operation.
** Yes, it is a better behavior
**** Figure applies Glyph operation on itself
* [RR] Package dependency ordering issue ([https://www.vistrails.org/ticket/829 #829])
*** Useful for matplotlib?
** Will try to fix
**** matplotlib currently find actors by searching upstream?
* [RR] Merging multithreaded-interpreter and JobManager
**** But this is more difficult since matplotlib is not function-based.
** Need examples of JobSubmission
** New "union" port type
* [TE] Merging automatic-loops-streaming
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value
** Try to merge with multithreaded-interpreter
**** Uses only first value in tuple
* Wiki menu looks strange in Chrome
**** Bokeh have many properties that support (list, string, Float)
** [TE] Will try to fix
*** Does not work with depth=1 types (e.g. Float of depth 1)
**** Turn Depth 1 types with known type into lists?


=== Feb 26, 2014 ===
* [TE] New bugfix release
** Finalize survey


=== April 20, 2016 ===
'''Updates'''
'''Updates'''
* Branch to be merged in: hidden-port-icon (v2.1)
* Merged upgrade fixes
* [RR] tabledata-merge-tabdata: added tests, fixed issues (todo: json)
** Now has infinite loop test
* [TE] Added package RemoteQ (BatchQ/PBS/Hadoop)
** loop test now correctly loops 5 times
** new Mac binary that includes everything here
*** remap_module used wrong (current) package version when creating ModuleDescriptor
* [DK] gmaps, climate proposal
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])


'''Items to Discuss'''
'''Items to Discuss'''
* Climate proposal
* New bugfix release
** recommendation and best practices
** Merge usagestats
** automatically segmenting the version tree, better modes of interaction
** Need to finalize survey
* [RR] tabledata
 
** selections, projections, joins
=== April 13, 2016 ===
** writing out table as csv?
'''Updates'''
** much improved bugs
 
** [DK] should merge aggregate module int
'''Items to Discuss'''
** [RR] working on different json representations
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])
* [TE] Hadoop configuration
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?
** has package configuration now
 
** still have machine module to hardcode a configuration for a workflow
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])
** http://www.vistrails.org/index.php/Hadoop_Package
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]
* Data cleaning/repo:
** Cherry-pick test to preserve source?
** always need the vistrails engine to run things
** Need maxPipelineFixAttempts option?
** scripting interface
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?
*** Would need to be moved into handle_invalid_pipeline
 
=== April 6, 2016 ===
'''Updates'''
* [TE] Query View improvements
** Done except for displaying "Version Info" and #1164
* [RR] Working on finishing new interpreter (finally!)


=== Feb 19, 2014 ===
'''Items to Discuss'''
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])
** Try latest upgrade first, then previous.
** If all fail, select latest upgrade?


=== March 30, 2016 ===
'''Updates'''
'''Updates'''
* [TE] automatic-loops-streaming works
* [TE] Hadoop package:
** Merged BatchQ-PBS and hadoop into RemoteQ
* [DK] improvements to tabledata, gmaps
* [RR] java-pkg is done


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] automatic-loops-streaming
** Add documentation
** Perhaps a matplotlib example where the plot updates as more data streams
** Do we want to add additional controlflow features?
*** Implemented while module, but this needs to be improved because of annotations
*** annotations don't invalidate cache
*** could include annotations in hasher for cache updates
*** how does if module work?
** [RR] pairwise for three ports?
*** the list modules do this, but we may need an interface for this
** http://dev.mygrid.org.uk/wiki/display/taverna/List+handlings
* [TE] Hadoop package:
** Best way to separate task and configuration parameters?
** URI is different for different clusters
** use package configuration variables
** move from checkbox to allow more general configuration?
** use configuration file for this?
* [RR] working to merge multithreading code with job-monitor, look at streaming
* [RR] java-pkg
** works for some classes (call constructors, setters, getters)
** cannot call any methods from VisTrails, don't want to mutate
** can call these methods from your own modules, also insert code in Python into these packages
** may add in an interface for loading jar files and wrapping them


=== Feb 11, 2014 ===
* [TE] Query View improvements
** Unified do_version_switch and get_upgrades as validate_version()
** Problems with query view using its own controller
*** Fixed module info not being displayed for query results
*** Still no version info for query results
*** Still an issue with version query view not being updated


=== March 23, 2016 ===
'''Updates'''
'''Updates'''
* [TE] automatic-loops-streaming - partially works
** Using a python generator for each streaming module
** TODO: Streaming Sinks
*** last module collecting values
** Logging/Feedback performance
** Demo?
* [DK] vtk-new-package
* [RR] java package: conflicts with dlls and jpype, going to work


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Hadoop package:
* [RR] Usage reporting almost ready to go
** a few installation issues, vistrails-hadoop pkg on git but was not accessible to public
** Need to record whether specific features are used: paramexpore, mashup, query
** same package for local cluster and aws? Yes, but need to change two lines, path to configuration files on server
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation
** could config files be set in a module instead or in the preferences?
** How to record size of vistrail?
** directories are hardcoded in hadoop package, need to be parameters
*** [DK] IdScope already walks over vistrail, use that
** anything that can change in different installations should be parameters
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines
** can be hard to figure out what does change
 
* [DK] merge tabledata-merge-tabdata, matplotlib-add-helpers?
=== March 16, 2016 ===
** examples are not very useful, no example with multiple plotline modules?
'''Updates'''
** Update matplotlib examples to better show off capabilities
* [DK] package loading speed, kwargs
** kwargs can be significantly slower
* java issues:
** [RR] should work, just a few installation issues
** machine learning in python: scikit-learn (http://scikit-learn.org/stable/)


=== Feb 5, 2014 ===
'''Items to Discuss'''
* [TE] Upgrades in Query Mode
** Now using code similar to do_version_switch
*** Uses handle_invalid_pipeline
*** Differences: Not using current_pipeline, flushes changes
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)
*** Integrate to avoid repeated code?
*** Use this for query mode?
* [TE] VisTrails 2.2.4 ready for release?
** tej package updated. Others?
* [JF] upgrades and the version tree
* [JF] provenance in Jupyter
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]


=== March 9, 2016 ===
'''Updates'''
'''Updates'''
* [TE] automatic-loops
** PortSpec-based version now works
*** Connections show how modules will be iterated
** Streaming is tricky
* [DK] Examples, merging tabledata


'''Items to Discuss'''
'''Items to Discuss'''
* mailing list question
* [TE] Fixing search mode
** no resopnse yet
** String query now matches all by default
* UV-CDAT: warnings when downsizing spreadsheet
*** This caused pipeline queries to never match
** problem: warnings will pop up a lot
** String query can search: User/Notes/Tagname/Time/Module name
** keep offscreen cache of (x,y) locations in spreadsheet?
*** Should we search module names by default? (Disabled by default because it is expensive)
** cache pipeline instead of full rendered pipeline
*** Fixed notes search
* [TE] streaming with branches becomes interesting
** '>' parameter queries in documentation but has been removed
** only want to update upstream branches once
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)
** Taverna has streaming-style (pipelining)
*** Upgrades still a problem
* PythonSource deleting port?
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)
* [RR] Java: possibly considering the matplotlib-style port creation versus java-style ports
*** [JF] Allow search on past tags
 
* [TE] Create new bugfix release
** Finish querying
** Path issue on windows


=== Jan 29, 2014 ===
* [DK] Look into sqlalchemy and new bundle branches


=== March 2, 2016 ===
'''Updates'''
'''Updates'''
* [TE] automatic-loops - Groups and Multi-level looping now works
* [TE] Fixed cache bug with port specs
* [DK] matplotlib-add-helpers: Rework the semi-automatic generators
* [TE] VisTrails Next
* [RR] java-pkg: bringing in old code, spreadsheet still works, probably going for dynamic generation of packages
** Most packages now work
** looking at jpype versus pyjnius
** No qgis for Python 3


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] str-format-module: in master. Merge in v2.1? Having to open the configuration window and click on a button is awkward
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]
** [DK] watch out for dynamically created format strings, could cause inconsistency at compute time if the format string doesn't match the ports
 
** Go ahead to merge since similar problem with PythonSource
=== February 23, 2016 ===
** What about using number of connections to enforce this? (set to 0 for config-style ports?)
'''Updates'''
* Streaming in VisTrails
* [TE] Fixed PythonSource 100% CPU bug
** Q: how is parallelism working in VisTrails? A: both theading and multiprocessing, but separate branch
* [TE] VisTrails Next
** for streaming, order of input and outut must be preserved
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples
** module can generate a number of pieces, a module that collects needs to know when the stream ends
*** Now uses REST service, tabledata, and Google Maps
** need to preserve order of map operations and
*** Web Service replaced with DownloadFile
** goal is to reduce the storage costs, want to make sure that we don't keep entire data in memory, don't generate the entire list at beginning
** hyperflow needs to know when a stream starts and when it ends: start and end token, hyperflow wraps data in a token
** Huy advises getting rid of the list and using tokens instead?
** interpreter can call compute-upstream and just tell interpreter to go upstream
** VTK streaming interpreter: might be in svn
** Remi: just be able to pass an object on the conection that knows the shape of the data and how to get next element (basically an iterator)
*** where do we call get-next? module in the collector calls
** main advantage of streaming is being able to do aggregation during the compute (otherwise, we're doing map-reduce)
** the iterator encapsulates the calls to the module's compute
** issue with logging? already rewritten for parallel, should work here
** Examples: triangle_area, image histograms
** Slowdown caused by lists? colors? logging?
** Schema changes: need to record depth, what type of semantics for combining port values
** Also probably means that we don't pass just raw values, need to wrap with depth and iterator
** Issues Huy raised:
*** issue with multithreading and streaming both enabled
*** triangle with modules: need to be careful about order with items coming from two different modules (e.g. BuildTable in Rémi's example)
* UV-CDAT


=== Jan 22, 2014 ===
'''Items to Discuss'''
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.
** New 2.2 release soon?
* [RR] Anonymous usage reports
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?
* noWorkflow - provenance for python scripts
** Recent work on tracking dependencies
* Also integrates ReproZip with Jupyter
** [http://mybinder.org mybinder]
*** Use with ReproZip to automatically build Jupyter notebook interfaces
* Daves comments on Jupyter integration
** It is possible to identify cell order on the server side
** Not trivial to track order of cells on client side
* How does Galaxy workflows relate to VisTrails?
** Packaging in Galaxy is hard
** Galaxy for biomedical workflows, difficulty making it general
* Action items
** How difficult to track cell dependencies in Jupyter notebooks?
** Use noWorkflow to track cell dependencies?
** How difficult to modify mybinder to use ReproZip packages?


=== February 17, 2016 ===
'''Updates'''
'''Updates'''
* [RR] Branches to be merged in: reload-disabled-package ([https://www.vistrails.org/ticket/714 #714]), ungroup-keep-disconnected-ports_/hybrid
* [TE] VisTrails 2.1.1 released
* [DK] matplotlib, uuid work


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Automatic Looping
* [TE] Fixed Group memory usage
** Using ListOf module as input will trigger the looping
** Modules in groups are supposed to be freed when done executing
*** Runtime type checking because we don't have a ListOfT class for each type.
*** Prevented by logger that stored references to the modules (fixed, now using id(module))
*** [DK] USGS interested in working with this
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)
*** Add a merge module to make it clear where the iteration stops and ends
* [TE] VisTrails Next
*** Have different type of ports
** Removed deprecated code
*** Remi to talk to Huy about streaming ideas
*** Core imports that should be gui imports
*** can we detect the iteration and draw those modules differently
*** GetInputFromPort etc.
*** decided that some fold/merge module would be useful, even if we don't always need to use it explicitly
**** PythonSources will be upgraded
* [RR] Ticking in matplotlib
*** Fixes for Python < 3
** formatters, locators, tick scale
*** fixes for VTK < 6
* [RR] Sort tickets? There are [https://www.vistrails.org/report/18 issues for the 2.0 milestone] and [https://www.vistrails.org/report/17 issues that affect released versions]. Tickets assigned to 2.1 are [https://www.vistrails.org/report/19 here].
** TODOs
* [DK] To contact USGS about the new persistence package and get feedback
*** Remove RemoteQ
* [DK] Merge weather and subway examples to use new tabledata package
**** Port Hadoop to tej (Already have pbs)
* Look into pandas http://pandas.pydata.org/
*** Merge package wrapper
** could be helpful to support some of the table operations
*** Finish SQLAlchemy branch
** in the future, we may want to support it for data analysis and stats (but for this, additional dependencies are required)
*** Finish new bundle branch
** [JF] Will contact Jeff and Matthias about moving to Python 3
* [JF] Query by Example is broken
* [JF] VisTrails Data Cleaning Project
** Use VisTrails to process and explore task-specific data while keeping provenance.
** Remi's Idea
*** Use VisTrails as the backend in a Jupyter notebook
*** Overcome problems with executing cells in random order
**** Interesting research topic, but little interest has yet been noted
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)
*** Impose restrictions to notebook cell boundaries
*** It will only work for functional operations
*** Can we fix, or detect, non-functional (mutating) operations?
*** Analyse code and data flow and see if it mutates data in unexpected ways?
*** Mutable operations can be created in many ways in Python
* Next meeting Tuesday 11:30


=== Jan 15, 2014 ===
=== February 10, 2016 ===
'''Updates'''


'''Items to Discuss'''
* VisTrails Next
** Test suite passes
*** Python 3 less deterministic, module upgrade order random.
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...
** All of our packages should run fine :)
** VTK thumbnail comparison works
*** Rendering bug on some systems
**** QVTKCell now inherits QGLWidget
*** Old thumbnails had wrong height due to spreadsheetwindow being too small
**** Fixed by regenerating thumbnails
*** Silenced deprecation messages when generating modules
*** Many new classes in VTK 7
** Remove deprecated code
*** Core imports that should be gui imports
*** GetInputFromPort etc.
**** Will break existing PythonSources
**** Try to upgrade?
*** Old-style packages
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
***** Read info.py without importing package?
*** Fixes for Python < 3
*** Fixes for Qt < 5
*** fixes for VTK < 7
*** Remove self port?
=== February 3, 2016 ===
'''Updates'''
'''Updates'''
* [RR] improves-logging:
** Pass exceptions to it directly for single line, or traceback.format_exc() for traceback
** WARNING level now printed in the console by default, -V 1: INFO (log() calls), -V 2: DEBUG (debug() calls)
** Messages view now shows whatever is selected, no matter the console log level
** Python warnings get captured
** Use warnings.warn(..., category=VistrailsWarning|VistrailsDeprecation) to warn once
** Backport to v2.1? (so deprecation warnings actually show up)
*** -> Yes, include in v2.1.1
* [TE] Hadoop package documented here: http://www.vistrails.org/index.php/Hadoop_Package
* [DK] UV-CDAT, LSU


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] input-module-no-subclass: input ports of type 'Module' shouldn't accept any output? These are used for 'self'->controlflow module connections...
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]
** controlflow modules have a function port that accepts a module
 
** check for modules that might use Module for something else
* VisTrails Next
** suggestion is to use Variant for any type being supported (StandardOutput, List, others)
** All tests but thumbnail comparison passes
** warns for now (doesn't actually disallow the connection until next version)
** Still issues with gui and more packages needs to be tested
** -> David wants to look at this more
** PythonSource and MplSource upgraded with lib2to3
* [RR] ungroup-keep-disconnected-ports: alternative ungrouping, keeps pipeline disconnected and preserves InputPort/OutputPort modules
** Keep contrib packages? Update?
** ideally, want to edit in place, could try to emulate the edit subworkflow code here, just have to save things back to the original vistrail
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]
** -> hybrid approach, materialize the unconnected ports
*** Supports Python 3
* Bugfix release? Would solve issues with server and parameter exploration. (also, logging improvement?)
*** Will hopefully get into conda and others soon
** [TE] Also fixes to workspace
*** Deprecates a bunch of modules
* [DK] font issue for OS X 10.9
 
* Look at Java support in Java?
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next
** can we leverage Rémi's past work here? on Weka
** Runs!
** pyjnius: cannot subclass a Java class in Python, should be able to create and call out to java classes
** Uses conda
** issue with Jython is the interface
*** Many levels of requirements
** also had java-based spreadsheet
**** apt packages
* also look at moving VTK to a matplotlib style generation
**** conda packages
**** python packages in requirements.txt
**** python packages in setup.py
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!
*** But useful if we want to use conda in our own build process.
** Lots of errors I do not get locally?


=== Jan 8, 2014 ===
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]
** Have you used any alternatives to vistrails?
*** Relevant? More options?
** How often do you use any of these particular features in VisTrails?
*** Add/Remove alternatives?
** How important would you rank the following potential improvements to VisTrails?
*** Add alternatives?


=== January 27, 2016 ===
'''Updates'''
'''Updates'''
* [TE] Hadoop package: Added -combine option for [http://www.vistrails.org/index.php/Cs9223_Mapreduce_Assignment Cs9223_Mapreduce_Assignment]
* New interpreter
** tricky to set up the package still, need documentation to understand how to assemble workflows from scratch
** Working prototype soon
* [RR] Logging improvements underway (branch 'improves-logging')
** Looping not completely done (and port combination mode: pairwise/cartesian...)
** Fixes some logging gotchas
** Need to integrate with VisTrails
** Better exception printing (pass them directly to critical|warning|log)
*** Compatibility with existing modules
** WARNING is now the default level for the console (only CRITICAL displays a popup)
*** Use log controller
* [RR] Alternative Mac .app ([https://docs.google.com/uc?id=0B3ucPz7GSthBZG5TS2pXUVdUWEk&export=download example dmg])
*** Want to cache at multiple levels (whole stream/single element of stream)
** Standard build of Python (currently as a Mac .framework)
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)
** Have to relink libraries with install_name_tool
** Seems to work
** Advantages: standard, full build of Python; pip will work
** Last problem: if Qt is installed, it gets loaded and conflicts with bundled Qt (I missed a dylib link)
* [DK] Work with USGS, refactoring version trees


'''Items to Discuss'''
'''Items to Discuss'''
* Colin's feedback on loops is that it has taken some time to wrap his head around what is going on
* User Survey
** what does Taverna do? automatically determines when a collection is being input and does iteration over the input and determines what output looks like
 
** look at what this would take, figure out what the best way to attack this ([[Automatic loops]])
* ZMQ failing on windows (vistrails-users)
* Colin also asked about use of persistence in another module


=== Jan 3, 2014 ===
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]
** Testsuite almost passes
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.
** Fixed issues
*** Signatures are now bytes (convert to string to show in widget)
*** Cannot compare different types, e.g.,  (None < 1)
**** PortSpec.depth and min/maxConn could be None. (Backport!)
**** cell.row could be None
*** Lots of unicode/bytes fixes to serializations
**** We have encryption implementations in the code! (keychain)
*** Lots of small fixes to package method differences
** Current issues
*** logging widget empty?
*** version tree signals messed up
*** provenance view legend icons black
*** Text do not fit in some widgets
*** VTK thumbnail comparison fails
**** VTK Cells have some faces in wrong direction?
****  Because running in VM?
** Installing on Ubuntu LTS/Latest Mint
*** Install latest matplotlib from pip
*** Install VTK 7.0.0rc2 from source
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)


=== January 20, 2016 ===
'''Updates'''
'''Updates'''


* [TE] VisTrails 2.1 released
* [DK] UUID branch, USGS work, upgrade recursion
** van Wijk on tree vis, and Buneman on XML Updates
'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Backporting features from Python 3 + PyQt5 + Qt5
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])
*** One unicode test failing
*** Problem saving to xml
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])
*** Updates print to functions 'from __future__ import print_function'
*** Updates exception calls to use 'as'
*** removes most usages of 'long' (mostly used unnecessarily)
**** Only needed for 32-bit support
*** Fixes relative imports (from __future__ import absolute_import)
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])
*** Updated all signals to new-style
* If we go 2 to 3, this could break packages
** means we should schedule any package format changes for the same time
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)
** putting package identifier as xml?
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)
** version the APIs?
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?
* file formats on 2 to 3:
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2
=== January 13, 2016 ===
'''Updates'''
'''Items to Discuss'''
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]
** Basic GUI and execution works
** Minimal work to support 2to3
** Hard to support both PyQt4 and PyQt5
=== January 6, 2016 ===
'''Updates'''
'''Items to Discuss'''
* Updating VisTrails dependencies
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.
**** Next version of VTK 6 is rumored to support Python 3
**** There is a development branch of VTK 6 supporting Python 3
*** Can travis build this?
**** We can skip tests for WebEngine and VTK if needed
** Python 3 [https://python3wos.appspot.com/ showstoppers]
*** No MySQL-python (sqlalchemy supports mysql through other drivers)
*** No suds (there are active forks but we could also drop support?)
* Options for python2/3 support:
** Interpreter could be python2/python3 based on packages
** six layer works but not the easiest to maintain (e.g. in reprozip)
* Drop support for win-64
* Update copyright headers to 2016


== Older meetings ==
== Older meetings ==
* [[Development/2015]]
* [[Development/2014]]
* [[Development/2013]]
* [[Development/2013]]
* [[Development/2012]]
* [[Development/2012]]
* [[Development/2011]]
* [[Development/2011]]
* [[Development/2010]]
* [[Development/2010]]

Latest revision as of 15:34, 30 October 2017

See also the Github wiki

2017

October 30, 2017

Updates

  • [DK] PyQt5
  • [RR] Running Python3 code

Items to Discuss

  • Installation:
    • Use conda? (macOS hack to have .app bundle load conda-installed vistrails)
    • Update conda recipe (pyqt4 restriction for now)
  • Interpreter
  • Release
  • Integration
    • hard to build same type of GUI in other apps that integrate core VisTrails (tend to use just the API)
    • independent pieces of the GUI could be reused if they were more independent.

September 6, 2017

Updates

  • [RR] Fixes for scikit-learn, tables, usagestats

Items to Discuss

  • Stripped-down version of VisTrails to showcase new features?
    • include meta-vistrail, bundles
    • hide mashups, extra widgets, latex mode, etc.
  • Interpreter

August 16, 2017

Updates

  • [DK] meta-vistrail
    • Working actions: normal action in vistrail, pruning version tree, tagging operations, and change
    • Interface

Items to Discuss

  • Notebook
  • Interpreter
  • Web visualization packages (e.g. bokeh)
  • Pipeline templates

August 2, 2017

Updates

  • [DK] meta-vistrail work
    • Extend Vistrail to MetaVistrail to support meta-actions
    • Vistrail becomes less tied to workflows so it can also supporting versioning vistrails

Items to Discuss

  • Interpreter
    • Using dask?
    • Streaming an issue
  • Notebooks
    • What is an output of a module? (need to expose a specific output port or the entire module)
    • Unifies some of the work from the script-workflow code

June 28, 2017

Updates

  • [DK] Testing UUID/SQLAlchemy branch
  • [DK] Working on meta-versions
    • would be nice to have pipeline and vistrail separate
    • [RR] Then do we only have meta-actions, any change to the pipeline is a meta-action that adds a version that does the change?
    • [DK] Yes?
  • [RR] Notebook interface (Vizier) making good progress, figuring out cell outputs

Items to Discuss

June 7, 2017

Updates

  • [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs

Items to Discuss

  • [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)
    • Can designate in module description an output port
    • Value has to be serializable
    • Should use the OutputModes work to handle multiple modes, configuration, selection, ...
  • [RR] Is VisTrails with SQL usable concurrently?
    • [DK] Probably not, demo from [TE] exists

May 10, 2017

Items to Discuss

  • Identifiers
  • Bundles & DB Code
  • Interpreter
  • Merge / Versioning Version Trees
  • Python3
  • Workings of List
  • Packages: Wrappers + new Versions (VTK7)
  • Script Conversion
  • Intermediate Outputs: reuse output module work?
    • would be useful for both workflows (run to a particular checkpoint)
    • and useful for notebook interfaces
  • SQL backend: does it allow for concurrent access?
    • Yes, real-time collaboration work for concurrent single vistrail access
    • Should be able to access different vistrails at once without issues

April 19, 2017

Updates

Items to Discuss

April 5, 2017

Updates

Items to Discuss

March 8, 2017

Updates

  • [RR] Added a looping indicator on modules #1217
    • This makes it easier to understand what's looping, no need to follow edges and count
  • [RR] Refactoring CLTools for integration in Docker package
  • [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)

Items to Discuss

  • [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers #1216
    • Can we move this up to the renderer modules?
    • Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)

February 24, 2017

Updates

  • [RR] Rich-text module documentation (#1210)
    • Went over packages and fixed format; maybe we should have a test to validate RST?
  • [RR] Single-instance shutdown with --remote-shutdown (#1204)
  • [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment

Items to Discuss

  • [RR] There are issues with list-depth
    • Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)
    • Issue with looping groups

January 25, 2017

Updates


Items to Discuss

  • Scaling VisTrails
  • sql-alchemy, uuid, bundle branches/merges

January 11, 2017

Updates

  • [RR] New VisTrails version in Debian [1]

Items to Discuss

  • [RR] Collaboration with school of medicine to port HiC-bench to VisTrails
    • Have homemade workflow system based on R scripts and symlinks
      • Each steps has multiple sets of parameters
      • Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step
      • Gets all results in directory structure step3-parameters/step2-parameters/step1-parameters/files
    • VisTrails improves provenance capture and makes editing pipelines easier
      • Need better support for aggregation (groupby operation might be good enough; pandas?)

2016

December 14, 2016

Updates

  • [RR] Single-instance code: over-engineered, but will work
    • (Have to avoid race conditions on startup, and OSes offer very different locking primitives)

Items to Discuss

  • Email from Matt Dirks: combobox should work, ask for more code?
  • Email from marin.nl:
    • looks like they would benefit from new interpreter
    • how to store their consolidated provenance when they exchange .vt files around?

November 30, 2016

Updates

Items to Discuss

  • [RR] Working on single instance code for MARIN (#1205)
    • Remote shutdown ready (#1204)
    • Still improving it
    • Want to return something more structured to the client (success/fail code, stdout, stderr)
    • Should be usable without Qt
  • [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason
  • [DK] Working on batch execution using console_mode
  • [DK] Settings aliased parameters when using core.api?

November 16, 2016

Updates

  • [RR] Issue #1200 from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)

Items to Discuss

  • Email from Pieter: executing pipelines with parameters from command-line using single-instance
    • Seems to work, main instance gets full command-line and alters configuration temporarily
    • Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful
    • Actions:
      • Can add switch to kill main instance
      • Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails
      • Add client/server code to core that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like gui.application.VistrailsApplicationSingleton#noninteractiveMode())?

November 2, 2016

Updates

  • New version 2.2-2 of Debian package, still wrong copyright, and broken bundle installation
    • Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch
  • Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)

Items to Discuss

October 5, 2016

Items to Discuss

  • working toward a 3.0 release; project board
    • unique ids
    • bundles
    • GUI fixes
    • wrapping updates
    • python3
    • interpreter?
  • command-line input parameters to provenance
    • [DK] we don't create a persisted workflow from those input params, i think
    • [DK] these are already defined as aliases so we treat them differently form a normal input param
    • [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases
    • Do the vistrails variables values get stored in provenance? (what if values change?)

September 14, 2016

Updates

Items to Discuss

August 31, 2016

Updates

  • [RR] Went over vgc.poly.edu/vistrails.org config:
    • Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
    • Fixed usersguide scripts (won't disappear again)
    • Moved some websites over to HTTPS
    • Websites
  • [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs (currently there) and users' guide under the same RTD "project"
  • [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that

Items to Discuss

  • [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
    • Get rid of more services? (svn, trac, ...)
  • [RR] Notebook execution works. How do we support more languages (than Python)?
    • Need: detection, metadata reader, code to input values
  • [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?

August 17, 2016

Updates

  • [TE] Fixed parameter order upgrade bug that breaks VTK examples.
  • [TE] Matplotlib and sklearn are exportable using module-to-script api
  • [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.

Items to Discuss

  • Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
  • Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
  • [TE] Wrap-up notes

August 10, 2016

Updates

Items to Discuss

  • [TE] Python Wrapper and scripting
    • All changes merged into python-wrapper branch
      • Tests pass
      • Still need to update sub-branches
    • Minor fixes
      • vtkPlot3DReader upgrade bug
      • Added tuple exporting
    • Most of vtk examples can be exported example
      • Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
        • Could be a bug in the old vistrail, but not sure
  • [DK] Subworkflows and uuids

August 3, 2016

Updates


Items to Discuss

  • [TE] Wrapper patches now stored as python code
    • Connection variables now lowercase
    • Can we use code as output variables?
      • E.g. "vtk_dataset_mapper.GetOutput(0)"
      • Could mutate output when output method is only supposed to be called once
      • Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)

July 27, 2016

Updates

  • [DK] DB bundle serialization implemented

Items to Discuss 

   def vtkInstance():
       self.vtkInstance = self

July 20, 2016

Updates

Items to Discuss

  • [TE] New patch method for python-wrapper example
    • A patch is a string template with self/input/output variables for patching a class method
    • Patches are stored in xml specification
    • Patches are assigned to methods in ClassSpec's
    • Used both for execution in vistrails and when exporting as script
      • Executed with "exec" when run in vistrails
      • Turns patches into natural calls in script
    • Wrapping VTK library no longer needed
      • Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)
    • Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)
    • Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)
    • TODO's
      • Add to export_as_script
      • Patch Modules that represent Python functions as well
  • [vistrails-users] Read data from SQL example?

July 13, 2016

Updates

  • [DK] bundles on sql-alchemy branch
    • Bundle, BundleMapping, and BundleSerializer
    • New BaseSerializers to deal with versions of directory, zip, db serializers
    • Can register the BundleSerializers with the base serializers
    • Translation between bundle versions
    • Bundle.add_object "just works" if an appropriate mapping was registered

Items to Discuss

  • [TE] Simplified running module using API for scripting example
    • API Module can be executed using compute method
      • Takes list of outputs to compute + inputs as keyword arguments
  • Status of general cell synchronization (vistrails-users)

July 6, 2016

Updates

  • [DK] Working on bundles

Items to Discuss

  • [TE] Using vistrails API in scripting example 1 (API with VTK) example 2 (DownloadFile only)
    • Call module in script using vistrails API
    • Can be used when no export method is defined.
  • [TE] Server news bug Critical?
    • [RR] No, unreleased
  • [DK] New bundle organization
    • BundleMapping contains individual mappings from raw objects to BundleObj
    • Must be separate in order to have in-memory, separate from serializer settings

June 29, 2016

Updates

Items to Discuss

  • [TE] Export VTK as python script
    • Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) example
    • Preludes (imports) still need improvements
    • Works with vtkInteractionHandler (clipping plane)
    • Resulting clipping plane is incorrect? (supposed to hide one side of the plane)
    • vtkRendererOutput Uses VTKRenderWindow as output
      • Should support different output modes?
      • Does not work with multiple outputs (Next cell is displayed when current is closed)
      • Uses wrapped vtkInstance (as vvtk)
        • Create new patch system with code that can be used both in vistrails and in a python script?

June 22, 2016

Updates

Items to Discuss

  • [TE] Fix for upgraded analogies (PR #1177)
    • Uses module location to find module remap (The best we can do right now?)
    • Also fixes remapped port names for connections to existing modules
    • May still fail on complex upgrades like adding a function on an upgraded module that has a different port name
  • [TE] Working on exporting wrapped libraries (VTK)

June 15, 2016

Updates

  • [TE] Added depth logic to python script exporter (full commit)
    • Fully implements connection merging and list looping

Items to Discuss

  • [TE] Making analogy work on upgraded workflows (#1175)
    • Analogy only works We need to use the non-upgraded action chain
    • But pipelines need to be valid (matcher uses module portspecs)
    • Tried to get upgrade remap from upgrade action, but this seems impossible
    • Use module location instead?
      • Location should be identical for one-to-one module upgrades

June 1, 2016

Updates

  • VisTrails 2.2.4 announced (finally)
    • 5 survey replies
    • 47 downloads last week

Items to Discuss

  • [TE] Export VTK as script (python-wrapper)
    • to_python_script
      • needs the pipeline (or at least the connections)
        • Should not generate code for any possible VTK method (exporter detects which ports is used in the source)
        • VTK needs to know connection order
        • Adding functions directly to the script results in nicer code
    • Add support for multiple input connections
    • Add support for list looping

May 25, 2016

Updates

  • Mailing-list fixed
  • Getting stats from the "server news" mechanism (via parser)

Items to Discuss

  • [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)
  • [TE] Union ports
    • Show union ports in pipeline (DEMO)

May 18, 2016

Items to Discuss

  • [TE] mailing list still down - Julio working on it
  • [TE] Improved union port
    • New schema 1.0.5 adds portitem.union string attribute
    • Works with pipeline view, parameter exploration view, and mashup view
    • Should work well for matplotlib
    • Use with VTK ports?
      • It uses [item_1,item_2] style ports usually with different tuple length.
  • [TE] Remove emacs dependency from db generate #1173

May 11, 2016

Items to Discuss

  • [TE] Union type #1172
    • GUI-based solution Branch union-port Demo?
      • Shows single port for all ports with same sort_key
      • Select specific type when adding/showing
      • Only one type should be visible on the module at any time
      • Making tuple parameters union types is probably too complicated
      • Should work with alternate specs like matplotlib (if sort_key is defined correctly)
  • [TE] Feature suggestion: Let connections connect to functions/parameters
    • Show functions and parameters as ports and toggle port visibility directly for functions and parameters
    • Use separate views for ports and functions (like before)
      • Allow rearranging of functions
    • Pros:
      • Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
        • We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.
        • Makes List module simpler
      • Connect to parameter in tuple
      • Functions that are visible on the module can connect to an actual function, and not just to the first one it finds.
      • Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.
    • Cons:
      • Interface more complicated and crowded
      • More difficult for users to grasp?
      • No immediate use case
      • Makes no union type for tuple parameters a bigger problem.
  • [TE] Publish user survey?
    • No response from vistrails release yet
    • Not announced on vistrails-users yet.

May 4, 2016

Updates

  • VisTrails 2.2.4 released

Items to Discuss

April 27, 2016

Updates

Items to Discuss

  • [TE] Wrapping Bokeh
    • New method type "operation"
      • Transforms Figure->Glyph->Output into Glyph->Figure->Output
        • Glyph sets function name and arguments as output
          • Not serializable, but should always be computed together with the subject of the operation.
        • Figure applies Glyph operation on itself
      • Useful for matplotlib?
        • matplotlib currently find actors by searching upstream?
        • But this is more difficult since matplotlib is not function-based.
    • New "union" port type
      • Turns "x(float), x(list)" into "x (float, list)" and uses first set value
        • Uses only first value in tuple
        • Bokeh have many properties that support (list, string, Float)
      • Does not work with depth=1 types (e.g. Float of depth 1)
        • Turn Depth 1 types with known type into lists?
  • [TE] New bugfix release
    • Finalize survey

April 20, 2016

Updates

  • Merged upgrade fixes
    • Now has infinite loop test
    • loop test now correctly loops 5 times
      • remap_module used wrong (current) package version when creating ModuleDescriptor
  • Fixed multi-step group upgrades (PR: #1168)

Items to Discuss

  • New bugfix release
    • Merge usagestats
    • Need to finalize survey

April 13, 2016

Updates

Items to Discuss

  • [TE] Chaining upgrades done (#1164)
    • Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?
  • [TE] Iterating handle_invalid_pipeline (#1165)
    • Not based on chain-upgrades-loop
    • Cherry-pick test to preserve source?
    • Need maxPipelineFixAttempts option?
      • The code is now smarter about recursion, so max attempts will "probably" never be reached.
    • Need debug messages?
      • Would need to be moved into handle_invalid_pipeline

April 6, 2016

Updates

  • [TE] Query View improvements
    • Done except for displaying "Version Info" and #1164
  • [RR] Working on finishing new interpreter (finally!)

Items to Discuss

  • [TE] Chaining upgrades (#1164)
    • Try latest upgrade first, then previous.
    • If all fail, select latest upgrade?

March 30, 2016

Updates

Items to Discuss

  • [TE] Query View improvements
    • Unified do_version_switch and get_upgrades as validate_version()
    • Problems with query view using its own controller
      • Fixed module info not being displayed for query results
      • Still no version info for query results
      • Still an issue with version query view not being updated

March 23, 2016

Updates

Items to Discuss

  • [RR] Usage reporting almost ready to go
    • Need to record whether specific features are used: paramexpore, mashup, query
      • [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation
    • How to record size of vistrail?
      • [DK] IdScope already walks over vistrail, use that
      • but total number of modules and versions doesn't give an idea of average size of individual pipelines

March 16, 2016

Updates

Items to Discuss

  • [TE] Upgrades in Query Mode
    • Now using code similar to do_version_switch
      • Uses handle_invalid_pipeline
      • Differences: Not using current_pipeline, flushes changes
      • New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)
      • Integrate to avoid repeated code?
      • Use this for query mode?
  • [TE] VisTrails 2.2.4 ready for release?
    • tej package updated. Others?
  • [JF] upgrades and the version tree
  • [JF] provenance in Jupyter
  • [DK] bundles: #1092

March 9, 2016

Updates

Items to Discuss

  • [TE] Fixing search mode
    • String query now matches all by default
      • This caused pipeline queries to never match
    • String query can search: User/Notes/Tagname/Time/Module name
      • Should we search module names by default? (Disabled by default because it is expensive)
      • Fixed notes search
    • '>' parameter queries in documentation but has been removed
    • Fixed module matches for upgraded pipelines (By not upgrading pipeline)
      • Upgrades still a problem
      • Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)
      • [JF] Allow search on past tags
  • [TE] Create new bugfix release
    • Finish querying
    • Path issue on windows
  • [DK] Look into sqlalchemy and new bundle branches

March 2, 2016

Updates

  • [TE] Fixed cache bug with port specs
  • [TE] VisTrails Next
    • Most packages now work
    • No qgis for Python 3

Items to Discuss

    • Always dereference symlinks? #1146

February 23, 2016

Updates

  • [TE] Fixed PythonSource 100% CPU bug
  • [TE] VisTrails Next
    • Updated NOAA Web Service examples
      • Now uses REST service, tabledata, and Google Maps
      • Web Service replaced with DownloadFile

Items to Discuss

  • [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.
    • New 2.2 release soon?
  • [RR] Anonymous usage reports
    • Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features #1154
  • [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?
  • noWorkflow - provenance for python scripts
    • Recent work on tracking dependencies
  • Also integrates ReproZip with Jupyter
    • mybinder
      • Use with ReproZip to automatically build Jupyter notebook interfaces
  • Daves comments on Jupyter integration
    • It is possible to identify cell order on the server side
    • Not trivial to track order of cells on client side
  • How does Galaxy workflows relate to VisTrails?
    • Packaging in Galaxy is hard
    • Galaxy for biomedical workflows, difficulty making it general
  • Action items
    • How difficult to track cell dependencies in Jupyter notebooks?
    • Use noWorkflow to track cell dependencies?
    • How difficult to modify mybinder to use ReproZip packages?

February 17, 2016

Updates

Items to Discuss

  • [TE] Fixed Group memory usage
    • Modules in groups are supposed to be freed when done executing
      • Prevented by logger that stored references to the modules (fixed, now using id(module))
      • Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)
  • [TE] VisTrails Next
    • Removed deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • PythonSources will be upgraded
      • Fixes for Python < 3
      • fixes for VTK < 6
    • TODOs
      • Remove RemoteQ
        • Port Hadoop to tej (Already have pbs)
      • Merge package wrapper
      • Finish SQLAlchemy branch
      • Finish new bundle branch
    • [JF] Will contact Jeff and Matthias about moving to Python 3
  • [JF] Query by Example is broken
  • [JF] VisTrails Data Cleaning Project
    • Use VisTrails to process and explore task-specific data while keeping provenance.
    • Remi's Idea
      • Use VisTrails as the backend in a Jupyter notebook
      • Overcome problems with executing cells in random order
        • Interesting research topic, but little interest has yet been noted
      • Interesting to study how such a notebook would be used (Is there notebook usage data available?)
      • Impose restrictions to notebook cell boundaries
      • It will only work for functional operations
      • Can we fix, or detect, non-functional (mutating) operations?
      • Analyse code and data flow and see if it mutates data in unexpected ways?
      • Mutable operations can be created in many ways in Python
  • Next meeting Tuesday 11:30

February 10, 2016

Updates

Items to Discuss

  • VisTrails Next
    • Test suite passes
      • Python 3 less deterministic, module upgrade order random.
    • [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...
    • All of our packages should run fine :)
    • VTK thumbnail comparison works
      • Rendering bug on some systems
        • QVTKCell now inherits QGLWidget
      • Old thumbnails had wrong height due to spreadsheetwindow being too small
        • Fixed by regenerating thumbnails
      • Silenced deprecation messages when generating modules
      • Many new classes in VTK 7
    • Remove deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • Will break existing PythonSources
        • Try to upgrade?
      • Old-style packages
        • Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
          • Read info.py without importing package?
      • Fixes for Python < 3
      • Fixes for Qt < 5
      • fixes for VTK < 7
      • Remove self port?

February 3, 2016

Updates

Items to Discuss

  • VisTrails Next
    • All tests but thumbnail comparison passes
    • Still issues with gui and more packages needs to be tested
    • PythonSource and MplSource upgraded with lib2to3
    • Keep contrib packages? Update?
    • VTK 7 is released
      • Supports Python 3
      • Will hopefully get into conda and others soon
      • Deprecates a bunch of modules
  • Travis builder for VisTrails Next
    • Runs!
    • Uses conda
      • Many levels of requirements
        • apt packages
        • conda packages
        • python packages in requirements.txt
        • python packages in setup.py
      • Can probably drop conda. Default builder does not have Qt5, but beta builder do!
      • But useful if we want to use conda in our own build process.
    • Lots of errors I do not get locally?
  • User Survey
    • Have you used any alternatives to vistrails?
      • Relevant? More options?
    • How often do you use any of these particular features in VisTrails?
      • Add/Remove alternatives?
    • How important would you rank the following potential improvements to VisTrails?
      • Add alternatives?

January 27, 2016

Updates

  • New interpreter
    • Working prototype soon
    • Looping not completely done (and port combination mode: pairwise/cartesian...)
    • Need to integrate with VisTrails
      • Compatibility with existing modules
      • Use log controller
      • Want to cache at multiple levels (whole stream/single element of stream)
      • Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)

Items to Discuss

  • User Survey
  • ZMQ failing on windows (vistrails-users)
  • [TE] Python 3 branch
    • Testsuite almost passes
      • Failing are unicode filenames, some users guide examples, and thumbnail comparisons.
    • Fixed issues
      • Signatures are now bytes (convert to string to show in widget)
      • Cannot compare different types, e.g., (None < 1)
        • PortSpec.depth and min/maxConn could be None. (Backport!)
        • cell.row could be None
      • Lots of unicode/bytes fixes to serializations
        • We have encryption implementations in the code! (keychain)
      • Lots of small fixes to package method differences
    • Current issues
      • logging widget empty?
      • version tree signals messed up
      • provenance view legend icons black
      • Text do not fit in some widgets
      • VTK thumbnail comparison fails
        • VTK Cells have some faces in wrong direction?
        • Because running in VM?
    • Installing on Ubuntu LTS/Latest Mint
      • Install latest matplotlib from pip
      • Install VTK 7.0.0rc2 from source
      • No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)

January 20, 2016

Updates

Items to Discuss

  • [TE] Backporting features from Python 3 + PyQt5 + Qt5
    • Merged unicode with master (unicode)
      • One unicode test failing
      • Problem saving to xml
    • Used python-future to run futurize stage 1 to update legacy (pre 2.6) code (unicode-futurized)
      • Updates print to functions 'from __future__ import print_function'
      • Updates exception calls to use 'as'
      • removes most usages of 'long' (mostly used unnecessarily)
        • Only needed for 32-bit support
      • Fixes relative imports (from __future__ import absolute_import)
    • Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 (unicode-futurized-qt5)
      • Updated all signals to new-style
  • If we go 2 to 3, this could break packages
    • means we should schedule any package format changes for the same time
    • e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* #1002)
    • any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)
    • putting package identifier as xml?
      • Python not so bad, plus some stuff in there is Python code (checking dependencies)
    • version the APIs?
      • RR planning on versioning vistrails_module.Module, no point versioning the whole package?
  • file formats on 2 to 3:
    • no real translation/upgrade, but non-ascii values that work on 3 won't load in 2

January 13, 2016

Updates

Items to Discuss

January 6, 2016

Updates Items to Discuss

  • Updating VisTrails dependencies
    • Successfully built Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7
      • VTK 7 is the first supporting Python 3 but is only in release candidate phase.
        • Next version of VTK 6 is rumored to support Python 3
        • There is a development branch of VTK 6 supporting Python 3
      • Can travis build this?
        • We can skip tests for WebEngine and VTK if needed
    • Python 3 showstoppers
      • No MySQL-python (sqlalchemy supports mysql through other drivers)
      • No suds (there are active forks but we could also drop support?)
  • Options for python2/3 support:
    • Interpreter could be python2/python3 based on packages
    • six layer works but not the easiest to maintain (e.g. in reprozip)
  • Drop support for win-64
  • Update copyright headers to 2016

Older meetings

Retrieved from "https://www.vistrails.org//index.php?title=Development&oldid=14736"