Difference between revisions of "Development"

From VistrailsWiki
Jump to navigation Jump to search
 
(214 intermediate revisions by 3 users not shown)
Line 5: Line 5:
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]


== 2015 ==
== 2017 ==
 
=== October 30, 2017 ===
 
'''Updates'''
 
* [DK] PyQt5
* [RR] Running Python3 code
 
'''Items to Discuss'''
 
* Installation:
** Use conda? (macOS hack to have .app bundle load conda-installed vistrails)
** Update conda recipe (pyqt4 restriction for now)
* Interpreter
* Release
* Integration
** hard to build same type of GUI in other apps that integrate core VisTrails (tend to use just the API)
** independent pieces of the GUI could be reused if they were more independent.
 
=== September 6, 2017 ===
 
'''Updates'''
 
* [RR] Fixes for scikit-learn, tables, usagestats
 
'''Items to Discuss'''
 
* Stripped-down version of VisTrails to showcase new features?
** include meta-vistrail, bundles
** hide mashups, extra widgets, latex mode, etc.
* Interpreter
 
=== August 16, 2017 ===
 
'''Updates'''
* [DK] meta-vistrail
** Working actions: normal action in vistrail, pruning version tree, tagging operations, and change
** Interface
 
'''Items to Discuss'''
* Notebook
* Interpreter
* Web visualization packages (e.g. bokeh)
** [https://github.com/VisTrails/VisTrails/pull/969 PR 969] has more details
* Pipeline templates
 
=== August 2, 2017 ===
 
'''Updates'''
* [DK] meta-vistrail work
** Extend Vistrail to MetaVistrail to support meta-actions
** Vistrail becomes less tied to workflows so it can also supporting versioning vistrails
 
'''Items to Discuss'''
 
* Interpreter
** Using dask?
** Streaming an issue
* Notebooks
** What is an output of a module? (need to expose a specific output port or the entire module)
** Unifies some of the work from the script-workflow code
 
=== June 28, 2017 ===
 
'''Updates'''
* [DK] Testing UUID/SQLAlchemy branch
* [DK] Working on meta-versions
** would be nice to have pipeline and vistrail separate
** [RR] Then do we only have meta-actions, any change to the pipeline is a meta-action that adds a version that does the change?
** [DK] Yes?
* [RR] Notebook interface (Vizier) making good progress, figuring out cell outputs
 
'''Items to Discuss'''
 
=== June 7, 2017 ===
 
'''Updates'''
* [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs
 
'''Items to Discuss'''
* [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)
** Can designate in module description an output port
** Value has to be serializable
** Should use the OutputModes work to handle multiple modes, configuration, selection, ...
* [RR] Is VisTrails with SQL usable concurrently?
** [DK] Probably not, demo from [TE] exists
 
=== May 10, 2017 ===
 
'''Items to Discuss'''
 
* Identifiers
* Bundles & DB Code
* Interpreter
* Merge / Versioning Version Trees
* Python3
* Workings of List
* Packages: Wrappers + new Versions (VTK7)
* Script Conversion
* Intermediate Outputs: reuse output module work?
** would be useful for both workflows (run to a particular checkpoint)
** and useful for notebook interfaces
* SQL backend: does it allow for concurrent access?
** Yes, real-time collaboration work for concurrent single vistrail access
** Should be able to access different vistrails at once without issues
 
=== April 19, 2017 ===
 
'''Updates'''
 
'''Items to Discuss'''
 
=== April 5, 2017 ===
 
'''Updates'''
 
'''Items to Discuss'''
 
=== March 8, 2017 ===
 
'''Updates'''
* [RR] Added a looping indicator on modules [https://github.com/VisTrails/VisTrails/pull/1217 #1217]
** This makes it easier to understand what's looping, no need to follow edges and count
* [RR] Refactoring CLTools for integration in Docker package
* [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)
 
'''Items to Discuss'''
* [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers [https://github.com/VisTrails/VisTrails/issues/1216 #1216]
** Can we move this up to the renderer modules?
** Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)
 
=== February 24, 2017 ===
 
'''Updates'''
* [RR] Rich-text module documentation ([https://github.com/VisTrails/VisTrails/pull/1210 #1210])
** Went over packages and fixed format; maybe we should have a test to validate RST?
* [RR] Single-instance shutdown with <tt>--remote-shutdown</tt> ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])
* [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment
 
'''Items to Discuss'''
* [RR] There are issues with list-depth
** Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)
** Issue with looping groups
 
=== January 25, 2017 ===
 
'''Updates'''
 
 
'''Items to Discuss'''
 
* Scaling VisTrails
* sql-alchemy, uuid, bundle branches/merges
 
=== January 11, 2017 ===
 
'''Updates'''
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]
 
'''Items to Discuss'''
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails
** Have homemade workflow system based on R scripts and symlinks
*** Each steps has multiple sets of parameters
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code>
** VisTrails improves provenance capture and makes editing pipelines easier
*** Need better support for aggregation (groupby operation might be good enough; pandas?)
 
== 2016 ==
=== December 14, 2016 ===


=== August 26, 2015 ===
'''Updates'''
'''Updates'''
* [TE] Added Job support to Parameter Explorations (Requested by Colin), and Mashups
* [RR] Single-instance code: over-engineered, but will work
** specify job ids, need to specify different ids for parameter explorations since they have the same version id
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)
** how to deal with parameters passed in on command line
 
** [General] provenance: should be creating a new version when we execute workflow with changed parameters
'''Items to Discuss'''
'''Items to Discuss'''
* Email from Matt Dirks: combobox should work, ask for more code?
* Email from marin.nl:
** looks like they would benefit from new interpreter
** how to store their consolidated provenance when they exchange .vt files around?
=== November 30, 2016 ===


=== August 19, 2015 ===
'''Updates'''
'''Updates'''
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]
* [TE] Working on Job Monitor documentation


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] JobMixin and JobMonitor: stable now?
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])
** Definitely needs more tests
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])
* [TE] Stop testing VisTrails 2.0?
** Still improving it
** Want to return something more structured to the client (success/fail code, stdout, stderr)
** Should be usable without Qt
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason
* [DK] Working on batch execution using console_mode
* [DK] Settings aliased parameters when using core.api?
 
=== November 16, 2016 ===


=== August 12, 2015 ===
'''Updates'''
'''Updates'''
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)
'''Items to Discuss'''
* Email from Pieter: executing pipelines with parameters from command-line using single-instance
** Seems to work, main instance gets full command-line and alters configuration temporarily
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful
** Actions:
*** Can add switch to kill main instance
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?
=== November 2, 2016 ===
'''Updates'''
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)
'''Items to Discuss'''
=== October 5, 2016 ===


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]
* [RR] JobMixin and JobMonitor: stable now?
** Definitely needs more tests
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]
* [RR] Build broken on Travis; because of IPython 4 released today? ([https://travis-ci.org/VisTrails/VisTrails/builds/75279795 build 992]; [https://github.com/VisTrails/VisTrails/issues/1123 #1123])


=== August 5, 2015 ===
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]
** unique ids
** bundles
** GUI fixes
** wrapping updates
** python3
** interpreter?
 
* command-line input parameters to provenance
** [DK] we don't create a persisted workflow from those input params, i think
** [DK] these are already defined as aliases so we treat them differently form a normal input param
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases
** Do the vistrails variables values get stored in provenance? (what if values change?)
 
=== September 14, 2016 ===
 
'''Updates'''
'''Updates'''
* [TE] Fixes to Jobs
** Could not delete jobs
** Could not run job in group
** Job not reset when calling ModuleSuspended
** Added deleting job from context menu
* [RR] Writing documentation for everything [https://github.com/VisTrails/VisTrails/pull/1105 #1105]


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)
** Did we know/contribute to this?
*** [DK] Yes. We wrote the text for the VisTrails chapter
** Drafted response
*** [DK] Looks good, will forward.
=== August 31, 2016 ===


=== July 29, 2015 ===
'''Updates'''
'''Updates'''
* RR still looking into new interpreter thing
* [RR] Went over vgc.poly.edu/vistrails.org config:
** Goal is to take out scheduling logic from Module so it can be split in multiple processes, and so that smarter strategies can be added in time
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
** This means some work on packages
** Fixed usersguide scripts (won't disappear again)
** Spreadsheet can live in kernel process? Still some UI stuff that will take work (changing configuration, persistent archive's viewer, ...)
** Moved some websites over to HTTPS
* [TE] Problems running examples [https://github.com/VisTrails/VisTrails/pull/1111 #1111]
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]
** Testing of more examples requires additional packages on the test machines.
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"
** Fixed faulty line-ending in PythonSource:s failing on Python 2.6.
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that
** Test suite now testing SUDSWebServices (If web service is down, test suite will fail)
** preferences.py test failed reloading 'dialogs' package, switched to using 'URL'.


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
** Get rid of more services? (svn, trac, ...)
* [RR] Notebook execution works. How do we support more languages (than Python)?
** Need: detection, metadata reader, code to input values
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?
=== August 17, 2016 ===


=== July 22, 2015 ===
'''Updates'''
'''Updates'''
* [TE] mailing lists back online
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.
* [TE] Working on [https://github.com/VisTrails/VisTrails/issues/1107 #1107]
* [TE] Matplotlib and sklearn are exportable using module-to-script api
* [RR] Working on ReproZip package
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] MongoDB package [https://github.com/VisTrails/VisTrails/pull/1106 #1106]
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
* [RR] Example for tej docs? [https://github.com/VisTrails/VisTrails/pull/1105 #1105]
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
* [TE] Wrap-up notes
 
=== August 10, 2016 ===


=== July 15, 2015 ===
'''Updates'''
'''Updates'''
* 2.2.2 released


'''Items to Discuss'''
'''Items to Discuss'''
* RR is considering executing everything in an IPython kernel (i.e. separate Python interpreter, like the one spawned for a notebook)
* [TE] Python Wrapper and scripting
** It's a separate process that we can restart without restarting the app/gui
** All changes merged into python-wrapper branch
** We can isolate the execution environment (e.g. for the server)
*** Tests pass
** We can run the whole thing remotely (if your data is elsewhere, just run VisTrails locally and the pipeline elsewhere)
*** Still need to update sub-branches
** Pipeline execution no longer makes the interface hang, it just makes the kernel hang (but that's fine)
** Minor fixes
** We can use notebooks as modules (probably way nicer than the PythonSource module)
*** vtkPlot3DReader upgrade bug
** We can run multiple kernels so long as the ports carry things that are serializable
*** Added tuple exporting
*** meaning we can put the multithreaded-interpreter without all the hacky parts it has now
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]
*** we can run IPython kernels in all the languages IPython supports, [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages currently 46]
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
**** Could be a bug in the old vistrail, but not sure
* [DK] Subworkflows and uuids
 
=== August 3, 2016 ===


=== July 8, 2015 ===
'''Updates'''
'''Updates'''


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Buildbot github hook not working after IP address change
* [TE] No reference to VistrailsApplication ([https://github.com/VisTrails/VisTrails/issues/1103 #1103])
** [RR] Added PR [https://github.com/VisTrails/VisTrails/pull/1104 #1104]
* [TE]  Reopening VT file after saving with bundled subworkflow won't offer subworkflow upgrade ([https://github.com/VisTrails/VisTrails/issues/1102 #1102])
** allow manual delete to fix right now
** fix this on top of the use-uuid branch
* [TE]  Release VisTrails 2.2.1? ([https://github.com/VisTrails/VisTrails/blob/v2.2/CHANGELOG CHANGELOG])
** [https://github.com/VisTrails/VisTrails/pull/1093 output-modules-use-global-dir]
** [https://github.com/VisTrails/VisTrails/milestones/version%202.2 Current issues for 2.2]


=== July 1, 2015 ===
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]
** Connection variables now lowercase
** Can we use code as output variables?
*** E.g. "vtk_dataset_mapper.GetOutput(0)"
*** Could mutate output when output method is only supposed to be called once
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)
 
=== July 27, 2016 ===
 
'''Updates'''
'''Updates'''
* [TE] PROV fixed
 
* [TE] Working on subworkflow issues
* [DK] DB bundle serialization implemented
* [RR] Considering reworking the controller (log vs exception problem, retained upgrades causing interferences) and also the interpreter (IPython?)


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]
** locale?
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches]
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]
** Still need to do something nicer for the patch formatting
*** patch keys currently contain non-python-variable characters but this can be fixed
*** Need to replace string template with python variables that is replacable using RedBaron
**** Patches could then look like:
    def vtkInstance():
        self.vtkInstance = self
=== July 20, 2016 ===


=== June 24, 2015 ===
'''Updates'''
'''Updates'''


'''Items to Discuss'''
'''Items to Discuss'''
* UV-CDAT
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]
* [TE] Can a cyclic workflow be valid? ([https://github.com/VisTrails/VisTrails/issues/1097 #1097])
** A patch is a string template with self/input/output variables for patching a class method
** focus on disabling the ability to create cyclic pipelines because more things break than just this with a cyclic pipeline
** Patches are stored in xml specification
* [RR] Relative paths ([https://github.com/VisTrails/VisTrails/pull/1057 #1057])
** Patches are assigned to methods in ClassSpec's
** This interacts with the new bundle; how do handle packing files inside the VT bundle?
** Used both for execution in vistrails and when exporting as script
*** Executed with "exec" when run in vistrails
*** Turns patches into natural calls in script
** Wrapping VTK library no longer needed
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)
** TODO's
*** Add to export_as_script
*** Patch Modules that represent Python functions as well
 
* [vistrails-users] Read data from SQL example?
 
=== July 13, 2016 ===


=== June 17, 2015 ===
'''Updates'''
'''Updates'''
* [DK] bundles on sql-alchemy branch
** Bundle, BundleMapping, and BundleSerializer
** New BaseSerializers to deal with versions of directory, zip, db serializers
** Can register the BundleSerializers with the base serializers
** Translation between bundle versions
** Bundle.add_object "just works" if an appropriate mapping was registered


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] current_version and reusing existing upgrades are broken ([https://github.com/VisTrails/VisTrails/issues/1095 ticket #1095])
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]
** Could be that export to PROV is using an unflushed upgrade pipeline?
** API Module can be executed using '''compute''' method
** current_version would then be correct.
*** Takes list of outputs to compute + inputs as keyword arguments
** It may work to flush the actions before exporting?
* Status of general cell synchronization (vistrails-users)
 
=== July 6, 2016 ===


=== June 10, 2015 ===
'''Updates'''
'''Updates'''
* DAT: fixed VTK issue on Linux and Mac
 
* Still crashes on Windows. Need help! Reminder: this works in the VisTrails spreadsheet (QCellPresenter), although no widget get changed there during a drag
* [DK] Working on bundles
** Is it a VTK bug?
** Is it simply impossible to change widgets during the drag, should we do it a different way?
** Did I miss something that is done in VisTrails but somehow not in DAT?
** Low prio, UV-CDAT doesn't run on Windows anyway


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 Updated pull request])
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]
* Single instance code and batch mode
** Call module in script using vistrails API
** Can be used when no export method is defined.
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?
**[RR] No, unreleased
* [DK] New bundle organization
** BundleMapping contains individual mappings from raw objects to BundleObj
** Must be separate in order to have in-memory, separate from serializer settings
 
=== June 29, 2016 ===


=== June 3, 2015 ===
'''Updates'''
'''Updates'''


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 pr])
* [TE] Export VTK as python script
** Requires fixes to command line parameters, Output modules, and batch mode
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]
** How to test this
** Preludes (imports) still need improvements
** Updated missing/outdated flags
** Works with vtkInteractionHandler (clipping plane)
** Fixed view issues when generating graphs
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)
* [TE] batch mode
** vtkRendererOutput Uses VTKRenderWindow as output
** SpreadsheetOutput not enabled in batch mode. Should we check is_running_gui instead?
*** Should support different output modes?
** Other instance setting flags from caller
*** Does not work with multiple outputs (Next cell is displayed when current is closed)
** Is graphsAsPdf replacing spreadsheetDumpPdf?
*** Uses wrapped vtkInstance (as vvtk)
** graphsAsPdf true by Default?
**** Create new patch system with code that can be used both in vistrails and in a python script?
** Batch mode executing by default (Not needed when generating graphs)
 
** Re-added workflowInfo as withWorkflowInfo for writing graph and xml workflow
=== June 22, 2016 ===
** [DK] batch mode should be outputting to files or stdout, shouldn't always trigger SpreadsheetMode
** execute flag, maybe make execution the default and allow a "--no-execute" if you only want to capture graphs, for example


=== May 27, 2015 ===
'''Updates'''
'''Updates'''


'''Items to Discuss'''
'''Items to Discuss'''
* [vistrails-users] API question
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])
**  http://lists.vistrails.org/pipermail/vistrails-users/2015-May/000357.html
** Uses module location to find module remap (The best we can do right now?)
* Status of DAT
** Also fixes remapped port names for connections to existing modules
** Can add VCS plot to DAT but need to work on configuration windows
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name
** Documentation for graphics templates for UV-CDAT/vcs?
 
* Qt support?
* [TE] Working on exporting wrapped libraries (VTK)
 
=== June 15, 2016 ===


=== May 20, 2015 ===
'''Updates'''
'''Updates'''
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])
** Fully implements connection merging and list looping


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Reviving the DAT, integrating scripting & porting UV-CDAT!
** On [https://github.com/VisTrails/DAT GitHub] ([https://github.com/VisTrails/DAT/issues issues])
** Merging 2 years of development taking longer than expected, but getting there. The plan is to get the patches in VisTrails and never fork again, we never want to get in UV-CDAT's situation (and don't need to).
** VTK cell works fine on Linux but there was flickering on Mac & Windows before; still issues on Mac (Windows status unknown)
** Can get a VCS plot soon (but will need VTK cell fix)
** Integrate in UV-CDAT's build system (so we have cdms, VCS, ...) -> RR can do this, low priority
** How do we integrate scripting?
*** We want to be able to seamlessly make changes to a plot by changing Python code
*** Define new plots by entering Python code without writing modules/packages?
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look
* [RR] Subworkflow issues
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen


=== May 13, 2015 ===
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])
** Analogy only works We need to use the non-upgraded action chain
** But pipelines need to be valid (matcher uses module portspecs)
** Tried to get upgrade remap from upgrade action, but this seems impossible
** Use module location instead?
*** Location should be identical for one-to-one module upgrades
 
=== June 1, 2016 ===
 
'''Updates'''
'''Updates'''
* [RR] Test skipping whitelist ([https://github.com/VisTrails/VisTrails/pull/1069 #1069]) -- low priority
* VisTrails 2.2.4 announced (finally)
* [RR] Custom matplotlib modules can't be compatible with both 2.1 and 2.2 ([https://github.com/VisTrails/VisTrails/issues/1067 #1067]); should be fixed for ALPS ([https://github.com/VisTrails/VisTrails/issues/1070 #1070])
** 5 survey replies
** 47 downloads last week


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look
* [TE] Export VTK as script (python-wrapper)
* [RR] Subworkflow issues
** to_python_script
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)
*** needs the pipeline (or at least the connections)
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)
**** Should not  generate code for any possible VTK method (exporter detects which ports is used in the source)
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue
**** VTK needs to know connection order
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades
**** Adding functions directly to the script results in nicer code
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen
** Add support for multiple input connections
** Add support for list looping
 
=== May 25, 2016 ===


=== May 7, 2015 ===
'''Updates'''
'''Updates'''
* [RR] Export/import workflow to Python working!
* Mailing-list fixed
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])


'''Items to Discuss'''
'''Items to Discuss'''
* [Claudio] UV-CDAT
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)
** The UV-CDAT project is the biggest user base of VisTrails
* [TE] Union ports
** VisTrails package management provides a lot of friction towards people plugging in their code
** Show union ports in pipeline (DEMO)
** Need to make it easy to integrate your random Python scripts in the system without having to deal with all the boilerplate, at least in the first step
 
** [RR] argues that modules are still good; UV-CDAT shouldn't move towards a purely script-based backend
=== May 18, 2016 ===
** [RR] export/import with Python could reduce a lot of that friction by allowing 1) to edit workflow as Python 2) to open up boxes automatically if needed code doesn't match actual modules
 
** ...
'''Items to Discuss'''
* [TE] mailing list still down - Julio working on it
* [TE] Improved union port
** New schema 1.0.5 adds portitem.union string attribute
** Works with pipeline view, parameter exploration view, and mashup view
** Should work well for matplotlib
** Use with VTK ports?
*** It uses [item_1,item_2] style ports usually with different tuple length.
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]
 
=== May 11, 2016 ===
 
'''Items to Discuss'''
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?
*** Shows single port for all ports with same sort_key
*** Select specific type when adding/showing
*** Only one type should be visible on the module at any time
*** Making tuple parameters union types is probably too complicated
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly)
 
* [TE] Feature suggestion: Let connections connect to functions/parameters
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters
** Use separate views for ports and functions (like before)
*** Allow rearranging of functions
** Pros:
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.
**** Makes List module simpler
*** Connect to parameter in tuple
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds.
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.
** Cons:
*** Interface more complicated and crowded
*** More difficult for users to grasp?
*** No immediate use case
*** Makes no union type for tuple parameters a bigger problem.
 
* [TE] Publish user survey?
** No response from vistrails release yet
** Not announced on vistrails-users yet.


=== April 29, 2015 ===
=== May 4, 2016 ===
'''Updates'''
'''Updates'''
* VisTrails 2.2.4 released


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] <strike>Abstractions</strike> subworkflows status ([https://github.com/VisTrails/VisTrails/labels/subworkflows tickets])
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]
* [RR] matplotlib compatibility (2.1 & 2.2), [https://github.com/VisTrails/VisTrails/issues/1067 #1067]
** RR to try and fix ALPS matplotlib modules


=== April 22, 2015 ===
=== April 27, 2016 ===
'''Updates'''
'''Updates'''
* 2.2.0 has been released!
* Windows issue (via email from Ryan)
** issue with manifest file (may be a new file in VTK6?)
** Tommy has regenerated new Windows builds
* Binaries, pypi, and conda released
* [RR] Export as script
* Python sources using VTK need to switch to SetInputData (users should be aware of this)


=== April 15, 2015 ===
'''Items to Discuss'''
* [TE] Wrapping Bokeh
** New method type "operation"
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output
**** Glyph sets function name and arguments as output
***** Not serializable, but should always be computed together with the subject of the operation.
**** Figure applies Glyph operation on itself
*** Useful for matplotlib?
**** matplotlib currently find actors by searching upstream?
**** But this is more difficult since matplotlib is not function-based.
** New "union" port type
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value
**** Uses only first value in tuple
**** Bokeh have many properties that support (list, string, Float)
*** Does not work with depth=1 types (e.g. Float of depth 1)
**** Turn Depth 1 types with known type into lists?
 
* [TE] New bugfix release
** Finalize survey
 
=== April 20, 2016 ===
'''Updates'''
'''Updates'''
* Ready for 2.2.0 (apart from binary/deps issues)
* Merged upgrade fixes
** Missing some libs (scikit-learn, tej, tdparser, SQLAlchemy+connectors
** Now has infinite loop test
** Windows: runvistrails.py is no longer used, so the PATH is wrong
** loop test now correctly loops 5 times
** Windows: pip is broken, but it probably wouldn't work anyway because of permission issues (disable this?)
*** remap_module used wrong (current) package version when creating ModuleDescriptor
* Queries, upgrades and getPipeline() usage ([https://github.com/VisTrails/VisTrails/issues/1054 #1054])
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])
** Getting a pipeline with getPipeline() is not safe: it might return an invalid pipeline
** This is used in many places throughout the code, like queries
** Upgrading would require going through the controller, but that creates new actions
* [TC] Avoid copying a module's output if it's used as input by exactly one downstream module ([https://github.com/VisTrails/VisTrails/issues/1060 #1060]) (useful for big numpy arrays you can update in-place)


=== April 8, 2015 ===
'''Items to Discuss'''
* New bugfix release
** Merge usagestats
** Need to finalize survey
 
=== April 13, 2016 ===
'''Updates'''
'''Updates'''
* [DK] Merged RR's changes for output modules (1012 and 1013)
** RR will merge remaining changes, then create v2.2 branch!


'''Items to Discuss'''
'''Items to Discuss'''
* BNL need numpy array to VTK image
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])
** Looks like VTK has some helpers for this
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?
** We will help if issues arise
** Will contribute back to VisTrails package
* Upgrade issue: [https://github.com/VisTrails/VisTrails/issues/1017 #1017]
** Automatic upgrades should happen between versions of provided upgrades
** Our existing upgrades work around this so it doesn't need to be 2.2.0
* Corner-case VTK modules
** No longer need VTKCell input port, so don't interfere with registry and API anymore
** Still work weirdly, people probably shouldn't use them
** But we have lots of clunky modules since we wrap the whole of VTK; some people might rely on this and know how to use them, let's keep them anyway
** Ready for 2.2.0


=== April 1, 2015 ===
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]
** Cherry-pick test to preserve source?
** Need maxPipelineFixAttempts option?
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?
*** Would need to be moved into handle_invalid_pipeline
 
=== April 6, 2016 ===
'''Updates'''
'''Updates'''
* [RR] UV-CDAT: bugfixing for 2.2, long-term plans: implement scripting import/export in VisTrails, port to UV-CDAT
* [TE] Query View improvements
* Possibly, try to move to regular VisTrails to use new features
** Done except for displaying "Version Info" and #1164
* [RR] Working on finishing new interpreter (finally!)


'''Items to Discuss'''
'''Items to Discuss'''
* 2.2 release: nothing much is pending anymore, release next week?
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])
** [TE] vtkExporter classes [https://github.com/VisTrails/VisTrails/issues/1032 #1032]
** Try latest upgrade first, then previous.
** [RR] Hiding upgrades in version tree might make it (or might be 2.2.1 so we can test it out) [https://github.com/VisTrails/VisTrails/issues/949 #949]
** If all fail, select latest upgrade?
** Output modules changes to go in
* add note to documentation about order of parameters in VTK
* add issue about exporter upgrades if not already there


=== March 25, 2015 ===
=== March 30, 2016 ===
'''Updates'''
'''Updates'''


'''Items to Discuss'''
'''Items to Discuss'''
* 2.2 release
** Checklist on Github: https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist
** Review output modules
** RR has a couple more issues to fix
** Ready to go -- sign app package for OSX? [https://github.com/VisTrails/VisTrails/issues/984 #984] We need access to the Apple Membership team


=== March 18, 2015 ===
* [TE] Query View improvements
** Unified do_version_switch and get_upgrades as validate_version()
** Problems with query view using its own controller
*** Fixed module info not being displayed for query results
*** Still no version info for query results
*** Still an issue with version query view not being updated
 
=== March 23, 2016 ===
'''Updates'''
'''Updates'''
* [TE] VTK6 works


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Release v2.1.5 with backported tabledata?
* [RR] Usage reporting almost ready to go
** MTA example needs updated tabledata (for JoinTables)
** Need to record whether specific features are used: paramexpore, mashup, query
** Google Maps package still not available
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation
* [RR] Work torward v2.2.0?
** How to record size of vistrail?
** changes:
*** [DK] IdScope already walks over vistrail, use that
*** new persistence
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines
*** API changes
*** output module changes (upgrades?), maintain cells but try to upgrade
*** not wrapping stuff
*** VTK6? yes
*** JobSubmission stuff?
*** relabeling for upgrades [https://github.com/VisTrails/VisTrails/issues/949 #949]
** makes sense, needs the tree view code to be updated, check selection
** See [https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist 2.2 checklist]
* Discussion of [https://github.com/VisTrails/VisTrails/issues/1016 #1016]
** plumbing between outputs and output modes, how to define a mode that works for many outputs without writing for each output?


=== March 11, 2015 ===
=== March 16, 2016 ===
'''Updates'''
'''Updates'''
* [RR] persistent_archive done; merge? ([https://github.com/VisTrails/VisTrails/pull/755 #755])
** note about the focus events for widgets
** TE be aware of file_archive for future binaries that include persistent_archive
* [TE] [https://github.com/VisTrails/VisTrails/pull/998 New VTK package] finished


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails? (see last week; decision needed)
* [TE] Upgrades in Query Mode
** RR email to JF about this
** Now using code similar to do_version_switch
** yes; email sent to Github
*** Uses handle_invalid_pipeline
* [RR] Switching order of output ports ([https://github.com/VisTrails/VisTrails/pull/1006 #1006])
*** Differences: Not using current_pipeline, flushes changes
** added port specs are sorted at a separate spot (Module.*_port_specs properties) than those in the registry (which are sorted in the registry), but those two lists are just combined without respect to sort keys
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)
** need to determine whether the two lists should be merged or remain distinct
*** Integrate to avoid repeated code?
** should make sure that order of input ports and output ports makes parallel connections for things with same order
*** Use this for query mode?
** DK suggests breaking backward compatibility here: workflows still run, can fix easily if a problem in an existing package.
* [TE] VisTrails 2.2.4 ready for release?
* [RR] Question about output modes ([https://github.com/VisTrails/VisTrails/pull/1007 #1007]), how to integrate in API ([https://github.com/VisTrails/VisTrails/pull/24 #24])
** tej package updated. Others?
** Should ImageFileMode be removed? ("image" is not a mode, "file" is)
* [JF] upgrades and the version tree
** ImageOutput missing?
* [JF] provenance in Jupyter
** Feel free to change how formats works
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]
* [TE] Test suite segfaults on Fedora 17 virtualbox. Install newer version? (Support ended in 2013)


=== March 4, 2015 ===
=== March 9, 2016 ===
'''Updates'''
'''Updates'''
* [TE] VTK wrapper
** Works on VTK 5.10
** Still need to test VTK 6
** New general wrappers for python functions and classes


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails?
* [TE] Fixing search mode
** When [http://i.imgur.com/UTY2qfg.png filing a pull request] from UV-CDAT/VisTrails, VisTrails/VisTrails is selected by default
** String query now matches all by default
** People keep forgetting to change the default ([https://github.com/VisTrails/VisTrails/pull/956 #956] [https://github.com/VisTrails/VisTrails/pull/968 #968] [https://github.com/VisTrails/VisTrails/pull/999 #999] [https://github.com/VisTrails/VisTrails/pull/1000 #1000] [https://github.com/VisTrails/VisTrails/pull/1003 #1003] [https://github.com/VisTrails/VisTrails/pull/1004 #1004] [https://github.com/VisTrails/VisTrails/pull/1005 #1005])
*** This caused pipeline queries to never match
** Only way to change that is to not have it marked as a fork of VisTrails/VisTrails
** String query can search: User/Notes/Tagname/Time/Module name
** Github staff [https://gist.github.com/remram44/b364c24ef1e462ce6e2f can make that change for us]; should we do it?
*** Should we search module names by default? (Disabled by default because it is expensive)
** Juliana: comments on this? visibility vs. convenience/annoyance for developers
*** Fixed notes search
* [RR] What about [https://github.com/VisTrails/VisTrails/issues/991 #991]?
** '>' parameter queries in documentation but has been removed
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)
*** Upgrades still a problem
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)
*** [JF] Allow search on past tags
 
* [TE] Create new bugfix release
** Finish querying
** Path issue on windows


=== February 25, 2015 ===
* [DK] Look into sqlalchemy and new bundle branches
 
=== March 2, 2016 ===
'''Updates'''
'''Updates'''
* [TE] Fixed cache bug with port specs
* [TE] VisTrails Next
** Most packages now work
** No qgis for Python 3


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] vtkviewcell for infovis support, can we unify with VTKCell?
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]
** need to test this
* [TE] vtk wrapping
** Mostly finished
** VTK 5.10 produce incorrect results with old wrapping
*** Old wrapper is based mostly on VTK 4
*** Most vtk_examples affected
*** [http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput]
*** should be able to upgrade from SetInput to SetInputData (need to drop GetOutput and replace with self ports)
*** can we change vtkInstance to just return self and not wrap things
** terminator example not working under 5.8?
* How does VTK wrapping fit into general wrapping framework?
* [RR] new persistence package


=== February 18, 2015 ===
=== February 23, 2016 ===
'''Updates'''
'''Updates'''
* [RR] New VisTrails API and IPython integration ([https://github.com/VisTrails/VisTrails/pull/24 #24])
* [TE] Fixed PythonSource 100% CPU bug
* [TE] VisTrails Next
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples
*** Now uses REST service, tabledata, and Google Maps
*** Web Service replaced with DownloadFile


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] VTK wrapping
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.
** [https://gist.github.com/rexissimus/16b74693545b5e7c1035 Benchmarking vtk package]
** New 2.2 release soon?
*** Old: 24.7 seconds
* [RR] Anonymous usage reports
*** New: 10.5 seconds (Except first time that adds 8 sec)
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]
**** The parsing that calls is_abstract (that tries to instanciate all vtk classes) is now only run the first time.
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?
**** get_items_from_sigstring takes 2 seconds, maybe we can use a lookup dict for already computed sigstrings?
* noWorkflow - provenance for python scripts
** Now using a general python function wrapper
** Recent work on tracking dependencies
*** VTK classes are wrapped into python function that does not depend on vistrails
* Also integrates ReproZip with Jupyter
*** VTK functions can be executed without vistrails
** [http://mybinder.org mybinder]
*** The spec maps functions into vistrails modules, but can also describe wrapping
*** Use with ReproZip to automatically build Jupyter notebook interfaces
*** A general python function wrapper that supports
* Daves comments on Jupyter integration
**** kwarg inputs
** It is possible to identify cell order on the server side
**** single, list, dict outputs
** Not trivial to track order of cells on client side
**** callback for progress reporting
* How does Galaxy workflows relate to VisTrails?
**** temporary file generator for using FilePool
** Packaging in Galaxy is hard
**** optional output generation
** Galaxy for biomedical workflows, difficulty making it general
*** Creating specs:
* Action items
***** Create spec by hand
** How difficult to track cell dependencies in Jupyter notebooks?
***** Auto-create spec outline (TODO) and manually finish it
** Use noWorkflow to track cell dependencies?
***** Dynamically create spec (VTK)
** How difficult to modify mybinder to use ReproZip packages?
***** Implement documentation wrappers (Can use scikitlearn wrapper to wrap numpydoc) (TODO)
***** Classes as bad functions needs to be wrapped in new functions before they are wrapped. This is different for each package.
****** Classes is hard: Like VTK, and matplotlib. Scikit-learn does still not wrap classes
***** Spec diffing and patching could be done using code from matplotlib.
** Still needs upgrades from old VTK package
*** Is it possible to dynamically wrap functions, e.g, you see a SetFunc and just remove the 'Set' prefix. Or do you need to create a complete mapping?


=== February 11, 2015 ===
=== February 17, 2016 ===
'''Updates'''
'''Updates'''
* Update from Friday's meeting
** discussed VisTrails internals
** discussed wrapping
*** xml discussion, hard to modify because tied to db code
*** TE has made it possible to add the schema-defined attributes to the intermediate representation
*** higher-level operations on the port specs
** make sure the simple case works
*** [JF] take a simple package with documentation and figure out what the base case for wrapping is


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] VTK wrapping
* [TE] Fixed Group memory usage
** Dynamic loading works
** Modules in groups are supposed to be freed when done executing
** Reading XML is fast enough, but [https://gist.github.com/rexissimus/8272d3c776ccfa115d82 unserializing data is slow]
*** Prevented by logger that stored references to the modules (fixed, now using id(module))
** Working on patterns for patching
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)
** Matplotlib has many advanced patterns like argument ordering, nested arguments, alternateSpecs, output types.
* [TE] VisTrails Next
*** Having all this in a general wrapper might confuse users?
** Removed deprecated code
** [RR] Delay module (except for identifiers) until you need it---e.g. don't deal with port specs, etc. until necessary
*** Core imports that should be gui imports
* Scripting Support [https://github.com/VisTrails/VisTrails/pull/950 #950]
*** GetInputFromPort etc.
** [RR] Issue with getting code from modules
**** PythonSources will be upgraded
** Design a simple solution
*** Fixes for Python < 3
** [JF] Couldn't you use modules as black boxes without conversion, just to call into modules/subworkflows easily from e.g. IPython?
*** fixes for VTK < 6
*** [RR] This is a job for the API, and a very separate use case. See [https://github.com/VisTrails/VisTrails/pull/24 #24]
** TODOs
*** Remove RemoteQ
**** Port Hadoop to tej (Already have pbs)
*** Merge package wrapper
*** Finish SQLAlchemy branch
*** Finish new bundle branch
** [JF] Will contact Jeff and Matthias about moving to Python 3
* [JF] Query by Example is broken
* [JF] VisTrails Data Cleaning Project
** Use VisTrails to process and explore task-specific data while keeping provenance.
** Remi's Idea
*** Use VisTrails as the backend in a Jupyter notebook
*** Overcome problems with executing cells in random order
**** Interesting research topic, but little interest has yet been noted
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)
*** Impose restrictions to notebook cell boundaries
*** It will only work for functional operations
*** Can we fix, or detect, non-functional (mutating) operations?
*** Analyse code and data flow and see if it mutates data in unexpected ways?
*** Mutable operations can be created in many ways in Python
* Next meeting Tuesday 11:30


=== February 4, 2015 ===
=== February 10, 2016 ===
'''Updates'''
'''Updates'''


'''Items to Discuss'''
'''Items to Discuss'''
* Wrapping
* VisTrails Next
** Format to use? Currently XML (like current matplotlib)
** Test suite passes
*** JSON and YAML have simple "to python dictionary" methods
*** Python 3 less deterministic, module upgrade order random.
*** But don't stream
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...
*** YAML a lot easier for humans
** All of our packages should run fine :)
** [DK] vtk-new-package also changes parameter names, creates enumerations
** VTK thumbnail comparison works
*** intermediate schema needs to be extensible
*** Rendering bug on some systems
*** packages will want to store there specific infos for compute() method generation
**** QVTKCell now inherits QGLWidget
*** also might have specs-altering info, like matplotlib's alternateSpec
*** Old thumbnails had wrong height due to spreadsheetwindow being too small
** representation to code , registry already has schema for some aspects
**** Fixed by regenerating thumbnails
** [RR] We might want to see if Module subclasses can be created lazily
*** Silenced deprecation messages when generating modules
*** no need to create all the classes just to register them in the registry and never actually use most of them
*** Many new classes in VTK 7
*** future effort
** Remove deprecated code
* [RR] Where should VisTrails packages live?
*** Core imports that should be gui imports
** tej installs as 'vistrailspkg.tej', TE installed it as 'userpackages.tej'
*** GetInputFromPort etc.
** Currently, standard packages are 'vistrails.packages.', user packages are 'userpackages.' and packages loaded through pkg_resources might be anything
**** Will break existing PythonSources
** [RR] Use 'vistrailspkg.' everywhere?
**** Try to upgrade?
** Long-term effort to simplify package distribution/installation (and have VisTrails get them automatically?)
*** Old-style packages
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
***** Read info.py without importing package?
*** Fixes for Python < 3
*** Fixes for Qt < 5
*** fixes for VTK < 7
*** Remove self port?


=== January 28, 2015 ===
=== February 3, 2016 ===
'''Updates'''
'''Updates'''
* T. Caswell to come visit on Fri 6 to discuss wrapping work


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] New VTK wrapping
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]
** Current code by DK seems a good deal faster
** Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code
** RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower
** The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own
** TC has its own code at [https://github.com/Nikea/VTTools/blob/master/vttools/wrap_lib.py github:VTTools] which parses numpy docstrings and generates modules, doesn't yet handle classes or persist anything
* Web crawler
** Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"
** RR would rather have the crawling be a job as far as VisTrails and tej are concerned
** The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize
*** Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached


=== January 21, 2015 ===
* VisTrails Next
** All tests but thumbnail comparison passes
** Still issues with gui and more packages needs to be tested
** PythonSource and MplSource upgraded with lib2to3
** Keep contrib packages? Update?
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]
*** Supports Python 3
*** Will hopefully get into conda and others soon
*** Deprecates a bunch of modules
 
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next
** Runs!
** Uses conda
*** Many levels of requirements
**** apt packages
**** conda packages
**** python packages in requirements.txt
**** python packages in setup.py
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!
*** But useful if we want to use conda in our own build process.
** Lots of errors I do not get locally?
 
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]
** Have you used any alternatives to vistrails?
*** Relevant? More options?
** How often do you use any of these particular features in VisTrails?
*** Add/Remove alternatives?
** How important would you rank the following potential improvements to VisTrails?
*** Add alternatives?
 
=== January 27, 2016 ===
'''Updates'''
'''Updates'''
* New interpreter
** Working prototype soon
** Looping not completely done (and port combination mode: pairwise/cartesian...)
** Need to integrate with VisTrails
*** Compatibility with existing modules
*** Use log controller
*** Want to cache at multiple levels (whole stream/single element of stream)
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Unified wrapping method discussion [https://github.com/VisTrails/VisTrails/issues/991 #991]
* User Survey
** TE to work on reusable method with intermediate representation, starting with VTK
 
* [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)
* ZMQ failing on windows (vistrails-users)
** AM's examples are enough
 
* [AM] scikit-learn package is done, merge it in? [https://github.com/VisTrails/VisTrails/pull/955#issuecomment-70286788 #955]
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]
** RR will merge
** Testsuite almost passes
* [RR] What should copyright headers say? [https://github.com/VisTrails/VisTrails/pull/994#issuecomment-70173935 #994]
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.
** Let's keep everything in there: Utah/Poly/NYU
** Fixed issues
*** Signatures are now bytes (convert to string to show in widget)
*** Cannot compare different types, e.g.,  (None < 1)
**** PortSpec.depth and min/maxConn could be None. (Backport!)
**** cell.row could be None
*** Lots of unicode/bytes fixes to serializations
**** We have encryption implementations in the code! (keychain)
*** Lots of small fixes to package method differences
** Current issues
*** logging widget empty?
*** version tree signals messed up
*** provenance view legend icons black
*** Text do not fit in some widgets
*** VTK thumbnail comparison fails
**** VTK Cells have some faces in wrong direction?
****  Because running in VM?
** Installing on Ubuntu LTS/Latest Mint
*** Install latest matplotlib from pip
*** Install VTK 7.0.0rc2 from source
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)


=== January 14, 2015 ===
=== January 20, 2016 ===
'''Updates'''
'''Updates'''
* [TE] Working on classifier
* [RR] Scripting integration, work in progress


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])
* [TE] Backporting features from Python 3 + PyQt5 + Qt5
** Let's talk next week, [AM] and [DK] are not here
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])
*** One unicode test failing
*** Problem saving to xml
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])
*** Updates print to functions 'from __future__ import print_function'
*** Updates exception calls to use 'as'
*** removes most usages of 'long' (mostly used unnecessarily)
**** Only needed for 32-bit support
*** Fixes relative imports (from __future__ import absolute_import)
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])
*** Updated all signals to new-style
* If we go 2 to 3, this could break packages
** means we should schedule any package format changes for the same time
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)
** putting package identifier as xml?
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)
** version the APIs?
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?
* file formats on 2 to 3:
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2


=== January 7, 2015 ===
=== January 13, 2016 ===
'''Updates'''
'''Updates'''


'''Items to Discuss'''
'''Items to Discuss'''
* make sure that we address critical issues, questions, and pending review branches in a timely manner
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]
* scripting support
** Basic GUI and execution works
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow
** Minimal work to support 2to3
** [RR] can translate from workflow to script, working on script to workflow
** Hard to support both PyQt4 and PyQt5
** will work for parameter value changes, structural changes require changes to the annotations
 
** need to publish best practices here
=== January 6, 2016 ===
** would be cool to do looping in scripts (easier interface than with workflows)
'''Updates'''
* notebook support (convert form notebook to workflow)
'''Items to Discuss'''
** RR will sync with FC on this
* Updating VisTrails dependencies
* Issue with console in built-from-scratch
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.
** binary has old version of iPython -> 1.0.0, should we update?
**** Next version of VTK 6 is rumored to support Python 3
* [TC] automated wrapping of numpy and scipy
**** There is a development branch of VTK 6 supporting Python 3
** discovered a bunch of malformed documentation in numpy and scipy
*** Can travis build this?
** has github repo for vistrails tools
**** We can skip tests for WebEngine and VTK if needed
** example modules wrap a bunch of R stuff (not baked in, just how things are)
** Python 3 [https://python3wos.appspot.com/ showstoppers]
** will be pushing wrapping logic up
*** No MySQL-python (sqlalchemy supports mysql through other drivers)
** port names forbidden (window and domain)
*** No suds (there are active forks but we could also drop support?)
** have an import hook to get from yaml directly to VisTrails Modules
* Options for python2/3 support:
** should work for any python modules with well-formed numpy docstrings.
** Interpreter could be python2/python3 based on packages
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)
** six layer works but not the easiest to maintain (e.g. in reprozip)
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)
* Drop support for win-64
* [TE] build and build scripts
* Update copyright headers to 2016
** completely automatic, buildbot
** need to set the build machines for the environment we want for the binary
** would virtualenv work here?
** [TC] anaconda can pin versions, potential path to test different configurations
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07
* package issues (see Remi's message)
* [TE] Scope of tej
** Support single ssh commands?
** Queue can be used as a remote machine (crawler is using queue.call*)
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07


== Older meetings ==
== Older meetings ==
* [[Development/2015]]
* [[Development/2014]]
* [[Development/2014]]
* [[Development/2013]]
* [[Development/2013]]

Latest revision as of 15:34, 30 October 2017

See also the Github wiki

2017

October 30, 2017

Updates

  • [DK] PyQt5
  • [RR] Running Python3 code

Items to Discuss

  • Installation:
    • Use conda? (macOS hack to have .app bundle load conda-installed vistrails)
    • Update conda recipe (pyqt4 restriction for now)
  • Interpreter
  • Release
  • Integration
    • hard to build same type of GUI in other apps that integrate core VisTrails (tend to use just the API)
    • independent pieces of the GUI could be reused if they were more independent.

September 6, 2017

Updates

  • [RR] Fixes for scikit-learn, tables, usagestats

Items to Discuss

  • Stripped-down version of VisTrails to showcase new features?
    • include meta-vistrail, bundles
    • hide mashups, extra widgets, latex mode, etc.
  • Interpreter

August 16, 2017

Updates

  • [DK] meta-vistrail
    • Working actions: normal action in vistrail, pruning version tree, tagging operations, and change
    • Interface

Items to Discuss

  • Notebook
  • Interpreter
  • Web visualization packages (e.g. bokeh)
  • Pipeline templates

August 2, 2017

Updates

  • [DK] meta-vistrail work
    • Extend Vistrail to MetaVistrail to support meta-actions
    • Vistrail becomes less tied to workflows so it can also supporting versioning vistrails

Items to Discuss

  • Interpreter
    • Using dask?
    • Streaming an issue
  • Notebooks
    • What is an output of a module? (need to expose a specific output port or the entire module)
    • Unifies some of the work from the script-workflow code

June 28, 2017

Updates

  • [DK] Testing UUID/SQLAlchemy branch
  • [DK] Working on meta-versions
    • would be nice to have pipeline and vistrail separate
    • [RR] Then do we only have meta-actions, any change to the pipeline is a meta-action that adds a version that does the change?
    • [DK] Yes?
  • [RR] Notebook interface (Vizier) making good progress, figuring out cell outputs

Items to Discuss

June 7, 2017

Updates

  • [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs

Items to Discuss

  • [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)
    • Can designate in module description an output port
    • Value has to be serializable
    • Should use the OutputModes work to handle multiple modes, configuration, selection, ...
  • [RR] Is VisTrails with SQL usable concurrently?
    • [DK] Probably not, demo from [TE] exists

May 10, 2017

Items to Discuss

  • Identifiers
  • Bundles & DB Code
  • Interpreter
  • Merge / Versioning Version Trees
  • Python3
  • Workings of List
  • Packages: Wrappers + new Versions (VTK7)
  • Script Conversion
  • Intermediate Outputs: reuse output module work?
    • would be useful for both workflows (run to a particular checkpoint)
    • and useful for notebook interfaces
  • SQL backend: does it allow for concurrent access?
    • Yes, real-time collaboration work for concurrent single vistrail access
    • Should be able to access different vistrails at once without issues

April 19, 2017

Updates

Items to Discuss

April 5, 2017

Updates

Items to Discuss

March 8, 2017

Updates

  • [RR] Added a looping indicator on modules #1217
    • This makes it easier to understand what's looping, no need to follow edges and count
  • [RR] Refactoring CLTools for integration in Docker package
  • [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)

Items to Discuss

  • [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers #1216
    • Can we move this up to the renderer modules?
    • Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)

February 24, 2017

Updates

  • [RR] Rich-text module documentation (#1210)
    • Went over packages and fixed format; maybe we should have a test to validate RST?
  • [RR] Single-instance shutdown with --remote-shutdown (#1204)
  • [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment

Items to Discuss

  • [RR] There are issues with list-depth
    • Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)
    • Issue with looping groups

January 25, 2017

Updates


Items to Discuss

  • Scaling VisTrails
  • sql-alchemy, uuid, bundle branches/merges

January 11, 2017

Updates

  • [RR] New VisTrails version in Debian [1]

Items to Discuss

  • [RR] Collaboration with school of medicine to port HiC-bench to VisTrails
    • Have homemade workflow system based on R scripts and symlinks
      • Each steps has multiple sets of parameters
      • Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step
      • Gets all results in directory structure step3-parameters/step2-parameters/step1-parameters/files
    • VisTrails improves provenance capture and makes editing pipelines easier
      • Need better support for aggregation (groupby operation might be good enough; pandas?)

2016

December 14, 2016

Updates

  • [RR] Single-instance code: over-engineered, but will work
    • (Have to avoid race conditions on startup, and OSes offer very different locking primitives)

Items to Discuss

  • Email from Matt Dirks: combobox should work, ask for more code?
  • Email from marin.nl:
    • looks like they would benefit from new interpreter
    • how to store their consolidated provenance when they exchange .vt files around?

November 30, 2016

Updates

Items to Discuss

  • [RR] Working on single instance code for MARIN (#1205)
    • Remote shutdown ready (#1204)
    • Still improving it
    • Want to return something more structured to the client (success/fail code, stdout, stderr)
    • Should be usable without Qt
  • [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason
  • [DK] Working on batch execution using console_mode
  • [DK] Settings aliased parameters when using core.api?

November 16, 2016

Updates

  • [RR] Issue #1200 from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)

Items to Discuss

  • Email from Pieter: executing pipelines with parameters from command-line using single-instance
    • Seems to work, main instance gets full command-line and alters configuration temporarily
    • Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful
    • Actions:
      • Can add switch to kill main instance
      • Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails
      • Add client/server code to core that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like gui.application.VistrailsApplicationSingleton#noninteractiveMode())?

November 2, 2016

Updates

  • New version 2.2-2 of Debian package, still wrong copyright, and broken bundle installation
    • Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch
  • Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)

Items to Discuss

October 5, 2016

Items to Discuss

  • working toward a 3.0 release; project board
    • unique ids
    • bundles
    • GUI fixes
    • wrapping updates
    • python3
    • interpreter?
  • command-line input parameters to provenance
    • [DK] we don't create a persisted workflow from those input params, i think
    • [DK] these are already defined as aliases so we treat them differently form a normal input param
    • [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases
    • Do the vistrails variables values get stored in provenance? (what if values change?)

September 14, 2016

Updates

Items to Discuss

August 31, 2016

Updates

  • [RR] Went over vgc.poly.edu/vistrails.org config:
    • Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
    • Fixed usersguide scripts (won't disappear again)
    • Moved some websites over to HTTPS
    • Websites
  • [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs (currently there) and users' guide under the same RTD "project"
  • [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that

Items to Discuss

  • [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
    • Get rid of more services? (svn, trac, ...)
  • [RR] Notebook execution works. How do we support more languages (than Python)?
    • Need: detection, metadata reader, code to input values
  • [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?

August 17, 2016

Updates

  • [TE] Fixed parameter order upgrade bug that breaks VTK examples.
  • [TE] Matplotlib and sklearn are exportable using module-to-script api
  • [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.

Items to Discuss

  • Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
  • Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
  • [TE] Wrap-up notes

August 10, 2016

Updates

Items to Discuss

  • [TE] Python Wrapper and scripting
    • All changes merged into python-wrapper branch
      • Tests pass
      • Still need to update sub-branches
    • Minor fixes
      • vtkPlot3DReader upgrade bug
      • Added tuple exporting
    • Most of vtk examples can be exported example
      • Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
        • Could be a bug in the old vistrail, but not sure
  • [DK] Subworkflows and uuids

August 3, 2016

Updates


Items to Discuss

  • [TE] Wrapper patches now stored as python code
    • Connection variables now lowercase
    • Can we use code as output variables?
      • E.g. "vtk_dataset_mapper.GetOutput(0)"
      • Could mutate output when output method is only supposed to be called once
      • Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)

July 27, 2016

Updates

  • [DK] DB bundle serialization implemented

Items to Discuss 

   def vtkInstance():
       self.vtkInstance = self

July 20, 2016

Updates

Items to Discuss

  • [TE] New patch method for python-wrapper example
    • A patch is a string template with self/input/output variables for patching a class method
    • Patches are stored in xml specification
    • Patches are assigned to methods in ClassSpec's
    • Used both for execution in vistrails and when exporting as script
      • Executed with "exec" when run in vistrails
      • Turns patches into natural calls in script
    • Wrapping VTK library no longer needed
      • Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)
    • Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)
    • Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)
    • TODO's
      • Add to export_as_script
      • Patch Modules that represent Python functions as well
  • [vistrails-users] Read data from SQL example?

July 13, 2016

Updates

  • [DK] bundles on sql-alchemy branch
    • Bundle, BundleMapping, and BundleSerializer
    • New BaseSerializers to deal with versions of directory, zip, db serializers
    • Can register the BundleSerializers with the base serializers
    • Translation between bundle versions
    • Bundle.add_object "just works" if an appropriate mapping was registered

Items to Discuss

  • [TE] Simplified running module using API for scripting example
    • API Module can be executed using compute method
      • Takes list of outputs to compute + inputs as keyword arguments
  • Status of general cell synchronization (vistrails-users)

July 6, 2016

Updates

  • [DK] Working on bundles

Items to Discuss

  • [TE] Using vistrails API in scripting example 1 (API with VTK) example 2 (DownloadFile only)
    • Call module in script using vistrails API
    • Can be used when no export method is defined.
  • [TE] Server news bug Critical?
    • [RR] No, unreleased
  • [DK] New bundle organization
    • BundleMapping contains individual mappings from raw objects to BundleObj
    • Must be separate in order to have in-memory, separate from serializer settings

June 29, 2016

Updates

Items to Discuss

  • [TE] Export VTK as python script
    • Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) example
    • Preludes (imports) still need improvements
    • Works with vtkInteractionHandler (clipping plane)
    • Resulting clipping plane is incorrect? (supposed to hide one side of the plane)
    • vtkRendererOutput Uses VTKRenderWindow as output
      • Should support different output modes?
      • Does not work with multiple outputs (Next cell is displayed when current is closed)
      • Uses wrapped vtkInstance (as vvtk)
        • Create new patch system with code that can be used both in vistrails and in a python script?

June 22, 2016

Updates

Items to Discuss

  • [TE] Fix for upgraded analogies (PR #1177)
    • Uses module location to find module remap (The best we can do right now?)
    • Also fixes remapped port names for connections to existing modules
    • May still fail on complex upgrades like adding a function on an upgraded module that has a different port name
  • [TE] Working on exporting wrapped libraries (VTK)

June 15, 2016

Updates

  • [TE] Added depth logic to python script exporter (full commit)
    • Fully implements connection merging and list looping

Items to Discuss

  • [TE] Making analogy work on upgraded workflows (#1175)
    • Analogy only works We need to use the non-upgraded action chain
    • But pipelines need to be valid (matcher uses module portspecs)
    • Tried to get upgrade remap from upgrade action, but this seems impossible
    • Use module location instead?
      • Location should be identical for one-to-one module upgrades

June 1, 2016

Updates

  • VisTrails 2.2.4 announced (finally)
    • 5 survey replies
    • 47 downloads last week

Items to Discuss

  • [TE] Export VTK as script (python-wrapper)
    • to_python_script
      • needs the pipeline (or at least the connections)
        • Should not generate code for any possible VTK method (exporter detects which ports is used in the source)
        • VTK needs to know connection order
        • Adding functions directly to the script results in nicer code
    • Add support for multiple input connections
    • Add support for list looping

May 25, 2016

Updates

  • Mailing-list fixed
  • Getting stats from the "server news" mechanism (via parser)

Items to Discuss

  • [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)
  • [TE] Union ports
    • Show union ports in pipeline (DEMO)

May 18, 2016

Items to Discuss

  • [TE] mailing list still down - Julio working on it
  • [TE] Improved union port
    • New schema 1.0.5 adds portitem.union string attribute
    • Works with pipeline view, parameter exploration view, and mashup view
    • Should work well for matplotlib
    • Use with VTK ports?
      • It uses [item_1,item_2] style ports usually with different tuple length.
  • [TE] Remove emacs dependency from db generate #1173

May 11, 2016

Items to Discuss

  • [TE] Union type #1172
    • GUI-based solution Branch union-port Demo?
      • Shows single port for all ports with same sort_key
      • Select specific type when adding/showing
      • Only one type should be visible on the module at any time
      • Making tuple parameters union types is probably too complicated
      • Should work with alternate specs like matplotlib (if sort_key is defined correctly)
  • [TE] Feature suggestion: Let connections connect to functions/parameters
    • Show functions and parameters as ports and toggle port visibility directly for functions and parameters
    • Use separate views for ports and functions (like before)
      • Allow rearranging of functions
    • Pros:
      • Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
        • We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.
        • Makes List module simpler
      • Connect to parameter in tuple
      • Functions that are visible on the module can connect to an actual function, and not just to the first one it finds.
      • Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.
    • Cons:
      • Interface more complicated and crowded
      • More difficult for users to grasp?
      • No immediate use case
      • Makes no union type for tuple parameters a bigger problem.
  • [TE] Publish user survey?
    • No response from vistrails release yet
    • Not announced on vistrails-users yet.

May 4, 2016

Updates

  • VisTrails 2.2.4 released

Items to Discuss

April 27, 2016

Updates

Items to Discuss

  • [TE] Wrapping Bokeh
    • New method type "operation"
      • Transforms Figure->Glyph->Output into Glyph->Figure->Output
        • Glyph sets function name and arguments as output
          • Not serializable, but should always be computed together with the subject of the operation.
        • Figure applies Glyph operation on itself
      • Useful for matplotlib?
        • matplotlib currently find actors by searching upstream?
        • But this is more difficult since matplotlib is not function-based.
    • New "union" port type
      • Turns "x(float), x(list)" into "x (float, list)" and uses first set value
        • Uses only first value in tuple
        • Bokeh have many properties that support (list, string, Float)
      • Does not work with depth=1 types (e.g. Float of depth 1)
        • Turn Depth 1 types with known type into lists?
  • [TE] New bugfix release
    • Finalize survey

April 20, 2016

Updates

  • Merged upgrade fixes
    • Now has infinite loop test
    • loop test now correctly loops 5 times
      • remap_module used wrong (current) package version when creating ModuleDescriptor
  • Fixed multi-step group upgrades (PR: #1168)

Items to Discuss

  • New bugfix release
    • Merge usagestats
    • Need to finalize survey

April 13, 2016

Updates

Items to Discuss

  • [TE] Chaining upgrades done (#1164)
    • Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?
  • [TE] Iterating handle_invalid_pipeline (#1165)
    • Not based on chain-upgrades-loop
    • Cherry-pick test to preserve source?
    • Need maxPipelineFixAttempts option?
      • The code is now smarter about recursion, so max attempts will "probably" never be reached.
    • Need debug messages?
      • Would need to be moved into handle_invalid_pipeline

April 6, 2016

Updates

  • [TE] Query View improvements
    • Done except for displaying "Version Info" and #1164
  • [RR] Working on finishing new interpreter (finally!)

Items to Discuss

  • [TE] Chaining upgrades (#1164)
    • Try latest upgrade first, then previous.
    • If all fail, select latest upgrade?

March 30, 2016

Updates

Items to Discuss

  • [TE] Query View improvements
    • Unified do_version_switch and get_upgrades as validate_version()
    • Problems with query view using its own controller
      • Fixed module info not being displayed for query results
      • Still no version info for query results
      • Still an issue with version query view not being updated

March 23, 2016

Updates

Items to Discuss

  • [RR] Usage reporting almost ready to go
    • Need to record whether specific features are used: paramexpore, mashup, query
      • [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation
    • How to record size of vistrail?
      • [DK] IdScope already walks over vistrail, use that
      • but total number of modules and versions doesn't give an idea of average size of individual pipelines

March 16, 2016

Updates

Items to Discuss

  • [TE] Upgrades in Query Mode
    • Now using code similar to do_version_switch
      • Uses handle_invalid_pipeline
      • Differences: Not using current_pipeline, flushes changes
      • New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)
      • Integrate to avoid repeated code?
      • Use this for query mode?
  • [TE] VisTrails 2.2.4 ready for release?
    • tej package updated. Others?
  • [JF] upgrades and the version tree
  • [JF] provenance in Jupyter
  • [DK] bundles: #1092

March 9, 2016

Updates

Items to Discuss

  • [TE] Fixing search mode
    • String query now matches all by default
      • This caused pipeline queries to never match
    • String query can search: User/Notes/Tagname/Time/Module name
      • Should we search module names by default? (Disabled by default because it is expensive)
      • Fixed notes search
    • '>' parameter queries in documentation but has been removed
    • Fixed module matches for upgraded pipelines (By not upgrading pipeline)
      • Upgrades still a problem
      • Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)
      • [JF] Allow search on past tags
  • [TE] Create new bugfix release
    • Finish querying
    • Path issue on windows
  • [DK] Look into sqlalchemy and new bundle branches

March 2, 2016

Updates

  • [TE] Fixed cache bug with port specs
  • [TE] VisTrails Next
    • Most packages now work
    • No qgis for Python 3

Items to Discuss

    • Always dereference symlinks? #1146

February 23, 2016

Updates

  • [TE] Fixed PythonSource 100% CPU bug
  • [TE] VisTrails Next
    • Updated NOAA Web Service examples
      • Now uses REST service, tabledata, and Google Maps
      • Web Service replaced with DownloadFile

Items to Discuss

  • [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.
    • New 2.2 release soon?
  • [RR] Anonymous usage reports
    • Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features #1154
  • [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?
  • noWorkflow - provenance for python scripts
    • Recent work on tracking dependencies
  • Also integrates ReproZip with Jupyter
    • mybinder
      • Use with ReproZip to automatically build Jupyter notebook interfaces
  • Daves comments on Jupyter integration
    • It is possible to identify cell order on the server side
    • Not trivial to track order of cells on client side
  • How does Galaxy workflows relate to VisTrails?
    • Packaging in Galaxy is hard
    • Galaxy for biomedical workflows, difficulty making it general
  • Action items
    • How difficult to track cell dependencies in Jupyter notebooks?
    • Use noWorkflow to track cell dependencies?
    • How difficult to modify mybinder to use ReproZip packages?

February 17, 2016

Updates

Items to Discuss

  • [TE] Fixed Group memory usage
    • Modules in groups are supposed to be freed when done executing
      • Prevented by logger that stored references to the modules (fixed, now using id(module))
      • Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)
  • [TE] VisTrails Next
    • Removed deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • PythonSources will be upgraded
      • Fixes for Python < 3
      • fixes for VTK < 6
    • TODOs
      • Remove RemoteQ
        • Port Hadoop to tej (Already have pbs)
      • Merge package wrapper
      • Finish SQLAlchemy branch
      • Finish new bundle branch
    • [JF] Will contact Jeff and Matthias about moving to Python 3
  • [JF] Query by Example is broken
  • [JF] VisTrails Data Cleaning Project
    • Use VisTrails to process and explore task-specific data while keeping provenance.
    • Remi's Idea
      • Use VisTrails as the backend in a Jupyter notebook
      • Overcome problems with executing cells in random order
        • Interesting research topic, but little interest has yet been noted
      • Interesting to study how such a notebook would be used (Is there notebook usage data available?)
      • Impose restrictions to notebook cell boundaries
      • It will only work for functional operations
      • Can we fix, or detect, non-functional (mutating) operations?
      • Analyse code and data flow and see if it mutates data in unexpected ways?
      • Mutable operations can be created in many ways in Python
  • Next meeting Tuesday 11:30

February 10, 2016

Updates

Items to Discuss

  • VisTrails Next
    • Test suite passes
      • Python 3 less deterministic, module upgrade order random.
    • [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...
    • All of our packages should run fine :)
    • VTK thumbnail comparison works
      • Rendering bug on some systems
        • QVTKCell now inherits QGLWidget
      • Old thumbnails had wrong height due to spreadsheetwindow being too small
        • Fixed by regenerating thumbnails
      • Silenced deprecation messages when generating modules
      • Many new classes in VTK 7
    • Remove deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • Will break existing PythonSources
        • Try to upgrade?
      • Old-style packages
        • Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
          • Read info.py without importing package?
      • Fixes for Python < 3
      • Fixes for Qt < 5
      • fixes for VTK < 7
      • Remove self port?

February 3, 2016

Updates

Items to Discuss

  • VisTrails Next
    • All tests but thumbnail comparison passes
    • Still issues with gui and more packages needs to be tested
    • PythonSource and MplSource upgraded with lib2to3
    • Keep contrib packages? Update?
    • VTK 7 is released
      • Supports Python 3
      • Will hopefully get into conda and others soon
      • Deprecates a bunch of modules
  • Travis builder for VisTrails Next
    • Runs!
    • Uses conda
      • Many levels of requirements
        • apt packages
        • conda packages
        • python packages in requirements.txt
        • python packages in setup.py
      • Can probably drop conda. Default builder does not have Qt5, but beta builder do!
      • But useful if we want to use conda in our own build process.
    • Lots of errors I do not get locally?
  • User Survey
    • Have you used any alternatives to vistrails?
      • Relevant? More options?
    • How often do you use any of these particular features in VisTrails?
      • Add/Remove alternatives?
    • How important would you rank the following potential improvements to VisTrails?
      • Add alternatives?

January 27, 2016

Updates

  • New interpreter
    • Working prototype soon
    • Looping not completely done (and port combination mode: pairwise/cartesian...)
    • Need to integrate with VisTrails
      • Compatibility with existing modules
      • Use log controller
      • Want to cache at multiple levels (whole stream/single element of stream)
      • Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)

Items to Discuss

  • User Survey
  • ZMQ failing on windows (vistrails-users)
  • [TE] Python 3 branch
    • Testsuite almost passes
      • Failing are unicode filenames, some users guide examples, and thumbnail comparisons.
    • Fixed issues
      • Signatures are now bytes (convert to string to show in widget)
      • Cannot compare different types, e.g., (None < 1)
        • PortSpec.depth and min/maxConn could be None. (Backport!)
        • cell.row could be None
      • Lots of unicode/bytes fixes to serializations
        • We have encryption implementations in the code! (keychain)
      • Lots of small fixes to package method differences
    • Current issues
      • logging widget empty?
      • version tree signals messed up
      • provenance view legend icons black
      • Text do not fit in some widgets
      • VTK thumbnail comparison fails
        • VTK Cells have some faces in wrong direction?
        • Because running in VM?
    • Installing on Ubuntu LTS/Latest Mint
      • Install latest matplotlib from pip
      • Install VTK 7.0.0rc2 from source
      • No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)

January 20, 2016

Updates

Items to Discuss

  • [TE] Backporting features from Python 3 + PyQt5 + Qt5
    • Merged unicode with master (unicode)
      • One unicode test failing
      • Problem saving to xml
    • Used python-future to run futurize stage 1 to update legacy (pre 2.6) code (unicode-futurized)
      • Updates print to functions 'from __future__ import print_function'
      • Updates exception calls to use 'as'
      • removes most usages of 'long' (mostly used unnecessarily)
        • Only needed for 32-bit support
      • Fixes relative imports (from __future__ import absolute_import)
    • Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 (unicode-futurized-qt5)
      • Updated all signals to new-style
  • If we go 2 to 3, this could break packages
    • means we should schedule any package format changes for the same time
    • e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* #1002)
    • any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)
    • putting package identifier as xml?
      • Python not so bad, plus some stuff in there is Python code (checking dependencies)
    • version the APIs?
      • RR planning on versioning vistrails_module.Module, no point versioning the whole package?
  • file formats on 2 to 3:
    • no real translation/upgrade, but non-ascii values that work on 3 won't load in 2

January 13, 2016

Updates

Items to Discuss

January 6, 2016

Updates Items to Discuss

  • Updating VisTrails dependencies
    • Successfully built Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7
      • VTK 7 is the first supporting Python 3 but is only in release candidate phase.
        • Next version of VTK 6 is rumored to support Python 3
        • There is a development branch of VTK 6 supporting Python 3
      • Can travis build this?
        • We can skip tests for WebEngine and VTK if needed
    • Python 3 showstoppers
      • No MySQL-python (sqlalchemy supports mysql through other drivers)
      • No suds (there are active forks but we could also drop support?)
  • Options for python2/3 support:
    • Interpreter could be python2/python3 based on packages
    • six layer works but not the easiest to maintain (e.g. in reprozip)
  • Drop support for win-64
  • Update copyright headers to 2016

Older meetings

Retrieved from "https://www.vistrails.org//index.php?title=Development&oldid=14736"