Difference between revisions of "Development"

From VistrailsWiki
Jump to navigation Jump to search
 
(72 intermediate revisions by 3 users not shown)
Line 4: Line 4:


See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]
== 2017 ==
=== October 30, 2017 ===
'''Updates'''
* [DK] PyQt5
* [RR] Running Python3 code
'''Items to Discuss'''
* Installation:
** Use conda? (macOS hack to have .app bundle load conda-installed vistrails)
** Update conda recipe (pyqt4 restriction for now)
* Interpreter
* Release
* Integration
** hard to build same type of GUI in other apps that integrate core VisTrails (tend to use just the API)
** independent pieces of the GUI could be reused if they were more independent.
=== September 6, 2017 ===
'''Updates'''
* [RR] Fixes for scikit-learn, tables, usagestats
'''Items to Discuss'''
* Stripped-down version of VisTrails to showcase new features?
** include meta-vistrail, bundles
** hide mashups, extra widgets, latex mode, etc.
* Interpreter
=== August 16, 2017 ===
'''Updates'''
* [DK] meta-vistrail
** Working actions: normal action in vistrail, pruning version tree, tagging operations, and change
** Interface
'''Items to Discuss'''
* Notebook
* Interpreter
* Web visualization packages (e.g. bokeh)
** [https://github.com/VisTrails/VisTrails/pull/969 PR 969] has more details
* Pipeline templates
=== August 2, 2017 ===
'''Updates'''
* [DK] meta-vistrail work
** Extend Vistrail to MetaVistrail to support meta-actions
** Vistrail becomes less tied to workflows so it can also supporting versioning vistrails
'''Items to Discuss'''
* Interpreter
** Using dask?
** Streaming an issue
* Notebooks
** What is an output of a module? (need to expose a specific output port or the entire module)
** Unifies some of the work from the script-workflow code
=== June 28, 2017 ===
'''Updates'''
* [DK] Testing UUID/SQLAlchemy branch
* [DK] Working on meta-versions
** would be nice to have pipeline and vistrail separate
** [RR] Then do we only have meta-actions, any change to the pipeline is a meta-action that adds a version that does the change?
** [DK] Yes?
* [RR] Notebook interface (Vizier) making good progress, figuring out cell outputs
'''Items to Discuss'''
=== June 7, 2017 ===
'''Updates'''
* [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs
'''Items to Discuss'''
* [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)
** Can designate in module description an output port
** Value has to be serializable
** Should use the OutputModes work to handle multiple modes, configuration, selection, ...
* [RR] Is VisTrails with SQL usable concurrently?
** [DK] Probably not, demo from [TE] exists
=== May 10, 2017 ===
'''Items to Discuss'''
* Identifiers
* Bundles & DB Code
* Interpreter
* Merge / Versioning Version Trees
* Python3
* Workings of List
* Packages: Wrappers + new Versions (VTK7)
* Script Conversion
* Intermediate Outputs: reuse output module work?
** would be useful for both workflows (run to a particular checkpoint)
** and useful for notebook interfaces
* SQL backend: does it allow for concurrent access?
** Yes, real-time collaboration work for concurrent single vistrail access
** Should be able to access different vistrails at once without issues
=== April 19, 2017 ===
'''Updates'''
'''Items to Discuss'''
=== April 5, 2017 ===
'''Updates'''
'''Items to Discuss'''
=== March 8, 2017 ===
'''Updates'''
* [RR] Added a looping indicator on modules [https://github.com/VisTrails/VisTrails/pull/1217 #1217]
** This makes it easier to understand what's looping, no need to follow edges and count
* [RR] Refactoring CLTools for integration in Docker package
* [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)
'''Items to Discuss'''
* [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers [https://github.com/VisTrails/VisTrails/issues/1216 #1216]
** Can we move this up to the renderer modules?
** Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)
=== February 24, 2017 ===
'''Updates'''
* [RR] Rich-text module documentation ([https://github.com/VisTrails/VisTrails/pull/1210 #1210])
** Went over packages and fixed format; maybe we should have a test to validate RST?
* [RR] Single-instance shutdown with <tt>--remote-shutdown</tt> ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])
* [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment
'''Items to Discuss'''
* [RR] There are issues with list-depth
** Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)
** Issue with looping groups
=== January 25, 2017 ===
'''Updates'''
'''Items to Discuss'''
* Scaling VisTrails
* sql-alchemy, uuid, bundle branches/merges
=== January 11, 2017 ===
'''Updates'''
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]
'''Items to Discuss'''
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails
** Have homemade workflow system based on R scripts and symlinks
*** Each steps has multiple sets of parameters
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code>
** VisTrails improves provenance capture and makes editing pipelines easier
*** Need better support for aggregation (groupby operation might be good enough; pandas?)


== 2016 ==
== 2016 ==
=== December 14, 2016 ===
'''Updates'''
* [RR] Single-instance code: over-engineered, but will work
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)
'''Items to Discuss'''
* Email from Matt Dirks: combobox should work, ask for more code?
* Email from marin.nl:
** looks like they would benefit from new interpreter
** how to store their consolidated provenance when they exchange .vt files around?
=== November 30, 2016 ===
'''Updates'''
'''Items to Discuss'''
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])
** Still improving it
** Want to return something more structured to the client (success/fail code, stdout, stderr)
** Should be usable without Qt
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason
* [DK] Working on batch execution using console_mode
* [DK] Settings aliased parameters when using core.api?
=== November 16, 2016 ===
'''Updates'''
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)
'''Items to Discuss'''
* Email from Pieter: executing pipelines with parameters from command-line using single-instance
** Seems to work, main instance gets full command-line and alters configuration temporarily
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful
** Actions:
*** Can add switch to kill main instance
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?
=== November 2, 2016 ===
'''Updates'''
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)
'''Items to Discuss'''
=== October 5, 2016 ===
'''Items to Discuss'''
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]
** unique ids
** bundles
** GUI fixes
** wrapping updates
** python3
** interpreter?
* command-line input parameters to provenance
** [DK] we don't create a persisted workflow from those input params, i think
** [DK] these are already defined as aliases so we treat them differently form a normal input param
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases
** Do the vistrails variables values get stored in provenance? (what if values change?)
=== September 14, 2016 ===
'''Updates'''
'''Items to Discuss'''
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)
** Did we know/contribute to this?
*** [DK] Yes. We wrote the text for the VisTrails chapter
** Drafted response
*** [DK] Looks good, will forward.
=== August 31, 2016 ===
'''Updates'''
* [RR] Went over vgc.poly.edu/vistrails.org config:
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
** Fixed usersguide scripts (won't disappear again)
** Moved some websites over to HTTPS
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that
'''Items to Discuss'''
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
** Get rid of more services? (svn, trac, ...)
* [RR] Notebook execution works. How do we support more languages (than Python)?
** Need: detection, metadata reader, code to input values
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?
=== August 17, 2016 ===
'''Updates'''
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.
* [TE] Matplotlib and sklearn are exportable using module-to-script api
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.
'''Items to Discuss'''
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
* [TE] Wrap-up notes
=== August 10, 2016 ===
'''Updates'''
'''Items to Discuss'''
* [TE] Python Wrapper and scripting
** All changes merged into python-wrapper branch
*** Tests pass
*** Still need to update sub-branches
** Minor fixes
*** vtkPlot3DReader upgrade bug
*** Added tuple exporting
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
**** Could be a bug in the old vistrail, but not sure
* [DK] Subworkflows and uuids
=== August 3, 2016 ===
=== August 3, 2016 ===


'''Updates'''
'''Updates'''
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]
 
** connection variables now lowercase


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]
** Connection variables now lowercase
** Can we use code as output variables?
*** E.g. "vtk_dataset_mapper.GetOutput(0)"
*** Could mutate output when output method is only supposed to be called once
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)


=== July 27, 2016 ===
=== July 27, 2016 ===
Line 577: Line 880:
* Drop support for win-64
* Drop support for win-64
* Update copyright headers to 2016
* Update copyright headers to 2016
== 2015 ==
=== December 30, 2015 ===
'''Updates'''
* Bokeh package finished
** Can output to Browser/IPython/Spreadsheet (Limited functionality in spreadsheet)
'''Items to Discuss'''
* Updating VisTrails dependencies
** Bokeh needs QWebEngine (Chromium) in Qt5 which requires PyQt5
*** Many plotting libraries only work in browsers
*** But Chromium may be a big and controversial dependency
*** Or create a browser version of the spreadsheet?
** Some users have requested Python 3
** Not backwards compatible?
** Target for VisTrails 3.0 with new interpreter?
=== December 16, 2015 ===
'''Updates'''
'''Items to Discuss'''
* Users list question
* [TE] Wrapping Bokeh
** Added bokeh property parser
*** complements numpydoc parser
*** Not all attributes are available, partially because I am using a pre-release version.
** Wrapped modules
*** bokeh.plotting finished (Backwards like matplotlib: Create plot then draw on it)
*** bokeh.charts mostly done (Missing arguments)
** QWebView mostly works (but no zooming or buttons)
*** It would be better to use external browser or QWebEngineView
** Added improvements to wrapper
*** Generalized port translations (Color/Path to native types)
*** Can use multiple docstring/property parsers simultaneously
=== December 9, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [RR] Looking into wrapper
** Will port TensorFlow
** Doing a pass on the wrapper code & doc
*** UX improvements
** Factorize high-level docstring parsers into core.wrapper? (sphinx, numpy, google)
** Problem with name vs module_name (fails sklearn tests because class_by_name() is broken)
** Looking into compressing specs, lazy registration of modules
=== December 2, 2015 ===
'''Updates'''
* [TE] Package wrapping
** Wrapped most of numpy/scipy using __all__
** Added class attribute and method access for classes (Can add to class or to separate inspector module)
** TODO: Document wrapping procedure
'''Items to Discuss'''
=== November 25, 2015 ===
'''Updates'''
* [RR] New interpreter is coming along
** Everything is a stream
** Task system, eventually work-stealing parallelism
** Streams no longer advance in lock-step
*** Allows for constructions like filter, join, sorted-merge
** depth>1 still to be tested, no current plan to allow this through the module interface
*** Through looping though, can do streams of streams
'''Items to Discuss'''
* [TE] Package wrapping
** Wrapped most numpy functions (no polynomials)
** and half of scipy classes and functions
** Added functions manually from documentation (no parseable lists?)
** Focusing on data flow constructs (no property/method access for classes)
*** Using spec diff (empty for now) and dynamic parser (stores spec in .vistrails/)
** Python class/function wrapper now stable
** Add google doc parser?
*** Is there a parser available?
=== November 18, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [TE] Package wrapping
** Numpy class wrapping works
*** Basic type is List, since most types are array_likes.
*** TODO: More classes and functions
*** Some operations are in-place and some aren't, and docstring not super clear
*** [RR] It seems only methods mutate, and most have a numpy.xx function equivalent; just manually go through the methods and remove the mutating ones?
** Class wrapper is modular
*** Docstring parser and type string parser can be customized
*** Classes can have optional attribute/method ports
**** Inspectors and attribute/method modules can be created separately
** Use PythonCalc as an example
*** Can be re-implemented as a function with a parseable docstring
*** [RR] PythonCalc is just an example, and it might actually make sense to get rid of it, or do a proper math package with scalar operations (as separate modules, no combobox)
** [RR] TensorFlow to use wrapping as well
*** Very simple (only types 'tensor' & 'variable')
=== November 11, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [TE] Package wrapping
** Wrapping numpy's classes using numpydoc (ndarray)
*** Wrapping constructor arguments, attribute getters/setters as module ports
**** Then how to access attributes afterwards?
***** Use input value port to class modules?
*** Wrap methods as modules? E.g., `ndarray.shape`.
*** Function wrapping is almost a subset of class wrapping
**** May be able to use the same parsing/execution methods
** VTK's non-getter/setter methods could be wrapped as modules
*** We could then remove extra logic in interpreter for keeping function order
** Bokeh uses autogenerated docstrings
*** We may be able to read the specification directly
* [RR] [https://github.com/VisTrails/VisTrails/pull/1138 TensorFlow package]
** Basic setup working, can execute the Mandelbrot example
** Will autogenerate the operations
=== November 4, 2015 ===
'''Updates'''
* [RR] Interpreter work requires a fix for [https://github.com/VisTrails/VisTrails/issues/1137 DB issue #1137]
* [DK] Kitware's Resonant [http://www.kitware.com/source/home/post/173]
** [https://github.com/girder/girder Girder]: data management system
** [https://github.com/Kitware/romanesco Romanesco]: execution engine, uses [http://www.celeryproject.org/ Celery] for task management
** Resonant Flow: web app for editing and executing workflows
'''Items to Discuss'''
* [TE] Added OSX Lion (10.7) VM on build machine using vagrant (seems ok with license?)
** New builds works on Lion
** Change the minimum requirement to 10.7?
* [TE] Package wrapping
* Added upgrade suggestions using 2 spec versions ([http://pastebin.com/30XUU8Gp example])
** using name edit distance to find matches
** Could use something better like port similarity for modules, and type similarity for ports
** Should write python upgrade code?
** Added spec to sklearn package
*** Can now diff spec versions and keep the spec static
*** There will be problems with downgrades and package versions
=== October 29, 2015 ===
'''Updates'''
* [TE] nightly-build-mac fixed (Needed to approve xcode license)
'''Items to Discuss'''
* [TE] matplotlib wrapper
** ported to general spec
** Diff can be used on general spec
*** Only needed minor changes
*** Fixed indexing bug that corrupted specs
*** Can now move diff tools to core/wrapper
** Add new plots?
*** New plots in mpl 1.3: eventplot
*** New plots in mpl 1.4: angle_spectrum, magnitude_spectrum, phase_spectrum', violinplot
*** No new plots in mpl 1.5 (from looking at boilerplate.py)
=== October 21, 2015 ===
'''Updates'''
* [RR] Alexis has arrived, will be working with [RR] on a more efficient interpreter
'''Items to Discuss'''
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package]
** Supporting multiple package versions
*** Check which version can be loaded [https://github.com/VisTrails/VisTrails/pull/1135 #1135]
*** We cannot show version requirements in the list of packages because old packages are loaded when importing the codepath
** Visual diff does not work well, but we can use the spec differ to see differences.
** Unifying vtk and matplotlib wrapping specs (WIP)
*** And porting matplotlibs spec differ to work on the general spec
*** Spec differ can then be used on vtk and other packages in the future
*** Will enable automatic upgrade generation
* [RR] Rework interpreter
** restore abstraction between interpreter/module code
** build looping, streaming, caching into the interpreter
** rework cache
** look into parallelism once basic functionality is there
=== October 14, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package] (adding numpydoc parser)
=== October 7, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [TE] [https://github.com/VisTrails/VisTrails/tree/fix-matplotlib-parser New matplotlib parser] (adding numpydoc parser)
** Added numpydoc attribute parser for plots.
*** TODO: Need better port spec reconciliation with call signature parser.
*** We can create a general numpydoc parser, but attribute types are unique to matplotlib.
** Needs package versioning
*** New spec will not support matplotlib < v1.4 due to changed path to axes classes.
*** Load spec version corresponding to installed matplotlib version?
*** Need version downgrades?
=== September 30, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [TE] Updating Matplotlib parser
** Matplotlib docstring parser fails on numpy docstrings
** I have [https://github.com/VisTrails/VisTrails/commit/eae8b0c16260810587e47c41cab53552cbb3cf0a added basic numpydoc parsing]
*** Only used by a few docstrings so far
*** Parsing uses many sources (class tables, signatures, ACCEPTS, method docstrings, definition parsing).
** Caswell said they were thinking about moving to traitlets, but this is not ready yet
=== September 23, 2015 ===
'''Updates'''
* [TE] VisTrails 2.2.3 released
** Also have pushed to PyPI, binstar, etc.
** Sourceforge vs. GitHub
*** should be able to host releases on GitHub
*** nightly binaries pushed to sf each night
*** old binaries?
* [TE] Re-wrapping MatPlotlib
** Keep static generation - Docstrings are brittle beween matplotlib versions
** Keep generating the executable classes - but create functions, not vistrails Modules
*** Why is there so much patching?
** Re-implemented Alternate PortSpec for InputPortSpec
*** It will now inherit specs from parent (No need to reimplement)
** Module upgrades?
'''Items to Discuss'''
=== September 16, 2015 ===
'''Updates'''
* [TE] Job Monitor tests and documentation done
** OK to leave document package in packages directory?
'''Items to Discuss'''
* [TE] Library wrapping: How to do code patching? [https://docs.google.com/drawings/d/1Sh3eOXot6v2y880jHKQrqXqAwr-0aoqYligg3z8G8MY/edit?usp=sharing Wrapping diagram]
** Insert code into module template (Old VTK code)
** Patch the library that is called (New VTK code)
** Somehow store code in spec and apply when executed?
*** Executed code needs at a minimum access to inputs, outputs, current class
*** Use standard input/output dict and obj reference that the code operates on?
*** Can this be done while keeping the execution abstract?
* Do new release now?
=== September 9, 2015 ===
'''Updates'''
'''Items to Discuss'''
* New release?
** [RR] wants reprounzip
** tej changes & doc
** job monitor & jobmixin fixes
** mongodb
** reprounzip
** warning: MplFigure type is int
** tabledata: add Reshape, DictoToTable & ListToTable, fix ListToTable with numpy arrays
** PythonSource can have same name for an input & output
** don't show spreadsheet at exit
=== September 2, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [TE] Limit autosaves? [https://github.com/VisTrails/VisTrails/pull/1126 #1126]
=== August 26, 2015 ===
'''Updates'''
* [TE] Added Job support to Parameter Explorations (Requested by Colin), and Mashups
** specify job ids, need to specify different ids for parameter explorations since they have the same version id
** how to deal with parameters passed in on command line
* [General] provenance: should be creating a new version when we execute workflow with changed parameters?
** currently, custom_params annotation stores this in provenance currently
** [RR] API doesn't record provenance if passing in parameters: http://git.io/vsAA5
'''Items to Discuss'''
=== August 19, 2015 ===
'''Updates'''
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]
* [TE] Working on Job Monitor documentation
'''Items to Discuss'''
* [RR] JobMixin and JobMonitor: stable now?
** Definitely needs more tests
* [TE] Stop testing VisTrails 2.0?
=== August 12, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]
* [RR] JobMixin and JobMonitor: stable now?
** Definitely needs more tests
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]
* [RR] Build broken on Travis; because of IPython 4 released today? ([https://travis-ci.org/VisTrails/VisTrails/builds/75279795 build 992]; [https://github.com/VisTrails/VisTrails/issues/1123 #1123])
=== August 5, 2015 ===
'''Updates'''
* [TE] Fixes to Jobs
** Could not delete jobs
** Could not run job in group
** Job not reset when calling ModuleSuspended
** Added deleting job from context menu
* [RR] Writing documentation for everything [https://github.com/VisTrails/VisTrails/pull/1105 #1105]
'''Items to Discuss'''
=== July 29, 2015 ===
'''Updates'''
* RR still looking into new interpreter thing
** Goal is to take out scheduling logic from Module so it can be split in multiple processes, and so that smarter strategies can be added in time
** This means some work on packages
** Spreadsheet can live in kernel process? Still some UI stuff that will take work (changing configuration, persistent archive's viewer, ...)
* [TE] Problems running examples [https://github.com/VisTrails/VisTrails/pull/1111 #1111]
** Testing of more examples requires additional packages on the test machines.
** Fixed faulty line-ending in PythonSource:s failing on Python 2.6.
** Test suite now testing SUDSWebServices (If web service is down, test suite will fail)
** preferences.py test failed reloading 'dialogs' package, switched to using 'URL'.
'''Items to Discuss'''
=== July 22, 2015 ===
'''Updates'''
* [TE] mailing lists back online
* [TE] Working on [https://github.com/VisTrails/VisTrails/issues/1107 #1107]
* [RR] Working on ReproZip package
'''Items to Discuss'''
* [RR] MongoDB package [https://github.com/VisTrails/VisTrails/pull/1106 #1106]
* [RR] Example for tej docs? [https://github.com/VisTrails/VisTrails/pull/1105 #1105]
=== July 15, 2015 ===
'''Updates'''
* 2.2.2 released
'''Items to Discuss'''
* RR is considering executing everything in an IPython kernel (i.e. separate Python interpreter, like the one spawned for a notebook)
** It's a separate process that we can restart without restarting the app/gui
** We can isolate the execution environment (e.g. for the server)
** We can run the whole thing remotely (if your data is elsewhere, just run VisTrails locally and the pipeline elsewhere)
** Pipeline execution no longer makes the interface hang, it just makes the kernel hang (but that's fine)
** We can use notebooks as modules (probably way nicer than the PythonSource module)
** We can run multiple kernels so long as the ports carry things that are serializable
*** meaning we can put the multithreaded-interpreter without all the hacky parts it has now
*** we can run IPython kernels in all the languages IPython supports, [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages currently 46]
=== July 8, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [TE] Buildbot github hook not working after IP address change
* [TE] No reference to VistrailsApplication ([https://github.com/VisTrails/VisTrails/issues/1103 #1103])
** [RR] Added PR [https://github.com/VisTrails/VisTrails/pull/1104 #1104]
* [TE]  Reopening VT file after saving with bundled subworkflow won't offer subworkflow upgrade ([https://github.com/VisTrails/VisTrails/issues/1102 #1102])
** allow manual delete to fix right now
** fix this on top of the use-uuid branch
* [TE]  Release VisTrails 2.2.1? ([https://github.com/VisTrails/VisTrails/blob/v2.2/CHANGELOG CHANGELOG])
** [https://github.com/VisTrails/VisTrails/pull/1093 output-modules-use-global-dir]
** [https://github.com/VisTrails/VisTrails/milestones/version%202.2 Current issues for 2.2]
=== July 1, 2015 ===
'''Updates'''
* [TE] PROV fixed
* [TE] Working on subworkflow issues
* [RR] Considering reworking the controller (log vs exception problem, retained upgrades causing interferences) and also the interpreter (IPython?)
'''Items to Discuss'''
=== June 24, 2015 ===
'''Updates'''
'''Items to Discuss'''
* UV-CDAT
* [TE] Can a cyclic workflow be valid? ([https://github.com/VisTrails/VisTrails/issues/1097 #1097])
** focus on disabling the ability to create cyclic pipelines because more things break than just this with a cyclic pipeline
* [RR] Relative paths ([https://github.com/VisTrails/VisTrails/pull/1057 #1057])
** This interacts with the new bundle; how do handle packing files inside the VT bundle?
=== June 17, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [TE] current_version and reusing existing upgrades are broken ([https://github.com/VisTrails/VisTrails/issues/1095 ticket #1095])
** Could be that export to PROV is using an unflushed upgrade pipeline?
** current_version would then be correct.
** It may work to flush the actions before exporting?
=== June 10, 2015 ===
'''Updates'''
* DAT: fixed VTK issue on Linux and Mac
* Still crashes on Windows. Need help! Reminder: this works in the VisTrails spreadsheet (QCellPresenter), although no widget get changed there during a drag
** Is it a VTK bug?
** Is it simply impossible to change widgets during the drag, should we do it a different way?
** Did I miss something that is done in VisTrails but somehow not in DAT?
** Low prio, UV-CDAT doesn't run on Windows anyway
'''Items to Discuss'''
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 Updated pull request])
* Single instance code and batch mode
=== June 3, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 pr])
** Requires fixes to command line parameters, Output modules, and batch mode
** How to test this
** Updated missing/outdated flags
** Fixed view issues when generating graphs
* [TE] batch mode
** SpreadsheetOutput not enabled in batch mode. Should we check is_running_gui instead?
** Other instance setting flags from caller
** Is graphsAsPdf replacing spreadsheetDumpPdf?
** graphsAsPdf true by Default?
** Batch mode executing by default (Not needed when generating graphs)
** Re-added workflowInfo as withWorkflowInfo for writing graph and xml workflow
** [DK] batch mode should be outputting to files or stdout, shouldn't always trigger SpreadsheetMode
** execute flag, maybe make execution the default and allow a "--no-execute" if you only want to capture graphs, for example
=== May 27, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [vistrails-users] API question
**  http://lists.vistrails.org/pipermail/vistrails-users/2015-May/000357.html
* Status of DAT
** Can add VCS plot to DAT but need to work on configuration windows
** Documentation for graphics templates for UV-CDAT/vcs?
* Qt support?
=== May 20, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [RR] Reviving the DAT, integrating scripting & porting UV-CDAT!
** On [https://github.com/VisTrails/DAT GitHub] ([https://github.com/VisTrails/DAT/issues issues])
** Merging 2 years of development taking longer than expected, but getting there. The plan is to get the patches in VisTrails and never fork again, we never want to get in UV-CDAT's situation (and don't need to).
** VTK cell works fine on Linux but there was flickering on Mac & Windows before; still issues on Mac (Windows status unknown)
** Can get a VCS plot soon (but will need VTK cell fix)
** Integrate in UV-CDAT's build system (so we have cdms, VCS, ...) -> RR can do this, low priority
** How do we integrate scripting?
*** We want to be able to seamlessly make changes to a plot by changing Python code
*** Define new plots by entering Python code without writing modules/packages?
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look
* [RR] Subworkflow issues
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen
=== May 13, 2015 ===
'''Updates'''
* [RR] Test skipping whitelist ([https://github.com/VisTrails/VisTrails/pull/1069 #1069]) -- low priority
* [RR] Custom matplotlib modules can't be compatible with both 2.1 and 2.2 ([https://github.com/VisTrails/VisTrails/issues/1067 #1067]); should be fixed for ALPS ([https://github.com/VisTrails/VisTrails/issues/1070 #1070])
'''Items to Discuss'''
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look
* [RR] Subworkflow issues
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen
=== May 7, 2015 ===
'''Updates'''
* [RR] Export/import workflow to Python working!
'''Items to Discuss'''
* [Claudio] UV-CDAT
** The UV-CDAT project is the biggest user base of VisTrails
** VisTrails package management provides a lot of friction towards people plugging in their code
** Need to make it easy to integrate your random Python scripts in the system without having to deal with all the boilerplate, at least in the first step
** [RR] argues that modules are still good; UV-CDAT shouldn't move towards a purely script-based backend
** [RR] export/import with Python could reduce a lot of that friction by allowing 1) to edit workflow as Python 2) to open up boxes automatically if needed code doesn't match actual modules
** ...
=== April 29, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [RR] <strike>Abstractions</strike> subworkflows status ([https://github.com/VisTrails/VisTrails/labels/subworkflows tickets])
* [RR] matplotlib compatibility (2.1 & 2.2), [https://github.com/VisTrails/VisTrails/issues/1067 #1067]
** RR to try and fix ALPS matplotlib modules
=== April 22, 2015 ===
'''Updates'''
* 2.2.0 has been released!
* Windows issue (via email from Ryan)
** issue with manifest file (may be a new file in VTK6?)
** Tommy has regenerated new Windows builds
* Binaries, pypi, and conda released
* [RR] Export as script
* Python sources using VTK need to switch to SetInputData (users should be aware of this)
=== April 15, 2015 ===
'''Updates'''
* Ready for 2.2.0 (apart from binary/deps issues)
** Missing some libs (scikit-learn, tej, tdparser, SQLAlchemy+connectors
** Windows: runvistrails.py is no longer used, so the PATH is wrong
** Windows: pip is broken, but it probably wouldn't work anyway because of permission issues (disable this?)
* Queries, upgrades and getPipeline() usage ([https://github.com/VisTrails/VisTrails/issues/1054 #1054])
** Getting a pipeline with getPipeline() is not safe: it might return an invalid pipeline
** This is used in many places throughout the code, like queries
** Upgrading would require going through the controller, but that creates new actions
* [TC] Avoid copying a module's output if it's used as input by exactly one downstream module ([https://github.com/VisTrails/VisTrails/issues/1060 #1060]) (useful for big numpy arrays you can update in-place)
=== April 8, 2015 ===
'''Updates'''
* [DK] Merged RR's changes for output modules (1012 and 1013)
** RR will merge remaining changes, then create v2.2 branch!
'''Items to Discuss'''
* BNL need numpy array to VTK image
** Looks like VTK has some helpers for this
** We will help if issues arise
** Will contribute back to VisTrails package
* Upgrade issue: [https://github.com/VisTrails/VisTrails/issues/1017 #1017]
** Automatic upgrades should happen between versions of provided upgrades
** Our existing upgrades work around this so it doesn't need to be 2.2.0
* Corner-case VTK modules
** No longer need VTKCell input port, so don't interfere with registry and API anymore
** Still work weirdly, people probably shouldn't use them
** But we have lots of clunky modules since we wrap the whole of VTK; some people might rely on this and know how to use them, let's keep them anyway
** Ready for 2.2.0
=== April 1, 2015 ===
'''Updates'''
* [RR] UV-CDAT: bugfixing for 2.2, long-term plans: implement scripting import/export in VisTrails, port to UV-CDAT
* Possibly, try to move to regular VisTrails to use new features
'''Items to Discuss'''
* 2.2 release: nothing much is pending anymore, release next week?
** [TE] vtkExporter classes [https://github.com/VisTrails/VisTrails/issues/1032 #1032]
** [RR] Hiding upgrades in version tree might make it (or might be 2.2.1 so we can test it out) [https://github.com/VisTrails/VisTrails/issues/949 #949]
** Output modules changes to go in
* add note to documentation about order of parameters in VTK
* add issue about exporter upgrades if not already there
=== March 25, 2015 ===
'''Updates'''
'''Items to Discuss'''
* 2.2 release
** Checklist on Github: https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist
** Review output modules
** RR has a couple more issues to fix
** Ready to go -- sign app package for OSX? [https://github.com/VisTrails/VisTrails/issues/984 #984] We need access to the Apple Membership team
=== March 18, 2015 ===
'''Updates'''
* [TE] VTK6 works
'''Items to Discuss'''
* [RR] Release v2.1.5 with backported tabledata?
** MTA example needs updated tabledata (for JoinTables)
** Google Maps package still not available
* [RR] Work torward v2.2.0?
** changes:
*** new persistence
*** API changes
*** output module changes (upgrades?), maintain cells but try to upgrade
*** not wrapping stuff
*** VTK6? yes
*** JobSubmission stuff?
*** relabeling for upgrades [https://github.com/VisTrails/VisTrails/issues/949 #949]
** makes sense, needs the tree view code to be updated, check selection
** See [https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist 2.2 checklist]
* Discussion of [https://github.com/VisTrails/VisTrails/issues/1016 #1016]
** plumbing between outputs and output modes, how to define a mode that works for many outputs without writing for each output?
=== March 11, 2015 ===
'''Updates'''
* [RR] persistent_archive done; merge? ([https://github.com/VisTrails/VisTrails/pull/755 #755])
** note about the focus events for widgets
** TE be aware of file_archive for future binaries that include persistent_archive
* [TE] [https://github.com/VisTrails/VisTrails/pull/998 New VTK package] finished
'''Items to Discuss'''
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails? (see last week; decision needed)
** RR email to JF about this
** yes; email sent to Github
* [RR] Switching order of output ports ([https://github.com/VisTrails/VisTrails/pull/1006 #1006])
** added port specs are sorted at a separate spot (Module.*_port_specs properties) than those in the registry (which are sorted in the registry), but those two lists are just combined without respect to sort keys
** need to determine whether the two lists should be merged or remain distinct
** should make sure that order of input ports and output ports makes parallel connections for things with same order
** DK suggests breaking backward compatibility here: workflows still run, can fix easily if a problem in an existing package.
* [RR] Question about output modes ([https://github.com/VisTrails/VisTrails/pull/1007 #1007]), how to integrate in API ([https://github.com/VisTrails/VisTrails/pull/24 #24])
** Should ImageFileMode be removed? ("image" is not a mode, "file" is)
** ImageOutput missing?
** Feel free to change how formats works
* [TE] Test suite segfaults on Fedora 17 virtualbox. Install newer version? (Support ended in 2013)
=== March 4, 2015 ===
'''Updates'''
* [TE] VTK wrapper
** Works on VTK 5.10
** Still need to test VTK 6
** New general wrappers for python functions and classes
'''Items to Discuss'''
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails?
** When [http://i.imgur.com/UTY2qfg.png filing a pull request] from UV-CDAT/VisTrails, VisTrails/VisTrails is selected by default
** People keep forgetting to change the default ([https://github.com/VisTrails/VisTrails/pull/956 #956] [https://github.com/VisTrails/VisTrails/pull/968 #968] [https://github.com/VisTrails/VisTrails/pull/999 #999] [https://github.com/VisTrails/VisTrails/pull/1000 #1000] [https://github.com/VisTrails/VisTrails/pull/1003 #1003] [https://github.com/VisTrails/VisTrails/pull/1004 #1004] [https://github.com/VisTrails/VisTrails/pull/1005 #1005])
** Only way to change that is to not have it marked as a fork of VisTrails/VisTrails
** Github staff [https://gist.github.com/remram44/b364c24ef1e462ce6e2f can make that change for us]; should we do it?
** Juliana: comments on this? visibility vs. convenience/annoyance for developers
* [RR] What about [https://github.com/VisTrails/VisTrails/issues/991 #991]?
=== February 25, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [TE] vtkviewcell for infovis support, can we unify with VTKCell?
** need to test this
* [TE] vtk wrapping
** Mostly finished
** VTK 5.10 produce incorrect results with old wrapping
*** Old wrapper is based mostly on VTK 4
*** Most vtk_examples affected
*** [http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput]
*** should be able to upgrade from SetInput to SetInputData (need to drop GetOutput and replace with self ports)
*** can we change vtkInstance to just return self and not wrap things
** terminator example not working under 5.8?
* How does VTK wrapping fit into general wrapping framework?
* [RR] new persistence package
=== February 18, 2015 ===
'''Updates'''
* [RR] New VisTrails API and IPython integration ([https://github.com/VisTrails/VisTrails/pull/24 #24])
'''Items to Discuss'''
* [TE] VTK wrapping
** [https://gist.github.com/rexissimus/16b74693545b5e7c1035 Benchmarking vtk package]
*** Old: 24.7 seconds
*** New: 10.5 seconds (Except first time that adds 8 sec)
**** The parsing that calls is_abstract (that tries to instanciate all vtk classes) is now only run the first time.
**** get_items_from_sigstring takes 2 seconds, maybe we can use a lookup dict for already computed sigstrings?
** Now using a general python function wrapper
*** VTK classes are wrapped into python function that does not depend on vistrails
*** VTK functions can be executed without vistrails
*** The spec maps functions into vistrails modules, but can also describe wrapping
*** A general python function wrapper that supports
**** kwarg inputs
**** single, list, dict outputs
**** callback for progress reporting
**** temporary file generator for using FilePool
**** optional output generation
*** Creating specs:
***** Create spec by hand
***** Auto-create spec outline (TODO) and manually finish it
***** Dynamically create spec (VTK)
***** Implement documentation wrappers (Can use scikitlearn wrapper to wrap numpydoc) (TODO)
***** Classes as bad functions needs to be wrapped in new functions before they are wrapped. This is different for each package.
****** Classes is hard: Like VTK, and matplotlib. Scikit-learn does still not wrap classes
***** Spec diffing and patching could be done using code from matplotlib.
** Still needs upgrades from old VTK package
*** Is it possible to dynamically wrap functions, e.g, you see a SetFunc and just remove the 'Set' prefix. Or do you need to create a complete mapping?
=== February 11, 2015 ===
'''Updates'''
* Update from Friday's meeting
** discussed VisTrails internals
** discussed wrapping
*** xml discussion, hard to modify because tied to db code
*** TE has made it possible to add the schema-defined attributes to the intermediate representation
*** higher-level operations on the port specs
** make sure the simple case works
*** [JF] take a simple package with documentation and figure out what the base case for wrapping is
'''Items to Discuss'''
* [TE] VTK wrapping
** Dynamic loading works
** Reading XML is fast enough, but [https://gist.github.com/rexissimus/8272d3c776ccfa115d82 unserializing data is slow]
** Working on patterns for patching
** Matplotlib has many advanced patterns like argument ordering, nested arguments, alternateSpecs, output types.
*** Having all this in a general wrapper might confuse users?
** [RR] Delay module (except for identifiers) until you need it---e.g. don't deal with port specs, etc. until necessary
* Scripting Support [https://github.com/VisTrails/VisTrails/pull/950 #950]
** [RR] Issue with getting code from modules
** Design a simple solution
** [JF] Couldn't you use modules as black boxes without conversion, just to call into modules/subworkflows easily from e.g. IPython?
*** [RR] This is a job for the API, and a very separate use case. See [https://github.com/VisTrails/VisTrails/pull/24 #24]
=== February 4, 2015 ===
'''Updates'''
'''Items to Discuss'''
* Wrapping
** Format to use? Currently XML (like current matplotlib)
*** JSON and YAML have simple "to python dictionary" methods
*** But don't stream
*** YAML a lot easier for humans
** [DK] vtk-new-package also changes parameter names, creates enumerations
*** intermediate schema needs to be extensible
*** packages will want to store there specific infos for compute() method generation
*** also might have specs-altering info, like matplotlib's alternateSpec
** representation to code , registry already has schema for some aspects
** [RR] We might want to see if Module subclasses can be created lazily
*** no need to create all the classes just to register them in the registry and never actually use most of them
*** future effort
* [RR] Where should VisTrails packages live?
** tej installs as 'vistrailspkg.tej', TE installed it as 'userpackages.tej'
** Currently, standard packages are 'vistrails.packages.', user packages are 'userpackages.' and packages loaded through pkg_resources might be anything
** [RR] Use 'vistrailspkg.' everywhere?
** Long-term effort to simplify package distribution/installation (and have VisTrails get them automatically?)
=== January 28, 2015 ===
'''Updates'''
* T. Caswell to come visit on Fri 6 to discuss wrapping work
'''Items to Discuss'''
* [TE] New VTK wrapping
** Current code by DK seems a good deal faster
** Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code
** RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower
** The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own
** TC has its own code at [https://github.com/Nikea/VTTools/blob/master/vttools/wrap_lib.py github:VTTools] which parses numpy docstrings and generates modules, doesn't yet handle classes or persist anything
* Web crawler
** Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"
** RR would rather have the crawling be a job as far as VisTrails and tej are concerned
** The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize
*** Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached
=== January 21, 2015 ===
'''Updates'''
'''Items to Discuss'''
* [RR] Unified wrapping method discussion [https://github.com/VisTrails/VisTrails/issues/991 #991]
** TE to work on reusable method with intermediate representation, starting with VTK
* [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)
** AM's examples are enough
* [AM] scikit-learn package is done, merge it in? [https://github.com/VisTrails/VisTrails/pull/955#issuecomment-70286788 #955]
** RR will merge
* [RR] What should copyright headers say? [https://github.com/VisTrails/VisTrails/pull/994#issuecomment-70173935 #994]
** Let's keep everything in there: Utah/Poly/NYU
=== January 14, 2015 ===
'''Updates'''
* [TE] Working on classifier
* [RR] Scripting integration, work in progress
'''Items to Discuss'''
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])
** Let's talk next week, [AM] and [DK] are not here
=== January 7, 2015 ===
'''Updates'''
'''Items to Discuss'''
* make sure that we address critical issues, questions, and pending review branches in a timely manner
* scripting support
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow
** [RR] can translate from workflow to script, working on script to workflow
** will work for parameter value changes, structural changes require changes to the annotations
** need to publish best practices here
** would be cool to do looping in scripts (easier interface than with workflows)
* notebook support (convert form notebook to workflow)
** RR will sync with FC on this
* Issue with console in built-from-scratch
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3
** binary has old version of iPython -> 1.0.0, should we update?
* [TC] automated wrapping of numpy and scipy
** discovered a bunch of malformed documentation in numpy and scipy
** has github repo for vistrails tools
** example modules wrap a bunch of R stuff (not baked in, just how things are)
** will be pushing wrapping logic up
** port names forbidden (window and domain)
** have an import hook to get from yaml directly to VisTrails Modules
** should work for any python modules with well-formed numpy docstrings.
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)
* [TE] build and build scripts
** completely automatic, buildbot
** need to set the build machines for the environment we want for the binary
** would virtualenv work here?
** [TC] anaconda can pin versions, potential path to test different configurations
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07
* package issues (see Remi's message)
* [TE] Scope of tej
** Support single ssh commands?
** Queue can be used as a remote machine (crawler is using queue.call*)
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07


== Older meetings ==
== Older meetings ==
* [[Development/2015]]
* [[Development/2014]]
* [[Development/2014]]
* [[Development/2013]]
* [[Development/2013]]

Latest revision as of 15:34, 30 October 2017

See also the Github wiki

2017

October 30, 2017

Updates

  • [DK] PyQt5
  • [RR] Running Python3 code

Items to Discuss

  • Installation:
    • Use conda? (macOS hack to have .app bundle load conda-installed vistrails)
    • Update conda recipe (pyqt4 restriction for now)
  • Interpreter
  • Release
  • Integration
    • hard to build same type of GUI in other apps that integrate core VisTrails (tend to use just the API)
    • independent pieces of the GUI could be reused if they were more independent.

September 6, 2017

Updates

  • [RR] Fixes for scikit-learn, tables, usagestats

Items to Discuss

  • Stripped-down version of VisTrails to showcase new features?
    • include meta-vistrail, bundles
    • hide mashups, extra widgets, latex mode, etc.
  • Interpreter

August 16, 2017

Updates

  • [DK] meta-vistrail
    • Working actions: normal action in vistrail, pruning version tree, tagging operations, and change
    • Interface

Items to Discuss

  • Notebook
  • Interpreter
  • Web visualization packages (e.g. bokeh)
  • Pipeline templates

August 2, 2017

Updates

  • [DK] meta-vistrail work
    • Extend Vistrail to MetaVistrail to support meta-actions
    • Vistrail becomes less tied to workflows so it can also supporting versioning vistrails

Items to Discuss

  • Interpreter
    • Using dask?
    • Streaming an issue
  • Notebooks
    • What is an output of a module? (need to expose a specific output port or the entire module)
    • Unifies some of the work from the script-workflow code

June 28, 2017

Updates

  • [DK] Testing UUID/SQLAlchemy branch
  • [DK] Working on meta-versions
    • would be nice to have pipeline and vistrail separate
    • [RR] Then do we only have meta-actions, any change to the pipeline is a meta-action that adds a version that does the change?
    • [DK] Yes?
  • [RR] Notebook interface (Vizier) making good progress, figuring out cell outputs

Items to Discuss

June 7, 2017

Updates

  • [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs

Items to Discuss

  • [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)
    • Can designate in module description an output port
    • Value has to be serializable
    • Should use the OutputModes work to handle multiple modes, configuration, selection, ...
  • [RR] Is VisTrails with SQL usable concurrently?
    • [DK] Probably not, demo from [TE] exists

May 10, 2017

Items to Discuss

  • Identifiers
  • Bundles & DB Code
  • Interpreter
  • Merge / Versioning Version Trees
  • Python3
  • Workings of List
  • Packages: Wrappers + new Versions (VTK7)
  • Script Conversion
  • Intermediate Outputs: reuse output module work?
    • would be useful for both workflows (run to a particular checkpoint)
    • and useful for notebook interfaces
  • SQL backend: does it allow for concurrent access?
    • Yes, real-time collaboration work for concurrent single vistrail access
    • Should be able to access different vistrails at once without issues

April 19, 2017

Updates

Items to Discuss

April 5, 2017

Updates

Items to Discuss

March 8, 2017

Updates

  • [RR] Added a looping indicator on modules #1217
    • This makes it easier to understand what's looping, no need to follow edges and count
  • [RR] Refactoring CLTools for integration in Docker package
  • [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)

Items to Discuss

  • [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers #1216
    • Can we move this up to the renderer modules?
    • Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)

February 24, 2017

Updates

  • [RR] Rich-text module documentation (#1210)
    • Went over packages and fixed format; maybe we should have a test to validate RST?
  • [RR] Single-instance shutdown with --remote-shutdown (#1204)
  • [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment

Items to Discuss

  • [RR] There are issues with list-depth
    • Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)
    • Issue with looping groups

January 25, 2017

Updates


Items to Discuss

  • Scaling VisTrails
  • sql-alchemy, uuid, bundle branches/merges

January 11, 2017

Updates

  • [RR] New VisTrails version in Debian [1]

Items to Discuss

  • [RR] Collaboration with school of medicine to port HiC-bench to VisTrails
    • Have homemade workflow system based on R scripts and symlinks
      • Each steps has multiple sets of parameters
      • Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step
      • Gets all results in directory structure step3-parameters/step2-parameters/step1-parameters/files
    • VisTrails improves provenance capture and makes editing pipelines easier
      • Need better support for aggregation (groupby operation might be good enough; pandas?)

2016

December 14, 2016

Updates

  • [RR] Single-instance code: over-engineered, but will work
    • (Have to avoid race conditions on startup, and OSes offer very different locking primitives)

Items to Discuss

  • Email from Matt Dirks: combobox should work, ask for more code?
  • Email from marin.nl:
    • looks like they would benefit from new interpreter
    • how to store their consolidated provenance when they exchange .vt files around?

November 30, 2016

Updates

Items to Discuss

  • [RR] Working on single instance code for MARIN (#1205)
    • Remote shutdown ready (#1204)
    • Still improving it
    • Want to return something more structured to the client (success/fail code, stdout, stderr)
    • Should be usable without Qt
  • [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason
  • [DK] Working on batch execution using console_mode
  • [DK] Settings aliased parameters when using core.api?

November 16, 2016

Updates

  • [RR] Issue #1200 from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)

Items to Discuss

  • Email from Pieter: executing pipelines with parameters from command-line using single-instance
    • Seems to work, main instance gets full command-line and alters configuration temporarily
    • Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful
    • Actions:
      • Can add switch to kill main instance
      • Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails
      • Add client/server code to core that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like gui.application.VistrailsApplicationSingleton#noninteractiveMode())?

November 2, 2016

Updates

  • New version 2.2-2 of Debian package, still wrong copyright, and broken bundle installation
    • Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch
  • Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)

Items to Discuss

October 5, 2016

Items to Discuss

  • working toward a 3.0 release; project board
    • unique ids
    • bundles
    • GUI fixes
    • wrapping updates
    • python3
    • interpreter?
  • command-line input parameters to provenance
    • [DK] we don't create a persisted workflow from those input params, i think
    • [DK] these are already defined as aliases so we treat them differently form a normal input param
    • [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases
    • Do the vistrails variables values get stored in provenance? (what if values change?)

September 14, 2016

Updates

Items to Discuss

August 31, 2016

Updates

  • [RR] Went over vgc.poly.edu/vistrails.org config:
    • Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
    • Fixed usersguide scripts (won't disappear again)
    • Moved some websites over to HTTPS
    • Websites
  • [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs (currently there) and users' guide under the same RTD "project"
  • [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that

Items to Discuss

  • [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
    • Get rid of more services? (svn, trac, ...)
  • [RR] Notebook execution works. How do we support more languages (than Python)?
    • Need: detection, metadata reader, code to input values
  • [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?

August 17, 2016

Updates

  • [TE] Fixed parameter order upgrade bug that breaks VTK examples.
  • [TE] Matplotlib and sklearn are exportable using module-to-script api
  • [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.

Items to Discuss

  • Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
  • Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
  • [TE] Wrap-up notes

August 10, 2016

Updates

Items to Discuss

  • [TE] Python Wrapper and scripting
    • All changes merged into python-wrapper branch
      • Tests pass
      • Still need to update sub-branches
    • Minor fixes
      • vtkPlot3DReader upgrade bug
      • Added tuple exporting
    • Most of vtk examples can be exported example
      • Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
        • Could be a bug in the old vistrail, but not sure
  • [DK] Subworkflows and uuids

August 3, 2016

Updates


Items to Discuss

  • [TE] Wrapper patches now stored as python code
    • Connection variables now lowercase
    • Can we use code as output variables?
      • E.g. "vtk_dataset_mapper.GetOutput(0)"
      • Could mutate output when output method is only supposed to be called once
      • Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)

July 27, 2016

Updates

  • [DK] DB bundle serialization implemented

Items to Discuss 

   def vtkInstance():
       self.vtkInstance = self

July 20, 2016

Updates

Items to Discuss

  • [TE] New patch method for python-wrapper example
    • A patch is a string template with self/input/output variables for patching a class method
    • Patches are stored in xml specification
    • Patches are assigned to methods in ClassSpec's
    • Used both for execution in vistrails and when exporting as script
      • Executed with "exec" when run in vistrails
      • Turns patches into natural calls in script
    • Wrapping VTK library no longer needed
      • Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)
    • Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)
    • Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)
    • TODO's
      • Add to export_as_script
      • Patch Modules that represent Python functions as well
  • [vistrails-users] Read data from SQL example?

July 13, 2016

Updates

  • [DK] bundles on sql-alchemy branch
    • Bundle, BundleMapping, and BundleSerializer
    • New BaseSerializers to deal with versions of directory, zip, db serializers
    • Can register the BundleSerializers with the base serializers
    • Translation between bundle versions
    • Bundle.add_object "just works" if an appropriate mapping was registered

Items to Discuss

  • [TE] Simplified running module using API for scripting example
    • API Module can be executed using compute method
      • Takes list of outputs to compute + inputs as keyword arguments
  • Status of general cell synchronization (vistrails-users)

July 6, 2016

Updates

  • [DK] Working on bundles

Items to Discuss

  • [TE] Using vistrails API in scripting example 1 (API with VTK) example 2 (DownloadFile only)
    • Call module in script using vistrails API
    • Can be used when no export method is defined.
  • [TE] Server news bug Critical?
    • [RR] No, unreleased
  • [DK] New bundle organization
    • BundleMapping contains individual mappings from raw objects to BundleObj
    • Must be separate in order to have in-memory, separate from serializer settings

June 29, 2016

Updates

Items to Discuss

  • [TE] Export VTK as python script
    • Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) example
    • Preludes (imports) still need improvements
    • Works with vtkInteractionHandler (clipping plane)
    • Resulting clipping plane is incorrect? (supposed to hide one side of the plane)
    • vtkRendererOutput Uses VTKRenderWindow as output
      • Should support different output modes?
      • Does not work with multiple outputs (Next cell is displayed when current is closed)
      • Uses wrapped vtkInstance (as vvtk)
        • Create new patch system with code that can be used both in vistrails and in a python script?

June 22, 2016

Updates

Items to Discuss

  • [TE] Fix for upgraded analogies (PR #1177)
    • Uses module location to find module remap (The best we can do right now?)
    • Also fixes remapped port names for connections to existing modules
    • May still fail on complex upgrades like adding a function on an upgraded module that has a different port name
  • [TE] Working on exporting wrapped libraries (VTK)

June 15, 2016

Updates

  • [TE] Added depth logic to python script exporter (full commit)
    • Fully implements connection merging and list looping

Items to Discuss

  • [TE] Making analogy work on upgraded workflows (#1175)
    • Analogy only works We need to use the non-upgraded action chain
    • But pipelines need to be valid (matcher uses module portspecs)
    • Tried to get upgrade remap from upgrade action, but this seems impossible
    • Use module location instead?
      • Location should be identical for one-to-one module upgrades

June 1, 2016

Updates

  • VisTrails 2.2.4 announced (finally)
    • 5 survey replies
    • 47 downloads last week

Items to Discuss

  • [TE] Export VTK as script (python-wrapper)
    • to_python_script
      • needs the pipeline (or at least the connections)
        • Should not generate code for any possible VTK method (exporter detects which ports is used in the source)
        • VTK needs to know connection order
        • Adding functions directly to the script results in nicer code
    • Add support for multiple input connections
    • Add support for list looping

May 25, 2016

Updates

  • Mailing-list fixed
  • Getting stats from the "server news" mechanism (via parser)

Items to Discuss

  • [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)
  • [TE] Union ports
    • Show union ports in pipeline (DEMO)

May 18, 2016

Items to Discuss

  • [TE] mailing list still down - Julio working on it
  • [TE] Improved union port
    • New schema 1.0.5 adds portitem.union string attribute
    • Works with pipeline view, parameter exploration view, and mashup view
    • Should work well for matplotlib
    • Use with VTK ports?
      • It uses [item_1,item_2] style ports usually with different tuple length.
  • [TE] Remove emacs dependency from db generate #1173

May 11, 2016

Items to Discuss

  • [TE] Union type #1172
    • GUI-based solution Branch union-port Demo?
      • Shows single port for all ports with same sort_key
      • Select specific type when adding/showing
      • Only one type should be visible on the module at any time
      • Making tuple parameters union types is probably too complicated
      • Should work with alternate specs like matplotlib (if sort_key is defined correctly)
  • [TE] Feature suggestion: Let connections connect to functions/parameters
    • Show functions and parameters as ports and toggle port visibility directly for functions and parameters
    • Use separate views for ports and functions (like before)
      • Allow rearranging of functions
    • Pros:
      • Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
        • We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.
        • Makes List module simpler
      • Connect to parameter in tuple
      • Functions that are visible on the module can connect to an actual function, and not just to the first one it finds.
      • Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.
    • Cons:
      • Interface more complicated and crowded
      • More difficult for users to grasp?
      • No immediate use case
      • Makes no union type for tuple parameters a bigger problem.
  • [TE] Publish user survey?
    • No response from vistrails release yet
    • Not announced on vistrails-users yet.

May 4, 2016

Updates

  • VisTrails 2.2.4 released

Items to Discuss

April 27, 2016

Updates

Items to Discuss

  • [TE] Wrapping Bokeh
    • New method type "operation"
      • Transforms Figure->Glyph->Output into Glyph->Figure->Output
        • Glyph sets function name and arguments as output
          • Not serializable, but should always be computed together with the subject of the operation.
        • Figure applies Glyph operation on itself
      • Useful for matplotlib?
        • matplotlib currently find actors by searching upstream?
        • But this is more difficult since matplotlib is not function-based.
    • New "union" port type
      • Turns "x(float), x(list)" into "x (float, list)" and uses first set value
        • Uses only first value in tuple
        • Bokeh have many properties that support (list, string, Float)
      • Does not work with depth=1 types (e.g. Float of depth 1)
        • Turn Depth 1 types with known type into lists?
  • [TE] New bugfix release
    • Finalize survey

April 20, 2016

Updates

  • Merged upgrade fixes
    • Now has infinite loop test
    • loop test now correctly loops 5 times
      • remap_module used wrong (current) package version when creating ModuleDescriptor
  • Fixed multi-step group upgrades (PR: #1168)

Items to Discuss

  • New bugfix release
    • Merge usagestats
    • Need to finalize survey

April 13, 2016

Updates

Items to Discuss

  • [TE] Chaining upgrades done (#1164)
    • Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?
  • [TE] Iterating handle_invalid_pipeline (#1165)
    • Not based on chain-upgrades-loop
    • Cherry-pick test to preserve source?
    • Need maxPipelineFixAttempts option?
      • The code is now smarter about recursion, so max attempts will "probably" never be reached.
    • Need debug messages?
      • Would need to be moved into handle_invalid_pipeline

April 6, 2016

Updates

  • [TE] Query View improvements
    • Done except for displaying "Version Info" and #1164
  • [RR] Working on finishing new interpreter (finally!)

Items to Discuss

  • [TE] Chaining upgrades (#1164)
    • Try latest upgrade first, then previous.
    • If all fail, select latest upgrade?

March 30, 2016

Updates

Items to Discuss

  • [TE] Query View improvements
    • Unified do_version_switch and get_upgrades as validate_version()
    • Problems with query view using its own controller
      • Fixed module info not being displayed for query results
      • Still no version info for query results
      • Still an issue with version query view not being updated

March 23, 2016

Updates

Items to Discuss

  • [RR] Usage reporting almost ready to go
    • Need to record whether specific features are used: paramexpore, mashup, query
      • [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation
    • How to record size of vistrail?
      • [DK] IdScope already walks over vistrail, use that
      • but total number of modules and versions doesn't give an idea of average size of individual pipelines

March 16, 2016

Updates

Items to Discuss

  • [TE] Upgrades in Query Mode
    • Now using code similar to do_version_switch
      • Uses handle_invalid_pipeline
      • Differences: Not using current_pipeline, flushes changes
      • New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)
      • Integrate to avoid repeated code?
      • Use this for query mode?
  • [TE] VisTrails 2.2.4 ready for release?
    • tej package updated. Others?
  • [JF] upgrades and the version tree
  • [JF] provenance in Jupyter
  • [DK] bundles: #1092

March 9, 2016

Updates

Items to Discuss

  • [TE] Fixing search mode
    • String query now matches all by default
      • This caused pipeline queries to never match
    • String query can search: User/Notes/Tagname/Time/Module name
      • Should we search module names by default? (Disabled by default because it is expensive)
      • Fixed notes search
    • '>' parameter queries in documentation but has been removed
    • Fixed module matches for upgraded pipelines (By not upgrading pipeline)
      • Upgrades still a problem
      • Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)
      • [JF] Allow search on past tags
  • [TE] Create new bugfix release
    • Finish querying
    • Path issue on windows
  • [DK] Look into sqlalchemy and new bundle branches

March 2, 2016

Updates

  • [TE] Fixed cache bug with port specs
  • [TE] VisTrails Next
    • Most packages now work
    • No qgis for Python 3

Items to Discuss

    • Always dereference symlinks? #1146

February 23, 2016

Updates

  • [TE] Fixed PythonSource 100% CPU bug
  • [TE] VisTrails Next
    • Updated NOAA Web Service examples
      • Now uses REST service, tabledata, and Google Maps
      • Web Service replaced with DownloadFile

Items to Discuss

  • [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.
    • New 2.2 release soon?
  • [RR] Anonymous usage reports
    • Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features #1154
  • [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?
  • noWorkflow - provenance for python scripts
    • Recent work on tracking dependencies
  • Also integrates ReproZip with Jupyter
    • mybinder
      • Use with ReproZip to automatically build Jupyter notebook interfaces
  • Daves comments on Jupyter integration
    • It is possible to identify cell order on the server side
    • Not trivial to track order of cells on client side
  • How does Galaxy workflows relate to VisTrails?
    • Packaging in Galaxy is hard
    • Galaxy for biomedical workflows, difficulty making it general
  • Action items
    • How difficult to track cell dependencies in Jupyter notebooks?
    • Use noWorkflow to track cell dependencies?
    • How difficult to modify mybinder to use ReproZip packages?

February 17, 2016

Updates

Items to Discuss

  • [TE] Fixed Group memory usage
    • Modules in groups are supposed to be freed when done executing
      • Prevented by logger that stored references to the modules (fixed, now using id(module))
      • Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)
  • [TE] VisTrails Next
    • Removed deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • PythonSources will be upgraded
      • Fixes for Python < 3
      • fixes for VTK < 6
    • TODOs
      • Remove RemoteQ
        • Port Hadoop to tej (Already have pbs)
      • Merge package wrapper
      • Finish SQLAlchemy branch
      • Finish new bundle branch
    • [JF] Will contact Jeff and Matthias about moving to Python 3
  • [JF] Query by Example is broken
  • [JF] VisTrails Data Cleaning Project
    • Use VisTrails to process and explore task-specific data while keeping provenance.
    • Remi's Idea
      • Use VisTrails as the backend in a Jupyter notebook
      • Overcome problems with executing cells in random order
        • Interesting research topic, but little interest has yet been noted
      • Interesting to study how such a notebook would be used (Is there notebook usage data available?)
      • Impose restrictions to notebook cell boundaries
      • It will only work for functional operations
      • Can we fix, or detect, non-functional (mutating) operations?
      • Analyse code and data flow and see if it mutates data in unexpected ways?
      • Mutable operations can be created in many ways in Python
  • Next meeting Tuesday 11:30

February 10, 2016

Updates

Items to Discuss

  • VisTrails Next
    • Test suite passes
      • Python 3 less deterministic, module upgrade order random.
    • [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...
    • All of our packages should run fine :)
    • VTK thumbnail comparison works
      • Rendering bug on some systems
        • QVTKCell now inherits QGLWidget
      • Old thumbnails had wrong height due to spreadsheetwindow being too small
        • Fixed by regenerating thumbnails
      • Silenced deprecation messages when generating modules
      • Many new classes in VTK 7
    • Remove deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • Will break existing PythonSources
        • Try to upgrade?
      • Old-style packages
        • Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
          • Read info.py without importing package?
      • Fixes for Python < 3
      • Fixes for Qt < 5
      • fixes for VTK < 7
      • Remove self port?

February 3, 2016

Updates

Items to Discuss

  • VisTrails Next
    • All tests but thumbnail comparison passes
    • Still issues with gui and more packages needs to be tested
    • PythonSource and MplSource upgraded with lib2to3
    • Keep contrib packages? Update?
    • VTK 7 is released
      • Supports Python 3
      • Will hopefully get into conda and others soon
      • Deprecates a bunch of modules
  • Travis builder for VisTrails Next
    • Runs!
    • Uses conda
      • Many levels of requirements
        • apt packages
        • conda packages
        • python packages in requirements.txt
        • python packages in setup.py
      • Can probably drop conda. Default builder does not have Qt5, but beta builder do!
      • But useful if we want to use conda in our own build process.
    • Lots of errors I do not get locally?
  • User Survey
    • Have you used any alternatives to vistrails?
      • Relevant? More options?
    • How often do you use any of these particular features in VisTrails?
      • Add/Remove alternatives?
    • How important would you rank the following potential improvements to VisTrails?
      • Add alternatives?

January 27, 2016

Updates

  • New interpreter
    • Working prototype soon
    • Looping not completely done (and port combination mode: pairwise/cartesian...)
    • Need to integrate with VisTrails
      • Compatibility with existing modules
      • Use log controller
      • Want to cache at multiple levels (whole stream/single element of stream)
      • Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)

Items to Discuss

  • User Survey
  • ZMQ failing on windows (vistrails-users)
  • [TE] Python 3 branch
    • Testsuite almost passes
      • Failing are unicode filenames, some users guide examples, and thumbnail comparisons.
    • Fixed issues
      • Signatures are now bytes (convert to string to show in widget)
      • Cannot compare different types, e.g., (None < 1)
        • PortSpec.depth and min/maxConn could be None. (Backport!)
        • cell.row could be None
      • Lots of unicode/bytes fixes to serializations
        • We have encryption implementations in the code! (keychain)
      • Lots of small fixes to package method differences
    • Current issues
      • logging widget empty?
      • version tree signals messed up
      • provenance view legend icons black
      • Text do not fit in some widgets
      • VTK thumbnail comparison fails
        • VTK Cells have some faces in wrong direction?
        • Because running in VM?
    • Installing on Ubuntu LTS/Latest Mint
      • Install latest matplotlib from pip
      • Install VTK 7.0.0rc2 from source
      • No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)

January 20, 2016

Updates

Items to Discuss

  • [TE] Backporting features from Python 3 + PyQt5 + Qt5
    • Merged unicode with master (unicode)
      • One unicode test failing
      • Problem saving to xml
    • Used python-future to run futurize stage 1 to update legacy (pre 2.6) code (unicode-futurized)
      • Updates print to functions 'from __future__ import print_function'
      • Updates exception calls to use 'as'
      • removes most usages of 'long' (mostly used unnecessarily)
        • Only needed for 32-bit support
      • Fixes relative imports (from __future__ import absolute_import)
    • Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 (unicode-futurized-qt5)
      • Updated all signals to new-style
  • If we go 2 to 3, this could break packages
    • means we should schedule any package format changes for the same time
    • e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* #1002)
    • any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)
    • putting package identifier as xml?
      • Python not so bad, plus some stuff in there is Python code (checking dependencies)
    • version the APIs?
      • RR planning on versioning vistrails_module.Module, no point versioning the whole package?
  • file formats on 2 to 3:
    • no real translation/upgrade, but non-ascii values that work on 3 won't load in 2

January 13, 2016

Updates

Items to Discuss

January 6, 2016

Updates Items to Discuss

  • Updating VisTrails dependencies
    • Successfully built Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7
      • VTK 7 is the first supporting Python 3 but is only in release candidate phase.
        • Next version of VTK 6 is rumored to support Python 3
        • There is a development branch of VTK 6 supporting Python 3
      • Can travis build this?
        • We can skip tests for WebEngine and VTK if needed
    • Python 3 showstoppers
      • No MySQL-python (sqlalchemy supports mysql through other drivers)
      • No suds (there are active forks but we could also drop support?)
  • Options for python2/3 support:
    • Interpreter could be python2/python3 based on packages
    • six layer works but not the easiest to maintain (e.g. in reprozip)
  • Drop support for win-64
  • Update copyright headers to 2016

Older meetings

Retrieved from "https://www.vistrails.org//index.php?title=Development&oldid=14736"