Difference between revisions of "Development"

From VistrailsWiki
Jump to navigation Jump to search
 
(362 intermediate revisions by 3 users not shown)
Line 3: Line 3:
</div>
</div>


== 2015 ==
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]
 
== 2017 ==
 
=== October 30, 2017 ===
 
'''Updates'''
 
* [DK] PyQt5
* [RR] Running Python3 code
 
'''Items to Discuss'''
 
* Installation:
** Use conda? (macOS hack to have .app bundle load conda-installed vistrails)
** Update conda recipe (pyqt4 restriction for now)
* Interpreter
* Release
* Integration
** hard to build same type of GUI in other apps that integrate core VisTrails (tend to use just the API)
** independent pieces of the GUI could be reused if they were more independent.
 
=== September 6, 2017 ===
 
'''Updates'''
 
* [RR] Fixes for scikit-learn, tables, usagestats
 
'''Items to Discuss'''
 
* Stripped-down version of VisTrails to showcase new features?
** include meta-vistrail, bundles
** hide mashups, extra widgets, latex mode, etc.
* Interpreter
 
=== August 16, 2017 ===
 
'''Updates'''
* [DK] meta-vistrail
** Working actions: normal action in vistrail, pruning version tree, tagging operations, and change
** Interface
 
'''Items to Discuss'''
* Notebook
* Interpreter
* Web visualization packages (e.g. bokeh)
** [https://github.com/VisTrails/VisTrails/pull/969 PR 969] has more details
* Pipeline templates
 
=== August 2, 2017 ===
 
'''Updates'''
* [DK] meta-vistrail work
** Extend Vistrail to MetaVistrail to support meta-actions
** Vistrail becomes less tied to workflows so it can also supporting versioning vistrails
 
'''Items to Discuss'''
 
* Interpreter
** Using dask?
** Streaming an issue
* Notebooks
** What is an output of a module? (need to expose a specific output port or the entire module)
** Unifies some of the work from the script-workflow code
 
=== June 28, 2017 ===
 
'''Updates'''
* [DK] Testing UUID/SQLAlchemy branch
* [DK] Working on meta-versions
** would be nice to have pipeline and vistrail separate
** [RR] Then do we only have meta-actions, any change to the pipeline is a meta-action that adds a version that does the change?
** [DK] Yes?
* [RR] Notebook interface (Vizier) making good progress, figuring out cell outputs
 
'''Items to Discuss'''
 
=== June 7, 2017 ===
 
'''Updates'''
* [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs
 
'''Items to Discuss'''
* [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)
** Can designate in module description an output port
** Value has to be serializable
** Should use the OutputModes work to handle multiple modes, configuration, selection, ...
* [RR] Is VisTrails with SQL usable concurrently?
** [DK] Probably not, demo from [TE] exists
 
=== May 10, 2017 ===
 
'''Items to Discuss'''
 
* Identifiers
* Bundles & DB Code
* Interpreter
* Merge / Versioning Version Trees
* Python3
* Workings of List
* Packages: Wrappers + new Versions (VTK7)
* Script Conversion
* Intermediate Outputs: reuse output module work?
** would be useful for both workflows (run to a particular checkpoint)
** and useful for notebook interfaces
* SQL backend: does it allow for concurrent access?
** Yes, real-time collaboration work for concurrent single vistrail access
** Should be able to access different vistrails at once without issues
 
=== April 19, 2017 ===
 
'''Updates'''
 
'''Items to Discuss'''
 
=== April 5, 2017 ===
 
'''Updates'''
 
'''Items to Discuss'''
 
=== March 8, 2017 ===
 
'''Updates'''
* [RR] Added a looping indicator on modules [https://github.com/VisTrails/VisTrails/pull/1217 #1217]
** This makes it easier to understand what's looping, no need to follow edges and count
* [RR] Refactoring CLTools for integration in Docker package
* [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)
 
'''Items to Discuss'''
* [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers [https://github.com/VisTrails/VisTrails/issues/1216 #1216]
** Can we move this up to the renderer modules?
** Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)
 
=== February 24, 2017 ===
 
'''Updates'''
* [RR] Rich-text module documentation ([https://github.com/VisTrails/VisTrails/pull/1210 #1210])
** Went over packages and fixed format; maybe we should have a test to validate RST?
* [RR] Single-instance shutdown with <tt>--remote-shutdown</tt> ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])
* [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment
 
'''Items to Discuss'''
* [RR] There are issues with list-depth
** Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)
** Issue with looping groups
 
=== January 25, 2017 ===
 
'''Updates'''
 
 
'''Items to Discuss'''
 
* Scaling VisTrails
* sql-alchemy, uuid, bundle branches/merges
 
=== January 11, 2017 ===
 
'''Updates'''
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]
 
'''Items to Discuss'''
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails
** Have homemade workflow system based on R scripts and symlinks
*** Each steps has multiple sets of parameters
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code>
** VisTrails improves provenance capture and makes editing pipelines easier
*** Need better support for aggregation (groupby operation might be good enough; pandas?)
 
== 2016 ==
=== December 14, 2016 ===
 
'''Updates'''
* [RR] Single-instance code: over-engineered, but will work
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)
 
'''Items to Discuss'''
* Email from Matt Dirks: combobox should work, ask for more code?
* Email from marin.nl:
** looks like they would benefit from new interpreter
** how to store their consolidated provenance when they exchange .vt files around?
 
=== November 30, 2016 ===


=== January 21, 2015 ===
'''Updates'''
'''Updates'''


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Examples for scikit-learn: [JF] has an old example using Weka with parameter exploration (not currently in source tree)
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])
** Still improving it
** Want to return something more structured to the client (success/fail code, stdout, stderr)
** Should be usable without Qt
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason
* [DK] Working on batch execution using console_mode
* [DK] Settings aliased parameters when using core.api?
 
=== November 16, 2016 ===


=== January 14, 2015 ===
'''Updates'''
'''Updates'''
* [TE] Working on classifier
 
* [RR] Scripting integration, work in progress
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])
** Let's talk next week, [AM] and [DK] are not here


=== January 7, 2015 ===
* Email from Pieter: executing pipelines with parameters from command-line using single-instance
** Seems to work, main instance gets full command-line and alters configuration temporarily
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful
** Actions:
*** Can add switch to kill main instance
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?
 
=== November 2, 2016 ===
 
'''Updates'''
'''Updates'''
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)


'''Items to Discuss'''
'''Items to Discuss'''
* make sure that we address critical issues, questions, and pending review branches in a timely manner
* scripting support
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow
** [RR] can translate from workflow to script, working on script to workflow
** will work for parameter value changes, structural changes require changes to the annotations
** need to publish best practices here
** would be cool to do looping in scripts (easier interface than with workflows)
* notebook support (convert form notebook to workflow)
** RR will sync with FC on this
* Issue with console in built-from-scratch
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3
** binary has old version of iPython -> 1.0.0, should we update?
* [TC] automated wrapping of numpy and scipy
** discovered a bunch of malformed documentation in numpy and scipy
** has github repo for vistrails tools
** example modules wrap a bunch of R stuff (not baked in, just how things are)
** will be pushing wrapping logic up
** port names forbidden (window and domain)
** have an import hook to get from yaml directly to VisTrails Modules
** should work for any python modules with well-formed numpy docstrings.
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)
* [TE] build and build scripts
** completely automatic, buildbot
** need to set the build machines for the environment we want for the binary
** would virtualenv work here?
** [TC] anaconda can pin versions, potential path to test different configurations
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07
* package issues (see Remi's message)
* [TE] Scope of tej
** Support single ssh commands?
** Queue can be used as a remote machine (crawler is using queue.call*)
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07


== 2014 ==
=== October 5, 2016 ===
 
'''Items to Discuss'''
 
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]
** unique ids
** bundles
** GUI fixes
** wrapping updates
** python3
** interpreter?
 
* command-line input parameters to provenance
** [DK] we don't create a persisted workflow from those input params, i think
** [DK] these are already defined as aliases so we treat them differently form a normal input param
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases
** Do the vistrails variables values get stored in provenance? (what if values change?)
 
=== September 14, 2016 ===


=== December 17, 2014 ===
'''Updates'''
'''Updates'''
* Lots of progress on sklearn package on Monday ([https://github.com/VisTrails/VisTrails/pull/955 #955])
** AM ran into issues with port orders
*** any manually specified port order should always be left-to-right. perhaps don't change the default port order yet
** sklearn modules uses different port shapes for its different types: Estimators (diamond), data (circle), parameters (default, square)
*** [RR] Looks good; generalize this?
*** do this for types of ports rather than try to infer for any port
** Issue with <tt>Pipeline</tt> module: how to specify a variable number of inputs that need a specified order
*** DK wrote something with <tt>ConcatenateString</tt> for this
*** problem is you need a UI for ordering the connections (order-connections branch assumed order they were added)


'''Items to Discuss'''
'''Items to Discuss'''
* scripting support
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)
** how to go from one to another
** Did we know/contribute to this?
* crawler progress [TE]
*** [DK] Yes. We wrote the text for the VisTrails chapter
** building a package to interact with the crawler
** Drafted response
* Two week break, Next Meeting is January 7
*** [DK] Looks good, will forward.
 
=== August 31, 2016 ===


=== December 10, 2014 ===
'''Updates'''
'''Updates'''
* [TE] bundles
* [RR] Went over vgc.poly.edu/vistrails.org config:
** Finished command-line and core API for handling bundle data
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
** controller has methods for adding data to bundle
** Fixed usersguide scripts (won't disappear again)
** [DK] why command lines? [TE] allow other tools like reprozip to build bundles
** Moved some websites over to HTTPS
** [DK] seems like could grow into a lot of command-line options
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]
** [TE] can also add the files as a directory
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"
** want to have call that allows user to specify what is saved in the bundle
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that
** [TE] should the user have access to add other parts of the bundle besides data? Probably not.
** [TE] working on storing packages in bundle and loading them from bundle?
* [TE] web crawler
** Using tej
*** Problems with closed sockets
*** No way to reconnect to queue
*** copy fails? TE to provide example to RR
** crawler works by submitting a job as a start and stop
** create a new completed job if it runs
** [RR] want to be able to get result even if not completed
* [RR] web package for d3.js etc
** [DK] looks good, works for me, github comment on supporting more specific use cases
** RR working on code editor


'''Items to Discuss'''
'''Items to Discuss'''
* Should source modules' code ports be control parameters? [https://github.com/VisTrails/VisTrails/issues/973 #973]
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
** think about how to support the raw cases where the source is computed upstream
** Get rid of more services? (svn, trac, ...)
** static typing
* [RR] Notebook execution works. How do we support more languages (than Python)?
* unicode/string issue from BNL?
** Need: detection, metadata reader, code to input values
* Remove Serializable: [https://github.com/VisTrails/VisTrails/pull/966 #966]
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?
** shuold we also remove the serialize calls in basic_modules.py
 
* [TC] auto-wrapping all of numpy into VisTrails
=== August 17, 2016 ===
** looks like matplotlib scraping
** ndarrays are labeled as lists, reason? no
** parameterized types? (have depth, could have Float with depth 2)
** do we convert the types? don't think so
** TC will add depth 0 ndarray
** override Module calls? potential issue with the depth computations?
** numpy has broadcasting to do internal parallel


=== December 3, 2014 ===
'''Updates'''
'''Updates'''
* [TE] bundles
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.
* [TE] Matplotlib and sklearn are exportable using module-to-script api
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.
 
'''Items to Discuss'''
'''Items to Discuss'''
* [TE] bundle extensions
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
** jobs stored in 'jobs' file
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
** Data stored as file references in data/ directory
* [TE] Wrap-up notes
*** Add with gui/api or a method like 'add File data to bundle'
 
*** Have some way to browse by directory
=== August 10, 2016 ===
** Packages stored in bundle
*** Add/delete package method
*** When loading, ask 'Do you want to import package in bundle' if not already existing.
*** [RR] need to fix reloading issues and conflicts with packages that already exist
*** [RR] Full name of a package can change (userpackages vs. vistrails.packages)
**** [DK] does this ever get saved anywhere? thought we only use "codename" which has no depth (dots)
** Package data (Like a Web Service wsdl stored in packagedata/my.package.identifier/
*** Packages can add/remove data as bundleobj file references
*** [JF] Do we need a package-specific directory?
*** decision to use the data directory for everything, no package-specific directory
* Scripting support
** [RR] pipeline->python translation needs some tweaking/testing, [https://github.com/VisTrails/VisTrails/pull/950 PR #950]
*** python->pipeline translation not done, we have some old code but might not be salvageable
** [RR] IPython notebooks integration still waiting for inputs/outputs definition in notebook format
*** we can probably use whatever metadata format in the meantime and reads theirs once it exists
*** send email to IPython folks
*** (the goal here is to wrap whole notebooks as VisTrails module, to string them together, still using IPython kernels to execute each notebook)
* integrating web things in VisTrails
** [RR] wants to take a look at this but working through reprozip backlog


=== November 26, 2014 ===
'''Updates'''
'''Updates'''
* [TE] improve-vt-bundle
 
** .vt load/save is working
** incrementally writing to vistrail file?
* [DK] add web inspector (merged)
'''Items to Discuss'''
'''Items to Discuss'''
* [JF] scope of filenames
* [TE] Python Wrapper and scripting
** [DK] what about copy-paste between vistrails? (hidden dependencies?)
** All changes merged into python-wrapper branch
*** [JF] shouldn't expect things to run here
*** Tests pass
*** need way for user to copy or extract from bundle
*** Still need to update sub-branches
** other option: have toggle to say that name is local to bundle?
** Minor fixes
** determine API first
*** vtkPlot3DReader upgrade bug
* [TE] job-fixes
*** Added tuple exporting
** RR working on improvements to job handling
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]
** [TE] what still needs to be done here?
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
** [RR] bug where need to dedup jobs, two workflows with same job (same subworkflow signature) will store job twice in job monitor and will check for both separately, and when finished, assumes one is still running
**** Could be a bug in the old vistrail, but not sure
** make job a first-class entity, work independently of workflow/vistrail
* [DK] Subworkflows and uuids
* [DK] Move to a new version of Qt for improved well cell support?
 
* [DK] Working with web frameworks/apps in VisTrails
=== August 3, 2016 ===
** [RR] start a local web server for this to avoid some issues
** way to move files computed upstream to a resource that the web server can access


=== November 19, 2014 ===
'''Updates'''
'''Updates'''


'''Items to Discuss'''
'''Items to Discuss'''
* [JF] Upgrade tree ([https://github.com/VisTrails/VisTrails/issues/949 #949])
** list view
** every node is a new one
** [JF] most natural way is to have same tag
** [JF] cannot reselect the old version
** Upgrade a node and have that change propagate
** Meta-actions?
** Save and save as tag actions (save moves the tag, maintain the history of a tag)
** [RR] Suggest collapsing the upgraded node into the current node (hide the un-upgraded node)
* [JF] Ability to refer to file names relative to the vt directory (related to packaging) -- this issue has arisen in the context of ReproZip
** flag in the File module to make things relative to the vt bundle?
** [TE] import data into vistrails?, truncate based on a map from file locations to bundle/local directory locations
* [TE] job-fixes ([https://github.com/VisTrails/VisTrails/pull/27 #27]) & tej
** we'll keep the job_id for now; tej exposes it
** changes to be merged in
** tej works, needs testing on actual clusters (NYU?), change name?
* [TE] figured out how most of improve-vt-bundle is supposed to work
** bundle has method to get_db_object, get_db_objs


=== November 12, 2014 ===
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]
** Connection variables now lowercase
** Can we use code as output variables?
*** E.g. "vtk_dataset_mapper.GetOutput(0)"
*** Could mutate output when output method is only supposed to be called once
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)
 
=== July 27, 2016 ===
 
'''Updates'''
'''Updates'''
* [RR] We moved to Github
 
* [RR] UV-CDAT moving to Github; UV-CDAT/VisTrails currently marked as fork
* [DK] DB bundle serialization implemented
** This means that opening a PR for a branch in that fork will target VisTrails/VisTrails by default
** People have to select UV-CDAT/VisTrails as base every time
** Unmark it as fork if accidents happen? (like [https://github.com/VisTrails/VisTrails/pull/956 #956])


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] We are on Github!
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]
** Please check [https://github.com/VisTrails/VisTrails/wiki/Development github-wiki/Development] for some guidelines on development
** locale?
** Most importantly, be sure to check out tickets marked [https://github.com/VisTrails/VisTrails/labels/@question question] or [https://github.com/VisTrails/VisTrails/labels/@pending-review pending-review]
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]
* Email notifications from Github
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches]  
** Right now vistrails-dev is watching the repository
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)
** This means the list gets an email whenever anything happens (except from pushes)
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]
** You can unsubscribe from issues, but you'll still get an email through the list
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.
** You will receive everything twice, and receive notifications from your own actions, because of the list
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]
** Is asking developers to [https://github.com/VisTrails/VisTrails/subscription watch the repo] enough?
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)
* [RR] BuildBot/Github interaction status?
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files
** [TE] buildbot is configured to use github with "git:" protocol.
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]
** [TE] Web hook is not working, hook is using "https:", but repository is using "git:"?
** Still need to do something nicer for the patch formatting
** [TE] Binary nightly build working for 2.1, will soon be merged into master
*** patch keys currently contain non-python-variable characters but this can be fixed
* [TE] Atomicity in Job Submission state graph ([https://github.com/VisTrails/VisTrails/pull/27 #27])
*** Need to replace string template with python variables that is replacable using RedBaron
* github shorthand? "fix" keywords close issues
**** Patches could then look like:
* feature branches move via pull requests
 
* Different releases: windows (4x), github and sourceforge
    def vtkInstance():
* Nightly releases? (should we push these to sourceforge?)
        self.vtkInstance = self
 
=== July 20, 2016 ===


=== November 5, 2014 ===
'''Updates'''
'''Updates'''
* [TE] setting up buildbots to build the binaries
 
** solved most of the issues
** working on Windows buildbot
'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Job submission
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]
** [https://docs.google.com/drawings/d/1NRHNwm48pgcAjMeat_zXNOLH-M4phPEkJ79TdyYaRKs/edit Flowchart]
** A patch is a string template with self/input/output variables for patching a class method
** JobMixin doesn't work for Rémi's use cases
** Patches are stored in xml specification
** Need to check on job more than once (based on signature, if server removed job, etc.)
** Patches are assigned to methods in ClassSpec's
** [TE] what if job failed or job has been submitted but isn't running?
** Used both for execution in vistrails and when exporting as script
** Pull Request: [https://github.com/VisTrails/VisTrails/pull/27 #27]
*** Executed with "exec" when run in vistrails
* [RR] Github migration
*** Turns patches into natural calls in script
** Notification on the Github web page to let users know not to open new tickets for a couple of days
** Wrapping VTK library no longer needed
* scikit-learn package:
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)
** Jorge wrote a package about this in 2012 at ORNL, need to talk with him
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)
* [DK] Interaction provenance
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)
** Current able to persist the state of a cell (e.g. Save Camera) if they align with parameters
** TODO's
** How would we capture interactions with the cells as they happen?
*** Add to export_as_script
*** May not correspond to a compute-time parameter (e.g. mouse drags)
*** Patch Modules that represent Python functions as well
*** Capturing low-level user interactions may be problematic if we don't know how an interaction sequence translates to a meaningful action
 
** [JF] try to link back to parameters (otherwise, how do we have anything meaningful)
* [vistrails-users] Read data from SQL example?
** [JF] Granularity issues
 
** Maya: hierarchy of actions, use geometry to try and infer this (analyze object and look for things that are separable)
=== July 13, 2016 ===


=== October 29, 2014 ===
'''Updates'''
'''Updates'''
* [RR] Build scripts
 
** Tommy working on release scripts (updating version numbers in build scripts and splash screen)
* [DK] bundles on sql-alchemy branch
*** [TE] prepare_release.py updates version numbers, branch, hash, and splash (requires inkscape)
** Bundle, BundleMapping, and BundleSerializer
** sdist and wheel ("PyPI" packages) fully automated, have [http://vistrails.poly.edu:9050/waterfall?show=pack-v2.1-sdist builder]
** New BaseSerializers to deal with versions of directory, zip, db serializers
** conda too, [http://vistrails.poly.edu:9050/waterfall?show=pack-v2.1-conda builder]
** Can register the BundleSerializers with the base serializers
** Windows build script needs lots of binaries in fixed locations, e.g. C:\Python27, C:\Users\vistrails\src\vtk, ... build difficult to reproduce
** Translation between bundle versions
*** could use conda here, but problems with the script
** Bundle.add_object "just works" if an appropriate mapping was registered
** Mac OS ???
*** [TE] Built with "make_app"
**** Needs installation of qt, vtk, etc., to build.
**** Building usersguide should be more contained
*** would py2app work on a condo-installed VisTrails?


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Weird error in UV-CDAT has me baffled ([https://github.com/UV-CDAT/uvcdat/issues/843#issuecomment-60843062 #843])
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]
** [DK] check on packages.uvcdat (generic name, could be overloaded) and/or the Variable name (also generic, could be overloaded)
** API Module can be executed using '''compute''' method
* [RR] JobSubmission and RemoteQ:
*** Takes list of outputs to compute + inputs as keyword arguments
** some minor issues with the way things are stored, mostly because uuids and other items should be removed
* Status of general cell synchronization (vistrails-users)
** [RR] caching output values in the vistrails was not a good idea
** big issue is still that RemoteQ is not best solution
** keep mapping on server to map something in module (pipeline_signature) to job and output
** VT keeps track of which workflow submitted job
** start_job method just calls arbitrary line of code to start job
* github migration
** [RR] to send github an email to ask what the best method might be
* problems with terminator.vt example?
** potential interaction handler issue?
* vt using temporary directory to store files
** disk cleaners can wipe these directories (USGS reported)
** question is whether disk cleaners *should* be doing that...


=== October 24, 2014 ===
=== July 6, 2016 ===
RR and TE meeting about job submission & RemoteQ status in VisTrails


http://www.vistrails.org/index.php/JobSubmission
'''Updates'''


Summary:
* [DK] Working on bundles
* some details to be fixed in RemoteQ package
 
* job result cache shouldn't be in vistrail
'''Items to Discuss'''
* the RemoteQ library is a problem since Troels doesn't maintain it
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]
** Call module in script using vistrails API
** Can be used when no export method is defined.
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?
**[RR] No, unreleased
* [DK] New bundle organization
** BundleMapping contains individual mappings from raw objects to BundleObj
** Must be separate in order to have in-memory, separate from serializer settings
 
=== June 29, 2016 ===


=== October 22, 2014 ===
'''Updates'''
'''Updates'''


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] improve-vt-bundle
* [TE] Export VTK as python script
** lack of documentation
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]
** two layers of serializers
** Preludes (imports) still need improvements
** serialize in more on-demand manner
** Works with vtkInteractionHandler (clipping plane)
** does auto-save save everything
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)
** database bundles: try to tie into sqlalchemy
** vtkRendererOutput Uses VTKRenderWindow as output
* Github migration
*** Should support different output modes?
** decide on how to move the open tickets
*** Does not work with multiple outputs (Next cell is displayed when current is closed)
** 26 first issues are already used, can just add original ones
*** Uses wrapped vtkInstance (as vvtk)
** Rémi needs access to the trac database (vistrails.org)
**** Create new patch system with code that can be used both in vistrails and in a python script?
** if possible, move everything
 
* [TE] New 2.1 release to fix examples?
=== June 22, 2016 ===
** upgrades had a bug so could affect existing (older) vistrails
** changes to script commands
** remove tilde files in release
** [RR] is release process automated?
*** [TE] run scripts in the git repository
*** putting release note inside vistrails
** TE to look into which steps are more difficult to change
* [TE] job-cache feedback?
** shareable jobs, but jobs are not persistent
* [TE] update Web Services examples
** look into RESTful services?
** should we deprecate the old examples?
** check the user's guide and edit if necessary
* [RR] VisTrails meant to written by developers and used by users
** writing packages is a bit hard
** developers want to use their own editors, specify which versions should be used
** still have to write your own wrappers
** if developers just want to integrate their tools, can be difficult
** get python program drag 'n drop into python source


=== October 15, 2014 ===
'''Updates'''
'''Updates'''
* [TE] job-cache (merged)
** Store jobs in vistrail
** Fixed jobs in automatic loops
** [RR] Probably shouldn't be in VisTrails
*** RemoteQ/BatchQ are nowhere to be found (except VisTrails's binaries), install procedure doesn't work, and unsupported
*** Not really used by anyone else ("proper" job submission, e.g. clusters, use PBS/...)
*** Sharing job status in vistrail is unsafe and WILL break if people make changes
*** Having that in the vistrail XML for the next release would force us to support it/provide upgrade paths
*** Suggest removing for now, while we think about this more


'''Items to Discuss'''
'''Items to Discuss'''
* (from last week)
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])
** Use Github and pull requests?
** Uses module location to find module remap (The best we can do right now?)
*** Have to find a way to import tickets from Trac
** Also fixes remapped port names for connections to existing modules
** Use a separate fork for UV-CDAT's vistrails?
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name
** Check with Claudio
 
*** Claudio says yes
* [TE] Working on exporting wrapped libraries (VTK)
* VisTrails and ViDA?
 
* [http://i.imgur.com/nqjTolG.jpg Notes] from meeting with F. Perez
=== June 15, 2016 ===
** Use whole notebook as modules in VisTrails (don't necessarily need to deconstruct cells)
*** IPython providing "kernels" (execution engine, behind notebooks) for 21 languages
*** connect notebooks that are of different languages
*** reduce serialization issues
*** implicit if two notebooks need to move outputs into inputs
*** They are discussing concepts of "notebook inputs" and "notebook outputs" right now
**** Kind of like our mashups
**** [http://mail.scipy.org/pipermail/ipython-dev/2014-October/thread.html#15145 nbparameterise thread]
*** Inter-notebook interactions probably not envisioned (but they'd like to be able to point to VisTrails for that)
*** [DK] interface workflow: how do users edit two different notebooks and connect in VisTrails (seems like a lot of windows)
*** restrictions on what can be serialized? notebooks need to be serialized properly
*** can you get a notebook without the stored outputs used by nbviewer?
** Using VisTrails workflows and capabilities from IPython
*** Generic Python API + some rich text augmentations for the notebook
*** Using VisTrails from notebooks using non-python kernels? IPC between kernels is not a goal for IPython folks
** Will try to follow their  [http://mail.scipy.org/mailman/listinfo/ipython-dev mailing-list] and [https://gitter.im/ipython/ipython gitter]


=== October 8, 2014 ===
'''Updates'''
'''Updates'''
* [TE] Store one job monitor per vistrail (almost done)
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])
** Easier to share running jobs
** Fully implements connection merging and list looping
** Jobs are shared between all workflows in a vistrail
** use new bundle structure
** what happens with local jobs?
*** doesn't really make sense to open a job started locally on another machine


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Use Github and pull requests?
** BNL uses them, UV-CDAT switched to that to
** [[Development Branches]] is hard to keep updated and offers no categorization/sorting
*** create pull request when new feature is started, do not have to wait until branch is completed
** Also, use separate fork for UV-CDAT's vistrails? [https://github.com/UV-CDAT/VisTrails UV-CDAT/VisTrails] is marked as fork but can have a different set of branches
*** vistrails.org might be the only UV-CDAT submodule that can't be served over https (see [https://gist.github.com/remram44/978b23f2eac006d5a5bb emails])
** user permissions?
** Juliana, thoughts on moving vistrails repository to github?
* need to keep testing 2.2 features (e.g. automatic looping, new widget interface) so we can prepare that release
* deprecation flag/error for old constant creation methods


=== October 1, 2014 ===
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])
** Analogy only works We need to use the non-upgraded action chain
** But pipelines need to be valid (matcher uses module portspecs)
** Tried to get upgrade remap from upgrade action, but this seems impossible
** Use module location instead?
*** Location should be identical for one-to-one module upgrades
 
=== June 1, 2016 ===
 
'''Updates'''
'''Updates'''
* [TE] Graphicsview font issues solved
* VisTrails 2.2.4 announced (finally)
** Preventing DPI changes on windows and using fixed values for module size calculations
** 5 survey replies
** Windows fonts always look bad
** 47 downloads last week


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] To review&merge: <tt>constant-classes</tt>, <tt>invalid-pipeline-hides-package-errors_/loops</tt> ([https://www.vistrails.org/ticket/901 #901]), <tt>version-flag</tt>
* [TE] Export VTK as script (python-wrapper)
** constant-classes: get rid of dynamic generation, use metaclass for constant class
** to_python_script
** version-flag: add --version flag back on argparse parser
*** needs the pipeline (or at least the connections)
** invalid-pipeline-hides-package-errors: switch to invalid pipeline and will try to fix it, handle upgrades
**** Should not  generate code for any possible VTK method (exporter detects which ports is used in the source)
*** actual error is not showed
**** VTK needs to know connection order
*** get into a deep recursion? upgrade to package and back
**** Adding functions directly to the script results in nicer code
* [RR] JobSubmission package status (does it work? is it useful?)
** Add support for multiple input connections
** should be able to remove it
** Add support for list looping
** [TE] different things to control batch job
 
* [TE] buidbot web interface got hung up
=== May 25, 2016 ===
* [TE] New features in 2.2 release (please extend/update)
** [RR] [[New features]]? Maybe we should maintain a CHANGELOG file in the repository
** Inline widgets ready? Need to test
** output modules
*** incremental change where we keep both cells and new output modules, then phase out output modules?
*** expose configuration keys as ports
* Changes to interpreter
** Simplify/Have different interpreter
** Support different environments
** Move update method from Module to interpreter
** Merge multithreaded interpreter
*** Automatic loops/streaming should work during incremental transition
** Different modules will support different modes of execution
*** have documentation for the best practices
* [TE] Error setting current_pipeline_view=None ([https://www.vistrails.org/changeset/968dede3c6bbe39aae9f37242af9cd2f0e61166e fix])
* [TC] have different namespaces with same modules
** using numpy docs to do most of the code generation


=== September 24, 2014 ===
'''Updates'''
'''Updates'''
* [TE] rewrite-startup integration finished
* Mailing-list fixed
* [TE] UV-CDAT bugfixes finished
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])
** need to make sure that the Aashish and/or Charles are aware the changes have been made
** [TE] notification on github


'''Items to Discuss'''
'''Items to Discuss'''
* [DK] parameterExploration flag: just run the latest of the selected version
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)
** may be nice to have a way to set parameters directly from the command-line like we do with output variables
* [TE] Union ports
* [RR] code issues found using pycharm
** Show union ports in pipeline (DEMO)
** there are a lot more issues than that, cf [http://build.remram.fr/builders/pylint/builds/1/steps/shell/logs/stdio/text pylint]
 
** use pycharm IDE or install pylint, pychecker to avoid committing basic mistakes
=== May 18, 2016 ===
* [TE] QGraphicsView size issues between Linux/Mac
** the text size is different even for the same fonts
** matplotlib ships with three fonts
** has logic with rules to determine what the closest fonts
** Qt has global configuration that can be hard to override
** terminator example has 15 parameters that are editable (adds 20+ pixels and messes up layout)
* [RR] all tests pass on unicode branch
** will try to keep in sync
** [TC] lots of people with strange paths need to use
* [JF] back and forth with scripts
** cloud execution
** have more rules for certain environment
** simplify the interpreter, have a different interpreter
** create a new direction for a restrictive environment
** have documentation for the best practices


=== September 17, 2014 ===
'''Items to Discuss'''
* [TE] mailing list still down - Julio working on it
* [TE] Improved union port
** New schema 1.0.5 adds portitem.union string attribute
** Works with pipeline view, parameter exploration view, and mashup view
** Should work well for matplotlib
** Use with VTK ports?
*** It uses [item_1,item_2] style ports usually with different tuple length.
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]
 
=== May 11, 2016 ===
 
'''Items to Discuss'''
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?
*** Shows single port for all ports with same sort_key
*** Select specific type when adding/showing
*** Only one type should be visible on the module at any time
*** Making tuple parameters union types is probably too complicated
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly)
 
* [TE] Feature suggestion: Let connections connect to functions/parameters
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters
** Use separate views for ports and functions (like before)
*** Allow rearranging of functions
** Pros:
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.
**** Makes List module simpler
*** Connect to parameter in tuple
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds.
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.
** Cons:
*** Interface more complicated and crowded
*** More difficult for users to grasp?
*** No immediate use case
*** Makes no union type for tuple parameters a bigger problem.
 
* [TE] Publish user survey?
** No response from vistrails release yet
** Not announced on vistrails-users yet.
 
=== May 4, 2016 ===
'''Updates'''
* VisTrails 2.2.4 released
 
'''Items to Discuss'''
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]
 
=== April 27, 2016 ===
'''Updates'''
 
'''Items to Discuss'''
* [TE] Wrapping Bokeh
** New method type "operation"
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output
**** Glyph sets function name and arguments as output
***** Not serializable, but should always be computed together with the subject of the operation.
**** Figure applies Glyph operation on itself
*** Useful for matplotlib?
**** matplotlib currently find actors by searching upstream?
**** But this is more difficult since matplotlib is not function-based.
** New "union" port type
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value
**** Uses only first value in tuple
**** Bokeh have many properties that support (list, string, Float)
*** Does not work with depth=1 types (e.g. Float of depth 1)
**** Turn Depth 1 types with known type into lists?
 
* [TE] New bugfix release
** Finalize survey
 
=== April 20, 2016 ===
'''Updates'''
* Merged upgrade fixes
** Now has infinite loop test
** loop test now correctly loops 5 times
*** remap_module used wrong (current) package version when creating ModuleDescriptor
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])
 
'''Items to Discuss'''
* New bugfix release
** Merge usagestats
** Need to finalize survey
 
=== April 13, 2016 ===
'''Updates'''
'''Updates'''
* [TE] Fixed database from command-line, server mode, command line flags
* [TE] Fixed slow view switching


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] unicode and python3 compatibility
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])
** [TC] <strike>Pull request doesn't change any behavior (text_type, or str)</strike>
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?
** [TC] Many changes from py3 were backported to py2, no 2to3 conversion
 
*** difference between text strings (always unicode) and byte strings (which are the default in py2)
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])
*** six.text_type which on py2 aliases to str and on py3 aliases to str (runs through unicode object initializer)
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]
** make sure to import unicode_literals from __future__
** Cherry-pick test to preserve source?
** [RR] more complicated than that. Let's convert everything to unicode in one go, and fix issues
** Need maxPipelineFixAttempts option?
*** [RR] issues with XML (de)serialization, locators, enum, configuration
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.
** look at a 2.2 release before adding unicode support (unicode may need a while to work bugs out)
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?
** [RR] don't move on Python 3 before unicode is supported. Probably no need to bring in six right now
*** Would need to be moved into handle_invalid_pipeline
* [TE] Parameter Widgets (function-tooltip) ready to merge
** LOD error
* [DK] improve-vt-bundle branch
** db/services/bundle.py exists on its own, need to link rest of code to use bundles
** sql-alchemy waiting, work on bundle first
* [TE] adding command-line options to the documentation, update in one place now (used to have pdf and HTML)
* [RR] Version flag missing in new startup, branch to add back, but argparse messes up the formatting
* [RR] Trying to load the same package twice or different versions of the same package breaks VisTrails
* [CS] Hire a couple of people for UV-CDAT, need someone to bring them up to date
** think about projects for them
** [DK] we are not sure the direction UV-CDAT will take after 2.0 release


=== September 10, 2014 ===
=== April 6, 2016 ===
'''Updates'''
'''Updates'''
* [RR] Backported tabledata package: https://github.com/remram44/tabledata-backport
* [TE] Query View improvements
* [TE] GUI Improvements
** Done except for displaying "Version Info" and #1164
** Added type checking in parameter view
* [RR] Working on finishing new interpreter (finally!)
** Switch to workflow info when no module selected
** Improved fitToView (used a square setting?)
* [TE] Edit widgets
** Hide when zooming out
** Improved proxied widgets
** Improved selection border
** Profiling
*** Nothing major for edit widgets
*** Fixed mark_list_depth calls when switching versions


'''Items to Discuss'''
'''Items to Discuss'''
* Workflow longevity: mta example no longer works
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])
** Fernando suggested offering an option that saves the package (or a description of the package) together with the workflow
** Try latest upgrade first, then previous.
** [DK] I agree, this had been talked about with the generic bundle support
** If all fail, select latest upgrade?
** [TE] Everything stored in a zipped vt file, could also store to a directory
*** [DK] some support for this
** [DK] to look over old code and point out what needs to be done for bundle support
* [RR] maybe work to distribute packages independent of VisTrails (or at least make moving between versions of packages easier)
* [RR] VisTrails integration with ReproUnzip: https://github.com/ViDA-NYU/reprozip/issues/69
** [DK] possible to do python install using smarter package installers?


=== September 3, 2014 ===
=== March 30, 2016 ===
'''Updates'''
'''Updates'''
* [TE] Fixed 1.0.3 schema and log upgrades
** [TE] Make sure that 1.0.3 has the right items across versions
** Keep startup only in master since it is only needed for upgrades


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Edit widgets demo
** don't show name of port for constants
** boolean widget doesn't look great (still native widget)
** different layout on Linux than Mac/Windows due to font?
* [DK] UV-CDAT build:
** Linux build and check for scipy issue (#590)
* python3?
* [TC] Good way to wrap up userpackages?
** using conda package manager (environment has directory and hard-linked into, need to be able to hard link into environment)
** [RR] is working to do a standard python package to recognize these automatically
** For userpackages, can configure userPackageDirectory to point to a hard-linked environment
** Preferences -> Paths & URLs -> Local Packages Directory
** everything to be in a "bubble" at BNL
* [ED] Parameter exploration:
** one variable that I care about but plot as 5x3 not 15x1 or 1x15, enhancement request
* [ED] Series of Qt widgets wrapped into VisTrails modules
** subclass SpreadsheetCell, Qt widgets use matplotlib
** [ED] to provide a demo on what is going on trac


=== August 27, 2014 ===
* [TE] Query View improvements
** Unified do_version_switch and get_upgrades as validate_version()
** Problems with query view using its own controller
*** Fixed module info not being displayed for query results
*** Still no version info for query results
*** Still an issue with version query view not being updated
 
=== March 23, 2016 ===
'''Updates'''
'''Updates'''
* [TE] VisTrails 2.1.4 released
** [RR] Will update [https://pypi.python.org/pypi/vistrails PyPI] and [https://binstar.org/remram/vistrails Conda]. Note that this currently uses its own branch for 2.1.x (<tt>deployment/v2.1</tt>), will be easier in 2.2


'''Items to Discuss'''
'''Items to Discuss'''
* [https://github.com/UV-CDAT/uvcdat/issues/520 UV-CDAT Bug #520] and [https://github.com/UV-CDAT/uvcdat/issues/557 #557]: make sure we don't show errors for packages that don't load (put these at info level)
* [RR] Usage reporting almost ready to go
** [RR] already created a branch with this name
** Need to record whether specific features are used: paramexpore, mashup, query
* matplotlib and UV-CDAT: should be able to include freetype in binaries
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation
* [https://www.vistrails.org/ticket/886 Bug 886]:
** How to record size of vistrail?
** merge in this branch, perhaps not 100% ideal but still better than what we had
*** [DK] IdScope already walks over vistrail, use that
** problem is that you sort of want to be able to do a bunch of actions and then raise InvalidPipeline so you don't have to do multiple try-catch
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines
* [https://www.vistrails.org/ticket/892#comment:2 Bug 892]:
** not pythonic to forceably squelch propagation
** should be a configuration option to do this
** add a filter to the root logger so that VisTrails messages can be controlled
** will think about this more
* [TC] blocker on python3 compatibility?
** [RR] unicode support should probably be done first: unicode from all the files, weird casts everywhere in the code because we used Qt before api v2
** from future unicode literals and str type makes the right thing happen on 2 and 3?
** BNL really wants 3 support, willing to work on
** Issues are [https://www.vistrails.org/ticket/673 #673 (unicode)] and [https://www.vistrails.org/ticket/674 #674 (python3)]
* Relayout programmatically?
** [http://git.io/MR0hZw layout_modules()] in controller


=== August 20, 2014 ===
=== March 16, 2016 ===
'''Updates'''
'''Updates'''
* [TE] Parameter Exploration bugfixes
** "Create Version" spreadsheet button fixed
** Missing date attribute in xml caused errors
** Issue opening vtl links


'''Items to Discuss'''
'''Items to Discuss'''
* [RR] Pending branches:
* [TE] Upgrades in Query Mode
** pipeline-validation ([https://www.vistrails.org/ticket/886 #886])
** Now using code similar to do_version_switch
*** (dep for fix-open-workflow (loading a workflow keeps the locator, so saving will overwrite))
*** Uses handle_invalid_pipeline
*** Ok, will merge
*** Differences: Not using current_pipeline, flushes changes
** logger-propagation ([https://www.vistrails.org/ticket/892 #892])
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)
*** Might be a better way? Discuss this more?
*** Integrate to avoid repeated code?
** hide-missing-dependencies ([https://www.vistrails.org/ticket/895 #895])
*** Use this for query mode?
*** Ok, will merge
* [TE] VisTrails 2.2.4 ready for release?
* [TE] New bugfix release? - Parameter Explorations and wrong usersguide in Mac binary
** tej package updated. Others?
* [TE] Module edit widgets demo
* [JF] upgrades and the version tree
** start with tooltips for type names rather than showing them right now
* [JF] provenance in Jupyter
** give warnings on input validation -- add to trac ([https://www.vistrails.org/ticket/898 #898])
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]
* [Thomas] matplotlib is meant to be used the other way
 
** [RR] problem: this is not the way our other plotting packages work (including VTK: data -> data -> renderer -> cell)
=== March 9, 2016 ===
** [RR] we've already released a matplotlib package (data -> plots -> figure -> cell) and can't really turn it around
'''Updates'''
** [RR] we all agree that current code is hacky (but works), see [https://www.vistrails.org/ticket/685 #685] then [https://www.vistrails.org/ticket/396 #396]


=== August 13, 2014 ===
'''Items to Discuss'''
'''Items to Discuss'''
* [TE] FileOpen event on Mac ([https://www.vistrails.org/ticket/889 #889])
* [TE] Fixing search mode
** We can pass argv to QApplication
** String query now matches all by default
* [TE] 1.0.3 schema differences ([https://www.vistrails.org/ticket/894 #894])
*** This caused pipeline queries to never match
** We can restore schema 1.0.3 on master from branch v1.2
** String query can search: User/Notes/Tagname/Time/Module name
*** Should we search module names by default? (Disabled by default because it is expensive)
*** Fixed notes search
** '>' parameter queries in documentation but has been removed
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)
*** Upgrades still a problem
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)
*** [JF] Allow search on past tags
 
* [TE] Create new bugfix release
** Finish querying
** Path issue on windows
 
* [DK] Look into sqlalchemy and new bundle branches


=== August 6, 2014 ===
=== March 2, 2016 ===
'''Updates'''
'''Updates'''
* [DK] bug fixes, improve-vt-bundle, uvcdat
* [TE] Fixed cache bug with port specs
* [TE] VisTrails Next
** Most packages now work
** No qgis for Python 3


'''Items to Discuss'''
'''Items to Discuss'''
* [LB] LSU, instrument building, tomography, interferometry system
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]
* [TC] image registration/processing, infrastructure at BNL
* [ED] working on data analysis at BNL, hoping to leverage VT for data provenance, user activity
* [LB] tomopy and VT integration
* [TC] how to integrate tomography software
* [LB] goal to communicate with others, data security for crowdLabs?
* [LB] improve parameter exploration
** [JF] different ways: whole workflows, loops inside workflows, multiple threads, implict looping
** [TE] add new examples of saved parameter explorations, different versions
* [RR] Bug [https://www.vistrails.org/ticket/886 #886] ("from_root" parameter for <tt>change_selected_version()</tt>)
* [RR] [[API Design|API redesign]], example: [http://nbviewer.ipython.org/gist/remram44/2a67e6269ba2f052a19d notebook], target: [http://nbviewer.ipython.org/gist/remram44/a59a6dbf2026a0433ede notebook]
** [DK] looks good, nicer string rep, improve access to and use of results is important
* Jeffrey Guenther - list handling


=== July 30, 2014 ===
=== February 23, 2016 ===
'''Updates'''
'''Updates'''
* [DK] UV-CDAT animation, sql-alchemy branch update
* [TE] Fixed PythonSource 100% CPU bug
* [TE] VisTrails Next
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples
*** Now uses REST service, tabledata, and Google Maps
*** Web Service replaced with DownloadFile
 
'''Items to Discuss'''
'''Items to Discuss'''
* [TE] parameters in workflow - demo
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.
** show functions on mouse_over
** New 2.2 release soon?
** move set functions to the top
* [RR] Anonymous usage reports
** inline parameter widgets
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]
*** show/hide all
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?
*** toggle in ports list to show/hide these widgets
* noWorkflow - provenance for python scripts
* API design [[API_Design|wiki page]]
** Recent work on tracking dependencies
** vistrails.api versus vistrails.core.api
* Also integrates ReproZip with Jupyter
** API is currently stateful, would like to change
** [http://mybinder.org mybinder]
** API calls using module names, parameter names instead of ids
*** Use with ReproZip to automatically build Jupyter notebook interfaces
** tighter integration with iPython (what were ideas from meeting last year with iPython devs)?
* Daves comments on Jupyter integration
* argparse mutually_exclusive_group bug
** It is possible to identify cell order on the server side
* provenance viewer: make panel wider
** Not trivial to track order of cells on client side
* parameter views: combo box with different parameter viewing types
* How does Galaxy workflows relate to VisTrails?
* change parameter not updating GUI
** Packaging in Galaxy is hard
* installing python packages into binary
** Galaxy for biomedical workflows, difficulty making it general
* modules that dynamically configure their ports
* Action items
** How difficult to track cell dependencies in Jupyter notebooks?
** Use noWorkflow to track cell dependencies?
** How difficult to modify mybinder to use ReproZip packages?


=== July 23, 2014 ===
=== February 17, 2016 ===
'''Updates'''
'''Updates'''
* [DK] output-modules merge, uvcdat, improve-vt-bundle


'''Items to Discuss'''
'''Items to Discuss'''
* API design [[API_Design|wiki page]]
* [TE] Fixed Group memory usage
* [TE] output-modules feedback
** Modules in groups are supposed to be freed when done executing
** Can we have 'Output' module that uses input value to determine output?
*** Prevented by logger that stored references to the modules (fixed, now using id(module))
*** [DK] Unsure what this means, we could have more than one input port, seems like the mode of output is independent of what the value is
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)
*** [RR] Infer from python types would take a lot of work, explicit easier to understand
* [TE] VisTrails Next
*** Usability vs. issues for developers
** Removed deprecated code
** Hide all but value port for outputmodules
*** Core imports that should be gui imports
*** Unsure here
*** GetInputFromPort etc.
** Make output mode be an enumeration rather than text entry (actually need to allow own entry)
**** PythonSources will be upgraded
**  Should vtkRendererOutput replace VTKCell?
*** Fixes for Python < 3
*** [DK] Need to see if display views are involved, interaction handler, etc.
*** fixes for VTK < 6
** Use module name as base name?
** TODOs
*** Also naming for parameters on command-line here
*** Remove RemoteQ
*** if you name module "flower", default base name is "flower"
**** Port Hadoop to tej (Already have pbs)
** Also consider possibly only generating one output based on this type of naming
*** Merge package wrapper
** Global imageFile format setting not working?
*** Finish SQLAlchemy branch
*** Put on trac
*** Finish new bundle branch
** Created WebGL output for vtkRenderer (As empty package)
** [JF] Will contact Jeff and Matthias about moving to Python 3
** Matplotlib WebAgg and MBAgg backends better for web
* [JF] Query by Example is broken
** Clash between mode identifiers
* [JF] VisTrails Data Cleaning Project
*** Register this on trac
** Use VisTrails to process and explore task-specific data while keeping provenance.
** [RR] how to load a specific mode if it isn't loaded?  How do we find it?
** Remi's Idea
* [TE] other ideas
*** Use VisTrails as the backend in a Jupyter notebook
** See other workflow systems: http://imgur.com/a/xY37B
*** Overcome problems with executing cells in random order
** Add icons to modules (graphical representation)
**** Interesting research topic, but little interest has yet been noted
** Add module by double-clicking in the pipeline view and writing module name
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)
** Edit parameters directly on the module
*** Impose restrictions to notebook cell boundaries
*** First step: move set parameters to top of list (or make more visible)
*** It will only work for functional operations
*** Second step: mine to find commonly used parameters
*** Can we fix, or detect, non-functional (mutating) operations?
** Better use of hovering to provide module information
*** Analyse code and data flow and see if it mutates data in unexpected ways?
*** Show parameters on right
*** Mutable operations can be created in many ways in Python
** [JF] Inspect workflows in the same window
* Next meeting Tuesday 11:30
* [DK] matplotlib feedback from Thomas Caswell
** figure top-down?
** see Tom's github fork of matplotlib of pyplot changes
** declarative syntax (MEP 25), goal for 1.5
** create artists in modules
* [RR] export-as-script
** It's mainly about renaming variables, to connect module and avoid collisions
** Currently gets code for script using package-provided to_python_script methods
** Using the [https://github.com/Psycojoker/redbaron redbaron] library for this (under early development)
** Need to handle the different types of input corresponding to different calls (get_input, get_input_list, force_get_input, ...)
* [BNL] Adding new palette tool
** search box with many fields to format query, send back dictionaries
* [BNL] Use numpy docstring format? yes
* [BNL] PEP whitespace? yes
* [BNL] python3 compatibility? no a big priority right now
* [BNL] PyQt new style connections: yes
* [BNL] Access to vistrails.org repository? yes


=== July 16, 2014 ===
=== February 10, 2016 ===
'''Updates'''
'''Updates'''
* [TE] VisTrails 2.1.3 released
* [DK] UV-CDAT


'''Items to Discuss'''
'''Items to Discuss'''
* Integrate patches from github
* VisTrails Next
** we push from vistrails.org to github
** Test suite passes
# git fetch git@github.com:VisTrails/VisTrails.git +refs/pull/17/head:refs/heads/pull17
*** Python 3 less deterministic, module upgrade order random.
# git checkout master
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...
# git merge pull17
** All of our packages should run fine :)
# git push origin master
** VTK thumbnail comparison works
** Synchrotron needs tools for computational pipelines
*** Rendering bug on some systems
* Discussion on scripting support
**** QVTKCell now inherits QGLWidget
** we are going to have to support a set of restricted operations
*** Old thumbnails had wrong height due to spreadsheetwindow being too small
** readability versus parsing
**** Fixed by regenerating thumbnails
** blocks vs. functions
*** Silenced deprecation messages when generating modules
* layered API:
*** Many new classes in VTK 7
** scripting api is base, interactive api has convenience methods that use state
** Remove deprecated code
** design document on API, [[API_Design|wiki page]]?
*** Core imports that should be gui imports
* Other requirements for scripting
*** GetInputFromPort etc.
** higher-level operations to find results linked to certain provenance
**** Will break existing PythonSources
* vt bundles
**** Try to upgrade?
** should we look into improve-vt-bundle and sql-alchemy branches?
*** Old-style packages
** design this (configurable bundles)
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
** have some hooks to add files to the vt bundle already
***** Read info.py without importing package?
* move-dist-directory branch
*** Fixes for Python < 3
** conflicts with python build directories
*** Fixes for Qt < 5
** does this affect build machines?
*** fixes for VTK < 7
*** Remove self port?


=== July 11, 2014 ===
=== February 3, 2016 ===
'''Updates'''
'''Updates'''


'''Items to Discuss'''
'''Items to Discuss'''
* Claudio wants to make it easy to edit PythonSource without VisTrails
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]
** [RR] Dropping url-encoding would allow you to edit them from XML? But they are also quoted because used in <parameter val="..."/>
* Scripting support
** Export workflow as standalone script:
*** Modules could provide their own logic, or we can make something up from compute() ast in some cases
*** Also, add annotations to script to make it easy to reconstruct (and module IDs, so we can update)
** Read back script as workflow
*** General case is hard
*** Building PythonSources from blocks might be doable
*** First step: simply reading some parameters from some modules in annotated script (i.e. has module ids)
* IPython notebook integration
** Wait on output-modules branch; then we can show visualizations in notebook
* API needs work
** API calls need to return things (to put them in notebook)
** Make initialization easier (shouldn't need to create application, etc)
** Currently stateful! (<code>app.new_vistrail(), ..., api.save_vistrails(f)</code> instead of <code>vt=Vistrail(), ..., vt.save(f)</code>)
** Should probably be in vistrails package directly


=== July 9, 2014 ===
* VisTrails Next
** All tests but thumbnail comparison passes
** Still issues with gui and more packages needs to be tested
** PythonSource and MplSource upgraded with lib2to3
** Keep contrib packages? Update?
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]
*** Supports Python 3
*** Will hopefully get into conda and others soon
*** Deprecates a bunch of modules
 
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next
** Runs!
** Uses conda
*** Many levels of requirements
**** apt packages
**** conda packages
**** python packages in requirements.txt
**** python packages in setup.py
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!
*** But useful if we want to use conda in our own build process.
** Lots of errors I do not get locally?
 
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]
** Have you used any alternatives to vistrails?
*** Relevant? More options?
** How often do you use any of these particular features in VisTrails?
*** Add/Remove alternatives?
** How important would you rank the following potential improvements to VisTrails?
*** Add alternatives?
 
=== January 27, 2016 ===
'''Updates'''
'''Updates'''
* [TE] Added logging of parameter explorations
* New interpreter
** Working prototype soon
** Looping not completely done (and port combination mode: pairwise/cartesian...)
** Need to integrate with VisTrails
*** Compatibility with existing modules
*** Use log controller
*** Want to cache at multiple levels (whole stream/single element of stream)
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)


'''Items to Discuss'''
'''Items to Discuss'''
* [TE] Crash when adding String module on Mac [https://www.vistrails.org/ticket/760 #760]
* User Survey
** Updated binary to Qt 4.8.6
** Release 2.1 with Qt 4.8.6 to fix String module crash
* [TE] Table cell - Use HTML cell and D3?
** Interactivity for free
** Can it work over multiple cells?
** Will work directly on web and crowdlabs
* [Claudio] Scripting support
** VisTrails is general but does not interact well with other systems
** What is not working, how can we make it easier?
** Need ability to work without the GUI
*** Construct workflows using utility functions that does not use the VisTrails GUI
** Convert python script to/from vistrails workflow
*** Show PythonSource contents in script
*** Start with workflow that do not show stuff at spreadsheet
** TODO:
*** Design a better mechanism to integrate VisTrails with other tools, starting with Python
**** Provide the ability to save a workflow in a format that can be edited using a text editor -- for example, save a workflow and use an editor to change the contents of the Python Source boxes in the workflow
*** Provide functionality similar to what was implemented in UV-CDAT, where workflows can be used as black-boxes from Python. See slide 41 in http://uvcdat.llnl.gov/media/pdf/VisTrails_UV-CDAT_course.pdf
*** For which applications would scripting support be useful?
*** Look at [http://cycling74.com/products/max max] and [http://grasshopper3d.com grasshopper3d]
*** Claudio wants to test how parallellization works in VisTrails
*** How would scripting support work?
*** Meet on Friday


=== July 2, 2014 ===
* ZMQ failing on windows (vistrails-users)
 
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]
** Testsuite almost passes
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.
** Fixed issues
*** Signatures are now bytes (convert to string to show in widget)
*** Cannot compare different types, e.g.,  (None < 1)
**** PortSpec.depth and min/maxConn could be None. (Backport!)
**** cell.row could be None
*** Lots of unicode/bytes fixes to serializations
**** We have encryption implementations in the code! (keychain)
*** Lots of small fixes to package method differences
** Current issues
*** logging widget empty?
*** version tree signals messed up
*** provenance view legend icons black
*** Text do not fit in some widgets
*** VTK thumbnail comparison fails
**** VTK Cells have some faces in wrong direction?
****  Because running in VM?
** Installing on Ubuntu LTS/Latest Mint
*** Install latest matplotlib from pip
*** Install VTK 7.0.0rc2 from source
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)
 
=== January 20, 2016 ===
'''Updates'''
'''Updates'''
* [TE] List handling improvements done
** Now List ports can connect to anything
** Only exception is List<->Variant


'''Items to Discuss'''
'''Items to Discuss'''
* [DK] List/Variant bug: [https://www.vistrails.org/ticket/871 #871]
* [TE] Backporting features from Python 3 + PyQt5 + Qt5
** [TE] Fixed
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])
** [TE] Why is Map output a Variant?
*** One unicode test failing
** looks like this was done because Fold produces a Variant (makes sense since fold can be a sum), but Map should always be a list)
*** Problem saving to xml
** this will allow depth 1 input ports to connect
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])
* [RR] added List.head as depth 1 port
*** Updates print to functions 'from __future__ import print_function'
** add documentation for this new functionality
*** Updates exception calls to use 'as'
* [DK] QPipelineScene.current_pipeline, controller refs [https://www.vistrails.org/ticket/869 #869]
*** removes most usages of 'long' (mostly used unnecessarily)
* [DK] PythonSource crash? [https://www.vistrails.org/ticket/760 #760]
**** Only needed for 32-bit support
** update Qt for newer binaries?
*** Fixes relative imports (from __future__ import absolute_import)
** pyside support?
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])
** Also, see [https://www.vistrails.org/ticket/737 #737]
*** Updated all signals to new-style
* [TE] Log Parameter Exploration? [https://www.vistrails.org/ticket/870 #870]
* If we go 2 to 3, this could break packages
** Very simple to enable, but we should annotate cell info
** means we should schedule any package format changes for the same time
* [DK] tabledata pandas support?
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])
** investigate on best way to make this available (data types, scripts, modules)?
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)
* [JF] SIGMOD: python is becoming lingua franca for data analysis, munging
** putting package identifier as xml?
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)
** version the APIs?
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?
* file formats on 2 to 3:
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2
 
=== January 13, 2016 ===
'''Updates'''
 
'''Items to Discuss'''
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]
** Basic GUI and execution works
** Minimal work to support 2to3
** Hard to support both PyQt4 and PyQt5
 
=== January 6, 2016 ===
'''Updates'''
'''Items to Discuss'''
* Updating VisTrails dependencies
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.
**** Next version of VTK 6 is rumored to support Python 3
**** There is a development branch of VTK 6 supporting Python 3
*** Can travis build this?
**** We can skip tests for WebEngine and VTK if needed
** Python 3 [https://python3wos.appspot.com/ showstoppers]
*** No MySQL-python (sqlalchemy supports mysql through other drivers)
*** No suds (there are active forks but we could also drop support?)
* Options for python2/3 support:
** Interpreter could be python2/python3 based on packages
** six layer works but not the easiest to maintain (e.g. in reprozip)
* Drop support for win-64
* Update copyright headers to 2016


== Older meetings ==
== Older meetings ==
* [[Development/2015]]
* [[Development/2014]]
* [[Development/2014]]
* [[Development/2013]]
* [[Development/2013]]

Latest revision as of 15:34, 30 October 2017

See also the Github wiki

2017

October 30, 2017

Updates

  • [DK] PyQt5
  • [RR] Running Python3 code

Items to Discuss

  • Installation:
    • Use conda? (macOS hack to have .app bundle load conda-installed vistrails)
    • Update conda recipe (pyqt4 restriction for now)
  • Interpreter
  • Release
  • Integration
    • hard to build same type of GUI in other apps that integrate core VisTrails (tend to use just the API)
    • independent pieces of the GUI could be reused if they were more independent.

September 6, 2017

Updates

  • [RR] Fixes for scikit-learn, tables, usagestats

Items to Discuss

  • Stripped-down version of VisTrails to showcase new features?
    • include meta-vistrail, bundles
    • hide mashups, extra widgets, latex mode, etc.
  • Interpreter

August 16, 2017

Updates

  • [DK] meta-vistrail
    • Working actions: normal action in vistrail, pruning version tree, tagging operations, and change
    • Interface

Items to Discuss

  • Notebook
  • Interpreter
  • Web visualization packages (e.g. bokeh)
  • Pipeline templates

August 2, 2017

Updates

  • [DK] meta-vistrail work
    • Extend Vistrail to MetaVistrail to support meta-actions
    • Vistrail becomes less tied to workflows so it can also supporting versioning vistrails

Items to Discuss

  • Interpreter
    • Using dask?
    • Streaming an issue
  • Notebooks
    • What is an output of a module? (need to expose a specific output port or the entire module)
    • Unifies some of the work from the script-workflow code

June 28, 2017

Updates

  • [DK] Testing UUID/SQLAlchemy branch
  • [DK] Working on meta-versions
    • would be nice to have pipeline and vistrail separate
    • [RR] Then do we only have meta-actions, any change to the pipeline is a meta-action that adds a version that does the change?
    • [DK] Yes?
  • [RR] Notebook interface (Vizier) making good progress, figuring out cell outputs

Items to Discuss

June 7, 2017

Updates

  • [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs

Items to Discuss

  • [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)
    • Can designate in module description an output port
    • Value has to be serializable
    • Should use the OutputModes work to handle multiple modes, configuration, selection, ...
  • [RR] Is VisTrails with SQL usable concurrently?
    • [DK] Probably not, demo from [TE] exists

May 10, 2017

Items to Discuss

  • Identifiers
  • Bundles & DB Code
  • Interpreter
  • Merge / Versioning Version Trees
  • Python3
  • Workings of List
  • Packages: Wrappers + new Versions (VTK7)
  • Script Conversion
  • Intermediate Outputs: reuse output module work?
    • would be useful for both workflows (run to a particular checkpoint)
    • and useful for notebook interfaces
  • SQL backend: does it allow for concurrent access?
    • Yes, real-time collaboration work for concurrent single vistrail access
    • Should be able to access different vistrails at once without issues

April 19, 2017

Updates

Items to Discuss

April 5, 2017

Updates

Items to Discuss

March 8, 2017

Updates

  • [RR] Added a looping indicator on modules #1217
    • This makes it easier to understand what's looping, no need to follow edges and count
  • [RR] Refactoring CLTools for integration in Docker package
  • [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)

Items to Discuss

  • [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers #1216
    • Can we move this up to the renderer modules?
    • Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)

February 24, 2017

Updates

  • [RR] Rich-text module documentation (#1210)
    • Went over packages and fixed format; maybe we should have a test to validate RST?
  • [RR] Single-instance shutdown with --remote-shutdown (#1204)
  • [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment

Items to Discuss

  • [RR] There are issues with list-depth
    • Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)
    • Issue with looping groups

January 25, 2017

Updates


Items to Discuss

  • Scaling VisTrails
  • sql-alchemy, uuid, bundle branches/merges

January 11, 2017

Updates

  • [RR] New VisTrails version in Debian [1]

Items to Discuss

  • [RR] Collaboration with school of medicine to port HiC-bench to VisTrails
    • Have homemade workflow system based on R scripts and symlinks
      • Each steps has multiple sets of parameters
      • Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step
      • Gets all results in directory structure step3-parameters/step2-parameters/step1-parameters/files
    • VisTrails improves provenance capture and makes editing pipelines easier
      • Need better support for aggregation (groupby operation might be good enough; pandas?)

2016

December 14, 2016

Updates

  • [RR] Single-instance code: over-engineered, but will work
    • (Have to avoid race conditions on startup, and OSes offer very different locking primitives)

Items to Discuss

  • Email from Matt Dirks: combobox should work, ask for more code?
  • Email from marin.nl:
    • looks like they would benefit from new interpreter
    • how to store their consolidated provenance when they exchange .vt files around?

November 30, 2016

Updates

Items to Discuss

  • [RR] Working on single instance code for MARIN (#1205)
    • Remote shutdown ready (#1204)
    • Still improving it
    • Want to return something more structured to the client (success/fail code, stdout, stderr)
    • Should be usable without Qt
  • [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason
  • [DK] Working on batch execution using console_mode
  • [DK] Settings aliased parameters when using core.api?

November 16, 2016

Updates

  • [RR] Issue #1200 from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)

Items to Discuss

  • Email from Pieter: executing pipelines with parameters from command-line using single-instance
    • Seems to work, main instance gets full command-line and alters configuration temporarily
    • Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful
    • Actions:
      • Can add switch to kill main instance
      • Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails
      • Add client/server code to core that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like gui.application.VistrailsApplicationSingleton#noninteractiveMode())?

November 2, 2016

Updates

  • New version 2.2-2 of Debian package, still wrong copyright, and broken bundle installation
    • Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch
  • Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)

Items to Discuss

October 5, 2016

Items to Discuss

  • working toward a 3.0 release; project board
    • unique ids
    • bundles
    • GUI fixes
    • wrapping updates
    • python3
    • interpreter?
  • command-line input parameters to provenance
    • [DK] we don't create a persisted workflow from those input params, i think
    • [DK] these are already defined as aliases so we treat them differently form a normal input param
    • [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases
    • Do the vistrails variables values get stored in provenance? (what if values change?)

September 14, 2016

Updates

Items to Discuss

August 31, 2016

Updates

  • [RR] Went over vgc.poly.edu/vistrails.org config:
    • Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
    • Fixed usersguide scripts (won't disappear again)
    • Moved some websites over to HTTPS
    • Websites
  • [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs (currently there) and users' guide under the same RTD "project"
  • [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that

Items to Discuss

  • [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
    • Get rid of more services? (svn, trac, ...)
  • [RR] Notebook execution works. How do we support more languages (than Python)?
    • Need: detection, metadata reader, code to input values
  • [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?

August 17, 2016

Updates

  • [TE] Fixed parameter order upgrade bug that breaks VTK examples.
  • [TE] Matplotlib and sklearn are exportable using module-to-script api
  • [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.

Items to Discuss

  • Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
  • Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
  • [TE] Wrap-up notes

August 10, 2016

Updates

Items to Discuss

  • [TE] Python Wrapper and scripting
    • All changes merged into python-wrapper branch
      • Tests pass
      • Still need to update sub-branches
    • Minor fixes
      • vtkPlot3DReader upgrade bug
      • Added tuple exporting
    • Most of vtk examples can be exported example
      • Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
        • Could be a bug in the old vistrail, but not sure
  • [DK] Subworkflows and uuids

August 3, 2016

Updates


Items to Discuss

  • [TE] Wrapper patches now stored as python code
    • Connection variables now lowercase
    • Can we use code as output variables?
      • E.g. "vtk_dataset_mapper.GetOutput(0)"
      • Could mutate output when output method is only supposed to be called once
      • Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)

July 27, 2016

Updates

  • [DK] DB bundle serialization implemented

Items to Discuss 

   def vtkInstance():
       self.vtkInstance = self

July 20, 2016

Updates

Items to Discuss

  • [TE] New patch method for python-wrapper example
    • A patch is a string template with self/input/output variables for patching a class method
    • Patches are stored in xml specification
    • Patches are assigned to methods in ClassSpec's
    • Used both for execution in vistrails and when exporting as script
      • Executed with "exec" when run in vistrails
      • Turns patches into natural calls in script
    • Wrapping VTK library no longer needed
      • Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)
    • Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)
    • Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)
    • TODO's
      • Add to export_as_script
      • Patch Modules that represent Python functions as well
  • [vistrails-users] Read data from SQL example?

July 13, 2016

Updates

  • [DK] bundles on sql-alchemy branch
    • Bundle, BundleMapping, and BundleSerializer
    • New BaseSerializers to deal with versions of directory, zip, db serializers
    • Can register the BundleSerializers with the base serializers
    • Translation between bundle versions
    • Bundle.add_object "just works" if an appropriate mapping was registered

Items to Discuss

  • [TE] Simplified running module using API for scripting example
    • API Module can be executed using compute method
      • Takes list of outputs to compute + inputs as keyword arguments
  • Status of general cell synchronization (vistrails-users)

July 6, 2016

Updates

  • [DK] Working on bundles

Items to Discuss

  • [TE] Using vistrails API in scripting example 1 (API with VTK) example 2 (DownloadFile only)
    • Call module in script using vistrails API
    • Can be used when no export method is defined.
  • [TE] Server news bug Critical?
    • [RR] No, unreleased
  • [DK] New bundle organization
    • BundleMapping contains individual mappings from raw objects to BundleObj
    • Must be separate in order to have in-memory, separate from serializer settings

June 29, 2016

Updates

Items to Discuss

  • [TE] Export VTK as python script
    • Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) example
    • Preludes (imports) still need improvements
    • Works with vtkInteractionHandler (clipping plane)
    • Resulting clipping plane is incorrect? (supposed to hide one side of the plane)
    • vtkRendererOutput Uses VTKRenderWindow as output
      • Should support different output modes?
      • Does not work with multiple outputs (Next cell is displayed when current is closed)
      • Uses wrapped vtkInstance (as vvtk)
        • Create new patch system with code that can be used both in vistrails and in a python script?

June 22, 2016

Updates

Items to Discuss

  • [TE] Fix for upgraded analogies (PR #1177)
    • Uses module location to find module remap (The best we can do right now?)
    • Also fixes remapped port names for connections to existing modules
    • May still fail on complex upgrades like adding a function on an upgraded module that has a different port name
  • [TE] Working on exporting wrapped libraries (VTK)

June 15, 2016

Updates

  • [TE] Added depth logic to python script exporter (full commit)
    • Fully implements connection merging and list looping

Items to Discuss

  • [TE] Making analogy work on upgraded workflows (#1175)
    • Analogy only works We need to use the non-upgraded action chain
    • But pipelines need to be valid (matcher uses module portspecs)
    • Tried to get upgrade remap from upgrade action, but this seems impossible
    • Use module location instead?
      • Location should be identical for one-to-one module upgrades

June 1, 2016

Updates

  • VisTrails 2.2.4 announced (finally)
    • 5 survey replies
    • 47 downloads last week

Items to Discuss

  • [TE] Export VTK as script (python-wrapper)
    • to_python_script
      • needs the pipeline (or at least the connections)
        • Should not generate code for any possible VTK method (exporter detects which ports is used in the source)
        • VTK needs to know connection order
        • Adding functions directly to the script results in nicer code
    • Add support for multiple input connections
    • Add support for list looping

May 25, 2016

Updates

  • Mailing-list fixed
  • Getting stats from the "server news" mechanism (via parser)

Items to Discuss

  • [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)
  • [TE] Union ports
    • Show union ports in pipeline (DEMO)

May 18, 2016

Items to Discuss

  • [TE] mailing list still down - Julio working on it
  • [TE] Improved union port
    • New schema 1.0.5 adds portitem.union string attribute
    • Works with pipeline view, parameter exploration view, and mashup view
    • Should work well for matplotlib
    • Use with VTK ports?
      • It uses [item_1,item_2] style ports usually with different tuple length.
  • [TE] Remove emacs dependency from db generate #1173

May 11, 2016

Items to Discuss

  • [TE] Union type #1172
    • GUI-based solution Branch union-port Demo?
      • Shows single port for all ports with same sort_key
      • Select specific type when adding/showing
      • Only one type should be visible on the module at any time
      • Making tuple parameters union types is probably too complicated
      • Should work with alternate specs like matplotlib (if sort_key is defined correctly)
  • [TE] Feature suggestion: Let connections connect to functions/parameters
    • Show functions and parameters as ports and toggle port visibility directly for functions and parameters
    • Use separate views for ports and functions (like before)
      • Allow rearranging of functions
    • Pros:
      • Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
        • We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.
        • Makes List module simpler
      • Connect to parameter in tuple
      • Functions that are visible on the module can connect to an actual function, and not just to the first one it finds.
      • Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.
    • Cons:
      • Interface more complicated and crowded
      • More difficult for users to grasp?
      • No immediate use case
      • Makes no union type for tuple parameters a bigger problem.
  • [TE] Publish user survey?
    • No response from vistrails release yet
    • Not announced on vistrails-users yet.

May 4, 2016

Updates

  • VisTrails 2.2.4 released

Items to Discuss

April 27, 2016

Updates

Items to Discuss

  • [TE] Wrapping Bokeh
    • New method type "operation"
      • Transforms Figure->Glyph->Output into Glyph->Figure->Output
        • Glyph sets function name and arguments as output
          • Not serializable, but should always be computed together with the subject of the operation.
        • Figure applies Glyph operation on itself
      • Useful for matplotlib?
        • matplotlib currently find actors by searching upstream?
        • But this is more difficult since matplotlib is not function-based.
    • New "union" port type
      • Turns "x(float), x(list)" into "x (float, list)" and uses first set value
        • Uses only first value in tuple
        • Bokeh have many properties that support (list, string, Float)
      • Does not work with depth=1 types (e.g. Float of depth 1)
        • Turn Depth 1 types with known type into lists?
  • [TE] New bugfix release
    • Finalize survey

April 20, 2016

Updates

  • Merged upgrade fixes
    • Now has infinite loop test
    • loop test now correctly loops 5 times
      • remap_module used wrong (current) package version when creating ModuleDescriptor
  • Fixed multi-step group upgrades (PR: #1168)

Items to Discuss

  • New bugfix release
    • Merge usagestats
    • Need to finalize survey

April 13, 2016

Updates

Items to Discuss

  • [TE] Chaining upgrades done (#1164)
    • Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?
  • [TE] Iterating handle_invalid_pipeline (#1165)
    • Not based on chain-upgrades-loop
    • Cherry-pick test to preserve source?
    • Need maxPipelineFixAttempts option?
      • The code is now smarter about recursion, so max attempts will "probably" never be reached.
    • Need debug messages?
      • Would need to be moved into handle_invalid_pipeline

April 6, 2016

Updates

  • [TE] Query View improvements
    • Done except for displaying "Version Info" and #1164
  • [RR] Working on finishing new interpreter (finally!)

Items to Discuss

  • [TE] Chaining upgrades (#1164)
    • Try latest upgrade first, then previous.
    • If all fail, select latest upgrade?

March 30, 2016

Updates

Items to Discuss

  • [TE] Query View improvements
    • Unified do_version_switch and get_upgrades as validate_version()
    • Problems with query view using its own controller
      • Fixed module info not being displayed for query results
      • Still no version info for query results
      • Still an issue with version query view not being updated

March 23, 2016

Updates

Items to Discuss

  • [RR] Usage reporting almost ready to go
    • Need to record whether specific features are used: paramexpore, mashup, query
      • [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation
    • How to record size of vistrail?
      • [DK] IdScope already walks over vistrail, use that
      • but total number of modules and versions doesn't give an idea of average size of individual pipelines

March 16, 2016

Updates

Items to Discuss

  • [TE] Upgrades in Query Mode
    • Now using code similar to do_version_switch
      • Uses handle_invalid_pipeline
      • Differences: Not using current_pipeline, flushes changes
      • New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)
      • Integrate to avoid repeated code?
      • Use this for query mode?
  • [TE] VisTrails 2.2.4 ready for release?
    • tej package updated. Others?
  • [JF] upgrades and the version tree
  • [JF] provenance in Jupyter
  • [DK] bundles: #1092

March 9, 2016

Updates

Items to Discuss

  • [TE] Fixing search mode
    • String query now matches all by default
      • This caused pipeline queries to never match
    • String query can search: User/Notes/Tagname/Time/Module name
      • Should we search module names by default? (Disabled by default because it is expensive)
      • Fixed notes search
    • '>' parameter queries in documentation but has been removed
    • Fixed module matches for upgraded pipelines (By not upgrading pipeline)
      • Upgrades still a problem
      • Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)
      • [JF] Allow search on past tags
  • [TE] Create new bugfix release
    • Finish querying
    • Path issue on windows
  • [DK] Look into sqlalchemy and new bundle branches

March 2, 2016

Updates

  • [TE] Fixed cache bug with port specs
  • [TE] VisTrails Next
    • Most packages now work
    • No qgis for Python 3

Items to Discuss

    • Always dereference symlinks? #1146

February 23, 2016

Updates

  • [TE] Fixed PythonSource 100% CPU bug
  • [TE] VisTrails Next
    • Updated NOAA Web Service examples
      • Now uses REST service, tabledata, and Google Maps
      • Web Service replaced with DownloadFile

Items to Discuss

  • [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.
    • New 2.2 release soon?
  • [RR] Anonymous usage reports
    • Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features #1154
  • [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?
  • noWorkflow - provenance for python scripts
    • Recent work on tracking dependencies
  • Also integrates ReproZip with Jupyter
    • mybinder
      • Use with ReproZip to automatically build Jupyter notebook interfaces
  • Daves comments on Jupyter integration
    • It is possible to identify cell order on the server side
    • Not trivial to track order of cells on client side
  • How does Galaxy workflows relate to VisTrails?
    • Packaging in Galaxy is hard
    • Galaxy for biomedical workflows, difficulty making it general
  • Action items
    • How difficult to track cell dependencies in Jupyter notebooks?
    • Use noWorkflow to track cell dependencies?
    • How difficult to modify mybinder to use ReproZip packages?

February 17, 2016

Updates

Items to Discuss

  • [TE] Fixed Group memory usage
    • Modules in groups are supposed to be freed when done executing
      • Prevented by logger that stored references to the modules (fixed, now using id(module))
      • Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)
  • [TE] VisTrails Next
    • Removed deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • PythonSources will be upgraded
      • Fixes for Python < 3
      • fixes for VTK < 6
    • TODOs
      • Remove RemoteQ
        • Port Hadoop to tej (Already have pbs)
      • Merge package wrapper
      • Finish SQLAlchemy branch
      • Finish new bundle branch
    • [JF] Will contact Jeff and Matthias about moving to Python 3
  • [JF] Query by Example is broken
  • [JF] VisTrails Data Cleaning Project
    • Use VisTrails to process and explore task-specific data while keeping provenance.
    • Remi's Idea
      • Use VisTrails as the backend in a Jupyter notebook
      • Overcome problems with executing cells in random order
        • Interesting research topic, but little interest has yet been noted
      • Interesting to study how such a notebook would be used (Is there notebook usage data available?)
      • Impose restrictions to notebook cell boundaries
      • It will only work for functional operations
      • Can we fix, or detect, non-functional (mutating) operations?
      • Analyse code and data flow and see if it mutates data in unexpected ways?
      • Mutable operations can be created in many ways in Python
  • Next meeting Tuesday 11:30

February 10, 2016

Updates

Items to Discuss

  • VisTrails Next
    • Test suite passes
      • Python 3 less deterministic, module upgrade order random.
    • [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...
    • All of our packages should run fine :)
    • VTK thumbnail comparison works
      • Rendering bug on some systems
        • QVTKCell now inherits QGLWidget
      • Old thumbnails had wrong height due to spreadsheetwindow being too small
        • Fixed by regenerating thumbnails
      • Silenced deprecation messages when generating modules
      • Many new classes in VTK 7
    • Remove deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • Will break existing PythonSources
        • Try to upgrade?
      • Old-style packages
        • Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
          • Read info.py without importing package?
      • Fixes for Python < 3
      • Fixes for Qt < 5
      • fixes for VTK < 7
      • Remove self port?

February 3, 2016

Updates

Items to Discuss

  • VisTrails Next
    • All tests but thumbnail comparison passes
    • Still issues with gui and more packages needs to be tested
    • PythonSource and MplSource upgraded with lib2to3
    • Keep contrib packages? Update?
    • VTK 7 is released
      • Supports Python 3
      • Will hopefully get into conda and others soon
      • Deprecates a bunch of modules
  • Travis builder for VisTrails Next
    • Runs!
    • Uses conda
      • Many levels of requirements
        • apt packages
        • conda packages
        • python packages in requirements.txt
        • python packages in setup.py
      • Can probably drop conda. Default builder does not have Qt5, but beta builder do!
      • But useful if we want to use conda in our own build process.
    • Lots of errors I do not get locally?
  • User Survey
    • Have you used any alternatives to vistrails?
      • Relevant? More options?
    • How often do you use any of these particular features in VisTrails?
      • Add/Remove alternatives?
    • How important would you rank the following potential improvements to VisTrails?
      • Add alternatives?

January 27, 2016

Updates

  • New interpreter
    • Working prototype soon
    • Looping not completely done (and port combination mode: pairwise/cartesian...)
    • Need to integrate with VisTrails
      • Compatibility with existing modules
      • Use log controller
      • Want to cache at multiple levels (whole stream/single element of stream)
      • Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)

Items to Discuss

  • User Survey
  • ZMQ failing on windows (vistrails-users)
  • [TE] Python 3 branch
    • Testsuite almost passes
      • Failing are unicode filenames, some users guide examples, and thumbnail comparisons.
    • Fixed issues
      • Signatures are now bytes (convert to string to show in widget)
      • Cannot compare different types, e.g., (None < 1)
        • PortSpec.depth and min/maxConn could be None. (Backport!)
        • cell.row could be None
      • Lots of unicode/bytes fixes to serializations
        • We have encryption implementations in the code! (keychain)
      • Lots of small fixes to package method differences
    • Current issues
      • logging widget empty?
      • version tree signals messed up
      • provenance view legend icons black
      • Text do not fit in some widgets
      • VTK thumbnail comparison fails
        • VTK Cells have some faces in wrong direction?
        • Because running in VM?
    • Installing on Ubuntu LTS/Latest Mint
      • Install latest matplotlib from pip
      • Install VTK 7.0.0rc2 from source
      • No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)

January 20, 2016

Updates

Items to Discuss

  • [TE] Backporting features from Python 3 + PyQt5 + Qt5
    • Merged unicode with master (unicode)
      • One unicode test failing
      • Problem saving to xml
    • Used python-future to run futurize stage 1 to update legacy (pre 2.6) code (unicode-futurized)
      • Updates print to functions 'from __future__ import print_function'
      • Updates exception calls to use 'as'
      • removes most usages of 'long' (mostly used unnecessarily)
        • Only needed for 32-bit support
      • Fixes relative imports (from __future__ import absolute_import)
    • Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 (unicode-futurized-qt5)
      • Updated all signals to new-style
  • If we go 2 to 3, this could break packages
    • means we should schedule any package format changes for the same time
    • e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* #1002)
    • any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)
    • putting package identifier as xml?
      • Python not so bad, plus some stuff in there is Python code (checking dependencies)
    • version the APIs?
      • RR planning on versioning vistrails_module.Module, no point versioning the whole package?
  • file formats on 2 to 3:
    • no real translation/upgrade, but non-ascii values that work on 3 won't load in 2

January 13, 2016

Updates

Items to Discuss

January 6, 2016

Updates Items to Discuss

  • Updating VisTrails dependencies
    • Successfully built Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7
      • VTK 7 is the first supporting Python 3 but is only in release candidate phase.
        • Next version of VTK 6 is rumored to support Python 3
        • There is a development branch of VTK 6 supporting Python 3
      • Can travis build this?
        • We can skip tests for WebEngine and VTK if needed
    • Python 3 showstoppers
      • No MySQL-python (sqlalchemy supports mysql through other drivers)
      • No suds (there are active forks but we could also drop support?)
  • Options for python2/3 support:
    • Interpreter could be python2/python3 based on packages
    • six layer works but not the easiest to maintain (e.g. in reprozip)
  • Drop support for win-64
  • Update copyright headers to 2016

Older meetings

Retrieved from "https://www.vistrails.org//index.php?title=Development&oldid=14736"