Development

See also the Github wiki

2015

January 28, 2015

Updates

• T Caswell to come visit on Fri 6 to discuss wrapping work

Items to Discuss

• [TE] New VTK wrapping
  • Current code by DK seems a good deal faster
  • Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code
  • RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower
  • The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own
• Web crawler
  • Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"
  • RR would rather have the crawling be a job as far as VisTrails and tej are concerned
  • The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize
    • Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached

January 21, 2015

Updates

Items to Discuss

• [RR] Unified wrapping method discussion #991
  • TE to work on reusable method with intermediate representation, starting with VTK
• [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)
  • AM's examples are enough
• [AM] scikit-learn package is done, merge it in? #955
  • RR will merge
• [RR] What should copyright headers say? #994
  • Let's keep everything in there: Utah/Poly/NYU

January 14, 2015

Updates

• [TE] Working on classifier
• [RR] Scripting integration, work in progress

Items to Discuss

• [RR] Unified wrapping method discussion (#991)
  • Let's talk next week, [AM] and [DK] are not here

January 7, 2015

Updates

Items to Discuss

• make sure that we address critical issues, questions, and pending review branches in a timely manner
• scripting support
  • [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow
  • [RR] can translate from workflow to script, working on script to workflow
  • will work for parameter value changes, structural changes require changes to the annotations
  • need to publish best practices here
  • would be cool to do looping in scripts (easier interface than with workflows)
• notebook support (convert form notebook to workflow)
  • RR will sync with FC on this
• Issue with console in built-from-scratch
  • [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3
  • binary has old version of iPython -> 1.0.0, should we update?
• [TC] automated wrapping of numpy and scipy
  • discovered a bunch of malformed documentation in numpy and scipy
  • has github repo for vistrails tools
  • example modules wrap a bunch of R stuff (not baked in, just how things are)
  • will be pushing wrapping logic up
  • port names forbidden (window and domain)
  • have an import hook to get from yaml directly to VisTrails Modules
  • should work for any python modules with well-formed numpy docstrings.
• [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)
• [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)
• [TE] build and build scripts
  • completely automatic, buildbot
  • need to set the build machines for the environment we want for the binary
  • would virtualenv work here?
  • [TC] anaconda can pin versions, potential path to test different configurations
  • Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled
• sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07
• package issues (see Remi's message)
• [TE] Scope of tej
  • Support single ssh commands?
  • Queue can be used as a remote machine (crawler is using queue.call*)
• SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07

2014

December 17, 2014

Updates

• Lots of progress on sklearn package on Monday (#955)
  • AM ran into issues with port orders
    • any manually specified port order should always be left-to-right. perhaps don't change the default port order yet
  • sklearn modules uses different port shapes for its different types: Estimators (diamond), data (circle), parameters (default, square)
    • [RR] Looks good; generalize this?
    • do this for types of ports rather than try to infer for any port
  • Issue with Pipeline module: how to specify a variable number of inputs that need a specified order
    • DK wrote something with ConcatenateString for this
    • problem is you need a UI for ordering the connections (order-connections branch assumed order they were added)

Items to Discuss

• scripting support
  • how to go from one to another
• crawler progress [TE]
  • building a package to interact with the crawler
• Two week break, Next Meeting is January 7

December 10, 2014

Updates

• [TE] bundles
  • Finished command-line and core API for handling bundle data
  • controller has methods for adding data to bundle
  • [DK] why command lines? [TE] allow other tools like reprozip to build bundles
  • [DK] seems like could grow into a lot of command-line options
  • [TE] can also add the files as a directory
  • want to have call that allows user to specify what is saved in the bundle
  • [TE] should the user have access to add other parts of the bundle besides data? Probably not.
  • [TE] working on storing packages in bundle and loading them from bundle?
• [TE] web crawler
  • Using tej
    • Problems with closed sockets
    • No way to reconnect to queue
    • copy fails? TE to provide example to RR
  • crawler works by submitting a job as a start and stop
  • create a new completed job if it runs
  • [RR] want to be able to get result even if not completed
• [RR] web package for d3.js etc
  • [DK] looks good, works for me, github comment on supporting more specific use cases
  • RR working on code editor

Items to Discuss

• Should source modules' code ports be control parameters? #973
  • think about how to support the raw cases where the source is computed upstream
  • static typing
• unicode/string issue from BNL?
• Remove Serializable: #966
  • shuold we also remove the serialize calls in basic_modules.py
• [TC] auto-wrapping all of numpy into VisTrails
  • looks like matplotlib scraping
  • ndarrays are labeled as lists, reason? no
  • parameterized types? (have depth, could have Float with depth 2)
  • do we convert the types? don't think so
  • TC will add depth 0 ndarray
  • override Module calls? potential issue with the depth computations?
  • numpy has broadcasting to do internal parallel

December 3, 2014

Updates

• [TE] bundles

Items to Discuss

• [TE] bundle extensions
  • jobs stored in 'jobs' file
  • Data stored as file references in data/ directory
    • Add with gui/api or a method like 'add File data to bundle'
    • Have some way to browse by directory
  • Packages stored in bundle
    • Add/delete package method
    • When loading, ask 'Do you want to import package in bundle' if not already existing.
    • [RR] need to fix reloading issues and conflicts with packages that already exist
    • [RR] Full name of a package can change (userpackages vs. vistrails.packages)
      • [DK] does this ever get saved anywhere? thought we only use "codename" which has no depth (dots)
  • Package data (Like a Web Service wsdl stored in packagedata/my.package.identifier/
    • Packages can add/remove data as bundleobj file references
    • [JF] Do we need a package-specific directory?
    • decision to use the data directory for everything, no package-specific directory
• Scripting support
  • [RR] pipeline->python translation needs some tweaking/testing, PR #950
    • python->pipeline translation not done, we have some old code but might not be salvageable
  • [RR] IPython notebooks integration still waiting for inputs/outputs definition in notebook format
    • we can probably use whatever metadata format in the meantime and reads theirs once it exists
    • send email to IPython folks
    • (the goal here is to wrap whole notebooks as VisTrails module, to string them together, still using IPython kernels to execute each notebook)
• integrating web things in VisTrails
  • [RR] wants to take a look at this but working through reprozip backlog

November 26, 2014

Updates

• [TE] improve-vt-bundle
  • .vt load/save is working
  • incrementally writing to vistrail file?
• [DK] add web inspector (merged)

Items to Discuss

• [JF] scope of filenames
  • [DK] what about copy-paste between vistrails? (hidden dependencies?)
    • [JF] shouldn't expect things to run here
    • need way for user to copy or extract from bundle
  • other option: have toggle to say that name is local to bundle?
  • determine API first
• [TE] job-fixes
  • RR working on improvements to job handling
  • [TE] what still needs to be done here?
  • [RR] bug where need to dedup jobs, two workflows with same job (same subworkflow signature) will store job twice in job monitor and will check for both separately, and when finished, assumes one is still running
  • make job a first-class entity, work independently of workflow/vistrail
• [DK] Move to a new version of Qt for improved well cell support?
• [DK] Working with web frameworks/apps in VisTrails
  • [RR] start a local web server for this to avoid some issues
  • way to move files computed upstream to a resource that the web server can access

November 19, 2014

Updates

Items to Discuss

• [JF] Upgrade tree (#949)
  • list view
  • every node is a new one
  • [JF] most natural way is to have same tag
  • [JF] cannot reselect the old version
  • Upgrade a node and have that change propagate
  • Meta-actions?
  • Save and save as tag actions (save moves the tag, maintain the history of a tag)
  • [RR] Suggest collapsing the upgraded node into the current node (hide the un-upgraded node)
• [JF] Ability to refer to file names relative to the vt directory (related to packaging) -- this issue has arisen in the context of ReproZip
  • flag in the File module to make things relative to the vt bundle?
  • [TE] import data into vistrails?, truncate based on a map from file locations to bundle/local directory locations
• [TE] job-fixes (#27) & tej
  • we'll keep the job_id for now; tej exposes it
  • changes to be merged in
  • tej works, needs testing on actual clusters (NYU?), change name?
• [TE] figured out how most of improve-vt-bundle is supposed to work
  • bundle has method to get_db_object, get_db_objs

November 12, 2014

Updates

• [RR] We moved to Github
• [RR] UV-CDAT moving to Github; UV-CDAT/VisTrails currently marked as fork
  • This means that opening a PR for a branch in that fork will target VisTrails/VisTrails by default
  • People have to select UV-CDAT/VisTrails as base every time
  • Unmark it as fork if accidents happen? (like #956)

Items to Discuss

• [RR] We are on Github!
  • Please check github-wiki/Development for some guidelines on development
  • Most importantly, be sure to check out tickets marked question or pending-review
• Email notifications from Github
  • Right now vistrails-dev is watching the repository
  • This means the list gets an email whenever anything happens (except from pushes)
  • You can unsubscribe from issues, but you'll still get an email through the list
  • You will receive everything twice, and receive notifications from your own actions, because of the list
  • Is asking developers to watch the repo enough?
• [RR] BuildBot/Github interaction status?
  • [TE] buildbot is configured to use github with "git:" protocol.
  • [TE] Web hook is not working, hook is using "https:", but repository is using "git:"?
  • [TE] Binary nightly build working for 2.1, will soon be merged into master
• [TE] Atomicity in Job Submission state graph (#27)
• github shorthand? "fix" keywords close issues
• feature branches move via pull requests
• Different releases: windows (4x), github and sourceforge
• Nightly releases? (should we push these to sourceforge?)

November 5, 2014

Updates

• [TE] setting up buildbots to build the binaries
  • solved most of the issues
  • working on Windows buildbot

Items to Discuss

• [RR] Job submission
  • Flowchart
  • JobMixin doesn't work for Rémi's use cases
  • Need to check on job more than once (based on signature, if server removed job, etc.)
  • [TE] what if job failed or job has been submitted but isn't running?
  • Pull Request: #27
• [RR] Github migration
  • Notification on the Github web page to let users know not to open new tickets for a couple of days
• scikit-learn package:
  • Jorge wrote a package about this in 2012 at ORNL, need to talk with him
• [DK] Interaction provenance
  • Current able to persist the state of a cell (e.g. Save Camera) if they align with parameters
  • How would we capture interactions with the cells as they happen?
    • May not correspond to a compute-time parameter (e.g. mouse drags)
    • Capturing low-level user interactions may be problematic if we don't know how an interaction sequence translates to a meaningful action
  • [JF] try to link back to parameters (otherwise, how do we have anything meaningful)
  • [JF] Granularity issues
  • Maya: hierarchy of actions, use geometry to try and infer this (analyze object and look for things that are separable)

October 29, 2014

Updates

• [RR] Build scripts
  • Tommy working on release scripts (updating version numbers in build scripts and splash screen)
    • [TE] prepare_release.py updates version numbers, branch, hash, and splash (requires inkscape)
  • sdist and wheel ("PyPI" packages) fully automated, have builder
  • conda too, builder
  • Windows build script needs lots of binaries in fixed locations, e.g. C:\Python27, C:\Users\vistrails\src\vtk, ... build difficult to reproduce
    • could use conda here, but problems with the script
  • Mac OS ???
    • [TE] Built with "make_app"
      • Needs installation of qt, vtk, etc., to build.
      • Building usersguide should be more contained
    • would py2app work on a condo-installed VisTrails?

Items to Discuss

• [RR] Weird error in UV-CDAT has me baffled (#843)
  • [DK] check on packages.uvcdat (generic name, could be overloaded) and/or the Variable name (also generic, could be overloaded)
• [RR] JobSubmission and RemoteQ:
  • some minor issues with the way things are stored, mostly because uuids and other items should be removed
  • [RR] caching output values in the vistrails was not a good idea
  • big issue is still that RemoteQ is not best solution
  • keep mapping on server to map something in module (pipeline_signature) to job and output
  • VT keeps track of which workflow submitted job
  • start_job method just calls arbitrary line of code to start job
• github migration
  • [RR] to send github an email to ask what the best method might be
• problems with terminator.vt example?
  • potential interaction handler issue?
• vt using temporary directory to store files
  • disk cleaners can wipe these directories (USGS reported)
  • question is whether disk cleaners *should* be doing that...

October 24, 2014

RR and TE meeting about job submission & RemoteQ status in VisTrails

http://www.vistrails.org/index.php/JobSubmission

Summary:

• some details to be fixed in RemoteQ package
• job result cache shouldn't be in vistrail
• the RemoteQ library is a problem since Troels doesn't maintain it

October 22, 2014

Updates

Items to Discuss

• [TE] improve-vt-bundle
  • lack of documentation
  • two layers of serializers
  • serialize in more on-demand manner
  • does auto-save save everything
  • database bundles: try to tie into sqlalchemy
• Github migration
  • decide on how to move the open tickets
  • 26 first issues are already used, can just add original ones
  • Rémi needs access to the trac database (vistrails.org)
  • if possible, move everything
• [TE] New 2.1 release to fix examples?
  • upgrades had a bug so could affect existing (older) vistrails
  • changes to script commands
  • remove tilde files in release
  • [RR] is release process automated?
    • [TE] run scripts in the git repository
    • putting release note inside vistrails
  • TE to look into which steps are more difficult to change
• [TE] job-cache feedback?
  • shareable jobs, but jobs are not persistent
• [TE] update Web Services examples
  • look into RESTful services?
  • should we deprecate the old examples?
  • check the user's guide and edit if necessary
• [RR] VisTrails meant to written by developers and used by users
  • writing packages is a bit hard
  • developers want to use their own editors, specify which versions should be used
  • still have to write your own wrappers
  • if developers just want to integrate their tools, can be difficult
  • get python program drag 'n drop into python source

October 15, 2014

Updates

• [TE] job-cache (merged)
  • Store jobs in vistrail
  • Fixed jobs in automatic loops
  • [RR] Probably shouldn't be in VisTrails
    • RemoteQ/BatchQ are nowhere to be found (except VisTrails's binaries), install procedure doesn't work, and unsupported
    • Not really used by anyone else ("proper" job submission, e.g. clusters, use PBS/...)
    • Sharing job status in vistrail is unsafe and WILL break if people make changes
    • Having that in the vistrail XML for the next release would force us to support it/provide upgrade paths
    • Suggest removing for now, while we think about this more

Items to Discuss

• (from last week)
  • Use Github and pull requests?
    • Have to find a way to import tickets from Trac
  • Use a separate fork for UV-CDAT's vistrails?
  • Check with Claudio
    • Claudio says yes
• VisTrails and ViDA?
• Notes from meeting with F. Perez
  • Use whole notebook as modules in VisTrails (don't necessarily need to deconstruct cells)
    • IPython providing "kernels" (execution engine, behind notebooks) for 21 languages
    • connect notebooks that are of different languages
    • reduce serialization issues
    • implicit if two notebooks need to move outputs into inputs
    • They are discussing concepts of "notebook inputs" and "notebook outputs" right now
      • Kind of like our mashups
      • nbparameterise thread
    • Inter-notebook interactions probably not envisioned (but they'd like to be able to point to VisTrails for that)
    • [DK] interface workflow: how do users edit two different notebooks and connect in VisTrails (seems like a lot of windows)
    • restrictions on what can be serialized? notebooks need to be serialized properly
    • can you get a notebook without the stored outputs used by nbviewer?
  • Using VisTrails workflows and capabilities from IPython
    • Generic Python API + some rich text augmentations for the notebook
    • Using VisTrails from notebooks using non-python kernels? IPC between kernels is not a goal for IPython folks
  • Will try to follow their mailing-list and gitter

October 8, 2014

Updates

• [TE] Store one job monitor per vistrail (almost done)
  • Easier to share running jobs
  • Jobs are shared between all workflows in a vistrail
  • use new bundle structure
  • what happens with local jobs?
    • doesn't really make sense to open a job started locally on another machine

Items to Discuss

• [RR] Use Github and pull requests?
  • BNL uses them, UV-CDAT switched to that to
  • Development Branches is hard to keep updated and offers no categorization/sorting
    • create pull request when new feature is started, do not have to wait until branch is completed
  • Also, use separate fork for UV-CDAT's vistrails? UV-CDAT/VisTrails is marked as fork but can have a different set of branches
    • vistrails.org might be the only UV-CDAT submodule that can't be served over https (see emails)
  • user permissions?
  • Juliana, thoughts on moving vistrails repository to github?
• need to keep testing 2.2 features (e.g. automatic looping, new widget interface) so we can prepare that release
• deprecation flag/error for old constant creation methods

October 1, 2014

Updates

• [TE] Graphicsview font issues solved
  • Preventing DPI changes on windows and using fixed values for module size calculations
  • Windows fonts always look bad

Items to Discuss

• [RR] To review&merge: constant-classes, invalid-pipeline-hides-package-errors_/loops (#901), version-flag
  • constant-classes: get rid of dynamic generation, use metaclass for constant class
  • version-flag: add --version flag back on argparse parser
  • invalid-pipeline-hides-package-errors: switch to invalid pipeline and will try to fix it, handle upgrades
    • actual error is not showed
    • get into a deep recursion? upgrade to package and back
• [RR] JobSubmission package status (does it work? is it useful?)
  • should be able to remove it
  • [TE] different things to control batch job
• [TE] buidbot web interface got hung up
• [TE] New features in 2.2 release (please extend/update)
  • [RR] New features? Maybe we should maintain a CHANGELOG file in the repository
  • Inline widgets ready? Need to test
  • output modules
    • incremental change where we keep both cells and new output modules, then phase out output modules?
    • expose configuration keys as ports
• Changes to interpreter
  • Simplify/Have different interpreter
  • Support different environments
  • Move update method from Module to interpreter
  • Merge multithreaded interpreter
    • Automatic loops/streaming should work during incremental transition
  • Different modules will support different modes of execution
    • have documentation for the best practices
• [TE] Error setting current_pipeline_view=None (fix)
• [TC] have different namespaces with same modules
  • using numpy docs to do most of the code generation

September 24, 2014

Updates

• [TE] rewrite-startup integration finished
• [TE] UV-CDAT bugfixes finished
  • need to make sure that the Aashish and/or Charles are aware the changes have been made
  • [TE] notification on github

Items to Discuss

• [DK] parameterExploration flag: just run the latest of the selected version
  • may be nice to have a way to set parameters directly from the command-line like we do with output variables
• [RR] code issues found using pycharm
  • there are a lot more issues than that, cf pylint
  • use pycharm IDE or install pylint, pychecker to avoid committing basic mistakes
• [TE] QGraphicsView size issues between Linux/Mac
  • the text size is different even for the same fonts
  • matplotlib ships with three fonts
  • has logic with rules to determine what the closest fonts
  • Qt has global configuration that can be hard to override
  • terminator example has 15 parameters that are editable (adds 20+ pixels and messes up layout)
• [RR] all tests pass on unicode branch
  • will try to keep in sync
  • [TC] lots of people with strange paths need to use
• [JF] back and forth with scripts
  • cloud execution
  • have more rules for certain environment
  • simplify the interpreter, have a different interpreter
  • create a new direction for a restrictive environment
  • have documentation for the best practices

September 17, 2014

Updates

• [TE] Fixed database from command-line, server mode, command line flags
• [TE] Fixed slow view switching

Items to Discuss

• [RR] unicode and python3 compatibility
  • [TC] Pull request doesn't change any behavior (text_type, or str)
  • [TC] Many changes from py3 were backported to py2, no 2to3 conversion
    • difference between text strings (always unicode) and byte strings (which are the default in py2)
    • six.text_type which on py2 aliases to str and on py3 aliases to str (runs through unicode object initializer)
  • make sure to import unicode_literals from __future__
  • [RR] more complicated than that. Let's convert everything to unicode in one go, and fix issues
    • [RR] issues with XML (de)serialization, locators, enum, configuration
  • look at a 2.2 release before adding unicode support (unicode may need a while to work bugs out)
  • [RR] don't move on Python 3 before unicode is supported. Probably no need to bring in six right now
• [TE] Parameter Widgets (function-tooltip) ready to merge
  • LOD error
• [DK] improve-vt-bundle branch
  • db/services/bundle.py exists on its own, need to link rest of code to use bundles
  • sql-alchemy waiting, work on bundle first
• [TE] adding command-line options to the documentation, update in one place now (used to have pdf and HTML)
• [RR] Version flag missing in new startup, branch to add back, but argparse messes up the formatting
• [RR] Trying to load the same package twice or different versions of the same package breaks VisTrails
• [CS] Hire a couple of people for UV-CDAT, need someone to bring them up to date
  • think about projects for them
  • [DK] we are not sure the direction UV-CDAT will take after 2.0 release

September 10, 2014

Updates

• [RR] Backported tabledata package: https://github.com/remram44/tabledata-backport
• [TE] GUI Improvements
  • Added type checking in parameter view
  • Switch to workflow info when no module selected
  • Improved fitToView (used a square setting?)
• [TE] Edit widgets
  • Hide when zooming out
  • Improved proxied widgets
  • Improved selection border
  • Profiling
    • Nothing major for edit widgets
    • Fixed mark_list_depth calls when switching versions

Items to Discuss

• Workflow longevity: mta example no longer works
  • Fernando suggested offering an option that saves the package (or a description of the package) together with the workflow
  • [DK] I agree, this had been talked about with the generic bundle support
  • [TE] Everything stored in a zipped vt file, could also store to a directory
    • [DK] some support for this
  • [DK] to look over old code and point out what needs to be done for bundle support
• [RR] maybe work to distribute packages independent of VisTrails (or at least make moving between versions of packages easier)
• [RR] VisTrails integration with ReproUnzip: https://github.com/ViDA-NYU/reprozip/issues/69
  • [DK] possible to do python install using smarter package installers?

September 3, 2014

Updates

• [TE] Fixed 1.0.3 schema and log upgrades
  • [TE] Make sure that 1.0.3 has the right items across versions
  • Keep startup only in master since it is only needed for upgrades

Items to Discuss

• [TE] Edit widgets demo
  • don't show name of port for constants
  • boolean widget doesn't look great (still native widget)
  • different layout on Linux than Mac/Windows due to font?
• [DK] UV-CDAT build:
  • Linux build and check for scipy issue (#590)
• python3?
• [TC] Good way to wrap up userpackages?
  • using conda package manager (environment has directory and hard-linked into, need to be able to hard link into environment)
  • [RR] is working to do a standard python package to recognize these automatically
  • For userpackages, can configure userPackageDirectory to point to a hard-linked environment
  • Preferences -> Paths & URLs -> Local Packages Directory
  • everything to be in a "bubble" at BNL
• [ED] Parameter exploration:
  • one variable that I care about but plot as 5x3 not 15x1 or 1x15, enhancement request
• [ED] Series of Qt widgets wrapped into VisTrails modules
  • subclass SpreadsheetCell, Qt widgets use matplotlib
  • [ED] to provide a demo on what is going on trac

August 27, 2014

Updates

• [TE] VisTrails 2.1.4 released
  • [RR] Will update PyPI and Conda. Note that this currently uses its own branch for 2.1.x (deployment/v2.1), will be easier in 2.2

Items to Discuss

• UV-CDAT Bug #520 and #557: make sure we don't show errors for packages that don't load (put these at info level)
  • [RR] already created a branch with this name
• matplotlib and UV-CDAT: should be able to include freetype in binaries
• Bug 886:
  • merge in this branch, perhaps not 100% ideal but still better than what we had
  • problem is that you sort of want to be able to do a bunch of actions and then raise InvalidPipeline so you don't have to do multiple try-catch
• Bug 892:
  • not pythonic to forceably squelch propagation
  • should be a configuration option to do this
  • add a filter to the root logger so that VisTrails messages can be controlled
  • will think about this more
• [TC] blocker on python3 compatibility?
  • [RR] unicode support should probably be done first: unicode from all the files, weird casts everywhere in the code because we used Qt before api v2
  • from future unicode literals and str type makes the right thing happen on 2 and 3?
  • BNL really wants 3 support, willing to work on
  • Issues are #673 (unicode) and #674 (python3)
• Relayout programmatically?
  • layout_modules() in controller

August 20, 2014

Updates

• [TE] Parameter Exploration bugfixes
  • "Create Version" spreadsheet button fixed
  • Missing date attribute in xml caused errors
  • Issue opening vtl links

Items to Discuss

• [RR] Pending branches:
  • pipeline-validation (#886)
    • (dep for fix-open-workflow (loading a workflow keeps the locator, so saving will overwrite))
    • Ok, will merge
  • logger-propagation (#892)
    • Might be a better way? Discuss this more?
  • hide-missing-dependencies (#895)
    • Ok, will merge
• [TE] New bugfix release? - Parameter Explorations and wrong usersguide in Mac binary
• [TE] Module edit widgets demo
  • start with tooltips for type names rather than showing them right now
  • give warnings on input validation -- add to trac (#898)
• [Thomas] matplotlib is meant to be used the other way
  • [RR] problem: this is not the way our other plotting packages work (including VTK: data -> data -> renderer -> cell)
  • [RR] we've already released a matplotlib package (data -> plots -> figure -> cell) and can't really turn it around
  • [RR] we all agree that current code is hacky (but works), see #685 then #396

August 13, 2014

Items to Discuss

• [TE] FileOpen event on Mac (#889)
  • We can pass argv to QApplication
• [TE] 1.0.3 schema differences (#894)
  • We can restore schema 1.0.3 on master from branch v1.2

August 6, 2014

Updates

• [DK] bug fixes, improve-vt-bundle, uvcdat

Items to Discuss

• [LB] LSU, instrument building, tomography, interferometry system
• [TC] image registration/processing, infrastructure at BNL
• [ED] working on data analysis at BNL, hoping to leverage VT for data provenance, user activity
• [LB] tomopy and VT integration
• [TC] how to integrate tomography software
• [LB] goal to communicate with others, data security for crowdLabs?
• [LB] improve parameter exploration
  • [JF] different ways: whole workflows, loops inside workflows, multiple threads, implict looping
  • [TE] add new examples of saved parameter explorations, different versions
• [RR] Bug #886 ("from_root" parameter for change_selected_version())
• [RR] API redesign, example: notebook, target: notebook
  • [DK] looks good, nicer string rep, improve access to and use of results is important
• Jeffrey Guenther - list handling

July 30, 2014

Updates

• [DK] UV-CDAT animation, sql-alchemy branch update

Items to Discuss

• [TE] parameters in workflow - demo
  • show functions on mouse_over
  • move set functions to the top
  • inline parameter widgets
    • show/hide all
    • toggle in ports list to show/hide these widgets
• API design wiki page
  • vistrails.api versus vistrails.core.api
  • API is currently stateful, would like to change
  • API calls using module names, parameter names instead of ids
  • tighter integration with iPython (what were ideas from meeting last year with iPython devs)?
• argparse mutually_exclusive_group bug
• provenance viewer: make panel wider
• parameter views: combo box with different parameter viewing types
• change parameter not updating GUI
• installing python packages into binary
• modules that dynamically configure their ports

July 23, 2014

Updates

• [DK] output-modules merge, uvcdat, improve-vt-bundle

Items to Discuss

• API design wiki page
• [TE] output-modules feedback
  • Can we have 'Output' module that uses input value to determine output?
    • [DK] Unsure what this means, we could have more than one input port, seems like the mode of output is independent of what the value is
    • [RR] Infer from python types would take a lot of work, explicit easier to understand
    • Usability vs. issues for developers
  • Hide all but value port for outputmodules
    • Unsure here
  • Make output mode be an enumeration rather than text entry (actually need to allow own entry)
  • Should vtkRendererOutput replace VTKCell?
    • [DK] Need to see if display views are involved, interaction handler, etc.
  • Use module name as base name?
    • Also naming for parameters on command-line here
    • if you name module "flower", default base name is "flower"
  • Also consider possibly only generating one output based on this type of naming
  • Global imageFile format setting not working?
    • Put on trac
  • Created WebGL output for vtkRenderer (As empty package)
  • Matplotlib WebAgg and MBAgg backends better for web
  • Clash between mode identifiers
    • Register this on trac
  • [RR] how to load a specific mode if it isn't loaded? How do we find it?
• [TE] other ideas
  • See other workflow systems: http://imgur.com/a/xY37B
  • Add icons to modules (graphical representation)
  • Add module by double-clicking in the pipeline view and writing module name
  • Edit parameters directly on the module
    • First step: move set parameters to top of list (or make more visible)
    • Second step: mine to find commonly used parameters
  • Better use of hovering to provide module information
    • Show parameters on right
  • [JF] Inspect workflows in the same window
• [DK] matplotlib feedback from Thomas Caswell
  • figure top-down?
  • see Tom's github fork of matplotlib of pyplot changes
  • declarative syntax (MEP 25), goal for 1.5
  • create artists in modules
• [RR] export-as-script
  • It's mainly about renaming variables, to connect module and avoid collisions
  • Currently gets code for script using package-provided to_python_script methods
  • Using the redbaron library for this (under early development)
  • Need to handle the different types of input corresponding to different calls (get_input, get_input_list, force_get_input, ...)
• [BNL] Adding new palette tool
  • search box with many fields to format query, send back dictionaries
• [BNL] Use numpy docstring format? yes
• [BNL] PEP whitespace? yes
• [BNL] python3 compatibility? no a big priority right now
• [BNL] PyQt new style connections: yes
• [BNL] Access to vistrails.org repository? yes

July 16, 2014

Updates

• [TE] VisTrails 2.1.3 released
• [DK] UV-CDAT

Items to Discuss

• Integrate patches from github
  • we push from vistrails.org to github

1. git fetch git@github.com:VisTrails/VisTrails.git +refs/pull/17/head:refs/heads/pull17
2. git checkout master
3. git merge pull17
4. git push origin master

• • Synchrotron needs tools for computational pipelines
• Discussion on scripting support
  • we are going to have to support a set of restricted operations
  • readability versus parsing
  • blocks vs. functions
• layered API:
  • scripting api is base, interactive api has convenience methods that use state
  • design document on API, wiki page?
• Other requirements for scripting
  • higher-level operations to find results linked to certain provenance
• vt bundles
  • should we look into improve-vt-bundle and sql-alchemy branches?
  • design this (configurable bundles)
  • have some hooks to add files to the vt bundle already
• move-dist-directory branch
  • conflicts with python build directories
  • does this affect build machines?

July 11, 2014

Updates

Items to Discuss

• Claudio wants to make it easy to edit PythonSource without VisTrails
  • [RR] Dropping url-encoding would allow you to edit them from XML? But they are also quoted because used in <parameter val="..."/>
• Scripting support
  • Export workflow as standalone script:
    • Modules could provide their own logic, or we can make something up from compute() ast in some cases
    • Also, add annotations to script to make it easy to reconstruct (and module IDs, so we can update)
  • Read back script as workflow
    • General case is hard
    • Building PythonSources from blocks might be doable
    • First step: simply reading some parameters from some modules in annotated script (i.e. has module ids)
• IPython notebook integration
  • Wait on output-modules branch; then we can show visualizations in notebook
• API needs work
  • API calls need to return things (to put them in notebook)
  • Make initialization easier (shouldn't need to create application, etc)
  • Currently stateful! (app.new_vistrail(), ..., api.save_vistrails(f) instead of vt=Vistrail(), ..., vt.save(f))
  • Should probably be in vistrails package directly

July 9, 2014

Updates

• [TE] Added logging of parameter explorations

Items to Discuss

• [TE] Crash when adding String module on Mac #760
  • Updated binary to Qt 4.8.6
  • Release 2.1 with Qt 4.8.6 to fix String module crash
• [TE] Table cell - Use HTML cell and D3?
  • Interactivity for free
  • Can it work over multiple cells?
  • Will work directly on web and crowdlabs
• [Claudio] Scripting support
  • VisTrails is general but does not interact well with other systems
  • What is not working, how can we make it easier?
  • Need ability to work without the GUI
    • Construct workflows using utility functions that does not use the VisTrails GUI
  • Convert python script to/from vistrails workflow
    • Show PythonSource contents in script
    • Start with workflow that do not show stuff at spreadsheet
  • TODO:
    • Design a better mechanism to integrate VisTrails with other tools, starting with Python
      • Provide the ability to save a workflow in a format that can be edited using a text editor -- for example, save a workflow and use an editor to change the contents of the Python Source boxes in the workflow
    • Provide functionality similar to what was implemented in UV-CDAT, where workflows can be used as black-boxes from Python. See slide 41 in http://uvcdat.llnl.gov/media/pdf/VisTrails_UV-CDAT_course.pdf
    • For which applications would scripting support be useful?
    • Look at max and grasshopper3d
    • Claudio wants to test how parallellization works in VisTrails
    • How would scripting support work?
    • Meet on Friday

July 2, 2014

Updates

• [TE] List handling improvements done
  • Now List ports can connect to anything
  • Only exception is List<->Variant

Items to Discuss

• [DK] List/Variant bug: #871
  • [TE] Fixed
  • [TE] Why is Map output a Variant?
  • looks like this was done because Fold produces a Variant (makes sense since fold can be a sum), but Map should always be a list)
  • this will allow depth 1 input ports to connect
• [RR] added List.head as depth 1 port
  • add documentation for this new functionality
• [DK] QPipelineScene.current_pipeline, controller refs #869
• [DK] PythonSource crash? #760
  • update Qt for newer binaries?
  • pyside support?
  • Also, see #737
• [TE] Log Parameter Exploration? #870
  • Very simple to enable, but we should annotate cell info
• [DK] tabledata pandas support?
  • investigate on best way to make this available (data types, scripts, modules)?
• [JF] SIGMOD: python is becoming lingua franca for data analysis, munging

Older meetings

• Development/2014
• Development/2013
• Development/2012
• Development/2011
• Development/2010