Command-line Arguments¶
Starting VisTrails via the Command Line¶
VisTrails supports a number of command-line arguments that let you modify certain attributes and behaviors of the program. When invoking VisTrails from the command line, the arguments are placed after the “run.py” filename. For example,
python vistrails/run.py -b
starts VisTrails in batch mode. Table table-batch-cli contains a complete list of the command line switches supported by vistrails. Each command line switch has both a short form and a long form. The two forms are logically equivalent, and which one you use is a matter of personal preference. The short form consists of a single minus sign “-” followed by a single letter. The longer form uses two minus signs “–” followed by a descriptive word. For example, the above command for batch mode could have been written as:
python vistrails/run.py --batch
In addition to the explicit switches listed in Table table-batch-cli, the VisTrails command line also lets you indicate the filename of the vistrail you wish to open. For example, assuming your “examples” directory is one level above your current working directory, this is how you would tell VisTrails to load the “lung.vt” example at startup:
python vistrails/run.py examples/lung.vt
Moreover, if you want VisTrails to start on a specific version of the pipeline within the vistrail, you can indicate that version’s tag name on the command line. The filename and version tag should be separated by a colon. For example, to start VisTrails with the colormap version of the “lung.vt” vistrail, use:
python vistrails/run.py examples/lung.vt:colormap
In the event that the version you want to open contains a space in its tag name, simply surround the entire “filename:tag” pair in double quotes. For example:
python vistrails/run.py "examples/lung.vt:Axial View"
You can also open up multiple vistrails at once by listing more than one vistrail file on the command line. This causes the vistrails to be opened in separate tabs, just as if you had opened them via the GUI. For example:
python vistrails/run.py examples/lung.vt examples/head.vt
You can specify version tags in conjunction with multiple filenames. Here is an example of an elaborate command-line invocation that opens two vistrails and sets each one to a specific version:
python vistrails/run.py "examples/lung.vt:Axial View" examples/head.vt:bone
usage: run.py [-h] [--version] [-S DIR] [--subworkflows-dir DIR]
[--data-dir DIR] [--package-dir DIR] [--user-package-dir DIR]
[--file-dir DIR] [--log-dir DIR] [--temporary-dir DIR]
[--thumbs-auto-save | --no-thumbs-auto-save]
[--thumbs-mouse-hover | --no-thumbs-mouse-hover]
[--thumbs-tags-only | --no-thumbs-tags-only]
[--thumbs-cache-dir DIR] [--thumbs-cache-size THUMBS.CACHESIZE]
[--host URL] [--port PORT] [--db DB] [--user USER]
[--rpc-server RPCSERVER] [--rpc-port RPCPORT]
[--rpc-log-file DIR] [--rpc-instances RPCINSTANCES]
[--multithread] [--rpc-config DIR] [--web-repository-url URL]
[--web-repository-user WEBREPOSITORYUSER]
[--job-check-interval JOBCHECKINTERVAL]
[--job-autorun JOBAUTORUN] [--job-list] [--job-info] [-E] [-b]
[-o DIR] [-p OUTPUTSETTINGS] [--parameters PARAMETERS]
[--parameter-exploration] [--show-window]
[--output-version-tree] [--output-pipeline-graph]
[--graphs-as-pdf] [--enable-usage] [--disable-usage]
[--maximize-windows | --no-maximize-windows]
[--multi-heads | --no-multi-heads]
[--show-splash | --hide-splash] [--auto-save | --no-auto-save]
[--db-default | --no-db-default] [--cache | --no-cache]
[--stop-on-error | --no-stop-on-error]
[--execution-log | --no-execution-log]
[--error-log | --no-error-log]
[--default-file-type DEFAULTFILETYPE] [-v DEBUGLEVEL]
[--show-vistrails-news | --hide-vistrails-news]
[--upgrades | --no-upgrades]
[--migrate-tags | --no-migrate-tags]
[--hide-upgrades | --no-hide-upgrades]
[--upgrade-delay | --no-upgrade-delay]
[--upgrade-module-fail-prompt | --no-upgrade-module-fail-prompt]
[--auto-connect | --no-auto-connect]
[--detach-history-view | --no-detach-history-view]
[--show-connection-errors | --hide-connection-errors]
[--show-variant-errors | --hide-variant-errors]
[--show-debug-popups | --hide-debug-popups]
[--show-scrollbars | --hide-scrollbars]
[--show-inline-parameter-widgets | --hide-inline-parameter-widgets]
[--shell-font-face SHELL.FONTFACE]
[--shell-font-size SHELL.FONTSIZE]
[--max-recent-vistrails MAXRECENTVISTRAILS]
[--view-on-load VIEWONLOAD]
[--custom-version-colors | --no-custom-version-colors]
[--fixed-custom-version-color-saturation | --no-fixed-custom-version-color-saturation]
[--enable-packages-silently | --no-enable-packages-silently]
[--load-packages | --no-load-packages]
[--install-bundles | --no-install-bundles]
[--install-bundles-with-pip | --no-install-bundles-with-pip]
[--repository-local-path DIR] [--repository-httpurl URL]
[--single-instance | --no-single-instance]
[--static-registry DIR]
[vistrail [vistrail ...]]
- Positional arguments:
vistrails Vistrail to open - Options:
--version show program’s version and exit -S, --dot-vistrails The location to look for VisTrails user configurations and storage. Defaults to ~/.vistrails. --subworkflows-dir The location where a user’s local subworkflows are stored. --data-dir The location that VisTrails uses as a default directory for data. --package-dir The directory to look for VisTrails core packages (use userPackageDir for user-defined packages). --user-package-dir The location for user-installed packages (defaults to ~/.vistrails/userpackages). --file-dir The location that VisTrails uses as a default directory for specifying files. --log-dir The path that indicates where log files should be stored. --temporary-dir The directory to use for temporary files generated by VisTrails. --thumbs-auto-save Whether to save thumbnails of results when executing VisTrails. --no-thumbs-auto-save Inverse of –thumbs-auto-save --thumbs-mouse-hover Whether to show thumbnails when hovering over a version in the version tree. --no-thumbs-mouse-hover Inverse of –thumbs-mouse-hover --thumbs-tags-only If True, only stores thumbnails for tagged versions. Otherwise, stores thumbnails for all versions. --no-thumbs-tags-only Inverse of –thumbs-tags-only --thumbs-cache-dir The directory to be used to cache thumbnails. --thumbs-cache-size The size (in MB) of the thumbnail cache. --host The hostname for the database to load the vistrail from. --port The port for the database to load the vistrail from. --db The name for the database to load the vistrail from. --user The username for the database to load the vistrail from. --rpc-server Hostname or ip address where this xml rpc server will work. --rpc-port Port where this xml rpc server will work. --rpc-log-file Log file for XML RPC server. --rpc-instances Number of other instances that vistrails should start. --multithread Server will start a thread for each request. --rpc-config Config file for server connection options. --web-repository-url The URL of the web repository that should be attached to VisTrails (e.g. www.crowdlabs.org). --web-repository-user The default username for logging into a VisTrails web repository like crowdLabs. --job-check-interval How often to check for jobs (in seconds, default=600). --job-autorun Run jobs automatically when they finish. --job-list List running workflows. --job-info List jobs in running workflow. -E, --no-execute Do not execute specified workflows. -b, --batch Run vistrails in batch mode instead of interactive mode. -o, --output-directory Directory in which to place output files -p, --output-settings One or more comma-separated key=value parameters. --parameters List of parameters to use when running workflow. --parameter-exploration Open and execute parameter exploration specified by the version argument after the .vt file. --show-window Show the main VisTrails window. --output-version-tree Output the version tree as an image. --output-pipeline-graph Output the workflow graph as an image. --graphs-as-pdf Generate graphs in PDF format instead of images --enable-usage Enable sending anonymous usage statistics --disable-usage Disable sending anonymous usage statistics --maximize-windows Whether the VisTrails windows should take up the entire screen space. --no-maximize-windows Inverse of –maximize-windows --multi-heads Whether or not to use multiple screens for VisTrails windows. --no-multi-heads Inverse of –multi-heads --show-splash Whether the VisTrails splash screen should be shown on startup. --hide-splash Inverse of –show-splash --auto-save Automatically save vistrails to allow recovery from crashes, etc. --no-auto-save Inverse of –auto-save --db-default Use a database as the default storage location for vistrails entities. --no-db-default Inverse of –db-default --cache Cache previous results so they may be used in future computations. --no-cache Inverse of –cache --stop-on-error Whether or not VisTrails stops executing the rest of the workflow if it encounters an error in one module. --no-stop-on-error Inverse of –stop-on-error --execution-log Track execution provenance when running workflows. --no-execution-log Inverse of –execution-log --error-log Write errors to a log file. --no-error-log Inverse of –error-log --default-file-type Defaults to .vt but could be .xml. -v, --debug-level How much information VisTrails should alert the user about (0: Critical errors only, 1: Critical errors and warnings, 2: Critical errors, warnings, and log messages). --show-vistrails-news Show news from VisTrails on startup. Each message will only be displayed once. --hide-vistrails-news Inverse of –show-vistrails-news --upgrades Whether to upgrade old workflows so they work with newer packages. --no-upgrades Inverse of –upgrades --migrate-tags Whether or not the tag on a workflow that was upgraded should be moved to point to the upgraded version. --no-migrate-tags Inverse of –migrate-tags --hide-upgrades Don’t show the “upgrade” nodes in the version tree. --no-hide-upgrades Inverse of –hide-upgrades --upgrade-delay Persist upgrade only after other changes. --no-upgrade-delay Inverse of –upgrade-delay --upgrade-module-fail-prompt Whether to alert the user when an upgrade may fail when upgrading a subworkflow. --no-upgrade-module-fail-prompt Inverse of –upgrade-module-fail-prompt --auto-connect Try to automatically connect a newly dragged in module to the rest of the workflow. --no-auto-connect Inverse of –auto-connect --detach-history-view Show the version tree in a separate window. --no-detach-history-view Inverse of –detach-history-view --show-connection-errors Alert the user if the value along a connection doesn’t match connection types. --hide-connection-errors Inverse of –show-connection-errors --show-variant-errors Alert the user if the value along a connection coming from a Variant output doesn’t match the input port. --hide-variant-errors Inverse of –show-variant-errors --show-debug-popups Always show the debug popups or only if there is a modal widget. --hide-debug-popups Inverse of –show-debug-popups --show-scrollbars Whether VisTrails should show scrollbars on the version tree and workflow canvases. --hide-scrollbars Inverse of –show-scrollbars --show-inline-parameter-widgets Show editable parameters inside modules. --hide-inline-parameter-widgets Inverse of –show-inline-parameter-widgets --shell-font-face The font to be used for the VisTrails console. --shell-font-size The font size used for the VisTrails console. --max-recent-vistrails How many recently opened vistrails should be stored for “Open Recent” access. --view-on-load Whether to show pipeline or history view when opening vistrail. Can be either appropriate/pipeline/history. --custom-version-colors Allow setting custom colors for versions, and display these colors in the version tree. --no-custom-version-colors Inverse of –custom-version-colors --fixed-custom-version-color-saturation Don’t change the saturation according to the age of the version if it has a custom color. --no-fixed-custom-version-color-saturation Inverse of –fixed-custom-version-color-saturation --enable-packages-silently Do not prompt the user to enable packages, just do so automatically. --no-enable-packages-silently Inverse of –enable-packages-silently --load-packages Whether to load the packages enabled in the configuration file. --no-load-packages Inverse of –load-packages --install-bundles Automatically try to install missing Python dependencies. --no-install-bundles Inverse of –install-bundles --install-bundles-with-pip Whether to try to use pip to install Python dependencies or use distribution support. --no-install-bundles-with-pip Inverse of –install-bundles-with-pip --repository-local-path Path used to locate packages available to be installed. --repository-httpurl URL used to locate packages available to be installed. --single-instance Whether or not VisTrails should only allow one instance to be running. --no-single-instance Inverse of –single-instance --static-registry If specified, VisTrails uses an XML file defining the VisTrails module registry to load modules instead of from the packages directly.
Specifying a User Configuration Directory¶
In addition to the default .vistrails directory, VisTrails allows you to create and use additional configuration directories. First, you will need to create a new directory. This is done by running:
python vistrails/run.py -S /path_to_new_directory/new_directory_name.
This will both create a new directory containing default configuration files and directories, and launch VisTrails, which will use the newly created files for configuration. The user is then free to add desired configurations to the new directory. Once a configuration directory exists, subsequent calls using the directory name (python vistrails/run.py -S /path_to_directory/existing_directory) will launch VisTrails using the ‘existing_directory’ for configuration and a new directory will not be created.
Note: If you would like to copy configuration directories, you must change the references in copy_of_directory/startup.xml to point to the new directory instead of the original.
Passing Database Parameters on the Command Line¶
As discussed in Chapter Connecting to a Database, VisTrails can read and write vistrails stored in a relational database as well as in a filesystem. VisTrails allows you to specify the name of the database server, the database name, the port number, and the username on the command line. This potentially saves you the trouble of filling out the same information on the database connection dialog. Note that, for security reasons, VisTrails does not allow you to include a database password on the command line; you must still type your password into the database connection dialog when VisTrails opens.
The last four rows of Table table-batch-cli show the command-line switches that pertain to database connectivity. Be advised that these switches were designed primarily for use by VTL files (see Section Using “Vistrail Link” Files) and as such, are not necessarily user-friendly. In particular, these switches are ignored unless you also specify the vistrail ID and version name on the command line. For example, to open the contour version of the the “spx” vistrail (whose ID is 5) from the database “vistrails” residing on the host “vistrails.sci.utah.edu” with a username of “vistrails”:
python vistrails/run.py --host=vistrails.sci.utah.edu --db=vistrails --user=vistrails 5:contour
Once VisTrails opens, you will be prompted to enter the password. Upon successful authentication, the vistrail is loaded from the database and opened to the pipeline corresponding to the specified version.
Using “Vistrail Link” Files¶
As discussed in Chapter Connecting to a Database, one of the advantages of storing your vistrails on a database is that you can collaborate with others without having to pass around a .vt file or force all users to use a shared filesystem. A disadvantage is that you need to remember the parameters with which to connect to the database. Using a “Vistrail Link” (VTL) file reduces this inconvenience, and also eliminates the need to include the associated command-line switches to connect to the database.
A VTL is a very small text (XML) file that contains the parameters required to load a vistrail from a database. VTL files are intended for use with a VisTrails-enabled wiki. You can open a VTL either by saving the file and passing its filename to the command line, or by configuring your web browser to do this for you. Here is the syntax for using a VTL file on the command line:
python vistrails/run.py sample.vtl
Internally, VisTrails parses the VTL file and loads the vistrail from the database exactly as if you had included its full parameter list on the command line.
Note:
VTL is a relatively new feature of VisTrails, and as such is neither fully developed nor completedly documented. Please contact the VisTrails development team with any bug reports and/or suggestions.
Running VisTrails in Batch Mode¶
Although VisTrails is primarily intended to be run as an interactive, graphical client application, it also supports non-interactive use. VisTrails can thus be invoked programmatically, eg as part of a shell script. You can tell VisTrails to start in non-interactive mode by using the “-b” or “–batch” command line switch when launching vistrails. [1]
Running VisTrails in non-interactive mode has little effect, however, without an additional command line argument indicating which vistrail to load. Since we are running VisTrails as part of a batch process, it only makes sense to execute vistrails whose output is something tangible, such as a file. A vistrail whose only output is an interactive rendering in a VTKCell, for instance, would not be well-suited for running in batch mode.
Consider the following example. The “offscreen.vt” vistrail (included in the “examples” directory) has a variety of output options, depending on which version you select in the History view (Figure The different versions of the offscreen.vt vistrail...). The version tagged only vtk displays its output as an interactive VTK rendering. The version tagged html creates a simple web page in the Spreadsheet. The offscreen version, however, outputs an image file named “image.png”. Since its output (a file) can be easily accessed outside of VisTrails, this version is an ideal candidate for running in batch mode. To try it, invoke VisTrails as shown, specifying both the name of the vistrail file and the desired version:
python vistrails/run.py -b examples/offscreen.vt:offscreen
The different versions of the “offscreen.vt” vistrail offer various forms of output.
As you would expect, this command runs to completion without opening any windows. Instead, it silently loads the requested pipeline, executes it, and closes. Assuming it ran correctly, this pipeline should have created a file named “image.png” in the current directory. When you view this file, it should resemble the picture in Figure Running the offscreen version of offscreeen.vt in batch mode....
Running the offscreen version of “offscreen.vt” in batch mode produces an image named “image.png”.
Running a Specific Workflow in Batch Mode¶
To run a specific workflow in batch mode, call VisTrails with the following options:
python vistrails/run.py -b path_to_vistrails_file:pipeline
where pipeline can be a version tag name or version id.
Note
If you downloaded the MacOS X bundle, you can run VisTrails from the command line via the following commands in the terminal. Change the current directory to wherever VisTrails was installed (often /Applications), and then type: Vistrails.app/Contents/MacOS/vistrails [<cmd_line_options>]
Running a Workflow with Specific Parameters¶
An alias is a name assigned to a parameter that allows you to reference that parameter in batch mode. An alias is created by clicking on the type of an existing parameter in VisTrails, then entering a name for it.
Example of creating an alias
Users can change workflow parameters that have an alias through the command line.
For example, offscreen pipeline in offscreen.vt always creates the file called image.png. If you want generate it with a different filename:
python vistrails/run.py -b examples/offscreen.vt:offscreen --parameters="filename=other.png"
filename in the example above is the alias name assigned to the parameter in the value method inside the String module. When running a pipeline from the command line, VisTrails will try to start the spreadsheet automatically if the pipeline requires it. For example, this other execution will also start the spreadsheet (attention to how $ characters are escaped when running on bash):
python vistrails/run.py -b examples/head.vt:aliases --parameters="isovalue=30\$&\$diffuse_color=0.8, 0.4, 0.2"
You can also execute more than one pipeline on the command line:
python vistrails/run.py -b examples/head.vt:aliases ../examples/spx.vt:spx \ --parameters="isovalue=30"
Use –parameters only once regardless the number of pipelines.
Accessing a Database in Batch Mode¶
As discussed in Section Passing Database Parameters on the Command Line, you can specify most of the parameters of your database connection on the command line, but the password must be entered through the GUI. This poses a problem for running VisTrails in non-interactive mode, since no database connection dialog will be opened. If your batch process needs to access vistrails stored on a database, the current workaround is to create a special account on the database (probably one with read-only access) that does not require a password, and use this account for connecting to the database in batch mode.
Using VisTrails as a Server¶
Using the VisTrails server mode, it is possible to execute workflows and control VisTrails through another application. For example, the CrowdLabs Web portal (http://www.crowdlabs.org) accesses a VisTrails sever to execute workflows, retrieve and display vistrail trees and workflows.
The way you access the server is by doing XML-RPC calls. In the current VisTrails release, we include a set of PHP scripts that can talk to a VisTrails server instance. They are in “extensions/http” folder. The files are reasonably well documented. Also, it should be not difficult to create python scripts to access the server (just use xmlrpclib module).
Note that the VisTrails server requires the provenance and workflows to be in a database. More detailed instructions on how to setup the server and the database are available in VisTrails Server Setup and in Setting up the database.
If what you want is just to execute a series of workflows in batch mode, a simpler solution would be to use the VisTrails client in batch mode (see Section Running VisTrails in Batch Mode).
Executing Workflows in Parallel¶
The VisTrails server can only execute pipelines in parallel if there’s more than one instance of VisTrails running. The command
self.rpcserver=ThreadedXMLRPCServer((self.temp_xml_rpc_options.server, self.temp_xml_rpc_options.port))
starts a multithreaded version of the XML-RPC server, so it will create a thread for each request received by the server. The problem is that Qt/PyQT doesn’t allow these multiple threads to create GUI objects. Only the main thread can. To overcome this limitation, the multithreaded version can instantiate other single threaded versions of VisTrails and put them in a queue, so workflow executions and other GUI-related requests, such as generating workflow graphs and history trees can be forwarded to this queue, and each instance takes turns in answering the request. If the results are in the cache, the multithreaded version answers the requests directly.
Note that this infrastructure works on Linux only. To make this work on Windows, you have to create a script similar to start_vistrails_xvfb.sh (located in the scripts folder) where you can send the number of other instances via command-line options to VisTrails. The command line options are:
python vistrails_server.py --host=<ADDRESS> --port=<PORT> -O<NUMBER_OF_OTHER_VISTRAILS_INSTANCES> [-M]&
If you want the main vistrails instance to be multithreaded, use the -M at the end.
After creating this script, update function start_other_instances in vistrails/gui/application_server.py lines 1007-1023 and set the script variable to point to your script. You may also have to change the arguments sent to your script (line 1016: for example, you don’t need to set a virtual display). You will need to change the path to the stop_vistrails_server.py script (on line 1026) according to your installation path.
Executing Parameter Explorations from the Command Line¶
Named parameter explorations can be executed from the command line in different ways using the -P flag. The parameter after the vistrail will then indicate the parameter exploration name in place of the workflow version. To open vistrails and execute a parameter exploration named “final” in terminator.vt run:
python vistrails/run.py -P terminator.vt:final
To only show the spreadsheet run:
python vistrails/run.py -P -i terminator.vt:final
To execute the spreadsheet in batch mode, and to output the spreadsheet as images to a directory, use the -b flag and specify a directory with the -e flag:
python vistrails/run.py -P -b -e ./final_images terminator.vt:final
This will create an image for each cell and also create a composite image for each sheet in the spreadsheet.
Finding Methods Via the Command Line¶
We have tried to make some methods more accessible in the console via an api. You can import the api via import api in the console and see the available methods with dir(api). To open a vistrail:
import api
api.open_vistrail_from_file('/Applications/VisTrails/examples/terminator.vt')
To execute a version of a workflow, you currently have to go through the controller:
api.select_version('Histogram')
api.get_current_controller().execute_current_workflow()
Currently, only a subset of VisTrails functionality is directly available from the api. However, since VisTrails is written in python, you can dig down starting with the VistrailsApplication or controller object to expose most of our internal methods. If you have suggestions for calls to be added to the api, please let us know.
One other feature that we’re working on, but is still in progress is the ability to construct workflows via the console. For example:
vtk = load_package('org.vistrails.vistrails.vtk')
vtk.vtkDataSetReader() # adds a vtkDataSetReader module to the pipeline
# click on the new module
a = selected_modules()[0] # get the one currently selected module
a.SetFile('/vistrails/examples/data/head120.vtk') # sets the SetFile\
parameter for the data set reader
b = vtk.vtkContourFilter() # adds a vtkContourFilter module to the\
pipeline and saves to var b
b.SetInputConnection0(a.GetOutputPort0()) # connects a's GetOutputPort0\
port to b's SetInputConnection0
Footnotes
| [1] | The parameter “-b” stands for “batch.” In this chapter, we use the terms “batch mode” and “non-interactive mode” synonymously. |