Archive

Matthias Troyer came to Poly to discuss his use of VisTrails and the problems he was facing with the persistence package.

Fernando, Juliana, Matthias and Remi met on 2013-09-24.

Summary

Persistence only used as a cache

Can’t delete stuff; he deletes and recreates the whole store
He wants to use it to archive correct result, without the other intermediate files that resulted from bogus workflow OR module code
He wants to be able to find his files afterwards. Git revision hash + file reference = impracticable
He doesn’t mind filenames being unreadable if he has some way of finding these from metadata (workflow name, vistrail query, or custom metadata from module code)

Conclusions:

Drop git. If we are only going to use it for storage, and keep a separate database to map from ref uuid/upstream hash to object hash, commits and branches are useless (and are a nuisance because we can’t rewrite history)
Use a flat object store with hashes (like git’s)
Use a database to associate hash filename (upstream modules hash?) to metadata: vistrails parameter, (execution info), custom metadata
Request: make this separate from VisTrails (and used by the new archive package) so that it can be used directly by other code, and used to find files in the store

MongoDB was suggested by Fernando. Its role is precisely to store key-value pairs associated to an id.

Cons:

This would required a filename/key/value table (or even, one per value type). JOINing could be painful.

Cons:

hstore is a key-value store as a single value (i.e. in a column).