Module Maki

module Maki: sig .. end

Maki: Persistent Incremental Computations

Maki is a system for memoizing costly OCaml functions using the disk. It requires the functions to be pure, that is, to always return the same result given that the set of dependencies declared by the function doesn't change.

status: experimental

This module is not thread-safe.

type 'a or_error = ('a, exn) Result.result 
type 'a lwt_or_error = 'a or_error Lwt.t 

Basic types

type path = string 
type program = string 
type time = float 

Controlling Parallelism

module Limit: sig .. end


Maki deals with functions that take and return values of any type that provides a 'a Value.t implementation; that is, values that we can serialize, unserialize, and hash. The reason is that values are stored on disk for memoization purposes.

module Value: sig .. end

On-Disk storage

We use a generic interface for on-disk storage, in the form of a dictionary string -> string. The default storage just uses one file per pair.

module Storage: Maki_storage

Time Utils

module Time: sig .. end

Memoized Functions

This is the heart of the library: a wrapper around pure functions from Maki.Value arguments to a Maki.Value-aware type. Such functions are supposed to always return the same value given the same arguments and dependencies (a typical dependency is an external program that might have several version).

The function is used to actually evaluate a wrapped function, or return its memoized result if the computation was done already.

We need to name functions because, from one execution to another, the results of a function call must be stored on disk. Names are used to map function calls to their result. If two different functions share the same name (even across programs), the results will be unpredictable.

type lifetime = [ `CanDrop | `Keep | `KeepFor of time | `KeepUntil of time ] 
lifetime for a cached value
val call : ?bypass:bool ->
?storage:Storage.t ->
?lifetime:lifetime ->
?limit:Limit.t ->
?tags:string list ->
name:string ->
deps:Value.t list ->
op:'res Value.ops -> (unit -> 'res Lwt.t) -> 'res or_error Lwt.t
Call the function iff its result has not been cached yet
bypass : if true, then cache is disabled
storage : the storage used for caching values (default Storage.get_default ())
lifetime : how long to keep the cached value (defautl: CanDrop)
limit : if given, call will acquire a handle from limit before calling the (potentially costly) function
name : the name of the function, should be unique!
deps : dependencies of the computation; if they change, the function will have to be re-computed. In other words, the function is supposed to be pure on the list of dependencies (same dependencies => same result)
val call_exn : ?bypass:bool ->
?storage:Storage.t ->
?lifetime:lifetime ->
?limit:Limit.t ->
?tags:string list ->
name:string ->
deps:Value.t list ->
op:'res Value.ops -> (unit -> 'res Lwt.t) -> 'res Lwt.t
Same as but raises the exception instead of wrapping it in Error

On-Storage Values

type cache_value 
val cache_value_of_bencode : Bencode.t -> cache_value or_error
val bencode_of_cache_value : cache_value -> Bencode.t
val cache_value_lifetime : cache_value -> lifetime
val cache_value_fun_name : cache_value -> string
function used to compute
val cache_value_deps : cache_value -> string list
val cache_value_data : cache_value -> string
the value itself
val cache_value_tags : cache_value -> string list
tags attached to this value


Garbage Collection for the stored values. It needs to be called explicitely

type gc_stats = {
   gc_kept : int;
   gc_removed : int;
val string_of_gc_stats : gc_stats -> string
val gc_storage : ?remove_file:bool -> Storage.t -> gc_stats or_error Lwt.t
gc_storage s removes uneeded values and uneeded dependencies, and returns some statistics. It might take a long time.
remove_file : if true, when a path result is removed, the file it corresponds to is also deleted from the file system


val last_mtime : path -> time or_error
Last modification time of the file
val sha1 : path -> Sha1.t Lwt.t
sha1 f hashes the file f
val sha1_of_string : string -> Sha1.t
hash the given string
val abspath : path -> path
Make the path absolute
val shell : ?timeout:float -> ?stdin:string -> string -> (string * string * int) Lwt.t
shell cmd runs the command cmd and returns stdout, sterr, errcode.
stdin : optional input to the sub-process
val shellf : ?timeout:float ->
?stdin:string ->
('a, Format.formatter, unit, (string * string * int) Lwt.t)
Pervasives.format4 -> 'a
Same as but with a format string. Careful with escaping!