Re: Caching in the dynamic database

On 02/12/2014 10:07 AM, Michael Ben Yosef wrote:
> Hi everyone,
>
> As we know, with library(persistency) one can persist (a chosen
> subset of) the dynamic database. It would also be nice to be able to
> implement caching when the total data grows larger than main memory,
> so that, for example, least recently used persistent clauses could
> be retracted to make way for new ones to be asserted.
>
> Is there any reason (e.g. JITI or dynamic database garbage
> collection characteristics) why it would be infeasible to frequently
> retract disk-persisted facts when they are not needed and re-assert
> them from disk when the are needed again?

I guess not. There are two issues of course: what does it mean if a
clause does not exist? If I have to go to a database for every clause
that does not exist to make sure it really does not exist, the advantage
is probably gone. Instead of using a simple file, one probably needs a
real database.  I guess this implies it makes sense if I have a clauses

my_clause(Id, <properties>)

that is queried as (+, -*).

Now, you could do something like this:

my_clause(Id, Properties) :-
	my_clause_cached(Id, Props0), !,
	Properties = Props0.
my_clause(Id, Properties) :-
	<db lookup>(Id, Props0),
	asserta(my_clause_cached(Id, Props0)),
	Properties = Props0.

Is that what you have in mind.

> What is the best way to estimate the current size of the dynamic
> database in memory for this purpose, i.e. so that caching policies
> can be based on how full memory is? It seems one should be able to do
> this with statistics/2, but I'm not sure which statistic to use. My
> installation gives
>
> ?- statistics(heapused, X). X = 0.
>
> i.e. heap used is not maintained - is there a compile-time option I
> should enable so that the system keeps track of it? I also get
>
> ?- statistics(memory, [Total, Free]). Total = 23768, Free = -23769.

Neither are still supported.

> and I'm not sure if that's correct, nor how to interpret it if it
> is. Under "codes" the table says
>
> "Total size of (virtual) executable code in words"
>
> I take it that that's VM words, not the number of *machine* words
> occupied by stored clauses, else it should be expressed in bytes as
> the table title suggests. Or could this somehow be used to estimate
> space usage?

Those are indead VM words.  A VM word is a machine word (32 or 64
bits).

Not sure of any of these keys make sense. You typically have to share
memory with other processes. How much is reasonable? What would be
pretty easy to add is a clause_property/2 that tells you how big a
clause is. Would that help? Probably you also want something to know
when it was last used. Not sure how to do that efficiently without using
extra space. In theory you could invent a time-stamping instruction.  Not
sure what a sensible API would look like though.

I'm surely interested to include support for this stuff, provided that
the general overhead can remain low.

	Cheers --- Jan