I've got a basic working version of the following, but I need to  
clean up the patch before I send it in for everyone to play with.   
Here's the basic idea though: I need to be able to iterate over large  
numbers of cache entries (currently 900,000, soon to be 1,000,000+).   
Currently we lock the cache and grab *every* item which currently is  
impossible with this many entries.  On smaller systems it's just  
slow, and holds up the other processes.

If we create an Iterator class then we can iterate over the APC cache  
by only grabbing say $chunk_size number of entries at a time.  Once  
we've iterated over the first chunk_size we can go back and grab  
chunk_size more from the cache.  This free's up the lock, and for  
most cases when we are searching or grabbing only 100 items it'll be  
significantly faster and lower memory usage.  It has the downside  
that fetches are not atomic, but if that functionality is needed  
apc_cache_info() or a $chunk_size = num_entries can be used instead.

proto: new APCIterator(string cache [, string regex [, int  
chunk_size]]);

// Toss some values in the cache
for($i=0; $i < 10000; $i++) { apc_store("key $i", "value $i"); }
for($i=0; $i < 5000; $i++) { apc_store("xkey $i", "value $i"); }

// Create an APC Iterator with regex '^xkey' on the 'user' cache.
$it = new APCIterator('user', '^xkey');

// Iterate through the cache up to 100 items
$count = 0;
foreach($it as $key=>$val) {
   echo "$key => ";
   var_dump($val);
   echo "\n";
   $count++;
   if ($count > 100) break;
}



I'm thinking that deletes could then possibly look something like:

$it = new APCIterator('file', '^/some/path/');      // fetch all  
files starting with /some/path
apc_delete_files($it);				    // delete all those, as  
apc_delete_files knows how to work with APCIterator class

apc_delete_files('/one/specific/file.php');	    // delete a single  
specific file a as a string name

apc_delete_files(array('file1.php', 'file2.php'));  // delete an  
array of specific file names


If we do need more options like globs and prefix etc, then it might  
be preferred if they go into the iterator so it can be used for other  
features, etc.  The downside of all this is that it's actually slower  
than just passing the regex directly into apc_delete_files as you're  
looking everything up twice.  Alternatively it could just become a  
glorified container for the regular expressions/matching mechanism as  
well.  I do tend to prefer this interface however, but what are  
other's thoughts on this?

-shire



On May 28, 2008, at 11:45 AM, shire wrote:

>
> Haven't had a chance to play with it just yet, but I'm thinking  
> that it would be a cleaner API if we could do away with the second  
> parameter constants.
> I'm assuming that you included the APC_DELETE_PREFIX and  
> APC_DELETE_GLOB for more efficient searches of the cache(?),  do  
> you know how much of a performance difference this makes in practice?
>
> I'm working on creating a function that will allow more efficient  
> fetching of cache entries entries by incrementally grabbing chunks  
> from the cache to decrease lock time, and will add regex support to  
> this.  It should be simple to pass in the results from this into  
> other functions to provide regex support.  I'll post it here as  
> soon as I get something working so ppl can comment.
>
> -shire
>
> On May 27, 2008, at 8:31 AM, David Sklar wrote:
>
>> Any thoughts on this patch and/or its likelihood of making it into  
>> the
>> next APC release?
>>
>> David
>>
>> On Tue, May 20, 2008 at 1:57 PM, David Sklar  
>> <david.sklar@gmail.com> wrote:
>>> A tarball of an initial implementation is up at
>>> http://www.sklar.com/files/apc_cache_delete.tar.gz
>>>
>>> (Seems lists.php.net qmail gets grumpy with messages > 30k)
>>>
>>> In the tarball, apc_cache_delete.diff is a diff against latest CVS;
>>> full copies of the
>>> modified files (php_apc.c, apc_cache.{ch}) are also included as well
>>> as some tests.
>>>
>>> The new user-level-function is apc_delete_files(). You pass it a
>>> delete-spec and an optional spec type that controls how the
>>> delete-spec is used, e.g.:
>>>
>>> apc_delete_files('/some/path/to/file.php', APC_DELETE_PATH);
>>> apc_delete_files('/some/path/to/file.php'); // no spec == exact  
>>> path match
>>> apc_delete_files('/some/string/prefix/for/path', APC_DELETE_PREFIX);
>>> apc_delete_files('/some/glob/expr?ssion.*', APC_DELETE_GLOB);
>>> apc_delete_files('@^/usr(/local)?/php/@', APC_DELETE_REGEX);
>>>
>>> I suppose a nice enhancement would be to allow the first argument be
>>> an array as you suggest below.
>>>
>>> David
>>>
>>> On Wed, May 14, 2008 at 5:04 PM, shire <shire@tekrat.com> wrote:
>>>>
>>>> Hey David,
>>>>
>>>> It's been on my list to code up a file removal function that can  
>>>> be accessed
>>>> from user space ( I haven't started so you've got my vote on  
>>>> this if you
>>>> want patch something up....).
>>>>
>>>> I'm also going to need to add a utility to grab both functions  
>>>> and user
>>>> variables via a regex function more effectively (I want to start  
>>>> on that
>>>> today) so I'd probably use the two functions in conjunction if I  
>>>> wanted to
>>>> do a regex delete, and make the argument to the delete function  
>>>> a string or
>>>> optional array so it's more flexible (ie: if you already have a  
>>>> list of
>>>> functions as an array).  I'm thinking though that searching  
>>>> would be most
>>>> easily implemented based upon prefixes rather than more  
>>>> complicated regex
>>>> expressions (falling back to a slower regex comparison would be  
>>>> possible),
>>>> does this fit your use case(s) as well?
>>>>
>>>> These two options may work well for another task I need which is  
>>>> removing
>>>> groupings of files that could be considered "packages".  But  
>>>> ideally if one
>>>> file was removed due to a TTL timeout etc, all it's sibling  
>>>> files would go
>>>> to, which I'm not sure yet how I see being implemented in a  
>>>> generic/flexible
>>>> way.
>>>>
>>>> -shire
>>>>
>>>> On May 14, 2008, at 8:39 AM, David Sklar wrote:
>>>>
>>>>> I'd like to add a function that removes a specific compiled  
>>>>> file from
>>>>> the cache. My initial thought is to have the function, given a
>>>>> filename:
>>>>>
>>>>> - use apc_cache_make_file_key() to determine the cache key for the
>>>>> filename
>>>>> - use apc_cache_find_slot() to determine the slot for the cache  
>>>>> key
>>>>> - call remove_slot() on the slot to either free it immediately  
>>>>> or put
>>>>> it on the deleted list if its refcount is > 0
>>>>> - and optionally have the function call process_pending_removals 
>>>>> () to
>>>>> flush the deleted list (if possible)
>>>>>
>>>>> Another possibility, more useful for bulk deletes (at least on
>>>>> fullpath entries) could be something like apc_delete_file 
>>>>> ($pattern)
>>>>> which deletes all files whose names match a given pattern.   
>>>>> This could
>>>>> work internally like apc_cache_clear() except that before calling
>>>>> remove_slot() on  a particular slot, it would compare the  
>>>>> pattern to
>>>>> the fullpath in the key and only remove if it matches.
>>>>>
>>>>> Do either/both of these approachese sound reasonable? Something
>>>>> obvious I'm missing?
>>>>>
>>>>> Thanks,
>>>>> David
>>>>>
>>>>> --
>>>>> APC Development Mailing List (http://pecl.php.net/APC)
>>>>> To unsubscribe, visit: http://www.php.net/unsub.php
>>>>
>>>>
>>>
>>
>> -- 
>> APC Development Mailing List (http://pecl.php.net/APC)
>> To unsubscribe, visit: http://www.php.net/unsub.php
>
>
> -- 
> APC Development Mailing List (http://pecl.php.net/APC)
> To unsubscribe, visit: http://www.php.net/unsub.php