Re: org.apache.xindice.core.filer.BTree

Todd Byrne wrote:
> Well this was an EOF exception so the database file was truncated, is
> there an away to recover the non corrupted documents?

I was thinking a bit about it.

It is relatively simple to add a basic sanity check into the filer.open() method 
to see if file was properly closed or not: length of the file should be exactly 
header + pageCount * pageSize. So it is easy to catch EOF condition.

In addition to this check, filer probably can write additional byte into the 
header to indicate clean shutdown. It can write 1 when file is opened, and 0 
right before it closed. So next time you open, if you see '1' in there, it means 
database was not properly shut down last time.

Once you know if file was shut down incorrectly, we could try recover data from 
it (starting with taking a backup, probably?)

For hash filer, it can walk through main table and collect all non empty 
records, and recover collision chains. BTree filer also can be traversed. So 
most of the information can be retrievable. Some, of course, could be lost.

Let me know if you want to implement any of the above.

Vadim

> Simple scan through the collection until the error occurs again?
> 
> I am going to work on some changes to propagate the IOException into the
> other methods.
> 
> Todd
> 
> Vadim Gritsenko wrote:
>> Todd Byrne wrote:
>>
>>> In getBTreeNode(long page, BTreeNode parent)(line ) if an exception gets
>>> thrown for what ever reason it just gets ignored and null returned. This
>>> seems problematic because nothing that calls this method checks for
>>> null. Worse seems getChildNode(int idx)(line 534) doesn't check and is
>>> called in about 11 places and none of their callers check for null.
>>>
>>> My hunch is to let the exception be thrown and then we will get meaning
>>> full stack traces instead of NPEs later and have to search the log files
>>> later. If an exception was thrown like PageNotFound we could catch it in
>>> the result code and gracefully skip the document.
>>>
>>> Thoughts Ideas?
>>
>> The only exception which should be thrown from this method is
>> IOException, as far as I can see. And I agree, I think it is better to
>> get IOException rather then rather pointless NPE.
>>
>> So for a quick fix, I'd try and change method signature(s) to include
>> IOException and pass it up instead of eating it.
>>
>>
>> It brings though a point of how xindice should handle exceptional
>> situations like this. If it is at all possible, it should recover. E.g.,
>> it is better to be able to extract some documents from damaged
>> collection then none at all.
>>
>>
>> Vadim