Ecco+EE Memory Limits >>New Info<<

Greetings.  Still testing and getting excellent information for the EE-loving side of the Ecco community.  A proper summary will follow but I'd like to have a discussion about "in what format and where" first.   When posting on Yahoo, I feel like all this work is gone in two days and likely forever.  Enough on this for now. 

Highlights .....

One test file now contains 60.7K Items and all 60.7K Items are Locked.  The file also contains 37K Date Items in one Date folder.  Both of these "overstuffed folders" made possible thanks to EE.  When freshly compacted, the file is about 50 MB.  It will expand to almost 70 MB during use.  

The file runs without a hitch.   Items can be Locked and Unlocked as needed and Search can locate a single word in a field of (guesstimate) 5  million words in under 4 seconds on an 8 year old computer running four OS's simultaneously and this search was on one of the slower virtual XP's, not the Win7 master OS.   

If I may make an editorial comment, this is "not too shabby".

The file was "scrubbed" (using Compact and Recover) occasionally while being built-up.  I believe this behavior simulates how one might use Ecco in real life - adding info daily and scrubbing perhaps quarterly.  While adding data,  I used Ecco - trying different filters, views, searches and sorts.  I was trying to blow Ecco up and why I've been calling these "stress tests".   Data was added in bulk through through 'Import'  as well as with individual line Items.   I observed when, where and how Ecco's tiny internal  memory registers would clog and how to keep this from happening.  Now the file is structured to Ecco's liking and even adding Filters and Notepads is fine.  There's only 4000 Items of headroom in this file and this Ecco+EE doesn't seem to mind.  Native Ecco will error out attempting to open this file.   

Note:  EE will successfully Scrub (Recover) this file the file BUT only if all filters are off and the scratch pad is deleted. Otherwise it Errors.  

Conclusion:  When running Recover on a full and problematic file, keep all filters off and delete Scratch, Scrap and Search note pads (views).
(@Lindsay Graham - maybe your case?)


Building the test file: There are about 30,000 Pointers (EE's "1-misc") in the file.  I'd hit 60K Pointers at 40K Items. To solve the headroom issue, I  converted several Text folders to Ext Text folders.  Beautiful solution Slang!

 I can collapse the 60K Items down to about 40K Items without loosing functionality.  I've done a "proof-of-concept" using a keyboard macro.  If this were my real-life file, I would figure out a way to do this conversion time-efficiently.   

Note: if not mindful, I can lock-up Ecco and loose work in just two clicks.  Probably a script or rule could keep this from happening better than a human trying to remember.  Adding a "anti-dumb move" script might be something the Ecco community could work on. 

There are now over 300 files on my hard drive and 50 scraps of paper on my desk.  I need to summarize soon and would like you - the Ecco community - to have a shot at easily referencing and adding to the knowledge base. The 'memory stress tests' need to be published, the (Schepers)  EE compendium updated and finalized, and EverEcco template to be Open Sourced.   Our moderator does a fine job keeping things running smoothly, but the Neo Yahoo forum structure is just not conducive to group effort.  It would be so much easier if the Yahoo Forum was any good!!!


Concluding: Before Slang & EE, Ecco's errors were mostly due to too many Items and/or Pointers.  Back then it was nearly impossible to know why errors were occuring.  Now with EE's Memory Usage window (and eccoMagic's Memory Report v2), with Memory and Corruption Guard enabled and detailed Folder Count Information* available,  files containing 60,000 Items can be commonplace.  


More when appropriate,

Mark



*  It could be easier to get this info, but it is 1000% better than nothing.  It is one of those "easy if you know how" things.  Also, I have a Folder Count template ready which makes reading the results easy.  If you bug me a bit (ie, if you have gotten this far), I will post.