Re: [Tiki-devel] Poor performance on doc (maybe dev too) hopefully fixed

Hi Ricardo

No, it's worse than that i'm afraid. Sadly someone added a feature (non optionally) that whenever a PDF (or print?) is requested, which anonymous can do on all the *.tiki.org sites) a token is issued in case anything on the page needs authentication (questionably imho to begin with).

Sadly these (supposedly) short life tokens don't ever seem to get removed - i tried with the commits listed below (and a couple of others) but in the real world they don't appear to have worked (worked fine in my localhost tests ;)

Stuff is crawling the tiki sites and generating a few tokens per minute, so these mount up and cause performance fails like we've been seeing for the past few years on our sites.

Hope you can help! :)

TIA

jonny





> On 4 Aug 2021, at 21:56, Ricardo Melo <[email protected]> wrote:
> 
> Maybe we just don't have the right indexes also?
> 
> Need to have a look.
> 
> Ricardo
> 
> On Tue, Aug 3, 2021 at 6:12 PM <[email protected]> wrote:
>> Thanks Jonny. That concurs with something Oliver wrote recently: "Also 
>> another note: after cleaning up most of the tokens in the token table, 
>> the response time for tiki.org is WAY better. We need an auto cleanup 
>> for this table. Right now: [ Execution time: 0.60 secs ]   [ Memory 
>> usage: 7.45MB ]   [ Queries: 386 in 0.29 secs ]   [ Server load: 1.02 ]"
>> 
>> 
>> Ricardo, Jorge and I will look into this one:
>> https://avan.tech/item65820--tiki-org-Old-expired-tokens-are-not-being-cleared-out
>> 
>> Oliver and Jonny: you can leave the tables on *.tiki.org grow as 
>> evidence, and some new code in 23.x will fix it shortly.
>> 
>> Thanks!
>> 
>> Marc
>> 
>> 
>> On 2021-08-03 18:46, Jonny Bradley via TikiWiki-devel wrote:
>>> > Hi all (inc [email protected] this time)
>>> > 
>>> > I thought i'd fixed the code that was supposed to clean out old
>>> > expired tiki_auth_tokens but it seems not... I just truncated the
>>> > tables again on doc, dev and themes as each one had a few hundred
>>> > thousand rows in it again, no time to try again now but this needs
>>> > fixing properly somehow...
>>> > 
>>> > jb
>>> > 
>>> > 
>>> > Message received 2021-06-23 15:44:14:
>>> > 
>>> >> Hi all again
>>> >> 
>>> >> I was trying to add some new screenies on doc and the file galleries 
>>> >> were being really slow again, so i had a look in the database and the 
>>> >> tiki_auth_tokens table (which i cleared out about a month ago) now has 
>>> >> 128,000 rows in it again - it seems auth tokens are being issued there 
>>> >> every 10-15 seconds for some strange reason (web crawlers indexing PDF 
>>> >> links maybe?) so the fix i made doesn't seem to be working...
>>> >> 
>>> >> Ok, seems the PDF generators were adding a token and then not cleaning 
>>> >> it up afterwards, and if no tokens are then used afterwards nothing 
>>> >> was then removing the expired ones, so i added deleteToken in 0672f897 
>>> >> and backported that to 22.x
>>> >> 
>>> >> Also i cleared out the tiki_auth_tokens again manually so we're able 
>>> >> to use the file galleries again - hope it didn't bork anything else!
>>> >> 
>>> >> Now where was i..?
>>> >> 
>>> >> jonny
>>> >> 
>>> >> 
>>> >> 
>>> >> 
>>> >>> On 24 May 2021, at 19:44, Jonny Bradley <[email protected]
>>> >>> wrote:
>>> >>> 
>>> >>> I started trying to fix the missing images on 
>>> >>> http://doc.tiki.org/PluginImg but needed to check in the file 
>>> >>> galleries, which have been broken there for months, so i 
>>> >>> investigated...
>>> >>> 
>>> >>> And eventually, found there were about 2 million expired auth tokens 
>>> >>> in the database (generated when pages are printed i think), due to 
>>> >>> some poor code in AuthTokens::getGroups, which i (hopefully) have 
>>> >>> fixed in:
>>> >>> 
>>> >>>    https://gitlab.com/tikiwiki/tiki/-/commit/5292fbdf
>>> >>> 
>>> >>> I've also backup up and removed all the expired tokens from the 
>>> >>> tiki_auth_tokens table on doc, and now we have filegals back again, 
>>> >>> woo!
>>> >>> 
>>> >>> e.g.
>>> >>> 
>>> >>>    https://doc.tiki.org/file5
>>> >>> 
>>> >>> Seems that table on dev also has about 1.5 million row, so i'll do 
>>> >>> the same expiry delete query there too...
>>> >>> 
>>> >>> And once the tikis there have updated (usually overnight i think) 
>>> >>> this shouldn't happen again - hopefully...
>>> >>> 
>>> >>> jonny
>>> >>> 
>>> > 
>>> > 
>>> > _______________________________________________
>>> > TikiWiki-devel mailing list
>>> > [email protected]
>>> > https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>> 
>>> 
>>> _______________________________________________
>>> TikiWiki-devel mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>> _______________________________________________
>>> TikiWiki-devel mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>