Re: [Gallery3] Wrapping up 3.0.5

Welcome, Shad, and thanks for all the work you've put into Gallery, especially on the jQuery update! I agree with focusing on jQuery 1.9 instead of putting effort into the contributed module front. 

I'd add that the type of notice proposed for codex and forum pages be added prominently to the downloads page. A message on the upgrader page in G3 would have the most visibility, a site status message wouldn't hurt.

The creation of an email list which includes contributed module developers (we don't have one, right?) might allow us to send due diligence notices for these types of changes in the future.  

- Chad


On Feb 19, 2013, at 3:52 PM, Shad Laws wrote:

> Hey everyone,
> 
> I just subscribed to the gallery-devel list.  Hello, everyone!
> 
> I agree that spending tons of time policing contributed modules instead of wrapping up the jQuery 1.9 update doesn't make much sense.  At the same time, if we do nothing, people currently using videos will upgrade their core to v3.0.5 and never realize that they're still running hacked v3.0.1 code for file_proxy and Item_Model, which imho also sounds like a bad/dangerous idea.  My thoughts are to:
> 
> - add a notice about the deprecation to the release notes and forum post for v3.0.5 (I can draft this up - when would you need it by?)
> - edit the codex module pages and forum posts for the deprecated modules and point to the notice above
> - (optional) add a bit to the core code that, upon upgrade, checks for the old modules and throws up a site message with a warning and a link to the notice above
> 
> In my opinion, editing the contributed modules themselves wouldn't have much of an impact, as the use case I imagine is where the user upgrades the core of their existing installation without touching their other modules.
> 
> Thoughts?
> 
> Thanks,
> Shad
> 
> 
> On 19 February 2013 19:38, Bharat Mediratta <[email protected]> wrote:
> cc'ing gallery-devel for general interest.  We should be having more conversations here, imo.
> 
> For those on -devel: Shad has done a lot of heavy lifting in the 3.0.5 release which is just about ready to go.  His work fixes up movie support considerably and obsoletes several contrib modules which were trying to work around these issues.
> 
> So the short answer is - we don't do anything.  Reasoning:
> - We really do care about our users having a nice experience
> - BUT we have limited time and have to care about the core functionality
> - if we spend our time policing -contrib modules it slows down release velocity
> - therefore we must focus on the things that only we can do and let the community do the rest
> 
> There will be some breakage, the community will react and fix the issues and move on.  In the meantime, we'll have the time to focus on 3.0.6 and the jQuery upgrade which is a far more impactful use of our time.
> 
> That's my rationale for why I'm not doing anything.  If you want to go hack up a couple of one-liners into those modules to check for a specific version of the G3 core and error out, etc - I'll be happy to field the pull requests :-)
> 
> -Bharat
> 
> 
> On Tue, Feb 19, 2013 at 5:26 AM, Shad Laws <shad-xpYdmXCiSuZWk0Htik3J/[email protected]> wrote:
> Hey Bharat,
> 
> One last thought occurs to me re: 3.0.5 - how should we handle deprecating the old, movie-related contributed modules?
> 
> videos
> - functionality: replaced by core core plus my new contributed "movie_tools" module
> - hack level: very high; replaces file_proxy controller, item model, and movieplayer view (doesn't even use MY_ to extend)
> - result of continued use: possibly dangerous or destructive
> 
> noffmpeg
> - functionality: replaced by core code
> - hack level: moderate; replaces movie helper (like videos, doesn't use MY_ to extend)
> - result of continued use: possibly dangerous, although less likely than videos
> 
> videothumb
> - functionality: not replaced, but doesn't take advantage of new API changes (and "movie_tools" will do the same job using the new API)
> - hack level: mild; gently uses its own controller and events, but bypasses movie::extract_frame and all of its new enhancements
> - result of continued use: can generate incorrect thumbs since it bypasses movie::extract_frame and its recent DAR fixes, etc.
> 
> videodimensions
> - functionality: not replaced, but no longer necessary since movie player is now auto-resized
> - hack level: very low; gently uses its own controller and events
> - result of continued use: possible mild data corruption, as users can write directly to the width and height entries of a movie without sanity checks
> 
> Take care,
> Shad
> 
> 
> On 18 February 2013 23:10, Shad Laws <shad-xpYdmXCiSuZWk0Htik3J/[email protected]> wrote:
> Git is being angry with me, so it's taking awhile for the tiny commit to make its way up there.  In any case, my edit is limited to the unit test for admin_watermark.  In other words, it technically has no bearing on the 3.0.5 release.
> 
> Shad
> 
> On 18 February 2013 22:15, Shad Laws <shad-xpYdmXCiSuZWk0Htik3J/[email protected]> wrote:
> Re: jQuery merge - Ah, gotcha.  Missed that detail.
> 
> Re: version numbering - If we kick the number up to 3.1, it could give us the excuse to do a few more things we'd like that could slightly break backward compatibility as well.  <shrug>
> 
> Re: watermarks unit test - looks like we broke the test with the system::delete_later addition.  The unit test looks for deleted files.  So, the unit test needs to call system::delete_marked_files before testing whether or not the files exist.  Pull request on its way...
> 
> Take care,
> Shad
> 
> 
> On 18 February 2013 21:35, Bharat Mediratta <[email protected]> wrote:
> 
> Just to be clear - I was expecting to merge in jQuery *after* 3.0.5 to give us plenty of bake time.  I'm sure there are plenty of small things we need to resolve with a change that large.  I was also thinking that maybe updating jQuery justifies going to 3.1.. not sure there.
> 
> In doing the release work I noticed that the admin watermarks test is broken:
> 
> +--------------------------------------------------------------------------------------------------+
> | Admin_Watermarks_Controller_Test                                                                 |
> +---------------------------------------------------------------------------------------+----------+
> | add_watermark_test                                                                    | PASS     |
> | add_watermark_reject_illegal_file_test                                                | FAIL     |
>   assert_false: Expected false, but was given (boolean) true.
>   /home/bharat/public_html/gallery3/modules/watermark/tests/Admin_Watermarks_Controller_Test.php (line 73)
> 
> | add_watermark_rename_legal_file_with_illegal_extension_test                           | PASS     |
> | add_watermark_reject_illegal_file_with_legal_extension_test                           | FAIL     |
>   assert_false: Expected false, but was given (boolean) true.
>   /home/bharat/public_html/gallery3/modules/watermark/tests/Admin_Watermarks_Controller_Test.php (line 120)
> 
> +=======================================================================================+==========+
> | Admin_Watermarks_Controller_Test         Score: 50  Total: 4   PASS: 2    FAIL: 2    ERROR: 0    |
> +==================================================================================================+
> 
>   TOTAL                                    Score: 99, Total: 328 PASS: 326  FAIL: 2    ERROR: 0  
> 
> 
> Ssh is hard from the plane, any idea what's going on here?  I fixed all the other tests.
> 
> -Bharat
> 
> 
> On Mon, Feb 18, 2013 at 10:33 AM, Shad Laws <shad-xpYdmXCiSuZWk0Htik3J/[email protected]> wrote:
> Hey Bharat,
> 
> Just did a bit more screwing around with IE, and I think the issues are smaller than I thought.  Apparently the real issue is with my IE config, not Gallery (it's fine on other machines).
> 
> In other words, let's roll!  Merge it in, spin it up, and let's go!  :-)
> 
> Have a safe flight,
> Shad
> 
> On 18 February 2013 19:26, Bharat Mediratta <[email protected]> wrote:
> 
> You're crushing it.  I think 3.0.5 is ready and we can merge the jquery branch up as soon as its out the door and do whatever fine tuning we need for 3.0.6.  I'm on a plane to NY now so maybe I can use this time productively by spinning up the 3.0.5 release.
> 
> -Bharat
> 
> 
> On Mon, Feb 18, 2013 at 9:42 AM, Shad Laws <shad-xpYdmXCiSuZWk0Htik3J/[email protected]> wrote:
> I made way more progress on this than I'd expected.  As far as I know, the *only* things left after the pull requests I have up are IE-related.  Specifically:
> 
> - add a function to check IE version without resorting to user agent sniffing or $.browser (already done).
> - put the sib_height fix for IE<=8 back in.
> - fix formatting of text in buttons/titlebars/etc.  It doesn't center vertically as it should.
> 
> Take care,
> Shad
> 
> 
> On 17 February 2013 23:03, Shad Laws <shad-xpYdmXCiSuZWk0Htik3J/[email protected]> wrote:
> Hey Bharat,
> 
> Nice work recently on your end, too!  No worries on keeping it short.  One of the reasons I had a bit more time to throw at this was because I was between a couple big projects on my end... and that's in the process of changing, so I certainly understand.  Still pushing to make sure we nail 3.0.5 in the very near future though!
> 
> Todo items:
> - tweak mark_file_for_delete.  I see your email as I write this - will revise in a sec.
> - "Search the gallery" text - I'll work on this tomorrow.  I'm not sure yet if it makes sense to use the specific album name or a generic one ("Search in Bob's Wedding" vs. "Search in current album"), as we'll likely need to use ellipses which could make long names more messy than useful.
> - do a quick pass and see if there are a couple easy things we can do to keep old themes from breaking so badly.  I'll peek at this tomorrow, too.
> - fix password strength indicator in change password screen.
> - fix IE formatting of text in buttons/titlebars (e.g. Add Photo dialog).  It doesn't center as it should.  FF and Chrome are fine.
> 
> If you have a moment, could you peek at the gallery.common.js bug in $.fn.gallery_context_menu?  All I've concluded from the time I've spent trying to debug it is that I'm no js guru :-).  Cut'n'pasted this blurb from previous email if you don't want to search for it: I've removed the use of a non-standard attribute (which is bound to make newer jQuery unhappy), but I still can't get the function to run only once.  addClass works, but hasClass doesn't.  I feel it's best to address the problem here as opposed to in ui.init.js or gallery.dialog.js, as this is the best way to ensure compatibility with older themes.
> 
> Probably not going to wrap this weekend, but perhaps before next weekend?
> 
> Hope all is well,
> Shad
> 
> 
> On 17 February 2013 05:50, Bharat Mediratta <[email protected]> wrote:
> 
> 
> 
> On Fri, Feb 15, 2013 at 11:54 AM, Shad Laws <shad-xpYdmXCiSuZWk0Htik3J/[email protected]> wrote:
> Hey Bharat,
> 
> Thanks much!  And you too - we're getting this thing done!  :-)
> 
> Re: security, digibug, and the bin dir, perfect.  Nicely done!
> 
> Re: the search-in-album feature, I already found myself wanting it on a couple occasions.  The use case was this: search for something -> click on album -> not it; search for something else -> no results found.  Since there was nothing saying I was searching only in the current album, it took me a few times to realize that the gallery wasn't broken.  Perhaps it'd make sense to change the "Search in gallery" text to "Search in album-name" at least?
> 
> That seems reasonable.
>  
> 
> Re: dialog box tweaking, yes, this is all for jQuery 1.9.x... and I'm *really* close to being done.  After the pull I'm sending right now (avail soon), there remains one bug: in gallery.common.js, the $.fn.gallery_context_menu function keeps running each time instead of only once.  As a result, anytime you hover over a context menu more than once, the resulting dialog boxes are totally broken.  I'm pretty sure this has to do with the changes in the way attr() works with newer jQuery 1.6+.  I spent over an hour on this one line of code and so far have failed, so I'm taking a pause :-).  I'd like to restrict the changes to the gallery.common.js file (as opposed to the theme's ui.init.js) to ensure that it doesn't break existing contributed themes.
> 
> I'll take a look at it.
>  
> 
> Re: autocomplete, this is also for 1.9.  I noticed in a couple places it still wasn't working (server add admin screen, password strength when changing passwords), but haven't done anything to see why just yet.  Thoughts?
> 
> The API changed and I only fixed this in a couple of places, let me do a review over all the occurrences.
>  
> 
> Re: a bunch of extra files, I just re-read how system::temp_filename works, and... dang.  Mea culpa.  The culprit is in the gallery_graphics helper, where a mod I made generates a temp filename to allow other modules to do it instead.  The problem was that I didn't realize that the files were actually being generated (albeit with 0 bytes).  Mea culpa.  I have to run for tonight, but I'll fix this tomorrow morning.  Sorry about that.
> 
> Re: var/logs, I think you're right.  It seems that the only time they're trimmed is when I go to admin/maintenance and run the "get rid of old stuff" task.  Good call.
> 
> I added some code to gallery_event::gallery_shutdown to do some pruning.
> 
> 
> So, here are my todo items I can think of:
> - Fix gallery_graphics.
> - Finish dialog box tweaking.  Sooooo close to being completely dialed in!
> - Test dialog boxes with Firefox and IE (I typically use Chrome), and try also with an old contributed theme or two.
> - Take one more look at the admin_watermarks / gallery.dialog.js workaround with rawurlencode and see if I can't do it more directly.
> 
> Thoughts?
> 
> Sounds good!  As you can tell from my short answers I'm still low on time but I'm doing my best to keep up with you :-)
>  
> 
> Take care,
> Shad
> 
> 
> On 15 February 2013 19:16, Bharat Mediratta <[email protected]> wrote:
> 
> You're doing an amazing job - keep it up!
> 
> I moved the digibug module over to contrib.  I think that's safe because we are very graceful when the module is missing so installs will have a database record for the module but it'll be inert.  I verified this in my production install which tracks 3.0.x.
> 
> Notes:
> - I'd like to leave the search-in-album feature alone for now, let's see if there's a demand for a toggle.  
> - I'll add the empty bin dir with a .htaccess and a README
> - I removed the digibug module
> - I've resolved all the security stuff
> 
> Questions:
> - What tweaking are you referring to on the dialog boxes and autocomplete?  Is that for jQuery 190?
> - I've noticed that there are thousands of resize_[A-z]*.jpg and test_[0-9a-f]*.jpg files in my var/tmp.  
> 
> 
> my dev box:
> $ find var/tmp -type f | cut -c1-12 | sort | uniq -c
>       1 var/tmp/.hta
>    2254 var/tmp/resi
>      10 var/tmp/rota
>    1097 var/tmp/test
> 
> my prod box:
> $ find var/tmp -type f | cut -c1-12 | sort | uniq -c
>       1 var/tmp/.hta
>       1 var/tmp/HTML
>       1 var/tmp/URI/
>      38 var/tmp/resi
>      45 var/tmp/rota
> 
> Any idea what's going on there?  Are you seeing the same?  We should probably sort this out and make sure it's not an issue in the new version before we overflow people's tmp dirs.
> 
> Also, check the number of files in your var/logs - I don't think we ever put something in place to prune that down.  I was thinking that every thousand requests or so we look to see if there's a log file over a week old and delete it.  
> 
> -Bharat
> 
> 
> 
> On Thu, Feb 14, 2013 at 2:54 PM, Shad Laws <shad-xpYdmXCiSuZWk0Htik3J/[email protected]> wrote:
> Hey Bharat,
> 
> Thanks for taking care of that!  I just finished the edits on 2003.  Can you do me a favor and pull it, then merge the master branch into the jquery_190 branch?  I'd like to see if I can eliminate the workaround with watermark dialog box error messages we've been using, and want to avoid causing collisions with the other admin_watermarks edits recently.
> 
> Re: digibug, according to http://sourceforge.net/apps/trac/gallery/ticket/2002, I think they went out of business...
> 
> I agree - we're getting really close!  Here's the stuff I can think of (feedback welcomed):
> - Add something to turn on/off the search-in-album feature
> - Add the empty bin directory and/or give it a readme and htaccess
> - Remove digibug (not sure how to do this gracefully - you want to grab it?)
> - Finish tweaking the dialog boxes (I can wrap this up)
> - Finish tweaking autocomplete (what's the current status?)
> - Other security stuff (I know you mentioned some possible issues - were they just jQuery-related, or is there something else?)
> 
> Perhaps even this weekend we can be done!
> 
> Take care,
> Shad
> 
> 
> 
> On 14 February 2013 20:49, Bharat Mediratta <[email protected]> wrote:
> 
> Just a quick note to say that I'm still trying to figure out what's up with Digibug.  I pulled all of your outstanding requests (great work!) and I think we're pretty close to ready to go now.  What do you think? 
> 
> 
> On Thu, Feb 14, 2013 at 7:03 AM, Shad Laws <shad-xpYdmXCiSuZWk0Htik3J/[email protected]> wrote:
> Hey Bharat,
> 
> When you get a moment, can you pull the jQuery-related pulls I have up?  I have a free block of time tomorrow morning and was hoping to use it to try and wrap up the dialog boxes.  It's getting close!
> 
> Thanks!
> 
> Take care,
> Shad
> 
> 
> 
> On 12 February 2013 17:52, Shad Laws <shad-xpYdmXCiSuZWk0Htik3J/[email protected]> wrote:
> Hey Bharat,
> 
> I think I almost have the last of the non-js commits I'd like to get in there:
> - follow-on 1994 (updating some comments to the already-made changes -- ready to pull)
> - 2001 (merged 2000 prior -- ready to pull)
> - 1988 (revised based on your feedback -- ready to pull)
> - Zend Guard debugging message (as far as I know we can't change this at run time, so I figured at least a message of why we're crashing would be useful)
> 
> The only two non-js things left I can think of are:
> - Digibug has left the building... bummer.
> - Search context.  The new search_within_album mode is cool, but not easily changed.  Also, the text still says "Search the gallery" which is a little misleading...
> 
> As for js, I started out by trying to take notes on the four different updates I know of (1.4.2, 1.7.2, Greydragon's 1.7.2, and the current 1.9).  Then, I bit off two pieces, which work well and are ready to pull:
> - Update the remainder of the libraries.
> - Updated the success callbacks, since <1.4 needed xhr to be set ahead of time.  This works well, refreshes things when they need to be refreshed, works with watermarks under Chrome (previously an issue with extra <pre> tags showing up), etc.
> 
> Next, I can dive into making the dialog box formatting a bit better.  Besides wrapping up your autocomplete work, what else do we have on our hit list?
> 
> Hope you had a great weekend in Tahoe!
> 
> Take care,
> Shad
> 
> 
> On 9 February 2013 23:47, Shad Laws <shad-xpYdmXCiSuZWk0Htik3J/[email protected]> wrote:
> Hey Bharat,
> 
> Sweet - enjoy Tahoe!  My bro is up there this weekend, too.  I miss weekends up there... and I miss dinner and drinks at Fire'n'Ice...
> 
> Thanks for taking a break to peek at the pulls.  I revised 1935 (more secure) and outlined my thoughts on 1988 (in the pull req).  If they were to meet approval, they've both tested and ready for pull.  I'll wrap up the last piece of thread 2 tomorrow morning.  Whether you see it first thing tomorrow morning or last thing before you go to bed tonight depends on how much fun your having :-)
> 
> Afterward, I'll plug into jQuery and see what I can do!
> 
> Take care,
> Shad
> 
> 
> On 9 February 2013 19:16, Bharat Mediratta <[email protected]> wrote:
> 
> 
> 
> On Sat, Feb 9, 2013 at 5:11 AM, Shad Laws <shad-xpYdmXCiSuZWk0Htik3J/[email protected]> wrote:
> Hey Bharat,
> 
> Seems like we might be entering a semi-feature-freeze state, and I thought I'd try and prioritize the work in progress.  Any and all feedback welcomed :-).
> 
> Excellent.  Thanks for summarizing.  I've been busy in the past week and am now having a guys weekend in Tahoe (read: too much meat and beer to function properly) but I'll try to get a little Gallery time in.
> 
>  
> In order of importance / current ROI...
> 
> Thread 1: Update jQuery.
> I see you started it - nice!  Of course this is a giant help, and likely addresses some potential security issues too (I know Drupal recently closed up holes arising from pre-1.6 versions).  Let me know if there's something I can do to help out - things to tackle, beta testing, debugging, etc.
> 
> Yep, I decided to plug away at it again.  In the past I've tried to be really disciplined about it - making sure that the provenance of all 3rd party source is set up in the vendor repo, etc.  This time I'm doing a hack-n-slash approach to just get it working.  Feel free to pull from that branch and make any number of fixes to it.  Let's just get it working any way we can.  Fix anything you want. 
>  
> 
> Thread 2: File type and mime type: agreement, extensibility, consistency...
> This is all of the placeholder, album cover, mime type, legal file, etc. work I've been working on.  I feel it's really close to being dialed-in, and we could really seal it with a couple more tweaks.
> Current pulls:
> - #1996 - Add blacklist to legal_file helper.
> - #1994 - Make get_file_metadata throw an exception if photo or movie is unidentifiable/illegal.
> Future pulls (dev done, but needs 1994 to be pulled first):
> - Make filename sanitizing more consistent (smash extensions, add extension if missing, make sure extension matches data file, etc.).
> 
> I'll try to get to these ASAP.
>  
> 
> Thread 3: Make FFmpeg/movies easy for users, and eliminate the need for less-than-ideal contributed modules that hack over core code and/or allow invalid data in the items table (videos, noffmpeg, videodimensions...)
> This has been another big thread of mine, and I feel we're really close to having it dialed-in, too.  There's just a few more tweaks needed, and then we can forever eliminate these modules and the laundry list of forum threads that say how non-straightforward this is.
> Current pulls:
> - #1935 - Make FFmpeg easier to install.
> - #1988 - Add allow_movie_uploads option ("always", "never", or "default").
> - #1997 - Correct movie dimensions and mime type improperly set by contributed modules.
> Future pulls (dev done, but needs 1935 and 1988 to be pulled first):
> - Add simple admin/movies screen, similar to simple admin/graphics screen, to see how we're configured and change the allow_movie_uploads setting.  This is more-or-less done on my end, but needs 1935 and 1988 to be pulled first.
> 
> I've been struggling with this one.  I've read over it a couple of times but haven't had time to really get into it.  In general, I'm worried that if we have an allow_movie_uploads set to "always" and ffmpeg stops working then people will have lots of errors.  Right now we verify the existence of ffmpeg once per request every time we use it (right?) so we don't have to worry about that.  That doesn't cost us too much and it seems like it's better.  Can you tell me the advantage of making this a setting?  Feel free to explain this in the pull req as well so that we keep it all in one place.  Um, I guess I should outline this issue in the pull req as well :-)
> 
> 
> Thread 4: JPG thumbs/resizes.
> While this is largely done on my end, it definitely has a lower ROI than the stuff above.  If we want to wrap 3.0.5 soon, the line can easily be drawn before this.
> Current pulls:
> - #1974 - Add option to make all resizes and/or thumbs jpg.
> Future pulls:
> - none - it's all captured in the big one above.
> 
> IIRC I thought this was a little invasive.. I'll look again.
>  
> 
> Thread 5: Extend image library to strip metadata, use progressive encoding, and turn off multithreading (compatibility issue for some users)
> I've worked a bit on this, but similar to the JPG thumb/resize thing, it definitely has a lower ROI at this point and can be punted to after 3.0.5.
> Current pulls:
> - none.
> Future pulls:
> - a couple.
> 
> After 3.0.5, imo.
> 
> 
>  
> 
> Thoughts?
> 
> Thanks!
> Shad
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_feb__[ g a l l e r y - d e v e l ]_________________________
> 
> [ list info/archive --> http://gallery.sf.net/lists.php ]
> [ gallery info/FAQ/download --> http://gallery.sf.net ]

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb

__[ g a l l e r y - d e v e l ]_________________________

[ list info/archive --> http://gallery.sf.net/lists.php ]
[ gallery info/FAQ/download --> http://gallery.sf.net ]