Re: ceph: add 'lazyio' mount option to kclient

Hi Xiubo,

The mount option plumbing looks good to me, but I have some concerns.

The main issue is where `CEPH_FILE_MODE_LAZY` gets added. `ceph_open()` computes
`fmode`/`wanted` and may either use existing caps or send the open request
before `ceph_init_file_info()` adds the LAZY bit. That means the MDS may not see
the LAZYIO want at open time, which leaves a gap in the case this option is
trying to cover. Could you set the LAZY bit before `ceph_caps_for_mode()` and
make sure the atomic-open path is covered too?

I also think the LAZYIO substitution in `try_get_cap_refs()` needs to be gated
on callers that actually requested LAZYIO. Otherwise an inode with one lazy fd
can let a non-lazy fd have CACHE/BUFFER wants satisfied by LAZYIO, which changes
the consistency semantics for that non-lazy caller.

The revoke-side accounting is subtle as well. Please double-check the
CACHE/BUFFER-then-LAZYIO revoke ordering and add coverage for dirty/cached pages
under LAZYIO, so we don't ack a revoke before the covered state is written back
or invalidated.

-- 
Alex Markuze