Re: CpuSet on PAGG (Re: PAGG in Open Source projects?)

Thank-you, Kaigai Kohei, for taking the time to explain
your motivation for preferring PAGG to hook fork/exit.

In your initial post a few days ago, you wrote:
> We want to use Job(and CSA) or CpuSet without specific patches.
> And, it's so important to adopt the PAGG framework into the stock kernel.

I agree that it would be nice if cpusets didn't require a specific
kernel patch.  It would make the job of getting cpusets accepted into
mainstream Linux much easier.

However I think that there is no way that you can use cpusets without
the specific cpuset patch.  Even with your patches to allow the cpuset
fork/exit hooks to be done using PAGG, you still have the other, more
specialized, cpuset hooks to consider.

>   (CpuSet-patch needs lockless references.)

I am a little surprised that the fork/exit cpuset hooks must
be lockless.

Are you talking about the cpuset patch that is in Andrew Morton's
*-mm kernels of the last few months, or some other CpuSet patch?

The cpuset_fork() just does an atomic_inc() of a reference count, so
doesn't care what locks are held when it is called.  But the
cpuset_exit() code can grab the cpuset semaphore if the last task using
a cpuset exits, when one needs to consider invoking notify_on_release.

I thought it was ok to nest semaphores inside semaphores (so long as you
respect an order, so that you can't deadlock), so I don't understand why
you needed to replace that pagg semaphore with an rcu section.

In your patch 3 of 3, you wrote:
>    Some functions, like as CpuSet, fit the PAGG framework, I think.

I am sure that my colleagues at SGI who are supporting PAGG hope that
you are right.  However I still don't see it.

Call-by-string-name dynamically evaluated invocations are not
necessarily better or worse than simple, hard coded, directly linked
function calls.  They _are_ more expensive, by far, and more complex and
obscure, which impairs the ease of both reading and debugging code. 
They _have_ to provide some balancing benefit to be justified.  If
something can be made entirely a loadable module, requiring no specific
patches (to use your nice phrase) then that might be such a benefit.

Until you can dynamically plug each of the following hooks:

    int cpuset_init(void);
    void cpuset_init_smp(void);
    void cpuset_fork(struct task_struct *p);
    void cpuset_exit(struct task_struct *p);
    const cpumask_t cpuset_cpus_allowed(const struct task_struct *p);
    void cpuset_init_current_mems_allowed(void);
    void cpuset_update_current_mems_allowed(void);
    void cpuset_restrict_to_mems_allowed(unsigned long *nodes);
    int cpuset_zonelist_valid_mems_allowed(struct zonelist *zl);
    int cpuset_zone_allowed(struct zone *z);
    struct file_operations proc_cpuset_operations;
    char *cpuset_task_status_allowed(struct task_struct *task, char *buffer);

I think that you will require a cpuset specific patch.

Am I missing something ??

Aside -- if you do value cpusets, please put in a good word for them
with Andrew Morton, on lkml perhaps.  He will _not_ further the advance
of cpusets unless others outside SGI ask for them eagerly.

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.650.933.1373, 1.925.600.0401