[Accel-config] Re: [PATCH] accel-config/test: load/unload uacce module at cleanup

On 4/6/2021 11:43 AM, Thomas, Ramesh wrote:
> On Tue, Apr 06, 2021 at 10:44:30AM -0700, Dave Jiang wrote:
>> On 4/6/2021 10:32 AM, Thomas, Ramesh wrote:
>>> On Tue, Apr 06, 2021 at 08:55:43AM -0700, Dave Jiang wrote:
>>>> On 4/5/2021 5:45 PM, ramesh.thomas(a)intel.com wrote:
>>>>> From: Ramesh Thomas <ramesh.thomas(a)intel.com>
>>>>>
>>>>> Cleanup unloads and reloads all idxd kernel modules. This is broken due
>>>>> to idxd_module depending on idxd module. The order of idxd_mdev and
>>>>> idxd_uacce modules are not fixed requiring additional checks during the
>>>>> process.
>>>>>
>>>>> Signed-off-by: Tony Zhu <tony.zhu(a)intel.com>
>>>>> Signed-off-by: Ramesh Thomas <ramesh.thomas(a)intel.com>
>>>> I would maybe add a comment explaining the weird behavior observed.
>>> I have a new finding. Even though "modprobe -r" returns failure and
>>> prints error message "modprobe: FATAL: Module idxd is in use.", the
>>> module is still unloaded. It does not give that error on the second
>>> module that is unloaded so I earlier thought it has to do with the
>>> order. This seems to be a bug in the driver and is easily reproducible.
>> Hmm.....if so it would be more in the device driver framework than the
>> driver. The driver only supply the init/remove callbacks. It doesn't
>> have any control over the load/unload mechanism. It can hold ref count
>> on the module, but we don't do that.
> It returns an error and prints "FATAL" in the message so it does look
> like a real bug. I think I know what is going on, though I don't know
> why. When the second module is unloaded, modprobe also unloads idxd.
> Which means whenever it unloads either idxd_mdev or idxd_uacce, it tries
> to unload idxd and fails if another module dependent on idxd is still
> loaded blocking idxd from getting unloaded. Do you think idxd should
> maintain a reference count or similar indication so modprobe does not
> try to unload it when a dependent module is unloaded while other
> dependent ones are still active?

Ah I guess that makes sense. But idxd actually knows nothing about the 
other drivers depending on it. So I don't think we can do anything about 
that. I think that is the behavior of modprobe -r where it tries to 
unload all the relevant modules. I think for our purpose, rmmod works 
better as it just unloads the module you specify. Here's from the man 
page. Essentially it tries to remove the other module. But I guess it's 
not smart enough to be aware of dependency from some other module on the 
module it attempts to remove.


        -r, --remove
            This option causes modprobe to remove rather than insert a 
module. If the modules it
            depends on are also unused, modprobe will try to remove them 
too. Unlike insertion, more
            than one module can be specified on the command line (it 
does not make sense to specify
            module parameters when removing modules).


>
>>
>>> Since it returns failure, the test script aborts with error. I will
>>> modify the workaround to not trap on the error and continue. That way we
>>> don't need the second unload attempt of the same module. I will add
>>> a comment describing this as a workaround for the above behavior.
>>>
>> Does rmmod complain? That's usually what I use.
> rmmod does not complain. For now I can us it instead of modprobe.
>
>