Re: FW: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL

On 3/26/23 02:21, Mike Rapoport wrote:
> Hi,
> 
> [..] >> One problem we experienced was occured in the combination of 
hot-remove and kerelspace allocation usecases.
>> ZONE_NORMAL allows kernel context allocation, but it does not allow hot-remove because kernel resides all the time.
>> ZONE_MOVABLE allows hot-remove due to the page migration, but it only allows userspace allocation.
>> Alternatively, we allocated a kernel context out of ZONE_MOVABLE by adding GFP_MOVABLE flag.
>> In case, oops and system hang has occasionally occured because ZONE_MOVABLE can be swapped.
>> We resolved the issue using ZONE_EXMEM by allowing seletively choice of the two usecases.
>> As you well know, among heterogeneous DRAM devices, CXL DRAM is the first PCIe basis device, which allows hot-pluggability, different RAS, and extended connectivity.
>> So, we thought it could be a graceful approach adding a new zone and separately manage the new features.
> 
> This still does not describe what are the use cases that require having
> kernel allocations on CXL.mem.
> 
> I believe it's important to start with explanation *why* it is important to
> have kernel allocations on removable devices.

Hi Mike,

not speaking for Kyungsan here, but I am starting to tackle hypervisor 
clustering and VM migration over cxl.mem [1].

And in my mind, at least one reason that I can think of having kernel 
allocations from cxl.mem devices is where you have multiple VH 
connections sharing the memory [2]. Where for example you have a user 
space application stored in cxl.mem, and then you want the metadata 
about this process/application that the kernel keeps on one hypervisor 
be "passed on" to another hypervisor. So basically the same way 
processors in a single hypervisors cooperate on memory, you extend that 
across processors that span over physical hypervisors. If that makes 
sense...


[1] A high-level explanation is at http://nil-migration.org
[2] Compute Express Link Specification r3.0, v1.0 8/1/22, Page 51, 
figure 1-4, black color scheme circle(3) and bars.

--
Peace can only come as a natural consequence
of universal enlightenment -Dr. Nikola Tesla