Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB)

On 8/17/12 7:15 AM, Marie Bacuno II wrote:
>
> On Aug 16, 2012, at 18:47, Garrett Cooper <[email protected]> wrote:
>
>> On Thu, Aug 16, 2012 at 6:44 PM, Garrett Cooper <[email protected]> wrote:
>>> On Thu, Aug 16, 2012 at 5:46 PM, Gezeala M. Bacuño II <[email protected]> wrote:
>>>> Hello fellow listers,
>>>>
>>>> On a server with 512GB RAM it appears that vm.kmem_size_max is not
>>>> being auto-tuned to use >329853485875 (~307GB).
>>>>
>>>> On this machine vm.kmem_size is equal to vm.kmem_size_max
>>>>
>>>> # from sysctl
>>>> vm.kmem_size_max: 329853485875
>>>> vm.kmem_size: 329853485875
>>>>
>>>> On a machine with 1GB of RAM, I have successfully set vm.kmem_size_max
>>>> to 330GB and vm.kmem_size automatically adjusts to 1GB even if I
>>>> manually set it in /boot/loader.conf.
>>>>
>>>> But on the machine with 512GB of RAM it just resets. For the machine
>>>> to boot, we need to go to the loader prompt and issue:
>>>>
>>>> OK set vm.kmem_size_max="300G"
>>>> OK boot
>>>>
>>>> On all PCBSD (8,9) or FreeBSD (8.1,8.2,9) machines we have,
>>>> vm.kmem_size_max is always set to 329853485875.
>>>>
>>>> How can I increase vm.kmem_size_max to use at least 500GB? And how is
>>>> 329853485875 determined (formula)? I need to increase vm.kmem_size_max
>>>> and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say
>>>> 490GB.
>>>>
>>>> I'm browsing thru the source code at
>>>> http://fxr.watson.org/fxr/search?v=FREEBSD9&string=vm.kmem_size_max
>>>> and I'm still trying to make sense of how vm.kmem_size_max is
>>>> computed.
>>>>
>>>> I have posted the same topic on forums.freebsd.org but I'm not getting
>>>> any recommendations.
>>>>
>>>> Please see the link for additional details:
>>>> http://forums.freebsd.org/showthread.php?t=33977
>>>
>>> Have you tried defining VM_KMEM_SIZE_MAX to your target value?
>>>
>>> Its architecture specific BTW... see
>>> sys/<architecture>/include/vmparam.h -- look for `VM_KMEM_SIZE_MAX`.
>>
>> Also, it's a tunable, not a sysctl... so you need to set the value in
>> /boot/loader.conf .
>> -Garrett
>
> Thanks for the quick reply.
>
> Yes, had it set on /boot/loader.conf and by trial and error on the loader prompt.
>
> We were able to bump it to 400G successfully. Tried 500G and 450G and the machine just spews out garbage in the screen.
>
> The latest output from "zfs-stats -a" with vm.kmem_size_max="400G" is in the forum: http://forums.freebsd.org/showthread.php?t=33977
>
> About the code, I am looking into amd64 arch. Still checking the values of the variables.. Can't just retrieve them using getconf. If you can point me to a doxygen like documentation appreciate it a lot.
>
> Where does the constant value 329853485875 came from?
>

It comes from this macro:

#define VM_KMEM_SIZE_MAX        ((VM_MAX_KERNEL_ADDRESS - \
     VM_MIN_KERNEL_ADDRESS + 1) * 3 / 5)

((1<<39) * 3 / 5) = 329853488332

AFAIK, VM_MAX_KERNEL_ADDRESS is limited to 512Gb.  May be it's time to 
increase it again.  I would asked kib@ or alc@ about that.

-- 
Andrey Zonov
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[email protected]"