Re: vm.kmem_size_max and vm.kmem_size capped at 329853485875 (~307GB)

On Mon, Aug 20, 2012 at 9:07 AM, Gezeala M. Bacuño II <[email protected]> wrote:
> On Mon, Aug 20, 2012 at 8:22 AM, Alan Cox <[email protected]> wrote:
>> On 08/18/2012 19:57, Gezeala M. Bacuño II wrote:
>>>
>>> On Sat, Aug 18, 2012 at 12:14 PM, Alan Cox<[email protected]>  wrote:
>>>>
>>>> On 08/17/2012 17:08, Gezeala M. Bacuño II wrote:
>>>>>
>>>>> On Fri, Aug 17, 2012 at 1:58 PM, Alan Cox<[email protected]>   wrote:
>>>>>>
>>>>>> vm.kmem_size controls the maximum size of the kernel's heap, i.e., the
>>>>>> region where the kernel's slab and malloc()-like memory allocators
>>>>>> obtain
>>>>>> their memory.  While this heap may occupy the largest portion of the
>>>>>> kernel's virtual address space, it cannot occupy the entirety of the
>>>>>> address
>>>>>> space.  There are other things that must be given space within the
>>>>>> kernel's
>>>>>> address space, for example, the file system buffer map.
>>>>>>
>>>>>> ZFS does not, however, use the regular file system buffer cache. The
>>>>>> ARC
>>>>>> takes its place, and the ARC abuses the kernel's heap like nothing
>>>>>> else.
>>>>>> So, if you are running a machine that only makes trivial use of a
>>>>>> non-ZFS
>>>>>> file system, like you boot from UFS, but store all of your data in ZFS,
>>>>>> then
>>>>>> you can dramatically reduce the size of the buffer map via boot loader
>>>>>> tuneables and proportionately increase vm.kmem_size.
>>>>>>
>>>>>> Any further increases in the kernel virtual address space size will,
>>>>>> however, require code changes.  Small changes, but changes nonetheless.
>>>>>>
>>>>>> Alan
>>>>>>
> <<snip>>
>>>> Your objective should be to reduce the value of "sysctl vfs.maxbufspace".
>>>> You can do this by setting the loader.conf tuneable "kern.maxbcache" to
>>>> the
>>>> desired value.
>>>>
>>>> What does your machine currently report for "sysctl vfs.maxbufspace"?
>>>>
>>> Here you go:
>>> vfs.maxbufspace: 54967025664
>>> kern.maxbcache: 0
>>
>>
>> Try setting kern.maxbcache to two billion and adding 50 billion to the
>> setting of vm.kmem_size{,_max}.
>>

2 : 50 ==>> is this the ratio for further tuning
kern.maxbcache:vm.kmem_size? Is kern.maxbcache also in bytes?

>
> Thank you. We'll try this and post back results.
>
>>> Other (probably) relevant values:
>>> vfs.hirunningspace: 16777216
>>> vfs.lorunningspace: 11206656
>>> vfs.bufdefragcnt: 0
>>> vfs.buffreekvacnt: 2
>>> vfs.bufreusecnt: 320149
>>> vfs.hibufspace: 54966370304
>>> vfs.lobufspace: 54966304768
>>> vfs.maxmallocbufspace: 2748318515
>>> vfs.bufmallocspace: 0
>>> vfs.bufspace: 10490478592
>>> vfs.runningbufspace: 0
>>>
>>> Let me know if you need other tuneables or sysctl values. Thanks a lot
>>> for looking into this.
>>>
>>

Setting the following as instructed, machine started successfully with
446GB for vm.kmem_size/_max.

kern.maxbcache: 2000000000
vm.kmem_size_max: 479496729600
vm.kmem_size: 479496729600

## auto-tuned
vfs.maxbufspace: 1999994880
...
vfs.hirunningspace: 16777216
vfs.lorunningspace: 11206656
vfs.bufdefragcnt: 0
vfs.buffreekvacnt: 2
vfs.bufreusecnt: 11511
vfs.hibufspace: 1999339520
vfs.lobufspace: 1999273984
vfs.maxmallocbufspace: 99966976
vfs.bufmallocspace: 0
vfs.bufspace: 377028608
vfs.runningbufspace: 0

## additional manual tuning
vfs.zfs.arc_max: 455521893120
vfs.zfs.arc_min: 227760946560
kern.ipc.semmni: 256
kern.ipc.semmns: 512
kern.ipc.semmnu: 256
kern.ipc.shm_use_phys: 1
kern.ipc.shmmax: 24000000000
kern.ipc.shmall: 5859375
kern.ipc.nmbclusters: 2560000
kern.maxfiles: 197248

We'll do some further tests and report back if there are any issues.

Thanks a lot!!
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[email protected]"