[SPDK] Re: Rocksdb over SPDK/Blobfs

The SPDK cache is only used for write buffering and readahead (for compaction).  So if your workload is only doing random GET operations, the SPDK cache will have little to no effect.

-Jim


On 11/25/20, 8:39 AM, "Dimitra Giantsidi" <dimitra.giantsidi(a)gmail.com> wrote:

    Hi Jim,

    Thanks for your prompt reply! That was really fast!
    Hmm.. since the kernel page cache has no limit, that makes sense.
    However, my workload size is estimated to be 4959.1 MB, namely, almost as
    much as the SPDK cache. I would assume that in this setting SPDK cache
    would host most of the data, so both versions should be equivalent (more or
    less), no?

    Assuming that SPDK cache can be as much as the system's memory (or at least
    bigger than the DB size) -- should I expect that SPDK performs better than
    NVMe driver?

    (I am running some experiments with the configurations you proposed and I
    will post them as soon as possible :) )


    Thank you for your help,
    Dimitra

    On Wed, 25 Nov 2020 at 17:07, Harris, James R <james.r.harris(a)intel.com>
    wrote:

    > Hi Dimitra,
    >
    > I think what's happening here is that your database is small enough that
    > it fits mostly (or even entirely) in the kernel page cache.  This means
    > that when using the kernel NVMe driver + filesystem, you are avoiding a lot
    > of reads to the SSD since the data is cached in memory.
    >
    > This SPDK GitHub issue describes this phenomenon a bit more:
    > https://github.com/spdk/spdk/issues/1555
    >
    > I would be interested to see the comparison if you make your database size
    > much bigger, such that it cannot all fit into page cache.
    >
    > Thanks,
    >
    > Jim
    >
    >
    > On 11/25/20, 6:30 AM, "Dimitra Giantsidi" <dimitra.giantsidi(a)gmail.com>
    > wrote:
    >
    >     Hi everyone,
    >
    >     I am running the rocksdb::RandomTransaction benchmark with *1 thread*
    > with
    >     the following specifications:
    >
    >     RocksDB:    version 5.14
    >     Date:       Wed Nov 25 13:05:48 2020
    >     CPU:        16 * Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
    >     CPUCache:   16384 KB
    >     Keys:       16 bytes each
    >     Values:     1024 bytes each (1024 bytes after compression)
    >     Entries:    5000000
    >     Prefix:    0 bytes
    >     Keys per prefix:    0
    >     RawSize:    4959.1 MB (estimated)
    >     FileSize:   4959.1 MB (estimated)
    >     Write rate: 0 bytes/second
    >     Read rate: 0 ops/second
    >     Compression: NoCompression
    >     Memtablerep: skip_list
    >     Perf Level: 1
    >
    > -----------------------------------------------------------------------------------------------------------------------
    >
    >     When using the kernel NVMe driver I have better performance (x 5
    > times). In
    >     particular the db_bench output is:
    >     NVMe driver: randomtransaction :      *16.727 micros/op 59782 ops/sec;*
    >      1.5 MB/s (transactions:5000000 aborts:0)
    >
    >     SPDK: randomtransaction :      *71.226 micros/op 14039 ops/sec;*    0.3
    >     MB/s (transactions:5000000 aborts:0)
    >
    >     I am using SPDK v18.01.1 and DPDK v18.02. The SPDK cache is 4GB and
    >     apparently I cannot increase it in my platform.
    >
    >     I am not sure how to explain this behavior. I would appreciate any
    >     help/comment on the result.
    >
    >     Thanks,
    >     Dimitra
    >
    >     --
    >     Giantsidi Dimitra
    >
    >     e-mail:dimitra.giantsidi(a)gmail.com
    >     _______________________________________________
    >     SPDK mailing list -- spdk(a)lists.01.org
    >     To unsubscribe send an email to spdk-leave(a)lists.01.org
    >
    > _______________________________________________
    > SPDK mailing list -- spdk(a)lists.01.org
    > To unsubscribe send an email to spdk-leave(a)lists.01.org
    >


    -- 
    Giantsidi Dimitra

    e-mail:dimitra.giantsidi(a)gmail.com
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org