Possible inefficiency of the serialization for reporting

Hello,

I would just like to share a few thoughts about the reporter. In
particular, I think that the current deserialization method for std::vector
in the typekits is inefficient for large arrays. I have a typekit where I
have:

1) 10-20 scalars
2) and a few std::vector arrays. Two are big (> 400 elements), one which
has cca 100 elements, and a few more with cca 10 samples. Each element is a
float.

My computation component spits data at 25 Hz. What I observed, together
with Ruben, is that corresponding reporter consumes 100% of CPU (single
core). In our app, we use netCDF reporter, and after some analysis I found
out that I miss quite some samples -- most of the time every 2nd sample.

Looking at a deployer log it is pretty much clear that serialization
decomposes each array into a sequence of scalars. I can see that this is
some legacy plus that it is "OK" for small arrays. For big arrays this is
overkill, simply because each time one sample (std::vector) has to be
logged, instead of being written to consecutive mem. locations each element
of the vector is appended to a separate array.

To be honest, I don't have a clue how netCDF works nor how reporting works
under the hood, I am just thinking why is reporter slow in my case.

BTW, I am using an Intel SSD (1-2 years old), so write times should be
fast. Looking at iotop, it says that write speed is < 500 kB/s.

Moreover, reading of the generated .nc files is quite slow. When I convert
this to .mat (either from python or MATLAB) reading times are much better
-- 10MB file loads instantly. I kinda believe if number of columns would be
smaller (std::vector -> one single entry in the netCDF file), read time
would be shorter -- just my intuition.

Finally, a question for devs: how hard would be to change serialization of
std::vector-like arrays?

Cheers,
Milan

-- 
Milan Vukov, PhD Student
KU Leuven, Department of Electrical Engineering (ESAT)
STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics
Kasteelpark Arenberg 10, bus 2446, B-3001 Leuven-Heverlee, Belgium
e-mail: [email protected]
url: http://homes.esat.kuleuven.be/~mvukov/
phone: +32-479-813256 (BE), +381-64-1541622 (SR)
disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm

-- 
Orocos-Dev mailing list
[email protected]
http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev