PCP QA and systemd ... what a nightmare

What I've observed over the past ugly and frustrating week ...

1. PCP QA scripts are expecting output from the init-rc scripts.  When
    these are run via systemctl that output is siphoned off, reformatted
    and written into a binary journal.  To get the text back reliably
    from the journals has proven impossible:
    - gratuitous white-space stripping changes the text
    - adding "..done" (or similar) annotation changes the text
    - journal writing is asynchronous, which means you cannot reliably
      tell when (a) any previous writes are done, nor (b) when the
      writes from the most recent init-rc script invocation are done
    - different platforms require different commands to dump some/all
      of the journal
    The only approach has been to dump&grep the journal before, run the
    init-rc script, sleep N (N=1 is not enough, N=2 may not be enough),
    dump&grep the journal after and diff the before and after dumps ...
    and I cannot make this work reliably for all tests on all platforms.

2. Some of the QA scripts are testing error paths or require
    non-standard setup (e.g. disable the primary pmlogger), and systemd
    just gets in the way for these tests, or prattles on like:
        Warning: pmlogger.service changed on disk. ...
    or tries to autorestart something I don't want restarted.

3. On some platforms, adding systemd into the mix has increased the
    running time for QA scripts that restart pmcd & pmlogger by something
    more than 30 seconds.  And there are more than 100 tests like this.
    Checking individual tests is painful, but a whole QA run may take
    over an hour longer.

What I've observed over and over is that new test "failures" are 100%
a result of systemd being in the mix and have nothing to do with the PCP
functionality we're trying to test.  And each test failure can only be
fixed by difficult test re-engineering in scenarios that are sometimes
race-dependent on a single platform and often different from one
platform to the next.

What I am proposing (and soliciting feedback on, if you're still reading 
this), is:

1. Build new tests to run only on systems with systemd in play that
    check the init-rc scripts work for starting and stopping PCP
    daemons.  Possibly add new tests for autorestart testing.

2. Otherwise treat systemd like the pox for the purposes of PCP QA
    testing and explicitly avoid it ... there is already a
    PCPQA_SYSTEMD guillotine in the code, waiting to be turned on.
    I will set PCPQA_SYSTEMD=no in the environment for all my testing,
    others are free to make their own choices.

Cheers,
The QA Pixie

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links:

You receive all messages sent to this group.

View/Reply Online (#15083): https://groups.io/g/pcp/message/15083
View All Messages In Topic (1): https://groups.io/g/pcp/topic/4346600
Mute This Topic: https://groups.io/mt/4346600?uid=174580
New Topic: https://groups.io/g/pcp/post
-=-=-
pcp mailing list
[email protected]
https://groups.io/g/pcp/messages
-=-=-
Change Your Subscription: https://groups.io/g/pcp/editsub?uid=174580
Group Home: https://groups.io/g/pcp
Contact Group Owner: [email protected]
Terms of Service: https://groups.io/static/tos
Unsubscribe: https://groups.io/g/pcp/leave/354243/563757577/xyzzy
-=-=-=-=-=-=-=-=-=-=-=-