pgCluu 3.0 has been published

This is a multi-part message in MIME format.
--------------1D98DF0A99168479014FE835
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Ubud, Bali - September 18th, 2019

**pgCluu 3.0 released**

PostgreSQL Cluster Utilization
----------------------------------------

pgCluu is a Perl program used to perform a full audit of a PostgreSQL
Cluster performances. It is divided in two parts, a collector used to
grab statistics on the PostgreSQL server using psql and sar, a reports
builder that will generate all HTML and charts output.

This release is a major release that adds lot of improvement and
new reports. It also improve performances and fully support a CGI
mode with dynamic content for realtime reports.

New or enhanced reports/features:

  * The CGI mode to provide dynamic reports on time selection is now
    production ready and has left the beta stage. See documentation
    on how to enable it. Debian packaging enable this mode by default.
  * Installation and configuration is fully managed from Makefile.PL
    script. This will help distribution packaging with additional
    configuration directives to control the behavior of pgcluu scripts
    with generic systemd service and timer files.
  * Add incremental mode to pgcluu. That mean that you don't have to
    remove the content of the report directory anymore before running
    pgcluu again and when pgcluu detect that you are running in this
    mode. The incremental mode is detected from statistic directory
    when you have used the --rotate-daily or --rotate-hourly option
    to pgcluu_collectd.
  * Rewrite Stats I/O reports to display read/hit per table or indexes
    during the audit period reported.
  * Move Xlog (Wal bytes written) report under replication menu as
    data comes from pg_stat_replication.
  * Explode System and Cluster reports into several reports dedicated
    files to limit the size of the HTML files in incremental mode.
  * Add collect and report of vm.nr_overcommit_hugepages.
  * Add report on pages scanned to report number of pages scanned by
    the kswapd daemon and number of pages scanned directly per second
    (pgscank/s and pgscand/s). It also reports the number of pages
    the system has reclaimed from cache (pagecache and swapcache) per
    second to satisfy its memory demands (pgsteal/s). On a second
    axis %vmeffto reports the the efficiency of page reclaim.
  * All pages reports have been moved as sub menu of the Page menu and
    pgfree has been removed from the page fault report.
  * Limit system cache statistics report to pgpgin/s and pgpgout/s and
    a new report is dedicated to Page fault statistics with majflt/s,
    minflt/s and pgfree/s dataset. Dataset minflt/s is calculated from
    sar output as result of "fault/s - majflt/s".
  * All statistics about pg_stat_user_* and pg_stat_xact_user_* views
    are now stored as a snapshot at start and at end of the audit
    period and each hour if you are running in incremental mode. The
    old behavior was to append the snapshot at each running interval
    to the data file but this is not necessary and this can use huge
    disk space if you have lot of table and indexes in your database.
    Backward compatibility is preserved.
  * New report of json versus jsonb columns in each database.
  * Add keepalive kernel configuration in SysInfo report.
  * Separate bgwriter "allocated" buffers in a dedicated report for
    better reading.
  * Add new report about estimated memory workload based on the value
    of Committed_AS from /proc/meminfo. Committed_AS is the total
    amount of memory estimated to complete the workload. This value
    represents the worst case scenario value, and also includes swap
    memory. The report show the memory workload aver the time.

New command line options:

  * Add option -x, --external-menu to pgcluu to save the common menu
    in menu.html and load it into each report using w3-include-html
    attribute from w3.js. This will only work if access to HTML report
    is through a Web server, not using the file:// protocol.
  * Add new option --retention to pgcluu to set number of rolling days
    to keep in report directory. Default is to store indefinitely.
  * Add option --retention NDAYS to pgcluu_collectd to set the number
    of rolling days to keep in data directory in incremental mode.
    Default is to store indefinitely.

New configuration directives :

  * Add configuration directive STATS_REPORT_CACHING to pgcluu.conf
    file. This directive must be enabled in CGI mode. In this case the
    pgcluu script must be run periodically at least each 5 minutes to
    build and update cached statistics. In cache mode pgcluu generate
    binary file in the statistics directory and nothing in the report
    directory. The pgcluu.service and pgcluu.timer can be use, pgcluu
    will read the configuration file and switch automatically to cache
    mode if STATS_REPORT_CACHING is enabled.
  * Add MAX_RENDERED_DAYS configuration directive to set the maximum
    number of days in a graph. Using default pgcluu_collectd 1 minute
    interval each, daily graph will have 1440 points. Having too much
    data to render is not a good point for performances with the
    current code. The user can set is own graph period, default is to
    limit to the first seven days of the selected period. This mean a
    maximum of 10080 points for one week, this seems a safe limit.

It also fixes some issues reported by users since last ten months.

-- 
Gilles Darold
http://www.darold.net/


--------------1D98DF0A99168479014FE835
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p>Ubud, Bali - September 18th, 2019<br>
      <br>
      **pgCluu 3.0 released**<br>
      <br>
      PostgreSQL Cluster Utilization<br>
      ----------------------------------------</p>
    <p>pgCluu is a Perl program used to perform a full audit of a
      PostgreSQL<br>
      Cluster performances. It is divided in two parts, a collector used
      to<br>
      grab statistics on the PostgreSQL server using psql and sar, a
      reports<br>
      builder that will generate all HTML and charts output.<br>
      <br>
    </p>
    <p>This release is a major release that adds lot of improvement and<br>
      new reports. It also improve performances and fully support a CGI<br>
      mode with dynamic content for realtime reports.</p>
    <p>New or enhanced reports/features:</p>
    <ul>
      <li>The CGI mode to provide dynamic reports on time selection is
        now<br>
        production ready and has left the beta stage. See documentation<br>
        on how to enable it. Debian packaging enable this mode by
        default.</li>
      <li>Installation and configuration is fully managed from
        Makefile.PL<br>
        script. This will help distribution packaging with additional<br>
        configuration directives to control the behavior of pgcluu
        scripts<br>
        with generic systemd service and timer files.</li>
      <li>Add incremental mode to pgcluu. That mean that you don't have
        to<br>
        remove the content of the report directory anymore before
        running<br>
        pgcluu again and when pgcluu detect that you are running in this<br>
        mode. The incremental mode is detected from statistic directory<br>
        when you have used the --rotate-daily or --rotate-hourly option<br>
        to pgcluu_collectd.</li>
      <li>Rewrite Stats I/O reports to display read/hit per table or
        indexes<br>
        during the audit period reported.</li>
      <li>Move Xlog (Wal bytes written) report under replication menu as<br>
        data comes from pg_stat_replication.</li>
      <li>Explode System and Cluster reports into several reports
        dedicated<br>
        files to limit the size of the HTML files in incremental mode.</li>
      <li>Add collect and report of vm.nr_overcommit_hugepages.</li>
      <li>Add report on pages scanned to report number of pages scanned
        by<br>
        the kswapd daemon and number of pages scanned directly per
        second<br>
        (pgscank/s and pgscand/s). It also reports the number of pages<br>
        the system has reclaimed from cache (pagecache and swapcache)
        per<br>
        second to satisfy its memory demands (pgsteal/s). On a second<br>
        axis %vmeffto reports the the efficiency of page reclaim.</li>
      <li>All pages reports have been moved as sub menu of the Page menu
        and<br>
        pgfree has been removed from the page fault report.</li>
      <li>Limit system cache statistics report to pgpgin/s and pgpgout/s
        and<br>
        a new report is dedicated to Page fault statistics with
        majflt/s,<br>
        minflt/s and pgfree/s dataset. Dataset minflt/s is calculated
        from<br>
        sar output as result of "fault/s - majflt/s".</li>
      <li>All statistics about pg_stat_user_* and pg_stat_xact_user_*
        views<br>
        are now stored as a snapshot at start and at end of the audit<br>
        period and each hour if you are running in incremental mode. The<br>
        old behavior was to append the snapshot at each running interval<br>
        to the data file but this is not necessary and this can use huge<br>
        disk space if you have lot of table and indexes in your
        database.<br>
        Backward compatibility is preserved.</li>
      <li>New report of json versus jsonb columns in each database.</li>
      <li>Add keepalive kernel configuration in SysInfo report.</li>
      <li>Separate bgwriter "allocated" buffers in a dedicated report
        for<br>
        better reading.</li>
      <li>Add new report about estimated memory workload based on the
        value<br>
        of Committed_AS from /proc/meminfo. Committed_AS is the total<br>
        amount of memory estimated to complete the workload. This value<br>
        represents the worst case scenario value, and also includes swap<br>
        memory. The report show the memory workload aver the time.</li>
    </ul>
    <p>New command line options:</p>
    <ul>
      <li>Add option -x, --external-menu to pgcluu to save the common
        menu<br>
        in menu.html and load it into each report using w3-include-html<br>
        attribute from w3.js. This will only work if access to HTML
        report<br>
        is through a Web server, not using the <a class="moz-txt-link-freetext" href="file://">file://</a> protocol.</li>
      <li>Add new option --retention to pgcluu to set number of rolling
        days<br>
        to keep in report directory. Default is to store indefinitely.</li>
      <li>Add option --retention NDAYS to pgcluu_collectd to set the
        number<br>
        of rolling days to keep in data directory in incremental mode.<br>
        Default is to store indefinitely.</li>
    </ul>
    <p>New configuration directives :</p>
    <ul>
      <li>Add configuration directive STATS_REPORT_CACHING to
        pgcluu.conf<br>
        file. This directive must be enabled in CGI mode. In this case
        the<br>
        pgcluu script must be run periodically at least each 5 minutes
        to<br>
        build and update cached statistics. In cache mode pgcluu
        generate<br>
        binary file in the statistics directory and nothing in the
        report<br>
        directory. The pgcluu.service and pgcluu.timer can be use,
        pgcluu<br>
        will read the configuration file and switch automatically to
        cache<br>
        mode if STATS_REPORT_CACHING is enabled.</li>
      <li>Add MAX_RENDERED_DAYS configuration directive to set the
        maximum<br>
        number of days in a graph. Using default pgcluu_collectd 1
        minute<br>
        interval each, daily graph will have 1440 points. Having too
        much<br>
        data to render is not a good point for performances with the<br>
        current code. The user can set is own graph period, default is
        to<br>
        limit to the first seven days of the selected period. This mean
        a<br>
        maximum of 10080 points for one week, this seems a safe limit.</li>
    </ul>
    <p>It also fixes some issues reported by users since last ten
      months.</p>
    <pre class="moz-signature" cols="72">-- 
Gilles Darold
<a class="moz-txt-link-freetext" href="http://www.darold.net/">http://www.darold.net/</a></pre>
  </body>
</html>

--------------1D98DF0A99168479014FE835--