Roundup 2.3.0b2 released

Hello Everybody:

I released Roundup 2.3.0b2 a few minutes ago.

(Beta 1 was broken and was using the 2.2.0 announcement.)

You can test it with:

  pip install roundup==2.3.0b2

in your venv. Docs for it are at:

   https://www.roundup-tracker.org/dev_docs/

and are up to date except that docs/upgrading.html seems to be cached
on an older version. Adding ?nocache=1 to the url returns the right
version. I wish there was some way to flush sourceforge's cache.

Please test the release. The last release escaped with a few bugs that
I had hoped would be caught during the alpha/beta period. Also since
the docs have been partly overhauled, checking those would be welcome
as well.

I am still planning on a 2.3.0 release on 13 July.

The announcement message, change log etc. is:

[...]

Among the notable improvements from the 2.2.0 release are:

* Dockerfile demo mode implemented.

* SQLite backends can use WAL mode to reduce blocking between readers
  and writers.

* Redis can be used for session database with SQLite and dbm
  backends. Provides a major performance improvement.

* roundup-mailgw can use OAUTH authentication to SMTP
  server. (roundup-mailgw command line options changed as a result.)

* Postgres full text index can now be enabled.

* Modifications to in-reply-to threading when there are multiple
  matches.

* Many updates to documentation to make it scannable, useful and
  work on mobile.

* Admin documentation includes a section on setting up Content
  Security Policy (CSP)

* REST now allows rate limiting headers to be accessed by client
  javascript.

* Default number of rounds for PBKDF2 updated to 2M to account for
  improvements in password crackers and CPU power.

* Support PBKDF2 with SHA512 for password storage

* Deprecate SSHA password hash function.

* roundup-admin reindex can be done in batches to managle load
  incurred by reindexing.

* roundup-admin can list avaailable templates and their installed
  locations.

* Crash fixes in detector handling, configuration handling, 

The file CHANGES.txt has a detailed list of feature additions and
bug fixes (51) for each release. The most recent changes from
there are at the end of this announcement. Also see the
information in doc/upgrading.txt.

If you find bugs, please report them to issues AT roundup-tracker.org
or create an account at https://issues.roundup-tracker.org and open a
new ticket. If you have patches to fix the issues they can be attached
to the email or uploaded to the tracker.

[...]

Recent Changes
==============

From 2.2.0 to 2.3.0b2

Fixed:
------

- Updated directions for verifying Roundup distribution using pgp.
- Dockerfile healthcheck fixed so it works when trackers are
  specified on command line. Also cleanup of unneeded
  packages. (John Rouillard)
- issue2551224 - Replace dbm db for sessions and otks when using
  sqlite. New databases are created for session data (db-session)
  and one time key data (db-otk). The data is ephemeral so no
  need to migrate. (John Rouillard)
- issue2551223 - Timestamps are truncated in mysql and postgresql
  for session and otk database tables. Modify db schema to use a
  numeric type that preserves more significant figures. See
  upgrading.txt for required steps. (John Rouillard)
- added more testing of BasicDatabase to support use of SQLite
  for that purpose. Had to fix memory, rdbms and dbm edge cases
  due to new tests. (John Rouillard)
- issue2551138 - roundup-server with ssl under python2 throws
  traceback on socket close. Not sure how this got fixed,
  but after fixing issue2551137 it was not an issue anymore.
- issue2551137 - roundup-server won't run with ssl under python3
  Fixed by using SocketIO and manually adding buffering io and
  catching SSL.ZeroReturnError indicating SSL has been shut down.
- add caching header for text/javascript in addition to depricated
  application/javascript. (John Rouillard)
- Enable postgres-fts: fix indexer-common::get_indexer so it returns a
  postgresql-fts Test code paths in get_indexer. (John Rouillard)
- Fix Postgres native-fts, implement a two phase initialization of the
  indexer. The native-fts one gets assigned after the database
  connection is open. (John Rouillard)
- fix crash if postgresql native-fts backend is asked to index content
  with null bytes. (John Rouillard)
- issue2551232 - modify in-reply-to threading when multiple matches
  Change how in-reply-to threading works in the mailgw. If there is
  more than one issue with a matching parent message, fall back to
  subject matching. See upgrading.txt for details. (John Rouillard)
- issue2551195 - port scripts from optparse to argparse (Ralf Schlatterbeck)
- issue2551246 - mitigation, document how -u doesn't work for
  roundup-admin. (John Rouillard)
- Document better that files in the template or static_files
  directories accessed via @@file are available to any user with the
  url. (John Rouillard)
- Fix final exception handler in roundup-server to send proper
  Content-Length header to the client. (John Rouillard)
- Fix traceback if Origin header is missing. (John Rouillard)
- issue2551250: Fix sorting of detectors even if there are two with the
  same name and priority (can happen if they are created in two
  different files).
- Fix Traceback when a numeric order attribute is empty (Ralf
  Schlatterbeck)
- Update some template schema files to assign Register permissions for the
  Anonymous user. Replaces the old Create permission. (John Rouillard)
- Allow '*' and explicit origins in allowed_api_origins. Only return 
  'Access-Control-Allow-Credentials' when not matching '*'. Fixes
  security issue with rest when using '*'.  (John Rouillard)
- issue2551263: In REST response expose rate limiting, sunset, allow
  HTTP headers to calling javascript.  (John Rouillard)
- issue2551257: When downloading an attached (user supplied file),
  make sure that an 'X-Content-Type-Options: nosniff' header is sent.
  (John Rouillard)
- issue2551252 - default number of rounds for PKDF2 password increased
  to 2,000,000.  (John Rouillard)
- issue2551251 - migrate/re-encrypt PBKDF2 password if stored
  password used a smaller number of rounds than set in
  password_pbkdf2_default_rounds.  (John Rouillard)
- upgrade from jquery-3.5.1 to jquery-3.6.3. Update user.help.html
  to new version.  (John Rouillard)
- Dockerfile scanned with hadolint. Fixed multiple issues. (John Rouillard)
- fix crash due to invalid initialization/reset of configuration.py
  option_validators. Crashed roundup-admin on second command if an
  option_validator was added by a detector or extension. (John Rouillard)
- Dockerfile uses dumb-init to properly wait for child/zombie
  processes. Defense against child process starting from detector
  and becoming a zombie when its roundup-server instance exits.
  (John Rouillard)
- Move installed frontend/Zope back to frontend/ZRoundup
  directory. This better identifies the directory when copied into
  the Zope framework. It also matches existing
  documentation. (John Rouilard)
- Multiple fixes/updates for installation documentation.
  Including docker shell/admin/demo mdoes. (John Rouillard)
- Invalid item identifiers passed to REST endpoint return a 404
  rather than a 400 error. E.G. /rest/data/issue/issue4 (rather
  than .../issue/4). (John Rouillard)

Features:
---------

- Add warning about limited Python 2 support lifetime to install and
  upgrading docs. (John Rouillard)
- Dockerfile supports demo mode for instant gratification
  8-). Also supports shell and admin mode (John Rouillard)
- Dockerfile build allows adding additional python packages via
  pip, setting UID tracker is run under. (John Rouillard)
- issue2551140 - Added redis as a session and otk database for use
  with anydbm and sqlite primary databases. (John Rouillard)
- issue2550559 - Pretty printing / formatting for Number types.
  Added pretty(format='%0.3f') method to NumberHTMLProperty to
  print numeric values. If value is None, return empty string
  otherwise str() of value. (John Rouillard)
- sqlite native-fts backend now uses the stopwords list in config.ini
  to filter words from queries. (Stopwords are still indexed so that
  phrase/proximity searches still work.) (John Rouillard)
- sqlite databases use WAL mode when *created* to improve read
  concurrency. Existing sqlite database still use rollback journal
  mode. See upgrading.txt for details. (John Rouillard)
- issue2551233 - create new roundup-admin command "templates" list all
  template names, location and descriptions. Should help find where
  /usr/share/roundup/templates is buried during some install
  mechanisms. Does not need a tracker home to run. (John Rouillard)
- Add OAuth authentication to the mailgw script. Now IMAPS can be used
  with OAuth as required by several large cloud providers. Move command
  line processing of the mailgw script to ``argparse``. Note that the
  command line options of the mailgw have changed, see upgrading.txt for
  details. (Ralf Schlatterbeck)
- issue2551243: schema-dump.py enhanced with anti-CSRF headers. Flake8
  cleanup and python2 support. (John Rouillard)
- issue2551253 - new password hash PBDKF2-SHA512 added. Not
  available by default. Follow directions in upgrading document
  to use.  (John Rouillard)
- roundup-admin migrate command reports the schema version.
- issue2551262 - the mail gateway subject prefix now allows spaces
  before/after prefix. Also allow spaces between classname and id
  number in prefix designator. So "[ issue 23   ] subject" is parsed
  like "[issue23] subject". (John Rouillard)
- [doc]: add section on implementing CSP for Roundup to admin
  doc. (John Rouillard)
- issue2551265 - deprecate SSHA password hash method. Users using SSHA
  passwords will have their passwords transprently upgraded to PBKDF2
  derived hash on next login. (John Rouillard)
- issue2551253 - Modify password PBKDF2 method to use SHA512. New
  hash function using PBKDF2-SHA512 available. Will be default in
  future. Directions for upgrading security by using it now is
  included in upgrading.txt. (John Rouillard)
- issue2551275 - Allow configuring max_children in roundup-server.
  When using roundup-server in fork mode, allow raising number of
  forked children above the default of 40. (Original patch by Joseph
  Myers, config settings and docs by John Rouillard.)
- roundup-admin genconfig does not need a tracker home to run. (John
  Rouillard)
- issue2551190 - Allow roundup-admin reindex to work in
  batches. Running roundup-admin -i ... reindex issue:1-1000 will
  reindex the first 1000 issues while reporting any missing issues
  in the range. Also completion progress is reported when indexing a
  specific class.


--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.