Re: Any input for some talk about usage of Debian in HPC

On 19/05/2024 12:12, Andreas Tille wrote:
> Hi,
> 
> I have an invitation to have some talk with the title
> 
>     Debian GNU/Linux for Scientific Research
> 
> Abstract:
> 
>     Over the past decade, Enterprise Linux has dominated large-scale
>     research computing infrastructure. However, recent developments have
>     sparked increased interest in community-led alternatives. Debian
>     GNU/Linux, a long-standing choice among researchers for supporting
>     scientific work, is experiencing a renewed interest for High-Throughput
>     Computing (HTC) and High-Performance Computing (HPC) applications.  This
>     presentation will provide an overview of how Debian is being utilized to
>     support scientific research and will include a case study showcasing the
>     migration of HTC operations from Enterprise Linux 7 (EL7) to Debian.
> 
> While I could talk about Debian Science and Debian Med in general it
> would be cool to reference to some real life examples where Debian is
> used in Science and what might be the reason to use Debian.

Hi, Andreas.

The Sanger Centre in the UK use Ubuntu + OpenStack + Ceph:

https://www.sanger.ac.uk/group/core-software-services/

I realise that it's not Debian, but it is based on Debian. I went there 
many years ago when they were running Debian on DEC Alpha AXP's, but 
they moved to CentOS because many other Academic HPC centres were using 
it, including ours when I worked at the University of Aberdeen.

This was not a good experience, and they decided to change to Ubuntu 
mainly because of the support provided by Canonical for OpenStack and 
Ceph. However, in my opinion, CentOS/RHEL is not a good platform for 
bioinformatics because the 'Enterprise' approach stifles innovation.

You can't ignore the host OS when you talk about HPC applications and 
the HEP (High Energy Physics) community put a lot of effort into 
developing good node provisioning systems and job-scheduling for HPC. 
Consequently, there was a significant bias towards support for HEP 
applications running under CentOS and less support for bioinformatics.

This was partly the motivation underlying our development of Bio-Linux 
in order to provide biologists with an alternative platform running on 
their own hardware instead of struggling to get the IT department to 
port the software they wanted to use to CentOS. In that respect the 
Debian-Med project was fundamentally important in helping biologists do 
their work outside of the centrally managed 'Enterprise' oriented IT 
policy imposed on us by Universities and Research Institutes.

The Sanger Centre provide a centrally managed HPC that is 
'biologist-friendly' and, I think, is an excellent model of how it 
should be done. However, it does not support the view that Debian should 
be the HPC OS because the main reason they chose Ubuntu was the 
commercial support for OpenStack and Ceph provided by Canonical.

> I personally would like to stress the "we package what we use" aspect
> and the "we mentor upstream to merge competence of the program with
> packaging skills" idea.  Any input would be welcome to cover more ideas.

As you might remember, I built and I advocate the use of 'departmental' 
or 'research-group' clusters. These are much more powerful than an 
individual biologists personal laptop, but are under the administrative 
control of the department or research group that funded their purchase.

In the past, I've used various HPC node-provisioning, cluster filesystem 
and job submission systems running under one version of another of 
Bio-Linux, now using your "med-bio" meta-package to provide 
bioinformatics software instead of the discontinued Bio-Linux packages.

However, I've recently set up a 3-node 'Proxmox-VE' cluster:

> https://www.proxmox.com/en/proxmox-virtual-environment/overview

[Proxmox is a GPL server management system based on Debian]

I'm using the Proxmox cluster for a bioinformatics in schools project 
with the University of Edinburgh:

> https://4273pi.org/

I'm also planning to use it for a new project with the IAEA in Vienna.

I think that giving biologists the choice of running the software they 
want under the OS they choose is very important when innovation is the 
priority of an organisation rather than centralisation of IT systems to 
reduce cost. You can, of course use Proxmox-VE as the node-provisioning 
and shared filesystem of an HPC cluster. Or, simply provide biologists 
with VMs running their OS of choice, administered by themselves e.g. a 
Bio-Linux VM or vanilla Debian etc. etc.

Finally, don't forget about Amdahl's Law:

> https://en.wikipedia.org/wiki/Amdahl%27s_law

There is really no such thing as an HPC or HTP 'application', because 
it's the underlying resource management system of an HPC cluster that 
provides the 'HP'. In my experience, most bioinformatics applications 
are 'embarrassingly' parallel and in this case processes do not 
communicate with each other. The 'HP' is achieved by managing the 
workflow efficiently using e.g. "Slurm" or "[Sun]Grid-Engine".

MPI-parallel processes are subject to diminishing returns as the number 
of processes increases as described by Amdahl's Law. Other 'Map-Reduce' 
workflows are also 'embarrassingly' parallel and GPGPU workflows while 
parallel on a given device are not HPC in the conventional sense despite 
nVidia's claims to the contrary, but are GPGPU-accelerated applications 
that are, typically, executed as embarrassingly parallel work-flows 
across clusters of servers with GPGPUs. All 'SLI' does is to combine 
GPGPUs locally on a server which, again, is not conventional HPC.

All of these technologies have fantastic potential, but I've been very 
frustrated and disappointed many times how difficult it is to utilise 
GPGPU for bioinformatics. It's only recently that e.g. Oxford Nanopore 
Technology and Pacbio have made effective use of GPGPU for base-calling 
long reads. The basic problem is that existing 'classic' bioinformatics 
analyses require re-writing to execute in parallel and it is only a few 
new bioinformatics algorithms that are worthwhile designing for GPGPU.

Bye,

   Tony.