Docker build simplification

Dave Page <[email protected]>
Newsgroups gmane.comp.db.postgresql.pgadmin.devel
Message-ID <CA+OCxoy3ZAKhPcWu1XZrsrT2WdM1uSn+fQUhz7kkMgLFTEujOQ@mail.gmail.com>
The attached patch simplifies the Docker container build so that it can be
done entirely using "docker build" in the root of a source tree. It also
enables building the container from a tarball, not just a git checkout.
This does mean that container builds may now also include any garbage in
the source tree from which they're built, but conversely offers the benefit
of making it easy to build a container with code changes in it without
having to commit first.

Akshay, can you look it over to make sure I haven't done anything obviously
stupid? Feedback from others is also welcome of course.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
docker_simplification.patch (application/octet-stream, 18.7 KB)
diff --git a/Dockerfile b/Dockerfile
new file mode 100644
index 00000000..fb833751
--- /dev/null
+++ b/Dockerfile
@@ -0,0 +1,171 @@
+########################################################################
+#
+# pgAdmin 4 - PostgreSQL Tools
+#
+# Copyright (C) 2013 - 2018, The pgAdmin Development Team
+# This software is released under the PostgreSQL Licence
+#
+#########################################################################
+
+#########################################################################
+# Create a Node container which will be used to build the JS components
+# and clean up the web/ source code
+#########################################################################
+
+FROM node:8-alpine AS app-builder
+
+RUN apk add --no-cache \
+    autoconf \
+    automake \
+    bash \
+    g++ \
+    libc6-compat \
+    libjpeg-turbo-dev \
+    libpng-dev \
+    make \
+    nasm \
+    git \
+    zlib-dev
+
+# Create the /pgadmin4 directory and copy the source into it. Explicitly
+# remove the node_modules directory as we'll recreate a clean version, as well
+# as various other files we don't want
+COPY web /pgadmin4/web
+COPY requirements.txt /pgadmin4/
+RUN rm -rf /pgadmin4/web/*.log \
+           /pgadmin4/web/config_*.py \
+           /pgadmin4/web/node_modules \
+           /pgadmin4/web/regression \
+           `find /pgadmin4/web -type d -name tests` \
+           `find /pgadmin4/web -type f -name .DS_Store`
+
+WORKDIR /pgadmin4/web
+
+# Build the JS vendor code in the app-builder, and then remove the vendor source.
+RUN npm install && \
+	npm audit fix && \
+	rm -f yarn.lock && \
+	yarn import && \
+	yarn audit && \
+	rm -f package-lock.json && \
+    yarn run bundle && \
+    rm -rf node_modules \
+           yarn.lock \
+           package.json \
+           .[^.]* \
+           babel.cfg \
+           webpack.* \
+           karma.conf.js \
+           ./pgadmin/static/js/generated/.cache
+
+#########################################################################
+# Now, create a documentation build container for the Sphinx docs
+#########################################################################
+
+FROM python:3.7-alpine3.9 as docs-builder
+
+# Install only dependencies absolutely required for documentation building
+RUN apk add --no-cache \
+        make \
+        build-base \
+        openssl-dev \
+        libffi-dev && \
+    pip install --no-cache-dir \
+        sphinx \
+        flask_security \
+        flask_paranoid \
+        python-dateutil \
+        flask_sqlalchemy \
+        flask_gravatar \
+        flask_migrate \
+        simplejson \
+        cryptography
+
+# Copy the docs from the local tree. Explicitly remove any existing builds that
+# may be present
+COPY docs /pgadmin4/docs
+COPY web /pgadmin4/web
+RUN rm -rf /pgadmin4/docs/en_US/_build
+
+# Build the docs
+RUN LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 make -C /pgadmin4/docs/en_US -f Makefile.sphinx html
+
+#########################################################################
+# Create additional builders to get all of the PostgreSQL utilities
+#########################################################################
+
+FROM postgres:9.4-alpine as pg94-builder
+FROM postgres:9.5-alpine as pg95-builder
+FROM postgres:9.6-alpine as pg96-builder
+FROM postgres:10-alpine as pg10-builder
+FROM postgres:11-alpine as pg11-builder
+
+#########################################################################
+# Assemble everything into the final container.
+#########################################################################
+
+FROM python:3.7-alpine3.9
+
+# Copy the PG binaries
+COPY --from=pg94-builder /usr/local/bin/pg_dump /usr/local/pgsql-9.4/
+COPY --from=pg94-builder /usr/local/bin/pg_dumpall /usr/local/pgsql-9.4/
+COPY --from=pg94-builder /usr/local/bin/pg_restore /usr/local/pgsql-9.4/
+COPY --from=pg94-builder /usr/local/bin/psql /usr/local/pgsql-9.4/
+
+COPY --from=pg95-builder /usr/local/bin/pg_dump /usr/local/pgsql-9.5/
+COPY --from=pg95-builder /usr/local/bin/pg_dumpall /usr/local/pgsql-9.5/
+COPY --from=pg95-builder /usr/local/bin/pg_restore /usr/local/pgsql-9.5/
+COPY --from=pg95-builder /usr/local/bin/psql /usr/local/pgsql-9.5/
+
+COPY --from=pg96-builder /usr/local/bin/pg_dump /usr/local/pgsql-9.6/
+COPY --from=pg96-builder /usr/local/bin/pg_dumpall /usr/local/pgsql-9.6/
+COPY --from=pg96-builder /usr/local/bin/pg_restore /usr/local/pgsql-9.6/
+COPY --from=pg96-builder /usr/local/bin/psql /usr/local/pgsql-9.6/
+
+COPY --from=pg10-builder /usr/local/bin/pg_dump /usr/local/pgsql-10/
+COPY --from=pg10-builder /usr/local/bin/pg_dumpall /usr/local/pgsql-10/
+COPY --from=pg10-builder /usr/local/bin/pg_restore /usr/local/pgsql-10/
+COPY --from=pg10-builder /usr/local/bin/psql /usr/local/pgsql-10/
+
+COPY --from=pg11-builder /usr/local/bin/pg_dump /usr/local/pgsql-11/
+COPY --from=pg11-builder /usr/local/bin/pg_dumpall /usr/local/pgsql-11/
+COPY --from=pg11-builder /usr/local/bin/pg_restore /usr/local/pgsql-11/
+COPY --from=pg11-builder /usr/local/bin/psql /usr/local/pgsql-11/
+
+WORKDIR /pgadmin4
+ENV PYTHONPATH=/pgadmin4
+
+# Copy in the code and docs
+COPY --from=app-builder /pgadmin4/web /pgadmin4
+COPY --from=app-builder /pgadmin4/requirements.txt /pgadmin4/requirements.txt
+COPY --from=docs-builder /pgadmin4/docs/en_US/_build/html/ /pgadmin4/docs
+
+# Install build-dependencies, build & install C extensions and purge deps in
+# one RUN step
+RUN apk add --no-cache --virtual \
+        build-deps \
+        build-base \
+        postgresql-dev \
+        libffi-dev \
+        linux-headers && \
+    apk add \
+        postfix \
+        postgresql-client \
+        postgresql-libs && \
+    pip install --upgrade pip && \
+    pip install --no-cache-dir -r requirements.txt && \
+    pip install --no-cache-dir gunicorn && \
+    apk del --no-cache build-deps
+
+# Copy the runner script
+COPY pkg/docker/run_pgadmin.py /pgadmin4
+COPY pkg/docker/entrypoint.sh /entrypoint.sh
+
+# Precompile and optimize python code to save time and space on startup
+RUN python -O -m compileall -x node_modules /pgadmin4
+
+# Finish up
+VOLUME /var/lib/pgadmin
+EXPOSE 80 443
+
+ENTRYPOINT ["/entrypoint.sh"]
diff --git a/Makefile b/Makefile
index 6001066b..6000fe1e 100644
--- a/Makefile
+++ b/Makefile
@@ -77,7 +77,7 @@ runtime:
 	cd runtime && qmake CONFIG+=release && make
 
 # Include all clean sub-targets in clean
-clean: clean-appbundle clean-docker clean-dist clean-docs clean-node clean-pip clean-src clean-runtime
+clean: clean-appbundle clean-dist clean-docs clean-node clean-pip clean-src clean-runtime
 	rm -rf web/pgadmin/static/js/generated/*
 	rm -rf web/pgadmin/static/js/generated/.cache
 	rm -rf web/pgadmin/static/css/generated/*
@@ -90,9 +90,6 @@ clean-runtime:
 clean-appbundle:
 	rm -rf mac-build/
 
-clean-docker:
-	rm -rf docker-build/
-
 clean-dist:
 	rm -rf dist/
 
@@ -109,7 +106,7 @@ clean-src:
 	rm -rf src-build/
 
 docker:
-	./pkg/docker/build.sh
+	docker build `python -c 'import web.config as c; print("-t {0} -t {0}:latest -t {0}:{1} -t {0}:{1}.{2}".format(c.APP_NAME.replace(" ", "").lower(), c.APP_RELEASE, c.APP_REVISION))'` .
 
 docs:
 	LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 $(MAKE) -C docs/en_US -f Makefile.sphinx html
diff --git a/pkg/docker/.dockerignore b/pkg/docker/.dockerignore
deleted file mode 100644
index 1b7ed264..00000000
--- a/pkg/docker/.dockerignore
+++ /dev/null
@@ -1,2 +0,0 @@
-pgadmin4/web/**/tests/
-pgadmin4/web/regression/
diff --git a/pkg/docker/Dockerfile b/pkg/docker/Dockerfile
deleted file mode 100644
index 2bf82a61..00000000
--- a/pkg/docker/Dockerfile
+++ /dev/null
@@ -1,98 +0,0 @@
-########################################################################
-#
-# pgAdmin 4 - PostgreSQL Tools
-#
-# Copyright (C) 2013 - 2018, The pgAdmin Development Team
-# This software is released under the PostgreSQL Licence
-#
-#########################################################################
-
-# First of all, build frontend with NodeJS in a separate builder container
-# Node-8 is supported by all needed C++ packages
-FROM node:8 AS node-builder
-
-COPY ./pgadmin4/web/ /pgadmin4/web/
-WORKDIR /pgadmin4/web
-
-RUN yarn install --cache-folder ./ycache --verbose && \
-    yarn run bundle && \
-    rm -rf ./ycache ./pgadmin/static/js/generated/.cache
-
-# Build Sphinx documentation in separate container
-FROM python:3.7-alpine3.9 as docs-builder
-
-# Install only dependencies absolutely required for documentation building
-RUN apk add --no-cache make build-base openssl-dev libffi-dev && \
-    pip install --no-cache-dir sphinx flask_security flask_paranoid python-dateutil flask_sqlalchemy flask_gravatar flask_migrate simplejson cryptography
-
-COPY ./pgadmin4/ /pgadmin4
-
-# Build the docs
-RUN LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 make -C /pgadmin4/docs/en_US -f Makefile.sphinx html
-
-# Get the PG binaries
-FROM postgres:9.4-alpine as pg94-builder
-FROM postgres:9.5-alpine as pg95-builder
-FROM postgres:9.6-alpine as pg96-builder
-FROM postgres:10-alpine as pg10-builder
-FROM postgres:11-alpine as pg11-builder
-
-# Then install backend, copy static files and set up entrypoint
-FROM python:3.7-alpine3.9
-
-# Copy the PG binaries
-COPY --from=pg94-builder /usr/local/bin/pg_dump /usr/local/pgsql-9.4/
-COPY --from=pg94-builder /usr/local/bin/pg_dumpall /usr/local/pgsql-9.4/
-COPY --from=pg94-builder /usr/local/bin/pg_restore /usr/local/pgsql-9.4/
-COPY --from=pg94-builder /usr/local/bin/psql /usr/local/pgsql-9.4/
-
-COPY --from=pg95-builder /usr/local/bin/pg_dump /usr/local/pgsql-9.5/
-COPY --from=pg95-builder /usr/local/bin/pg_dumpall /usr/local/pgsql-9.5/
-COPY --from=pg95-builder /usr/local/bin/pg_restore /usr/local/pgsql-9.5/
-COPY --from=pg95-builder /usr/local/bin/psql /usr/local/pgsql-9.5/
-
-COPY --from=pg96-builder /usr/local/bin/pg_dump /usr/local/pgsql-9.6/
-COPY --from=pg96-builder /usr/local/bin/pg_dumpall /usr/local/pgsql-9.6/
-COPY --from=pg96-builder /usr/local/bin/pg_restore /usr/local/pgsql-9.6/
-COPY --from=pg96-builder /usr/local/bin/psql /usr/local/pgsql-9.6/
-
-COPY --from=pg10-builder /usr/local/bin/pg_dump /usr/local/pgsql-10/
-COPY --from=pg10-builder /usr/local/bin/pg_dumpall /usr/local/pgsql-10/
-COPY --from=pg10-builder /usr/local/bin/pg_restore /usr/local/pgsql-10/
-COPY --from=pg10-builder /usr/local/bin/psql /usr/local/pgsql-10/
-
-COPY --from=pg11-builder /usr/local/bin/pg_dump /usr/local/pgsql-11/
-COPY --from=pg11-builder /usr/local/bin/pg_dumpall /usr/local/pgsql-11/
-COPY --from=pg11-builder /usr/local/bin/pg_restore /usr/local/pgsql-11/
-COPY --from=pg11-builder /usr/local/bin/psql /usr/local/pgsql-11/
-
-WORKDIR /pgadmin4
-ENV PYTHONPATH=/pgadmin4
-
-#Copy in the docs and JS/CSS bundles
-COPY --from=node-builder /pgadmin4/web/pgadmin/static/js/generated/ /pgadmin4/pgadmin/static/js/generated/
-COPY --from=docs-builder /pgadmin4/docs/en_US/_build/html/ /pgadmin4/docs/
-
-# Install build-dependencies, build & install C extensions and purge deps in one RUN step
-# so that deps do not increase the size of resulting image by remaining in layers
-COPY ./pgadmin4/requirements.txt /pgadmin4
-RUN apk add --no-cache --virtual build-deps build-base postgresql-dev libffi-dev linux-headers && \
-    apk add postfix postgresql-client postgresql-libs && \
-    pip install --upgrade pip && \
-    pip install --no-cache-dir -r requirements.txt && \
-    pip install --no-cache-dir gunicorn && \
-    apk del --no-cache build-deps
-
-# Copy the code
-COPY ./pgadmin4/web /pgadmin4
-COPY ./run_pgadmin.py /pgadmin4
-COPY ./entrypoint.sh /entrypoint.sh
-
-# Precompile and optimize python code to save time and space on startup
-RUN python -O -m compileall /pgadmin4
-
-# Finish up
-VOLUME /var/lib/pgadmin
-EXPOSE 80 443
-
-ENTRYPOINT ["/entrypoint.sh"]
diff --git a/pkg/docker/README b/pkg/docker/README
index accbc48b..28a9ff90 100644
--- a/pkg/docker/README
+++ b/pkg/docker/README
@@ -1,129 +1,24 @@
-This directory contains the files required to create a docker container running pgAdmin.
+This directory contains the files required to create a docker container running
+pgAdmin.
 
 Building
 ========
 
-Whilst you can just use the Dockerfile directly, it requires that various pre-configuration steps are performed, for
-example, the pgAdmin web code must be copied to `./web`, Sphinx documentation source must be copied to `./docs`
-and `requirements.txt` is also expected to be in this directory.
+From the top level directory of the pgAdmin source tree, simply run:
 
-The recommended (and easy) way to build the container is to do:
+docker build .
 
-```console
-cd $PGADMIN_SRC/
-workon pgadmin-venv
-make docker
-```
+You can also run 'make docker', which will call 'docker build .' but also tag
+the image like:
 
-This will call the build script $PGADMIN_SRC/pkg/docker/build.sh which will prepare a staging directory containing all
-the required files, then build the container and push it to your repo.
+pgadmin4 pgadmin4:latest pgadmin4:4 pgadmin4:4.12
+
+WARNING: The build should be run in a CLEAN source tree. Whilst some potentially
+         dangerous files such as config_local.py or log files will be explicitly
+         excluded from the final image, other files will not be.
 
 Running
 =======
 
-Environment Variables
----------------------
-
-The container will accept the following variables at startup:
-
-*PGADMIN_DEFAULT_EMAIL*
-
-This is the email address used when setting up the initial administrator account
-to login to pgAdmin. This variable is required and must be set at launch time.
-
-*PGADMIN_DEFAULT_PASSWORD*
-
-This is the password used when setting up the initial administrator account to
-login to pgAdmin. This variable is required and must be set at launch time.
-
-*PGADMIN_ENABLE_TLS*
-
-Default: <null>
-
-If left un-set, the container will listen on port 80 for connections in plain
-text. If set to any value, the container will listen on port 443 for TLS
-connections.
-
-When TLS is enabled, a certificate and key must be provided. Typically these
-should be stored on the host file system and mounted from the container. The
-expected paths are /certs/server.crt and /certs/server.key
-
-*PGADMIN_LISTEN_PORT*
-
-Default: 80 or 443 (if TLS is enabled)
-
-Allows the port that the server listens on to be set to a specific value rather
-than using the default.
-
-*GUNICORN_THREADS*
-
-Default: 25
-
-Adjust the number of threads the Gunicorn server uses to handle incoming
-requests. This should typically be left as-is, except in highly loaded systems
-where it may be increased.
-
-Mapped Files and Directories
-----------------------------
-
-The following files or directories can be mapped from the container onto the
-host machine to allow configuration to be customised and shared between
-instances:
-
-*/var/lib/pgadmin*
-
-This is the working directory in which pgAdmin stores session data, user files,
-configuration files, and it's configuration database. Mapping this directory
-onto the host machine gives you an easy way to maintain configuration between
-
-invocations of the container.
-
-*/pgadmin4/config_local.py*
-
-This file can be used to override configuration settings in pgAdmin. Settings
-found in config.py can be overridden with deployment specific values if
-required.
-
-*/pgadmin4/servers.json*
-
-If this file is mapped, server definitions found in it will be loaded at launch
-time. This allows connection information to be pre-loaded into the instance of
-pgAdmin in the container.
-
-*/certs/server.cert*
-
-If TLS is enabled, this file will be used as the servers TLS certificate.
-
-*/certs/server.key*
-
-If TLS is enabled, this file will be used as the key file for the servers TLS
-certificate.
-
-Examples
---------
-
-Run a simple container over port 80:
-
-```
-    docker pull dpage/pgadmin4
-    docker run -p 80:80 \
-        -e "[email protected]" \
-        -e "PGADMIN_DEFAULT_PASSWORD=SuperSecret" \
-        -d dpage/pgadmin4
-```
-Run a TLS secured container using a shared config/storage directory in
-/private/var/lib/pgadmin on the host, and servers pre-loaded from
-/tmp/servers.json on the host:
-
-```
-    docker pull dpage/pgadmin4
-    docker run -p 443:443 \
-        -v "/private/var/lib/pgadmin:/var/lib/pgadmin" \
-        -v "/path/to/certificate.cert:/certs/server.cert" \
-        -v "/path/to/certificate.key:/certs/server.key" \
-        -v "/tmp/servers.json:/servers.json" \
-        -e "[email protected]" \
-        -e "PGADMIN_DEFAULT_PASSWORD=SuperSecret" \
-        -e "PGADMIN_ENABLE_TLS=True" \
-        -d dpage/pgadmin4
-```
+See the documentation at docs/en_US/container_deployment.rst for information on
+running the container.
diff --git a/pkg/docker/build.sh b/pkg/docker/build.sh
deleted file mode 100755
index 5cd07c97..00000000
--- a/pkg/docker/build.sh
+++ /dev/null
@@ -1,61 +0,0 @@
-#!/bin/bash
-
-########################################################################
-#
-# pgAdmin 4 - PostgreSQL Tools
-#
-# Copyright (C) 2013 - 2019, The pgAdmin Development Team
-# This software is released under the PostgreSQL Licence
-#
-#########################################################################
-
-# Runtime checks
-if [ ! -d runtime -a ! -d web ]; then
-    echo This script must be run from the top-level directory of the source tree.
-    exit 1
-fi
-
-if [ ! -d .git -a ! -f .git/config ]; then
-    echo This script must be run from a git checkout of the source tree.
-    exit 1
-fi
-
-# Get the required package info
-APP_RELEASE=`grep "^APP_RELEASE" web/config.py | cut -d"=" -f2 | sed 's/ //g'`
-APP_REVISION=`grep "^APP_REVISION" web/config.py | cut -d"=" -f2 | sed 's/ //g'`
-APP_NAME=`grep "^APP_NAME" web/config.py | cut -d"=" -f2 | sed "s/'//g" | sed 's/^ //'`
-APP_LONG_VERSION=${APP_RELEASE}.${APP_REVISION}
-APP_SHORT_VERSION=`echo ${APP_LONG_VERSION} | cut -d . -f1,2`
-APP_SUFFIX=`grep "^APP_SUFFIX" web/config.py | cut -d"=" -f2 | sed 's/ //g' | sed "s/'//g"`
-if [ ! -z ${APP_SUFFIX} ]; then
-    export APP_LONG_VERSION=${APP_LONG_VERSION}-${APP_SUFFIX}
-fi
-CONTAINER_NAME=`echo ${APP_NAME} | sed 's/ //g' | awk '{print tolower($0)}'`
-
-# Output basic details to show we're working
-echo Building Docker Container for ${APP_NAME} version ${APP_LONG_VERSION}...
-
-# Create/clearout the build directory
-echo Creating/cleaning required directories...
-if [ -d docker-build ]; then
-    rm -rf docker-build
-fi
-
-mkdir -p docker-build/pgadmin4
-
-# Build the clean tree
-echo Copying source tree...
-git archive HEAD -- docs web requirements.txt | tar xvf - -C docker-build/pgadmin4
-
-# Copy the Docker specific assets into place
-cp pkg/docker/Dockerfile \
-    pkg/docker/entrypoint.sh \
-    pkg/docker/run_pgadmin.py \
-    pkg/docker/.dockerignore \
-    docker-build/
-
-# Build the container
-docker build docker-build -t ${CONTAINER_NAME} \
-                          -t ${CONTAINER_NAME}:latest \
-                          -t ${CONTAINER_NAME}:${APP_RELEASE} \
-                          -t ${CONTAINER_NAME}:${APP_LONG_VERSION}
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.