How do you handle JDK/JRE patch updates for Java apps on K8s?
Background: in Mesos (https://en.wikipedia.org/wiki/Apache_Mesos) times, we used to keep the JDK on the runner nodes. When a CVE or patch came out, we updated the host JDK, and all apps picked it up. That was convenient for fast security rollouts. On k8s, almost everyone I see bakes the JDK into the container image, which means: new JDK → rebuild base image → rebuild app images (or at least rebuild base) → push → roll out. That is reliable and reproducible, but it is impossible to update the JDK version for, for example, 2000 apps quickly.
Questions I have for people who run Java on k8s at scale:
Do you rebuild images for every JDK patch?
If so, how do you keep the pipeline fast/automated?
What approaches we talked about (still looking for something better):
- Rebuild images on every JDK patch (CI pipeline that automatically bumps base image + rebuilds): reproducible but heavy and slow.
- Host-provided JDK (like Mesos) via hostPath or a shared volume (every path version must be available): fast patches, but brittle (node drift, version chaos between k8s nodes, less reproducible, potential security/permission problems).
- Base, standard image for all java apps (alpine+java) that our platform updates and init container downloading user app on startup, so that we can update it in the background.
- Sidecar or init-container that places a JDK into a shared volume, and the app container uses that volume: mutable runtime without rebuilding images — how well does this work in practice?
I think option/bullet 2 keeps things as streamlined as possible - base standard image for java apps that apps then use as their base when they build their container images to run in k8s containers. Maybe there's a way to tag base images with major.minor so that apps just pull that image and if its cached its faster and if there was a major.minor.patch just applied and you're the first app to deploy then that app pays the penalty of waiting for the underlying base image to be rebuilt.
A volume is an interesting idea but are there possible issues for OS library dependencies like openssl, database drivers, etc. that might not be the right version for a JDK mounted on a shared volume?
The new images are then rolled out in exactly the same controlled manner as any other new release.
Also regarding Quarkus, that's java stuff. I don't use it, but I know people do use it for shortcomings of Java in k8s. Perhaps GraalVM is the thing that makes it worth-while, idk.
There's also a distroless Java container from Google that could be of value.