What we learned #

Migrating from one ingress controller to another is not a trivial task!

Context #

The news that the best-effort maintenance of the Ingress NGINX Controller would end in the first quarter of 2026 sparked a considerable amount of discussion among the community about what the best replacement for this controller is.

Because “what the best replacement is” depends on the specifics and maturity levels of the infrastructure stack, for the near future, 2i2c has decided to migrate to the official NGINX Ingress Controller. However, this choice will have to be challenged in the future, because the Kubernetes Ingress API itself was frozen when the INGRESS NGINX Controller was sunset, with the Kubernetes project recommending the Gateway API instead. So, we may need/want to migrate to the Gateway API down the road.

Migration #

Our migration plan has all the technical details about the steps we took towards the actual controller migration, so this blog post will not go into those aspects. Instead, it will describe the approach we took to achieve the least amount of downtime for our communities, as well as list the challenges that we couldn’t anticipate during the planning phase.

Least amount of downtime #

In our initial setup, we were using the LoadBalancer service provided by the ingress controller. This meant that migrating from one controller to another, meant updating all DNS records to point to the new controller’s LB. During this time, the cluster infrastructure would have been unavailable to users.

So, in order to minimize the downtime, we have introduced a new LoadBalancer ingress service, independent of the ingress controller. This service provides a static external IP address (the entrypoint into each cluster) that points to the ingress controller ClusterIP service. This allows us to simply point this LB at a new service pod whenever we need to switch to the new controller/gateway.

Challenges #

The tracking issue on our infrastructure repo tracks almost all the tasks (both expected and unexpected) related to the migration. But here is a summary of the ones that took us by surprise:

1. On one hub, the reverse proxy was unable to communicate with binderhub-service. This resulted in 502 errors. This also pointed us to identify a websocket failure between the frontend and the user server due to missing ingress annotations. The problem and solution are described in the incident report.

2. Manually updating all DNS records is, of course, error-prone. So, we ended up missing one. The full incident report is available to be checked-out.

3. The F5 NGINX Ingress controller came with an unexpected behaviour. When certificates needed to be renewed, it was trying to create (new) temporary ingress objects. The new controller does not support multiple ingresses naively sharing the same host. So, we had to update all ingress objects needing TLS, to allow cert-manager to mutate the ingress via edit-in-place. More about it in this incident report.

4. Renewing certificates was still failing

• Even though ingress objects were updated to allow cert-manager to edit-in-place, this did not trigger the update of the certificate itself, which meant that nginx-ingress still tried to create temporary ingresses. Manually reconciling the Certificate resource with the Ingress fixed this. More in the incident report and this discussion.
• Our Prometheus instances were being authenticated at the ingress controller level. This made the ACME challenge resolution during renewal encounter HTTP 401 responses. So we had to move the authentication from the ingress controller level to the Prometheus service itself. Although a beneficial change, this generated more work and more integrations to fix 2i2c-org/infrastructure#8123.

Acknowledgements #

• To MinRK for adding this feature to Dask-Gateway and Yuvi for brainstorming the best migration approach
• To Angus for helping with the migration and troubleshooting the issues