• NOMAD CoE

Exposing services to the internet

To access the different services one has to expose them to internet users. From the outset, we decided to use a single hostname for them, and distinguish them using path prefixes.

To do this something like a reverse proxy (a server that forwards requests to other servers) is needed. We decided to use Nginx for this purpose. Nginx is an efficient web browser (using an event approach like the apache event MPM) and is widely used as a reverse proxy. Using it, we can leverage a large community using it in similar ways.

The configuration, and common static files live in the frontend-nginx git repository.

Services

To run the services we mostly use Kubernetes, see nomad/Deploy for how to start them. As described in nomad/Kubernetes every service object has a fixed name. Inside the namespace the service can be reached simply through its name at its port. A service with a node port is available on all nodes of the cluster under the same port. This simplifies connecting to it from outside the cluster, and avoids the bottleneck of having a single entry point to reach the service, as the number of nodes scales with the cluster. Still outside the kubernetes cluster, without using the cluster DNS, the host on which the service will be reachable will be installation specific. Furthermore, initial normal installations of kubernetes with network interfaces (CNI) that did not need special hardware were not able to fix the node port to be used, only randomly chosen node ports were supported. Finally, for technical or historical reasons, some services run in other kubernetes clusters or outside Kubernetes.

Templates

Hardcoding the ports and hostnames in the Nginx configuration (which we initially did) is not really sustainable. This forced us to choose a more flexible solution, with handlebars templates that has placeholders for this information. The nginx.conf.in file contains the configuration template. It checks if there is information about each service, and if present adds the configuration for it. The configurations of the main production and development configuration are also kept in git to be able to quickly see the changes, and restore previous configurations.

Service info files

Service info files are json files consisting in a dictionary with the service names as key, and a list of service infos as value.

Each service info is an object that can have various attributes. The main ones are:

• nodes: a list of the hostnames of the nodes that can be used to reach the service
• ports: a list of port objects, the attribute nodePort if given contains the port to use on the nodes to reach the service

For Kubernetes the service info files can be generated by the serviceDumper function of the container-manager (see nomad/ContainerManager for a description of the container manager itself). The deploy.sh script generates yaml files also to create a service-info file in a standard place.

Multiple service info files can be merged (that is the reason the value corresponding to a service is a list). This is a very useful feature: services from multiple Kubernetes clusters, and even some manual setup can live in separate files, and be updated independently, while the final configuration still depends on all of them.

The templates take the information about the services from one or more service info files, and generates a configuration file. See the Readme

Scaling

Supercomputing centers do expose many servers to the internet, and it is difficult to get them to expose computing nodes (understandably due to security concerns). Also a single server has quite a bit of bandwidth. Until now the frontend wasn’t the bottleneck, still if that should become an issue, the simplest way out is to have multiple frontends that expose different nodes (currently we always use the first).

Alternatives

Recently kubernetes introduced ingresses. These try to address the same issue we have (exposing services), but only within kubernetes. This is a relatively new solution (thus still changing), but works well if some nodes are exposed to the internet. As ingress controller one can use an Nginx based solution or other options like traefik (see also below). An ingress can be smarter in keeping a user talking to the same endpoint of the service, but if the host exposed is not part of the cluster the setup is more cumbersome, and if other services also need to be exposed one needs an extra reverse proxy anyhow, negating many of the advantages, while keeping the setup burden.

traefik is a new solution to the routing problem that tries to address many of the shortcoming of other solutions. It can also be an ingress controller. It is relatively new, so it might change, but seems an interesting option for the future.

For our purposes the more standard nginx solution is well tested, robust and covers our current needs, so we stayed with it.

Conclusions

We use an Nginx based solution to expose our services. Its configuration uses handlebars templates, and the information about the services can come from multiple files generated automatically from the kubernetes information.

It enables a quick switch (automatizable, but currently still manual for security reasons) between different configurations, enabling for example the switch between different installations with minimal downtime (but a connected user might still notice it).

All the code for it is available in the frontend-nginx git repository

• Nginx