Running Multiple NGINX Ingress Controllers in AKS

In some of the previous articles of this blog, we went through the process of installing NGINX as the Ingress Controller in AKS clusters. Either for applications that should be available directly from the public internet, or for applications that should only be accessed from networks considered internal.

In the majority of the cases, however, and given that an AKS cluster is an environment that is designed to host multiple applications and provide economy at scale, additional ingress controllers may be required.

In this post, we're going to go through the process of deploying an additional NGINX ingress controller that is going to be used for internal applications. The below diagram depicts the desired outcome:

The Angular application is published via the public NGINX through the Azure Load Balancer that has been assigned a public IP. The .NET app is published by a different set of NGINX pods that are deployed in a different namespace and their ingress controller service is connected to another Azure Load Balancer that uses private IPs.

Let's see it in action! As always, the first step is to deploy the necessary resources on Azure. Clone the repository and use the deploy.sh script to start the deployment. In case you need to adjust the names to your environment, edit the script or the main.bicep file.

When the deployment finishes, there should be four new resource groups in your subscription:

The networking group contains all the networking-related resources like the vNet and the bastion host, the VM contains a virtual machine to perform tests and manage the cluster, and the AKS and AKS-MC, the AKS resource and its managed cluster.

In this example, the second NGINX will be deployed in a private manner, in order to publish applications to internal networks (see this post). This requires AKS to be granted permission on the vNet, easily done using the vNetAccess.sh script. 

Get the credentials to the cluster using the getAKSCredentials.sh script and you're good to start with the NGINX deployments!

We're going to deploy the NGINX ingress controllers the usual way, that is using the default Helm chart. There are going to be some additional parameters however since we need to configure the names of some classes and the ingress controller, to avoid conflicts.

To deploy the first ingress controller, switch to the 101-K8s-NGINX-Public-001 folder, apply the 1-namespace.yaml file, and run the 2-helm.sh script. The deployment of the first controller is now complete.

The second controller is deployed following the same steps, but using the files from the 111-K8s-NGINX-Internal-001 folder. The only differences between the files are the names of the namespaces and the additional parameters to the Helm chart, as shown below: 

The parameters marked in the red rectangle are the ones that essentially differentiate the second NGINX from the first one. We are using a different ingress class name and a different name for the controller.

If you get the ingress classes available in the cluster after the deployment of the second controller, you should get something similar to the below:

As you can see, we have two different classes implemented by two different controllers.

Now that we have completed the deployment of the two NGINX ingress controllers, it's time to deploy a couple of applications and publish them using ingress resources. 

The 201-K8s-App-Public-001 and 211-K8s-App-Internal-001 folders contain the necessary files to deploy two applications, one that is going to be using the public-facing NGINX, and one for the private NGINX. The application that is deployed is essentially the same, the only differences are the namespace and the class of the ingress to use.

For the public application, the ingress controller is declared to be of type nginx, which is the default ingress type that was installed as part of the default installation of the first ingress controller.

This class will result in the ingress being processed by the public NGINX controller.

The ingress class of the second application's ingress resource is set to internal-nginx, thus the ingress resource will be processed by the second NGINX controller. 

If you take a closer look at the Helm scripts parameters for the installation of the second NGINX, you'll notice that internal-nginx is the value for the ingress class resource name!

The result of the above deployments would be two NGINX services, one in each NGINX namespace:

As expected, the service of the default NGINX has been assigned a public IP and the service of the internal NGINX has been assigned an IP from the specific vNet on Azure.

Moving on to the ingress resources - the way that applications are exposed outside the AKS cluster, we have two ingress resources, one for each application:

The configuration of the ingresses seems to be right, given that the names of the classes and IP addresses are configured correctly. If for any reason your second ingress is not getting an IP, make sure you have assigned the permissions to the AKS cluster on the vNet and also allow some time to pass.

To verify that the ingresses work as expected, you may use the management virtual machine and bastion host that are deployed as part of the Bicep template to access each application:

Things are looking good, each application is coming from the respective namespace (the page shows the OS name of the server that is starting with "app" for one app and "app2" for the other).

Well, this is how you deploy multiple NGINX ingress controllers in the same Kubernetes cluster!

The two posts below will help you understand more about the role of NGINX in AKS:

All of the files used in this demo are available on my Github repository over here.

Happy coding!

Popular posts from this blog

Domain Controller Machine Password Reset

Image
On my lab environment, I've configured two Active Directory sites since most enterprises have offices in more that one places. My lab however is not running 24/7 and the domain controllers in the second site are rarely turned on in order to save resources. This leads to issues with the Active Directory replication such as the "The target principal name is incorrect" error when I execute:  repadmin /syncall /AdeP. To remedy the issue, we have to reset the machine password of the domain controller that has been offline. First off, we are going to stop and disable the Kerberos Key Distribution Center (kdc) service on the problematic domain controller, in our case DC4. There may be some tickets in the cache so we should also clear them using klist purge Now it's time to change the machine password of the domain controller using the command netdom resetpwd /s:dc3 /ud:lab\administrator /pd:* Replace the "lab\administrator" with an account on your...
Keep reading

Managing Active Directory User Certificates using PowerShell

I first came across user certificates when I was working with email certificates a few years ago and I have to admit that I had trouble updating the certificates on the objects! Most organizations have a Microsoft Active Directory Certification Authority that issues the certificates used internally. When a certificate is issued to a user, the Microsoft Certificate Service saves the public key in Active Directory. The userCertificate attribute is a multi-valued attribute that contains the DER-encoded X509v3 certificates issued to the user. Although we rarely need to pay attention to this attribute, there are cases where we have to update it. To make things easier, I've written PowerShell functions to Get, Remove, Import and Export the certificates on that field. To get the list of certificates for an object, use the Get-ActiveDirectoryObjectCertificate function: PS C:\> Get-ActiveDirectoryObjectCertificate -UserPrincipalName cpolydorou@lab.local DistinguishedName  ...
Keep reading

Configuring a Certificate on Exchange Receive Connector

Today's article is about configuring Exchange receive connectors with specific certificates. Out of the box, Exchange uses self signed certificates to provide TLS secured mail flow. This will definitely be an issue if you expose the SMTP protocol to client computers since they won't trust the certificate. In this article we are going to configure a certificate that was issued by a third part authority to the Client Frontend receive connector. We'll start with getting the thumbprint of the certificate using the Get-ExchangeCertificate cmdlet: [PS] C:\> Get-ExchangeCertificate Thumbprint                               Subject ----------                               ------- 241B864DC82C664FECBA18B8D54987AAFB65B4C2 CN=*.lab.com, ... D4D210886B34E690191A1F008C78FDD0E7325DD4 CN=Exchange2013A 960171662EB26116...
Keep reading