[Hosting HPC in Azure] 2.4 -- Login Node

How to Change the Subnet where Login Nodes are?

The cluster should be terminated.

CycleCloud Wizard -> Arrays -> Click login row and Edit.

Edit the Subnet ID.

How to Connect To and Set Up Login Node?

CycleCloud launches login nodes as virtual machine scale set (VMSS). We cannot just assign a Network Security Group and associate a public IP to it. We need to use load balancer to connect to individual VMs and NAT gateway for outbound internet connections for VMSS.

Bind VMSS to Load Balancer Public IP

1. Azure Portal -> Virtual Machine -> Virtual Machine Scale Set -> Load Balancing -> Create

   Form	How to
   Type	Public
   Protocol	TCP (since we are using SSH)
   Frontend port	number (ssh -p number user@host)
   Backend port	22 (for SSH)

2. Virtual Machine Scale Set -> Load Balancing -> Load Balancing Rules
   • Click Details, the state should be up.

   Make sure it is using TCP, and the backend port and health probe points to the same port. Need to make sure health probe can work, so that load balancer know where to distribute the work load. If it didn’t work, then all of the network are block.

3. Load Balancing -> Backend pools, if you see Azure says go upgrade the backend pool, then upgrade the VMSS.

4. Add Network Security Group (NSG) to both the subnet and VMSS. The NSG should have port 22 for SSH, which is the backend port we set in previous step.

   • Virtual Machine Scale Set -> Netwwork Settings -> Create and Create inbound rule
   • Mind the NSG name, then go to Virtual Network -> Subnet -> Network Security Group drop down box

By now, we should be able to log in to log in node:

ssh -p <frontend_port> user@<public_ip_of_load_balancer>

But we don’t have internet access yet:

curl -I https://www.google.com # print nothing

NAT Gateway for Outbound Internet

1. Azure Portal -> NAT Gateways -> Create

   • Outbound IP: Add public IP address or prefixes
   • Networking: select the VNet and subnet where the login nodes are at.

On login node:

curl -I https://www.google.com # print the page

If we allocate a compute node and test the internet access, the compute node can also access the internet, which is bad for security reasons.

I have no idea why compute nodes that are in different subnet can also access to the internet. To block the outbound internet access for the compute nodes, I have to create a route table to drop all the outbound packages. See Compute Node.

FAQs

No discoverable nodes found while starting the cluster in CycleCloud Wizard

• Description: No discoverable nodes found
• Details:

    File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/converge/__init__.py", line 429, in execute
        chef_enabled = _update_config(goal_state, runlist, slurm_role)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/converge/__init__.py", line 289, in _update_config
        jetpack.util.mounts.update_addresses(config['cyclecloud'].get('mounts', {}))
    File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/util/mounts.py", line 7, in update_addresses
        mount['address'] = jetpack.util.get_cshared_ip()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/util/__init__.py", line 777, in get_cshared_ip
        raise Exception("No discoverable nodes found")
  

I think this is because login node is searching or waiting for scheduler node. After scheduler successfully launched, the error disappear.