This month Proximity Placement Groups have been announced as a public preview offer and this post is here to tell you about them.
For a long time, we’ve been using availability sets to bring our applications as close as possible to ensure the lowest latency possible however this scenario couldn’t always be achieved because the cloud is shared with multiple customers and you’re deploying resources in tandem with other people. This means that if you’re looking to run a latency-sensitive application in Azure then Availability Sets or Availability Zones are not always the answer.
I’ve been holding off this post for a while now to gather more information, to figure out how things can be done better and so on. This post is a culmination of experiences I’ve had with Service Fabric, how I solved them and hopefully solve your issues before you have a disaster on your hands.
Starting from the beginning. What is Service Fabric?
Service Fabric is a distributed systems platform that makes it easy to package, deploy, and manage scalable and reliable microservices and containers. Service Fabric also addresses the significant challenges in developing and managing cloud native applications. Developers and administrators can avoid complex infrastructure problems and focus on implementing mission-critical, demanding workloads that are scalable, reliable, and manageable. Service Fabric represents the next-generation platform for building and managing these enterprise-class, tier-1, cloud-scale applications running in containers.
Source: Azure docs
That being said, I have a small list of recommendations that should be enforced in practice so that you don’t repeat the mistakes I had to repair.
Network troubleshooting in the cloud was always a pain. Let’s talk about the Azure Network Watcher and what it can do for you.
Running workloads in the cloud can be very easy but when it comes to troubleshooting something you don’t have access to can prove to be quite a challenge.
As you may know, you’re not dealing with you’re regular on-premises network stack that you’re used to but you’re dealing with software-defined networking or SDN for short. This means that everything is virtualized and you don’t need to managed switches, routers or any other type of networking equipment. In the cloud, you manage virtual networks, subnets, IPs, VPN devices, network rules and so on but on a software level.
While everything is nice and fun with SDN, you will encounter in Azure most of the networking problems that you encounter on-premises. Emphasis on most. You will not deal with hardware problems, VLANs, STP and so on but you will deal with firewall rules, routes, priorities, wrong topologies.
Issues that you might encounter in Azure networking:
Loss of network connection
VM cannot connect to a service
VPN Gateway is not connecting to the on-premises server
VMs across VNET Peering cannot connect
Everything is wrong; Nothing works 🙂
Sounds familiar? It’s mostly the same as on-premises but you’re dealing with a different technology stack.
What are my options?
In Azure, there’s a nice piece of free technology called Network Watcher which allows you to do network debugging and figure out most of the time, where the network problem is. When I say most of the time, I’m saying that there are those cases where you do everything in your power and still cannot figure out where’s the issue.
Enabling Network Watcher:
This is the easy part; Go to All Services in Azure and type in Network Watcher then in the overview blade select the region where it should be enabled.
Network Watcher capabilities
With Network Watcher you have the following capabilities:
Network Performance Monitor
IP Flow verification
Next Hop validation
Effective NSG rules
Network topology gives you a network diagram of actual Virtual network in scope. You select the subscription, RG and Virtual Network where you require the diagram and you get a network topology of what’s connected and how it’s connected.
This allows you to visually map how you’re network is deployed in Azure which means that you and obviously it helps when somebody requests a very detailed network diagram.
Connection monitor allows you to set up a continuous endpoint monitor which gives you metrics about the connections over a period of time.
To set up a connection monitor, you need to press on Add and then specify what you need to monitor.
If you’re selecting an Azure VM as the source then the AzureNetworkWatcheExtesion will be installed to that VM. One thing to watch out when you’re specifying an Azure VM in the source pane is that you will be able only to select Azure VMs that are part of the same VNET. Not a peered network or anything else. The workaround to this problem would be to just specify an IP address and that’s it.
Once you start the connection monitor, you will get prompted with some historical data in graph and data from connection metrics and status. In the example from above, you can see that I set up a connection monitor from the Azure VM to 220.127.116.11 and you can see that the VM can communicate with Google DNS and the return time is 2 MS (1MS round-trip). If I would have had a problem I would have known when the problem happened and validate in the Azure Monitor logs what happened.
IP Flow Verify
IP Flow Verify lets you validate the configuration of your Network Security Group rules; It requires you to input five packet details (Protocol, Direction, SourceIP, SourcePort, DestinationIP and DestinationPort)
Once provided, Network Watcher|IP Flow Verify will do an NSG check to validate if the connection succeeds or fails. This basically validates your NSG configuration and not your VM firewall. If it succeeds or fails it’s going to tell you which rule was hit.
Next Hop is one of the simplest tools available in Network Watcher; It’s basically a tool that tells you which where the packet will go. It also gives you which route table is affecting the packet route.
Some examples of Next Hop are: VNET Peer, Internet, VNET or Network Appliance (NGFW)
Packet Capture or Network Capture is the be all end all of the network troubleshooting. This tool allows you to very detailed captures of what’s happening on a target VM.
Before diving into this tool, it’s recommended to have a storage account ready for the captures because it’s much fast to get and distribute the data that’s being monitored. Otherwise, you have the option of saving the file locally on the VM but you’re not so flexible at that point.
When you’re specifying the details for the packet capture, you have the possibility of filtering out irrelevant traffic as shown in the picture above.
What about the others?
This article wants to show getting the basics of network monitoring in Azure. There’s Network Performance Monitor which is a whole beast itself and requires a hefty deployment to show off the possibilities. VPN Troubleshoot which gives you a log of what’s happening so you can debug VPN settings (on-premises ones mostly) and the logging part which will be another article as the scope is different there.
That being said. Thanks for reading and have a good one!
Security is not something to kid about and when it comes to cloud, you have to be very through when you’re deploying your cloud infrastructure. Which means that you are still required to do defense in depth, use anti-malware systems, configure extended monitoring, logging and reporting mechanisms. When you’re going to the cloud, you have to be aware of the Shared Responsibility Matrix which applies to any cloud provider.
As you can see in the image above, you as a customer still have a responsibility to secure your cloud environment. So those skills you’ve developed while working on-premises will still be of value in the cloud.
The subject for today’s topic is managing Network Security Groups using a feature in Azure, called Application Security Groups.
What are they?
Application Security Groups are a mechanism to group virtual machines that reside in the same virtual network and apply Network Security Groups to them.
The way you deployed NSGs in Azure subscription was that you would assign them to a network interface or a subnet and then configure them in a granular manner, based on the deployment type. The reality was that it’s utopic to do this cleanly and it gets messy after a couple of months. So ASGs came to the rescue where it helped you group a set of VMs based on roles like Web, DB, Middleware etc. and apply NSG Allow / Deny rules on them.
By using an ASG, you simply your management overhead by just adding the VMs that you create in those groups and automatically you get the security policies applied from your NSG.
Creating / using Application Security Groups is easy. Go to the Azure Portal -> Create a resource -> Type in Application Security Group and press create.
Or you can simply use Powershell
#PS Example for creating Application Security Groups.
$testAsg = New-AzureRmApplicationSecurityGroup -ResourceGroupName asgTest -Name testAsg -Location westeurope
After you’ve created the ASG, the next thing that you need to do is to assign it to some VMs, which can be done via the Portal or PS.
#PS Example for attaching ASGs
$Nic = Get-AzureRmNetworkInterface -Name test134 -ResourceGroupName asgtest
$Nic.IpConfigurations.ApplicationSecurityGroups = $testAsg
Set-AzureRmNetworkInterface -NetworkInterface $Nic
Next step is to add or modify your inbound / outbound rules to use those new ASGs you’ve created. Doing that is very simple and you can also do it via the portal or CLI.
As you can see, it’s pretty easy to secure your VMs, and considering that it can become a pain to manage the NSGs even for simple deployments. I’m not even talking about very complex ARM deployments which deploy tens of VMs and link them together 🙂
Did this problem ever happen to you? If yes, then you know that the way to solving this issue is by booting the distro into the Single User mode. But how do you do that in Azure? Well Serial Console to the rescue!
Usually this is easily solvable using the Run Command or by using the Reset Password blade but in this case imagine that they don’t work. This is the case of the SAP deployment using the RHEL VMs. You cannot do anything if you’ve lost access and if the VM crashes it’s even worse.
You need to get to grub so you can boot the VM in single user mode. The problem here is that the VM is very fast for the serial console to connect and press the ESC button in the magic moment.
So what can you do?
The solution to that problem is to stop the VM without de-allocating it. This means that the VM on the Hyper-V server in the backend is not deleted but preserved. This means that you can have the serial console in standby to have a chance at that magic moment. How do you know that? Check figs 1 and 2.
#stop the VM without de-allocation.
Stop-AzureRMVM -Name $VMNAME -ResourceGroup $RESOURCEGROUP -StayProvisioned -Verbose
Once you’ve gotten to the screens that the VM is starting, this is what you need to watch for and then mash the ESC button:
Once you’ve managed to enter GRUB, you’re home free to reset the password using the steps below Press e in the Serial Console to edit the first OS line.
Go to the kernel line which starts with linux16
Add rd.break to the end of the line which will break the boot cycle. If selinux is enabled then add rd.break enforcing=0
Exit GRUB and reboot with the rd.break command saved by pressing ctrl x
During this reboot, the VM will go into the Emergency Mode where you have to mount the systemroot using the “mount -o remount,rw /sysroot” command.
This will boot you in single user mode, where you will have to type in chroot /sysroot to switch into the sysroot jail and then reset the password for the root user with passwd
Edit the sshd_config file “nano /etc/ssh/sshd_config” using your preferred editor so you enable root access using the Serial Console by setting PermitRootLogin yes
Once you’re done, reboot the VM and you’ve gotten root access.
After you’re done resetting all the passwords, installing all the agents so you’re not confronted with this again, set PermitRootLogin no and you’re golden 🙂