ARM Template - Creating NGINX Webfarm with Custom Script Extension

A friend of mine recently started working with Azure and loved it once he got the hang of it. I encouraged him to start using PowerShell to automate various Azure operations but it didn’t quite stick with him on the first try. He started automating Azure operations using the Azure CLI and while it’s not a bad tool, it’s quite lacking in features compared to PowerShell and I’m pretty sure that it will not be maintained much longer since Microsoft open sourced PowerShell and gave the Linux / Mac community a taste. The funny part of this story is that he’s a Windows user, uses Windows 10 and yet he’s still using Azure CLI.

Where am I going with this? While giving him some tips on how to deploy some production / staging environments in Azure, I saw how he was automating resource creation in Azure using the CLI. He basically created a golden image in a storage account and with 35 lines of Azure CLI code, he was provisioning the environments. That made me cringe and motivated me to teach him how to do it using ARM templates and custom scripts.

Long story short, I taught him the basics of ARM Templates and to give him a nudge in the right direction, I created an ARM Template that served his needs and to share with the community, I have attached it in this blog post as well ?

In a nutshell, the ARM Template creates a storage account, a load balancer, a public IP, a specific number of CentOS 7.2 Virtual Machines with one data disk per instance and uses a custom bash script that configures the VMs. The template also configures the load balancer to probe ports 80 and 443 and set NAT rules for SSH on each VM instance.

The template has one simple requirement. Your custom script has to be in a storage account.

You can obtain the FileUri using the Azure Portal or Storage Explorer and the CommandToExecute should contain sh filename.sh and additional parameters if required.

The bash script attached in this post does a couple of things:

1. Disables SELINUX
2. Adds the latest EPEL (Extra Packages for Enterprise Linux) for version 7.
3. Installs NGINX, PHP 5.6 and some PHP 5.6 Packages.
4. Installs oAuth
5. Does some required configuration for NGINX to work
6. Configures the data disk
7. Some performance oriented settings for the web server.

One thing to note: Do not try to do a YUM upgrade / update. It will break the process.

Now without further ado, here’s the ARM Template and the bash script.

{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",

  "parameters": {
    "environmentPrefixName": {
      "type": "string",
      "maxLength": 7,
      "metadata": {
        "description": "Prefix of the environment."
      },
      "defaultValue": "easc"
    },
    "vNetResourceGroup": {
      "type": "string",
      "metadata": {
        "description": "Name of resource group with VNET."
      },
      "defaultValue": "VirtualNetworks"
    },
    "vNetName": {
      "type": "string",
      "metadata": {
        "description": "Name of Virtual Network where you want to deploy the VMs."
      },
      "defaultValue": "VMNetwork"
    },

    "vNetSNName": {
      "type": "string",
      "metadata": {
        "description": "Name of Virtual Network Subnet where you want to deploy the VMs."
      },
      "defaultValue": "WebFarm"
    },

    "adminUserName": {
      "type": "string",
      "metadata": {
        "description": "User name for local administrator account."
      },
      "defaultValue": "ecadmin"
    },
    "adminPassword": {
      "type": "securestring",
      "metadata": {
        "description": "Password for local adminstrator account."
      },
      "defaultValue": "PASS@word123"
    },
    "vmWebCount": {
      "type": "int",
      "minValue": 2,
      "maxValue": 9,
      "metadata": {
        "description": "Number of frontend virtual machines behind a loadbalancer."
      },
      "defaultValue": 2
    },
    "fileUris": {
      "type": "string",
      "metadata": {
        "description": "URL to bash script"
      }
    },
    "commandToExecute": {
      "type": "string",
      "metadata": {
        "description": "Command to execute script"
      },
      "defaultValue": "sh install_56.sh"
    },
    "customScriptStorageAccountName": {
      "type": "string",
      "metadata": {
        "description": "Name of the custom script storage account"
      }
    },
    "customScriptStorageAccountKey": {
      "type": "securestring",
      "metadata": {
        "description": "Key for referenced storage account"
      }
    }
  },
  "variables": {
    "computeApiVersion": "2016-03-30",
    "networkApiVersion": "2016-06-01",
    "storageApiVersion": "2016-01-01",

    "imagePublisher": "OpenLogic",
    "imageVersion": "latest",
    "imageSKU": "7.2",
    "imageOffer": "CentOS",

    "publicIPAddressName": "[concat(parameters('environmentPrefixName'), '.LBPUBLICIP')]",
    "publicIPAddressID": "[resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName'))]",
    "lbID": "[resourceId('Microsoft.Network/loadBalancers',variables('lbWebName'))]",
    "location": "[resourceGroup().location]",

    "saVhdName": "[concat(parameters('environmentPrefixName'), uniqueString(resourceGroup().id))]",
    "storageAccountType": "Standard_LRS",
    "vNetID": "[resourceId(parameters('vNetResourceGroup'), 'Microsoft.Network/virtualNetworks', parameters('vNetName'))]",
    "vNetSNRef": "[concat(variables('vNetID'), '/subnets/', parameters('vNetSNName'))]",
    "vmWeb0xComputerName": "[concat(parameters('environmentPrefixName'), 'web0')]",
    "asWebName": "[concat(parameters('environmentPrefixName'), '.ASWEB')]",
    "lbWebName": "[concat(parameters('environmentPrefixName'), '.LBWEB')]",
    "nicWeb0xName": "[concat(parameters('environmentPrefixName'), '.NICWEB0')]",
    "vmWeb0xName": "[concat(parameters('environmentPrefixName'), '.VMWEB0')]",
    "vmWebSize": "Standard_F2"
  },

  "resources": [
    {
      "apiVersion": "[variables('storageApiVersion')]",
      "type": "Microsoft.Storage/storageAccounts",
      "name": "[variables('saVhdName')]",
      "location": "[variables('location')]",
      "sku": {
        "name": "[variables('storageAccountType')]"
      },
      "kind": "Storage",
      "properties": {}
    },
    {
      "apiVersion": "[variables('computeApiVersion')]",
      "type": "Microsoft.Compute/availabilitySets",
      "name": "[variables('asWebName')]",
      "location": "[variables('location')]",
      "dependsOn": [],
      "properties": {
      }
    },

    {
      "apiVersion": "[variables('networkApiVersion')]",
      "type": "Microsoft.Network/publicIPAddresses",
      "name": "[variables('publicIPAddressName')]",
      "location": "[resourceGroup().location]",
      "properties": {
        "publicIPAllocationMethod": "Dynamic"
      }
    },
    {
      "apiVersion": "[variables('networkApiVersion')]",
      "type": "Microsoft.Network/loadBalancers",
      "name": "[variables('lbWebName')]",
      "location": "[variables('location')]",
      "dependsOn": [
        "[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'))]"
      ],
      "properties": {
        "frontendIPConfigurations": [
          {
            "name": "FrontEnd",
            "properties": {
              "publicIPAddress": {
                "id": "[variables('publicIPAddressID')]"
              }
            }
          }
        ],
        "backendAddressPools": [
          {
            "name": "BackEnd"
          }
        ],
        "inboundNatRules": [
        ],
        "loadBalancingRules": [
          {
            "name": "HTTPS",
            "properties": {
              "frontendIPConfiguration": {
                "id": "[concat(resourceId('Microsoft.Network/loadBalancers/', variables('lbWebName')),'/frontendIPConfigurations/FrontEnd')]"
              },
              "backendAddressPool": {
                "id": "[concat(resourceId('Microsoft.Network/loadBalancers/', variables('lbWebName')),'/backendAddressPools/BackEnd')]"
              },
              "protocol": "Tcp",
              "frontendPort": 443,
              "backendPort": 443,
              "enableFloatingIP": false,
              "idleTimeoutInMinutes": 5,
              "loadDistribution": "Default",
              "probe": {
                "id": "[concat(resourceId('Microsoft.Network/loadBalancers/', variables('lbWebName')),'/probes/httpsProbe')]"
              }
            }
          },
          {
            "name": "HTTP",
            "properties": {
              "frontendIPConfiguration": {
                "id": "[concat(resourceId('Microsoft.Network/loadBalancers/', variables('lbWebName')),'/frontendIPConfigurations/FrontEnd')]"
              },
              "backendAddressPool": {
                "id": "[concat(resourceId('Microsoft.Network/loadBalancers/', variables('lbWebName')),'/backendAddressPools/BackEnd')]"
              },
              "protocol": "Tcp",
              "frontendPort": 80,
              "backendPort": 80,
              "enableFloatingIP": false,
              "idleTimeoutInMinutes": 5,
              "loadDistribution": "Default",
              "probe": {
                "id": "[concat(resourceId('Microsoft.Network/loadBalancers/', variables('lbWebName')),'/probes/httpProbe')]"
              }
            }
          }
        ],
        "probes": [
          {
            "name": "httpsProbe",
            "properties": {
              "protocol": "Http",
              "port": 443,
              "requestPath": "/",
              "intervalInSeconds": 5,
              "numberOfProbes": 2
            }
          },
          {
            "name": "httpProbe",
            "properties": {
              "protocol": "Http",
              "port": 80,
              "requestPath": "/",
              "intervalInSeconds": 5,
              "numberOfProbes": 2
            }
          }
        ]
      }
    },
    {
      "apiVersion": "[variables('networkApiVersion')]",
      "type": "Microsoft.Network/loadBalancers/inboundNatRules",
      "name": "[concat(variables('lbWebName'), '/', 'SSH-VM', copyIndex())]",
      "location": "[resourceGroup().location]",
      "copy": {
        "name": "lbNatLoop",
        "count": "[parameters('vmWebCount')]"
      },
      "dependsOn": [
        "[concat('Microsoft.Network/loadBalancers/', variables('lbWebName'))]"
      ],
      "properties": {
        "frontendIPConfiguration": {
          "id": "[concat(resourceId('Microsoft.Network/loadBalancers/', variables('lbWebName')),'/frontendIPConfigurations/FrontEnd')]"
        },
        "protocol": "tcp",
        "frontendPort": "[copyIndex(5000)]",
        "backendPort": 22,
        "enableFloatingIP": false
      }
    },

    {
      "apiVersion": "[variables('networkApiVersion')]",
      "type": "Microsoft.Network/networkInterfaces",
      "name": "[concat(variables('nicWeb0xName'), copyIndex())]",
      "location": "[variables('location')]",
      "dependsOn": [
        "[resourceId('Microsoft.Network/loadBalancers/', variables('lbWebName'))]",
        "[concat('Microsoft.Network/loadBalancers/', variables('lbWebName'), '/inboundNatRules/', 'SSH-VM', copyIndex())]"
      ],
      "properties": {
        "ipConfigurations": [
          {
            "name": "ipconfig1",
            "properties": {
              "privateIPAllocationMethod": "Dynamic",
              "subnet": {
                "id": "[variables('vNetSNRef')]"
              },
              "loadBalancerBackendAddressPools": [
                {
                  "id": "[concat(resourceId('Microsoft.Network/loadBalancers/', variables('lbWebName')),'/backendAddressPools/BackEnd')]"
                }
              ],
              "loadBalancerInboundNatRules": [
                {
                  "id": "[concat(variables('lbID'),'/inboundNatRules/SSH-VM', copyindex())]"
                }
              ]
            }
          }
        ],
        "dnsSettings": {
          "dnsServers": [
          ]
        }
      },
      "copy": {
        "name": "vmCopy",
        "count": "[parameters('vmWebCount')]"
      }
    },
    {
      "apiVersion": "[variables('computeApiVersion')]",
      "type": "Microsoft.Compute/virtualMachines",
      "name": "[concat(variables('vmWeb0xName'), copyIndex())]",
      "location": "[variables('location')]",
      "dependsOn": [
        "[concat('Microsoft.Storage/storageAccounts/', variables('saVhdName'))]",
        "[concat('Microsoft.Compute/availabilitySets/', variables('asWebName'))]",
        "[concat('Microsoft.Network/networkInterfaces/', concat(variables('nicWeb0xName'), copyIndex()))]"
      ],
      "properties": {
        "availabilitySet": {
          "id": "[resourceId('Microsoft.Compute/availabilitySets', variables('asWebName'))]"
        },
        "hardwareProfile": {
          "vmSize": "[variables('vmWebSize')]"
        },
        "storageProfile": {
          "imageReference": {
            "publisher": "[variables('imagePublisher')]",
            "offer": "[variables('imageOffer')]",
            "sku": "[variables('imageSKU')]",
            "version": "[variables('imageVersion')]"
          },
          "osDisk": {
            "name": "[concat(concat( variables('vmWeb0xName'), copyIndex() ), '-osdisk')]",
            "vhd": {
              "uri": "[concat('http://', variables('saVhdName'), '.blob.core.windows.net/vhds/', concat(concat( variables('vmWeb0xName'), copyIndex() ), '-osdisk.vhd'))]"
            },
            "caching": "ReadWrite",
            "createOption": "FromImage"
          },

          "dataDisks": [
            {
              "name": "[concat(concat( variables('vmWeb0xName'), copyIndex() ), '-datadisk')]",
              "diskSizeGB": "1023",
              "lun": 0,
              "vhd": {
                "uri": "[concat('http://', variables('saVhdName'), '.blob.core.windows.net/vhds/', concat(concat( variables('vmWeb0xName'), copyIndex() ), '-datadisk.vhd'))]"
              },
              "caching": "None",
              "createOption": "Empty"
            }
          ]
        },
        "osProfile": {
          "computerName": "[concat(variables('vmWeb0xComputerName'), copyIndex())]",
          "adminUsername": "[parameters('adminUserName')]",
          "adminPassword": "[parameters('adminPassword')]"
        },
        "networkProfile": {
          "networkInterfaces": [
            {
              "id": "[resourceId('Microsoft.Network/networkInterfaces', concat(variables('nicWeb0xName'), copyIndex()))]",
              "properties": { "primary": true }
            }
          ]
        },
        "diagnosticsProfile": {
          "bootDiagnostics": {
            "enabled": "true",
            "storageUri": "[concat('http://', variables('saVhdName'), '.blob.core.windows.net')]"
          }
        }
      },
      "resources": [
      ],
      "copy": {
        "name": "vmCopy",
        "count": "[parameters('vmWebCount')]"
      }
    },
    {
      "apiVersion": "[variables('computeApiVersion')]",
      "type": "Microsoft.Compute/virtualMachines/extensions",
      "name": "[concat(concat(variables('vmWeb0xName'), copyIndex()),'/CustomScript')]",
      "location": "[variables('location')]",
      "dependsOn": [
        "[resourceId('Microsoft.Compute/virtualMachines/', concat(variables('vmWeb0xName'), copyIndex()))]"
      ],
      "properties": {
        "publisher": "Microsoft.Azure.Extensions",
        "type": "CustomScript",
        "typeHandlerVersion": "2.0",
        "autoUpgradeMinorVersion": true,
        "settings": {
          "fileUris": "[split(parameters('fileUris'), ' ')]"
        },
        "protectedSettings": {
          "storageAccountName": "[parameters('customScriptStorageAccountName')]",
          "storageAccountKey": "[parameters('customScriptStorageAccountKey')]",
          "commandToExecute": "[parameters('commandToExecute')]"
        }
      },
      "copy": {
        "name": "vmCopy",
        "count": "[parameters('vmWebCount')]"
      }
    }
  ]
}

#!/bin/bash

echo "Disabling SELINUX"

setenforce 0 >> /tmp/setenforce.out
cat /etc/selinux/config > /tmp/beforeSelinux.out
sed -i 's^SELINUX=enforcing^SELINUX=disabled^g' /etc/selinux/config || true
cat /etc/selinux/config > /tmp/afterSeLinux.out

setenforce 0

yum makecache fast

rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm

yum -y install nginx18 php56w-fpm git nano wget

yum -y install php56w-mysql php56w-pear php56w-bcmath php56w-gd php56w-pdo php56w-mcrypt php56w-soap php56w-mbstring php56w-opcache php56w-devel gcc mariadb

echo "oauth install"

printf "\n" | pecl install oauth

echo -n extension=oauth.so >> /etc/php.ini

echo "Restarting and setting nginx to start on boot"

systemctl restart nginx; systemctl enable nginx

echo "Setting user and group = nginx in PHP-FPM"

sed -i 's/apache/nginx/g' /etc/php-fpm.d/www.conf

chown nginx:root /var/log/php-fpm -R

echo "Restarting PHP-FPM"

systemctl restart php-fpm; systemctl enable php-fpm

echo "Creating SSL folder in nginx root"
mkdir /etc/nginx/ssl

echo "n
p
1


p
w

" | fdisk /dev/sdc

mkfs -t ext4 /dev/sdc1

mkdir /data

mount /dev/sdc1 /data

var=$(blkid /dev/sdc1 -s UUID | awk -F'UUID="|"' '{print $2}')

echo >> /etc/fstab "UUID=$var /data ext4 defaults,noatime,data=writeback,barrier=0,nobh,errors=remount-ro 0 2"

echo "Creating a www folder on the data drive"

mkdir -p /data/www/

echo " Giving ownership to nginx on the WWW folder in the attached data disk"

chown nginx:nginx -R /data/www

echo "Enabling Swap"

sed -i 's/ResourceDisk.EnableSwap=n/ResourceDisk.EnableSwap=y/g' /etc/waagent.conf

sed -i 's/ResourceDisk.SwapSizeMB=0/ResourceDisk.SwapSizeMB=16384/g' /etc/waagent.conf

echo "* soft nofile 65536" >> /etc/security/limits.conf

echo "* hard nofile 65536" >> /etc/security/limits.conf

echo "* soft nproc 65536" >> /etc/security/limits.conf

echo "* hard nproc 65536" >>  /etc/security/limits.conf

echo "Setting maximum open file limit to 65536"

ulimit -SHn 65536

ulimit -SHu 65536

echo “ulimit -SHn 65536” >>/etc/rc.local

echo “ulimit -SHu 65536” >>/etc/rc.local

echo "Setting Time Zone To Europe/Bucharest"

timedatectl set-timezone Europe/Bucharest

echo "Optimizing TCP Stack"

echo net.ipv4.tcp_sack=1 >> /etc/sysctl.conf
echo net.core.rmem_max=4194304 >> /etc/sysctl.conf
echo net.core.wmem_max=4194304 >> /etc/sysctl.conf
echo net.core.rmem_default=4194304 >> /etc/sysctl.conf
echo net.core.wmem_default=4194304 >> /etc/sysctl.conf
echo net.core.optmem_max=4194304 >> /etc/sysctl.conf
echo net.ipv4.tcp_rmem="4096 87380 4194304" >> /etc/sysctl.conf
echo net.ipv4.tcp_wmem="4096 65536 4194304" >> /etc/sysctl.conf
echo net.ipv4.tcp_low_latency=1 >> /etc/sysctl.conf
sed -i "s/defaults        0 0/defaults,noatime        0 0/" /etc/fstab