Change Management

One Time Setup for Azure

One time infra changes needed for Azure deployment

Register Application

Register an application that will allow deployment python script to be used from the build and staging servers

1. Go to the Azure portal and sign in using your credentials.
2. Look for "App registration" in azure services. Click on the "App registration". If you do not see this option check with the system admin team to provide the necessary permissions/roles in Azure active directory (Azure AD). Typically users under role Application Developer, Application Administrator, Cloud Application Administrator, Global Administrator, have access to this option.
3. This will navigate you to the app registration home screen. Under "All applications" tab you would find existing app registrations (if any).
4. Click on "New registration". This will navigate you to the "Register an application" page.
5. Mention name, who can use this application on access this App. For example, use scribble-deploy as name. Select "Accounts in this organizational directory only - Single tenant"
6. Click on "Register". This will create the app registration.
7. You will be navigated to the app details page where you can find the "Application (client) ID". Copy this ID and add it to siteconf. This will be used as the client_id in the deploy config. Also, this page will show Directory (tenant) ID. Copy this ID and add it to siteconf. This will be used as the tenant_id in the deploy config.
8. Select certifications and secrets from left pane. You will be navigated to certificates and secrets page. Select Client secrets tab and click on New client secret.
9. Right page will open up a dialogue to specify client secret description and expiry. Please specify description for example, Client secrets for scribble deploy. Please refer to organization policy for expiry. Click on Add.
10. Client secrets tab would show secret id, value for the new secret created. Note down the value field from client secret. This will be used as the client_secret in the deploy script. You would need to create new client secret if you lose the value.

Grant Permissions

Grant necessary permissions for the Management Application

1. If you are using azure disk mount based deployment following permissions needs to be granted for the Application (Client) id generated in the steps above.

   [
     "Microsoft.Compute/disks/read",
     "Microsoft.Compute/disks/write",
     "Microsoft.Compute/disks/delete",
     "Microsoft.Compute/disks/beginGetAccess/action",
     "Microsoft.Compute/snapshots/read",
     "Microsoft.Compute/snapshots/write",
     "Microsoft.Compute/snapshots/delete",
     "Microsoft.Compute/virtualMachines/read",
     "Microsoft.Compute/virtualMachines/write",
     "Microsoft.Network/networkInterfaces/join/action",
     "Microsoft.Resources/subscriptions/resourcegroups/read"
   ]
   

2. As a reference following script is available to define respective role definitions, role assignments in your resource groups. https://github.com/scribbledata/scribble-utils/blob/main/azure-server/role_assignments.sh

   Granting this permissions would require specific roles which allows Microsoft.Authorization/roleAssignments/write at the resource group level. If you do not have this permission granted please contact your IT/infra team so that they can grant these permissions for the application (client) id generated in the previous steps. 3. You would need to adjust the values specific to your environment.

   SUBSCRIPTION_ID="<Add the correct Subscription ID>"
   APPLICATION_CLIENT_ID="<This is obtained from the previous steps>"
   RESOURCE_GROUP="<Add the correct Resource Group>"
   

Create Blob storage container

1. Create storage account in Azure portal. This step is optional. You can reuse the existing storage account if you want. Sample command from azure cli is given below.

   az storage account create --name <your-storage-account-name> --resource-group <your-resource-group-name> --location eastus

Name of the storage account can be of your choice. 2. Create blob storage container under the storage account. Sample command from azure cli is given below. This blob storage container will be used for storing datasets uploaded by scribble. We recommend to create separate container so that permission management is separate from other blob storage containers.

   az storage container create --name <your-container-name> --account-name <your-storage-account-name> --auth-mode login

Name of the container can be of your choice. 3. Assign a blob data contributor role to the application (client) id generated in the previous steps. This will allow the application to upload datasets to the blob storage container.

az role assignment create --role "Storage Blob Data Contributor" --assignee <APPLICATION_CLIENT_ID> --scope /subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.Storage/storageAccounts/<STORAGE_ACCOUNT_NAME>/blobServices/default/containers/<CONTAINER_NAME>

Replace the placeholders with the actual values.

Adding Azure Credentials to the VM

1. SSH to the VM.
2. Create directory for storing the credentials

      mkdir -p ~/.private/credentials/azure/
      cd ~/.private/credentials/azure/
   

3. Create template for credentials.json using following command:

   echo '{
     "azure_deploy_credentials": {
       "tenant_id": "<MODIFY THIS VALUE>",
       "client_id": "<MODIFY THIS VALUE>",
       "client_secret": "<MODIFY THIS VALUE>"
     }
   }' > credentials.json
   

4. Check contents of credentials.json

   cat credentials.json
   

   Sample output

   {
     "azure_deploy_credentials": {
       "tenant_id": "<MODIFY THIS VALUE>",
       "client_id": "<MODIFY THIS VALUE>",
       "client_secret": "<MODIFY THIS VALUE>"
     }
   }
   

5. Create template for config.json using following command:

   echo '{
    "azure_deploy_config": {
      "subscription_id": "<MODIFY THIS VALUE>",
      "build_resource_group_name": "<MODIFY THIS VALUE>",
      "target_resource_group_name": "<MODIFY THIS VALUE>",
      "target_vm_name": "<MODIFY THIS VALUE>",
      "location": "<MODIFY THIS VALUE>",
      "build_server_vm": "<MODIFY THIS VALUE>",
       "disk_size_gb": {
         "code": 20,
         "config": 1,
         "data": 100
       },
        "blob_storage":{
          "storage_account_name": "<MODIFY THIS VALUE>",
          "container_name": "<MODIFY THIS VALUE>"
        }
    }
   }' > config.json
   

6. Check contents of config.json

   cat config.json
   

   Sample output

   {
      "azure_deploy_config": {
        "subscription_id": "<MODIFY THIS VALUE>",
        "build_resource_group_name": "<MODIFY THIS VALUE>",
        "target_resource_group_name": "<MODIFY THIS VALUE>",
        "target_vm_name": "<MODIFY THIS VALUE>",
        "location": "<MODIFY THIS VALUE>",
        "build_server_vm": "<MODIFY THIS VALUE>",
        "disk_size_gb": {
           "code": 20,
           "config": 1,
           "data": 100
        },
        "blob_storage":{
          "storage_account_name": "<MODIFY THIS VALUE>",
          "container_name": "<MODIFY THIS VALUE>"
        }
      }
   }
   

7. Edit values in credentials.json.

   1. tenant_id : This is the Directory (tenant) ID from the app registration details. (Step 7 in register-application).
   2. client_id : This is the Application (client) ID from the app registration details. (Step 7 in register-application).
   3. client_secret : This is the value from the client secret details. (Step 10 in register-application).

8. Edit values in config.json.

   1. subscription_id : This is the subscription id of the azure account. This can be found in the azure portal.
   2. build_resource_group_name : This is the resource group name where the build-server VM is present.
   3. target_resource_group_name : This is the resource group name where the staging-server VM is present. This can be same as the build resource group.
   4. target_vm_name : This is name of staging-server VM. Look at the azure portal for the vm name.
   5. location : This is azure region to be used for creating the disk. e.g. eastus.
   6. build_server_vm : This is the name of the VM for the build server. Look at the azure portal for the vm name.
   7. disk_size_gb : This is the disk size in GB for the code, config and data disks. This is used for creating the disks.
   8. blob_storage : This is the blob storage details.
      1. storage_account_name : This is the storage account name where the container is created. (Step 1 in create-blob-storage-container))
      2. container_name : This is the container name where the datasets are uploaded. (Step 2 in create-blob-storage-container).

9. Encrypt azure credentials using following command.

   ~/enrich/deploy/bin/encrypt.sh cred.zip ~/.private/credentials/azure/credentials.json ~/.private/credentials/azure/config.json
   

   This script will ask for a user input for the password for encryption. Please choose strong password which is mix of alphabets, numbers and special characters. Sample output

   ./bin/encrypt.sh cred.zip .private/credentials/azure/credentials.json .private/credentials/azure/config.json
   Password?#my@very_verylong?password123
   
   7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
   p7zip Version 16.02 (locale=en_IN,Utf16=on,HugeFiles=on,64 bits,12 CPUs 13th Gen Intel(R) Core(TM) i5-1335U (B06A3),ASM,AES-NI)
   
   Open archive: cred.zip
   --
   Path = cred.zip
   Type = zip
   Physical Size = 571
   
   Scanning the drive:
   2 files, 531 bytes (1 KiB)         
   
   Updating archive: cred.zip
   
   Items to compress: 2
   
   
   Files read from disk: 2
   Archive size: 572 bytes (1 KiB)
   Everything is Ok
   Encrypted creds in cred.zip
   

   This will create encrypted file cred.zip in the current directory.

10. Remove the unencrypted files.

    rm  ~/.private/credentials/azure/credentials.json ~/.private/credentials/azure/config.json