Volume-based Change

Hasper supports a second approach to change manage involving volumes. The idea is to take snapshots of volumes and deploy them.

Common setup for azure

Common setup for azure

One time infra changes needed for Azure deployment

Register Application

Register an application that will allow deployment python script to be used from the build and staging servers

1. Go to the Azure portal and sign in using your credentials.
2. Look for "App registration" in azure services. Click on the "App registration". If you do not see this option check with the system admin team to provide the necessary permissions/roles in Azure active directory (Azure AD). Typically users under role Application Developer, Application Administrator, Cloud Application Administrator, Global Administrator, have access to this option.
3. This will navigate you to the app registration home screen. Under "All applications" tab you would find existing app registrations (if any).
4. Click on "New registration". This will navigate you to the "Register an application" page.
5. Mention name, who can use this application on access this App. For example, use scribble-deploy as name. Select "Accounts in this organizational directory only - Single tenant"
6. Click on "Register". This will create the app registration.
7. You will be navigated to the app details page where you can find the "Application (client) ID". Copy this ID and add it to siteconf. This will be used as the client_id in the deploy config. Also, this page will show Directory (tenant) ID. Copy this ID and add it to siteconf. This will be used as the tenant_id in the deploy config.
8. Select certifications and secrets from left pane. You will be navigated to certificates and secrets page. Select Client secrets tab and click on New client secret.
9. Right page will open up a dialogue to specify client secret description and expiry. Please specify description for example, Client secrets for scribble deploy. Please refer to organization policy for expiry. Click on Add.
10. Client secrets tab would show secret id, value for the new secret created. Note down the value field from client secret. This will be used as the client_secret in the deploy script. You would need to create new client secret if you lose the value.

Grant Permissions

Grant necessary permissions for the Management Application

1. If you are using azure disk mount based deployment following permissions needs to be granted for the Application (Client) id generated in the steps above.

   [
     "Microsoft.Compute/disks/read",
     "Microsoft.Compute/disks/write",
     "Microsoft.Compute/disks/delete",
     "Microsoft.Compute/disks/beginGetAccess/action",
     "Microsoft.Compute/snapshots/read",
     "Microsoft.Compute/snapshots/write",
     "Microsoft.Compute/snapshots/delete",
     "Microsoft.Compute/virtualMachines/read",
     "Microsoft.Compute/virtualMachines/write",
     "Microsoft.Network/networkInterfaces/join/action",
     "Microsoft.Resources/subscriptions/resourcegroups/read"
   ]
   

2. As a reference following script is available to define respective role definitions, role assignments in your resource groups. https://github.com/scribbledata/scribble-utils/blob/main/azure-server/role_assignments.sh

   Granting this permissions would require specific roles which allows Microsoft.Authorization/roleAssignments/write at the resource group level. If you do not have this permission granted please contact your IT/infra team so that they can grant these permissions for the application (client) id generated in the previous steps. 3. You would need to adjust the values specific to your environment.

   SUBSCRIPTION_ID="<Add the correct Subscription ID>"
   APPLICATION_CLIENT_ID="<This is obtained from the previous steps>"
   RESOURCE_GROUP="<Add the correct Resource Group>"
   

Create Blob storage container

1. Create storage account in Azure portal. This step is optional. You can reuse the existing storage account if you want. Sample command from azure cli is given below.

   az storage account create --name <your-storage-account-name> --resource-group <your-resource-group-name> --location eastus

Name of the storage account can be of your choice. 2. Create blob storage container under the storage account. Sample command from azure cli is given below. This blob storage container will be used for storing datasets uploaded by scribble. We recommend to create separate container so that permission management is separate from other blob storage containers.

   az storage container create --name <your-container-name> --account-name <your-storage-account-name> --auth-mode login

Name of the container can be of your choice. 3. Assign a blob data contributor role to the application (client) id generated in the previous steps. This will allow the application to upload datasets to the blob storage container.

az role assignment create --role "Storage Blob Data Contributor" --assignee <APPLICATION_CLIENT_ID> --scope /subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.Storage/storageAccounts/<STORAGE_ACCOUNT_NAME>/blobServices/default/containers/<CONTAINER_NAME>

Replace the placeholders with the actual values.

Setup for build server

1. SSH to the VM.
2. Create directory for storing the credentials

      mkdir -p ~/.private/credentials/azure/
      cd ~/.private/credentials/azure/
   

3. Create template for credentials.json using following command:

   echo '{
     "azure_deploy_credentials": {
       "tenant_id": "<MODIFY THIS VALUE>",
       "client_id": "<MODIFY THIS VALUE>",
       "client_secret": "<MODIFY THIS VALUE>"
     }
   }' > credentials.json
   

4. Check contents of credentials.json

   cat credentials.json
   

   Sample output

   {
     "azure_deploy_credentials": {
       "tenant_id": "<MODIFY THIS VALUE>",
       "client_id": "<MODIFY THIS VALUE>",
       "client_secret": "<MODIFY THIS VALUE>"
     }
   }
   

5. Create template for config.json using following command:

   echo '{
    "azure_deploy_config": {
      "subscription_id": "<MODIFY THIS VALUE>",
      "build_resource_group_name": "<MODIFY THIS VALUE>",
      "target_resource_group_name": "<MODIFY THIS VALUE>",
      "target_vm_name": "<MODIFY THIS VALUE>",
      "location": "<MODIFY THIS VALUE>",
      "build_vm_name": "<MODIFY THIS VALUE>",
      "disk_sku": "StandardSSD_LRS",
      "disk_size_gb": {
         "code": 20,
         "config": 1,
         "data": 100
      },
      "blob_storage":{
         "storage_account_name": "<MODIFY THIS VALUE>",
         "container_name": "<MODIFY THIS VALUE>"
      }
    }
   }' > config.json
   

6. Check contents of config.json

   cat config.json
   

   Sample output

   {
      "azure_deploy_config": {
        "subscription_id": "<MODIFY THIS VALUE>",
        "build_resource_group_name": "<MODIFY THIS VALUE>",
        "target_resource_group_name": "<MODIFY THIS VALUE>",
        "target_vm_name": "<MODIFY THIS VALUE>",
        "location": "<MODIFY THIS VALUE>",
        "build_vm_name": "<MODIFY THIS VALUE>",
        "disk_sku": "StandardSSD_LRS",
        "disk_size_gb": {
           "code": 30,
           "config": 1,
           "data": 100
        },
        "blob_storage":{
          "storage_account_name": "<MODIFY THIS VALUE>",
          "container_name": "<MODIFY THIS VALUE>"
        }
      }
   }
   

7. Edit values in credentials.json.

   1. tenant_id : This is the Directory (tenant) ID from the app registration details. (Step 7 in register-application).
   2. client_id : This is the Application (client) ID from the app registration details. (Step 7 in register-application).
   3. client_secret : This is the value from the client secret details. (Step 10 in register-application).

8. Edit values in config.json.

   1. subscription_id : This is the subscription id of the azure account. This can be found in the azure portal. If you have multiple subscriptions, you can select the subscription you want to use for a particular deployment; you can check output of the following command. Select the suitable subscription id from the list.

      az account list --output table
      

   2. build_resource_group_name : This is the resource group name where the build-server VM is present. You can check the list of resource groups in your subscription using the following command.

      az group list --output table

   3. target_resource_group_name : This is the resource group name where the staging-server VM is present. This can be same as the build resource group.

   4. target_vm_name : This is name of staging-server VM. Look at the azure portal for the vm name. Deployment assumes that the VM of specified size is already provised. This is target value would be different for staging and production environment. For config on staging use vm name of the staging. For config file on production use the vm name of the production. This value will be read by the upgrade script running on the server. Note that, this value could be different than hostname. It is used by azure commands to identify the VM. You can check the list of VMs using the following command

      az vm list --resource-group scribble-dev-azure --output table
      

   5. location : This is azure region to be used for creating the disk. e.g. eastus. You can find it using the following command

      az group list --output table
      

   6. disk_sku: The SKU of the disk. The SKU determines the disk performance and pricing. The following are the possible values for this parameter: StandardSSD_LRS, Premium_LRS, Standard_LRS, UltraSSD_LRS, Standard_ZRS, Premium_ZRS, StandardSSD_ZRS, UltraSSD_ZRS. LRS stands for Locally Redundant Storage and ZRS stands for Zone Redundant Storage.

      NOTE: We recommend StandardSSD_LRS

   7. build_vm_name : This is the name of the VM for the build server. Look at the azure portal for the vm name.

   8. disk_size_gb : This is the disk size in GB for the code, config and data disks. This is used for creating the disks.

   9. blob_storage : This is the blob storage details.

      1. storage_account_name : This is the storage account name where the container is created. (Step 1 in create-blob-storage-container))
      2. container_name : This is the container name where the datasets are uploaded. (Step 2 in create-blob-storage-container).

9. Download encrypt utility script to encrypt the credentials.

   wget https://raw.githubusercontent.com/scribbledata/scribble-utils/main/credentials-manager/encrypt-util.sh
   chmod +x encrypt-util.sh
   

10. Encrypt azure credentials using following command.

    ./encrypt-util.sh ~/.private/credentials/azure/credentials.json ~/.private/credentials/azure/config.json
    

    This script will ask for a user input for the password for encryption. Please choose strong password which is mix of alphabets, numbers and special characters. Sample output for this command is as follows:

    ./encrypt-util.sh credentials.json config.json
    Encrypting given files using user provided password.
    Please provide the password of your choice.
    Password: #my@very_verylong?password123
    
    7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
    p7zip Version 16.02 (locale=en_IN,Utf16=on,HugeFiles=on,64 bits,12 CPUs 13th Gen Intel(R) Core(TM) i5-1335U (B06A3),ASM,AES-NI)
    
    Open archive: cred.zip
    --
    Path = cred.zip
    Type = zip
    Physical Size = 792
    
    Scanning the drive:
    1 file, 216 bytes (1 KiB)
    
    Updating archive: cred.zip
    
    Items to compress: 1
    
    
    Files read from disk: 1
    Archive size: 792 bytes (1 KiB)
    Everything is Ok
    
    7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
    p7zip Version 16.02 (locale=en_IN,Utf16=on,HugeFiles=on,64 bits,12 CPUs 13th Gen Intel(R) Core(TM) i5-1335U (B06A3),ASM,AES-NI)
    
    Open archive: cred.zip
    --
    Path = cred.zip
    Type = zip
    Physical Size = 792
    
    Scanning the drive:
    1 file, 652 bytes (1 KiB)
    
    Updating archive: cred.zip
    
    Items to compress: 1
    
    
    Files read from disk: 1
    Archive size: 792 bytes (1 KiB)
    Everything is Ok
    Encrypted creds in cred.zip
    Testing decryption using given password.
    
    7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
    p7zip Version 16.02 (locale=en_IN,Utf16=on,HugeFiles=on,64 bits,12 CPUs 13th Gen Intel(R) Core(TM) i5-1335U (B06A3),ASM,AES-NI)
    
    Scanning the drive for archives:
    1 file, 792 bytes (1 KiB)
    
    Extracting archive: cred.zip
    --
    Path = cred.zip
    Type = zip
    Physical Size = 792
    
    Everything is Ok
    
    Files: 2
    Size:       868
    Compressed: 792
    Decrypted creds from cred.zip to 'decrypted' directory
    File config.json matches
    File credentials.json matches
    All files match
    Encryption, decryption, and testing successful.
    Cleaning up the files.
    Cleaning up the original files.
    Removing original file: credentials.json
    Removing original file: config.json
    Removing decrypted directory.
    

    This will create encrypted file cred.zip in the current directory.

Setup for staging server

1. SSH to the VM.
2. Create directory for storing the credentials

      mkdir -p ~/.private/credentials/azure/
      cd ~/.private/credentials/azure/
   

3. Create template for credentials.json using following command:

   echo '{
     "azure_deploy_credentials": {
       "tenant_id": "<MODIFY THIS VALUE>",
       "client_id": "<MODIFY THIS VALUE>",
       "client_secret": "<MODIFY THIS VALUE>"
     }
   }' > credentials.json
   

4. Check contents of credentials.json

   cat credentials.json
   

   Sample output

   {
     "azure_deploy_credentials": {
       "tenant_id": "<MODIFY THIS VALUE>",
       "client_id": "<MODIFY THIS VALUE>",
       "client_secret": "<MODIFY THIS VALUE>"
     }
   }
   

5. Create template for config.json using following command:

   echo '{
    "azure_deploy_config": {
      "subscription_id": "<MODIFY THIS VALUE>",
      "build_resource_group_name": "<MODIFY THIS VALUE>",
      "target_resource_group_name": "<MODIFY THIS VALUE>",
      "target_vm_name": "<MODIFY THIS VALUE>",
      "location": "<MODIFY THIS VALUE>",
      "build_vm_name": "<MODIFY THIS VALUE>",
      "disk_sku": "StandardSSD_LRS",
      "disk_size_gb": {
         "code": 20,
         "config": 1,
         "data": 100
      },
      "blob_storage":{
         "storage_account_name": "<MODIFY THIS VALUE>",
         "container_name": "<MODIFY THIS VALUE>"
      }
    }
   }' > config.json
   

6. Check contents of config.json

   cat config.json
   

   Sample output

   {
      "azure_deploy_config": {
        "subscription_id": "<MODIFY THIS VALUE>",
        "build_resource_group_name": "<MODIFY THIS VALUE>",
        "target_resource_group_name": "<MODIFY THIS VALUE>",
        "target_vm_name": "<MODIFY THIS VALUE>",
        "location": "<MODIFY THIS VALUE>",
        "build_vm_name": "<MODIFY THIS VALUE>",
        "disk_sku": "StandardSSD_LRS",
        "disk_size_gb": {
           "code": 20,
           "config": 1,
           "data": 100
        },
        "blob_storage":{
          "storage_account_name": "<MODIFY THIS VALUE>",
          "container_name": "<MODIFY THIS VALUE>"
        }
      }
   }
   

7. Edit values in credentials.json.

   1. tenant_id : This is the Directory (tenant) ID from the app registration details. (Step 7 in register-application).
   2. client_id : This is the Application (client) ID from the app registration details. (Step 7 in register-application).
   3. client_secret : This is the value from the client secret details. (Step 10 in register-application).

8. Edit values in config.json.

   1. subscription_id : This is the subscription id of the azure account. This can be found in the azure portal.
   2. build_resource_group_name : This is the resource group name where the build-server VM is present.
   3. target_resource_group_name : This is the resource group name where the staging-server VM is present. This can be same as the build resource group.
   4. target_vm_name : This is name of staging-server VM. Look at the azure portal for the vm name.
   5. location : This is azure region to be used for creating the disk. e.g. eastus.
   6. build_vm_name : This is the name of the VM for the build server. Look at the azure portal for the vm name.
   7. disk_size_gb : This is the disk size in GB for the code, config and data disks. This is used for creating the disks.
   8. blob_storage : This is the blob storage details.
      1. storage_account_name : This is the storage account name where the container is created. (Step 1 in create-blob-storage-container))
      2. container_name : This is the container name where the datasets are uploaded. (Step 2 in create-blob-storage-container).
9. Download encrypt utility script to encrypt the credentials.

   wget https://raw.githubusercontent.com/scribbledata/scribble-utils/main/credentials-manager/encrypt-util.sh
   chmod +x encrypt-util.sh
   

10. Encrypt azure credentials using following command.

    ./encrypt-util.sh ~/.private/credentials/azure/credentials.json ~/.private/credentials/azure/config.json
    

    This script will ask for a user input for the password for encryption. Please choose strong password which is mix of alphabets, numbers and special characters.

Sample output for this command is as follows:

./encrypt-util.sh credentials.json config.json
Encrypting given files using user provided password.
Please provide the password of your choice.
Password: #my@very_verylong?password123

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_IN,Utf16=on,HugeFiles=on,64 bits,12 CPUs 13th Gen Intel(R) Core(TM) i5-1335U (B06A3),ASM,AES-NI)

Open archive: cred.zip
--
Path = cred.zip
Type = zip
Physical Size = 792

Scanning the drive:
1 file, 216 bytes (1 KiB)

Updating archive: cred.zip

Items to compress: 1


Files read from disk: 1
Archive size: 792 bytes (1 KiB)
Everything is Ok

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_IN,Utf16=on,HugeFiles=on,64 bits,12 CPUs 13th Gen Intel(R) Core(TM) i5-1335U (B06A3),ASM,AES-NI)

Open archive: cred.zip
--
Path = cred.zip
Type = zip
Physical Size = 792

Scanning the drive:
1 file, 652 bytes (1 KiB)

Updating archive: cred.zip

Items to compress: 1


Files read from disk: 1
Archive size: 792 bytes (1 KiB)
Everything is Ok
Encrypted creds in cred.zip
Testing decryption using given password.

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_IN,Utf16=on,HugeFiles=on,64 bits,12 CPUs 13th Gen Intel(R) Core(TM) i5-1335U (B06A3),ASM,AES-NI)

Scanning the drive for archives:
1 file, 792 bytes (1 KiB)

Extracting archive: cred.zip
--
Path = cred.zip
Type = zip
Physical Size = 792

Everything is Ok

Files: 2
Size:       868
Compressed: 792
Decrypted creds from cred.zip to 'decrypted' directory
File config.json matches
File credentials.json matches
All files match
Encryption, decryption, and testing successful.
Cleaning up the files.
Cleaning up the original files.
Removing original file: credentials.json
Removing original file: config.json
Removing decrypted directory.

This will create encrypted file cred.zip in the current directory.

Setup for production server

1. SSH to the VM.
2. Create directory for storing the credentials

      mkdir -p ~/.private/credentials/azure/
      cd ~/.private/credentials/azure/
   

3. Create template for credentials.json using following command:

   echo '{
     "azure_deploy_credentials": {
       "tenant_id": "<MODIFY THIS VALUE>",
       "client_id": "<MODIFY THIS VALUE>",
       "client_secret": "<MODIFY THIS VALUE>"
     }
   }' > credentials.json
   

4. Check contents of credentials.json

   cat credentials.json
   

   Sample output

   {
     "azure_deploy_credentials": {
       "tenant_id": "<MODIFY THIS VALUE>",
       "client_id": "<MODIFY THIS VALUE>",
       "client_secret": "<MODIFY THIS VALUE>"
     }
   }
   

5. Create template for config.json using following command:

   echo '{
    "azure_deploy_config": {
      "subscription_id": "<MODIFY THIS VALUE>",
      "build_resource_group_name": "<MODIFY THIS VALUE>",
      "target_resource_group_name": "<MODIFY THIS VALUE>",
      "target_vm_name": "<MODIFY THIS VALUE>",
      "location": "<MODIFY THIS VALUE>",
      "build_vm_name": "<MODIFY THIS VALUE>",
      "disk_sku": "StandardSSD_LRS",
      "disk_size_gb": {
         "code": 20,
         "config": 1,
         "data": 100
      },
      "blob_storage":{
         "storage_account_name": "<MODIFY THIS VALUE>",
         "container_name": "<MODIFY THIS VALUE>"
      }
    }
   }' > config.json
   

6. Check contents of config.json

   cat config.json
   

   Sample output

   {
      "azure_deploy_config": {
        "subscription_id": "<MODIFY THIS VALUE>",
        "build_resource_group_name": "<MODIFY THIS VALUE>",
        "target_resource_group_name": "<MODIFY THIS VALUE>",
        "target_vm_name": "<MODIFY THIS VALUE>",
        "location": "<MODIFY THIS VALUE>",
        "build_vm_name": "<MODIFY THIS VALUE>",
        "disk_sku": "StandardSSD_LRS",
        "disk_size_gb": {
           "code": 20,
           "config": 1,
           "data": 100
        },
        "blob_storage":{
          "storage_account_name": "<MODIFY THIS VALUE>",
          "container_name": "<MODIFY THIS VALUE>"
        }
      }
   }
   

7. Edit values in credentials.json.

   1. tenant_id : This is the Directory (tenant) ID from the app registration details. (Step 7 in register-application).
   2. client_id : This is the Application (client) ID from the app registration details. (Step 7 in register-application).
   3. client_secret : This is the value from the client secret details. (Step 10 in register-application).

8. Edit values in config.json.

   1. subscription_id : This is the subscription id of the azure account. This can be found in the azure portal.
   2. build_resource_group_name : This is the resource group name where the build-server VM is present.
   3. target_resource_group_name : This is the resource group name where the production-server VM is present. This can be same as the build resource group.
   4. target_vm_name : This is name of production-server VM. Look at the azure portal for the vm name.
   5. location : This is azure region to be used for creating the disk. e.g. eastus.
   6. build_vm_name : This is the name of the VM for the build server. Look at the azure portal for the vm name.
   7. disk_size_gb : This is the disk size in GB for the code, config and data disks. This is used for creating the disks.
   8. blob_storage : This is the blob storage details.
      1. storage_account_name : This is the storage account name where the container is created. (Step 1 in create-blob-storage-container))
      2. container_name : This is the container name where the datasets are uploaded. (Step 2 in create-blob-storage-container).
9. Download encrypt utility script to encrypt the credentials.

   wget https://raw.githubusercontent.com/scribbledata/scribble-utils/main/credentials-manager/encrypt-util.sh
   chmod +x encrypt-util.sh
   

10. Encrypt azure credentials using following command.

    ./encrypt-util.sh ~/.private/credentials/azure/credentials.json ~/.private/credentials/azure/config.json
    

    This script will ask for a user input for the password for encryption. Please choose strong password which is mix of alphabets, numbers and special characters.

Sample output for this command is as follows:

./encrypt-util.sh credentials.json config.json
Encrypting given files using user provided password.
Please provide the password of your choice.
Password: #my@very_verylong?password123

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_IN,Utf16=on,HugeFiles=on,64 bits,12 CPUs 13th Gen Intel(R) Core(TM) i5-1335U (B06A3),ASM,AES-NI)

Open archive: cred.zip
--
Path = cred.zip
Type = zip
Physical Size = 792

Scanning the drive:
1 file, 216 bytes (1 KiB)

Updating archive: cred.zip

Items to compress: 1


Files read from disk: 1
Archive size: 792 bytes (1 KiB)
Everything is Ok

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_IN,Utf16=on,HugeFiles=on,64 bits,12 CPUs 13th Gen Intel(R) Core(TM) i5-1335U (B06A3),ASM,AES-NI)

Open archive: cred.zip
--
Path = cred.zip
Type = zip
Physical Size = 792

Scanning the drive:
1 file, 652 bytes (1 KiB)

Updating archive: cred.zip

Items to compress: 1


Files read from disk: 1
Archive size: 792 bytes (1 KiB)
Everything is Ok
Encrypted creds in cred.zip
Testing decryption using given password.

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_IN,Utf16=on,HugeFiles=on,64 bits,12 CPUs 13th Gen Intel(R) Core(TM) i5-1335U (B06A3),ASM,AES-NI)

Scanning the drive for archives:
1 file, 792 bytes (1 KiB)

Extracting archive: cred.zip
--
Path = cred.zip
Type = zip
Physical Size = 792

Everything is Ok

Files: 2
Size:       868
Compressed: 792
Decrypted creds from cred.zip to 'decrypted' directory
File config.json matches
File credentials.json matches
All files match
Encryption, decryption, and testing successful.
Cleaning up the files.
Cleaning up the original files.
Removing original file: credentials.json
Removing original file: config.json
Removing decrypted directory.

This will create encrypted file cred.zip in the current directory.