Successful start of IaC – automated testing tools

The Business Value of Infrastructure as Code

As DevOps grows, it helps to know how it works. One of the big things in DevOps is “infrastructure as code” aka IaC. This means that you treat your infrastructure the exact same as you would treat your application code. So you’ll check it into version control, write tests for it, and make sure that it doesn’t diverge from what you have across multiple environments. Infrastructure deployments with IaC are repeatable and prevent runtime issues caused by configuration drift or missing dependencies. However, human error, particularly as long as manual processes are still around, cannot be removed. But it’s still possible to avoid critical issues upon deployment and prevent security leaks of sensitive data inside of IaC.

In DevoVision we always recommend to our customers prepare CI to build for IaC, regardless of technology stack usage in the project. The classic CI process looks like: build, test, publish artifacts. The same as in application development, IaC should have similar steps. IaC scan tooling can be easily integrated into the CI system and made recommendations before deploying code even to the development environment.

Terraform

Terraform validate command – it is a good practice to always run terraform validate against your terraform files before pushing them to your version control system. Also, this level of validation should be a part of CI pipeline.

Out of the box: integrate one of 3rd party open source scanning tools. The following tools provide static analysis for Terraform files:

Example of the integrated steps to Azure DevOps with Checkov scanning tool:

## Checkov Test Run
- bash: |
    docker run --rm -t -v $(System.DefaultWorkingDirectory):/terraform bridgecrew/checkov --directory /terraform --skip-check CKV_AZURE_35,CKV_AZURE_13 -o junitxml > $(System.DefaultWorkingDirectory)/junit.xml
    sed -i '$d' $(System.DefaultWorkingDirectory)/junit.xml
    sed -i '$d' $(System.DefaultWorkingDirectory)/junit.xml
    displayName: 'Checkov Static Code Analysis'
    continueOnError: true 

## Publish Test results in Azure DevOps 
- task: PublishTestResults@2
    displayName: 'Publish Checkov Test Results'
    condition: succeededOrFailed()
    inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: 'junit.xml'
    searchFolder: '$(System.DefaultWorkingDirectory)'
    mergeTestREsults: false 
    failTaskOnFailedTests: true 
    publishRunAttachments: true

Test results:

ARM

The ARM Template Tool Kit is a static code analyzer for ARM templates created by Microsoft. It’s an open-source PowerShell library that you can use to validate your templates against a series of test cases. These test cases are generic and designed to validate that your templates are following best practice.

## Run ARM TTK Test
- task: RunARMTTKTests@1
  displayName: "Run ARM Template Unit Tests"
  inputs:
    templatelocation: $(System.DefaultWorkingDirectory)\arm
    resultLocation: '$(System.DefaultWorkingDirectory)\results'
    skipTests: 'IDs Should Be Derived From ResourceIDs,Location Should Not Be Hardcoded,apiVersions Should Be Recent,Parameters Must Be Referenced,DeploymentTemplate Must Not Contain Hardcoded Uri,Outputs Must Not Contain Secrets,DependsOn Best Practices,ResourceIds should not contain'

## Publish Test results in Azure DevOps
- task: PublishTestResults@2
  displayName: "Publish Test Results"
  inputs:
    testResultsFormat: 'NUnit'
    testResultsFiles: '$(System.DefaultWorkingDirectory)\results\*-armttk.xml'
  condition: always()

Test results:

What benefit do you think Infrastructure as Code will bring to your Business?

Use the Contacts Form section and let us know your thoughts.

As more companies are looking to move to the cloud, there exists a great opportunity to adopt IaC. This approach allows for a consistent and clear look at what is being created, where, and most importantly, why. If you’re using IaC, then you’ll have an advantage in getting your infrastructure and applications out there.

Resources:

  1. https://docs.microsoft.com/en-us/azure/developer/terraform/best-practices-integration-testing
  2. https://github.com/sam-cogan/arm-ttk-extension

AKS Cluster Backup and Restore

Why worry about Backup and Restore?

It’s important when dealing with data to always have in mind what would happen, both to you and your customer, if that data were to be lost and were no longer available. As you can imagine, the outcome is rarely optimal. When it comes to cloud computing, concepts such as Disaster Recovery and High Availability are often discussed and put into practice for this very reason. If any of the data storage in AKS cluster were to fail, we need to ensure that we have a backup data disk from which we can restore and continue running normally.

What resources should I be backing up?

The only resources that need to be backed up from an AKS resource group are the mounted Persistent Storage disk resources. This means any OS Disk resources, often labeled aks-agentpool-NUMBER-1_ID, or other VM components, NSG, Load Balancers, etc need not be snapshot as they do not contain any data we need. For an example, see the picture below.

AKS Cluster

What format should the data be backed up in?

The backed-up data should be snapshot of the desired disk.

How do I go about backing up my AKS Cluster?

As there is currently no way to natively back up persistent storage in Azure, you must manually snapshot each disk you wish to back up. Ideally this would be done by writing a Snapshot Persistent Storage Disks task where applicable. Please refer to Dynamically create and use a persistent volume with Azure disks in Azure Kubernetes Service 1 for more information and steps on how to do so. Keep in mind, you will need the subscription ID and resource group name for appreciate AKS resource group, as well as the kubectl tool.

How do I go about restoring data I have backed up?

Again, as there is currently no way to natively back up persistent storage in Azure, you must manually restore each disk you’d like from each individual snapshot.

Resources:

  1. https://docs.microsoft.com/en-us/azure/aks/azure-disks-dynamic-pv

So, what is Azure Machine Learning?

What is machine learning?

Machine learning is a data science technique that allows computers to use existing data to forecast future behaviors, outcomes and trends. By using machine learning, computers learn without being explicitly programmed. Forecasts or predictions from machine learning can make apps and devices smarter. For example, when you shop online, machine learning helps recommend other products you might want based on what you have bought. Or when your credit card is swiped, machine learning compares the transaction to a database of transactions and helps detect fraud. And when your robot vacuum cleaner vacuums a room, machine learning helps it decide whether the job is done.

What is Azure Machine Learning service? 

Azure Machine Learning service provides a cloud-based environment you can use to prep data, train, test, deploy manage and track machine learning models.

Azure machine learning

Azure Machine Learning service fully supports open-source technologies. So, you can use tens of thousands of open-source Python packages with machine learning components. Examples are PyTorch, TensorFlow and scikit-learn. Support for rich tools makes it easy to interactively explore and prepare data and then develop and test models. Examples are Jupyter notebooks or the Azure Machine Learning for Visual Studio Code extension. Azure Machine Learning service also includes features that automate the model generation and tuning to help you create models with ease, efficiency and accuracy.  

By using Azure Machine Learning service, you can start training on your local machine and then scale out to the cloud.

With many available compute targets, like Azure Machine Learning Compute and Azure Databricks and with advanced hyperparameter tuning services, you can build better models faster by using the power of the cloud.

When you have the right model, you can easily deploy it in a container such as Docker. So, it’s simple to deploy to Azure Container Instances or Azure Kubernetes Service. Or you can use the container in your own deployments, either on-premises or in the cloud. For more information, see the article on how to deploy and where.  

You can manage the deployed models and track multiple runs as you experiment to find the best solution. After it’s deployed, your model can return predictions in real-time or asynchronously on large quantities of data. And with advanced machine learning pipelines, you can collaborate on all the steps of data preparation, model training and evaluation and deployment.

What can I do with Azure Machine Learning service?

Using the main Python SDK and the Data Prep SDK for Azure Machine Learning as well as open-source Python packages, you can build and train highly accurate machine learning and deep-learning models yourself in an Azure Machine Learning Service Workspace. You can choose from many machine learning components available in open0source Python packages, such as the following examples:

  • Scikit-learn; 
  • TensorFlow;
  • PyTorch;
  • CNTK;
  • MXNet.

Azure Machine Learning service can also autotrain a model and autotune it for you. After you have a model, you use it to create a container, such as Docker, that can be deployed locally for testing. After testing is done, you can deploy the model as a production web service in either Azure Container Instances or Azure Kubernetes Service. Then you can manage your deployed models by using the Azure Machine Learning SDK for Python or the Azure portal. You can evaluate model metrics, retrain and redeploy new versions of the model, all while tracking the model experiments.

Deploy models with the Azure Machine Learning service

The Azure Machine Learning SDK provides several ways you can deploy your trained model. You can deploy models to the following compute targets:

Deploy models

Azure cost saving opportunities

Cost-effectiveness is one of the advertised benefits of cloud computing. However, a survey of 100 IT decision-makers 1 in companies with 500 or more employees conducted by NetEnrich found that top cloud computing issues are:

  • Security (68%)
  • Cost overruns (59%)
  • Cost of recruiting cloud professionals (48%)

What you can do about it?

Manage resources appropriately:

  • Shutdown during weekend when no one is using the environment.

For development and test environments, do you need a full set of data in your databases?

  • For development and test environments, you can streamline data volumes upon database refresh from production to work with a smaller dataset.
  • Right/Downsize development resources that are under-utilized.

Remove Azure resources that are no longer needed.

Receive a discount on your Azure services by purchasing resource reservations (savings can be up to 72%).

Azure Cosmos Database overview

Azure Cosmos Database should be used when:

  • An Azure SQL Database is not a feasible option;
  • The solution is globally distributed – Data can be replicated to the geolocation from where users are accessing, which helps in serving data quickly with low latency;
  • Low latency – Cosmos DB guarantees 10 milliseconds latency at the 99th percentile for reads and writes for all consistency levels;
  • Horizontally scalable – Ability to handle the increased load by adding more servers to the cluster;
  • High availability is needed – Cosmos DB provide 99.999% availability for both reads and writes for multi-region accounts with multi-region writes;
  • Multi-model database service is needed – Document store, Graph DBMS, Key Value store, Columnar store.

Azure SQL database general recommendations

1) When designing a solution that leverage an Azure SQL Database, create a single database with multiple schemas instead of multiple databases within a solution.

2) Based on testing various configuration for different workloads and data volumes, below are recommended configurations:

  • For Meta Stores, utilize Standard S0 database (10DTUs);
  • For a simple application, utilize the Standard S1 database (20 DTUs) in Development and Test Environments. Use Premium P1 databases (125 DTUs) only in Production;
  • For medium complexity applications, utilize Standard S3 Databases (100 DTUs) in Development and Test Environments. Use Premium P1 databases (125 DTUs) only in Production;
  • For Business-Critical applications, utilize Gen5(16 vCores) for complex workloads in Production.

Other ways? Azure SQL with auto-pause settings

“Azure SQL Database serverless automatically scales compute for single databases based on workload demand and bills for compute used per second.  Serverless also provides an option to automatically pause the database during inactive usage periods when only storage costs are billed” 2 – Microsoft says, but how it’s in real life?

So, to check this I will create two similar Azure SQL Databases, but with only one difference – Auto-Pause Enabled and Auto-Pause Disabled option.

For Database with Available Auto-pause delay I choose the following configuration settings:

 Auto-pause delay

Wait some time until results appear in dashboard…

Auto-Pause Disable
Azure SQL Serverless Auto-Pause Disable
Azure SQL Serverless Auto-Pause Enabled
Azure SQL Serverless Auto-Pause Enabled

Cost comparing between:

As it was just a simple test, the results are exceeding all expectations. Cost-saving is more than 57%.

Consideration

Serverless will not work for all cases:

  • IOPs are limited;
  • A bad serverless implementation can actually increase your actual costs;
  • Application code needs to be adapted to serverless (retry logic);
  • SSMS connectivity can keep database awake, spending money;
  • No way to force a pause state.

Resources:

  1. https://www.globenewswire.com/news-release/2019/01/17/1701128/0/en/Enterprise-IT-Focused-on-Moving-More-Workloads-to-Cloud-in-2019.html
  2. https://azure.microsoft.com/en-us/updates/update-to-azure-sql-database-serverless-providing-even-greater-price-optimization/  

DevOps transformation

Just what is DevOps?

DevOps is a mindset, a culture, and a set of technical practices. It provides communication, integration, automation and close cooperation among all the people needed to plan, develop, test, deploy, release and maintain the solution.

DevOps brings business, development and operations stakeholders together to streamline IT delivery. DevOps focuses on engineering culture and automation practices for development and operations. DevOps is a combination of new ways of collaboration paired with advanced possibilities of automation reducing manual efforts and accelerating the software delivery process.  

Common product development challenges that drive the adoption of DevOps

  • Monolithic products – 9%
  • Inconsistent environments – 19%
  • Long release cycles – 19%
  • Poor code quality – 19%
  • Reduce production incidents – 10%
  • Inefficient development processes – 19%
  • Migrate from legacy tools – 5%

(re) Imagine what if you could…

DevOps Transformation

DevOps enables delivery agility, increased software quality and improve cross-team collaboration

Agile: build the right thing:

  • Prioritize on business value
  • Iterate to learn
  • Cross functional teams

DevOps: build the thing right:

  • Automate software delivery
  • Engineering excellence practices
  • Improve and automate operations
DevOps transformation

Both Agile and DevOps are as much about changing culture and changing organizational behavior as they are about new skills acquisition and technology & tools. For this reason, it’s the combination of tooling, training, coaching and transformation (with a heavy emphasis on organizational change management) that will be needed to break down the walls between the different pockets in the organization and to change the culture to a more cross-functional and highly collaborative way of working. This will enable an end-to-end value delivery ecosystem that involves the entire enterprise.

Culture change

Every business will become a software business, build applications, use advanced analytics and provide SaaS services.” – Satya Nadella, CEO of Microsoft. 

Action steps to be done

People:

  • Cross-train on technologies and applications to better leverage bandwidth;
  • Develop consistent improvement mindset for ruthless automation;
  • Move IT interactions closer to the business.

Processes:

  • Automate where applicable;
  • Lean process requirements;
  • Shift from first-in first-out to value based on work prioritization.

Tools

  • Integrate all work intake systems into a central work management tool;
  • Develop a pipeline automation framework to speed up routine tasks.

Want to get more? Reach out DevoVision team to start DevOps journey!