Azure Sentinel News
  • Home
    • Home – Layout 1
    • Home – Layout 2
    • Home – Layout 3
  • Security and Compliance
  • SOC
  • Threat Intelligence
  • Security Ochestration & Automated Response
  • SOAR
  • Security Operations
  • Artificial Intelligence
No Result
View All Result
  • Home
    • Home – Layout 1
    • Home – Layout 2
    • Home – Layout 3
  • Security and Compliance
  • SOC
  • Threat Intelligence
  • Security Ochestration & Automated Response
  • SOAR
  • Security Operations
  • Artificial Intelligence
No Result
View All Result
Azure Sentinel News
No Result
View All Result
Home Security and Compliance

HPC on Microsoft Azure: A Practical Guide

Azure Sentinel News Editor by Azure Sentinel News Editor
November 25, 2020
in Security and Compliance
0
Enriching Windows Security Events with Parameterized Function
3.2kViews
340 Shares Share on Facebook Share on Twitter

Deploying critical workloads like genomic sequencing, oil and gas, or electronic design automation requires speed, resilience and high-availability. High Performance Computing (HPC) offerings from Azure enable you to leverage cloud HPC, which can significantly reduce your costs while providing workload distribution and scalability.

Why Run HPC in the Cloud?

You can run HPC workloads in all major public or private clouds. Some cloud providers, like Azure, offer ready-to-use HPC clusters for machine learning, big data analytics, which you can leverage for any industry.

Benefits of HPC in the cloud include:

  • Availability—the high-availability of the cloud can reduce interruptions in HPC workload operations. High-availability also provides data protection since it is usually achieved through mirrored data centers.
  • Scalability—cloud controls can help you avoid paying for unused resources by increasing or decreasing storage, memory, and processing power of your HPC workloads as needed.
  • Cost—cloud infrastructure eliminates the need to purchase, maintain, and host servers. Outsourcing resources instead of buying reduces upfront costs and potentially removes technical debt. 
  • Workload distribution—using containerized microservices in the cloud enables you to orchestrate distributed HPC workloads. Containers can also help you move existing workloads and tools to the cloud.

Running HPC in the cloud requires components like low latency, batch scheduling features, high-throughput storage, bare-metal support, userspace communication functionality, high-speed interconnects, and clustered instances. 

How Do HPC Deployments Work in Microsoft Azure?

You can deploy HPC in Azure as an extension of on-premise deployments or entirely in the cloud. You can also run workloads simultaneously on-premise and in the cloud for hybrid deployments. Another option is to use Azure deployments only as burst capacity for larger workloads.

In Azure, you can distribute low-latency networking between your HPC nodes by using Remote Direct Memory Access (RDMA) or InfiniBand. To set up your nodes in Azure, you can use GPU-enabled Virtual Machines (VMs) with NVIDIA GPUs or customized compute-intensive VMs. The type of node you choose will depend on your budget and specific compute needs.

HPC in Azure components include:

  • HPC head node—a VM that operates as the central managing server for your HPC cluster. You can use it to schedule workloads and jobs to your worker nodes.
  • VM Scale Sets service—enables you to create multiple virtual machines with features for autoscaling, load balancing, and deployment across availability zones. You can use scale sets to run Hadoop, Cassandra, Cloudera, MongoDB, and Mesos.
  • Virtual network—a network that links your head node to your storage and compute nodes via IP. You can use this network to control the infrastructure and the traffic between subnets. Virtual Network creates an isolated environment for your deployment with secure connections via ExpressRoute OR IPsec VPN.
  • Storage—Azure storage options include disk, file, and blob storage. You can also use hybrid storage solutions or data lakes. You can add the Avere vFXT service to improve storage performance. Avere vFXT enables you to deploy file system workloads with low latency through an advanced cache feature.
  • Azure Resource Manager—a deployment and management service that enables you to use script files or templates to deploy HPC applications. It is the standard resource management tool in Azure.

Managing HPC Deployments in Azure 

Azure’s HPC deployment management services are native offerings. You can integrate them with other Azure services and use Azure support when needed.

Azure Batch

Azure Batch is a compute management and job scheduling service for HPC applications. To use this service, you need to configure your VM pool and upload your workloads. Azure Batch then provisions VMs, assigns and runs tasks, and monitors progress. You can use Batch to set policies for job scheduling and autoscale your deployment.

Microsoft HPC Pack

HPC Pack is a centralized management and deployment interface that enables you to manage your VM cluster schedule jobs, and monitor processes. You can use HPC Pack to provision and manage network infrastructure and VMs. HPC Pack is helpful for migrating existing, on-premise HPC workloads to the cloud. You can either migrate your workloads completely to the cloud or use HPC Pack to manage a hybrid deployment.

Azure CycleCloud

Azure CycleCloud is an enterprise-grade tool for managing and orchestrating HPC clusters on Azure. You can use CycleCloud to provision HPC infrastructure, and run and schedule jobs at any scale. CycleCloud enables you to create different file systems types in Azure and mount them to the compute cluster nodes to support HPC workloads.

Best Practices for Using HPC on Azure

The following guidelines can help you create and use large Azure deployments with your HPC cluster.

Split deployments to multiple services

Split large HPC deployments into several smaller-sized deployments by using multiple Azure services. You should use less than 500 to 700 virtual machine instances in each service. Larger deployments can lead to deployment timeouts, loss of live instances, and issues with virtual IP address swapping.

When dividing deployments, you get the following benefits:

  • Flexibility—in stopping and starting groups of nodes.
  • Shut down unused instances—if a job is no longer running.
  • Find available nodes in Azure clusters—when you use extra large instances.
  • Multiple Azure data centers—for business continuity and disaster recovery scenarios.

Use several Azure storage accounts for different node deployments

For applications that are constrained by I/O, use several Azure storage accounts to deploy large HPC nodes simultaneously. However, you should use these storage accounts only for node provisioning. For instance, you need to configure separate Azure storage accounts to move job and task data to and from the head node.

Change the number of proxy node instances

Proxy nodes are Azure worker role instances that enable communication between Azure nodes and on-premises head nodes. Proxy nodes are automatically added to each deployment of Azure node from an HPC cluster. The demand for resources on the proxy nodes depends on the number of deployed nodes in Azure and the running jobs on those nodes. 

You should increase the number of proxy nodes in a large Azure deployment. You can scale up or down the number of proxy nodes by using the Azure Management Portal. The advisable number of proxy nodes for a single deployment is ten.

Use HPC Client Utilities to connect remotely to the head node

The performance of the head node can be negatively impacted when operating under a heavy load, or when many users are connected with remote desktop connections. Instead of using Remote Desktop Services (RDS) to connect users to the head node, users should install HPC Pack Client Utilities on their workstations. These utilities enable users to access the cluster by using remote tools.

Conclusion

Organizations need the ability to process data in real time for purposes like live sporting event streaming, machine learning applications, or big data analytics. HPC in the cloud provides a fast, and highly reliable IT infrastructure to process, store, and analyze massive amounts of data. To get the most out of your HPC workload in Azure, you need to split deployments to multiple services, use multiple Azure storage accounts for node deployments, and change the number of proxy node instances.

Reference: https://wire19.com/hpc-on-microsoft-azure-practical-guide/

Azure Sentinel News Editor

Azure Sentinel News Editor

Related Posts

Vectra AI and Microsoft partner on security integration
Security and Compliance

Replay Now Available – Microsoft Security Insights 036: Azure Sentinel with Rod Trent

February 8, 2021
What’s new: Microsoft Teams connector in Public Preview
Security and Compliance

eBook Available for Managing Azure Sentinel with PowerShell

January 6, 2021
Microsoft is quietly becoming a cybersecurity powerhouse
Security and Compliance

Official Azure Sentinel PowerShell Module Released

January 4, 2021
Next Post
New Azure VMware Solution now generally available in Asia

New Azure VMware Solution now generally available in Asia

Microsoft’s John Thompson and VMware’s Sanjay Poonen share a similar view of the security landscape

Microsoft’s John Thompson and VMware’s Sanjay Poonen share a similar view of the security landscape

Microsoft bolsters threat prevention capabilities for enterprises

The Cloud Collective deploys Microsoft 365 for NRMA Insurance

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Follow Us

  • 21.8M Fans
  • 81 Followers

Recommended

Microsoft is quietly becoming a cybersecurity powerhouse

Official Azure Sentinel PowerShell Module Released

2 months ago
Microsoft improves Azure’s security to protect your business

How to Enable the Microsoft Teams Public Preview for Azure Sentinel

2 months ago
Enriching Windows Security Events with Parameterized Function

Gain Compliance, Posture, and Protection Insights with this Azure Security Center Related Workbook

3 months ago
Playbooks & Watchlists Part 1: Inform the subscription owner

Playbooks & Watchlists Part 1: Inform the subscription owner

5 months ago

Instagram

    Please install/update and activate JNews Instagram plugin.

Categories

  • AI & ML
  • Artificial Intelligence
  • Incident Response
  • IR
  • KQL
  • Security and Compliance
  • Security Ochestration & Automated Response
  • Security Operations
  • SIEM
  • SOAR
  • SOC
  • Threat Intelligence
  • Uncategorized

Topics

anomaly automation Azure Azure DevOps Azure Security Center Azure Sentinel Azure Sentinel API Azure Sentinel Connector BlueVoyant Call cybersecurity Detection file GitHub Hunting Huntingy IAC incident response Incident Triage infrastructure as code Investigation jupyter LAQueryLogs MDR Microsoft microsoft 365 mssp Multitenancy Notebooks Pester Playbooks PowerShell python Records Security Sentinel Sharing SIEM signin Supply Chain teams Threat hunting Watchlists Workbooks XDR
No Result
View All Result

Highlights

New Items of Note on the Azure Sentinel GitHub Repo

Tuning the MCAS Analytics Rule for Azure Sentinel: System Alerts and Feature Deprecation

New Search Capability for Azure Sentinel Incidents

Follow-up: Microsoft Tech Talks Practical Sentinel : A Day in the Life of a Sentinel Analyst

Changes in How Running Hunting Queries Works in Azure Sentinel

Azure Sentinel can now Analyze All Available Azure Active Directory Log Files

Trending

What’s new: Microsoft Teams connector in Public Preview
IR

How to Generate Azure Sentinel Incidents for Testing

by Azure Sentinel News Editor
February 26, 2021
0

Do you want to generate an Incident in Azure Sentinel for testing/demoing? Here’s a couple easy ways...

What’s new: Microsoft Teams connector in Public Preview

Azure Sentinel Notebooks Loses It’s Preview Tag

February 25, 2021
Microsoft’s newest sustainable datacenter region coming to Arizona in 2021

The Holy Grail of Azure Sentinel Data Connections: The Azure Service Diagnostic Setting

February 22, 2021
Microsoft’s newest sustainable datacenter region coming to Arizona in 2021

New Items of Note on the Azure Sentinel GitHub Repo

February 18, 2021
Microsoft’s newest sustainable datacenter region coming to Arizona in 2021

Tuning the MCAS Analytics Rule for Azure Sentinel: System Alerts and Feature Deprecation

February 17, 2021

We bring you the best, latest and perfect Azure Sentinel News, Magazine, Personal Blogs, etc. Visit our landing page to see all features & demos.
LEARN MORE »

Recent News

  • How to Generate Azure Sentinel Incidents for Testing February 26, 2021
  • Azure Sentinel Notebooks Loses It’s Preview Tag February 25, 2021
  • The Holy Grail of Azure Sentinel Data Connections: The Azure Service Diagnostic Setting February 22, 2021

Categories

  • AI & ML
  • Artificial Intelligence
  • Incident Response
  • IR
  • KQL
  • Security and Compliance
  • Security Ochestration & Automated Response
  • Security Operations
  • SIEM
  • SOAR
  • SOC
  • Threat Intelligence
  • Uncategorized

[mc4wp_form]

Copyright © 2020 - Azure Sentinel News

No Result
View All Result
  • Home
  • Security and Compliance
  • SOC
  • Threat Intelligence
  • Security Ochestration & Automated Response
  • SOAR
  • Security Operations
  • Artificial Intelligence

Copyright © 2020 Azure Sentinel News