Mission100 Azure Data Engineer Course By Deepak Goyal
https://adeus.azurelib.com
Email at: admin@azurelib.com
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/
what are the different ways to connect to azure account
1. Azure Portal
●
Description: The Azure Portal is a web-based interface where you can manage and monitor your Azure
services.
●
Usage: Simply go to https://portal.azure.com, sign in with your Azure account credentials, and you'll
have access to all your Azure resources.
●
Best For: Managing resources, creating services, monitoring activity, and performing administrative
tasks via a graphical interface.
2. Azure CLI (Command-Line Interface)
●
Description: Azure CLI is a cross-platform command-line tool that allows you to manage your Azure
resources directly from your terminal or command prompt.
●
Usage: After installing Azure CLI, you can connect by running the command az login, which will
open a web browser for you to sign in.
●
Best For: Automating tasks, scripting, and managing Azure resources via commands. It's popular
among developers and DevOps engineers.
3. Azure PowerShell
●
Description: Azure PowerShell is a set of modules that allow you to manage Azure resources using
PowerShell scripts and commands.
●
Usage: After installing Azure PowerShell, you can connect by running Connect-AzAccount, which
will prompt you to sign in through a web browser.
●
Best For: Scripting and automating Azure resource management tasks, especially for those familiar
with PowerShell.
4. Azure Mobile App
●
Description: The Azure mobile app lets you monitor and manage your Azure resources on the go.
●
Usage: Download the Azure app from the App Store or Google Play, sign in with your Azure credentials,
and you can monitor resources, check alerts, and perform basic management tasks.
●
Best For: Quick checks and light management tasks from a mobile device.
Mission100 Azure Data Engineer Course By Deepak Goyal
https://adeus.azurelib.com
Email at: admin@azurelib.com
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/
5. Azure Cloud Shell
●
Description: Azure Cloud Shell is an online, browser-based shell that provides you with a commandline experience in the Azure portal.
●
Usage: Available directly in the Azure Portal, you can choose either Bash or PowerShell environments
to manage Azure resources without installing anything locally.
●
Best For: Quick command-line tasks without needing to install anything on your local machine.
6. SDKs and APIs
●
Description: Azure provides Software Development Kits (SDKs) and REST APIs that allow you to
connect to and manage Azure services programmatically.
●
Usage: By using Azure SDKs available for various programming languages (like Python, .NET, Java, etc.),
you can connect to Azure directly within your application code.
●
Best For: Developers building applications that need to interact with Azure services programmatically.
Azure Regions:
An Azure region is a specific geographical area where Microsoft has data centers to host cloud
services and resources. Each region consists of multiple data centers located within a defined perimeter
and connected through a dedicated, low-latency network.
Mission100 Azure Data Engineer Course By Deepak Goyal
https://adeus.azurelib.com
Email at: admin@azurelib.com
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/
●
Purpose: Regions allow you to deploy your resources closer to your users to reduce latency,
meet data residency requirements, and ensure compliance with local regulations.
●
Examples of Azure Regions:
o East US (Virginia)
o West Europe (Netherlands)
o Southeast Asia (Singapore)
o Australia East (New South Wales)
o Central India (Pune)
Azure offers over 60 regions globally, making it one of the most extensive cloud networks in the
world.
Availability Zones:
An Availability Zone is a physically separate location within an Azure region. Each Availability Zone
consists of one or more data centers equipped with independent power, cooling, and networking. By
deploying resources across multiple Availability Zones, you can ensure high availability and fault
tolerance for your applications.
●
●
●
Purpose: Availability Zones are designed to protect your applications and data from data
center failures within a region. If one zone goes down, the other zones in the region continue to
operate, minimizing downtime.
Structure: Typically, an Azure region will have three or more Availability Zones. These zones
are interconnected with high-speed, private fiber-optic networks.
Examples of Services Using Availability Zones:
o Virtual Machines: You can deploy VMs across multiple zones to ensure that if one
zone fails, your application remains available in the other zones.
o Managed Disks: Zone-redundant storage (ZRS) replicates your data across multiple
zones to ensure durability.
o Load Balancers: Azure Load Balancer can distribute traffic across VMs in different
zones, providing high availability.
Mission100 Azure Data Engineer Course By Deepak Goyal
https://adeus.azurelib.com
Email at: admin@azurelib.com
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/
What are Resource Groups in Azure?
A Resource Group in Azure is a logical container that holds related Azure resources. These resources
might include virtual machines, storage accounts, virtual networks, databases, and more. Resource
groups help you manage and organize your Azure resources effectively.
Key Features of Resource Groups:
1. Logical Organization:
o Resource groups allow you to group together resources that share the same lifecycle or
purpose. For example, all resources related to a specific application or project can be placed in
the same resource group, making it easier to manage, monitor, and deploy.
2. Resource Management:
o You can perform management tasks on the resource group as a whole, such as applying
policies, setting access controls, or deleting the entire group. This simplifies operations,
especially when dealing with multiple resources.
Mission100 Azure Data Engineer Course By Deepak Goyal
https://adeus.azurelib.com
Email at: admin@azurelib.com
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/
3. Access Control:
o Azure Role-Based Access Control (RBAC) can be applied at the resource group level, allowing
you to define who has permission to manage the resources within the group. This provides a
centralized way to manage permissions.
4. Cost Management:
o Since all resources in a resource group can be tracked together, it becomes easier to monitor
and manage costs associated with a specific application or project.
5. Lifecycle Management:
o Resources within a resource group typically share the same lifecycle, which means you can
deploy, update, or delete all the resources together. For instance, when a project ends, you
can delete the resource group, and all its resources will be deleted as well.
6. Dependency Management:
o When deploying resources via templates (like Azure Resource Manager (ARM) templates), the
resources in a resource group can be deployed in a specific order, with dependencies defined
between them.
Important Considerations:
●
●
●
●
Location: The resource group itself has a location (region), which determines where its
metadata is stored. However, the resources within the group can be in different regions. It's
generally recommended to keep resources in the same region for performance reasons.
Naming: Resource groups should be named in a way that reflects their purpose, such as RGProduction-ApplicationName or RG-Dev-ProjectX, to make them easily identifiable.
Scope: A resource can only belong to one resource group at a time, but you can move resources
between groups if needed.
Limits: While there are limits on the number of resource groups and resources per group, these
limits are generally high and sufficient for most use cases.
Example Use Cases:
●
●
Application Lifecycle: You might create a resource group for a web application that includes
resources like a web server, database, and storage account. When you update the application,
you can update all related resources together.
Development and Testing: For a development environment, you could create a resource group
that contains all the necessary resources (VMs, databases, etc.) for testing purposes. Once
testing is complete, you can delete the resource group to clean up all associated resources.
●
Cost Management: By grouping all resources for a specific department or project, you can
easily track the costs associated with that group and manage budgets accordingly.
In summary, resource groups in Azure are a fundamental feature for organizing, managing, and
controlling access to your cloud resources. They help streamline operations and provide a structured
way to manage your Azure environment.
What is a Storage Account in Azure?
An Azure Storage Account is a service provided by Microsoft Azure that allows you to store and
manage various types of data in the cloud. It acts as a container that provides a namespace for Azure
Storage services and gives you access to Azure's scalable, secure, and durable storage options.
Mission100 Azure Data Engineer Course By Deepak Goyal
https://adeus.azurelib.com
Email at: admin@azurelib.com
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/
Key Features of Azure Storage Accounts:
1. Data Services: An Azure Storage Account can hold several types of data services, including:
o
o
o
o
o
Blob Storage: For storing unstructured data such as images, videos, and documents. It’s
particularly useful for storing files that are accessed over the internet.
File Storage: Provides fully managed file shares in the cloud that are accessible via the Server
Message Block (SMB) protocol. It’s like having a network file share in the cloud.
Queue Storage: For storing large numbers of messages that can be accessed from anywhere
via authenticated calls. It’s often used for task scheduling and asynchronous processing.
Table Storage: A NoSQL key-value store for structured data. It’s designed to handle large
amounts of data with simple query operations.
Disk Storage: Provides persistent, durable, and high-performance storage for Azure Virtual
Machines (VMs) in the form of managed disks.
2. Scalability: Azure Storage Accounts are designed to scale automatically to meet your needs.
Whether you need to store a few gigabytes or several petabytes, Azure Storage can handle the
load.
3. Durability and Redundancy: Azure provides multiple options for ensuring that your data is
highly available and durable:
o
o
o
o
Locally-Redundant Storage (LRS): Copies your data three times within a single data center in a
region.
Zone-Redundant Storage (ZRS): Copies your data synchronously across three Azure availability
zones in a region.
Geo-Redundant Storage (GRS): Copies your data to a secondary region, hundreds of miles
away from the primary location, for disaster recovery.
Read-Access Geo-Redundant Storage (RA-GRS): Provides read access to the data in the
secondary region, in addition to the features of GRS.
Mission100 Azure Data Engineer Course By Deepak Goyal
https://adeus.azurelib.com
Email at: admin@azurelib.com
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/
4. Security:
o Encryption: All data stored in Azure Storage is encrypted by default. Azure uses both serverside and client-side encryption to protect your data.
o Access Control: You can control access to your storage account using Azure Active Directory
(AAD) and shared access signatures (SAS), allowing you to define who can access your data and
for how long.
o Firewall and Virtual Network Integration: You can restrict access to your storage account to
specific networks or IP ranges for additional security.
5. Cost-Effectiveness: Azure Storage offers different pricing tiers based on the type of storage,
redundancy option, and access patterns (hot, cool, or archive tiers). This flexibility allows you
to optimize costs based on your storage needs.
6. Access Methods: Data in an Azure Storage Account can be accessed via several methods:
o
o
o
o
Azure Portal: A graphical user interface for managing your storage account and its contents.
Azure CLI/PowerShell: Command-line tools for automating and managing your storage
resources.
Azure SDKs: Software Development Kits (SDKs) for various programming languages (like .NET,
Java, Python) that allow you to interact with Azure Storage programmatically.
REST API: Provides direct access to Azure Storage services for custom integration.
Types of Storage Accounts:
●
General-purpose v2 (GPv2): Supports all the latest features and is the recommended type for most
scenarios. It provides access to all Azure Storage services.
●
Blob Storage Account: Specifically optimized for blob storage, with tiering options (hot, cool, and
archive) to optimize cost based on access frequency.
●
File Storage Account: Optimized for Azure Files, supporting premium file shares.
●
BlockBlobStorage Account: Designed for workloads with high transaction rates or that require
consistent, low-latency data access.
Use Cases:
●
Storing Files and Documents: Store and access files, images, videos, and other unstructured data.
●
Backup and Restore: Use Azure Storage as a backup destination for on-premises or cloud-based
systems.
●
Disaster Recovery: Ensure data availability with geo-redundant storage options.
●
Big Data Analytics: Store large datasets for analysis with tools like Azure Data Lake Analytics or Azure
Synapse Analytics.
●
Web and Mobile Applications: Host and serve content such as web pages, videos, or static files
directly from Azure Storage.
How to Create a Storage Account:
1. Sign in to the Azure Portal.
2. Navigate to 'Storage Accounts' and click on 'Create'.
Mission100 Azure Data Engineer Course By Deepak Goyal
https://adeus.azurelib.com
Email at: admin@azurelib.com
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/
3. Configure your storage account settings, such as choosing the resource group, name, region, and
performance tier.
4. Select the desired redundancy option (LRS, ZRS, GRS, or RA-GRS).
5. Review and create the storage account.
Once created, you can start using your storage account to store and manage data in the cloud.
To keep history you can enable disable the soft delete feature.
Encryption in Azure Storage Account
Azure Storage provides robust encryption features to protect your data at rest. This ensures that all
your data is automatically encrypted before it is stored and decrypted before it is retrieved, without
requiring any additional configuration or management from you.
Microsoft-managed keys: By default, Azure manages the encryption keys for you, which simplifies
key management and ensures that your data is protected using Microsoft-managed keys.
Customer-managed keys (CMK): If you prefer more control, you can manage your own
encryption keys using Azure Key Vault. This gives you full control over the key lifecycle, including
rotation and revocation. Customer-managed keys can also be used for auditing purposes, as you have
visibility into the key usage.
Mission100 Azure Data Engineer Course By Deepak Goyal
https://adeus.azurelib.com
Email at: admin@azurelib.com
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/
Access Tiers in Azure Storage Account
Azure Storage offers different access tiers to help you optimize the cost of storing data based on how
frequently the data is accessed. These access tiers allow you to store your data in a way that aligns with
its usage patterns, enabling you to balance cost and performance.
Here’s a breakdown of the Azure Storage access tiers:
1. Hot Tier:
●
Description: The Hot tier is designed for data that is accessed frequently. It offers the lowest access
latency and the highest throughput, making it ideal for data that needs to be accessed and processed
regularly.
●
Use Cases:
o Active datasets, such as files and databases that are accessed frequently.
o Content that is frequently updated or queried, like transaction logs.
o Data for applications that require low latency access, such as web and mobile apps.
●
Cost: Higher storage costs compared to the Cool and Archive tiers, but lower access costs.
2. Cool Tier:
●
Description: The Cool tier is optimized for data that is infrequently accessed but needs to be stored for
at least 30 days. It offers lower storage costs than the Hot tier but higher access costs.
●
Use Cases:
o Data that is not accessed frequently but still needs to be available for occasional access, like
backups, archived data, or media content that is accessed seasonally.
Mission100 Azure Data Engineer Course By Deepak Goyal
https://adeus.azurelib.com
Email at: admin@azurelib.com
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/
o Data that is stored for compliance or business continuity purposes.
●
Cost: Lower storage costs than the Hot tier, but higher costs for data access and retrieval.
3. Archive Tier:
●
Description: The Archive tier is intended for data that is rarely accessed and can tolerate higher
retrieval times. This tier offers the lowest storage costs but the highest costs and latency for data
retrieval.
●
Use Cases:
o Long-term archival data, such as compliance records, legal documents, or historical data that
may be accessed once in a while.
o Data that needs to be kept for extended periods but is not likely to be needed frequently.
●
Cost: The lowest storage cost among all tiers, but the highest cost and latency for data access. Data in
the Archive tier must be rehydrated (moved to the Hot or Cool tier) before it can be accessed.
ADLS vs Blob Storage
Feature
Purpose
Data Hierarchy
Performance
Security
Access Protocols
Analytics Integration
Data Types
Cost Structure
Use Cases
Azure Data Lake Storage (ADLS)
Optimized for big data analytics
workloads
Supports hierarchical namespace (folders
and subfolders)
Designed for high throughput and
optimized performance for big data
scenarios
Granular access control with Azure Active
Directory (AAD) integration
Supports Hadoop Distributed File System
(HDFS) and POSIX-compliant ACLs
Built-in support for analytics engines like
Azure HDInsight, Azure Databricks, and
Azure Synapse
Optimized for structured, semistructured, and unstructured data
Higher cost, especially for hierarchical
namespace features
Big data analytics, complex data
processing, enterprise-level data lakes
Azure Blob Storage
General-purpose object storage for
unstructured data
Flat namespace; no native support for
directories
High performance, but not specifically
optimized for big data
Role-based access control (RBAC) and
shared access signatures (SAS)
Standard REST APIs, SDKs, and tools like
Azure Storage Explorer
Limited integration with big data analytics
engines
Best for unstructured data (images, videos,
documents, etc.)
Generally lower cost, suitable for simple
storage needs
Backup, archiving, web applications, media
storage
Mission100 Azure Data Engineer Course By Deepak Goyal
https://adeus.azurelib.com
Email at: admin@azurelib.com
Ask Queries here: https://www.linkedin.com/in/deepak-goyal-93805a17/