Azure Key Vault: Securely Managing Secrets in ADF Pipelines

 

🔐 Azure Key Vault: Securely Managing Secrets in ADF Pipelines

🏗️ Introduction

Azure Key Vault (AKV) is a cloud service that securely stores secrets like API keys, passwords, and database connection strings. Integrating Azure Data Factory (ADF) pipelines with Azure Key Vault ensures enhanced security by managing sensitive information centrally.

This guide provides step-by-step instructions to:

  • Create an Azure Key Vault.
  • Store secrets securely.
  • Integrate Key Vault with ADF pipelines.
  • Retrieve credentials from Key Vault within ADF.

🏛️ Step 1: Creating Azure Key Vault

  1. 🔍 Search for Azure Key Vault in the Azure portal.
  2. ➕ Click Create Key Vault.
  3. Configure the settings:
    • Subscription: Free Tier
    • Resource Group:divyalearn_az
    • Key Vault Name: mysecurevault22
    • Region: Central India
    • Pricing Tier: Standard
  4. ✅ Click Next
  5. Access configures:
    • Permission Model: Vault Access Policy
    • Access Policy: + create
    • Create An Access Policy: Configure from Template (key secret certificate rotation permission) 
  6. principal:
    • Select principal: Rock2025(azure data factory)
    • Next Next other things are optional
    • Review and create => create
    • It is like AWS IAM, but IAM roles is dead easy but azure not

🔑 Step 2: Adding Secrets to Key Vault

  1. Navigate to mysecurevault22 "go to resources" in Azure Key Vault.
  2. Click Object=> Secrets → + Generate/Import.
  3. Configure the secret:
    • Name: MSSQLConnection
    • Password: Mypassword.02
    • Content value(optional):Mypassword
    • Value: jdbc:sqlserver://divyamysqldb.database.windows.net:1433;database=mysql13febdb;user=divya;password=YourSecurePassword;
  4. ✅ Click Create.

🔹 Note: Avoid hardcoding sensitive information. Always store connection strings and passwords in Key Vault.

🔗 Step 3: Assigning Permissions to ADF

  1. Navigate to mysecurevault → Access Policies.
  2. Click + Add Access Policy.
  3. Set permissions:
    • Secret Permissions: Get, List
  4. Assign Azure Data Factory (ADF) as the Principal.
  5. ✅ Click Save.

⚙️ Step 4: Integrating Key Vault with Azure Data Factory

🔄 Linking Key Vault to ADF

  1. Open Azure Data Factory → Manage.
  2. Click Linked Services → + New Linked Service.
  3. Select Azure Key Vault.
  4. Configure settings:
    • Name: LS_Acess_KV
    • Azure Subscription: Free Tier
    • Key Vault Name: mysecurevault22
    • Authentication Type: System-assigned managed identity
    • Test Connection
    • ✅ Click Create.

🔍 Retrieving Secrets in ADF

  1. Navigate to Linked Services → LS_acess_kv.
  2. Modify Authentication to Azure Key Vault.
  3. In Connection String, select Use Key Vault.
  4. Choose Secret Name: MSSQLConnection.
  5. ✅ Click Test Connection → Publish.

🏗️ Step 5: Using Key Vault in ADF Pipeline

  1. Navigate to ADF → Author → Pipelines → + New.
  2. Drag & drop Copy Activity.
  3. Configure Source:
    • Dataset: ds_asldata
  4. Configure Sink:
    • Dataset: (Create New) → SQL Server
    • Linked Service: LS_SqlServer1 (Key Vault Secured)
    • Table Name: asltab
  5. ✅ Save and Publish the pipeline.
  6. Click Trigger Now to execute.

🎯 Conclusion

By integrating Azure Key Vault with Azure Data Factory, we have securely managed credentials and ensured data security. This approach eliminates the need to store sensitive information within the pipeline, enhancing security and compliance.

🚀 Stay Connected for more Azure AWS blog "Happy Valentine Day to All"😊

Comments

Post a Comment

Popular posts from this blog

AWS Athena, AWS Lambda, AWS Glue, and Amazon S3 – Detailed Explanation

AWS Athena, AWS Lambda, AWS Glue, and Amazon S3 – Detailed Explanation AWS provides various serverless services to handle data processing, querying, and storage efficiently. Let's break down each of these services with a structured approach. 1. AWS Athena Definition: AWS Athena is an interactive query service that allows users to analyze data directly in Amazon S3 using standard SQL. It is a serverless service, meaning you don’t need to manage infrastructure. Purpose: Allows querying structured and semi-structured data stored in Amazon S3. Uses Presto as its underlying engine to execute SQL queries efficiently. Enables quick ad-hoc analysis without setting up a database or cluster. Advantages: ✅ Serverless – No need to provision or manage servers. ✅ Cost-effective – Pay only for the amount of data scanned. ✅ Supports various formats – Works with CSV, JSON, ORC, Parquet, Avro stored in S3. ✅ Integrates with BI tools – Can be used with Power BI, Tableau, AWS QuickSight ...
Read more

Kafka Integrated with Spark Structured Streaming

Image
Kafka Integrated with Spark Structured Streaming 🚀 Apache Kafka Apache Kafka is an open-source data streaming platform that stores, processes, and analyzes large amounts of real-time data. It is used to build real-time data pipelines and applications that can adapt to data streams. 🔥 Event Streaming Event streaming is the digital equivalent of the human body's central nervous system. It is the technological foundation for the "always-on" world, where businesses are increasingly software-defined and automated. Technically, event streaming involves capturing data in real-time from event sources like databases, sensors, mobile devices, cloud services, and applications. It then stores, processes, and routes these events to different destinations as needed, ensuring continuous data flow. ⚡ Kafka as an Event Streaming Platform Kafka combines three key capabilities: 📤 Publishing and subscribing to streams of events. 💾 Storing streams of events durably and reliably. ⏳ Proc...
Read more

Azure Data Factory: Copying Data from ADLS to MSSQL

🚀 Azure Data Factory: Copying Data from ADLS to MSSQL 🔑 Key Vault & Initial Setup To copy data from Azure Data Lake Storage (ADLS) to Microsoft SQL Server (MSSQL) using Azure Data Factory (ADF), follow these structured steps. 🌐 Step 1: Creating Azure Data Factory (ADF) & Linking Services 🏗️ Creating an Azure Data Factory 🔍 Search for "Data Factory" in the Azure portal. ➕ Click "Create Data Factory". 📦 Select the Free Tier. 📂 Attach the resource group: rock2025 . 🏷️ Provide a name, e.g., learnazure (lowercase, no special characters or numbers). ✅ Click "Review + Create" and proceed with deployment. 🔗 Linking Azure Data Lake Storage to Data Factory Navigate to Azure Data Factory → Launch ADF Studio . In the Manage section: Click Linked Services → + New Linked Service . Select Azure Data Lake Storage Gen2 . Provide a name, e.g., LS_FEB13 . Choose the subscription (Free Tier). Select the Storage Account (e.g., divyabucket02...
Read more