1.11

Comments

Post a Comment

Popular posts from this blog

🌐Filtering and Copying Files Dynamically in Azure Data Factory (ADF)

Image
  Filtering and Copying Files Dynamically in Azure Data Factory (ADF) Introduction In Azure Data Factory (ADF) , automating file processing is a common use case. This guide demonstrates how to: Retrieve file metadata (including filenames inside a folder). Filter files based on today’s date. Apply advanced filter conditions such as contains() , startswith() , and endswith() . Copy only filtered files dynamically using ForEach and Copy Data activities. Step 1: Set Variable for Today's Date The first step is to create a Set Variable activity to store today’s date in ddMMyyyy format. Activity: Set Variable Variable Name: dt Value: @formatDateTime(utcNow(), 'ddMMyyyy') Explanation utcNow() fetches the current UTC time. formatDateTime(utcNow(), 'ddMMyyyy') converts it into a day-month-year format. Example Output: { "name": "dt", "value": "28022025" } Step 2: Get Metadata of Folder Contents ...
Read more

🔥Apache Spark Architecture with RDD & DAG

  Apache Spark Architecture with RDD & DAG Apache Spark follows a master-slave architecture, designed for fast, distributed data processing. It consists of three main components: Driver Node (Master) Cluster Manager Executors (Workers) Additionally, two important internal components play a crucial role in execution: RDD (Resilient Distributed Dataset) DAG (Directed Acyclic Graph) 1. Driver Node (Master) The Driver Node is responsible for coordinating and executing a Spark application. Responsibilities of the Driver: Starts a SparkSession (entry point of Spark). Divides the program into tasks and schedules them. Sends tasks to executors for execution. Monitors task execution and collects results. Working of the Driver Node: User submits a Spark job using spark-submit . Driver requests resources from the Cluster Manager. The job is converted into a DAG (Directed Acyclic Graph). The DAG is split into Stages, and each Stage contains Tasks. The Cluster Man...
Read more

🚀End-to-End Data Flow Pipeline using Apache NiFi, Kafka-Spark Structured Streaming, and Snowflake

🚀 End-to-End Data Flow Pipeline using Apache NiFi, Kafka-Spark Structured Streaming, and Snowflake 💬 Personal Note: 🌟 I was unwell for a while, which caused a pause in my blogging journey. However, I’m feeling much better now and back on track. From now on, I will be posting blogs consistently. Thank you all for your support! 🙏✨ 🔄 Flow of Data in this Pipeline: Server (https://randomuser.me/api/) ↓ (REST API) Apache NiFi (InvokeHTTP Processor) ↓ Kafka (Kafka Brokers - PublishKafkaRecord_2_6 Processor) ↓ Consumer (Kafka Structured Streaming - Spark) ↓ Snowflake (Data Storage) 🌐 Project Overview: This project demonstrates a real-time data streaming pipeline that integrates data collection, processing, and storage using industry-standard tools: 🌐 Data Collection: Fetched from randomuser.me using Apache NiFi’s InvokeHTTP processor. 🏭 Streaming Data: Pushed into Kafka using PublishKafkaRecord_2_6 . ⚡ Data Processing: Apache Spark Structured Streaming co...
Read more