Mastering PySpark: Top 100 Functions for Data Engineers

Apache Spark has revolutionized big data processing with its distributed computing capabilities. PySpark, the Python API for Spark, is widely used for data engineering, analysis, and machine learning. This blog will cover the top 100 PySpark functions every data engineer should know, along with a practice dataset to apply these concepts.

Index of PySpark Functions

1. Basic DataFrame Operations

  • Creating an Empty DataFrame
  • Converting RDD to DataFrame
  • Converting DataFrame to Pandas
  • Displaying Data (show())
  • Defining Schema (StructType & StructField)

2. Column & Row Operations

  • Column Class
  • Selecting Columns (select())
  • Collecting Data (collect())
  • Adding & Modifying Columns (withColumn())
  • Renaming Columns (withColumnRenamed())
  • Filtering Data (where() & filter())
  • Dropping Columns & Duplicates (drop(), dropDuplicates())
  • Sorting Data (orderBy(), sort())
  • Grouping Data (groupBy())
  • Joining DataFrames (join())
  • Union Operations (union(), unionAll(), unionByName())

3. User-Defined Functions (UDFs) & Transformations

  • Creating UDFs
  • Transforming Data (transform(), apply())
  • Mapping Functions (map(), flatMap())
  • Iterating Over Rows (foreach())

4. Sampling & Null Handling

  • Random Sampling (sample(), sampleBy())
  • Handling Missing Values (fillna(), fill())

5. Pivoting & Partitioning

  • Pivoting Data (pivot())
  • Partitioning Data (partitionBy())

6. Map & Dictionary Operations

  • Using MapType for Key-Value Pairs

7. Inbuilt Functions

  • col(), lit(), when(), isNull(), isNotNull(), between(), like(), rlike(), alias(), cast(), expr()

8. Aggregate Functions

  • Count (count(), countDistinct(), approx_count_distinct())
  • Sum (sum(), sumDistinct())
  • Average (avg())
  • Min & Max (min(), max())
  • First & Last (first(), last())
  • Standard Deviation & Variance (stddev(), variance())
  • Collecting Data into Lists & Sets (collect_list(), collect_set())
  • Correlation & Covariance (corr(), covar_pop(), covar_samp())
  • Statistical Measures (kurtosis(), skewness(), approxQuantile())

9. Window Functions

  • Rank-Based Functions (rank(), dense_rank(), ntile(), row_number())
  • Lead & Lag Functions (lead(), lag())
  • Percentile Functions (percent_rank())
  • Window Definitions (window())

10. String Functions

  • Concatenation (concat(), concat_ws())
  • String Length & Case (length(), lower(), upper())
  • Trimming & Padding (trim(), ltrim(), rtrim(), pad(), rpad(), lpad())
  • String Manipulation (reverse(), substring(), substr(), split(), regexp_extract(), regexp_replace())
  • Finding & Replacing Characters (instr(), translate())
  • Encoding & Decoding (encode(), decode())
  • Formatting & Capitalization (format_number(), initcap())

11. Date & Time Functions

  • Current Date & Time (current_date(), current_timestamp())
  • Date Arithmetic (date_add(), date_sub())
  • Extracting Date Parts (year(), month(), dayofmonth(), dayofweek(), dayofyear(), hour(), minute(), second())
  • Formatting Dates (date_format(), trunc(), date_trunc())
  • Working with Timestamps (to_date(), to_timestamp(), from_unixtime(), unix_timestamp())
  • Month & Week Operations (add_months(), months_between(), weekofyear())
  • Finding Specific Days (last_day(), next_day())

12. Array Functions

  • Creating & Manipulating Arrays (array(), array_contains(), array_distinct(), array_intersect(), array_union())

Practice Dataset

To help you get hands-on experience with these functions, here is a sample dataset:

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, IntegerType, StringType, DoubleType

data = [(i, f"Person_{i}", 20 + (i % 30), ["Engineer", "Doctor", "Teacher", "Artist", "Scientist"][i % 5], 3000.0 + (i * 100)) for i in range(1, 51)]

schema = StructType([
    StructField("id", IntegerType(), True),
    StructField("name", StringType(), True),
    StructField("age", IntegerType(), True),
    StructField("profession", StringType(), True),
    StructField("salary", DoubleType(), True)
])

spark = SparkSession.builder.appName("PySpark Practice").getOrCreate()
df = spark.createDataFrame(data, schema)
df.show()

Conclusion

Mastering these PySpark functions will significantly improve your efficiency in handling large datasets. This guide, along with the practice dataset, will help you build confidence and proficiency in PySpark. Try implementing these functions and experiment with different scenarios!

Happy coding!

Comments

Post a Comment

Popular posts from this blog

AWS Athena, AWS Lambda, AWS Glue, and Amazon S3 – Detailed Explanation

AWS Athena, AWS Lambda, AWS Glue, and Amazon S3 – Detailed Explanation AWS provides various serverless services to handle data processing, querying, and storage efficiently. Let's break down each of these services with a structured approach. 1. AWS Athena Definition: AWS Athena is an interactive query service that allows users to analyze data directly in Amazon S3 using standard SQL. It is a serverless service, meaning you don’t need to manage infrastructure. Purpose: Allows querying structured and semi-structured data stored in Amazon S3. Uses Presto as its underlying engine to execute SQL queries efficiently. Enables quick ad-hoc analysis without setting up a database or cluster. Advantages: ✅ Serverless – No need to provision or manage servers. ✅ Cost-effective – Pay only for the amount of data scanned. ✅ Supports various formats – Works with CSV, JSON, ORC, Parquet, Avro stored in S3. ✅ Integrates with BI tools – Can be used with Power BI, Tableau, AWS QuickSight ...
Read more

Kafka Integrated with Spark Structured Streaming

Image
Kafka Integrated with Spark Structured Streaming 🚀 Apache Kafka Apache Kafka is an open-source data streaming platform that stores, processes, and analyzes large amounts of real-time data. It is used to build real-time data pipelines and applications that can adapt to data streams. 🔥 Event Streaming Event streaming is the digital equivalent of the human body's central nervous system. It is the technological foundation for the "always-on" world, where businesses are increasingly software-defined and automated. Technically, event streaming involves capturing data in real-time from event sources like databases, sensors, mobile devices, cloud services, and applications. It then stores, processes, and routes these events to different destinations as needed, ensuring continuous data flow. ⚡ Kafka as an Event Streaming Platform Kafka combines three key capabilities: 📤 Publishing and subscribing to streams of events. 💾 Storing streams of events durably and reliably. ⏳ Proc...
Read more

Azure Data Factory: Copying Data from ADLS to MSSQL

🚀 Azure Data Factory: Copying Data from ADLS to MSSQL 🔑 Key Vault & Initial Setup To copy data from Azure Data Lake Storage (ADLS) to Microsoft SQL Server (MSSQL) using Azure Data Factory (ADF), follow these structured steps. 🌐 Step 1: Creating Azure Data Factory (ADF) & Linking Services 🏗️ Creating an Azure Data Factory 🔍 Search for "Data Factory" in the Azure portal. ➕ Click "Create Data Factory". 📦 Select the Free Tier. 📂 Attach the resource group: rock2025 . 🏷️ Provide a name, e.g., learnazure (lowercase, no special characters or numbers). ✅ Click "Review + Create" and proceed with deployment. 🔗 Linking Azure Data Lake Storage to Data Factory Navigate to Azure Data Factory → Launch ADF Studio . In the Manage section: Click Linked Services → + New Linked Service . Select Azure Data Lake Storage Gen2 . Provide a name, e.g., LS_FEB13 . Choose the subscription (Free Tier). Select the Storage Account (e.g., divyabucket02...
Read more