SQL & PySpark Equivalence: A Comprehensive Guide
Structured Query Language (SQL) and PySpark are both powerful tools for handling large-scale
data processing. SQL is widely used for querying and manipulating structured data in relational
databases, while PySpark, built on Apache Spark, is designed for distributed computing and big
data analytics.
Understanding the equivalence between SQL and PySpark is crucial for data engineers and analysts
working in hybrid environments where both technologies are used. SQL provides a declarative way
to interact with data, whereas PySpark leverages Resilient Distributed Datasets (RDDs) and
DataFrames to perform transformations and actions efficiently across distributed systems.
This guide presents a side-by-side comparison of key SQL operations and their equivalent PySpark
implementations. It covers data selection, filtering, aggregations, joins, window functions,
performance optimizations, and more, helping professionals seamlessly transition between the
two technologies.
1.Data Types
SQL Data Type
PySpark Equivalent
INT
IntegerType()
BIGINT
LongType()
FLOAT
FloatType()
DOUBLE
DoubleType()
CHAR(n) / VARCHAR(n) StringType()
DATE
DateType()
TIMESTAMP
TimestampType()
2.Database & Table Operations
Concept
Create
Database
Use
Database
Drop
Database
Show
Databases
SQL Query
CREATE DATABASE
db_name;
PySpark Equivalent
spark.sql("CREATE DATABASE db_name")
USE db_name;
spark.catalog.setCurrentDatabase("db_n
ame")
DROP DATABASE
db_name;
spark.sql("DROP DATABASE db_name")
SHOW DATABASES;
spark.sql("SHOW DATABASES").show()
CREATE TABLE
df.write.format("parquet").saveAsTable
table_name (col1
("table_name")
INT, col2 STRING);
DROP TABLE
spark.sql("DROP TABLE IF EXISTS
Drop Table table_name;
table_name")
TRUNCATE TABLE
Truncate
spark.sql("TRUNCATE TABLE table_name")
table_name;
Table
Create
Table
Concept
Describe
Table
Show
Tables
SQL Query
DESCRIBE TABLE
table_name;
SHOW TABLES;
PySpark Equivalent
df.printSchema()
spark.sql("SHOW TABLES").show()
3. Table Alterations
Concept
Add
Column
Rename
Column
Drop
Column
SQL Query
PySpark Equivalent
ALTER TABLE table_name
df.withColumn("col3",
ADD COLUMN col3 STRING;
lit(None).cast("string"))
ALTER TABLE table_name
df.withColumnRenamed("old_name"
RENAME COLUMN old_name TO
, "new_name")
new_name;
ALTER TABLE table_name
df.drop("col3")
DROP COLUMN col3;
4. Partitioning & Bucketing
Concept
Create
Partitioned
Table
Insert into
Partitioned
Table
Create
Bucketed
Table
SQL Query
PySpark Equivalent
CREATE TABLE table_name
df.write.partitionBy("col3").fo
(col1 INT, col2 STRING)
rmat("parquet").saveAsTable("ta
PARTITIONED BY (col3
ble_name")
STRING);
INSERT INTO table_name
PARTITION
df.write.mode("append").partiti
(col3='value') SELECT
onBy("col3").saveAsTable("table
col1, col2 FROM
_name")
source_table;
CREATE TABLE table_name
df.write.bucketBy(10,
(col1 INT, col2 STRING)
"col1").saveAsTable("table_name
CLUSTERED BY (col1)
")
INTO 10 BUCKETS;
5. Views (Temporary & Permanent)
Concept
SQL Query
CREATE VIEW view_name
Create View AS SELECT * FROM
table_name;
Drop View
DROP VIEW view_name;
PySpark Equivalent
df.createOrReplaceTempView("view
_name")
spark.sql("DROP VIEW IF EXISTS
view_name")
CREATE GLOBAL VIEW
Create
view_name AS SELECT *
Global View FROM table_name;
df.createGlobalTempView("view_na
me")
Show Views SHOW VIEWS;
spark.sql("SHOW VIEWS").show()
6. Schema Management
Concept
Define
Schema
Manually
Check
Schema
Change
Column
Data Type
SQL Query
PySpark Equivalent
from pyspark.sql.types import
StructType, StructField, IntegerType,
StringType, DateType
CREATE TABLE
schema =
table_name (col1
StructType([StructField("col1",
INT, col2 STRING,
IntegerType(), True),
col3 DATE);
StructField("col2", StringType(),
True), StructField("col3", DateType(),
True)])
DESCRIBE TABLE
df.printSchema()
table_name;
ALTER TABLE
table_name ALTER df.withColumn("col1",
COLUMN col1 TYPE col("col1").cast("bigint"))
BIGINT;
7. File-Based Table Operations
Concept
Save as
Parquet
SQL Query
N/A (Implicit in Hive)
PySpark Equivalent
df.write.format("parquet").save("
path/to/parquet")
CREATE TABLE table_name
df.write.format("delta").save("pa
Save as
USING DELTA LOCATION
th/to/delta")
Delta Table 'path';
df.write.format("csv").option("he
Save as CSV N/A
ader", True).save("path/to/csv")
df.write.format("json").save("pat
Save as
N/A
h/to/json")
JSON
df.write.format("orc").save("path
Save as
N/A
/to/orc")
ORC
8.Basic SELECT Queries
Concept
Select specific
columns
Select all
columns
Distinct values
WHERE
condition
ORDER BY
SQL Query
PySpark Equivalent
SELECT column1, column2 FROM df.select("column1",
table;
"column2")
SELECT * FROM table;
df.select("*")
SELECT DISTINCT column FROM df.select("column").disti
table;
nct()
SELECT * FROM table WHERE
df.filter(col("column")
column = 'value';
== 'value')
SELECT * FROM table ORDER BY
df.sort("column")
column;
Concept
LIMIT rows
COUNT rows
SQL Query
PySpark Equivalent
SELECT * FROM table LIMIT n; df.limit(n)
SELECT COUNT(*) FROM table; df.count()
9. Aggregate Functions
Concept
SQL Query
PySpark Equivalent
SELECT SUM(column) FROM table; df.agg(sum("column"))
SUM
SELECT AVG(column) FROM table; df.agg(avg("column"))
AVG
SELECT MAX(column) FROM table; df.agg(max("column"))
MAX
SELECT MIN(column) FROM table; df.agg(min("column"))
MIN
10. String Functions
Concept
String Length
Convert to
Uppercase
Convert to
Lowercase
Concatenate
Strings
Trim String
Substring
SQL Query
SELECT LEN(column) FROM
table;
SELECT UPPER(column) FROM
table;
SELECT LOWER(column) FROM
table;
SELECT CONCAT(string1,
string2) FROM table;
SELECT TRIM(column) FROM
table;
SELECT SUBSTRING(column,
start, length) FROM table;
PySpark Equivalent
df.select(length(col("colu
mn")))
df.select(upper(col("colum
n")))
df.select(lower(col("colum
n")))
df.select(concat(col("stri
ng1"), col("string2")))
df.select(trim(col("column
")))
df.select(substring(col("c
olumn"), start, length))
11. Date & Time Functions
Concept
Current Date
Current
Timestamp
CAST /
CONVERT
SQL Query
SELECT CURDATE();
SELECT NOW();
SELECT CAST(column AS
datatype) FROM table;
PySpark Equivalent
df.select(current_date())
df.select(current_timestam
p())
df.select(col("column").ca
st("datatype"))
12. Conditional Logic
Concept
SQL Query
SELECT IF(condition,
IF (Conditional value1, value2) FROM
Logic)
table;
SELECT
COALESCE
COALESCE(column1,
PySpark Equivalent
df.select(when(condition,
value1).otherwise(value2))
df.select(coalesce(col("column
1"), col("column2"),
Concept
SQL Query
column2, column3)
FROM table;
PySpark Equivalent
col("column3")))
13. Join, Grouping & Pivoting
Concept
JOIN
GROUP
BY
PIVOT
SQL Query
SELECT * FROM table1 JOIN
table2 ON table1.column =
table2.column;
SELECT column,
agg_function(column) FROM
table GROUP BY column;
PIVOT (agg_function(column)
FOR pivot_column IN
(values));
PySpark Equivalent
df1.join(df2, "column")
df.groupBy("column").agg(agg
_function("column"))
df.groupBy("pivot_column").p
ivot("column").agg(agg_funct
ion)
14. Logical Operators
Concept
AND / OR
IS NULL / IS
NOT NULL
LIKE
BETWEEN
SQL Query
SELECT * FROM table
WHERE column1 = value
AND column2 > value;
SELECT * FROM table
WHERE column IS NULL;
SELECT * FROM table
WHERE column LIKE
'value%';
SELECT * FROM table
WHERE column BETWEEN
value1 AND value2;
PySpark Equivalent
df.filter((col("column1") ==
value) & (col("column2") >
value))
df.filter(col("column").isNul
l())
df.filter(col("column").like(
"value%"))
df.filter((col("column") >=
value1) & (col("column") <=
value2))
15. Set Operations
Concept
UNION
UNION
ALL
SQL Query
PySpark Equivalent
SELECT column FROM table1 UNION
df1.union(df2).select(
SELECT column FROM table2;
"column")
SELECT column FROM table1 UNION ALL df1.unionAll(df2).sele
SELECT column FROM table2;
ct("column")
16. Window Functions
Concept
RANK /
DENSE_RANK /
ROW_NUMBER
SQL Query
SELECT column,
RANK() OVER (ORDER
BY column) FROM
PySpark Equivalent
df.select("column",
rank().over(Window.orderBy("co
lumn")).alias("rank"))
Concept
SQL Query
PySpark Equivalent
table;
17. Common Table Expressions (CTEs)
Concept
SQL Query
PySpark Equivalent
WITH cte1 AS (SELECT df.createOrReplaceTempView("cte1"
CTE (Common * FROM table1)
)
Table
SELECT * FROM cte1
df_cte1 = spark.sql("SELECT *
Expressions)
WHERE condition;
FROM cte1 WHERE condition")
18. Window Functions
Window functions allow calculations across a set of table rows related to the current row.
Concept
SQL Query
PySpark Equivalent
SELECT column, RANK()
df.withColumn("rank",
OVER (PARTITION BY col2
rank().over(Window.partitionBy("c
RANK()
ORDER BY column) FROM
ol2").orderBy("column")))
table;
SELECT column,
DENSE_RANK() OVER
df.withColumn("dense_rank",
DENSE_RA (PARTITION BY col2
dense_rank().over(Window.partitio
NK()
ORDER BY column) FROM
nBy("col2").orderBy("column")))
table;
SELECT column,
ROW_NUMBER() OVER
df.withColumn("row_number",
ROW_NU (PARTITION BY col2
row_number().over(Window.partitio
MBER()
ORDER BY column) FROM
nBy("col2").orderBy("column")))
table;
SELECT column,
df.withColumn("lead_value",
LEAD(column, 1) OVER
lead("column",
(PARTITION BY col2
LEAD()
1).over(Window.partitionBy("col2"
ORDER BY column) FROM
).orderBy("column")))
table;
SELECT column,
df.withColumn("lag_value",
LAG(column, 1) OVER
lag("column",
(PARTITION BY col2
LAG()
1).over(Window.partitionBy("col2"
ORDER BY column) FROM
).orderBy("column")))
table;