Questions tagged [pyspark]
The Spark Python API (PySpark) exposes the Apache Spark programming model to Python.
pyspark
40,854
questions
0
votes
1
answer
20
views
Deduplication based on other dataframe columns
I have pyspark df1 as
+----+
|name|
+----+
| A|
| B|
| C|
| D|
+----+
and df2 as
+------+------+
|name_a|name_b|
+------+------+
| A| B|
| B| A|
| C| A|
| A| C|...
- 81
1
vote
1
answer
16
views
avg() over a whole dataframe causing different output
I see that dataframe.agg(avg(Col) works fine, but when i calculate avg() over a window over whole column(not using any partition), i see different results based on which column i use with orderBy.
...
- 1,685
0
votes
0
answers
12
views
java.io.UncheckedIOException: io.netty.channel.StacklessClosedChannelException while writing to adx
I'm trying to write some data to azure data explorer table, but getting the below exception.
The adx cluster is open to all the networks and could see the connectivity between databricks and adx is ...
- 56
0
votes
2
answers
22
views
How to get nested xml structure as a string from an xml document using xpath in pyspark dataframe?
I have a dataframe with a string datatype column with XML string. Now I want to create a new column with a nested XML structure from the original column. For this, I tried using XPath in PySpark.
...
- 13
0
votes
0
answers
17
views
Spark Shell: spark.executor.extraJavaOptions is not allowed to set Spark options
I created this Spark Shell programm, but i got this error while running it:
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
Install the latest PowerShell for new features ...
-3
votes
0
answers
19
views
Write filter on DataFrame in Spark Scala on mulitple different columns
I have 3 columns in my data frame that I want to run my filter on.
Filter conditions:
dataframe.filter(col(ID) =!= X) || col(y) =!= null || col (y) =!= col(z))
Requirement is :
To exclude data from ...
- 333
0
votes
3
answers
19
views
How to drop records after date based on condition
I'm looking for an elegant way to drop all records in a DataFrame that occur before the latest occurrence of 'TEST_COMPONENT' being 'UNSATISFACTORY', based on their 'TEST_DT' value for each ID.
For ...
0
votes
0
answers
20
views
Unable to read a dataframe from s3
I am getting the following error:
24/07/25 21:29:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/07/25 21:29:53 ...
- 7
0
votes
0
answers
19
views
Error while trying to show the results using PySpark show function
I am trying to show my results in PySpark.
I am using Spark 3.5.1 and PySpark 3.5.1 with Java 8 installed and everything is well set.
Some answers suggesting adding this:
import findspark
findspark....
- 56
0
votes
0
answers
24
views
Can we create multiple Spark executors within a single driver node on a Databricks cluster?
I have a power user compute with a single driver node and I'm trying to parallelize forecasting across multiple series by aggregating the data and doing a groupBy and then an apply on the groupBy.
The ...
0
votes
1
answer
25
views
Pandas UDF to derive new column
In Spark/Databricks, I have a pandas dataframe with a string column. I need to perform multiple actions on this column (data cleansing type stuff), and produce a new column from the result.
Here's my ...
- 8,613
0
votes
0
answers
17
views
Read multiple files parallel into separate dataframe in Pyspark
I am trying to read large txt files into dataframe. Each file is 10-15 GB in size,
as the IO is taking long time. I want to read multiple file in parallel and get them in separate dataframe.
I tried ...
- 401
0
votes
0
answers
13
views
Could not find py4J jar at
I am trying to run my PMML pre-trained model in Python 3.9, but no matter what I do I have this error Could Not find py4j jar at.
None of the solutions provided on the blogs are working. And, even if ...
1
vote
1
answer
35
views
Replacing carriage return and line feeds in a text file using Azure Synapse
I am currently working on a project where I have to get data from an Excel file in SharePoint, this file has multiple tabs, and I am using Azure Synapse Analytics to call a SharePoint API to get each ...
- 23
0
votes
0
answers
12
views
Spark OverwritePartitions always triggering shuffle of data
I have a spark job that is reading from an oracle data source and writing to a iceberg table.
There are multiple queries executing in multi-threading, each query hits just one partition (in iceberg).
...
- 1