Create submit_pyspark_job_to_driver_node_group_cluster.py #13513

Supriya-Koppa · 2025-07-22T07:40:37Z

Sample code to submit a PySpark job to a Dataproc driver node group cluster.

Description

Fixes #

Note: Before submitting a pull request, please open an issue for discussion if you are not associated with Google.

Checklist

I have followed Sample Guidelines from AUTHORING_GUIDE.MD
README is updated to include all relevant information
Tests pass: nox -s py-3.9 (see Test Environment Setup)
Lint pass: nox -s lint (see Test Environment Setup)
These samples need a new API enabled in testing projects to pass (let us know which ones)
These samples need a new/updated env vars in testing projects set to pass (let us know which ones)
This sample adds a new sample directory, and I updated the CODEOWNERS file with the codeowners for this sample
This sample adds a new Product API, and I updated the Blunderbuss issue/PR auto-assigner with the codeowners for this sample
Please merge this PR for me once it is approved

Sample code to submit a PySpark job to a Dataproc driver node group cluster.

snippet-bot · 2025-07-22T07:40:44Z

Here is the summary of changes.

You are about to add 1 region tag.

submit_pyspark_job_to_driver_node_group_cluster.py:26, tag dataproc_submit_pyspark_job_to_driver_node_group_cluster

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

Refresh this comment

gemini-code-assist

Summary of Changes

Hello @Supriya-Koppa, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request adds a new, self-contained Python sample to the repository. Its primary purpose is to provide a clear and executable example for users looking to programmatically submit PySpark jobs to Dataproc clusters, specifically those configured with driver node groups, and to illustrate how to manage the job's resource allocation and retrieve its output.

Highlights

New Sample Code: Introduced a new Python sample, submit_pyspark_job_to_driver_node_group_cluster.py, demonstrating how to submit a PySpark job to a Google Cloud Dataproc cluster that utilizes a driver node group.
Dataproc Job Submission: The sample showcases the use of the google.cloud.dataproc_v1 client library to configure and submit a PySpark job, including specifying DriverSchedulingConfig for memory and vcores.
Output Retrieval: The code includes logic to retrieve and print the job's output from the Google Cloud Storage bucket where Dataproc stores job results.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request adds a new Python sample for submitting a PySpark job to a Dataproc driver node group cluster. The code is a good starting point. My review includes suggestions for improving the script's usability by implementing command-line argument parsing, enhancing robustness with proper error handling, and increasing clarity and consistency through documentation and style adjustments. Addressing these points will make the sample more robust and user-friendly.

submit_pyspark_job_to_driver_node_group_cluster.py

gemini-code-assist · 2025-07-22T07:43:13Z

submit_pyspark_job_to_driver_node_group_cluster.py

+    my_project_id = "your_cluster"  # <-- REPLACE THIS
+    my_region = "us-central1"        # <-- REPLACE THIS
+    my_cluster_name = "your-node-group-cluster" # <-- REPLACE THIS


The script uses hardcoded placeholder values, which requires users to manually edit the file before running it. To make the sample easier to use and consistent with other samples in the repository, it should parse command-line arguments.

Additionally, the placeholder value for my_project_id is "your_cluster", which is confusing. It should be something like "your-project-id".

Please consider replacing the current implementation with one that uses sys.argv for argument parsing.

if __name__ == "__main__": import sys if len(sys.argv) != 4: sys.exit( f"Usage: python {sys.argv[0]} <project_id> <region> <cluster_name>" ) project_id = sys.argv[1] region = sys.argv[2] cluster_name = sys.argv[3] submit_job(project_id, region, cluster_name)

corrected the placeholder value for my_project_id.

I intend the main function to be as is.

gemini-code-assist · 2025-07-22T07:43:13Z

submit_pyspark_job_to_driver_node_group_cluster.py

+#    python submit_pyspark_job_to_driver_node_group_cluster.py \
+#        --project_id <PROJECT_ID> --region <REGION> \
+#        --cluster_name <CLUSTER_NAME>


The Usage example suggests named arguments (--project_id), but other samples in this directory (e.g., submit_job.py) use positional arguments. To maintain consistency across samples, consider updating this to reflect positional arguments.

Suggested change

# python submit_pyspark_job_to_driver_node_group_cluster.py \

# --project_id <PROJECT_ID> --region <REGION> \

# --cluster_name <CLUSTER_NAME>

# python submit_pyspark_job_to_driver_node_group_cluster.py \

# <PROJECT_ID> <REGION> <CLUSTER_NAME>

left as is, referred to https://github.com/GoogleCloudPlatform/python-docs-samples/blob/HEAD/dataproc/snippets/submit_spark_job_to_driver_node_group_cluster.py

gemini-code-assist · 2025-07-22T07:43:13Z

submit_pyspark_job_to_driver_node_group_cluster.py

+
+import re
+
+


Per PEP 8, there should be only one blank line separating logical import groups (in this case, standard library re from third-party google.cloud). Please remove the extra blank line here to improve code style consistency.

submit_pyspark_job_to_driver_node_group_cluster.py

gemini-code-assist · 2025-07-22T07:43:14Z

submit_pyspark_job_to_driver_node_group_cluster.py

+    # Create the job config. 'main_jar_file_uri' can also be a
+    # Google Cloud Storage URL.


This comment mentions 'main_jar_file_uri', which is relevant for Spark jobs (Java/Scala). Since this is a PySpark job, the comment should refer to 'main_python_file_uri' to avoid confusion.

# Create the job config. 'main_python_file_uri' can also be a

Added a END region tag

Improving readability

Supriya-Koppa · 2025-07-23T06:28:09Z

hi @glasnt could you please review this PR?

it is similar to #13423

Thanks!

glasnt

Thank you for your first submission to the python-docs-samples repo! 🎉

There are some changes required to this file, including:

moving it to the correct sample location for this product: dataproc/snippets/
including testing (you should be able to reference tests from similar samples in the dataproc folder as a starting point)

glasnt · 2025-07-24T00:21:34Z

submit_pyspark_job_to_driver_node_group_cluster.py

@@ -0,0 +1,91 @@
+#!/usr/bin/env python


This file needs to be moved to the dataproc/snippets folder.

Also, please clarify that the existing "submit_spark_job_to_driver_node_group_cluster.py" sample is different to this "submit_pyspark_job_to_driver_node_group_cluster.py` sample.

glasnt · 2025-07-24T00:23:29Z

submit_pyspark_job_to_driver_node_group_cluster.py

+    # Create the job config. 'main_python_file_uri' can also be a
+    # Google Cloud Storage URL.


In this case, the value already is a Storage URL. This comment should be updated.

glasnt · 2025-07-24T00:23:57Z

submit_pyspark_job_to_driver_node_group_cluster.py

+    job = {
+        "placement": {"cluster_name": cluster_name},
+        "pyspark_job": {
+            "main_python_file_uri": "gs://dataproc-examples/pyspark/hello-world/hello-world.py"


Where does this object come from? Is there open source code for this object?

glasnt · 2025-07-24T00:25:38Z

submit_pyspark_job_to_driver_node_group_cluster.py

+    my_project_id = "your_project_id"  # <-- REPLACE THIS
+    my_region = "your_region"        # <-- REPLACE THIS
+    my_cluster_name = "your_node_group_cluster" # <-- REPLACE THIS
+
+    submit_job(my_project_id, my_region, my_cluster_name)


Please change this main method to copy the functionality from the spark job, which will remove the "REPLACE THIS" values. This can also help if you use command-line testing

python-docs-samples/dataproc/snippets/submit_spark_job_to_driver_node_group_cluster.py

Line 93 in e06d3cf

parser = argparse.ArgumentParser(

Create submit_pyspark_job_to_driver_node_group_cluster.py

0ced200

Sample code to submit a PySpark job to a Dataproc driver node group cluster.

Supriya-Koppa requested review from a team as code owners July 22, 2025 07:40

product-auto-label bot added the samples Issues that are directly related to samples. label Jul 22, 2025

gemini-code-assist bot reviewed Jul 22, 2025

View reviewed changes

Supriya-Koppa added 2 commits July 22, 2025 13:23

Update submit_pyspark_job_to_driver_node_group_cluster.py

1921249

Added a END region tag

Update submit_pyspark_job_to_driver_node_group_cluster.py

ca20564

Improving readability

glasnt requested changes Jul 24, 2025

View reviewed changes

		# Create the job config. 'main_jar_file_uri' can also be a
		# Google Cloud Storage URL.

		# Create the job config. 'main_python_file_uri' can also be a
		# Google Cloud Storage URL.

Create submit_pyspark_job_to_driver_node_group_cluster.py #13513

Are you sure you want to change the base?

Create submit_pyspark_job_to_driver_node_group_cluster.py #13513

Uh oh!

Conversation

Supriya-Koppa commented Jul 22, 2025

Description

Checklist

Uh oh!

snippet-bot bot commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

gemini-code-assist bot Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gemini-code-assist bot Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Supriya-Koppa commented Jul 23, 2025

Uh oh!

glasnt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

snippet-bot bot commented Jul 22, 2025 •

edited

Loading