Manages Hadoop-based clusters and jobs on Google Cloud Platform.

Specifies the type and number of accelerator cards attached to the instances of an instance. See GPUs on Compute Engine (https://cloud.google.com/compute/docs/gpus/).

Environment details of a Saprk Application.

Details of a particular job associated with Spark Application

A summary of Spark Application

Details of a query for a Spark Application

SparkPlanGraph for a Spark Application execution limited to maximum 10000 clusters.

Stage Attempt for a Stage of a Spark Application

RDD operation graph for a Spark Application Stage limited to maximum 10000 clusters.

Environment details of a Saprk Application.

Details of a particular job associated with Spark Application

A summary of Spark Application

Details of a query for a Spark Application

SparkPlanGraph for a Spark Application execution limited to maximum 10000 clusters.

Stage Attempt for a Stage of a Spark Application

RDD operation graph for a Spark Application Stage limited to maximum 10000 clusters.

A request to analyze a batch workload.

Metadata describing the Analyze operation.

Specific attempt of an application.

Details about the Environment that the application is running in.

High level information corresponding to an application.

Authentication configuration for a workload is used to set the default identity for the workload execution. The config specifies the type of identity (service account or user) that will be used by workloads to access resources on the project(s).

Autoscaling Policy config associated with the cluster.

Describes an autoscaling policy for Dataproc cluster autoscaler.

Autotuning configuration of the workload.

Node group identification and configuration information.

Auxiliary services configuration for a Cluster.

Basic algorithm for autoscaling.

Basic autoscaling configurations for YARN.

A representation of a batch workload in the service.

Metadata describing the Batch operation.

Associates members, or principals, with a role.

Native Build Info

A request to cancel a job.

Describes the identifying information, config, and status of a Dataproc cluster

The cluster config.

Contains cluster daemon metrics, such as HDFS and YARN stats.Beta Feature: This report is available for testing purposes only. It may be changed before final release.

The cluster operation triggered by a workflow.

Metadata describing the operation.

The status of the operation.

A selector that chooses target cluster for jobs based on metadata.

The status of a cluster and its instances.

Cluster to be repaired

Confidential Instance Config for clusters using Confidential VMs (https://cloud.google.com/compute/confidential-vm/docs)

Consolidated summary about executors used by the application.

Defines the root interface for all clients that generate credentials for calling Google APIs. All clients should implement this interface.

Dataproc metric config.

A request to collect cluster diagnostic information.

The location of diagnostic output.

Specifies the config of boot disk and attached disk options for a group of VM instances.

Driver scheduling configuration.

A generic empty message that you can re-use to avoid defining duplicated empty messages in your APIs. A typical example is to use it as the request or the response type of an API method. For instance: service Foo { rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty); }

Encryption settings for the cluster.

Endpoint config for this cluster

Environment configuration for a workload.

Execution configuration for a workload.

Resources used per executor used by the application.

Executor resources consumed by a stage.

Details about executors used by the application.

Represents a textual expression in the Common Expression Language (CEL) syntax. CEL is a C-like expression language. The syntax and semantics of CEL are documented at https://github.com/google/cel-spec.Example (Comparison): title: "Summary size limit" description: "Determines if a summary is less than 100 chars" expression: "document.summary.size() < 100" Example (Equality): title: "Requestor is owner" description: "Determines if requestor is the document owner" expression: "document.owner == request.auth.claims.email" Example (Logic): title: "Public documents" description: "Determine whether the document should be publicly visible" expression: "document.type != 'private' && document.type != 'internal'" Example (Data Manipulation): title: "Notification string" description: "Create a notification string with a timestamp." expression: "'New message received at ' + string(document.create_time)" The exact variables and functions that may be referenced within an expression are determined by the service that evaluates it. See the service documentation for additional information.

Native SQL Execution Data

A Dataproc job for running Apache Flink applications on YARN.

Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.

Request message for GetIamPolicy method.

Encapsulates settings provided to GetIamPolicy.

The cluster's GKE config.

Parameters that describe cluster nodes.

A GkeNodeConfigAcceleratorConfig represents a Hardware Accelerator request for a node pool.

GkeNodePoolAutoscaling contains information the cluster autoscaler needs to adjust the size of the node pool to the current cluster usage.

The configuration of a GKE node pool used by a Dataproc-on-GKE cluster (https://cloud.google.com/dataproc/docs/concepts/jobs/dataproc-gke#create-a-dataproc-on-gke-cluster).

GKE node pools that Dataproc workloads run on.

Encryption settings for encrypting workflow template job arguments.

A Dataproc job for running Apache Hadoop MapReduce (https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html) jobs on Apache Hadoop YARN (https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/YARN.html).

A Dataproc job for running Apache Hive (https://hive.apache.org/) queries on YARN.

Identity related configuration, including service account based secure multi-tenancy user mappings.

A request to inject credentials into a cluster.

Metrics about the input data read by the task.

Instance flexibility Policy allowing a mixture of VM shapes and provisioning models.

Configuration for the size bounds of an instance group, including its proportional size to other groups.

The config settings for Compute Engine resources in an instance group, such as a master or worker group.

A reference to a Compute Engine instance.

Defines machines types and a rank to which the machines types belong.

Defines a mapping from machine types to the number of VMs that are created with each machine type.

A request to instantiate a workflow template.

Represents a time interval, encoded as a Timestamp start (inclusive) and a Timestamp end (exclusive).The start must be less than or equal to the end. When the start equals the end, the interval is empty (matches no time). When both start and end are unspecified, the interval matches any time.

A Dataproc job resource.

Data corresponding to a spark job.

Job Operation metadata.

Dataproc job config.

Encapsulates the full scoping used to reference a job.

Job scheduling options.

Data related to Jobs page summary

Dataproc job status.

Jupyter configuration for an interactive session.

Specifies Kerberos related configuration.

The configuration for running the Dataproc cluster on Kubernetes.

The software configuration for this Dataproc cluster running on Kubernetes.

Specifies the cluster auto-delete schedule configuration.

A response to a request to list autoscaling policies in a project.

A list of batch workloads.

The list of all clusters in a project.

A list of jobs in a project.

The response message for Operations.ListOperations.

A list of interactive sessions.

A list of session templates.

A response to a request to list workflow templates in a project.

The runtime logging config of the job.

Cluster that is managed by the workflow.

Specifies the resources used to actively manage an instance group.

Specifies a Metastore configuration.

A Dataproc custom metric.

Deprecated. Used only for the deprecated beta. A full, namespace-isolated deployment target for an existing GKE cluster.

Native SQL Execution Data

Dataproc Node Group. The Dataproc NodeGroup resource is not related to the Dataproc NodeGroupAffinity resource.

Node Group Affinity for clusters using sole-tenant node groups. The Dataproc NodeGroupAffinity resource is not related to the Dataproc NodeGroup resource.

Metadata describing the node group operation.

Specifies an executable to run on a fully configured node and a timeout period for executable completion.

indicating a list of workers of same type

This resource represents a long-running operation that is the result of a network API call.

A job executed by the workflow.

Metrics about the data written by the task.

Configuration for parameter validation.

Auxiliary services configuration for a workload.

A Dataproc job for running Apache Pig (https://pig.apache.org/) queries on YARN.

An Identity and Access Management (IAM) policy, which specifies access controls for Google Cloud resources.A Policy is a collection of bindings. A binding binds one or more members, or principals, to a single role. Principals can be user accounts, service accounts, Google groups, and domains (such as G Suite). A role is a named list of permissions; each role can be an IAM predefined role or a user-created custom role.For some types of Google Cloud resources, a binding can also specify a condition, which is a logical expression that allows access to a resource only if the expression evaluates to true. A condition can add constraints based on attributes of the request, the resource, or both. To learn which resources support conditions in their IAM policies, see the IAM documentation (https://cloud.google.com/iam/help/conditions/resource-policies).JSON example: { "bindings": [ { "role": "roles/resourcemanager.organizationAdmin", "members": [ "user:mike@example.com", "group:admins@example.com", "domain:google.com", "serviceAccount:my-project-id@appspot.gserviceaccount.com" ] }, { "role": "roles/resourcemanager.organizationViewer", "members": [ "user:eve@example.com" ], "condition": { "title": "expirable access", "description": "Does not grant access after Sep 2020", "expression": "request.time < timestamp('2020-10-01T00:00:00.000Z')", } } ], "etag": "BwWWja0YfJA=", "version": 3 } YAML example: bindings: - members: - user:mike@example.com - group:admins@example.com - domain:google.com - serviceAccount:my-project-id@appspot.gserviceaccount.com role: roles/resourcemanager.organizationAdmin - members: - user:eve@example.com role: roles/resourcemanager.organizationViewer condition: title: expirable access description: Does not grant access after Sep 2020 expression: request.time < timestamp('2020-10-01T00:00:00.000Z') etag: BwWWja0YfJA= version: 3 For a description of IAM and its features, see the IAM documentation (https://cloud.google.com/iam/docs/).

Pool Data

A Dataproc job for running Presto (https://prestosql.io/) queries. IMPORTANT: The Dataproc Presto Optional Component (https://cloud.google.com/dataproc/docs/concepts/components/presto) must be enabled when the cluster is created to submit a Presto job to the cluster.

Process Summary

Additional options for Dataproc#projectsLocationsAutoscalingPoliciesList.

Additional options for Dataproc#projectsLocationsBatchesCreate.

Additional options for Dataproc#projectsLocationsBatchesList.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsAccessEnvironmentInfo.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsAccessJob.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsAccess.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsAccessSqlPlan.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsAccessSqlQuery.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsAccessStageAttempt.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsAccessStageRddGraph.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsSearchExecutors.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsSearchExecutorStageSummary.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsSearchJobs.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsSearch.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsSearchSqlQueries.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsSearchStageAttempts.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsSearchStageAttemptTasks.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsSearchStages.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsSummarizeExecutors.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsSummarizeJobs.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsSummarizeStageAttemptTasks.

Additional options for Dataproc#projectsLocationsBatchesSparkApplicationsSummarizeStages.

Additional options for Dataproc#projectsLocationsOperationsList.

Additional options for Dataproc#projectsLocationsSessionsCreate.

Additional options for Dataproc#projectsLocationsSessionsDelete.

Additional options for Dataproc#projectsLocationsSessionsList.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsAccessEnvironmentInfo.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsAccessJob.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsAccess.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsAccessSqlPlan.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsAccessSqlQuery.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsAccessStageAttempt.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsAccessStageRddGraph.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsSearchExecutors.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsSearchExecutorStageSummary.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsSearchJobs.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsSearch.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsSearchSqlQueries.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsSearchStageAttempts.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsSearchStageAttemptTasks.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsSearchStages.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsSummarizeExecutors.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsSummarizeJobs.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsSummarizeStageAttemptTasks.

Additional options for Dataproc#projectsLocationsSessionsSparkApplicationsSummarizeStages.

Additional options for Dataproc#projectsLocationsSessionTemplatesList.

Additional options for Dataproc#projectsLocationsWorkflowTemplatesDelete.

Additional options for Dataproc#projectsLocationsWorkflowTemplatesGet.

Additional options for Dataproc#projectsLocationsWorkflowTemplatesInstantiateInline.

Additional options for Dataproc#projectsLocationsWorkflowTemplatesList.

Additional options for Dataproc#projectsRegionsAutoscalingPoliciesList.

Additional options for Dataproc#projectsRegionsClustersCreate.

Additional options for Dataproc#projectsRegionsClustersDelete.

Additional options for Dataproc#projectsRegionsClustersList.

Additional options for Dataproc#projectsRegionsClustersNodeGroupsCreate.

Additional options for Dataproc#projectsRegionsClustersPatch.

Additional options for Dataproc#projectsRegionsJobsList.

Additional options for Dataproc#projectsRegionsJobsPatch.

Additional options for Dataproc#projectsRegionsOperationsList.

Additional options for Dataproc#projectsRegionsWorkflowTemplatesDelete.

Additional options for Dataproc#projectsRegionsWorkflowTemplatesGet.

Additional options for Dataproc#projectsRegionsWorkflowTemplatesInstantiateInline.

Additional options for Dataproc#projectsRegionsWorkflowTemplatesList.

Properties of the workload organized by origin.

Defines how Dataproc should create VMs with a mixture of provisioning models.

Configuration for PyPi repository

A configuration for running an Apache PySpark (https://spark.apache.org/docs/latest/api/python/getting_started/quickstart.html) batch workload.

A Dataproc job for running Apache PySpark (https://spark.apache.org/docs/latest/api/python/index.html#pyspark-overview) applications on YARN.

Quantile metrics data related to Tasks. Units can be seconds, bytes, milliseconds, etc depending on the message type.

A list of queries to run on a cluster.

Details about RDD usage.

A grouping of nodes representing higher level constructs (stage, job etc.).

A directed edge representing dependency between two RDDs.

Graph representing RDD dependencies. Consists of edges and a root cluster.

A node in the RDD operation graph. Corresponds to a single RDD.

Information about RDD partitions.

Overall data about RDD storage.

Validation based on regular expressions.

A request to repair a cluster.

Configuration for dependency repositories

Reservation Affinity for consuming Zonal reservation.

A request to resize a node group.

Resource profile that contains information about all the resources required by executors and tasks.

Runtime configuration for a workload.

Runtime information about workload execution.

List of Executors associated with a Spark Application.

List of Executors associated with a Spark Application Stage.

A list of Jobs associated with a Spark Application.

List of all queries for a Spark Application.

A list of summary of Spark Applications

A list of Stage Attempts for a Stage of a Spark Application.

List of tasks for a stage of a Spark Application

A list of stages associated with a Spark Application.

List of Executors associated with a Spark Application.

List of Executors associated with a Spark Application Stage.

A list of Jobs associated with a Spark Application.

List of all queries for a Spark Application.

A list of summary of Spark Applications

A list of Stage Attempts for a Stage of a Spark Application.

List of tasks for a stage of a Spark Application

A list of stages associated with a Spark Application.

Security related configuration, including encryption, Kerberos, etc.

A representation of a session.

Metadata describing the Session operation.

Historical state information.

A representation of a session template.

Request message for SetIamPolicy method.

Shielded Instance Config for clusters using Compute Engine Shielded VMs (https://cloud.google.com/security/shielded-cloud/shielded-vm).

Shuffle data read by the task.

Shuffle data written by task.

Specifies the selection and config of software inside the cluster.

A summary of Spark Application

A configuration for running an Apache Spark (https://spark.apache.org/) batch workload.

Spark connect configuration for an interactive session.

Spark History Server configuration for the workload.

A Dataproc job for running Apache Spark (https://spark.apache.org/) applications on YARN.

A graph used for storing information of an executionPlan of DataFrame.

Represents a tree of spark plan.

Represents a directed edge in the spark plan tree from child to parent.

Represents a node in the spark plan tree.

Wrapper user to represent either a node or a cluster.

A configuration for running an Apache SparkR (https://spark.apache.org/docs/latest/sparkr.html) batch workload.

A Dataproc job for running Apache SparkR (https://spark.apache.org/docs/latest/sparkr.html) applications on YARN.

A configuration for running Apache Spark SQL (https://spark.apache.org/sql/) queries as a batch workload.

A Dataproc job for running Apache Spark SQL (https://spark.apache.org/sql/) queries.

Basic autoscaling configurations for Spark Standalone.

Outer message that contains the data obtained from spark listener, packaged with information that is required to process it.

Details of the speculation task when speculative execution is enabled.

SQL Execution Data

Metrics related to SQL execution.

Data related to tasks summary for a Spark Stage Attempt

Data corresponding to a stage.

Metrics about the input read by the stage.

Stage Level Aggregated Metrics

Metrics about the output written by the stage.

Shuffle data read for the stage.

Shuffle data written for the stage.

Data related to Stages page summary

A request to start a cluster.

Configuration to handle the startup of instances during cluster create and update process.

Historical state information.

The Status type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by gRPC (https://github.com/grpc). Each Status message contains three pieces of data: error code, error message, and error details.You can find out more about this error model and how to work with it in the API Design Guide (https://cloud.google.com/apis/design/errors).

A request to stop a cluster.

Stream Block Data.

Streaming

A request to submit a job.

Consolidated summary of executors for a Spark Application.

Summary of a Spark Application jobs.

Summary of tasks for a Spark Application stage attempt.

Summary of a Spark Application stages.

Consolidated summary of executors for a Spark Application.

Summary of a Spark Application jobs.

Summary of tasks for a Spark Application stage attempt.

Summary of a Spark Application stages.

Data corresponding to tasks created by spark.

Executor Task Metrics

Resources used per task created by the application.

A configurable parameter that replaces one or more fields in the template. Parameterizable fields: - Labels - File uris - Job properties - Job arguments

• Script variables - Main class (in HadoopJob and SparkJob) - Zone (in ClusterSelector)

A request to terminate an interactive session.

Request message for TestIamPermissions method.

Response message for TestIamPermissions method.

A Dataproc job for running Trino (https://trino.io/) queries. IMPORTANT: The Dataproc Trino Optional Component (https://cloud.google.com/dataproc/docs/concepts/components/trino) must be enabled when the cluster is created to submit a Trino job to the cluster.

Usage metrics represent approximate total resources consumed by a workload.

The usage snapshot represents the resources consumed by a workload at a specified time.

Annotatated property value.

Validation based on a list of allowed values.

The Dataproc cluster config for a cluster that does not directly control the underlying compute resources, such as a Dataproc-on-GKE cluster (https://cloud.google.com/dataproc/docs/guides/dpgke/dataproc-gke-overview).

The workflow graph.

A Dataproc workflow template resource.

The workflow node.

A Dataproc workflow template resource.

Specifies workflow execution target.Either managed_cluster or cluster_selector is required.

Write Spark Application data to internal storage systems

Response returned as an acknowledgement of receipt of data.

Write Spark Application data to internal storage systems

Response returned as an acknowledgement of receipt of data.

A YARN application created by a job. Application information is a subset of org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto.Beta Feature: This report is available for testing purposes only. It may be changed before final release.