WorkerPool
import type { WorkerPool } from "https://googleapis.deno.dev/v1/dataflow:v1b3.ts";
Describes one particular pool of Cloud Dataflow workers to be instantiated by the Cloud Dataflow service in order to perform the computations required by a job. Note that a workflow job may use multiple pools, in order to match the various computational requirements of the various stages of the job.
§Properties
Settings for autoscaling of this WorkerPool.
The default package set to install. This allows the service to select a default set of packages which are useful to worker harnesses written in a particular language.
Size of root disk for VMs, in GB. If zero or unspecified, the service will attempt to choose a reasonable default.
Type of root disk for VMs. If empty or unspecified, the service will attempt to choose a reasonable default.
Configuration for VM IPs.
Machine type (e.g. "n1-standard-1"). If empty or unspecified, the service will attempt to choose a reasonable default.
Network to which VMs will be assigned. If empty or unspecified, the service will use the network "default".
The number of threads per worker harness. If empty or unspecified, the service will choose a number of threads (according to the number of cores on the selected machine type for batch, or 1 by convention for streaming).
Number of Google Compute Engine workers in this pool needed to execute the job. If zero or unspecified, the service will attempt to choose a reasonable default.
The action to take on host maintenance, as defined by the Google Compute Engine API.
Set of SDK harness containers needed to execute this pipeline. This will only be set in the Fn API path. For non-cross-language pipelines this should have only one entry. Cross-language pipelines will have two or more entries.
Subnetwork to which VMs will be assigned, if desired. Expected to be of the form "regions/REGION/subnetworks/SUBNETWORK".
Settings passed through to Google Compute Engine workers when using the standard Dataflow task runner. Users should ignore this field.
Sets the policy for determining when to turndown worker pool. Allowed
values are: TEARDOWN_ALWAYS
, TEARDOWN_ON_SUCCESS
, and TEARDOWN_NEVER
.
TEARDOWN_ALWAYS
means workers are always torn down regardless of whether
the job succeeds. TEARDOWN_ON_SUCCESS
means workers are torn down if the
job succeeds. TEARDOWN_NEVER
means the workers are never torn down. If
the workers are not torn down by the service, they will continue to run and
use Google Compute Engine VM resources in the user's project until they are
explicitly terminated by the user. Because of this, Google recommends using
the TEARDOWN_ALWAYS
policy except for small, manually supervised test
jobs. If unknown or unspecified, the service will attempt to choose a
reasonable default.