About waynez

waynez

I created an autoscaling policy and attached to a cluster with 2 primary worker nodes (2 core with 8gb). I ran a PySpark code with number of executors set to 50, but it did not trigger autoscaling.Went through the documentations, most is on the theor...

waynez

I did go thru the console to create a dataproc batch, but can't find a way in UI to enable custom metrics on Spark.

waynez

@jangemmarAlso Scale up / down factor ranges from 0 to 1.0, correct? Fraction of average pending memory in the last cooldown period for which to add workers. A scale-up factor of 1.0 will result in scaling up so that there is no pending memory remain...

waynez

Thanks for the answer and I will give a try. The documentation is not clear as I thought the worker configuration did not include the 2ndary workers, my intention was to keep primary worker nodes as 2 and scale only with 2ndary workers.

waynez

ms4446,I am working on Dataproc Serverless with the same goal to get resource usage metrics for the past batches. Metrics Explorer by default shows the metrics only "active" for the last 25 hr. How to get the raw data from the same source used by Met...

waynez

Damian,I am working on Dataproc Serverless. The customer metrics are enabled by default. However, I am not clear about the concept of "active". It seems that I can add a filter on project ID and batch ID within limited time frame. Otherwise, there ar...

My Stats

kutluhang's Bio

Badges waynez Earned

Recent Activity

Autoscaling not working for Dataproc GCE cluster

How to enable custom metrics such as spark for Dataproc Serverless batches and sessions

Re: Autoscaling not working for Dataproc GCE cluster

Re: Autoscaling not working for Dataproc GCE cluster

Re: Historical Resource Consumption Metrics on Dataproc Ephemeral Clusters

Re: How to activate Dataproc Job metrics