Describe the issue
I'm trying to override the autoscale setting in a specific target to use a single-node cluster.
Configuration
resources:
jobs: &job-config
xxx_job: &xxx_job
name: xxx_job
tags: &tags
ver: ver1
tasks:
- task_key: tk1
job_cluster_key: &tk1_cluster_key tk1_cluster
job_clusters: &job_clusters
- job_cluster_key: tk1_cluster
new_cluster: &tk1_cluster
spark_version: 16.4.x-scala2.13
spark_conf: &spark_conf
spark.databricks.delta.stalenessLimit: 20000
spark.sql.shuffle.partitions: auto
custom_tags: &cluster_custom_tags
Team: X-Team
node_type_id: r5d.large
autoscale:
min_workers: 1
max_workers: 2
bundle:
name: bundleX
targets:
t1:
t2:
resources:
jobs:
xxx_job:
job_clusters:
- job_cluster_key: tk1_cluster
new_cluster:
node_type_id: r5d.4xlarge
# nullify autoscale settings on this singleNode cluster
autoscale: null
num_workers: 0
spark_conf:
<<: *spark_conf
spark.databricks.cluster.profile: singleNode
spark.master: "local[*]"
custom_tags:
<<: *cluster_custom_tags
Task: apiband-bola-attack
ResourceClass: SingleNode
Steps to reproduce the behavior
- Run a summary of
t1 see the default value for autoscale:
$ databricks -p test bundle summary -t t1 -o json | gron | grep new_cluster
json.resources.jobs.xxx_job.job_clusters[0].new_cluster = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.autoscale = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.autoscale.max_workers = 2;
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.autoscale.min_workers = 1;
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.custom_tags = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.custom_tags.Team = "X-Team";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.node_type_id = "r5d.large";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf["spark.databricks.delta.stalenessLimit"] = "20000";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf["spark.sql.shuffle.partitions"] = "auto";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_version = "16.4.x-scala2.13";
- Run a summary for
t2 and see that the autoscale settings are not nullified.
$ databricks -p test bundle summary -t t2 -o json | gron | grep new_cluster
json.resources.jobs.xxx_job.job_clusters[0].new_cluster = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.autoscale = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.autoscale.max_workers = 2;
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.autoscale.min_workers = 1;
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.custom_tags = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.custom_tags.ResourceClass = "SingleNode";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.custom_tags.Task = "apiband-bola-attack";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.custom_tags.Team = "X-Team";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.node_type_id = "r5d.4xlarge";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.num_workers = 0;
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf["spark.databricks.cluster.profile"] = "singleNode";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf["spark.databricks.delta.stalenessLimit"] = "20000";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf["spark.master"] = "local[*]";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf["spark.sql.shuffle.partitions"] = "auto";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_version = "16.4.x-scala2.13";
Expected Behavior
No autoscale settings in target t2
Actual Behavior
The autoscale setting is deep merged into t2
OS and CLI version
$ databricks --version
Databricks CLI v1.4.0
Is this a regression?
No.
Debug Logs
Attached to the bug, although not relevant to this issue, it doesn't show anything about deep merge.
Describe the issue
I'm trying to override the
autoscalesetting in a specific target to use a single-node cluster.Configuration
Steps to reproduce the behavior
t1see the default value for autoscale:t2and see that the autoscale settings are not nullified.Expected Behavior
No
autoscalesettings in targett2Actual Behavior
The
autoscalesetting is deep merged intot2OS and CLI version
Is this a regression?
No.
Debug Logs
Attached to the bug, although not relevant to this issue, it doesn't show anything about deep merge.