Skip to content

deep merge can't nullify autoscale settings in target. #5701

Description

@enceladus-beep

Describe the issue

I'm trying to override the autoscale setting in a specific target to use a single-node cluster.

Configuration

resources:
  jobs: &job-config
    xxx_job: &xxx_job
      name: xxx_job
      tags: &tags
        ver: ver1
      tasks:
        - task_key: tk1
          job_cluster_key: &tk1_cluster_key tk1_cluster
      job_clusters: &job_clusters
        - job_cluster_key: tk1_cluster
          new_cluster: &tk1_cluster
            spark_version: 16.4.x-scala2.13
            spark_conf: &spark_conf
              spark.databricks.delta.stalenessLimit: 20000
              spark.sql.shuffle.partitions: auto
            custom_tags: &cluster_custom_tags
              Team: X-Team
            node_type_id: r5d.large
            autoscale:
              min_workers: 1
              max_workers: 2


bundle:
  name: bundleX

targets:
  t1:
  
  t2: 
    resources:
      jobs:
        xxx_job:
          job_clusters:
            - job_cluster_key: tk1_cluster
              new_cluster:
                node_type_id: r5d.4xlarge
                # nullify autoscale settings on this singleNode cluster
                autoscale: null
                num_workers: 0
                spark_conf:
                  <<: *spark_conf
                  spark.databricks.cluster.profile: singleNode
                  spark.master: "local[*]"
                custom_tags:
                  <<: *cluster_custom_tags
                  Task: apiband-bola-attack
                  ResourceClass: SingleNode

Steps to reproduce the behavior

  1. Run a summary of t1 see the default value for autoscale:
$ databricks -p test bundle summary -t t1 -o json | gron | grep new_cluster
json.resources.jobs.xxx_job.job_clusters[0].new_cluster = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.autoscale = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.autoscale.max_workers = 2;
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.autoscale.min_workers = 1;
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.custom_tags = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.custom_tags.Team = "X-Team";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.node_type_id = "r5d.large";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf["spark.databricks.delta.stalenessLimit"] = "20000";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf["spark.sql.shuffle.partitions"] = "auto";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_version = "16.4.x-scala2.13";
  1. Run a summary for t2 and see that the autoscale settings are not nullified.
$ databricks -p test bundle summary -t t2 -o json | gron | grep new_cluster
json.resources.jobs.xxx_job.job_clusters[0].new_cluster = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.autoscale = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.autoscale.max_workers = 2;
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.autoscale.min_workers = 1;
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.custom_tags = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.custom_tags.ResourceClass = "SingleNode";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.custom_tags.Task = "apiband-bola-attack";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.custom_tags.Team = "X-Team";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.node_type_id = "r5d.4xlarge";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.num_workers = 0;
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf = {};
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf["spark.databricks.cluster.profile"] = "singleNode";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf["spark.databricks.delta.stalenessLimit"] = "20000";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf["spark.master"] = "local[*]";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_conf["spark.sql.shuffle.partitions"] = "auto";
json.resources.jobs.xxx_job.job_clusters[0].new_cluster.spark_version = "16.4.x-scala2.13";

Expected Behavior

No autoscale settings in target t2

Actual Behavior

The autoscale setting is deep merged into t2

OS and CLI version

$ databricks --version 
Databricks CLI v1.4.0

Is this a regression?

No.

Debug Logs

Attached to the bug, although not relevant to this issue, it doesn't show anything about deep merge.

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugSomething isn't workingDABsDABs related issues

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions