Why You Shouldn’t Use Databricks Asset Bundles (DAB)

The Context

For the past few years, Databricks has been actively pushing Databricks Asset Bundles (DAB) as the deployment solution for data projects. The pitch is appealing: give data teams autonomy, simplify deployments, avoid Terraform’s complexity. But the reality is quite different.

DAB in Brief

Databricks Asset Bundles is a deployment tool that allows you to package and deploy Databricks resources (jobs, workflows, notebooks, etc.) via YAML configuration files. The idea is to offer a “simpler” alternative to Terraform for data teams who don’t master Infrastructure as Code.

Terraform in Brief

Terraform is the reference tool for Infrastructure as Code, developed by HashiCorp. It allows you to describe, version, and deploy any infrastructure through declarative code (HCL). The Databricks provider for Terraform has existed for years and covers (almost) all platform resources.

The Argument: “Data Engineers Don’t Master Terraform”

This is the argument we hear constantly to justify DAB: “Terraform is too complicated for data teams, it’s an infra tool, DEs (Data Engineers) don’t know how to use it.” In other words: “Each to their own perimeter, each to their own responsibilities, each to their own working methods.”

The reality? It takes one day to learn the basics of Terraform and be autonomous. Just one day. You won’t be an expert, but you’ll know how to create and deploy 80% of what you need. I’ve experienced it, my colleagues have experienced it, dozens of Data Engineers have experienced it.

If we estimate that a Data Engineer is incapable of learning HCL syntax in one day, we have a serious recruitment or training problem, not a tooling problem.

Why DAB is a Bad Idea

1. A Terraform Wrapper… Without Its Features

DAB is nothing more than a wrapper around the Databricks Terraform provider. Except that instead of benefiting from all of Terraform’s power, you inherit a watered-down, restricted version with fewer features.

2. No Modules, No Code Reuse

Terraform allows you to create reusable modules to standardize your deployments. Need a Databricks job pattern with monitoring, alerting, and retry? Create a module, call it everywhere.

With DAB? You’ll be copy-pasting your YAML configuration every time.

3. Remote State? Pushed into the Databricks Workspace

The Terraform state is one of the most critical elements of your infrastructure. With Terraform, you version it in S3, Gitlab… with encryption, locking, and complete history.

With DAB? The state is pushed directly into the Databricks workspace. No native versioning, no accessible history. You’re forced to create a custom backup job to avoid losing everything.

4. No Native Linting or Security Analysis

Terraform benefits from a mature ecosystem: terraform validate, tflint, checkov, kics, tfsec… Dozens of tools to analyze your code’s quality and security before even deploying it.

With DAB? You have to create your own custom rules.

5. Mandatory Target/Environment Specification

DAB forces you to explicitly specify targets and environments in your code. This is exactly the opposite of the fundamental IaC principle: code should be agnostic of the environment it runs in.

With Terraform, your code stays the same, only variables change. Dev, staging, prod: same code, different configurations. Clean, maintainable, scalable.

6. Include Limited to a Single Level

Need to structure your code with multiple inclusions? Terraform does it natively with nested modules.

DAB? Only one level of include. This limitation constrains your architecture and prevents you from structuring your code in a modular and scalable way as you would naturally do with Terraform.

7. Incomplete Service Principal Coverage

Not all Databricks resources are supported by DAB for creating Service Principals. You’ll have to juggle between DAB and other tools to manage your identities. What consistency!

8. Technology Mixing as Soon as You Leave Databricks Perimeter

Real scenario: your Databricks workflow needs to read data from Kafka or OpenSearch. You need an AWS IAM role (let’s remember that Databricks only works with AWS, Azure & GCP).

With Terraform? Everything in the same code: IAM role, policies, Databricks job, permissions. A single terraform apply.

With DAB? You have to mix DAB for Databricks and Terraform (or other) for IAM. Two tools, two workflows, two states, two CI/CDs. A nightmare.

9. No Native Multi-Environment Mono-Repo

Impossible natively to have a mono-repo that manages deployment artifacts for all your environments. Impossible for repo A to call a DAB artifact published in repo B.

With Terraform? Remote modules, Terraform registries, everything has been there for years.

┌─────────────────────────────────────────┐
│         With Terraform                   │
│                                          │
│  Repo A (modules)                        │
│    └─ modules/                           │
│         ├─ databricks-job/               │
│         └─ databricks-cluster/           │
│                                          │
│  Repo B (prod)                           │
│    └─ main.tf                            │
│         └─ module "job" {                │
│             source = "git::repo-a"       │
│           }                              │
└─────────────────────────────────────────┘

┌─────────────────────────────────────────┐
│         With DAB                         │
│                                          │
│  Repo A                                  │
│    └─ databricks.yml                     │
│         (copy-paste required)            │
│                                          │
│  Repo B                                  │
│    └─ databricks.yml                     │
│         (code duplication)               │
│                                          │
│  ❌ No reuse possible                    │
└─────────────────────────────────────────┘

10. A Dedicated CI/CD to Create and Maintain

You’ll be forced to create a specific CI/CD for DAB. While your other infrastructure components already use Terraform with tested and proven pipelines.

More complexity, more maintenance, more documentation to write.

So Why Does DAB Exist?

I’m convinced that Databricks is sincere in their approach: simplify deployment for Data Engineers and give them more autonomy. The intention is commendable.

But the result isn’t there. By wanting to create a “simpler” tool, they created a limited wrapper that restricts Terraform’s functionality without providing real added value. In the end, you lose flexibility, maintainability, and compatibility with the existing IaC ecosystem.

The fundamental problem remains the same: training a data team on Terraform takes one day. Working around DAB’s limitations will waste your DevOps team’s time.

Conclusion

Databricks Asset Bundles is a regression. It’s a limited Terraform wrapper that makes you lose functionality, flexibility, and maintainability, all for a “simplification” that isn’t one.

If your data teams don’t know how to use Terraform, train them. One day is enough. You’ll gain autonomy, quality, and consistency with the rest of your infrastructure.