Changelog (Stable)¶

All stable and notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Note

To view documentation for a specific version, please click the 'docs' badges under the specific changelog entry.

View Beta Releases View Archived Releases

[v0.11.2] - 2025-03-12¶

Hot fixes¶

Fixed an issue introduced in v0.11.1 which effectively forced the src_eff attribute to be required unintentionally #255

[v0.11.1] - 2025-03-06¶

New¶

dbt Versions¶

👍 Officially tested on dbt v1.9.x Note: AutomateDV running with dbt v1.9 is currently untested on MS SQLServer due to compatibility issues. Use at your own risk.

All Platforms¶

Fixed an edge case where data would be ignored when loading Satellites #233
Added improved support for out-of-sequence loading by making use of the src_eff parameter when provided #253

[v0.11.0] - 2024-06-07¶

The 'Jossy' release¶

This release is dedicated to our friend and colleague at Datavault who recently left the company. Your hard work, dedication, and significant contributions to this release and to AutomateDV over the years have been invaluable. You will always be remembered and greatly missed! Thank you ❤️

New¶

dbt Versions¶

👍 Officially tested on dbt 1.8.x (on all platforms except MS SQLServer, which does not yet support 1.8.x)

All Platforms¶

QUALIFY Statement Usage: Improved across all applicable platforms. Previously, the in-line WINDOW + QUALIFY syntax was not used correctly; instead, columns were materialized before applying QUALIFY. This issue has now been resolved, resulting in a noticeable performance improvement.

Google BigQuery & Databricks¶

Native Hashing now available, hashes no longer stored as strings! docs
Hashes now correctly use the BYTES type on BigQuery (instead of STRING)
Hashes now correctly use the BINARY type on Databricks (instead of STRING)

Enable this with the new enable_native_hashes global variable.
Note this behaviour is opt-in to avoid forcing a breaking change

Fixed¶

All Platforms¶

Fix for multi-column src_pk fields not working with Ghost Records correctly (#232)
Fixed an issue introduced in 0.10.0 which caused a column not found error during incremental satellite loads (#227)
Fixed an issue where setting enable_ghost_records to false resulted in the use of local Ghost Records with incorrect values in PITs (#196)
Fixed an issue where some loading edge-cases would trigger a cartesian JOIN in Effectivity Satellites
Fixed an edge case for duplicate ghost records during base loads in Satellites

Databricks¶

Fixed #219 as a result of the new Native Hashing functionality described above.

Microsoft SQLServer¶

Resolved an issue caused by the transition from CONCAT_WS to CONCAT in v0.9.1 (#188)
CONCAT_WS is now correctly used on Databricks, MS SQLServer, Snowflake, and Postgres. For Google BigQuery, which does not support CONCAT_WS, CONCAT continues to be used.

Deprecations¶

PITs & Bridges¶

We are planning a major overhaul and re-implementation of these macros to address performance issues and ensure compliance with DV 2.0 standards. They are currently not meeting their intended purposes effectively.

Custom Materializations¶

In version 0.10.0, we introduced new functionality that replaces the vault_insert_by_x materializations (improved satellite loading etc.). These materializations were initially designed to provide an option for rapid iterative development of incremental loading patterns in local environments for development and testing, allowing users to bypass the need for a comprehensive PSA or delta-loading solution. In this release, v0.11.0, we are deprecating the vault_insert_by_x materializations to encourage the use of more robust solutions.

[v0.10.2] - 2024-02-27¶

New¶

dbt Versions¶

Officially tested on dbt 1.7.x

All Platforms¶

A new SHA-1 Hashing option is now available for all platforms!* docs

*Not available on Postgres due to requiring pgcrypto extension¶

Fixed¶

All Platforms¶

Fixed an issue where the payload-exclusion feature was not enable for payload definitions in Multi-Active Satellites #217
Fixed an issue where in some cases duplicates would be loaded into Satellites #221

SQLServer/Postgres¶

Fixed multiple SQL code case inconsistencies causing compilation issues in SQLServer (#211, #209)

Notes¶

dbt SQLServer 1.7.x has caused the vault_insert_by_x materialisations provided by AutomateDV to behave inconsistently - both materialisations have been temporarily disabled in SQLServer only and will raise an error, and output an error message when used on SQLServer only.

[v0.10.1] - 2023-08-28¶

Fixes¶

All Platforms¶

Fixed the case where repeating a load would cause duplicates when using the Satellites as released in 0.10.0 (#207)
Implemented as a new Behaviour Flag for Satellites apply_source_filter
- Read more in loading best practises
- Read more on sat() macro Behaviour Flags

[v0.10.0] - 2023-08-14¶

New¶

All Platforms¶

Reference Tables (ref_table macro)
- Tutorial

Improved¶

All Platforms¶

Satellite loading limitations resolved

Migration Guide
Updated Loading Guide

[v0.9.7] - 2023-07-19¶

New¶

Databricks and Postgres¶

Databricks and Postgres are now fully supported in v0.9.7!

Transactional Links (t_link macro)
Effectivity Satellites (eff_sat macro)
Multi-active Satellites (ma_sat macro)
Extended Tracking Satellites (xts macro)
Point in Time tables (PITs) (pit macro)
Bridges (bridge macro)

Fixes¶

All platforms¶

Error handling and error messages¶

Cases where escape characters are empty now correctly use the platform default instead.
Cases where too many iterations (>100,000) would occur for custom vault_insert_by_x materialisations on SQLServer now raise an error.

Postgres¶

Fixed a hashing length bug (#176)

[v0.9.6] - 2023-05-16¶

The rebrand update! - dbtvault is now AutomateDV¶

Changes¶

Macros are now called using automate_dv instead of dbtvault. e.g. automate_dv.hub(...)
The default system_record_value is now AUTOMATE_DV_SYSTEM instead of DBTVAULT_SYSTEM

[v0.9.5] - 2023-03-22¶

Fixes¶

All platforms¶

Added error handling for when the number of iterations in vault_insert_by_x exceeds 100,000 (#175)
Fixed a regression in PITs where an incorrect join was causing a performance hit and in some cases, incorrect data
Fixed an issue causing 'LOADING...' log messages to appear when running dbt docs generate or dbt docs serve
Fixed a bug in the vault_insert_by_period materialisation affecting executions with 'hour' as the period (#178)

SQLServer¶

Fixed a minor casing issue in the SQLServer eff_sat macro (#182)

Databricks¶

Fixed an issue related to #183 but for MD5 hashing in Databricks

[v0.9.4] - 2023-02-16¶

This is a minor hotfix update. More bug fixes to come soon!

Fixes¶

Binary type not defaulting correctly (Snowflake) (#183)

[v0.9.3] - 2023-01-27¶

Fixes¶

Updated packages.yml for compatibility with dbt-utils 1.0.0

Notes¶

Fully tested (and passing) with dbt-utils 1.0.0 and dbt 1.3.2

Thank you to our community¶

Thank you to all those who were being patient for this release. The delay was due to us wanting to release this with a few other bug fixes and new features. This additional content is being released at a later date so that we could get this dbt-utils fix out to our community sooner.

[v0.9.2] - 2022-12-22¶

Fixes¶

Hotfixes for issues with Ghost Record creation under certain circumstances (#173,#174)

In other news: Happy Holidays!

[v0.9.1] - 2022-12-16¶

New¶

New Features:¶

Ghost Records -> docs

Hashing Casing config (#123) -> Docs

dbt Versions¶

dbt support updated to 1.3.x

dbt-utils support updated to 0.9.x (1.0.x will be officially supported shortly)

Fixes¶

Escaping¶

We have made significant changes to how escaping now works as per #168. We believe this will fix the bugs collected in this master issue.

Whilst we have tested this extensively, we cannot yet be 100% this has fixed every edge case. Please bear with us as we collect community feedback. We welcome your feedback on this!

Related issues:

Casing¶

In addition to the above, we have also done an overhaul of casing in our templates. Users should now not experience any unwanted casing changes. As above, please provide feedback if any issues are found!

Related issues:

Other¶

Fixed a few edge cases where excludes for payload and hashdiffs would not work as expected

Behind the scenes¶

Major re-factor of Hashing to improve maintainability, readability and extensibility. The functionality remains the same and should not affect users

Docs¶

Split best practises into separate pages for ease of navigation and to reduce clutter
Moved old release notes to a new "archived" releases page

[v0.9.0] - 2022-09-13¶

New Platform Support¶

Databricks¶

Staging

Hubs

Links

Satellites

Postgres**¶

Staging

Hubs

Links

Satellites

Note

There are currently limitations on Postgres. Click here for more details

New Features¶

Introducing:

Payload column exclusion: Satellite's payload can now be configured to select all columns, except a user-defined list using an exclude configuration (Datavault-UK/automate-dv#128)

Null Business Key Handling: Now users can provide a configuration in their staging tables to handle NULL keys elegantly, according to business needs (Datavault-UK/automate-dv#133)

Extra Columns Parameter: All table macros now provide a src_extra_columns parameter which allows users to add extra columns outside the standard template for business needs.

More logging: Hubs and Links now provide additional logging about the number of sources they are loading from. Minor but helpful! This is our first step towards giving our users more information.

Fixes¶

Fixed an edge case for vault_insert_by_period when the staging table and the target table were in different databases (Datavault-UK/automate-dv#121)

Removed the uppercase conversion in the staging macro (Datavault-UK/automate-dv#122, Datavault-UK/automate-dv#134)

Fixed an issue where duplicate records (same hashdiff) would sometimes be loaded into a Satellite (Datavault-UK/automate-dv#126)

Disabled automatic column name escaping in derived columns when using the stage() macro. Escaping can now be configured on a case-by-case basis for each column to escape when they are reserved words etc. (Datavault-UK/automate-dv#114, Datavault-UK/automate-dv#141)

Breaking changes¶

Read our 0.83 to 0.9.0 migration guide

Behind the scenes¶

Re-factor PIT and Bridge macros to ensure better maintainability and readability

Docs¶

Moved stage configuration details to a new page
Updated packages behind the scenes for security and bug fixes
Created landing pages for sections, which should make navigation easier, e.g. getting started is now the home page when clicking 'Tutorials' in the menu, instead of having to click twice.

Thanks¶

View on GitHub

[v0.8.3] - 2022-05-10¶

All existing macros are now supported by all platforms!

New¶

General¶

dbt 1.1.x support

Google BigQuery and MS SQL Server¶

PITs (pit macro)
Bridges (bridge macro)

Fixed¶

Effectivity Satellites¶

Fixed an issue affecting auto-end-dating in flip-flop situations eff_sat (#115)

Staging¶

Fixed an issue where hashed columns with lower-case columns provided to an exclude_columns config, behaved incorrectly (#110)

[v0.8.2] - 2022-03-14¶

More Google BigQuery and MS SQL Server support, plus fixes!

New¶

Google BigQuery and MS SQL Server¶

T-Links (t_link macro)
Effectivity Satellites (eff_sat macro)
Multi-Active Satellites (ma_sat macro)
Extended Tracking Satellites (xts macro)

See our Platform support matrix for more details.

Fixed¶

Fixed a bug where vault_insert_by_period would give an error during incremental loads (#108)
Fixed vault_insert_by_x issues for MS SQL Server
Fixed (increased) datetime precision in max_datetime for Google BigQuery

[v0.8.1] - 2022-02-22¶

HOTFIX RELEASE

Fixed¶

Fixed a bug where vault_insert_by_rank unintentionally used logic from vault_insert_by_period when in full-refresh mode or replacing an existing view.

[v0.8.0] - 2022-02-21¶

This is a big release for AutomateDV. It's the first time we are releasing support for new platforms! Please welcome to the AutomateDV family, Google BigQuery and MS SQL Server!

This is just the start, and we're excited to bring even more platforms (and further support for existing platforms) to you in the future!

New¶

Google BigQuery and MS SQL Server¶

Hubs (hub macro)
Links (link macro)
Satellites (sat macro)

New

Platform support matrix

All platforms¶

Column Escaping (#28 , #23)
Docs: AutomateDV now automatically surrounds all column names with quotes. This is to allow for columns with reserved words, spaces, and other oddities. The type of quotes is configurable, please refer to the docs linked above.

[v0.7.9] - 2021-12-13¶

Dependencies¶

Breaking¶

dbt v1.0.0 Support
- Support for earlier dbt versions (dbt <=0.21.0) removed. This is as a result of the upgrade to dbt v1.0.0. How do I upgrade my project?
Updated to dbt_utils v0.8.0 (for dbt 1.0.0 compatibility)

New¶

Table structures¶

Point in Time tables Tutorial - Macro docs
Bridge tables Tutorial - Macro docs
Extended Tracking Satellites (XTS) Tutorial - Macro docs

Materialisations¶

Custom materialisation for PITs Docs
Custom materialisation for Bridges Docs

Behind the Scenes¶

More test coverage for incremental loading.
Improved consistency and support for Composite PKs.
Significantly simplified Multi-Active Satellite (MAS) logic.

Bug Fixes¶

Multi-Active Satellite record duplication under some circumstances #50

[v0.7.8] - 2021-10-25¶

Dependencies¶

dbt 0.21.0 support
dbt utils package dependency now has a version range (sorry!)

Fixes¶

Effectivity Satellites with auto-end-dating off now handle the use case where records may already be end-dated in the staging layer, as a result of loading data 'manually' end-dated by business rules.

Features¶

Rank column configurations in stage macro (ranked_columns):¶

Provide ASC or DESC for an order_by column Read More
Configure the ranking to use DENSE_RANK() or RANK() Read More

Configuration for hash strings¶

[v0.7.7] - 2021-08-24¶

Re-release of v0.7.6.1 to ensure deployment to dbt Hub

[v0.7.6.1] - 2021-07-14¶

Hotfix for 0.7.6 to remove unintentionally added macros from the beta branch. #36

Installing¶

Note: This version cannot be installed via dbt hub, please install as follows:

packages:
  - git: "https://github.com/Datavault-UK/automate-dv.git"
    revision: v0.7.6.1

[v0.7.6] - 2021-07-13¶

Updated to dbt 0.20.0 and incorporated adapter.dispatch changes (#32)

[v0.7.5] - 2021-06-10¶

New structures¶

Multi-Active Satellites Read More

Bug Fixes¶

Fixed a bug where an Effectivity Satellite with multiple DFKs or SDKs would incorrectly handle changes in the corresponding link records, meaning one-to-many relationships were not getting handled as intended.

Improvements¶

Added support for multiple order_by or partition_by columns when creating ranked columns in the stage or ranked_columns macros.
Performance improvement for the Satellite macro, which aims to reduce the number of records handled in the initial selection of records from the source data.

[v0.7.4] - 2021-03-27¶

Bug Fixes¶

Fixed NULL handling bugs in Hubs, Links and Satellites (#26)
Fixed a bug where Effectivity Satellites would incorrectly end-date (with auto-end-dating enabled) records other than the latest, resulting in duplicate end-date records for previously end-dated records.

Improvements¶

Added check for matching primary key when inserting new satellite records in the sat macro. This removes the requirement to add the natural key to the hashdiff, but it is still recommended. Read More

Quality of Life¶

Payload in Transactional (Non-Historised) Links now optional
Effective From in Satellites now optional

[v0.7.3] - 2021-01-28¶

Updated dbt to v0.19.0
Updated dbt utils to 0.6.4