Skip to content

Changelog (Stable)

All stable and notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Note

To view documentation for a specific version, please click the 'docs' badges under the specific changelog entry.

View Beta Releases View Archived Releases


[v0.11.0] - 2024-06-07

Documentation Status dbt Versions

The 'Jossy' release

This release is dedicated to our friend and colleague at Datavault who recently left the company. Your hard work, dedication, and significant contributions to this release and to AutomateDV over the years have been invaluable. You will always be remembered and greatly missed! Thank you ❤️

New

dbt Versions

👍 Officially tested on dbt 1.8.x (on all platforms except MS SQLServer, which does not yet support 1.8.x)

All Platforms

  • QUALIFY Statement Usage: Improved across all applicable platforms. Previously, the in-line WINDOW + QUALIFY syntax was not used correctly; instead, columns were materialized before applying QUALIFY. This issue has now been resolved, resulting in a noticeable performance improvement.

Google BigQuery & Databricks

  • Native Hashing now available , hashes no longer stored as strings! docs
  • Hashes now correctly use the BYTES type on BigQuery (instead of STRING)
  • Hashes now correctly use the BINARY type on Databricks (instead of STRING)

⏩ Enable this with the new enable_native_hashes global variable.
Note this behaviour is opt-in to avoid forcing a breaking change

Fixed

All Platforms

  • Fix for multi-column src_pk fields not working with Ghost Records correctly (#232)
  • Fixed an issue introduced in 0.10.0 which caused a column not found error during incremental satellite loads (#227)
  • Fixed an issue where setting enable_ghost_records to false resulted in the use of local Ghost Records with incorrect values in PITs (#196)
  • Fixed an issue where some loading edge-cases would trigger a cartesian JOIN in Effectivity Satellites
  • Fixed an edge case for duplicate ghost records during base loads in Satellites

Databricks

  • Fixed #219 as a result of the new Native Hashing functionality described above.

Microsoft SQLServer

  • Resolved an issue caused by the transition from CONCAT_WS to CONCAT in v0.9.1 (#188)
  • CONCAT_WS is now correctly used on Databricks, MS SQLServer, Snowflake, and Postgres. For Google BigQuery, which does not support CONCAT_WS, CONCAT continues to be used.

Deprecations

PITs & Bridges

We are planning a major overhaul and re-implementation of these macros to address performance issues and ensure compliance with DV 2.0 standards. They are currently not meeting their intended purposes effectively.

Custom Materializations

In version 0.10.0, we introduced new functionality that replaces the vault_insert_by_x materializations (improved satellite loading etc.). These materializations were initially designed to provide an option for rapid iterative development of incremental loading patterns in local environments for development and testing, allowing users to bypass the need for a comprehensive PSA or delta-loading solution. In this release, v0.11.0, we are deprecating the vault_insert_by_x materializations to encourage the use of more robust solutions.


[v0.10.2] - 2024-02-27

Documentation Status dbt Versions

New

dbt Versions

👍 Officially tested on dbt 1.7.x

All Platforms

  • A new SHA-1 Hashing option is now available for all platforms!* docs
*Not available on Postgres due to requiring pgcrypto extension

Fixed

All Platforms

  • Fixed an issue where the payload-exclusion feature was not enable for payload definitions in Multi-Active Satellites #217
  • Fixed an issue where in some cases duplicates would be loaded into Satellites #221

SQLServer/Postgres

  • Fixed multiple SQL code case inconsistencies causing compilation issues in SQLServer (#211, #209)

Notes

  • dbt SQLServer 1.7.x has caused the vault_insert_by_x materialisations provided by AutomateDV to behave inconsistently - both materialisations have been temporarily disabled in SQLServer only and will raise an error, and output an error message when used on SQLServer only.

[v0.10.1] - 2023-08-28

Documentation Status dbt Versions

Fixes

All Platforms


[v0.10.0] - 2023-08-14

Documentation Status dbt Versions

New

All Platforms

Improved

All Platforms

Satellite loading limitations resolved 🎉🎉🎉


[v0.9.7] - 2023-07-19

Documentation Status dbt Versions

New

Databricks and Postgres

Databricks and Postgres are now fully supported in v0.9.7!

Fixes

All platforms

Error handling and error messages

  • Cases where escape characters are empty now correctly use the platform default instead.
  • Cases where too many iterations (>100,000) would occur for custom vault_insert_by_x materialisations on SQLServer now raise an error.

Postgres

  • Fixed a hashing length bug (#176)

[v0.9.6] - 2023-05-16

Documentation Status dbt Versions

The rebrand update! - dbtvault is now AutomateDV

Changes

  • Macros are now called using automate_dv instead of dbtvault. e.g. automate_dv.hub(...)
  • The default system_record_value is now AUTOMATE_DV_SYSTEM instead of DBTVAULT_SYSTEM

[v0.9.5] - 2023-03-22

Documentation Status dbt Versions

Fixes

All platforms

  • Added error handling for when the number of iterations in vault_insert_by_x exceeds 100,000 (#175)
  • Fixed a regression in PITs where an incorrect join was causing a performance hit and in some cases, incorrect data
  • Fixed an issue causing 'LOADING...' log messages to appear when running dbt docs generate or dbt docs serve
  • Fixed a bug in the vault_insert_by_period materialisation affecting executions with 'hour' as the period (#178)

SQLServer

  • Fixed a minor casing issue in the SQLServer eff_sat macro (#182)

Databricks

  • Fixed an issue related to #183 but for MD5 hashing in Databricks

[v0.9.4] - 2023-02-16

Documentation Status

This is a minor hotfix update. More bug fixes to come soon! 😄

Fixes

  • Binary type not defaulting correctly (Snowflake) (#183)

[v0.9.3] - 2023-01-27

Documentation Status

Fixes

  • Updated packages.yml for compatibility with dbt-utils 1.0.0

Notes

  • Fully tested (and passing) with dbt-utils 1.0.0 and dbt 1.3.2

Thank you to our community

Thank you to all those who were being patient for this release. The delay was due to us wanting to release this with a few other bug fixes and new features. This additional content is being released at a later date so that we could get this dbt-utils fix out to our community sooner.


[v0.9.2] - 2022-12-22

Documentation Status

Fixes

  • Hotfixes for issues with Ghost Record creation under certain circumstances (#173,#174)

In other news: Happy Holidays!


[v0.9.1] - 2022-12-16

Documentation Status

New

New Features:

🌟 Ghost Records -> docs

🌟 Hashing Casing config (#123) -> Docs

dbt Versions

👍 dbt support updated to 1.3.x

👍 dbt-utils support updated to 0.9.x (1.0.x will be officially supported shortly)

Fixes

Escaping

We have made significant changes to how escaping now works as per #168. We believe this will fix the bugs collected in this master issue.

Whilst we have tested this extensively, we cannot yet be 100% this has fixed every edge case. Please bear with us as we collect community feedback. We welcome your feedback on this!

Related issues:

Casing

In addition to the above, we have also done an overhaul of casing in our templates. Users should now not experience any unwanted casing changes. As above, please provide feedback if any issues are found!

Related issues:

Other

  • Fixed a few edge cases where excludes for payload and hashdiffs would not work as expected

Behind the scenes

  • Major re-factor of Hashing to improve maintainability, readability and extensibility. The functionality remains the same and should not affect users

Docs

  • Split best practises into separate pages for ease of navigation and to reduce clutter
  • Moved old release notes to a new "archived" releases page

[v0.9.0] - 2022-09-13

Documentation Status

New Platform Support

Databricks

✨ Staging

✨ Hubs

✨ Links

✨ Satellites

Postgres**

✨ Staging

✨ Hubs

✨ Links

✨ Satellites

Note

There are currently limitations on Postgres. Click here for more details

New Features

Introducing:

🌟 Payload column exclusion: Satellite's payload can now be configured to select all columns, except a user-defined list using an exclude configuration (Datavault-UK/automate-dv#128)

🌟 Null Business Key Handling: Now users can provide a configuration in their staging tables to handle NULL keys elegantly, according to business needs (Datavault-UK/automate-dv#133)

🌟 Extra Columns Parameter: All table macros now provide a src_extra_columns parameter which allows users to add extra columns outside the standard template for business needs.

🌟 More logging: Hubs and Links now provide additional logging about the number of sources they are loading from. Minor but helpful! This is our first step towards giving our users more information.

Fixes

✅ Fixed an edge case for vault_insert_by_period when the staging table and the target table were in different databases (Datavault-UK/automate-dv#121)

✅ Removed the uppercase conversion in the staging macro (Datavault-UK/automate-dv#122, Datavault-UK/automate-dv#134)

✅ Fixed an issue where duplicate records (same hashdiff) would sometimes be loaded into a Satellite (Datavault-UK/automate-dv#126)

✅ Disabled automatic column name escaping in derived columns when using the stage() macro. Escaping can now be configured on a case-by-case basis for each column to escape when they are reserved words etc. (Datavault-UK/automate-dv#114, Datavault-UK/automate-dv#141)

Breaking changes

Behind the scenes

  • Re-factor PIT and Bridge macros to ensure better maintainability and readability

Docs

  • Moved stage configuration details to a new page
  • Updated packages behind the scenes for security and bug fixes
  • Created landing pages for sections, which should make navigation easier, e.g. getting started is now the home page when clicking 'Tutorials' in the menu, instead of having to click twice.

Thanks

View on GitHub


[v0.8.3] - 2022-05-10

Documentation Status

All existing macros are now supported by all platforms!

New

General

  • dbt 1.1.x support

Google BigQuery and MS SQL Server

Fixed

Effectivity Satellites
  • Fixed an issue affecting auto-end-dating in flip-flop situations eff_sat (#115)
Staging
  • Fixed an issue where hashed columns with lower-case columns provided to an exclude_columns config, behaved incorrectly (#110)

[v0.8.2] - 2022-03-14

Documentation Status

More Google BigQuery and MS SQL Server support, plus fixes!

New

Google BigQuery and MS SQL Server

See our Platform support matrix for more details.

Fixed

  • Fixed a bug where vault_insert_by_period would give an error during incremental loads (#108)
  • Fixed vault_insert_by_x issues for MS SQL Server
  • Fixed (increased) datetime precision in max_datetime for Google BigQuery

[v0.8.1] - 2022-02-22

Documentation Status

HOTFIX RELEASE

Fixed

  • Fixed a bug where vault_insert_by_rank unintentionally used logic from vault_insert_by_period when in full-refresh mode or replacing an existing view.

[v0.8.0] - 2022-02-21

Documentation Status

This is a big release for AutomateDV. It's the first time we are releasing support for new platforms! Please welcome to the AutomateDV family, Google BigQuery and MS SQL Server!

This is just the start, and we're excited to bring even more platforms (and further support for existing platforms) to you in the future!

New

Google BigQuery and MS SQL Server

  • Hubs (hub macro)
  • Links (link macro)
  • Satellites (sat macro)

New

Platform support matrix

All platforms

  • Column Escaping (#28 , #23)
  • Docs: AutomateDV now automatically surrounds all column names with quotes. This is to allow for columns with reserved words, spaces, and other oddities. The type of quotes is configurable, please refer to the docs linked above.

[v0.7.9] - 2021-12-13

Documentation Status

Dependencies

Breaking

  • dbt v1.0.0 Support
  • Updated to dbt_utils v0.8.0 (for dbt 1.0.0 compatibility)

New

Table structures

Materialisations

  • Custom materialisation for PITs Docs
  • Custom materialisation for Bridges Docs

Behind the Scenes

  • More test coverage for incremental loading.
  • Improved consistency and support for Composite PKs.
  • Significantly simplified Multi-Active Satellite (MAS) logic.

Bug Fixes

  • Multi-Active Satellite record duplication under some circumstances #50

[v0.7.8] - 2021-10-25

Documentation Status

Dependencies

  • dbt 0.21.0 support
  • dbt utils package dependency now has a version range (sorry!)

Fixes

  • Effectivity Satellites with auto-end-dating off now handle the use case where records may already be end-dated in the staging layer, as a result of loading data 'manually' end-dated by business rules.

Features

Rank column configurations in stage macro (ranked_columns):

  • Provide ASC or DESC for an order_by column Read More
  • Configure the ranking to use DENSE_RANK() or RANK() Read More

Configuration for hash strings

Read More

  • Concatenation string can now be user defined
  • Null placeholder string can now be user defined

[v0.7.7] - 2021-08-24

  • Re-release of v0.7.6.1 to ensure deployment to dbt Hub

[v0.7.6.1] - 2021-07-14

  • Hotfix for 0.7.6 to remove unintentionally added macros from the beta branch. #36

Installing

Note: This version cannot be installed via dbt hub, please install as follows:

1
2
3
packages:
  - git: "https://github.com/Datavault-UK/automate-dv.git"
    revision: v0.7.6.1

[v0.7.6] - 2021-07-13

Documentation Status

  • Updated to dbt 0.20.0 and incorporated adapter.dispatch changes (#32)

[v0.7.5] - 2021-06-10

Documentation Status

New structures

Bug Fixes

  • Fixed a bug where an Effectivity Satellite with multiple DFKs or SDKs would incorrectly handle changes in the corresponding link records, meaning one-to-many relationships were not getting handled as intended.

Improvements

  • Added support for multiple order_by or partition_by columns when creating ranked columns in the stage or ranked_columns macros.
  • Performance improvement for the Satellite macro, which aims to reduce the number of records handled in the initial selection of records from the source data.

[v0.7.4] - 2021-03-27

Documentation Status

Bug Fixes

  • Fixed NULL handling bugs in Hubs, Links and Satellites (#26)
  • Fixed a bug where Effectivity Satellites would incorrectly end-date (with auto-end-dating enabled) records other than the latest, resulting in duplicate end-date records for previously end-dated records.

Improvements

  • Added check for matching primary key when inserting new satellite records in the sat macro. This removes the requirement to add the natural key to the hashdiff, but it is still recommended. Read More

Quality of Life

  • Payload in Transactional (Non-Historised) Links now optional
  • Effective From in Satellites now optional

[v0.7.3] - 2021-01-28

Documentation Status

  • Updated dbt to v0.19.0
  • Updated dbt utils to 0.6.4