Destination Databricks: Support managed tables by suhomud · Pull Request #23965 · airbytehq/airbyte (original) (raw)

What

Destination Databricks: Support new data source type Managed tables.

How

In order to support Databricks Managed tables new DatabricksManagedTablesDestination.java is implemented based on AbstractJdbcDestination.java with a slight modification of JdbcSqlOperations. On the UI we can see the new data source type [Recommended] Managed tables without datasource property since it has to be modified on the Databricks side:

Additional changes

• In origin implementation the Databricks port was ignored for some reason. Updated DATABRICKS url in DatabaseDriver.java to custom port.
• spec.json was updated with new field database to specify a Databricks Catalog. So now it looks like:

• Some refactoring was added to navigate through connectors's code more easily.
• Added Utility clases to avoid code duplications.
• Nice Java 14 Records Class feature was added for DatabricksDestinationConfig.java.

Recommended reading order

1. spec.json
2. DatabricksManagedTablesDestination.java - The new datasource type based on AbstractJdbcDestination.java.
3. DatabricksSqlOperations.java
4. DatabricksStorageType.java
5. DatabaseDriver.java - New url to support custom port.
6. DatabricksDestinationConfig.java - Modified default schema to default instead of public since Databricks creates default schema when Catalog is created. Also changed it to Java record type which looks much better then all of those getters-setters staff.
7. DatabricksStorageConfigProvider.java - Renamed and refactored. Note: return switch is feature of Java 17.
8. DatabricksDestinationResolver.java
9. DatabricksExternalStorageBaseDestination.java - Renamed from DatabricksBaseDestination since now it is used by external data source type only.
10. DatabricksConstants.java - Made utility class with constants
11. DatabricksDatabaseUtil.java - Utility class with shared code
12. DatabricksS3DestinationAcceptanceTest.java - As discussed was disabled for now since it looks like the external tables are not fully working. Should be revisit after external data source will be fixed.
13. DatabricksUtilTest.java - Integration tests util. Note: there is an issue with retrieving the data which was magically fixed. See code comments to know the details. This fix exists only for integration tests
14. DatabricksManagedTablesDestinationAcceptanceTest.java - Note: if you want to run it locally you should take a new secret from google secret manager SECRET_DESTINATION-DATABRICKS__MANAGED_TABLES_CREDS. Additionally all Databricks secrets were updated.
15. Others changes are mostly related to new folder structure, re-naming, small refactoring changes.

🚨 User Impact 🚨

Since the spec.json has changed I think it should break change for existing connections if there is any.