December 12, 2024 Lakehouse Studio Product Update Release Notes

This release (Release 2024.12.12, v1.5) introduces a series of new features, enhancements, and bug fixes. Please note that these updates will be rolled out in phases to the following regions, and will be completed within one to two weeks from the release date, depending on your region.

Tencent Cloud Beijing
Tencent Cloud Shanghai
Tencent Cloud Guangzhou
Alibaba Cloud Shanghai
Alibaba Cloud Singapore
AWS Beijing
AWS Singapore

Breaking Changes

Permission Change: This release implements fine-grained permission control optimization. The existing instance_sre and workspace_sre roles no longer have task development and publishing permissions across all workspaces; their permissions have been reduced to read-only. After this adjustment, only workspace_admin and workspace_dev roles can create, edit, configure scheduling properties, and submit/publish within a workspace. This change does not affect existing role accounts' data access permissions, nor does it affect the normal execution of scheduled tasks.
Data Quality: Data quality check result data will now be retained for a maximum of 3 months.

New Features

Product-Wide: Added visual development, scheduling orchestration, and operations monitoring support for Databricks task nodes (SQL and Notebook).
Data Sources: Added Amazon Redshift data source, supporting data writing via batch sync tasks.
Data Sync: Unified resource pool — sync-type compute clusters can now be configured for use in tasks.
Task Development: Added intelligent parsing for scheduling dependencies and task outputs of batch sync tasks.
Monitoring & Alerting: Added monitoring and alerting for schedule delay metrics of periodic task instances.
Compute Cluster: Added sync-type compute clusters, which can be selected for use in batch and real-time data sync tasks.

Improvements

Product-Wide: Optimized feature permission points for built-in roles. Based on the latest built-in roles, functional permission restrictions have been adjusted across different product scenarios (including workspace, development, task operations, data sources, clusters, job history, etc.).
Account Center: Optimized the bar chart display of the 30-day billing summary on the account home page.
Data Sync: For scenarios where the source table has more fields than the target table, redundant source fields are now listed.
Data Sync: In batch sync task configuration, field mapping now defaults to same-name column mapping.
Data Sync: In multi-table real-time sync mirror mode, all databases and tables are now unselected by default.
Task Development: Page loading optimization — resolved page lag issues when opening batch integration and task group DAGs.
Task Development: Added horizontal layout mode for task group DAGs, improving task arrangement and page utilization.
Task Development: Fixed an issue where entering parameter values became abnormal due to excessively long parameter names when running code with parameters in development.
Operations Center: The instance operations search box now supports searching for temporary instances by task name.
Operations Center: Optimized batch sync task log information — complete error logs are now exposed to help locate key diagnostic information.
Compute Cluster: Supports more fine-grained specification adjustments to improve resource utilization. Resource specification is now expressed in numeric form with CRU (Compute Resource Unit) instead of codes like XSmall and Large, for example: 1CRU, 2CRU, etc.

Bug Fixes

Data Sync: Fixed an issue where multi-table real-time sync tasks would error after being deleted on another page; after task deletion, the page now redirects to a new page by default.
Data Sync: Fixed an issue in batch sync tasks where tar.gz files were imported successfully but the data was incorrect.
Task Development: Fixed an issue in the Operations Center instance detail page where clicking the "Error Status" category failed to navigate correctly when the task lineage had many levels.
Task Development: Fixed an issue where scheduling parameters and external parameters interfered with each other in the task development interface.

Known Limitations

Sync-type compute clusters currently do not support viewing cluster load and usage information.