feat: refactor data masking module#589
Conversation
…ities - cloudbeaver 通过依赖注入集成数据脱敏 - Introduced `SQLResultMasker` interface for masking SQL results during execution. - Updated `CloudbeaverUsecase` to utilize the new `SQLResultMasker` for enhanced data privacy. - Refactored `buildTaskIdAssocDataMasking` to accept a `taskMaskingContext` for improved masking context management. - Integrated masking task checks within the GraphQL distributor to ensure sensitive data is appropriately handled during operations. - Enhanced the initialization of `CloudbeaverService` to include the new SQL result masking functionality.
- 新增配置脱敏任务的权限 - Introduced a new operation permission for masking audit with UID `700038`. - Updated the permission name for desensitization to "配置脱敏任务". - Enhanced localization support by adding descriptions and names for the new masking audit permission. - Refactored permission handling to accommodate the new permission type in various service components.
… repository - 从数据源配置中移除是否开启脱敏,移动到脱敏任务,由数据源是否开启脱敏任务判断是否开启脱敏 - Removed `IsMaskingSwitch` from `DBService`, `BizDBServiceArgs`, and related methods to streamline the configuration. - Introduced `MaskingTaskRepo` interface for managing masking task existence and status. - Updated `DBServiceUsecase` to include `maskingTaskRepo` for enhanced masking task management. - Adjusted various service methods to reflect the removal of masking switch logic, ensuring cleaner code and improved maintainability.
- Introduced a new method `GetURL` in the `ProxyTarget` struct to safely retrieve the URL as a string. - The method checks for a nil URL and returns an empty string if it is not set, enhancing the usability of the `ProxyTarget` type.
…k repository - 依赖注入 - Added `maskingConfigRepo` and `maskingTaskRepo` to the `DataExportWorkflowUsecase` for improved data masking capabilities. - Updated the constructor for `DataExportWorkflowUsecase` to include new dependencies, facilitating better management of masking configurations and tasks. - Introduced a new file `data_masking_ce.go` to define methods related to data masking, returning errors for unsupported operations in the current context. - Refactored `DMSService` to initialize the new masking configuration repository and update the data masking use case, ensuring a cohesive integration of data masking functionalities.
…g templates - 接口定义新增与调整 - Removed `IsEnableMasking` field from `DBService`, `UpdateDBService`, and `ImportDBService` to streamline configuration. - Introduced new API endpoints for managing masking rules and templates, enhancing data masking capabilities. - Added methods for listing, adding, updating, and deleting masking templates and sensitive data discovery tasks in the `DMSController`. - Updated routing to accommodate new masking functionalities, ensuring better organization and maintainability of the codebase.
- Revised descriptions for existing endpoints to enhance clarity, including responses for listing masking rules and approval requests. - Introduced new endpoints for managing masking approval requests, including listing pending requests and processing decisions. - Added definitions for new request and response structures related to masking templates and sensitive data discovery tasks in the Swagger documentation. - Improved organization of API routes under the `/masking` namespace for better maintainability.
- 移除不再需要的旧代码 - Removed `data_masking_ce.go`, `data_masking_ee.go`, and associated configuration files to streamline the codebase. - Eliminated `data_masking_conf_ee.yml` and `data_masking_rule_in_ee.yml` as part of the cleanup process. - Deleted `masking_ce.go`, `masking_ee.go`, and related service files to enhance maintainability and focus on core functionalities.
…overy task repositories - 补充依赖注入接口和空实现 - Added `DataExportMaskingConfigRepo` interface to define data masking configuration methods. - Implemented `SensitiveDataDiscoveryTaskRepo` with methods for checking task existence and listing task statuses. - Enhanced `dataMaskingUsecase` struct to include a discovery task use case for better task management.
- Changed the x-go-package path in Swagger files to reflect the new structure. - Removed unused properties from ListMemberRoleWithOpRange definition to streamline the API. - Added new properties for database service host and port in ListSensitiveDataDiscoveryTasksData to enhance data discovery capabilities. - Updated related Go structs to include new fields for database service host and port, ensuring consistency across the codebase.
- 新增接口,用于获取可以创建脱敏任务的数据源列表 - 调整接口,支持获取全局中脱敏支持的数据源类型 - Added a new endpoint to list creatable database services for sensitive data discovery tasks, allowing users to specify project UID and optional query parameters for pagination and filtering. - Introduced a function support filter in the existing global DB services tips endpoint to return database types based on specified functionality. - Updated Swagger documentation to reflect new API definitions and parameters, ensuring clarity and consistency across the API. - Enhanced related Go structs and service methods to support the new functionality, improving the overall data masking capabilities.
- 增加敏感数据扫描任务的终止状态 - Changed the x-go-package path in Swagger files to the new structure for consistency. - Added "STOPPED" status to the task status descriptions and enums in both Swagger JSON and YAML files. - Updated related Go constants and methods to include the new "STOPPED" status for sensitive data discovery tasks, enhancing the API's functionality.
…urations - 优化敏感数据扫描流程,不重复扫描用户已确认字段 - 排除Oracle系统表 - 区分用户更新和系统更新 - Updated the `ConfigureMaskingRules` method to accept a `currentUserUid` parameter, allowing for user-specific masking rule configurations. - Modified the `BatchUpsertDiscoveryResults` method to handle user updates distinctly from system updates, ensuring better control over masking configurations. - Introduced a new method `ConfigureMaskingRulesByUser` in the `MaskingRuleManagementUsecase` to facilitate user-specific rule management. - Enhanced the `SensitiveDiscoveryUsecase` to exclude previously configured columns during sensitive data discovery, improving the accuracy of the discovery process. - Updated the Oracle metadata collector to include additional default schemas for exclusion, refining the schema collection process.
…task management - 调整更新敏感数据扫描任务的接口,改为Action模式,简化状态流转。移除状态机 - 修复swagger定义错误 - Corrected enum values in Swagger files for consistency, changing "[data_masking]" to "data_masking". - Updated x-go-package paths to reflect the new structure across multiple definitions. - Enhanced the UpdateSensitiveDataDiscoveryTaskReq to use an action-based approach, replacing the task requirement with an action field for better clarity in task management. - Added new properties to ListMemberRoleWithOpRange for member group and operation permissions, improving the API's functionality.
PR Reviewer Guide 🔍(Review updated until commit 978b650)
|
PR Code Suggestions ✨Explore these optional code suggestions:
|
- Modified the pattern in the check-pr-files.yml to exclude _ee.md files from changes, enhancing the file change prevention mechanism.
|
Persistent review updated to latest commit 978b650 |
|
Failed to generate code suggestions for PR |
关联的 issue
https://github.com/actiontech/dms-ee/issues/736
link https://github.com/actiontech/dms-ee/pull/765
assign @LordofAvernus
描述你的变更
实现脱敏模块1期需求,主要功能包括:
测试
见⏩进展:数据脱敏1期 的产品验收和自测部分
确认项(pr提交后操作)
Tip
请在指定复审人之前,确认并完成以下事项,完成后✅
not_compatibleneed_update_doc