Skip to content

feat: support external history tables #18194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

dqhl76
Copy link
Collaborator

@dqhl76 dqhl76 commented Jun 19, 2025

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

close: #18146
follow up: #18185

allow to configure storage for history tables

Configuration:

[log.history]
on = true

[log.history.storage]
type = "s3"

[log.history.storage.s3]
bucket = "testbucket"
root = "history_table_test"
endpoint_url = "http://127.0.0.1:9900"
access_key_id = "minioadmin"
secret_access_key = "minioadmin"

[[log.history.tables]]
table_name = "query_history"

Note:
When the storage config is change, the old tables will be dropped.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Jun 19, 2025
@dqhl76 dqhl76 marked this pull request as ready for review June 20, 2025 03:00
@dqhl76 dqhl76 requested a review from zhang2014 June 20, 2025 03:00
@zhang2014 zhang2014 marked this pull request as draft June 20, 2025 03:58
@dqhl76
Copy link
Collaborator Author

dqhl76 commented Jun 20, 2025

This PR support Internal history table automatically convert to External history table based on config. It will drop the old tables and create new.

However, for External to Internal, and different External storage config, it will return error and exit

Databend Query start failure, cause: InvalidConfig. Code: 2002, Text = [HISTORY-TABLES] Cannot convert external history table to internal table, please drop the tables and stage first..
Databend Query start failure, cause: InvalidConfig. Code: 2002, Text = [HISTORY-TABLES] Cannot change storage parameters of external history table, please drop the tables and stage first..

@dqhl76 dqhl76 force-pushed the external-storage-2 branch from 892158a to 5637ee8 Compare June 20, 2025 07:50
@dqhl76 dqhl76 marked this pull request as ready for review June 20, 2025 10:28
@dqhl76 dqhl76 requested a review from Copilot June 20, 2025 10:28
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for configuring external storage for history tables, enabling the conversion of internal history tables to external ones while providing the necessary tests and updated logic.

  • Implements external history table support via new configuration and connection handling in the query service.
  • Adds new tests and scripts to validate both internal and external history table behaviors.
  • Updates various modules (meta, global history log, alter table, external storage connection) to support the external storage configuration.

Reviewed Changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/logging/test-logs.sh Removed redundant log table check to fit with new external config.
tests/logging/test-history-tables.sh Added tests for external history tables.
tests/logging/history_table_external.toml New config file for external history table definition.
src/query/service/src/history_tables/meta.rs Added asynchronous semaphore acquisition helper.
src/query/service/src/history_tables/global_history_log.rs Extended prepare logic to support external storage and reset checks.
src/query/service/src/history_tables/external.rs New module for external storage connection and URI generation.
src/query/service/src/history_tables/alter_table.rs Updated table alteration and reset checks for external configuration.
src/common/tracing/src/predefined_tables/* Minor updates to export new functions.
src/binaries/query/entry.rs Updated initialization error handling for history tables.
.github/actions/test_logs/action.yml Enhanced CI action to setup and test external storage configurations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature: allow separate storage configuration for History Tables
1 participant