Added BigQuery Metastore Catalog #2068
Conversation
rambleraptor
force-pushed
the
metadata_store
branch
from
3c94c83 to
e8adf49
Compare
|
@rambleraptor user here, I tried to use bigquery metastore recently when it was announced it has added rest API interface, I am just wondering, why you need to add support for pyiceberg if it is already using rest API ? |
rambleraptor
force-pushed
the
metadata_store
branch
from
e8adf49 to
87e0734
Compare
|
Edit: looks like that is supported via |
| GCP_PROJECT_ID = "gcp.project-id" | ||
| GCP_LOCATION = "gcp.location" | ||
| GCP_CREDENTIALS_LOCATION = "gcp.credentials-location" | ||
| GCP_CREDENTIALS_INFO = "gcp.credentials-info" |
There was a problem hiding this comment.
upstream defines these as
public static final String CREDENTIAL_FILE = "gcp.bigquery.credential-file";
public static final String PROJECT_ID = "gcp.bigquery.project-id";
public static final String GCP_LOCATION = "gcp.bigquery.location";
public static final String LIST_ALL_TABLES = "gcp.bigquery.list-all-tables";
| gcp_credentials = None | ||
| if credentials_location: | ||
| gcp_credentials = service_account.Credentials.from_service_account_file(credentials_location) | ||
| elif credentials_info_str: | ||
| try: |
There was a problem hiding this comment.
credentials_location will take precedence if the user specifies both -- do we want to allow a user to specify both? idrk
There was a problem hiding this comment.
are special credentials needed to run this in CI?
Rationale for this change
This PR brings BigQuery Metastore support to Python after it was merged into the Java implementation.
This allows Iceberg catalog functionality to be backed by BigQuery. It supports creating/deleting/listing namespaces (datasets in BigQuery terminology), creating/deleting/listing tables, and registering tables.
This is my first PR of size to iceberg-python, so any advice would be appreciated!
Are these changes tested?
Integration and unit tests included.
Are there any user-facing changes?
Introduces a new Catalog type.