Unify Catalog API methods to Async Style

[dantengsky]

Summary

Unify Catalog API methods to async, remove the internal block_on workaround.

• Background

  No offense, I am wondering why all read funcs in the trait Catalog are not asyncs, observing that there are several do_blocks in the implementation.

  Originally posted by @zhongzc in issue 1251, catalog as meta-store client & TableMetaCache #1212 (comment)

[BohuTANG]

Let me have a try.

[dantengsky]

Thanks bro

[BohuTANG]

I have try and found lots of codes need change to async, so thinking a question: what the benefit of changing to async is?
At present, it is better to keep it as it is 😢
cc: @dantengsky

[dantengsky]

yes, lots of code involved, let's postpone this issue

[dantengsky]

May I leave this issue open?

[BohuTANG]

May I leave this issue open?

Sure

[leiysky]

@BohuTANG @dantengsky

I have some concern on this.

According to my experience, the most usage of catalog is in optimizer, which is canonically synchronous component. Thus I have to rewrite the code into asynchronous or do block_on by myself.

So maybe it's better to provide only synchronous API in Catalog trait?

[BohuTANG]

@BohuTANG @dantengsky

I have some concern on this.

According to my experience, the most usage of catalog is in optimizer, which is canonically synchronous component. Thus I have to rewrite the code into asynchronous or do block_on by myself.

So maybe it's better to provide only synchronous API in Catalog trait?

Thank you for reminding @leiysky
For catalog, sync is more make sense here.
@dantengsky what do you think?

[dantengsky]

I am not sure about it.

On the one hand, the Catalog is backed by flight service(for the time being) which is async, thus it is easier/more natural for Catalog API to go with async style.

On the other hand:

• if the optimizer depends on Catalog heavily while searching for better plans (correct me if I am mistaken), it would be better to make catalog API sync.
• Since nothing prevents us from encapsulating an async service as sync APIs, and with cache enabled, we may have bot efficient impls and elegant APIs.

Also, It may depend on the way we provide statistical information to the optimizer (again, correct me if I am mistaken). for the time being, as we discussed before, I am considering embedding statistics into the table metadata, which is NOT directly embedded in the MetaStore, but saved in object storage. Maybe the optimizer should consult the index layer while planning. I mean, the extended/enhanced index layer, which can provide table statistics (min, max, cardinality, row num, size, etc).

[leiysky]

• if the optimizer depends on Catalog heavily while searching for better plans (correct me if I am mistaken), it would be better to make catalog API sync.

The workload of Catalog accessing can be catagorized as:

• Name resolution, only need logical information about database(e.g. tables, column definitions, functions, snapshots if we support syntax like AS OF [timestamp]). Generally, logical information lookup won't become a bottleneck.
• Partition pruning, which is performed during plan enumuration. Depending on design of storage, the cost of partition information accessing can vary a lot. This is the part we should take care of.

Take micro-partition in Snowflake as an example, each micro-partition is in size between 50MB-500MB. Thus for a table in size of 1TB, there are about 20,000 micro-partitions at most. If we want to maintain min/max/count/card information for each column(assume that all of the columns are int32), says we have 100 columns in a table, so each micro-partition will take 4 * 4 * 100 = 1600 bytes. Therefore, it will take at most 32MB to store all of partition information in memory, which is not unacceptable.

If datafuse-store isn't a "shocking" product, the statistic information size won't be much different from that of Snowflake.

Besides, the qps of each user is limited to tens, which means it won't be much stressful for metadata service.

I am considering embedding statistics into the table metadata, which is NOT directly embedded in the MetaStore, but saved in object storage.

A brainstorm, maybe we can store statistics data in internal system table, and always cache them in storage node, so we can build our own statistic component(with much complicated statistics information like histogram) on top of it.

[BohuTANG]

First, let's determine what's topic discusses?

1. the catalog async/sync style
   or
2. the catalog design

If (2), I guess we can talk it in another issue :)

[leiysky]

First, let's determine what's topic discusses?

1. the catalog async/sync style

or

2. the catalog design

If (2), I guess we can talk it in another issue :)

Yeah, I was getting off the topic🤣.

My points are:

• Logical metadata can be cached easily, so it's okay to provide blocking API.
• Partition information and other statistics information can be seperated from metadata service, thus we can design the API later. But they will likely be able to provide blocking API with acceptable overhead as well.

[BohuTANG]

1. blocking API makes sense, we can change it if the optimizer(or others) required.
2. partition information and statistics info is not in the metadata service(MetaStore) by design in datasue arch.

Make working first :)

[dantengsky]

Take micro-partition ...... Thus for a table in size of 1TB, there are about 20,000 micro-partitions at most .......... it will take at most 32MB to store all of partition information in memory, which is not unacceptable.

For a table in size of 1TB, I do agree with you. But the same page you referenced, tells "very large tables, which can be comprised of millions, or even hundreds of millions, of micro-partitions."

the qps of each user is limited to tens

imo, this is not a nice feature :D

[dantengsky]

please allow me to transform this issue into a discussion

[BohuTANG]

Catalog sync way is ok now after PR #1652