Metrics for improved memory pool diagnosability

[adityamandaleeka]

Now that we have meters with dimensions it should be possible to emit per connection metrics on bytes buffered/consumed.

This may be too granular, so another approach we should consider is aggregating these somehow (one idea is to do it per request method).

cc @davidfowl @NickCraver

Additional context

No response

[davidfowl]

Some obvious ones:

Metric Name	Description	Unit	Example
TotalAllocatedMemory	Total memory allocated for blocks since the pool was created.	Bytes	5242880 (5 MB)
CurrentMemory	Memory currently held in the pool (available for reuse).	Bytes	1048576 (1 MB)

The other suggestion was a histogram of memory usage per request (we need to figure out the buckets).

Spoke to @JamesNK and came up with a special (suggested by @NickCraver) histogram counter that we could turn on per request that shows memory usage with the request method as a dimension.

The histogram bucket would start at 1024 and go up in powers of 2 (we can use 14 buckets), then the max value would be 16 megs (which would be kinda crazy 😄).

[BrennanConroy]

PeakAllocatedMemory
AllocationRate (more like usage rate since a majority of memory is pooled?) (e.g. get 4kb, return 4kb, get 4kb all in a second, would be 8kb/sec)

[adityamandaleeka]

Superseded by #61594