Sample code to show how can hwloc be used to print latency /bandwidth for initiators and targets

[aghozzo]

I couldn't find any sample C code that shows how to extract HMAT table attributes or APIs that can be used and called directly from a hello world C code .

can someone please provide a pointer on how to run and test sample code to get this information or if there are already exisiting examples ?

Thanks

[bgoglin]

Hello
Bandwidth and Latency are part of the "memory attributes" functionnality. "lstopo --memattrs" will show bandwidth/latency on the command-line if available (likely not available unless you have an IceLake platform).
The API is in hwloc/memattrs.h, most of the current doc is in the comments there (and a little bit in https://www.open-mpi.org/projects/hwloc/doc/v2.8.0/a00366.php#topoattrs_memattrs)
There's an example in tests/hwloc/memattrs.c (but it's quite complicated because it queries AND modifies memattrs).
I'll try to add a simple example under the doc directory.
Brice

[bgoglin]

Would this example help https://github.com/bgoglin/hwloc/blob/master/doc/examples/memory-attributes.c ?

It shows this on my laptop (no heterogeneous memory, no HMAT):

$ doc/examples/memory-attributes         
There are 1 NUMA nodes
Core L#0 cpuset = 0x00000005
Found 1 local NUMA nodes
NUMA node L#0 P#0 is local to core L#0
  bandwidth is unknown
  latency is unknown
Couldn't find best NUMA node for bandwidth to core L#0

And this on a machine with 2 kinds of memory and HMAT:

$ doc/examples/memory-attributes
There are 8 NUMA nodes
Core L#0 cpuset = 0x00000001
Found 2 local NUMA nodes
NUMA node L#0 P#0 is local to core L#0
  bandwidth = 62100 MiB/s
  latency = 191 ns
NUMA node L#1 P#4 is local to core L#0
  bandwidth = 69100 MiB/s
  latency = 227 ns
Best bandwidth NUMA node for core L#0 is L#1 P#4
Allocated buffer 0x560991f25b30

[aghozzo]

Hi Brice,

First of all let me thank you for the quick reply and code . Excellent support , appreciated
the example is a great start. I will run few test cases and check.

Please keep the issue open for a week or so to add feedback or if there is a different area for Q/A discussion we can use that.

Again Thanks a bunch

[bgoglin]

This issue will be autoclosed by github when I commit the new example to the repository, but feel free to ask other questions here, I'll be notified.

[aghozzo]

Hi

Does hwloc work in qemu enviroment setup with HMAT >?

ls /sys/firmware/acpi/tables
APIC CEDT DSDT FACP FACS HMAT HPET MCFG SRAT WAET data dynamic

lstopo
Machine (5955MB total)
Package L#0
NUMANode L#0 (P#0 981MB)
L3 L#0 (16MB) + L2 L#0 (4096KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#0)
Package L#1
NUMANode L#1 (P#1 1952MB)
L3 L#1 (16MB) + L2 L#1 (4096KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1 (P#1)
Group0 L#0
NUMANode L#2 (P#2 3023MB)

when i run the sample :

There are 3 NUMA nodes
Core L#0 cpuset = 0x00000001
Found 1 local NUMA nodes
NUMA node L#0 P#0 (subtype (null)) is local to core L#0
bandwidth is unknown
latency is unknown
Couldn't find best NUMA node for bandwidth to core L#0

[bgoglin]

Yes I use qemu quite intensively for simulating strange memory subsystems. But it doesn't populate HMAT with anything useful by default. There's an example of what I've been using in section "HMAT performance attributes" of https://github.com/open-mpi/hwloc/wiki/Simulating-complex-memory-with-Qemu
As you see, you have to manually define the bandwidth and/or latency of every couple (CPU initiator, memory target). That's quite boring. The qemu patch cited at the end of this section was only applied upstream recently, you don't have it in your qemu, hence you must pass initiator=0 or 1 after memdev=ram2.
If you don't want to setup all this, I can provide some XML topology that hwloc may load at runtime instead of reading the platform topology. Something like this:

$ HWLOC_XMLFILE=/path/to/hwloc/git/tests/hwloc/xml/64intel64-fakeKNL-SNC4-hybrid.xml doc/examples/memory-attributes
There are 8 NUMA nodes
Core L#0 cpuset = 0x00010001,0x00010001
Found 2 local NUMA nodes
NUMA node L#0 P#0 (subtype (null)) is local to core L#0
  bandwidth = 22500 MiB/s
  latency is unknown
NUMA node L#1 P#7 (subtype MCDRAM) is local to core L#0
  bandwidth = 90000 MiB/s
  latency is unknown
Best bandwidth NUMA node for core L#0 is L#1 P#7
Allocated buffer 0x7f761b67f010 on best node

[aghozzo]

Hi Brice,

The XML file idea is brilliant .
I was able to re-create the example you just provided ( I didn't get latency like you) .

just 2 more questions :
is there a fake xml file that fake/shows latency and bandwidth ?
is there a documentation/explanation for these xml files as a guidance on what they are instead of going to each one and read the file ?

[bgoglin]

I don't have a XML with latency in the repo. But adding latencies isn't very hard:

• modify the XML it manually: duplicate the Bandwidth memattr section at the end, rename into Latency and change the values.
• or use hwloc-annotate to add individual latencies: something like hwloc-annotate input.xml output.xml -- numa:1 -- memattr Latency package:0 100.
  There's a long example for customizing a XML topology in https://www.open-mpi.org/projects/hwloc/doc/v2.8.0/a00373.php#faq_create_asymmetric.
  There's no documentation for the XML file contents itself. It was not meant to be manually modified (even if I do it very often).

[aghozzo]

I don't have a XML with latency in the repo. But adding latencies isn't very hard:

• modify the XML it manually: duplicate the Bandwidth memattr section at the end, rename into Latency and change the values.
• or use hwloc-annotate to add individual latencies: something like hwloc-annotate input.xml output.xml -- numa:1 -- memattr Latency package:0 100.
  There's a long example for customizing a XML topology in https://www.open-mpi.org/projects/hwloc/doc/v2.8.0/a00373.php#faq_create_asymmetric.
  There's no documentation for the XML file contents itself. It was not meant to be manually modified (even if I do it very often).

Hi,

Thanks for the Reply , much appreciated

I tried to modify the XML file

• I tried replacing/renaming the word "Bandwidth" with "Latency" . didn't work
• I tried duplicating the last section and changed the values and the name to Latency. didn't work.

when I changed the "Flag=6" I start seeing the Latency values :) . Thanks

[bgoglin]

Ah sorry, I forgot that you need to change the "flags" after Bandwidth. For latency it's 6 instead of 5 (because best latency is lowest while best bandwidth is highest).

[aghozzo]

@bgoglin

Hi,

I have 2 questions :

1. Is there a mechanism to show Initiators/targets matrix ( example bandwidth/latency/ or distance from any core to ALL NUMA nodes >?

hwloc_get_local_numanode_objs(topology, &initiator, &numa_nodes, nodes,HWLOC_LOCAL_NUMANODE_FLAG_ALL);
Even though i do FLAG_ALL , i still only get attributes for local NUMA nodes not the ones outside the package (if that is the right term to use) .

I like to know the penalty in terms of attributes for a CPU core to access memory outside of its local socket./package . (something like NUMA node local latency + latency to go off socket to access the memory )

2. how the system deal with CPU-less NUMA nodes (memory extension NUMA nodes) ? are those considered local to all CPU cores ?