tracing failed with error "failed to create BPF map 'aggs': Too big"

[rganesan]

I was trying a simple oneliner from the dtrace book on a Linux box and got this:

# dtrace -n 'syscall:::entry { @sc[execname, probefunc] = count(); }'                                                                                    
dtrace: description 'syscall:::entry ' matched 325 probes                             
dtrace: could not enable tracing: failed to create BPF map 'aggs': Too big  
# rpm -q dtrace                                                     
dtrace-2.0.1-1.el9.x86_64

This is on a custom Linux kernel 5.10 on AlmaLinux 9. I haven't tried reproducing the problem on a newer kernel. I will do so when I get a chance and report back.

[kvanhees]

I would suggest running your command with the -xdebug option and looking for lines that say ' Creating BPF maps'. Reporting what that says might help debug this. The 'Too big' error comes from the MAP_CREATE bpf operation returning E2BIG.

[euloh]

By default, each per-CPU agg map is 4M. How many CPUs? One thing to try is to reduce the agg size. E.g., -xaggsize=64k. Play around. You have two strings (execname and probefunc) in the aggregation key. Strings are long (256 by default) and you might not need so much. So, you can try -xstrsize=32 in conjunction with the -xaggsize setting. That is, if the agg maps are too big, you might get by with the aggsize lowered. Lowering strsize simply helps you fit more aggs into that space.

[rganesan]

Thank you for your helpful hints. There are 4 cpus. Here's the relevant output from -xdebug:

libdtrace DEBUG 1732698160: Creating BPF map 'state' (ksz 4, vsz 4, sz 7)             
libdtrace DEBUG 1732698160: BPF map 'state' is FD 4                                   
libdtrace DEBUG 1732698160: Creating BPF meta map 'aggs' (ksz 4, sz 4)                
libdtrace DEBUG 1732698160:     storing BPF maps (ksz 526, vsz 16, sz 7738)           
bpf DEBUG 1732698160: failed to create BPF map 'aggs': Too big                        
dtrace: could not enable tracing: failed to create BPF map 'aggs': Too big            

-xaggsize=64k didn't help, but -xstrsize=32 worked (without setting -xaggsize=64k)

[ezannoni]

@rganesan Is this solved now, can it be closed?

[rganesan]

Thank you for the help. "-xstrsize=32" workaround does fix the issue for me but "-xaggsize=64k" didn't, in fact I couldn't get it working no matter what "-xaggsize" value I provide. So I think it's worth further investigation; maybe make "-xstrsize=32" the default or at least document this caveat explicitly.

…

On Wed, Dec 18, 2024 at 8:07 PM Elena Zannoni ***@***.***> wrote: @rganesan <https://github.com/rganesan> Is this solved now, can it be closed? — Reply to this email directly, view it on GitHub <#114 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRNVRLMNDJROT3DVG335GT2GHIX3AVCNFSM6AAAAABSRHWNI6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJSGE3TGNZYGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Ganesan Rajagopal