setup() granularity and subtracting related tasks

[tylerjereddy]

One tricky thing to deal with is state-dependent benchmarks--sorting a list or handling a StringIO object as I mentioned for NumPy and pandas benchmarks.

The problem is that setup() runs between repeats, but not between iterations within repeats, so if you point to an object like a list you want to sort or a StringIO object you want to read in, the work on that object will already be done in subsequent iterations and the timings will do less or close to nothing depending on the task being timed in time_operation.

I don't think there's any obvious way around this (since that's how timeit works -- single setup then a bunch of iterations with no setups in between). However, one possibility might be exposing convenient functionality to subtract away the timing of any operations you aren't interested in as implied by Raymond Hettinger in a comment about sort benchmarking.

cc @stefanv , @mattip

[pv]

Set number = 1 and increase repeat to improve the statistical accuracy.

The "subtract the setup time" approach I think is relevant if the timer accuracy is limited, which I don't think is usually the case with Python.

[tylerjereddy]

Pretty sure I still had trouble getting unique StringIO objects when setting number and repeat attributes within the setup() of parametrized benchmarks within class objects. I guess I'll try again with that approach--do we have documented examples of setting these attributes within a setup()? All I could find was for an isolated test_ function I think.

So is:

def setup(self, param1, param2):
    self.number = 1
    self.repeat = 1

guaranteed to run just once for each parameter? (or once plus 1 repeat?)

And then, from a design standpoint, I wonder how well manually setting those values actually works if the parameters range over a few order of magnitude (which timeit would normally handle automatically by adjusting the iteration (number) count, I think).

[tylerjereddy]

If I set repeat = 1 and number = 1 as shown below the time_benchmark function seems to effectively ignore my hardcoded specifications for repeats and iterations per repeat (using a relatively recent version of the asv dev branch):

 67 class LoadtxtCSVComments(Benchmark):                                                                                       
 70                                                                                 
 71     params = [10, int(1e2), int(1e4), int(1e5)]                                 
 72     param_names = ['num_lines']                                                 
 73                                                                                 
 74     def setup(self, num_lines):                                                 
 75         self.repeat = 1                                                         
 76         self.number = 1                                                         
 77         data = [u'1,2,3 # comment'] * num_lines                                                   
 82         self.data_comments = StringIO(u'\n'.join(data))                         
 83         print("running setup for num_lines:", num_lines)                        
 84                                                                                 
 85     def time_comment_loadtxt_csv(self, num_lines):                                                              
 95         print("running bench test proper for num_lines:", num_lines)            
 96         self.data_comments.seek(0)                                              
 97         np.loadtxt(self.data_comments,                                          
 98                    delimiter=',') 

output in log file with print functions being called way more than expected for the iterations:
log.txt

[tylerjereddy]

If I just print from the setup() to clean up the output:

                For parameters: 10
                running setup for num_lines: 10
                running setup for num_lines: 10
                
                For parameters: 100
                running setup for num_lines: 100
                running setup for num_lines: 100
                
                For parameters: 10000
                running setup for num_lines: 10000
                running setup for num_lines: 10000
                
                For parameters: 100000
                running setup for num_lines: 100000

Looks like setup() only gets called once for the largest set, but the entire suite runs in way less than 60 seconds. I suppose there may be warm up runs in some cases or something?!

[pv]

Yes, there is warmup if you are using asv 0.3.dev, see the documentation.

[pv]

Also, there's no need to set `repeat = 1` unless you really want to sample only a single data point. You can leave it at the default value (0) otherwise.

[pv]

In addition, you can set ``` time_comment_loadtxt_csv.number = 1 ``` or ``` number = 1 ``` in the class definition, instead of setting the values dynamically in setup().

[tylerjereddy]

This was mostly for debug purposes; I see that setting warmup_time = 0 seems to remove the calls I wasn't expecting in a debug context. Looks like repeat and number don't affect warmup behavior, which I suppose is sensible. Closing, thanks