Slow calculation and memory leaking with 0.5.1

[jkfm]

Hi all,

I am using pvlib to calculate many pv arrays and do this by looping over the individual system specs. Since updating on 0.5.1, I noticed much longer calculation times and a steady increase in memory usage.

From a quick glance at the changes from 0.5.0 I couldn't determine an obvious cause for this behavior. Did I miss something?

My parameters for the calculation are
ModelChain(system, location, dc_model='singlediode', ac_model='snlinverter', spectral_model='no_loss', temp_model='sapm', aoi_model='ashrae', solar_position_method='nrel_numba', transposition_model='haydavies', losses_model='pvwatts', clearsky_model='simplified_solis', orientation_strategy=None)

[wholmgren]

It could be related to fixing ModelChain to actually use solar_position_method, especially since you're specifying nrel_numba. See #377 and #379. I suggest setting it to nrel_numpy and retesting the performance of your code. Also make sure that you're consistently using nrel_numba for every single solar position calculation in your whole script. This is because the spa module needs to be reimported when the method changes, and this takes some time, especially when the module needs to be JIT compiled again. Note that methods such as Location.get_clearsky will use the default solar position method (nrel_numpy) unless explicitly told otherwise.

Is pvlib the only thing that's changed in your environment?

[jkfm]

Thanks for your suggestions! I will look into them.

Updating pvlib was not the only change that occurred, but after reverting to 0.5.0 the behavior was normal again.

[markcampanelli]

Looks like you are using the single diode model, so it also could be my changes to support ideal devices.

[wholmgren]

@jkfm did you get a chance to look into what might have caused the pvlib performance problem?

[jkfm]

Not yet :-( But I will report back.

[cwhanse]

@jkfm any reports back from your investigation? We'd like to fix the issue you experienced, if we can narrow down the cause.

[jkfm]

Hey all, sorry for the delay. I am now looking into the problem.

I wrote a script which replicates my actual use case but is much simpler: https://gist.github.com/jkfm/6c5668da8f28403bf37268cda4afeb05

The code for following memory usage is from here.

With 30 iterations (my RAM can't handle more ;-) ), the code has the following stats:
0.5.0:
duration 4s,
memory usage:

0.5.1:
duration: 1min 47s
memory usage:

I realise, that the memory usage is not really representative as such, but I guess the trend is clear.

[jkfm]

As the script with 0.5.1 also throws different warnings I suspected the reason for the lag to be in v_from_i and i_from_v in pvsystem.py

I manually changed only those two functions back to their state from 0.5.0 et voilà:

Duration: 4.8s
Memory usage:

So I guess the error has to do with changes in v_from_i and i_from_v

@wholmgren
I also checked your suggestion regarding solar_position_method. There is no significant difference between nrel_numba and nrel_numpy.

Some more information regarding the warnings:
With the old code, I get these warnings once during the first iteration:
/pvlib/pvsystem.py:1958: RuntimeWarning: invalid value encountered in true_divide I = -V/(Rs + Rsh) - (nNsVth/Rs)*lambertwterm + Rsh*(IL + I0)/(Rs + Rsh)
/pvlib/pvsystem.py:1869: RuntimeWarning: overflow encountered in exp argW = I0 * Rsh / nNsVth * np.exp(Rsh * (-I + IL + I0) / nNsVth)

With the new code, I get this warning for each iteration:
/pvlib/pvsystem.py:1919: RuntimeWarning: overflow encountered in exp (Gsh[idx_p]*a[idx_p]))

[cwhanse]

Confirmed. I get different numbers for mem_usage using your script, and not as dramatic as your results for v0.5.1, but the trend is the same: since v0.5.1 we have a memory leak.

[wholmgren]

Thanks @jkfm for this very detailed report!

[wholmgren]

good candidate for #530

[cwhanse]

Looked into this a bit more with tracemalloc, the memory leak appears to depend on using numba with spa, because the memory usage is stable with solar_position_method = nrel_numpy. That is with pvlib v0.6.0.

[wholmgren]

Interesting. I thought the earlier report put the blame on iv functions.

…

On Tue, Nov 6, 2018 at 8:43 AM Cliff Hansen ***@***.***> wrote: Looked into this a bit more with tracemalloc, the memory leak appears to depend on using numba with spa, because the memory usage is stable with solar_position_method = nrel_numpy. That is with pvlib v0.6.0. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#401 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AELiRw63XnnDQOzqd3ihJSj7qf3uWdhsks5usa4egaJpZM4RCDHw> .