https://github.com/pandas-dev/pandas/pull/52459 did this for `to_pydatetime`, so we should do the same for `to_pytimedelta`