Skip to content
This repository was archived by the owner on Mar 13, 2022. It is now read-only.

Use a higher value for connecion_pool_maxsize. #29

Merged
merged 1 commit into from
Aug 23, 2017
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions configuration.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@

from __future__ import absolute_import

import multiprocessing
import logging
import sys

Expand Down Expand Up @@ -83,11 +84,12 @@ def __init__(self):
# Set this to True/False to enable/disable SSL hostname verification.
self.assert_hostname = None
# urllib3 connection pool's maximum number of connections saved
# per pool. Increasing this is useful for cases when you are
# making a lot of possibly parallel requests to the same host,
# which is often the case here.
# When set to `None`, will default to whatever urllib3 uses
self.connection_pool_maxsize = None
# per pool. urllib3 uses 1 connection as default value, but this is
# not the best value when you are making a lot of possibly parallel
# requests to the same host, which is often the case here.
# cpu_count * 5 is used as default value to increase performance
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a note saying we're doing this because this is the default for ThreadPoolExecutor?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added, thanks.

# This is used because it's the default value for ThreadPoolExecutor
self.connection_pool_maxsize = multiprocessing.cpu_count() * 5
# http proxy setting
self.http_proxy_url = None

Expand Down