urlopen URL with unescaped space

[wichert]

BPO	14826
Nosy	@gpshead, @orsenthil, @pitrou, @tiran, @merwok, @vadmium, @miss-islington
PRs	bpo-30458: Disallow control chars in http URLs. #12755 bpo-14826: document that URLopener quotes fullurl. #12758 [3.7] bpo-14826: document that URLopener quotes fullurl. (GH-12758) #12759 [3.6] bpo-14826: document that URLopener quotes fullurl. (GH-12758) #12760
Files	urllib-quote-14826.patch urllib-request.patch: Followup patch for urllib-quote-14826.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/orsenthil'
closed_at = None
created_at = <Date 2012-05-16.08:54:33.063>
labels = ['type-feature', 'library']
title = 'urlopen URL with unescaped space'
updated_at = <Date 2019-04-10.09:30:36.137>
user = 'https://bugs.python.org/wichert'

bugs.python.org fields:

activity = <Date 2019-04-10.09:30:36.137>
actor = 'miss-islington'
assignee = 'orsenthil'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2012-05-16.08:54:33.063>
creator = 'wichert'
dependencies = []
files = ['26311', '26314']
hgrepos = []
issue_num = 14826
keywords = ['patch']
message_count = 24.0
messages = ['160811', '160930', '164945', '164955', '164957', '164958', '164966', '164969', '164973', '164975', '164976', '164980', '164981', '164982', '165023', '165046', '165047', '165049', '165050', '255525', '268985', '339836', '339839', '339841']
nosy_count = 12.0
nosy_names = ['gregory.p.smith', 'orsenthil', 'pitrou', 'jerub', 'christian.heimes', 'wichert', 'eric.araujo', 'antlong', 'rosslagerwall', 'python-dev', 'martin.panter', 'miss-islington']
pr_nums = ['12755', '12758', '12759', '12760']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue14826'
versions = ['Python 3.6']

[wichert]

There appears to be an odd networking issue with how urllib2 sends HTTP requests. Downloading an image from maw.liquifire.com gives an error:

$ python -c 'import urllib2 ; urllib2.urlopen("http://maw.liquifire.com/maw?set=image[2302.000.13314 a]&call=url[file:325x445]")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1180, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline()
  File "/usr/lib/python2.7/socket.py", line 447, in readline
    data = self._sock.recv(self._rbufsize)
socket.error: [Errno 104] Connection reset by peer

Downloading the same image using wget works fine:

$ wget 'http://maw.liquifire.com/maw?set=image[2302.000.13314 a]&call=url[file:325x445]'             
--2012-05-16 10:53:27--  http://maw.liquifire.com/maw?set=image[2302.000.13314%20a]&call=url[file:325x445]
Resolving maw.liquifire.com (maw.liquifire.com)... 184.169.78.6
Connecting to maw.liquifire.com (maw.liquifire.com)|184.169.78.6|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11393 (11K) [image/jpeg]
Saving to: `maw?set=image[2302.000.13314 a]&call=url[file:325x445]'

100%[======================================>] 11,393 --.-K/s in 0.003s

2012-05-16 10:53:27 (3.49 MB/s) - `maw?set=image[2302.000.13314 a]&call=url[file:325x445]' saved [11393/11393]

[antlong]

http://maw.liquifire.com/maw?set=image[2302.000.13314%20a]&call=url[file:325x445] works properly. Notice the %20 instead of ' '

[jerub]

Here is a patch that uses the same quoting logic in urllib.request.Request.__init__ as is used by urllib.request.URLopener.open()

[python-dev]

New changeset 01c8d800efd2 by Senthil Kumaran in branch '3.2':
Fix bpo-14826 - make urllib.request.Request quoted url consistent with URLOpener open method.
http://hg.python.org/cpython/rev/01c8d800efd2

New changeset e6bb919b2623 by Senthil Kumaran in branch 'default':
Fix bpo-14826 - make urllib.request.Request quoted url consistent with URLOpener open method.
http://hg.python.org/cpython/rev/e6bb919b2623

[python-dev]

New changeset d931a3b64fd6 by Senthil Kumaran in branch '2.7':
Fix bpo-14826 - make urllib.request.Request quoted url consistent with URLOpener open method.
http://hg.python.org/cpython/rev/d931a3b64fd6

[orsenthil]

Thanks for the patch, Stephen.

[rosslagerwall]

It looks like this broke the build bots:
http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%202.7/builds/66/steps/test/logs/stdio

[jerub]

Here's a followup patch that fixes the trunk build for me.

This will unbreak the builds as well as fixing this bug, but it should be investigated why URLopener calls to_bytes() and Request does not. Ideally this interface should be consistent.

[orsenthil]

It seems to me that toBytes in urllib was introduce to restrict the
allowance of urls which were sent as unicode strings. We wanted urls
to be ascii strings in Python2.

http://mail.python.org/pipermail/python-bugs-list/2000-November/002779.html

And quoting to toBytes / to_bytes is actually the problem here, as
cookielib test cases is sending a unicode character which ascii
encoding fails to operate on. I am thinking that we should arrive at a
solution which brings consistency and fixes any previous mistakes. In
3.3, I think, rework of to_bytes may also be a good solution, in 2.7
and 3.2, I think stephen's attached patch is in good lines.
Practically, the quote is more important than the failure at toBytes
by sending an unicode url.

[merwok]

I’m not sure urllib should accept invalid (non-escaped) URLs; a higher-level application can do so, but for the low-level stdlib module it is more debatable.

[orsenthil]

Yeah, I am thinking so as well in that case, the test_cookielib.py
test case may need a change.