print(s) fails on Windows with long strings

[casevh]

BPO	11395
Nosy	@terryjreedy, @amauryfa, @pitrou, @eryksun
Files	test_writeconsole.patch wconsole_large.patch test_wconsole_binlarge.patch winconsole_large_py33.patch winconsole_large_py33_direct.patch io_write.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-03-20.22:50:41.369>
created_at = <Date 2011-03-04.05:47:42.980>
labels = ['OS-windows', 'type-crash']
title = 'print(s) fails on Windows with long strings'
updated_at = <Date 2020-04-13.22:30:55.729>
user = 'https://bugs.python.org/casevh'

bugs.python.org fields:

activity = <Date 2020-04-13.22:30:55.729>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2011-03-20.22:50:41.369>
closer = 'vstinner'
components = ['Windows']
creation = <Date 2011-03-04.05:47:42.980>
creator = 'casevh'
dependencies = []
files = ['21003', '21012', '21013', '21021', '21025', '21030']
hgrepos = []
issue_num = 11395
keywords = ['patch']
message_count = 36.0
messages = ['130029', '130033', '130034', '130036', '130037', '130038', '130039', '130041', '130042', '130043', '130044', '130045', '130047', '130048', '130095', '130100', '130133', '130135', '130158', '130186', '130187', '130189', '130197', '130202', '130212', '130214', '130249', '130251', '130252', '131558', '131561', '132284', '224606', '224608', '366237', '366243']
nosy_count = 10.0
nosy_names = ['terry.reedy', 'amaury.forgeotdarc', 'pitrou', 'casevh', 'davidsarah', 'neologix', 'santoso.wijaya', 'python-dev', 'eryksun', 'Drekin']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue11395'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2', 'Python 3.3']

[casevh]

Python 3.2 fails when printing long strings.

C:\Python32>python
Python 3.2 (r32:88445, Feb 20 2011, 21:30:00) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> print("a"*66000)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 12] Not enough space
>>>

Some observations:

1. 3.2 on Linux prints just fine.
2. 2.7.1 and 3.1.3 on Windows x64 are fine
3. The 32-bit interpreter for 3.2 also fails.
4. On 32-bit Windows, a length of 62733 works correctly but 62734, and higher, fail.
5. On 64-bit Windows, the output is visibly corrupted when the length reaches 62801 but the error does not occur until the length reaches 65536.
6. While experimenting with various lengths, I was able to crash the interpreter once.

[amauryfa]

Likewise, this fails with 3.2::
import os
os.write(1, b"a" * 66000)

[neologix]

It's probably a Windows limitation regarding the number of bytes that can be written to stdout in one write.
As for the difference between python versions, what does
python -c "import sys; print(sys.getsizeof('a'))" return ?

[amauryfa]

no, it works with 3.2b2 (r32b2:87398), and fails with 3.2 final (r32:88445)

[vstinner]

Extract of issue bpo-1602:
<< WriteConsoleW has one bug that I know of, which is that it <a href="http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1232"\>fails when writing more than 26608 characters at once</a>. That's easy to work around by limiting the amount of data passed in a single call. >>

I suppose that os.write(1) does indirectly write into the Windows console which has this limit.

[vstinner]

Extract of the WriteConsole Function:

<< The storage for this buffer is allocated from a shared heap for the process that is 64 KB in size. The maximum size of the buffer will depend on heap usage. >>
http://msdn.microsoft.com/en-us/library/ms687401(VS.85).aspx

Ah ah, that's funny, "depend on heap usage" :-)

[amauryfa]

This changed with r87824

[vstinner]

This changed with r87824

Yes, I changed Python to open all files in binary mode. With Python < 3.2, you can open sys.std* streams in binary mode using -u command line option (u like unbuffered, not Unicode ;-)).

[amauryfa]

Indeed, Python3.1 fails with the -u option.

Before r87824, the C call to write() performed CRLF conversion. In the implementation of MSVCRT, a local buffer is used (1025 chars in vs8.0, 5*1024 in vs10.0), so WriteFile is called with small sizes.
Since r87824 (or with -u), no such conversion occurs, and WriteFile is called with the full buffer.

[vstinner]

Anyway, use os.write() to write unicode into the Windows console is not the right thing to do. We should use WriteConsoleW(): bpo-1602 is the correct fix for this issue.

[amauryfa]

I'm writing bytes here: os.write(1, b"b" * 66000)
And WriteConsole has the same issue.

[vstinner]

And WriteConsole has the same issue.

print() (sys.stdout and sys.stderr) should use WriteConsoleW() and use small chunks (smaller than 64 KB, I don't know the safest size).

[pitrou]

IIUC, this is a Windows bug? Is there any easy workaround for us?

[amauryfa]

It may be a windows bug, but it's also an a python regression!
A fix is to limit the number of chars:

===================================================================

--- D:/py3k/Modules/_io/fileio.c   (revision 87824)
+++ D:/py3k/Modules/_io/fileio.c   (copie de travail)
@@ -712,6 +712,8 @@
         errno = 0;
         len = pbuf.len;
 #if defined(MS_WIN64) || defined(MS_WINDOWS)
+        if (len > 32000 && isatty(self->fd))
+            len = 32000;
         if (len > INT_MAX)
             len = INT_MAX;
         n = write(self->fd, pbuf.buf, (int)len);

On my system, errors start at ~52200 (why?). I hope that 32K is low enough... MSVCRT's write() (version vs10.0) uses a buffer of 5K.