TCP Receive Stall - Please provide insight on where I might debug

[jhinklejr]

Issue: TCP socket used to receive 50 meg file download from a server stalls with receive wndo of zero.

Application code uses select() to wait for data to be received. Select has 20 sec timout associated with it to detect a stall or disconnect with server. Timeouts are counted and when a level is reached - the connection is closed for lack of communication.

The application fails about 40% of the time.

In all cases wireshark shows communication with server has stopped because the esp keeps sending the same ack with a wndo of zero. After the server receives 9 of these acks -- it stops trying to send.

Issue -- the select is not firing. The select is not returning with a timeout or with data.

This issue existed using ESP8266 RTOS SDK V2.0 and is what has driven me to V30 IDF version -- at least here I can attempt to resolve it.

Since I have not found any issue of this associated with lwip -- I suspect the issue is with the port to the ESP --- both in the V2 and V3 RTOS releases.

What has Espressif done in porting "select()" from lwip to the esp8266? What file/function would be the best to start looking at? Also -- since a timeout is not firing -- any thoughts on how an esp port would affect that also?

Thanks in advance for any help.

[donghengqaz]

Maybe it is the problem of TCP, I will report the problem to TCP/IP developer.

[jhinklejr]

I got some wireshark logs that show the esp slowing WAY down to process an in-coming packet -- it shows 2 full secs delay from receiving a packet until responding with an ack

[jhinklejr]

Ignore last statement --- response time OK -- it was a lag in wireshark

[jhinklejr]

I have found that most all of the default/suggested queue/mailbox/buffer size and number of buffers are OK for small amounts of data to be communicated via TCP. All those entities above had to be dramatically increased in size for lwip to work when moving 50meg of data.

With the small size recommended as default - lwip will stall or lockup randomly. I don't think the lwip team has done a great deal of boundary testing with small number of buffers/queue, etc.

I've bypassed the menuconfig and set my options directly in lwipopts.h.

You have some of your menuconfig limits set too low. I wanted a TCP WNDO of 14600 -- can't do that in menuconfig -- had to do it directly in lwipopts.h.

You can close this out as it is resolved as stated in this post.

[outermanle]

@jhinklejr I met the same issue in esp-idf master. Could you give me some advise to prevent this issue?

my case as below:

1. server: tcpdump show server has receive the esp32 client packets and server send ack immeditaly, but after that server no longer receive data from esp32 client until 3s timeout and active close the socket.
2. esp32 client: tcp debug log show esp32 client send packets to server and wait a long time(2-3s) to receive ack from server, so esp32 client wait and not send data more until ack(from server) received, but server has send ack already, and esp client wait a long time to receive ack, wifi driver drop this packets due to RF issue?
3. this will lead to server close socket and esp32 client read -1 and return.
4. server: tcpdump shows server often need to send twice for not receive ack from esp client, during sync-ack, data-ack, fin-ack to esp client.

How can I debug this issue? Drop packets for wifi driver or wifi RF?
or change lwip tcp para retry time 3s to a small one?

tcp.c
    pcb->rto = 3000 / TCP_SLOW_INTERVAL;
    pcb->sv = 3000 / TCP_SLOW_INTERVAL;

Thanks!

[jhinklejr]

In my case - I was only the Client. Here is my lwipopts.h file showing the changes I made.

Again - note -- Server is going to be more on transmit than Client which is receive.

lwipopts.h.txt

[donghengqaz]

I has modify the LWIP kconfig TCP window default value max to 14600, and thanks for your restitution.