fopen:Too many open files

[warkior]

Hi there,

We've been using this tool for a little while now on our site and it generally works great. However recently it started breaking after a random period of time. It runs well for a while, and then suddenly we start getting exceptions such as Illuminate\Broadcasting\BroadcastException: Failed to connect to Pusher

We are running an Ubuntu server via Forge with:
PHP: 7.3
laravel/framework: 6.20
pusher/pusher-php-server: v4.1.5
beyondcode/laravel-websockets: 1.9.0

I ran a debug log for a while on the actual websockets process and found this in the log...

exception `RuntimeException` thrown: `Unable to read from stream: stream_get_contents(): SSL operation failed with code 5. OpenSSL Error messages:
error:02001018:system library:fopen:Too many open files
error:20074002:BIO routines:file_ctrl:system lib
error:140DC002:SSL routines:use_certificate_chain_file:system lib`.

Can someone tell me what this means? Is the websocket server leaving connections open and over time it is just filling up?

[malanx]

Hi @warkior I have a guess,

Too many connections, you can verify if this is the issue in your Pusher Dashboard (there is a nice chart).

The thing that is causing the issue, my guess is bug #362 however you may genuinely be hitting that limit with your users.
-- You can try upgrading to v2 of this library in a test environment to verify the issue however I wouldn't recommend using it in production until the official release is ready.

If that isn't your problem you may just be hitting a resource limit on your server, you can google "ulimit" and your ubuntu distribution to find more details.

I hope one of those solved your issue :)

[warkior]

Hi @CodeJunkieio

Thanks for your suggestion. I'll monitor the stats for a bit here and see what I come up with. I did already find one thing which may or may not have impacted this. I was missing an APP_NAME key in my env file which seemed to result in stats not being recorded to the websockets stats table. Not sure why it had that effect since that value isn't stored in that table, but when I added it things seemed to start getting recorded again.

All the best!

[warkior]

Hmm. Nope. Actually the stats problem was NOT caused by the APP_NAME key but rather this issue: #624

in config -- 'perform_dns_lookup' => true,

[warkior]

@CodeJunkieio

Well, I'm not sure how many is too many. I feel like this isn't all that huge. But I'm not entirely sure how to read it either. Note the gap. That's where the websocket server appears to crash and stop receiving connections.

[warkior]

As an update to this, it DOES look like we were hitting the ulimit barrier of 1024 open files. I discovered this by checking ls -al /proc/{PID}/fd to find all the sockets associated with the websockets process. It seems to eventually fill up to the 1024 limit and then just stop responding to requests. I have increased the limit, but I feel this isn't really a proper solution. It seems to me it will just take longer to fill up.

This leads me to believe that the websockets code is not properly closing the connections to the server. Is it possible there is some memory/socket leak that would cause this? In the stats I DO see vacated and disconnected entries, leading me to believe the clients ARE disconnecting correctly.

Anyone have insight on why this might be happening?

[warkior]

I've increased the ulimit as documented here but still have the issue.
https://beyondco.de/docs/laravel-websockets/faq/deploying

I'm running it via Laravel Forge and Supervisor so I adjusted the ulimit for the forge user under which it runs. Still experimenting to see if I can find the solution.

[malanx]

That sounds about right. As far as limit, Pusher has a hard limit, it looks like you're not even getting close to the free tier limit so I would agree with the sentiment that Pusher is not the issue with the problems you're having

[warkior]

Still no success. I've tried setting the ulimit for the forge user.

Tried changing limits via minfds=10240 as documented here:
https://beyondco.de/docs/laravel-websockets/basic-usage/starting

Tried starting the process using something like bash -c "ulimit -n 10000; php artisan websocket..."

No success. I'm ready to chalk it up to version 1.9 bugs, but when I tried installing the 2.0 branch I just got errors. Not sure what's up with that.

[rennokki]

In some cases, there are hidden parts in the OS that somehow overwrites the defined user limits. It's more of a Linux problem rather than the package not being able to handle the file limits.

If using Supervisor, make sure to check minfds: https://ma.ttias.be/increase-the-number-of-open-files-for-jobs-managed-by-supervisord/

[tobischulz]

@rennokki, this helped me a lot. Supervisor has its default limit to 1024. Updated the minfds in supervisord.conf and restart. Now the new limits are active.

Thank you very much!

[rennokki]

Fixed in the previously mentioned PR.