yz/ibrowse configuration and interaction with Solr

[wbrown]

Currentlly, Yokozuna uses the ibrowse application to stage index updates to Solr. The default ibrowse settings are decidedly non-optimal with a default per host:port pair maximum of 10 sessions and 10 pipeline depth.

As the Solr indexes grow to millions of entries, the latency of the index updates exerts a backpressure upon Riak, slowing down key updates and PUTs. htop shows a back and forth pattern of Riak and Solr exchanging CPU cycles. At this point, Riak performance completely tanks, especially with the LevelDB backend.

Working with @evanmcc -- we found that the ibrowse author doesn't like to do configuration like the rest of the Erlang world, unfortunately. He suggested a configuration line update, that I had to alter -- hooray for documentation that doesn't match reality.

${RIAK_HOME}/lib/ibrowse-4.0.1/priv/ibrowse.conf had the following line added:

{dest, "localhost", 8093, 100, 100000, []}.

Note that {127,0,0,1} in place of localhost did not work. It really wants the string hostname, probably because yz is using localhost. By the way, if you do this, this will default to IPV6 for localhost on some systems, which is an interesting and potentially problematic side effect.

The above settings set the connection maximum to 100 with a pipeline depth of 10000. I restarted Riak, and sure enough, I got that amount of connections which is an indicator that I may need to raise the limit higher for my throughput of about 400-500 keys indexed a second.

[root@altostratus log]#  netstat -anp | grep 8093 | grep EST | wc -l200
200

Checking for TIME_WAIT revealed hundreds of connections left over, which is indicative that there is no pipelining.

[root@altostratus log]#  netstat -anp | grep 8093 | grep TIME_WAIT | wc -l
594

@coderoshi confirmed my suspicion. This is problematic for deployments that index hundreds or thousands of keys a second with the overhead of establishing a socket connection for every index update.

So, I'd suggest the following changes:

• Add a more reasonable default max session value for localhost:8093
• Use HTTP pipelining so that we don't run out of socket descriptors/file handles and also so we don't incur the punishing latency and overhead of a TCP connection for every key update.

[rzezeski]

@wbrown Thank you very much for this informative issue. I always had a suspicion the current ibrowse setup was not optimal but I have yet to analyze it as closely as you have. Things are really busy right now but I hope someone might be able to dig into this issue in the coming weeks. Feel free to submit your own patch if you feel inclined to.

[wbrown]

@rzezeski I'd be happy to submit a patch for the configuration file. It may not be optimal for other use cases though. Writing HTTP pipelining into the Yokozuna interface to Solr is probably a bit beyond my current time budget though, as I'd have to learn more Erlang. :)

[rzezeski]

@wbrown Yea in this case I would want it configurable via our standard configuration. Which would require editing the configuration schema file and passing to ibrowse. But it sounds like you and @evanmcc discovered passing the pool and pipeline size programmatically doesn't work. Perhaps a patch to ibrowse is also in order. I'm not sure, I need more info and need to look at the ibrowse code.

[rzezeski]

@wbrown

A quick look at ibrowse suggests that it automatically does the pipelining. I'm not sure the excess TIME_WAIT is necessarily indicative of requests not being pipelined. The local Solr process will have connections originating from two different sources:

1. the localhost when a user query comes in on that node;
2. a remote host when a Solr distributed search is being executed.

I wonder if it is case 2 that is causing those excess connections to be created. Can you verify with netstat that all TIME_WAIT on 8093 are localhost => localhost.

[wbrown]

@rzezeski Yes, ibrowse does do the pipelining. However, this does not appear to be in effect, as Yokozuna appears to be doing each update request individually, and causing a new socket connect via ibrowse.

There are no searches going on while I am doing this testing -- all connections to Solr are coming via localhost. The higher I raise the socket limit for ibrowse, the more the lingering TIME_WAITs increase.

[rzezeski]

Oh, could it be because, according to RFC 2616, you are not supposed to pipeline non-idempotent requests? Yokozuna uses POST to send the update.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html

That said, keep alive should be in play and those connections should be re-used. The requests just wouldn't be pipelined. I'm not sure why it would not reuse existing conns. Perhaps there is a bug with ibrowse connection reuse.

Another thing to note is that the jetty config (lib/yokozuna/priv/solr/etc/jetty.xml) has the connection thread pool set to 10,000 max threads. Just for laughs I am curious what happens if you set that equal to ibrowse connection pool size and set the lowResourceMaxIdleTime equal to the maxIdleTime on the socket connector.

[wbrown]

Setting up for this test -- setting the connection thread pool to 150 max threads. I don't see any configuration option or default in ibrowse that tells me what the maxIdleTime would be though.

On Mar 5, 2014, at 4:57 PM, Ryan Zezeski notifications@github.com wrote:

Oh, could it be because, according to RFC 2616, you are not supposed to pipeline non-idempotent requests? Yokozuna uses POST to send the update.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html

That said, keep alive should be in play and those connections should be re-used. The requests just wouldn't be pipelined. I'm not sure why it would not reuse existing conns. Perhaps there is a bug with ibrowse connection reuse.

Another thing to note is that the jetty config (lib/yokozuna/priv/solr/etc/jetty.xml) has the connection thread pool set to 10,000 max threads. Just for laughs I am curious what happens if you set that equal to ibrowse connection pool size and set the lowResourceMaxIdleTime equal to the maxIdleTime on the socket connector.

—
Reply to this email directly or view it on GitHub.

[wbrown]

Nevermind -- found the parameter you're asking for, in Solr. Here goes.

[rzezeski]

I don't see any configuration option or default in ibrowse that tells me what the maxIdleTime would be though.

That's a good point. I'm not sure what the defaults are for connection evictions in ibrowse.

[wbrown]

Wiith ibrowse set to allow up to 150 connections, we get 150 connections to Solr as shown below.

[root@altostratus ~]# netstat -anp | grep :8093 | grep EST | wc -l
300

We get lots of TIME_WAITs still:

[root@altostratus ~]# netstat -anp | grep :8093 | grep TIME_WAIT | wc -l
2140

One parameter that I changed that seems to have helped was increasing Solr's Acceptors value to 10. Which would make sense, as the default is 2, and if it's busy handling TCP handshakes, Riak would get backed up. But it does make the TIME_WAITs apparently worse, which makes sense as well.

To sum up, the pertinent values are:

• maxThreads to 150
• Acceptors to 10
• lowResourcesMaxIdleTime to 50000