Performance slow?

[markuswinkler]

Hi!

I recently looked into my database performance and was wondering if my results are somewhat slow.

A single query.get(user.id) resolves in 20ms, which seems to be rather high,
regarding that the whole system and database is basically idling (indexes on _id are set).
There are only 8 users entries. It doesn't make a difference if I run it with a local
database/server or on AWS/MongoDBAtlas.

The performance for a query.get with 4 includes is about 40-50ms.

Is this the normal performance range to expect?

Steps to reproduce

Parse.Cloud.define("test", function(request, response) {
	const tStart=Date.now();

	let user = request.user;
	let query = new Parse.Query(Parse.User);
	query.get(user.id, {useMasterKey: true}).then(function(result) {
			let time = Date.now()-tStart;

			console.log(time);
			response.success({"time": time});
		},
		function(error) {
			response.error(error)
		});
});

Expected Results

< 1ms

Actual Outcome

around 20ms

Environment Setup

• Server

  • parse-server version : 2.7.4
  • remote server: AWS

• Database

  • MongoDB version: 3.4.13 wired tiger, hosted on MongoDB Atlas

[flovilmart]

Expecting results under 1ms is a bit too much I guess, in our case, we're responding in 40ms at 95th percentile, which is more than acceptable, considering most of the calls are authenticated.

[flovilmart]

The performance for a query.get with 4 includes is about 40-50ms.

When you do 4 includes, there are 4 additional queries done in the database hopefully run in parallel.

Those values are still in an acceptable range for me. You could run an APM and identify the bottlenecks. Probably the DB call latency and JSON serialization / deserialization will be the most consuming element of your calls.

[flovilmart]

Also, consider when doing you 'Parse.Query' you're issuing and HTTP reuqest to parse server, and not a direct call to the internals of parse-server so that overhead has to be considered.

Can you please report the time you see with PARSE_SERVER_ENABLE_EXPERIMENTAL_DIRECT_ACCESS=1 as an env variable?

[markuswinkler]

Ah, ok. I just didn't know what to expect.
I read some mongoDB examples where the update time for 1.5 million records is around
1:15:00, resulting in around 3ms/update.
Does parse-server keep the connection to the DB persistent or does it have to authenticate for every request?

Enabling PARSE_SERVER_ENABLE_EXPERIMENTAL_DIRECT_ACCESS=1 made a HUGE impact. All request/save times are basically half now, around 10/11 ms.
How stable is that feature?

(Also, I really mean no offense, parse-server is a truly fantastic product, I just have no comparison values here, trying to find a baseline).

[flovilmart]

Updates and reads are completely a different game :)

Parse adds some overhead, schema validation, ACL, authentication etc.. all of that add a certain amount of overhead.

As for the stability of the feature, it should be fine. But may not fit every workload as everything stays in memory on the same serverz

[markuswinkler]

That makes a lot of sense, thanks for the explanation!

So basically the worst that can happen is that the server just consumes a lot more memory. I will watch the memory consumption then.

[flovilmart]

Just remember to pass masterKey / sessionToken, also IIRC there are some edge cases with loggingIn cloud code (which should not be supported).

[markuswinkler]

Thank you, good to know. I made my client intentionally very dumb and rights limited, the only data manipulation is happening via cloud code calls that all need to run with masterKey internally since the database is completely locked down so I should be safe then. :)

Since we are on the topic of performance optimizations, how big is the gain in using cacheAdapter: redisCache as lined out in the get started documentation? Can redis run on the same machine? Just curious.

[flovilmart]

It would depend on your workload there may be a cost because of the failed cache lookups, that are slower than in memory. Redis is useful to cache your user’s session data and roles. Each authenticated call (with a session token) yield a lookup for the session, the user as well as the roles. As you see with our metrics, we use redis to cache it all, so it speeds up our calls.

Redis is the last thing you want to put in as after it, there isn’t much more optimisations you can add to parse-server without custom code.

Note that we also cache those in memory, so it’s only useful if you deploy multiple instances and don’t use session affinity at load balancer.

Scaling parse-server follows the same best practices as scaling any nodejs / express apps. If you have multiple cores on your server, leverage them through node’s cluster first or the cli with the —cluster option etc...

[markuswinkler]

Thanks for the thorough answer!

[mman]

@flovilmart Thanks for the explanation. I wonder what happens when you deploy multiple parse servers behind a load balancer with PARSE_SERVER_ENABLE_EXPERIMENTAL_DIRECT_ACCESS=1 and without using redisCache and without using any session affinity.

Will the backend instance verify sessionToken correctly against mongo every time?

Or will it only lookup in the memory on the given backend instance?

Can I still expect everything to work as expected? Meaning every sessionId passed in via either backend parse server instance still correctly validates against mongo every time a call is made?

thanks,
Martin

[flovilmart]

Meaning every sessionId passed in via either backend parse server instance still correctly validates against mongo every time a call is made?

Yes, sessions will always be validated.
The best way to look into what’s going on is to perhaps put an APM like new relic; and look for the DB calls. In a previous series of load testing, we realized that JSON serialization of large objects was a large source of CPU consumption, and was inducing latency in the responses as serialization is blocking the node process.

In our case, we decided not to go with the direct access but to let internal cloud code calls reach another server, as it was the most optimal solution for our workload

[Amit-synergytop]

We are using AWS lambda functions and MongoDB Atlas with parse server.
We are facing slow API response issue and we are trying to determine where is the issue. As I did some RND on it and get conclusion that parse server responding slow.
So can you please help me in determining the actual problem or suggest me any tool through which I can determine it by my self.