Download with `getData()` stalls for large file on slow network

[vzukanov]

New Issue Checklist

• I am not disclosing a vulnerability.
• I am not just asking a question.
• I have searched through existing issues.
• I can reproduce the issue with the latest version of Parse Server and the Parse Android SDK.

Issue Description

Parse Android SDK hangs when issuing getData() call for large file on slow network.

Steps to reproduce

Store a large file in Parse and try to get its data synchronously (on background thread) using getData() call when a mobile device is connected to slow network (or slowed down emulator).

Actual Outcome

getData() call hangs indefinitely.

Expected Outcome

getData() should either succeed, or throw an exception.

Environment

Parse Android SDK

• SDK version: 3.0.0
• Operating system version: Android 11

Server

• Parse Server version: 5.1.1
• Operating system: Ubuntu 20.04
• Local or remote host (AWS, Azure, Google Cloud, Heroku, Digital Ocean, etc): Digital Ocean

Database

• System (MongoDB or Postgres): MongoDB
• Database version: 3.6.8

Logs

Nginx last entry in access.log when Andorid SDK hangs on slow network:

85.250.100.140 - - [23/Mar/2022:19:43:41 +0000] "GET /parse/classes/Book/Cq1hFjlytH HTTP/2.0" 200 267 "-" "Parse Android SDK API Level 30"

On faster network (LTE), I Nginx's access.log shows the following:

85.250.100.140 - - [23/Mar/2022:19:45:34 +0000] "GET /parse/classes/Book/Cq1hFjlytH HTTP/2.0" 200 267 "-" "Parse Android SDK API Level 30" "-"
85.250.100.140 - - [23/Mar/2022:19:45:37 +0000] "GET /parse/files/appid/67dfb4b64a2682765d9b34f34e5bfe00_shvilim-14.pdf HTTP/2.0" 200 28613718 "-" "Parse Android SDK API Level 30" "-"

Additional info

First, I thought that the problem is client-side timeout. So, I increased it:

        Parse.initialize(new Parse.Configuration.Builder(this)
                .applicationId("<appid>")
                .server("<url>")
                .enableLocalDataStore()
                .allowCustomObjectId()
                .clientBuilder(
                        new OkHttpClient.Builder()
                                .readTimeout(60, TimeUnit.SECONDS)
                                .writeTimeout(60, TimeUnit.SECONDS)
                )
                .build()

Unfortunately, did not work.

Then I increased timeouts for Nginx:

http {                                                                                                                  
                                                                                                                        
    proxy_read_timeout 300;                                                                                             
    proxy_connect_timeout 300;                                                                                          
    proxy_send_timeout 300;                                                                                             
    send_timeout 300;  

    ...
}

Did not work either.

This is a critical problem for us. If Parse Android SDK would just throw an exception instead of hanging it would make the situtation much better.

[parse-github-assistant]

Thanks for opening this issue!

• 🚀 You can help us to fix this issue faster by opening a pull request with a failing test. See our Contribution Guide for how to make a pull request, or read our New Contributor's Guide if this is your first time contributing.

[mtrezza]

• It's not clear what you mean by "hanging indefinitely" - do you know whether the file upload is still progressing but maybe very slowly so that it just seems to stall indefinitely?
• What behavior are you expecting if the upload speed is very slow?

[vzukanov]

Providing more information below.

First, when I wrote "hanging indefinitely", I meant that literally: the calling thread is blocked and never released. It's not about waiting a bit longer.

That's the state of my prefetch-thread as seen in the thread dump:

"prefetch-thread@17783" prio=5 tid=0x1e7 nid=NA waiting
  java.lang.Thread.State: WAITING
	 blocks prefetch-thread@17783
	  at java.lang.Object.wait(Object.java:-1)
	  at java.lang.Object.wait(Object.java:442)
	  at java.lang.Object.wait(Object.java:568)
	  at com.parse.boltsinternal.Task.waitForCompletion(Task.java:583)
	  - locked <0x45bc> (a java.lang.Object)
	  at com.parse.ParseTaskUtils.wait(ParseTaskUtils.java:28)
	  at com.parse.ParseFile.getData(ParseFile.java:374)
	  at com.myapp.initialization.PrefetchDataFlowSync.prefetchMaterial(PrefetchDataFlowSync.java:233)

As a workaround, I refactored the code from calling ParseFile.getData() to using regular OkHttp and downloading the file from a URL obtained with ParseFile.getUrl():

            Request request = new Request.Builder().url(materialFile.getUrl()).build();
            OkHttpClient client = new OkHttpClient.Builder().build();
            try (Response response = client.newCall(request).execute()) {
                fileHelper.writeDataToFile(response.body().byteStream(), localMaterialFilePath);
            } catch (IOException e) {
                MyLogger.e("failed to cache the file: " + fileName, e);
            }

This resulted in this explicit TimeoutException:

2022-03-24 10:48:41.114 6601-6816/com.myapp.android E/PrefetchDataFlowSync: [prefetch-thread] failed to cache the file: f7040f68aa136062a2e9aa9ab832e630_Shvilim_14.pdf
    java.net.SocketTimeoutException: timeout
        at okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.kt:677)
        at okhttp3.internal.http2.Http2Stream$StreamTimeout.exitAndThrowIfTimedOut(Http2Stream.kt:686)
        at okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.kt:143)
        at okhttp3.internal.http2.Http2ExchangeCodec.readResponseHeaders(Http2ExchangeCodec.kt:96)
        at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:106)
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:79)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
        at com.myapp.initialization.PrefetchDataFlowSync.prefetchMaterial(PrefetchDataFlowSync.java:236)
        at com.myapp.initialization.PrefetchDataFlowSync.prefetchMaterials(PrefetchDataFlowSync.java:207)
        at com.myapp.initialization.PrefetchDataFlowSync.prefetchDataSync(PrefetchDataFlowSync.java:108)
        at com.myapp.initialization.PrefetchDataFlow.lambda$prefetchDataAndNotify$0$PrefetchDataFlow(PrefetchDataFlow.java:51)
        at com.myapp.initialization.-$$Lambda$PrefetchDataFlow$qV9NPeJSXMVjJSRt70U9cghW590.run(Unknown Source:2)
        at java.lang.Thread.run(Thread.java:923)

Then I tried to increase the timeouts of this custom OkHttpClient to 60 seconds (just like I tried with Parse's OkHttpClient), but the thread got stuck again. Just to be sure, I let the app run for another 10 minutes. Nothing. Similar picture in the thread dump:

"prefetch-thread@17606" prio=5 tid=0x27b nid=NA waiting
  java.lang.Thread.State: WAITING
	 blocks prefetch-thread@17606
	  at java.lang.Object.wait(Object.java:-1)
	  at java.lang.Object.wait(Object.java:442)
	  at java.lang.Object.wait(Object.java:568)
	  at okhttp3.internal.http2.Http2Stream.waitForIo$okhttp(Util.kt:550)
	  at okhttp3.internal.http2.Http2Stream$FramingSource.read(Http2Stream.kt:378)
	  - locked <0x45a6> (a okhttp3.internal.http2.Http2Stream)
	  at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.kt:276)
	  at okio.RealBufferedSource$inputStream$1.read(RealBufferedSource.kt:158)
	  at java.io.InputStream.read(InputStream.java:101)
	  at java.nio.file.Files.copy(Files.java:2908)
	  at java.nio.file.Files.copy(Files.java:3027)
	  at com.myapp.common.files.FileHelper.writeDataToFile(FileHelper.java:175)

As a last resort I tried to override the version of OkHttp used by Parse (4.9.1) to the latest 4.9.3, but it didn't change the situation.

IMO, all this means that the problem is either in OkHttp, or my backend, or both. Even assuming that the backend misbehaves, OkHttp should throw TimeoutException instead of hanging, right?

Empirically, I found that the threshold between TimeoutException and hanging is around 30 seconds (for timeout setting of 25 seconds I get the exception, but for 30 the thread gets stuck).

I guess the next steps from my side should be:

1. Bring OkHttp's maintainers' attention to this issue and ask for their input
2. Open a ticket for Parse Server to debug my backned issue (look like something there, potentially Parse Server itself, fails on long requests)

Any additional ideas?

[vzukanov]

In addition, @mtrezza, the new title you assigned to this issue can be misleading because I'm having troubles with downloading the data, not uploading.

[swankjesse]

Is there a URL I can use to reproduce? What's your nginx version?

[vzukanov]

@swankjesse , thanks for jumping in.

I'm using Ngnix 1.20.2 (LMK if you want to see its config files).

I wouldn't like to share our URLs here, so I'll PM a URL to a test file to you on Twitter.

For your convenience, that's the minimal code sample that reproduces the problem on my end (regardless of Parse):

        String threadName = "test-thread";
        new Thread(() -> {
            Log.d(threadName, "caching the file");
            Request request = new Request.Builder().url("<URL>").build();
            OkHttpClient client = new OkHttpClient.Builder()
                    .readTimeout(40, TimeUnit.SECONDS)
                    .writeTimeout(40, TimeUnit.SECONDS)
                    .build();
            try (Response response = client.newCall(request).execute()) {
                String localFilePath = getContext().getFilesDir().getPath() + File.separator + "test.pdf";
                fileHelper.writeDataToFile(response.body().byteStream(), localFilePath);
                Log.d(threadName, "cached material file at: " + localFilePath);
            } catch (IOException e) {
                Log.e(threadName, "failed to cache the file", e);
            }
        }, threadName).start();

[mtrezza]

I'm having troubles with downloading the data, not uploading.

Thanks for noting and providing the additional details.

Even assuming that the backend misbehaves, OkHttp should throw TimeoutException instead of hanging, right?

Yes I would say so.

[parse-github-assistant]

The label type:bug cannot be used in combination with type:question.