Can't open / sync big files on external storage (webdav)

[GAS85]

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Steps to reproduce

1. Add webdav external storage
2. Tries to open big file there (e.g. 800 MB) or Sync folder via client
3. See an error
4. Check the logs and see:

GuzzleHttp\Exception\ConnectException: cURL error 28: Operation timed out after 30000 milliseconds with 379564427 out of 838860800 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://magentacloud.de/remote.php/webdav/System/NewTea.avi

Seems it is related to #30262 and #29969 with #26071
5. Use Workaround as per #26071 (comment)
6. See that Web UI is now opening, but with a huge delay (whole file being downloaded), but client sync does not work still.

Expected behaviour

Do not download files (as per #30262) when folder simply opened, use cache instead.
Do not download files when properties only needed (last modification date, or size)

Actual behaviour

External Storage could not be synchronized, even via Desktop Client it simply does not open and not synced any more.
All files downloaded to /tmp or RAM and causing timeout.

Server configuration

Operating system: Ubuntu 20.04

Web server: Apache/2.4.51

Database: 10.3.32-MariaDB

PHP version: 7.4.26

Nextcloud version: 22.2.3.0

Updated from an older Nextcloud/ownCloud or fresh install: updated

Where did you install Nextcloud from: Official

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - accessibility: 1.8.0
  - activity: 2.15.0
  - admin_audit: 1.12.0
  - audioplayer: 3.2.2
  - bruteforcesettings: 2.2.0
  - calendar: 3.0.0
  - camerarawpreviews: 0.7.15
  - checksum: 1.1.2
  - cloud_federation_api: 1.5.0
  - comments: 1.12.0
  - contacts: 4.0.6
  - contactsinteraction: 1.3.0
  - cospend: 1.4.3
  - dashboard: 7.2.0
  - data_request: 1.9.0
  - dav: 1.19.0
  - deck: 1.5.5
  - drawio: 1.0.1
  - external: 3.9.0
  - extract: 1.3.2
  - facerecognition: 0.8.5
  - federatedfilesharing: 1.12.0
  - files: 1.17.0
  - files_automatedtagging: 1.12.0
  - files_downloadactivity: 1.11.1
  - files_external: 1.13.0
  - files_mindmap: 0.0.25
  - files_pdfviewer: 2.3.1
  - files_retention: 1.11.1
  - files_rightclick: 1.1.0
  - files_sharing: 1.14.0
  - files_trashbin: 1.12.0
  - files_versions: 1.15.0
  - files_videoplayer: 1.11.0
  - firstrunwizard: 2.11.0
  - forms: 2.4.0
  - gpxedit: 0.0.14
  - gpxpod: 4.3.0
  - integration_github: 1.0.2
  - integration_gitlab: 1.0.3
  - keeweb: 0.6.8
  - logreader: 2.7.0
  - lookup_server_connector: 1.10.0
  - mail: 1.11.0
  - maps: 0.1.9
  - nextcloud_announcements: 1.11.0
  - notes: 4.2.0
  - notifications: 2.10.1
  - oauth2: 1.10.0
  - ocdownloader: 1.7.12
  - password_policy: 1.12.0
  - phonetrack: 0.6.9
  - photos: 1.4.0
  - polls: 3.3.0
  - previewgenerator: 3.3.0
  - privacy: 1.6.0
  - provisioning_api: 1.12.0
  - recommendations: 1.1.0
  - serverinfo: 1.12.0
  - settings: 1.4.0
  - sharebymail: 1.12.0
  - side_menu: 2.0.1
  - spreed: 12.1.2
  - support: 1.5.0
  - survey_client: 1.10.0
  - systemtags: 1.12.0
  - text: 3.3.0
  - theming: 1.13.0
  - twofactor_backupcodes: 1.11.0
  - twofactor_totp: 6.1.0
  - unsplash: 1.2.4
  - updatenotification: 1.12.0
  - user_status: 1.2.0
  - video_converter: 1.0.3
  - viewer: 1.6.0
  - weather_status: 1.2.0
  - workflowengine: 2.4.0
Disabled:
  - circles
  - encryption
  - federation
  - files_accesscontrol
  - flowupload
  - gpxmotion
  - impersonate
  - integration_jira
  - radio
  - recognize
  - user_ldap
  - weather

Nextcloud configuration:

Config report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "22.2.3.0",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "filesystem_check_changes": 0,
        "filelocking.enabled": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 1.5
        },
        "default_phone_region": "DE",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "theme": "",
        "logfile": "\/var\/nextcloud\/data\/nextcloud.log",
        "loglevel": 1,
        "trashbin_retention_obligation": "14, auto",
        "versions_retention_obligation": "14, auto",
        "data-fingerprint": "7be51475e95ea13b254cd880319d98ec",
        "enable_previews": true,
        "enabledPreviewProviders": [
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\GIF",
            "OC\\Preview\\BMP",
            "OC\\Preview\\XBitmap",
            "OC\\Preview\\Movie",
            "OC\\Preview\\PDF",
            "OC\\Preview\\MP3",
            "OC\\Preview\\TXT",
            "OC\\Preview\\MarkDown"
        ],
        "preview_max_x": 1920,
        "preview_max_y": 1080,
        "jpeg_quality": 90,
        "auth.bruteforce.protection.enabled": true,
        "simpleSignUpLink.shown": false,
        "mail_smtpsecure": "tls",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "twofactor_enforced": true,
        "twofactor_enforced_groups": [
            "admin"
        ],
        "twofactor_enforced_excluded_groups": [],
        "has_rebuilt_cache": true,
        "updater.release.channel": "stable",
        "mail_sendmailmode": "smtp"
    }
}

Are you using external storage, if yes which one: local/webdav

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Logs

Nextcloud log (data/nextcloud.log)

Nextcloud log

{"reqId":"3bdICeP14G6vPYqfCKqQ","level":3,"time":"2021-12-15T09:30:33+00:00","remoteAddr":"","user":"--","app":"files","method":"","url":"--","message":"cURL error 28: Operation timed out after 30000 milliseconds with 379564427 out of 838860800 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://magentacloud.de/remote.php/webdav/System/NewTea.avi","userAgent":"--","version":"22.2.3.0","exception":{"Exception":"GuzzleHttp\\Exception\\ConnectException","Message":"cURL error 28: Operation timed out after 30000 milliseconds with 379564427 out of 838860800 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://magentacloud.de/remote.php/webdav/System/NewTea.avi","Code":0,"Trace":[{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":158,"function":"createRejection","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":110,"function":"finishError","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlHandler.php","line":47,"function":"finish","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/lib/private/Http/Client/DnsPinMiddleware.php","line":146,"function":"__invoke","class":"GuzzleHttp\\Handler\\CurlHandler","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php","line":35,"function":"OC\\Http\\Client\\{closure}","class":"OC\\Http\\Client\\DnsPinMiddleware","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":31,"function":"__invoke","class":"GuzzleHttp\\PrepareBodyMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php","line":71,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":63,"function":"__invoke","class":"GuzzleHttp\\RedirectMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/HandlerStack.php","line":75,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":331,"function":"__invoke","class":"GuzzleHttp\\HandlerStack","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":168,"function":"transfer","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":187,"function":"requestAsync","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Http/Client/Client.php","line":223,"function":"request","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/DAV.php","line":356,"function":"get","class":"OC\\Http\\Client\\Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/LocalTempFileTrait.php","line":65,"function":"fopen","class":"OC\\Files\\Storage\\DAV","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/LocalTempFileTrait.php","line":48,"function":"toTmpFile","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Common.php","line":269,"function":"getCachedFile","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":367,"function":"getLocalFile","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php","line":374,"function":"getLocalFile","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":367,"function":"getLocalFile","class":"OC\\Files\\Storage\\Wrapper\\Availability","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Check/FileMimeType.php","line":131,"function":"getLocalFile","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Check/FileMimeType.php","line":109,"function":"getActualValue","class":"OCA\\WorkflowEngine\\Check\\FileMimeType","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Service/RuleMatcher.php","line":245,"function":"executeCheck","class":"OCA\\WorkflowEngine\\Check\\FileMimeType","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Service/RuleMatcher.php","line":180,"function":"check","class":"OCA\\WorkflowEngine\\Service\\RuleMatcher","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Service/RuleMatcher.php","line":128,"function":"getMatchingOperations","class":"OCA\\WorkflowEngine\\Service\\RuleMatcher","type":"->"},{"file":"/var/www/nextcloud/apps/files_automatedtagging/lib/Operation.php","line":95,"function":"getFlows","class":"OCA\\WorkflowEngine\\Service\\RuleMatcher","type":"->"},{"file":"/var/www/nextcloud/apps/files_automatedtagging/lib/Listener/CacheListener.php","line":43,"function":"checkOperations","class":"OCA\\FilesAutomatedTagging\\Operation","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php","line":75,"function":"handle","class":"OCA\\FilesAutomatedTagging\\Listener\\CacheListener","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":251,"function":"__invoke","class":"OC\\EventDispatcher\\ServiceEventListener","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":73,"function":"callListeners","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":88,"function":"dispatch","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":100,"function":"dispatch","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Cache.php","line":324,"function":"dispatchTyped","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":294,"function":"insert","class":"OC\\Files\\Cache\\Cache","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":224,"function":"addToCache","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":427,"function":"scanFile","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":388,"function":"handleChildren","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":340,"function":"scanChildren","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":510,"function":"scan","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":521,"function":"OC\\Files\\Cache\\{closure}","class":"OC\\Files\\Cache\\Scanner","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":509,"function":"runBackgroundScanJob","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Utils/Scanner.php","line":185,"function":"backgroundScan","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/apps/files/lib/BackgroundJob/ScanFiles.php","line":89,"function":"backgroundScan","class":"OC\\Files\\Utils\\Scanner","type":"->"},{"file":"/var/www/nextcloud/apps/files/lib/BackgroundJob/ScanFiles.php","line":125,"function":"runScanner","class":"OCA\\Files\\BackgroundJob\\ScanFiles","type":"->"},{"file":"/var/www/nextcloud/lib/private/BackgroundJob/Job.php","line":51,"function":"run","class":"OCA\\Files\\BackgroundJob\\ScanFiles","type":"->"},{"file":"/var/www/nextcloud/lib/private/BackgroundJob/TimedJob.php","line":58,"function":"execute","class":"OC\\BackgroundJob\\Job","type":"->"},{"file":"/var/www/nextcloud/cron.php","line":127,"function":"execute","class":"OC\\BackgroundJob\\TimedJob","type":"->"}],"File":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","Line":210,"CustomMessage":"--"},"id":"61b9f1e0b44fa"}

Nextcloud Desktop Client log

Nextcloud Client log

||MagentaCloud|2|2|1638616226|_invalid_|0|10006743xxxxxxxxxxx|4||0|0|0||
#=#=#=# Syncrun finished 2021-12-15T13:59:38Z (last step: 157 msec, total: 2134 msec)
#=#=#=# Syncrun started 2021-12-15T13:59:40Z
#=#=#=#=# Propagation starts 2021-12-15T14:00:24Z (last step: 43733 msec, total: 43733 msec)
#=#=#=# Syncrun finished 2021-12-15T14:00:24Z (last step: 24 msec, total: 43757 msec)
#=#=#=# Syncrun started 2021-12-15T14:01:03Z
#=#=#=#=# Propagation starts 2021-12-15T14:01:03Z (last step: 371 msec, total: 371 msec)
||MagentaCloud|8|2|1638616226|61ab4ca2b0b9c|0|10006743xxxxxxxxxxx|4||0|0|0||
#=#=#=# Syncrun finished 2021-12-15T14:01:03Z (last step: 28 msec, total: 399 msec)

[GAS85]

Also I can't open e.g. Videos from the external Storages via UI any more...

How it works before:

1. Go to external storage
2. Click on Video file
3. Open immediately - nextlcoud works aka Proxy to the external Storage.

How it works now:

1. Go to external storage
2. Click on Video file
3. See that Traffic goes up and memory usage grows:
   
4. Check /tmp Video files are there:

ls -lah /tmp | grep "www-data"
-rw-------  1 www-data www-data 800M Dec 16 08:52 oc_tmp_uLpKpT-.avi
-rw-------  1 www-data www-data  50M Dec 16 08:53 php0NKva5
-rw-------  1 www-data www-data 100M Dec 16 08:53 phpYRlgBW
-rw-------  1 www-data www-data 152M Dec 16 08:53 phptIAg9A
-rw-------  1 www-data www-data 200M Dec 16 08:53 phpwbuY14

5. Nothing will be transferred till full Video downloaded from the external storage.
   
6. Close Video window, but traffic on backend will not be stopp till Video fully downloaded.
   
7. Nextcloud works aka caching contend provider to the external Storage.

[liejuntao001]

Add a data point.
I upgraded from 21.0.4 to 22.2.3.
The problem didn't happen in 21.0.4.
With 22.2.3 it's a new behavior that every time when user try to download a file from the webdav external storage, the file needs to be downloadeded to /tmp from the server side, then the web download could start.
If the server download could not finish within 30 seconds, the error happens.
A related, but might be different problem, the webdav external storage speed is degraded in 22 comparing to 21.0.4.
When do a full scan (occ scan ), it takes 10 minutes in 21 but 16 minutes in 22.

[liejuntao001]

This problem is still in 22.2.5.
When the user access a file and the WebDAV client is involved, Nextcloud would copy the whole file from remote to local /tmp folder. If the file is large, the delay could be very long and trigger curl time out.
This applies to the external webdav storage, and to the federation sharing.

[LukaszSellmann]

Problem still occurs in 23.0.2
External samba storage, folders with files larger than 512mb doesn't sync on entering them and shows as empty.
Command occ fieles:scan doesn't sync them as well.

[sk1p]

I'm pretty sure I've tracked down the issue. In #26266 the way Guzzle HTTP clients are constructed was changed from the default, which chose the handler based on the environment, to always use the curl handler. Just using the curl handler doesn't seem to support the "stream" argument like the default handler stack does - you need to somehow directly pass in the sink option, otherwise the response is written to a temporary file.

The following patch, on top of 23.0.3.2 works around this issue, although it also reverts the security improvements from said PR:

diff -ruN ./1/lib/private/Http/Client/ClientService.php ./0/lib/private/Http/Client/ClientService.php 
--- ./1/lib/private/Http/Client/ClientService.php   2022-03-21 13:02:41.000000000 +0000 
+++ ./0/lib/private/Http/Client/ClientService.php   2022-06-02 19:31:07.114812663 +0000 
@@ -68,11 +68,7 @@ 
         * @return Client 
         */ 
        public function newClient(): IClient { 
-               $handler = new CurlHandler(); 
-               $stack = HandlerStack::create($handler); 
-               $stack->push($this->dnsPinMiddleware->addDnsPinning()); 
- 
-               $client = new GuzzleClient(['handler' => $stack]); 
+               $client = new GuzzleClient(); 
 
                return new Client( 
                        $this->config,

@LukasReschke @rullzer could you have a look?

[kesselb]

Good finding 👍

php-curl is a hard requirement for guzzle.

But you are right we are always using CurlHandler. The default implementation does a bit more: https://github.com/nextcloud/3rdparty/blob/ce3a3cedfd36dd914ec103a62b8a0f47e950a0e1/guzzlehttp/guzzle/src/Utils.php#L77-L106

Index: lib/private/Http/Client/ClientService.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/lib/private/Http/Client/ClientService.php b/lib/private/Http/Client/ClientService.php
--- a/lib/private/Http/Client/ClientService.php	(revision e5309201f7479be1464fe10c71e4a039dddd26b9)
+++ b/lib/private/Http/Client/ClientService.php	(date 1654617117944)
@@ -63,8 +63,7 @@
 	 * @return Client
 	 */
 	public function newClient(): IClient {
-		$handler = new CurlHandler();
-		$stack = HandlerStack::create($handler);
+		$stack = HandlerStack::create();
 		$stack->push($this->dnsPinMiddleware->addDnsPinning());
 
 		$client = new GuzzleClient(['handler' => $stack]);

Does this work too?

[uellue]

@kesselb Confirmed to work! 👍