Nextcloud fails to handle large files when using S3 object store (with fix)

[kaymes]

Steps to reproduce

1. Configure Nextcloud to use Amazon's S3 as Primary object store.
2. Upload a file which is larger than 5 GB.
3. Check the log to see a fatal error containing Amazon's message "Your proposed upload exceeds the maximum allowed size"

Expected behaviour

Nextcloud should have obeyed Amazon's 5 GB limit on single transfer uploads and split the file into a multi part upload.

Actual behaviour

Nextcloud attempted to upload the file in a single transfer despite being larger than 5 GB.

Server configuration

Operating system:
Ubuntu

Web server:
Apache

Database:
MariaDB

PHP version:
7

Nextcloud version: (see Nextcloud admin page)
12.03

Updated from an older Nextcloud/ownCloud or fresh install:
fresh install

Where did you install Nextcloud from:
nextcloud.com

Signing status:

Signing status

Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

No errors have been found.

List of activated apps:

App list

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your Nextcloud installation folder

Nextcloud configuration:

Config report

<?php
$CONFIG = array (
  'objectstore' => 
  array (
    'class' => 'OC\\Files\\ObjectStore\\S3',
    'arguments' => 
    array (
      'bucket' => 'XXX',
      'autocreate' => false,
      'key' => 'XXX',
      'secret' => 'XXX',
      'use_ssl' => true,
      'region' => 'ap-southeast-2',
      'hostname' => 's3-ap-southeast-2.amazonaws.com',
      'upload_min_part_size' => 1024*1024*100,
    ),
  ),
  'passwordsalt' => 'XXX',
  'secret' => 'XXX',
  'trusted_domains' => 
  array (
    0 => 'nc.kseiler.de',
  ),
  'datadirectory' => '/var/www/nextcloud/data',
  'overwrite.cli.url' => 'XXX',
  'dbtype' => 'mysql',
  'version' => '12.0.3.3',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'XXX',
  'dbpassword' => 'XXX',
  'installed' => true,
  'instanceid' => 'XXX',
  'mail_smtpmode' => 'smtp',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'XXX',
  'mail_smtpport' => '465',
  'mail_from_address' => 'noreply',
  'mail_domain' => 'XXX',
  'mail_smtpsecure' => 'ssl',
  'mail_smtpname' => 'XXX',
  'mail_smtppassword' => 'XXX',

  'htaccess.RewriteBase' => '/',

  'filelocking.ttl' => 60*60,

);

**Are you using external storage, if yes which one:** local/smb/sftp/... S3

Are you using encryption: yes/no
No

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
No

Nextcloud log (data/nextcloud.log)

Nextcloud log

{"reqId":"Og22UrP9l27qiKbLh9Nu","level":3,"time":"2017-10-09T05:12:40+00:00","remoteAddr":"127.0.0.1","user":"XXX","app":"objectstore","method":"PUT","url":"\/remote.php\/dav\/files\/XXX","message":"Could not create object urn:oid:112521 for files\/XXX: {\"Exception\":\"Aws\\\\S3\\\\Exception\\\\EntityTooLargeException\",\"Message\":\"Your proposed upload exceeds the maximum allowed size\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/files_external\\\/3rdparty\\\/aws-sdk-php\\\/Aws\\\/Common\\\/Exception\\\/NamespaceExceptionFactory.php(76): Aws\\\\Common\\\\Exception\\\\NamespaceExceptionFactory->createException('Aws\\\\\\\\S3\\\\\\\\Exceptio...', Object(Guzzle\\\\Http\\\\Message\\\\EntityEnclosingRequest), Object(Guzzle\\\\Http\\\\Message\\\\Response), Array)\\n#1 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/files_external\\\/3rdparty\\\/aws-sdk-php\\\/Aws\\\/Common\\\/Exception\\\/ExceptionListener.php(55): Aws\\\\Common\\\\Exception\\\\NamespaceExceptionFactory->fromResponse(Object(Guzzle\\\\Http\\\\Message\\\\EntityEnclosingRequest), Object(Guzzle\\\\Http\\\\Message\\\\Response))\\n#2 [internal function]: Aws\\\\Common\\\\Exception\\\\ExceptionListener->onRequestError(Object(Guzzle\\\\Common\\\\Event), 'request.error', Object(Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher))\\n#3 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/symfony\\\/event-dispatcher\\\/EventDispatcher.php(174): call_user_func(Array, Object(Guzzle\\\\Common\\\\Event), 'request.error', Object(Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher))\\n#4 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/symfony\\\/event-dispatcher\\\/EventDispatcher.php(43): Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher->doDispatch(Array, 'request.error', Object(Guzzle\\\\Common\\\\Event))\\n#5 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Message\\\/Request.php(589): Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher->dispatch('request.error', Object(Guzzle\\\\Common\\\\Event))\\n#6 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Message\\\/Request.php(378): Guzzle\\\\Http\\\\Message\\\\Request->processResponse(Array)\\n#7 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Message\\\/EntityEnclosingRequest.php(49): Guzzle\\\\Http\\\\Message\\\\Request->setState('complete', Array)\\n#8 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMulti.php(286): Guzzle\\\\Http\\\\Message\\\\EntityEnclosingRequest->setState('complete', Array)\\n#9 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMulti.php(244): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->processResponse(Object(Guzzle\\\\Http\\\\Message\\\\EntityEnclosingRequest), Object(Guzzle\\\\Http\\\\Curl\\\\CurlHandle), Array)\\n#10 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMulti.php(227): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->processMessages()\\n#11 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMulti.php(211): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->executeHandles()\\n#12 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMulti.php(105): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->perform()\\n#13 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Curl\\\/CurlMultiProxy.php(91): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->send()\\n#14 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Http\\\/Client.php(282): Guzzle\\\\Http\\\\Curl\\\\CurlMultiProxy->send()\\n#15 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/files_external\\\/3rdparty\\\/aws-sdk-php\\\/Aws\\\/Common\\\/Client\\\/AbstractClient.php(262): Guzzle\\\\Http\\\\Client->send(Object(Guzzle\\\\Http\\\\Message\\\\EntityEnclosingRequest))\\n#16 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Service\\\/Client.php(136): Aws\\\\Common\\\\Client\\\\AbstractClient->send(Object(Guzzle\\\\Http\\\\Message\\\\EntityEnclosingRequest))\\n#17 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Service\\\/Command\\\/AbstractCommand.php(153): Guzzle\\\\Service\\\\Client->execute(Object(Aws\\\\S3\\\\Command\\\\S3Command))\\n#18 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Service\\\/Command\\\/AbstractCommand.php(189): Guzzle\\\\Service\\\\Command\\\\AbstractCommand->execute()\\n#19 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzle\\\/guzzle\\\/src\\\/Guzzle\\\/Service\\\/Client.php(76): Guzzle\\\\Service\\\\Command\\\\AbstractCommand->getResult()\\n#20 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/files_external\\\/3rdparty\\\/aws-sdk-php\\\/Aws\\\/Common\\\/Client\\\/AbstractClient.php(107): Guzzle\\\\Service\\\\Client->__call('PutObject', Array)\\n#21 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/ObjectStore\\\/S3.php(90): Aws\\\\Common\\\\Client\\\\AbstractClient->__call('putObject', Array)\\n#22 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/ObjectStore\\\/ObjectStoreStorage.php(399): OC\\\\Files\\\\ObjectStore\\\\S3->writeObject('urn:oid:112521', Resource id #83)\\n#23 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/ObjectStore\\\/ObjectStoreStorage.php(306): OC\\\\Files\\\\ObjectStore\\\\ObjectStoreStorage->writeBack('\\\/tmp\\\/oc_tmp_4OI...', 'files\\\/Dropbox\\\/P...')\\n#24 [internal function]: OC\\\\Files\\\\ObjectStore\\\\ObjectStoreStorage->OC\\\\Files\\\\ObjectStore\\\\{closure}()\\n#25 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/files_external\\\/3rdparty\\\/icewind\\\/streams\\\/src\\\/CallbackWrapper.php(109): call_user_func(Object(Closure))\\n#26 [internal function]: Icewind\\\\Streams\\\\CallbackWrapper->stream_close()\\n#27 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php(135): fclose(Resource id #78)\\n#28 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/Directory.php(151): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->put(Resource id #71)\\n#29 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(1096): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Directory->createFile('internal (all)....', Resource id #71)\\n#30 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(525): Sabre\\\\DAV\\\\Server->createFile('files\\\/kaymes\\\/Dr...', Resource id #71, NULL)\\n#31 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpPut(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#32 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#33 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(479): Sabre\\\\Event\\\\EventEmitter->emit('method:PUT', Array)\\n#34 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#35 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Server.php(258): Sabre\\\\DAV\\\\Server->exec()\\n#36 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/dav\\\/appinfo\\\/v2\\\/remote.php(33): OCA\\\\DAV\\\\Server->exec()\\n#37 \\\/var\\\/www\\\/nextcloud\\\/remote.php(162): require_once('\\\/var\\\/www\\\/nextcl...')\\n#38 {main}\",\"File\":\"\\\/var\\\/www\\\/nextcloud\\\/apps\\\/files_external\\\/3rdparty\\\/aws-sdk-php\\\/Aws\\\/Common\\\/Exception\\\/NamespaceExceptionFactory.php\",\"Line\":91}","userAgent":"davfs2\/1.5.2 neon\/0.30.1","version":"12.0.3.3"}

[kaymes]

In ./lib/private/Files/ObjectStore/S3.php, the method S3->writeObject() uses the putObject() method of the S3Client which only does single uploads. Instead, the upload() method should be used to automatically create multi part uploads if the file is larger than a certain threshold.

I replaced the function by the following code:

function writeObject($urn, $stream) {
                $opts = array();
                if (isset($this->params['upload_min_part_size'])) {
                        $opts['min_part_size'] = $this->params['upload_min_part_size'];
                }
                $this->getConnection()->upload($this->bucket, $urn, $stream, 'private', $opts);
        }

The user can even set the parameter upload_min_part_size in the config.php to manually set the threshold (which has to be between 5 MB and 5 GB). Amazon recommends to consider multi part uploads for all files above 100 MB. But smaller might benefit, too because multi part uploads can happen using concurrent connections (The default of the aws code is up to 3 connections for normal files and doing multipart from 5 MB).

In the current development code on github, the writeObject method seems to have been moved into lib/private/Files/ObjectStore/S3ObjectTrait.php. I believe, the fix would still work though.

[nickvergessen]

Should be fixed by #6602