Error when upload file >5G on S3 as external storage

[neomed]

Ubuntu server 16.04
Nexcloud 16.0.2
S3 Amazon mounted as external storage

Whe i upload file more than 5G, i receive this error on log, but file is copied correctly.

Error	files_external	Aws\S3\Exception\S3Exception: Error executing "CopyObject" on "https://xxxxxx.s3.eu-central-1.amazonaws.com/folder/video.mp4"; AWS HTTP error: Client error: PUT https://xxxxxxx.s3.eu-central-1.amazonaws.com/folder/video.mp4 resulted in a 400 Bad Request response: InvalidRequestThe specified copy source is larger than the maximum allowable size for a cop (truncated...) InvalidRequest (client): The specified copy source is larger than the maximum allowable size for a copy source: 5368709120 - InvalidRequestThe specified copy source is larger than the maximum allowable size for a copy source: 5368709120E48037AA295DF6E6

[mdwn]

Looks related to #15579.

[mdwn]

All, I'd implore you to look at this sooner rather than later -- being unable to store files larger than 1 GB in an S3 backed Nextcloud instance basically invalidates my use case for Nextcloud. I can't imagine I'm the only one.

[kesselb]

You can store individual objects of up to 5 TB in Amazon S3. You create a copy of your object up to 5 GB in size in a single atomic operation using this API. However, for copying an object greater than 5 GB, you must use the multipart upload Upload Part - Copy API. For conceptual information, see Copy Object Using the REST Multipart Upload API in the Amazon Simple Storage Service Developer Guide.

https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html

[kesselb]

Index: apps/files_external/lib/Lib/Storage/AmazonS3.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- apps/files_external/lib/Lib/Storage/AmazonS3.php	(revision ecd0d39c304597c77639600a40db25e86fa5a126)
+++ apps/files_external/lib/Lib/Storage/AmazonS3.php	(date 1567598476858)
@@ -36,6 +36,7 @@
 namespace OCA\Files_External\Lib\Storage;
 
 use Aws\Result;
+use Aws\S3\ObjectCopier;
 use Aws\S3\S3Client;
 use Aws\S3\Exception\S3Exception;
 use Icewind\Streams\CallbackWrapper;
@@ -548,12 +549,13 @@
 		$path2 = $this->normalizePath($path2);
 
 		if ($this->is_file($path1)) {
+			$objectCopier = new ObjectCopier(
+				$this->getConnection(),
+				['Bucket' => $this->bucket, 'Key' => $this->cleanKey($path1)],
+				['Bucket' => $this->bucket, 'Key' => $this->cleanKey($path2)]
+			);
 			try {
-				$this->getConnection()->copyObject(array(
-					'Bucket' => $this->bucket,
-					'Key' => $this->cleanKey($path2),
-					'CopySource' => S3Client::encodeKey($this->bucket . '/' . $path1)
-				));
+				$objectCopier->copy();
 				$this->testTimeout();
 			} catch (S3Exception $e) {
 				\OC::$server->getLogger()->logException($e, ['app' => 'files_external']);

Something like this should do the trick. Needs further testing.

[mdwn]

@kesselb Happy to test this manually on my local install. I'll apply the patch and let you know.

[mdwn]

This didn't work for me, unfortunately. Looking at my own logs, what I'm hitting may be unrelated to this bug actually. Not sure what other issue it may map to.

{"reqId":"8JQCVD34eai7GotmFz1t","level":3,"time":"2019-09-04T17:45:20+00:00","remoteAddr":"151.196.189.6","user":"mike","app":"no app in context","method":"MOVE","url":"\/remote.php\/dav\/uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","message":{"Exception":"Exception","Message":"Stream from assembly node shorter than expected, got 294912 bytes, expected 10485760","Code":0,"Trace":[{"function":"stream_read","class":"OCA\\DAV\\Upload\\AssemblyStream","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/Wrapper.php","line":91,"function":"fread","args":[null,8192]},{"file":"\/var\/www\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/CallbackWrapper.php","line":98,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"Icewind\\Streams\\CallbackWrapper","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/Wrapper.php","line":91,"function":"fread","args":[null,8192]},{"file":"\/var\/www\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/CountWrapper.php","line":90,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"Icewind\\Streams\\CountWrapper","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/Wrapper.php","line":91,"function":"fread","args":[null,8192]},{"file":"\/var\/www\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/CallbackWrapper.php","line":98,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"Icewind\\Streams\\CallbackWrapper","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/psr7\/src\/Stream.php","line":218,"function":"fread","args":[null,8192]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/psr7\/src\/LimitStream.php","line":150,"function":"read","class":"GuzzleHttp\\Psr7\\Stream","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud\/3rdparty\/aws\/aws-sdk-php\/src\/HashingStream.php","line":38,"function":"read","class":"GuzzleHttp\\Psr7\\LimitStream","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/psr7\/src\/functions.php","line":378,"function":"read","class":"Aws\\HashingStream","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud\/3rdparty\/aws\/aws-sdk-php\/src\/S3\/MultipartUploader.php","line":117,"function":"GuzzleHttp\\Psr7\\copy_to_stream","args":[{"stream":{"stream":{"__class__":"GuzzleHttp\\Psr7\\Stream"},"__class__":"GuzzleHttp\\Psr7\\LimitStream"},"__class__":"Aws\\HashingStream"},{"__class__":"GuzzleHttp\\Psr7\\Stream"}]},{"file":"\/var\/www\/nextcloud\/3rdparty\/aws\/aws-sdk-php\/src\/Multipart\/AbstractUploader.php","line":53,"function":"createPart","class":"Aws\\S3\\MultipartUploader","type":"->","args":[false,6]},{"file":"\/var\/www\/nextcloud\/3rdparty\/aws\/aws-sdk-php\/src\/CommandPool.php","line":55,"function":"getUploadCommands","class":"Aws\\Multipart\\AbstractUploader","type":"->","args":[{"__class__":"Closure"}]},{"function":"Aws\\{closure}","class":"Aws\\CommandPool","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/EachPromise.php","line":187,"function":"next","class":"Generator","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/EachPromise.php","line":213,"function":"advanceIterator","class":"GuzzleHttp\\Promise\\EachPromise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/EachPromise.php","line":161,"function":"step","class":"GuzzleHttp\\Promise\\EachPromise","type":"->","args":[1]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":203,"function":"GuzzleHttp\\Promise\\{closure}","class":"GuzzleHttp\\Promise\\EachPromise","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":156,"function":"callHandler","class":"GuzzleHttp\\Promise\\Promise","type":"::","args":[1,"*** sensitive parameter replaced ***",[{"__class__":"GuzzleHttp\\Promise\\Promise"},{"__class__":"Closure"},{"__class__":"Closure"}]]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/TaskQueue.php","line":47,"function":"GuzzleHttp\\Promise\\{closure}","class":"GuzzleHttp\\Promise\\Promise","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/guzzle\/src\/Handler\/CurlMultiHandler.php","line":98,"function":"run","class":"GuzzleHttp\\Promise\\TaskQueue","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/guzzle\/src\/Handler\/CurlMultiHandler.php","line":125,"function":"tick","class":"GuzzleHttp\\Handler\\CurlMultiHandler","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":246,"function":"execute","class":"GuzzleHttp\\Handler\\CurlMultiHandler","type":"->","args":["*** sensitive parameter replaced ***"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":223,"function":"invokeWaitFn","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":267,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":225,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":267,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":225,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":62,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Coroutine.php","line":65,"function":"wait","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":246,"function":"GuzzleHttp\\Promise\\{closure}","class":"GuzzleHttp\\Promise\\Coroutine","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":223,"function":"invokeWaitFn","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":267,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":225,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":62,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/aws\/aws-sdk-php\/src\/Multipart\/AbstractUploadManager.php","line":83,"function":"wait","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php","line":92,"function":"upload","class":"Aws\\Multipart\\AbstractUploadManager","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/lib\/private\/Files\/ObjectStore\/ObjectStoreStorage.php","line":452,"function":"writeObject","class":"OC\\Files\\ObjectStore\\S3","type":"->","args":["urn:oid:40206",null]},{"file":"\/var\/www\/nextcloud\/lib\/private\/Files\/Storage\/Wrapper\/Wrapper.php","line":630,"function":"writeStream","class":"OC\\Files\\ObjectStore\\ObjectStoreStorage","type":"->","args":["files\/Live Recordings\/Isenmor\/2019-09-01\/00000001.WAV",null,null]},{"file":"\/var\/www\/nextcloud\/lib\/private\/Files\/Storage\/Wrapper\/Wrapper.php","line":630,"function":"writeStream","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["files\/Live Recordings\/Isenmor\/2019-09-01\/00000001.WAV",null,null]},{"file":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/File.php","line":191,"function":"writeStream","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["files\/Live Recordings\/Isenmor\/2019-09-01\/00000001.WAV",null]},{"file":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/Directory.php","line":156,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[null]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Tree.php","line":316,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["00000001.WAV",null]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Tree.php","line":130,"function":"copyNode","class":"Sabre\\DAV\\Tree","type":"->","args":[{"__class__":"OCA\\DAV\\Upload\\FutureFile"},{"__class__":"OCA\\DAV\\Connector\\Sabre\\Directory"},"00000001.WAV"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Tree.php","line":161,"function":"copy","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","files\/mike\/Live Recordings\/Isenmor\/2019-09-01\/00000001.WAV"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php","line":642,"function":"move","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","files\/mike\/Live Recordings\/Isenmor\/2019-09-01\/00000001.WAV"]},{"function":"httpMove","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"absoluteUrl":"https:\/\/www.phragma.org\/remote.php\/dav\/uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php","line":105,"function":"call_user_func_array","args":[[{"__class__":"Sabre\\DAV\\CorePlugin"},"httpMove"],[{"absoluteUrl":"https:\/\/www.phragma.org\/remote.php\/dav\/uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":479,"function":"emit","class":"Sabre\\Event\\EventEmitter","type":"->","args":["method:MOVE",[{"absoluteUrl":"https:\/\/www.phragma.org\/remote.php\/dav\/uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"https:\/\/www.phragma.org\/remote.php\/dav\/uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Server.php","line":316,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/apps\/dav\/appinfo\/v2\/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/remote.php","line":163,"args":["\/var\/www\/nextcloud\/apps\/dav\/appinfo\/v2\/remote.php"],"function":"require_once"}],"File":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Upload\/AssemblyStream.php","Line":157,"CustomMessage":"--"},"userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/76.0.3809.132 Safari\/537.36","version":"16.0.4.1"}
{"reqId":"8JQCVD34eai7GotmFz1t","level":4,"time":"2019-09-04T17:45:20+00:00","remoteAddr":"151.196.189.6","user":"mike","app":"webdav","method":"MOVE","url":"\/remote.php\/dav\/uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","message":{"Exception":"Sabre\\DAV\\Exception","Message":"Stream from assembly node shorter than expected, got 294912 bytes, expected 10485760","Code":0,"Trace":[{"file":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/File.php","line":243,"function":"convertToSabreException","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[{"__class__":"Exception"}]},{"file":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/Directory.php","line":156,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[null]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Tree.php","line":316,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["00000001.WAV",null]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Tree.php","line":130,"function":"copyNode","class":"Sabre\\DAV\\Tree","type":"->","args":[{"__class__":"OCA\\DAV\\Upload\\FutureFile"},{"__class__":"OCA\\DAV\\Connector\\Sabre\\Directory"},"00000001.WAV"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Tree.php","line":161,"function":"copy","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","files\/mike\/Live Recordings\/Isenmor\/2019-09-01\/00000001.WAV"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php","line":642,"function":"move","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","files\/mike\/Live Recordings\/Isenmor\/2019-09-01\/00000001.WAV"]},{"function":"httpMove","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"absoluteUrl":"https:\/\/www.phragma.org\/remote.php\/dav\/uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php","line":105,"function":"call_user_func_array","args":[[{"__class__":"Sabre\\DAV\\CorePlugin"},"httpMove"],[{"absoluteUrl":"https:\/\/www.phragma.org\/remote.php\/dav\/uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":479,"function":"emit","class":"Sabre\\Event\\EventEmitter","type":"->","args":["method:MOVE",[{"absoluteUrl":"https:\/\/www.phragma.org\/remote.php\/dav\/uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"https:\/\/www.phragma.org\/remote.php\/dav\/uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Server.php","line":316,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/apps\/dav\/appinfo\/v2\/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/remote.php","line":163,"args":["\/var\/www\/nextcloud\/apps\/dav\/appinfo\/v2\/remote.php"],"function":"require_once"}],"File":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/File.php","Line":634,"Previous":{"Exception":"Exception","Message":"Stream from assembly node shorter than expected, got 294912 bytes, expected 10485760","Code":0,"Trace":[{"function":"stream_read","class":"OCA\\DAV\\Upload\\AssemblyStream","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/Wrapper.php","line":91,"function":"fread","args":[null,8192]},{"file":"\/var\/www\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/CallbackWrapper.php","line":98,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"Icewind\\Streams\\CallbackWrapper","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/Wrapper.php","line":91,"function":"fread","args":[null,8192]},{"file":"\/var\/www\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/CountWrapper.php","line":90,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"Icewind\\Streams\\CountWrapper","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/Wrapper.php","line":91,"function":"fread","args":[null,8192]},{"file":"\/var\/www\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/CallbackWrapper.php","line":98,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"Icewind\\Streams\\CallbackWrapper","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/psr7\/src\/Stream.php","line":218,"function":"fread","args":[null,8192]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/psr7\/src\/LimitStream.php","line":150,"function":"read","class":"GuzzleHttp\\Psr7\\Stream","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud\/3rdparty\/aws\/aws-sdk-php\/src\/HashingStream.php","line":38,"function":"read","class":"GuzzleHttp\\Psr7\\LimitStream","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/psr7\/src\/functions.php","line":378,"function":"read","class":"Aws\\HashingStream","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud\/3rdparty\/aws\/aws-sdk-php\/src\/S3\/MultipartUploader.php","line":117,"function":"GuzzleHttp\\Psr7\\copy_to_stream","args":[{"stream":{"stream":{"__class__":"GuzzleHttp\\Psr7\\Stream"},"__class__":"GuzzleHttp\\Psr7\\LimitStream"},"__class__":"Aws\\HashingStream"},{"__class__":"GuzzleHttp\\Psr7\\Stream"}]},{"file":"\/var\/www\/nextcloud\/3rdparty\/aws\/aws-sdk-php\/src\/Multipart\/AbstractUploader.php","line":53,"function":"createPart","class":"Aws\\S3\\MultipartUploader","type":"->","args":[false,6]},{"file":"\/var\/www\/nextcloud\/3rdparty\/aws\/aws-sdk-php\/src\/CommandPool.php","line":55,"function":"getUploadCommands","class":"Aws\\Multipart\\AbstractUploader","type":"->","args":[{"__class__":"Closure"}]},{"function":"Aws\\{closure}","class":"Aws\\CommandPool","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/EachPromise.php","line":187,"function":"next","class":"Generator","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/EachPromise.php","line":213,"function":"advanceIterator","class":"GuzzleHttp\\Promise\\EachPromise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/EachPromise.php","line":161,"function":"step","class":"GuzzleHttp\\Promise\\EachPromise","type":"->","args":[1]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":203,"function":"GuzzleHttp\\Promise\\{closure}","class":"GuzzleHttp\\Promise\\EachPromise","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":156,"function":"callHandler","class":"GuzzleHttp\\Promise\\Promise","type":"::","args":[1,"*** sensitive parameter replaced ***",[{"__class__":"GuzzleHttp\\Promise\\Promise"},{"__class__":"Closure"},{"__class__":"Closure"}]]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/TaskQueue.php","line":47,"function":"GuzzleHttp\\Promise\\{closure}","class":"GuzzleHttp\\Promise\\Promise","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/guzzle\/src\/Handler\/CurlMultiHandler.php","line":98,"function":"run","class":"GuzzleHttp\\Promise\\TaskQueue","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/guzzle\/src\/Handler\/CurlMultiHandler.php","line":125,"function":"tick","class":"GuzzleHttp\\Handler\\CurlMultiHandler","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":246,"function":"execute","class":"GuzzleHttp\\Handler\\CurlMultiHandler","type":"->","args":["*** sensitive parameter replaced ***"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":223,"function":"invokeWaitFn","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":267,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":225,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":267,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":225,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":62,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Coroutine.php","line":65,"function":"wait","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":246,"function":"GuzzleHttp\\Promise\\{closure}","class":"GuzzleHttp\\Promise\\Coroutine","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":223,"function":"invokeWaitFn","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":267,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":225,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php","line":62,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/3rdparty\/aws\/aws-sdk-php\/src\/Multipart\/AbstractUploadManager.php","line":83,"function":"wait","class":"GuzzleHttp\\Promise\\Promise","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php","line":92,"function":"upload","class":"Aws\\Multipart\\AbstractUploadManager","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/lib\/private\/Files\/ObjectStore\/ObjectStoreStorage.php","line":452,"function":"writeObject","class":"OC\\Files\\ObjectStore\\S3","type":"->","args":["urn:oid:40206",null]},{"file":"\/var\/www\/nextcloud\/lib\/private\/Files\/Storage\/Wrapper\/Wrapper.php","line":630,"function":"writeStream","class":"OC\\Files\\ObjectStore\\ObjectStoreStorage","type":"->","args":["files\/Live Recordings\/Isenmor\/2019-09-01\/00000001.WAV",null,null]},{"file":"\/var\/www\/nextcloud\/lib\/private\/Files\/Storage\/Wrapper\/Wrapper.php","line":630,"function":"writeStream","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["files\/Live Recordings\/Isenmor\/2019-09-01\/00000001.WAV",null,null]},{"file":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/File.php","line":191,"function":"writeStream","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["files\/Live Recordings\/Isenmor\/2019-09-01\/00000001.WAV",null]},{"file":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/Directory.php","line":156,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[null]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Tree.php","line":316,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["00000001.WAV",null]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Tree.php","line":130,"function":"copyNode","class":"Sabre\\DAV\\Tree","type":"->","args":[{"__class__":"OCA\\DAV\\Upload\\FutureFile"},{"__class__":"OCA\\DAV\\Connector\\Sabre\\Directory"},"00000001.WAV"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Tree.php","line":161,"function":"copy","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","files\/mike\/Live Recordings\/Isenmor\/2019-09-01\/00000001.WAV"]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php","line":642,"function":"move","class":"Sabre\\DAV\\Tree","type":"->","args":["uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","files\/mike\/Live Recordings\/Isenmor\/2019-09-01\/00000001.WAV"]},{"function":"httpMove","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"absoluteUrl":"https:\/\/www.phragma.org\/remote.php\/dav\/uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php","line":105,"function":"call_user_func_array","args":[[{"__class__":"Sabre\\DAV\\CorePlugin"},"httpMove"],[{"absoluteUrl":"https:\/\/www.phragma.org\/remote.php\/dav\/uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":479,"function":"emit","class":"Sabre\\Event\\EventEmitter","type":"->","args":["method:MOVE",[{"absoluteUrl":"https:\/\/www.phragma.org\/remote.php\/dav\/uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"\/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"https:\/\/www.phragma.org\/remote.php\/dav\/uploads\/mike\/web-file-upload-6b1ca89cec171267eab37f3e53afd9cc-1567611039325\/.file","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Server.php","line":316,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/apps\/dav\/appinfo\/v2\/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"\/var\/www\/nextcloud\/remote.php","line":163,"args":["\/var\/www\/nextcloud\/apps\/dav\/appinfo\/v2\/remote.php"],"function":"require_once"}],"File":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Upload\/AssemblyStream.php","Line":157},"CustomMessage":"--"},"userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/76.0.3809.132 Safari\/537.36","version":"16.0.4.1"}

[kesselb]

You are using object storage right? The patch above is for s3 as external storage (because the title of the issue says external storage). This might work for @neomed 🏓

For object storage this needs to be fixed in a different way. The object storage works identical for every storage backend. A copy operation is a read and write operation.

Your stack trace shows that the object storage is already using AbstractUploadManager and should support writing files up to any size. Would you mind to add the https://github.com/nextcloud/server/blob/master/.github/ISSUE_TEMPLATE/Bug_report.md issue template with your information to #15579? (I think its similar to your issue)

[kesselb]

I think its similar to your issue

Nope. I'm wrong about that ;) The other issue says it's using S3 as primary object storage but the stacktrace says external storage (like here).

Would you mind to add the https://github.com/nextcloud/server/blob/master/.github/ISSUE_TEMPLATE/Bug_report.md issue template

If you cannot find a open issue here about object storage and s3 please open a new issue.

[szaimen]

I'm closing this issue due to inactivity. If this is still happening please make sure to upgrade to the latest version. After that, feel free to reopen.