Difference Between Streamed Upload and Multipart Upload
Updated at: Mar 24, 2022 GMT+08:00
Performing a Multipart Upload
If yous have whatever questions during development, post them on the Issues folio of GitHub. For details about parameters and usage of each API, see the API Reference
To upload a large file, multipart upload is recommended. Multipart upload is applicable to many scenarios, including:
- Files to be uploaded are larger than 100 MB.
- The network status is poor. Connexion to the OBS server is constantly downward.
- Sizes of files to be uploaded are uncertain.
Multipart upload consists of iii phases:
- Initialize a multipart upload (ObsClient.initiateMultipartUpload).
- Upload parts one by 1 or concurrently (ObsClient.uploadPart).
- Combine parts (ObsClient.completeMultipartUpload) or abort the multipart upload (ObsClient.abortMultipartUpload).
Initializing a Multipart Upload
Before upload, you demand to notify OBS of initializing a multipart upload. This operation will return an upload ID (globally unique identifier) created past the OBS server to identify the multipart upload. Yous tin utilize this upload ID to initiate related operations, such as aborting the multipart upload, listing multipart uploads, and listing uploaded parts.
You tin call ObsClient.initiateMultipartUpload to initialize a multipart upload.
// Import the OBS library. // Use npm to install the client. var ObsClient = require('esdk-obs-nodejs'); // Utilise source codes to install the client. // var ObsClient = require('./lib/obs'); // Create an instance of ObsClient. var obsClient = new ObsClient({ access_key_id: '*** Provide your Admission Fundamental ***', secret_access_key: '*** Provide your Secret Primal ***', server : 'https://your-endpoint' }); obsClient.initiateMultipartUpload({ Saucepan : 'bucketname', Cardinal : 'objectname', ContentType : 'text/evidently', Metadata : {'property' : 'property-value'} }, (err, result) => { if(err){ console.mistake('Error-->' + err); }else{ console.log('Condition-->' + consequence.CommonMsg.Condition); if(issue.CommonMsg.Status < 300 && result.InterfaceResult){ panel.log('UploadId-->' + effect.InterfaceResult.UploadId); } } }); 
- When initializing a multipart upload, you tin can apply the ContentType and Metadata parameters to respectively set the MIME type and customize the metadata of an object, too the object name and owning bucket.
- After the API for initializing a multipart upload is chosen, the upload ID volition exist returned. This ID will be used in follow-up operations.
Uploading a Part
After initializing a multipart upload, you can specify the object name and upload ID to upload a part. Each upload part has a part number (ranging from 1 to 10000). For parts with the same upload ID, their function numbers are unique and identify their comparative locations in the object. If you use the aforementioned role number to upload ii parts, the latter one being uploaded volition overwrite the former. Except for the part last uploaded whose size ranges from 0 to five GB, sizes of the other parts range from 100 KB to 5 GB. Parts are uploaded in random order and can be uploaded through different processes or machines. OBS will combine them into the object based on their function numbers.
Yous can call ObsClient.uploadPart to upload a part.
// Import the OBS library. // Utilize npm to install the customer. var ObsClient = require('esdk-obs-nodejs'); // Use source codes to install the client. // var ObsClient = crave('./lib/obs'); // Create an instance of ObsClient. var obsClient = new ObsClient({ access_key_id: '*** Provide your Access Key ***', secret_access_key: '*** Provide your Secret Key ***', server : 'https://your-endpoint' }); obsClient.uploadPart({ Bucket:'bucketname', Key:'objectname', // Prepare the part number, which ranges from 1 to 10000. PartNumber:i, // Set the upload ID. UploadId:'upload id from initiateMultipartUpload', // Set the large file to be uploaded. localfile is the path of the local file to be uploaded. Yous need to specify the file name. SourceFile: 'localfile', // Set the part size. PartSize: v * 1024 * 1024, // Set the start offset. Outset: 0 }, (err, upshot) => { if(err){ console.log('Error-->' + err); }else{ console.log('Status-->' + upshot.CommonMsg.Condition); if(consequence.CommonMsg.Condition < 300 && result.InterfaceResult){ console.log('ETag-->' + outcome.InterfaceResult.ETag); } } });
- Use the PartNumber parameter to specify the part number, the UploadId parameter to specify the globally unique ID, the SourceFile parameter to specify the to-be-uploaded file, the PartSize parameter to set the part size, and the Offset parameter to set the kickoff offset of the file.
- Except the office concluding uploaded, other parts must be larger than 100 KB. Office sizes volition not exist verified during upload because which one is terminal uploaded is not identified until parts are combined.
- OBS will return ETags (MD5 values) of the received parts to users.
- You lot can utilise the ContentMD5 parameter to set the MD5 value for the object to be uploaded, which can be provided to the OBS server for information integrity verification.
- Role numbers range from one to 10000. If a role number exceeds this range, OBS will return a 400 Bad Asking error.
- The minimum part size supported by an OBS iii.0 bucket is 100 KB, and the minimum function size supported by an OBS 2.0 bucket is 5 MB. Y'all are advised to perform multipart upload to OBS 3.0 buckets.
Combining Parts
After all parts are uploaded, call the API for combining parts to generate the object. Before this performance, valid part numbers and ETags of all parts must be sent to OBS. Later on receiving this information, OBS verifies the validity of each part one by one. Later all parts pass the verification, OBS combines these parts to course the final object.
You can call ObsClient.completeMultipartUpload to combine parts.
// Import the OBS library. // Employ npm to install the client. var ObsClient = require('esdk-obs-nodejs'); // Use source codes to install the client. // var ObsClient = require('./lib/obs'); // Create an instance of ObsClient. var obsClient = new ObsClient({ access_key_id: '*** Provide your Admission Primal ***', secret_access_key: '*** Provide your Secret Key ***', server : 'https://your-endpoint' }); obsClient.completeMultipartUpload({ Saucepan:'bucketname', Key:'objectname', // Set up the upload ID. UploadId:'upload id from initiateMultipartUpload', Parts: [{'PartNumber':1,'ETag':'etag value from uploadPart'}] }, (err, result) => { if(err){ console.log('Mistake-->' + err); }else{ console.log('Condition-->' + result.CommonMsg.Status); } });
- Utilise the UploadId parameter to specify the upload ID for the multipart upload and the Parts parameter to specify the list of part numbers and ETags. Content in the list is displayed in the ascending order by part number.
- Part numbers can exist inconsecutive.
Concurrently Uploading Parts
Multipart upload is mainly used for large file upload or when the network condition is poor. The following sample code shows how to concurrently upload large files in multipart style:
// Import the OBS library. // Use npm to install the client. const ObsClient = require('esdk-obs-nodejs'); // Use source codes to install the customer. // var ObsClient = require('./lib/obs'); const fs = crave('fs'); // Create an case of ObsClient. const obsClient = new ObsClient({ access_key_id: '*** Provide your Admission Key ***', secret_access_key: '*** Provide your Hugger-mugger Key ***', server : 'https://your-endpoint' }); const Bucket = 'bucketname' const Key = 'objectname' // Specify the part size. const DEFAULT_PART_SIZE = 9 * 1024 * 1024; // Set the number of concurrent parts to 20. const CONCURRENCY = 20 // Gear up multipart upload parameters. const preparePartParams = (Bucket, Key, UploadId) => (sampleFile, partSize = DEFAULT_PART_SIZE) => { try { const fileSize = fs.lstatSync(sampleFile).size; const partCount = fileSize % partSize === 0 ? Math.floor(fileSize / partSize) : Math.flooring(fileSize / partSize) + 1; const uploadPartParams = []; // Specify the concurrent upload. for (let i = 0; i < partCount; i++) { // Start position of parts in the file permit Get-go = i * partSize; // Part size permit currPartSize = (i + 1 === partCount) ? fileSize - Offset : partSize; // Part number permit PartNumber = i + 1; uploadPartParams.push({ Bucket, Key, PartNumber, UploadId, Commencement, SourceFile: sampleFile, PartSize: currPartSize, }); } return ({ uploadPartParams, fileSize }); } catch (error) { panel.log(error) } } /** * uploadSuccessSize: Size of the parts that have been uploaded * uploadSuccessCount: Number of the parts that take been uploaded * concurrency: Current concurrency */ let uploadSuccessSize = 0; let uploadSuccessCount = 0; permit concurrency = 0 const parts = []; const uploadPart = (uploadPartParam, otherUploadPartInfo) => { const partCount = otherUploadPartInfo.partCount; const fileSize = otherUploadPartInfo.fileSize; concurrency++; return obsClient .uploadPart(uploadPartParam) .then(upshot => { const { PartNumber, PartSize } = uploadPartParam; if (result.CommonMsg.Status < 300) { uploadSuccessCount++; uploadSuccessSize += PartSize; console.log(`the current concurrent count is ${concurrency} | uploaded segment: ${uploadSuccessCount}/${partCount}. the progress is ${((uploadSuccessSize / fileSize) * 100).toFixed(2)}% | the partNumber ${PartNumber} upload successed.`) parts.push button({ PartNumber, ETag: event.InterfaceResult.ETag }); } else { console.log(result.CommonMsg.Code, parts) } concurrency--; }).catch(function (err) { console.log(err); throw err; }) } const uploadFile = (sourceFile) => { obsClient.initiateMultipartUpload({ Bucket, Key }).then(res => { const Status = res.CommonMsg.Status; const UploadId = res.InterfaceResult.UploadId; if (typeof Status === 'number' && Status > 300) { console.log(`initiateMultipartUpload failed! Status:${Status}`); return; } const partParams = preparePartParams(Bucket, Cardinal, UploadId)(sourceFile) const uploadPartParams = partParams.uploadPartParams; const fileSize = partParams.fileSize; const partCount = uploadPartParams.length; const otherUploadPartInfo = { fileSize, partCount } // Call the parallel upload function. parallelFunc(uploadPartParams, (param) => uploadPart(param, otherUploadPartInfo), CONCURRENCY) .then(() => { obsClient.completeMultipartUpload({ Saucepan, Key, UploadId, Parts: parts.sort((a, b) => a.PartNumber - b.PartNumber) }, (err, result) => { if (err) { console.log('Error-->' + err); } else { console.log('Status-->' + effect.CommonMsg.Condition); } }); }) }).catch(office (err) { console.log(err) }) } /** * Implement the parallel execution. * @param {Array} params Parameter array of the callback function * @param {Hope} promiseFn Callback function * @param {number} limit Number of parallel parts */ const parallelFunc = (params, promiseFn, limit) => { render new Promise((resolve) => { let concurrency = 0; let finished = 0; const count = params.length; const run = (param) => { concurrency++; promiseFn(param) .and so(() => { concurrency--; drainQueue(); finished++ if (finished === count) { resolve() } }); } const drainQueue = () => { while (params.length > 0 && concurrency < limit) { var param = params.shift(); run(param); } } drainQueue(); }) } uploadFile('localfile');
When uploading a large file in multipart style, you need to utilize the Kickoff and PartSize parameters to fix the start and stop positions of each part in the file.
If at that place are as well many concurrent parts, a timeout error may occur due to network instability or other causes. In such case, you lot need to limit the number of concurrent parts.
Aborting a Multipart Upload
After a multipart upload is aborted, you lot cannot employ its upload ID to perform whatever operation and the uploaded parts will be deleted past OBS.
When an object is existence uploaded in multi-part mode or an object fails to be uploaded, parts generated in the saucepan. These parts occupy your storage space. You tin cancel the multi-function uploading task to delete unnecessary parts, thereby saving the storage space.
Y'all can call ObsClient.abortMultipartUpload to abort a multipart upload.
// Import the OBS library. // Apply npm to install the client. var ObsClient = require('esdk-obs-nodejs'); // Utilise source codes to install the client. // var ObsClient = require('./lib/obs'); // Create an instance of ObsClient. var obsClient = new ObsClient({ access_key_id: '*** Provide your Access Primal ***', secret_access_key: '*** Provide your Undercover Primal ***', server : 'https://your-endpoint' }); obsClient.abortMultipartUpload({ Bucket:'bucketname', Fundamental:'objectname', // Set the upload ID. UploadId:'upload id from initiateMultipartUpload', }, (err, outcome) => { if(err){ console.log('Error-->' + err); }else{ console.log('Status-->' + effect.CommonMsg.Condition); } }); Listing Uploaded Parts
You lot can call ObsClient.listParts to list successfully uploaded parts of a multipart upload.
The following table describes the parameters involved in this API.
| Parameter | Clarification |
|---|---|
| UploadId | Upload ID, which globally identifies a multipart upload. The value is in the returned issue of ObsClient.initiateMultipartUpload. |
| MaxParts | Maximum number of parts that tin can be listed per page. |
| PartNumberMarker | Part number after which listing uploaded parts begins. Simply parts whose function numbers are larger than this value will exist listed. |
- Listing parts in simple mode
// Import the OBS library. // Use npm to install the customer. var ObsClient = require('esdk-obs-nodejs'); // Use source codes to install the client. // var ObsClient = crave('./lib/obs'); // Create an example of ObsClient. var obsClient = new ObsClient({ access_key_id: '*** Provide your Access Key ***', secret_access_key: '*** Provide your Secret Fundamental ***', server : 'https://your-endpoint' }); // List uploaded parts. uploadId is obtained from initiateMultipartUpload. obsClient.listParts({ Bucket : 'bucketname', Primal: 'objectname', UploadId : 'upload id from initiateMultipartUpload' }, (err, result) => { if(err){ console.log('Mistake-->' + err); }else{ console.log('Status-->' + result.CommonMsg.Condition); if(result.CommonMsg.Status < 300 && event.InterfaceResult){ for(let i=0;i<result.InterfaceResult.Parts.length;i++){ panel.log('Part['+ i +']:'); // Part number, specified during the upload panel.log('PartNumber-->' + event.InterfaceResult.Parts[i]['PartNumber']); // Time when the part was last uploaded panel.log('LastModified-->' + result.InterfaceResult.Parts[i]['LastModified']); // Function ETag console.log('ETag-->' + result.InterfaceResult.Parts[i]['ETag']); // Part size console.log('Size-->' + result.InterfaceResult.Parts[i]['Size']); } } } });
- A maximum of 1000 parts tin can exist returned each time. If an upload of the specific upload ID contains more than one thousand parts and InterfaceResult.IsTruncated is true in the returned outcome, non all parts are returned. In such cases, you can use InterfaceResult.NextPartNumberMarker to obtain the start position for next list.
- If you want to obtain all parts involved in a specific upload ID, you can use the paging mode for list.
- Listing all parts
If the number of parts of a multipart upload is larger than 1000, you can use the post-obit sample lawmaking to list all parts.
// Import the OBS library. // Use npm to install the client. var ObsClient = require('esdk-obs-nodejs'); // Employ source codes to install the client. // var ObsClient = require('./lib/obs'); // Create an example of ObsClient. var obsClient = new ObsClient({ access_key_id: '*** Provide your Access Key ***', secret_access_key: '*** Provide your Hush-hush Key ***', server : 'https://your-endpoint' }); var listAll = (partNumberMarker) => { // List uploaded parts. uploadId is obtained from initiateMultipartUpload. obsClient.listParts({ Saucepan : 'bucketname', Key: 'objectname', UploadId : 'upload id from initiateMultipartUpload', PartNumberMarker : partNumberMarker }, (err, result) => { if(err){ panel.log('Error-->' + err); }else{ console.log('Status-->' + event.CommonMsg.Status); if(issue.CommonMsg.Status < 300 && result.InterfaceResult){ for(let i=0;i<result.InterfaceResult.Parts.length;i++){ console.log('Part['+ i +']:'); // Part number, specified during the upload console.log('PartNumber-->' + upshot.InterfaceResult.Parts[i]['PartNumber']); // Time when the part was last uploaded console.log('LastModified-->' + effect.InterfaceResult.Parts[i]['LastModified']); // Part ETag panel.log('ETag-->' + event.InterfaceResult.Parts[i]['ETag']); // Part size console.log('Size-->' + result.InterfaceResult.Parts[i]['Size']); } if(result.InterfaceResult.IsTruncated === 'true'){ listAll(effect.InterfaceResult.NextPartNumberMarker); } } } }); }; listAll(); Listing Multipart Uploads
You can call ObsClient.listMultipartUploads to listing multipart uploads. The following table describes parameters involved in ObsClient.listMultipartUploads.
| Parameter | Description |
|---|---|
| Prefix | Prefix that the object names in the multipart uploads to be listed must contain |
| Delimiter | Character used to group object names involved in multipart uploads. If the object name contains the Delimiter parameter, the graphic symbol string from the start character to the first Delimiter parameter in the object proper name is grouped under a single upshot element, CommonPrefix. (If a prefix is specified in the asking, the prefix must exist removed from the object name.) |
| MaxUploads | Maximum number of returned multipart uploads. The value ranges from 1 to chiliad. If the value is not in this range, 1000 uploads are returned by default. |
| KeyMarker | Object name to start with when listing multipart uploads |
| UploadIdMarker | Upload ID after which the multipart upload listing begins. It is effective only when used with KeyMarker so that multipart uploads after UploadIdMarker of KeyMarker will be listed. |
- Listing multipart uploads in simple fashion
// Import the OBS library. // Apply npm to install the client. var ObsClient = require('esdk-obs-nodejs'); // Use source codes to install the client. // var ObsClient = require('./lib/obs'); // Create an instance of ObsClient. var obsClient = new ObsClient({ access_key_id: '*** Provide your Admission Key ***', secret_access_key: '*** Provide your Secret Key ***', server : 'https://your-endpoint' }); obsClient.listMultipartUploads({ Bucket : 'bucketname' }, (err, effect) => { if(err){ console.log('Error-->' + err); }else{ console.log('Condition-->' + result.CommonMsg.Status); if(result.CommonMsg.Condition < 300 && result.InterfaceResult){ for(permit i=0;i<result.InterfaceResult.Uploads.length;i++){ console.log('Uploads[' + i + ']'); console.log('UploadId-->' + result.InterfaceResult.Uploads[i]['UploadId']); console.log('Cardinal-->' + result.InterfaceResult.Uploads[i]['Key']); console.log('Initiated-->' + result.InterfaceResult.Uploads[i]['Initiated']); } } } });
- Information about a maximum of k multipart uploads can exist returned each fourth dimension. If a bucket contains more than 1000 multipart uploads and InterfaceResult.IsTruncated is true in the returned effect, non all uploads will be returned. In such cases, you tin can employ InterfaceResult.NextKeyMarker and InterfaceResult.NextUploadIdMarker to obtain the start position for next listing.
- If y'all want to obtain all multipart uploads in a saucepan, yous can employ the paging list.
- Listing all multipart uploads
// Import the OBS library. // Use npm to install the client. var ObsClient = require('esdk-obs-nodejs'); // Use source codes to install the client. // var ObsClient = crave('./lib/obs'); // Create an instance of ObsClient. var obsClient = new ObsClient({ access_key_id: '*** Provide your Access Central ***', secret_access_key: '*** Provide your Secret Key ***', server : 'https://your-endpoint' }); var listAll = (keyMarker, uploadIdMarker) => { obsClient.listMultipartUploads({ Bucket : 'bucketname', KeyMarker : keyMarker, UploadIdMarker : uploadIdMarker }, (err, result) => { if(err){ console.log('Error-->' + err); }else{ console.log('Status-->' + consequence.CommonMsg.Status); if(effect.CommonMsg.Status < 300 && result.InterfaceResult){ for(let i=0;i<result.InterfaceResult.Uploads.length;i++){ console.log('Uploads[' + i + ']'); console.log('UploadId-->' + result.InterfaceResult.Uploads[i]['UploadId']); console.log('Key-->' + issue.InterfaceResult.Uploads[i]['Cardinal']); console.log('Initiated-->' + result.InterfaceResult.Uploads[i]['Initiated']); } if(result.InterfaceResult.IsTruncated === 'true'){ listAll(result.InterfaceResult.NextKeyMarker, result.InterfaceResult.NextUploadIdMarker); } } } }); } listAll(); Did you lot find this folio helpful?
Submit successfully!
Thanks for your feedback. Your feedback helps brand our documentation improve.
Failed to submit the feedback. Please try again subsequently.
Source: https://support.huaweicloud.com/intl/en-us/sdk-nodejs-devg-obs/obs_29_0407.html
0 Response to "Difference Between Streamed Upload and Multipart Upload"
Post a Comment