如何使用HTTP API先分块,再分片上传到七牛云

Open matadorr opened this issue 7 years ago • 1 comments

附加信息：

false 上传空间: adrfiles 上传方式: 分片上传

liguo2018-09-04 10:50 分片上传文件,按照API REFERENCE中的【创建块】【上传片】【创建文件】三个接口，

素材:10M大小的文件即1010241024 byte

1,为了清晰理解,使用物理切分,切分成100Kb大小的片

2,调用https://up-z2.qiniup.com/mkblk/4194304创建块,4M大小的块(官方要求),并且附带第一片100k,

七牛返回:{

"ctx": "ujpuQjPLdjS1J6IK8OKHKAGWpcRsNRFRFe3f0lCUlpBmE8eSjVE_f08BvwNCYi1hYmNmLTQ0YzEtYTEzOC1jZWYxNTdkYTVlMzAtLQ0KkwAAHqYAABKWAAAB8gAAAd0AAAFbAAAGWgAACg8AACEAAADhkAEAAAAAAAAAQADhkAEAZWdJQUFIWl9qVnZabzBJQQ==", "checksum": "2VO0sjb_-9F6i1bYTl1MS62IwVA=", "crc32": 480655653, "offset": 102625, "host": "https://up-z2.qiniup.com", "expired_at": 1536632669 } 3,上传第二片,

https://up-z2.qiniup.com/bput/ujpuQjPLdjS1J6IK8OKHKAGWpcRsNRFRFe3f0lCUlpBmE8eSjVE_f08BvwNCYi1hYmNmLTQ0YzEtYTEzOC1jZWYxNTdkYTVlMzAtLQ0KkwAAHqYAABKWAAAB8gAAAd0AAAFbAAAGWgAACg8AACEAAADhkAEAAAAAAAAAQADhkAEAZWdJQUFIWl9qVnZabzBJQQ==/102625

以此类推,每次上传片都使用上一次返回的ctx和nextchunkOffset构造url

问题,当前块即将传完的时候,报错

{ "error": "too many data to read, block capacity is 4194304 bytes, 4105031 byte(s) used" } 附代码:

//块级控制信息 String ctx = ""; List<String> ctxs = new ArrayList<>(10); Long nextChunckOffset = 0L; Long uploadedSize = 0L; Long totalSize = 4194304L2+2097153L; Integer index = 0; File file = null; Long chunkSize=1001024L; //附加参数 StringMap params = new StringMap(); final String returnBody = "{"key":"$(key)","hash":"$(etag)","fsize":"$(fsize)"" + ","fname":"$(fname)","mimeType":"$(mimeType)"}"; //七牛云配置 Configuration configuration = new Configuration(Zone.zone2()); Client client = new Client(configuration); Auth auth = Auth.create(qiniuAccessKey, qiniuSecretKey); Map<String, Object> headersMap = new HashMap<>(10); String token = auth.uploadToken(bucket, null, 3600, new StringMap().put("returnBody", returnBody)); headersMap.put("Authorization", "UpToken " + token);

while (uploadedSize < totalSize) { index++; if (uploadedSize % (4 * 1 << 20) == 0) { uploadUrl += "/mkblk/4194304"; file = new File("E:\sample_part_" + index + ".mp4"); JSONObject json = post(uploadUrl, file, headersMap, Collections.EMPTY_MAP); uploadUrl = json.getString("host"); nextChunckOffset = json.getLong("offset"); ctx = json.getString("ctx"); ctxs.add(ctx); } else { uploadUrl += "/bput/" + ctx + "/" + nextChunckOffset; file = new File("E:\sample_part_" + index + ".mp4"); JSONObject json = post(uploadUrl, file, headersMap, Collections.EMPTY_MAP); uploadUrl = json.getString("host"); nextChunckOffset = json.getLong("offset"); ctx = json.getString("ctx"); } uploadedSize +=chunkSize ;

} uploadUrl += "/mkfile/"+totalSize+"/test/"; String ctxsStr = Joiner.on(",").skipNulls().join(ctxs); JSONObject uploadResult = post(uploadUrl, headersMap, ctxsStr);

七牛云工程师2018-09-04 10:50 您的问题我们已收到，会尽快为您查看。请您耐心等待，谢谢！

liguo2018-09-04 10:57 疑问:

1,为什么我102400的分片文件上传后占用了102625?

2,我第二片的文件是不是应该从102625开始?

3,如果这样的话,4M的块怎么完整拼齐,因为4M与102625无法整除

liguo2018-09-04 11:05 既然是每片大小是102625,我总共4M的块,能放多少片就放多少片,如下,但是合并文件的时候报错{ "error": "unexpected file size" } String uploadUrl = "https://up-z2.qiniup.com"; //块级控制信息 String ctx = ""; List<String> ctxs = new ArrayList<>(10); Long nextChunckOffset = 0L; Long uploadedSize = 0L; Long totalSize = 1010241024L; Integer index = 0; File file = null; Long chunkSize=100*1024L; Long remainBlockSize=0L; //附加参数 StringMap params = new StringMap(); final String returnBody = "{"key":"$(key)","hash":"$(etag)","fsize":"$(fsize)"" + ","fname":"$(fname)","mimeType":"$(mimeType)"}"; //七牛云配置 Configuration configuration = new Configuration(Zone.zone2()); Client client = new Client(configuration); Auth auth = Auth.create(qiniuAccessKey, qiniuSecretKey); Map<String, Object> headersMap = new HashMap<>(10); String token = auth.uploadToken(bucket, null, 3600, new StringMap().put("returnBody", returnBody)); headersMap.put("Authorization", "UpToken " + token);

while (uploadedSize < totalSize) { index++; if (remainBlockSize<chunkSize) { uploadUrl += "/mkblk/4194304"; file = new File("E:\workspace\git-workspace\ocean-common\group-file\video\sample_part_" + index + ".mp4"); JSONObject json = post(uploadUrl, file, headersMap, Collections.EMPTY_MAP); uploadUrl = json.getString("host"); nextChunckOffset = json.getLong("offset"); ctx = json.getString("ctx"); ctxs.add(ctx); remainBlockSize=410241024L; } else { uploadUrl += "/bput/" + ctx + "/" + nextChunckOffset; file = new File("E:\workspace\git-workspace\ocean-common\group-file\video\sample_part_" + index + ".mp4"); JSONObject json = post(uploadUrl, file, headersMap, Collections.EMPTY_MAP); uploadUrl = json.getString("host"); nextChunckOffset = json.getLong("offset"); ctx = json.getString("ctx"); } remainBlockSize-=chunkSize; uploadedSize +=chunkSize ; } uploadUrl += "/mkfile/"+totalSize+"/test/"; String ctxsStr = Joiner.on(",").skipNulls().join(ctxs); JSONObject uploadResult = post(uploadUrl, headersMap, ctxsStr); System.out.println(uploadResult);

七牛云工程师2018-09-04 11:32 您好：

分片上传参考：

https://developer.qiniu.com/kodo/manual/1650/chunked-upload

SDK源码：

https://github.com/qiniu

liguo2018-09-04 11:36 这些我看过了,sdk只有分块,没有分片

liguo2018-09-04 11:38 可不可以帮我发一个调用上面mkblk bput mkfile成功的demo

七牛云工程师2018-09-04 11:56 您好：

我们的SDK源码都在发您的链接里面了，您自己多看一下呢？

liguo2018-09-04 12:37 java-sdk并没有分片的逻辑,麻烦帮我找个有分片逻辑的sdk

liguo2018-09-04 12:55 分片的http api怎么传都报错

七牛云工程师2018-09-04 13:44 您好：

您有这需求的话，可以提交一下issue:https://github.com/qiniu/java-sdk/issues

Sep 04 '18 05:09 matadorr

不建议您自行实现分块上传，七牛的 java sdk 中已经自行实现了分块上传，https://github.com/qiniu/java-sdk/blob/master/src/main/java/com/qiniu/storage/ResumeUploader.java，如果您的大于 Configuration 的 putThreshold 就会自动分块上传

假设您要上传 11MB 的文件，分片上传实现的原理是

除了文件的最后一个分块之外，前面每个块都需要是 4MB
SDK 中块内不会进一步分片
如果您需要在块内进行分片的话（假设分片的大小是 2MB），那么效果如下
- 第一个请求 mkblk/4MB，且只上传 2MB，content-length 也设置为 2MB，拿到 ctx0
- 第二个请求 bput/ctx0/2MB，且也只上传 2MB，content-length 也设置为 2MB，拿到 ctx1
- 这样第一个分块的 4MB 就上传完了
- 第三个请求是新的分块，mkblk/4MB，也只上传新的 2MB，content-length 也设置为 2MB，拿到 ctx2
- 第四个请求 bput/ctx2/2MB，只上传 2MB，content-length 也设置为 2MB，拿到 ctx3
- 第五个请求因为还剩 3MB 了，也小于 4MB 的完整分块，所以就用 mkblk/3MB，但是只上传 2MB，content-length 也设置为 2MB，拿到 ctx4
- 第六个请求，将最后的 1MB 进行上传，bput/ctx4/1MB，content-length 设置为 1MB，拿到 ctx5
- 第七个请求，mkfile/11MB/... ，body 为 ctx0,ctx1,ctx2,...,ctx5

Jul 21 '20 10:07 forrest-mao