Corruption with file upload service ....

#1

Ok, so after many hours of swearing at PDF.js for not being able to display certain PDF files, it turns out that the Crossbar.io file upload service, in conjunction with Resumable.js, is under certain circumstances, corrupting uploads. After a little more investigation, the issue seems to be a function of “chunkSize”.

If the size of the file is more than 2x"chunkSize", then resulting uploaded file ends up being corrupted.

I’ve flagged this with Resumable.js, but I’m relatively confident this is a Crossbar issue.

This is my config.json;

“upload”: {
“type” : “upload”,
“realm” : “realm1”,
“role” : “anonymous”,
“directory” : “…/uploads”,
“temp_directory”: “…/uploads/tmp”,
“form_fields” : {
“file_name”: “resumableFilename”,
“mime_type”: “resumableType”,
“total_size”: “resumableTotalSize”,
“chunk_number”: “resumableChunkNumber”,
“chunk_size”: “resumableChunkSize”,
“total_chunks”: “resumableTotalChunks”,
“content”: “file”,
“on_progress”: “on_progress”,
“session”: “session”
},
“options”: {
“max_file_size”: 209715200,
“file_permissions”: “0644”,
“file_types”: [".csv", “.txt”, “.pdf”, “.img”,".png"]
}
}

``

The handler code is relatively innocuous;

var r = new Resumable({
target : ‘upload’,
chunkSize : 200*1024, // THIS MAKES ALL THE DIFFERENCE
forceChunkSize : false,
simultaneousUploads : 3,
testChunks : false,
query: {
on_progress: ‘app.candidate.channel’
session: ionman.session.id
}
});

``

In this example files of up to 400K upload fine, whereas a 450K file will be corrupted.

0 Likes

#2

HI Gareth!

First of all sorry for the late answer. I’ve filed an issue referencing your post, and hope that Marco, who added the file upload, will be able to take a look at this soon.

Regards,

Alex

···

Am Samstag, 19. Dezember 2015 02:56:21 UTC+1 schrieb Gareth Bult:

Ok, so after many hours of swearing at PDF.js for not being able to display certain PDF files, it turns out that the Crossbar.io file upload service, in conjunction with Resumable.js, is under certain circumstances, corrupting uploads. After a little more investigation, the issue seems to be a function of “chunkSize”.

If the size of the file is more than 2x"chunkSize", then resulting uploaded file ends up being corrupted.

I’ve flagged this with Resumable.js, but I’m relatively confident this is a Crossbar issue.

This is my config.json;

“upload”: {
“type” : “upload”,
“realm” : “realm1”,
“role” : “anonymous”,
“directory” : “…/uploads”,
“temp_directory”: “…/uploads/tmp”,
“form_fields” : {
“file_name”: “resumableFilename”,
“mime_type”: “resumableType”,
“total_size”: “resumableTotalSize”,
“chunk_number”: “resumableChunkNumber”,
“chunk_size”: “resumableChunkSize”,
“total_chunks”: “resumableTotalChunks”,
“content”: “file”,
“on_progress”: “on_progress”,
“session”: “session”
},
“options”: {
“max_file_size”: 209715200,
“file_permissions”: “0644”,
“file_types”: [".csv", “.txt”, “.pdf”, “.img”,".png"]
}
}

``

The handler code is relatively innocuous;

var r = new Resumable({
target : ‘upload’,
chunkSize : 200*1024, // THIS MAKES ALL THE DIFFERENCE
forceChunkSize : false,
simultaneousUploads : 3,
testChunks : false,
query: {
on_progress: ‘app.candidate.channel’
session: ionman.session.id
}
});

``

In this example files of up to 400K upload fine, whereas a 450K file will be corrupted.

0 Likes

#3

Hi Gareth. I’ll take a look into this. Also sorry for the late reply. (I was not properly subscribed to this channel here.)
Could you tell me if you are using the python 3 or python 2 version of crossbar?

Best,

Marko

0 Likes

#4

I tested with python 2 and python 3 with your config and with text and pdf files. Each Time I compared the original file with the uploaded file using

cmp original.pdf uploaded.pdf

``

I could not reproduce your error. Could you provide some more information like your system config. (OS on receiving server and sending client.)

Also the output of

crossbar version

``

Thanks,

Marko

0 Likes

#5

Gareth what version of crossbar are you using, there was a bug when file uploading was first introduced which caused corruption but I believe a fix was added recently.

0 Likes

#6

Hi Guys,

Sorry for the delay.

I’m using;

Python 2.7.10

Crossbar.io 0.11.1

Autobahn 0.10.9

I’ve tried using newer versions of Crossbar, but each time I get fatal errors and have had to back-track. This combination seems fair stable, but newer versions seem to have compatibility issues between Crossbar and Autobahn. I did swap a couple of messages with Tobias on this a little while ago, but as of a few days ago it’s still a terminal issue.

I would love to use Python3, but I lean heavily on the WSGI code and last time I looked (recently) this still hasn’t been ported. I’ve three months ploughed into the current solution and probably a year or so’s effort on the framework leading up to it … I’d be a little sorry to see wsgi “go”, especially as I’m not aware of any alternatives (?) [Essentially I need to use Flask or similar]

crossbar version

Automatically choosing optimal Twisted reactor

Running on Linux and optimal reactor (epoll) was installed.


/ |__)/ \/__/`|) /\ |__) |/ \

_,| \/././|)/~~| . |_/

Crossbar.io : 0.11.1

Autobahn : 0.10.9

UTF8 Validator : autobahn

XOR Masker : autobahn

JSON Codec : stdlib

MsgPack Codec : msgpack-python-0.4.6

Twisted : 15.5.0-EPollReactor

Python : 2.7.10/CPython

OS : Linux-4.2.0-16-generic-x86_64-with-Ubuntu-15.10-wily

Machine : x86_64

···

On Sunday, 17 January 2016 06:15:10 UTC, Greg Keys wrote:

Gareth what version of crossbar are you using, there was a bug when file uploading was first introduced which caused corruption but I believe a fix was added recently.

0 Likes