Too many files open error using file upload service ...

#1

Hi, it would appear (!) that the file upload service is leaving something “open” … if I upload a bunch of files, typically in bunches of around 15-20, then after a relatively short period of file I see this on the server;

2016-07-08T16:23:52+0100 [Router 16137] Unhandled Error
Traceback (most recent call last):
File “/usr/local/lib/pypy2.7/dist-packages/crossbar/worker/process.py”, line 273, in

File “/usr/local/lib/pypy2.7/dist-packages/crossbar/worker/process.py”, line 261, in run

File “/usr/local/lib/pypy2.7/dist-packages/twisted/internet/base.py”, line 1194, in run

File “/usr/local/lib/pypy2.7/dist-packages/twisted/internet/base.py”, line 1203, in mainLoop

— —
File “/usr/local/lib/pypy2.7/dist-packages/twisted/internet/base.py”, line 825, in runUntilCurrent

File “/usr/local/lib/pypy2.7/dist-packages/txaio/_common.py”, line 99, in _notify_bucket

File “/usr/local/lib/pypy2.7/dist-packages/txaio/_common.py”, line 95, in notify_one_chunk

exceptions.RuntimeError: Error(s) processing call_later bucket:
[Errno 24] Too many open files

``

This is “game over”, and I need to restart Crossbar to continue.

Crossbar version is 0.13.2.

Am I doing something wrong, or is this a bug?

If I check /proc/<pid/fd for the crossbar process, it’s showing 1024 active file handles …

0 Likes

#2

Hi Gareth,

···

On Friday, July 8, 2016 at 10:36:58 AM UTC-5, Gareth Bult wrote:

Hi, it would appear (!) that the file upload service is leaving something “open” … if I upload a bunch of files, typically in bunches of around 15-20, then after a relatively short period of file I see this on the server;

2016-07-08T16:23:52+0100 [Router 16137] Unhandled Error
Traceback (most recent call last):
File “/usr/local/lib/pypy2.7/dist-packages/crossbar/worker/process.py”, line 273, in

File “/usr/local/lib/pypy2.7/dist-packages/crossbar/worker/process.py”, line 261, in run

File “/usr/local/lib/pypy2.7/dist-packages/twisted/internet/base.py”, line 1194, in run

File “/usr/local/lib/pypy2.7/dist-packages/twisted/internet/base.py”, line 1203, in mainLoop

— —
File “/usr/local/lib/pypy2.7/dist-packages/twisted/internet/base.py”, line 825, in runUntilCurrent

File “/usr/local/lib/pypy2.7/dist-packages/txaio/_common.py”, line 99, in _notify_bucket

File “/usr/local/lib/pypy2.7/dist-packages/txaio/_common.py”, line 95, in notify_one_chunk

exceptions.RuntimeError: Error(s) processing call_later bucket:
[Errno 24] Too many open files

``

This is “game over”, and I need to restart Crossbar to continue.

Crossbar version is 0.13.2.

Am I doing something wrong, or is this a bug?

If I check /proc/<pid/fd for the crossbar process, it’s showing 1024 active file handles …

Looks like you are running out of available file descriptors for the process in question. The default per-process maximum is typically 1024, which is exactly what is being reported here. This may not be a fd leak, you might just be causing the system to attempt to allocate fds faster than they are being released. Try upping the number of per-process fds, until your test succeeds, and see what the max number of fds allocated turns out to be. Then see if the fds are released at some point after your test completes.

This kind of thing frequently occurs when someone attempts to load-test a server without tuning the server for the expected load, and it usually isn’t a leak.

Hope this helps,

L. Daniel Burr

0 Likes

#3

Hi Daniel,

I’m afraid that’s not it, those file descriptors are never (so far as I can see) released and the entire (crossbar) server locks up until it’s restarted.

If I upload 20 files. Wait 30 seconds, upload another 20 files, wait 30 seconds, I should be able to repeat the process indefinitely without running out of sockets. There is no processing time, Crossbar should be releasing sockets as soon as the upload is complete (i.e. immediately) , otherwise this is a potential DDOS vector.

I guess what was worrying me was whether I should be explicitly doing something on the server to mark uploads complete, but I can’t see anything in the docs. I’ve had a problem for a long time where occasionally Crossbar locks up for no apparent reason and I have to restart, I’m pretty sure this is the issue I’ve been seeing, occasional uploads mount up until it eventually runs out of sockets and hangs.

As another aside, I can’t see to get the server-side “on_progress” to work.
If I manually publish to the topic using the “on complete” handler it works fine, but if add the topic as an “on_progress” either to the INPUT tag on the form or as a parameter to “query”, I get nothing.

Regards,

Gareth.

0 Likes

#4

Hi Gareth,

- FDs are for both files and sockets
- to tune up max FDs: http://crossbar.io/docs/Network-Tuning/
- I am not aware of CB leaking FDs in general (our public demo instance has been running for like a full month uninterrupted, I have tested with 100k's of connections), but:
- specifically, the file upload service is unfinished and buggy. don't use it. I have no time / motivation to fix it - 3rd party contrib.

Cheers,
/Tobias

···

Am 08.07.2016 um 19:26 schrieb Gareth Bult:

Hi Daniel,

I'm afraid that's not it, those file descriptors are *never* (so far as I
can see) released and the entire (crossbar) server locks up until it's
restarted.

If I upload 20 files. Wait 30 seconds, upload another 20 files, wait 30
seconds, I should be able to repeat the process indefinitely without
running out of sockets. There is no processing time, Crossbar should be
releasing sockets as soon as the upload is complete (i.e. immediately) ,
otherwise this is a potential DDOS vector.

I guess what was worrying me was whether I should be explicitly doing
something on the server to mark uploads complete, but I can't see anything
in the docs. I've had a problem for a long time where occasionally Crossbar
locks up for no apparent reason and I have to restart, I'm pretty sure this
is the issue I've been seeing, occasional uploads mount up until it
eventually runs out of sockets and hangs.

As another aside, I can't see to get the server-side "on_progress" to work.
If I manually publish to the topic using the "on complete" handler it works
fine, but if add the topic as an "on_progress" either to the INPUT tag on
the form or as a parameter to "query", I get nothing.

Regards,
Gareth.

0 Likes

#5

Hi Tobias,

I’m not suggesting for a second that Crossbar itself if leaking FD’s, this is purely a file-upload-service issue as far as I can see.

Ok, totally appreciate what you’re saying, although I wasn’t aware the upload service was a 3rd party contrib … will see if I can come up with an alternative solution without re-inventing too many wheels … :slight_smile:

Thanks,

Gareth.

0 Likes

#6

Hi Gareth,

···

On Friday, July 8, 2016 at 1:37:34 PM UTC-5, Gareth Bult wrote:

Hi Tobias,

I’m not suggesting for a second that Crossbar itself if leaking FD’s, this is purely a file-upload-service issue as far as I can see.

Ok, totally appreciate what you’re saying, although I wasn’t aware the upload service was a 3rd party contrib … will see if I can come up with an alternative solution without re-inventing too many wheels … :slight_smile:

Thanks,

Gareth.

I’d suggest implementing your own file upload resource, and specifically overriding the request factory for the Site instance that owns the resource, so that you can explicitly control the way that fds pertaining to the uploaded files themselves are handled.

Also, as a minimal test, I’d replace the file upload resource with an instance of twisted.web.resource.Resource that does no processing of the POST payload at all, and see if you still have the fd problem.

Hope this helps,

L. Daniel Burr

0 Likes

#7

Hi,

I think I have a good solution … when I’m happy it works all the time I’ll post it … :slight_smile:

Regards,

Gareth.

0 Likes

#8

Ok, this seems to work … does not require any Crossbar configuration … and can be tweaked to provide progress information … how does this look in principle;

Stage # 1, Input field / button / object of your choice.

``

“OnChange” event builds up a list of files in _this.fileList;

_this.fileList = [];
var fileList = document.getElementById(“fupload”);
fileList.addEventListener(“change”, function (e) {
for (var i = 0; i < this.files.length; i++) {
_this.fileList.push(this.files[i]);
}
}, false);

``

Then when “Upload” / “Submit” / whatever is clicked;

file_upload: function(node, files) {
var _this = this;
for(var i=0; i<files.length; i++) {
var file = files[i];
var reader = new FileReader();
reader.fileName = file.name;
var blob = file.slice();
reader.onloadend = function(evt) {
if (evt.target.readyState == FileReader.DONE) {
var success = function(data) {
console.log(“OK”);
}
name = evt.target.fileName;
text = evt.target.result;
_this.call(‘ion.app.send_file’,{name:name, text:text}, success);
}
};
reader.readAsBinaryString(blob);
};

``

So all we need server side is;

@inlineCallbacks
def ion_app_send_file(self, argv, session=None, user=None):
“”“Create an asset”""
file = argv.get(‘name’, None)
text = argv.get(‘text’, None)

if not file or not root or not text:
    returnValue(self.fail('Invalid parameters'))
···
#
#
#        
returnValue(self.ok())        

``

0 Likes

#9

Gah! I hate it when for some reason Google decides to post an article before you’re finished typing, then refuses to let you edit … anyway, there are local artifacts in this code, but essentially it does work and seems to be able to handle batches of files Ok. If I write some chunking code, I should be able to add a “progress” function … doesn’t have the theoretical feature set of “resumable”, but over reliable connections it does make for a “simpler” solution (?) Getting away from HTTP / file upload configurations on the Crossbar config is also a plus … :slight_smile:

0 Likes

#10

Ok, for anyone interested, this seems to be working, although it could probably benefit from some error trapping and checking;

Call this after selecting the files to upload, so typically tag it to your “submit” or “upload” button …

Parameters;

  • “node” - application specific reference, typically “where to store the file”
  • “files” - array of JS File objects returned by ‘change’ event on local file browser object (<INPUT…>)
  • “progress” - function to call with progress updates, parameters are “file name” and “overall percent complete”
    (ideal for “progress bar” display, will do a 0-100 progress bar and change the file name displayed as it goes)

Server side you need two RPC topics;

  • “start”, parameters are (node,name,size)
  • “chunk”, parameters are (data, bool(finished))

Subject to “I hate it because it doesn’t work with IE6”, in principle this seems like quite a nice alternative to work with … and it’s pure WAMP … :slight_smile:

    file_upload: function(node, files, progress) {
        var _this = this, _size = 0, _index = 0, _transfer = 0, _chunk_size = 4096;
        for(var i=0; i<files.length; i++) _size += files[i].size;          
        var transfer_init = function() {
            var reader = new FileReader();                  
            var blob = files[_index].slice();
            reader.onloadend = function(evt) {
                if (evt.target.readyState == FileReader.DONE) {                                                
                    var send = function(data) {
                        _chunk = _text.slice(0,_chunk_size);
                        _text = _text.slice(_chunk_size+1);                            
                        var transfer_next = function() {
                            _transfer += _chunk.length;
                            var _percent = parseInt((100*_transfer) / _size);
                            if(progress) progress(files[_index].name,_percent);
                            if(!done) send();
                            else {                                    
                                if(_index<files.length-1) {
                                    _index++;
                                    transfer_init();
                                } else
                                    if(progress) progress(files[_index].name,100);
                            }    
                        };
                        var done = _text.length == 0;
                        _this.call('ion.app.send_file_chunk',{chunk: _chunk, done: done}, transfer_next);                        
                    };
                    var _text = evt.target.result;
                    _this.call('ion.app.send_file_start',{
                        node: node,
                        name: files[_index].name,
                        size: files[_index].size
                    }, send);                        
                }
            };                
            reader.readAsBinaryString(blob);                            
        }
        if(progress) progress(files[_index].name,0);
        transfer_init();
    },

``

0 Likes