WGSI performance

#1

Hi, I’ve been using “flask” with the WSGI interface for some time, and not knowing much about WSGI just blindly followed the Crossbar examples, which has worked no problem. I noticed today the performance wasn’t ‘great’ (~250 requests/sec) and idly wondered if it was possible to use anything else, and if anything else would be quicker.

I came across something called Falcon, which seems to be an alternative to Flask, which to my surprise worked immediately with Crossbar.

https://falconframework.org/

Is there any particular reason why I should not use Falcon or why Flask is “better” ???
(or does anyone know of anything better/quicker?)

This is my demo code, whereas it doesn’t deliver the promised performance listed on the Falcon website, it does seem to give 2x the speed of the equivalent Flask;

import os
import falcon

BASE = os.getcwd() + ‘/…/static/’

class ServeFile:
def init(self, branch, mimetype):
self.mimetype = mimetype
self.branch = branch

def on_get(self, req, resp, name=''):
    if not len(name): name = 'index.html'
    path = BASE + self.branch + '/' + name

    with open(path, 'rb') as f:
        resp.body = f.read()
        resp.append_header('content-type', self.mimetype)
        resp.append_header('content-length', f.tell())

app = falcon.API()
app.add_route(’/’, ServeFile(‘html’,‘text/html’))
app.add_route(’/app/{name}’, ServeFile(‘app’,‘text/html’))
app.add_route(’/css/{name}’, ServeFile(‘css’,‘text/css’))
app.add_route(’/fonts/{name}’, ServeFile(‘fonts’,‘font/opentype’))
app.add_route(’/html/{name}’, ServeFile(‘html’,‘text/html’))
app.add_route(’/images/{name}’, ServeFile(‘images’,‘img/png’))
app.add_route(’/j2/{name}’, ServeFile(‘j2’,‘text/html’))
app.add_route(’/js/{name}’, ServeFile(‘js’,‘text/javascript’))

``

And in config.json;

“paths”: {
“/”: {
“type”: “wsgi”,
“module”: “lib_wsgi”,
“object”: “app”
}
}

``

0 Likes

#2

I would recommend you NOT serve static files with whatever Python framework you’re using.

NginX (or even Apache) are much, much, much, much better at this.

For a production deployment you should always ensure that your Python code is not doing things it shouldn’t be. This includes serving files. Serving files is a pretty mindless task and there really is no need to use a dynamic web framework to perform a task that will be doing the exact same thing every time.

For your sanity and others, serve your static files properly.

···

On Wednesday, 24 August 2016 18:55:35 UTC+2, Gareth Bult wrote:

Hi, I’ve been using “flask” with the WSGI interface for some time, and not knowing much about WSGI just blindly followed the Crossbar examples, which has worked no problem. I noticed today the performance wasn’t ‘great’ (~250 requests/sec) and idly wondered if it was possible to use anything else, and if anything else would be quicker.

I came across something called Falcon, which seems to be an alternative to Flask, which to my surprise worked immediately with Crossbar.

https://falconframework.org/

Is there any particular reason why I should not use Falcon or why Flask is “better” ???
(or does anyone know of anything better/quicker?)

This is my demo code, whereas it doesn’t deliver the promised performance listed on the Falcon website, it does seem to give 2x the speed of the equivalent Flask;

import os
import falcon

BASE = os.getcwd() + ‘/…/static/’

class ServeFile:
def init(self, branch, mimetype):
self.mimetype = mimetype
self.branch = branch

def on_get(self, req, resp, name=''):
    if not len(name): name = 'index.html'
    path = BASE + self.branch + '/' + name

    with open(path, 'rb') as f:
        resp.body = f.read()
        resp.append_header('content-type', self.mimetype)
        resp.append_header('content-length', f.tell())

app = falcon.API()
app.add_route(’/’, ServeFile(‘html’,‘text/html’))
app.add_route(’/app/{name}’, ServeFile(‘app’,‘text/html’))
app.add_route(’/css/{name}’, ServeFile(‘css’,‘text/css’))
app.add_route(’/fonts/{name}’, ServeFile(‘fonts’,‘font/opentype’))
app.add_route(’/html/{name}’, ServeFile(‘html’,‘text/html’))
app.add_route(’/images/{name}’, ServeFile(‘images’,‘img/png’))
app.add_route(’/j2/{name}’, ServeFile(‘j2’,‘text/html’))
app.add_route(’/js/{name}’, ServeFile(‘js’,‘text/javascript’))

``

And in config.json;

“paths”: {
“/”: {
“type”: “wsgi”,
“module”: “lib_wsgi”,
“object”: “app”
}
}

``

0 Likes

#3

Hi, Adam, in principle I totally agree …

I do however have a number of requirements that in practice mean it’s more convenient to serve some things via HTTP using Crossbar’s WSGI service, sometimes these documents are not ‘static’.
(you will appreciate my ‘demo’ is just a demo, it could just as easily supply dynamic data …)

In these instances, I would be interested in any experience anyone may have using alternatives to Flask, Falcon in particular …

Incidentally, putting a caching NGINX in front of Flask also works quite well from a development / convenience perspective.

0 Likes

#4

Falcon is supposed to be faster than Flask but again for serving static content with it is not a good use-case for maximum performance.

···

On Thursday, 25 August 2016 10:19:27 UTC+2, Gareth Bult wrote:

Hi, Adam, in principle I totally agree …

I do however have a number of requirements that in practice mean it’s more convenient to serve some things via HTTP using Crossbar’s WSGI service, sometimes these documents are not ‘static’.
(you will appreciate my ‘demo’ is just a demo, it could just as easily supply dynamic data …)

In these instances, I would be interested in any experience anyone may have using alternatives to Flask, Falcon in particular …

Incidentally, putting a caching NGINX in front of Flask also works quite well from a development / convenience perspective.

0 Likes

#5

Hi Adam,

As I said, I was looking for something “faster” than Flask, I don’t need “maximum performance” … I guess if I was I’d be wanting a message router written in assembler rather than Python. When serving a small number of files very infrequently, footprint, maintenance and convenience are all way ahead of performance. Given I am going to serve up content via HTTP/WSGI, if Falcon is smaller than Flask and twice as quick, subject to any downsides it seems like a better option…

0 Likes

#6

Hi Gareth,

Hi Adam,

As I said, I was looking for something "faster" than Flask, I don't need
"maximum performance" ... I guess if I was I'd be wanting a message router
written in assembler rather than Python. When serving a small number of

There is similar gain from running Crossbar.io under PyPy, namely the JIT will dynamically generate native machine code for hot spots.

files very infrequently, footprint, maintenance and convenience are all way
ahead of performance. Given I am going to serve up content via HTTP/WSGI,
if Falcon is smaller than Flask and twice as quick, subject to any
downsides it seems like a better option...

If the user Web resource run in a Crossbar.io Web transport is stateless, then it can be scaled on multi-core like so

https://github.com/crossbario/crossbarexamples/tree/master/benchmark/web/.crossbar

Regarding performance when serving static files from Crossbar.io: currently, it'll be definitely slower like 3-5x. A large factor slowing stuff done in Crossbar.io is the fact that there is no in memory caching. Hawkie is working on adding caching.

Btw: Crossbar.io supports HTTP/2 serving Web resources.

Combine this with the new Universal Transport allows you to run WAMP/WebSocket, WAMP/RawSocket and Web resouces via HTTP/1.1 or 2 all on the same port!

For public facing transport running in production, usually one will want to run the universal transport on TLS.

And for even more privacy, there is an alpha feature for enabling end-to-end encryption.

I would say this is a pretty neat feature set =)

/Tobias

···

Am 25.08.2016 um 20:33 schrieb Gareth Bult:

0 Likes

#7

Hi,

There is similar gain from running Crossbar.io under PyPy, namely the
JIT will dynamically generate native machine code for hot spots.

Ok, slightly off-topic ( :wink: ) but while were here … I did read the literature on PyPy and supporting info
on Crossbar.io, and as it was recommended, I converted my live App earlier in the year from CPython to Pypy.
(as the benchmarks do make it sound SO attractive)

In real life however it was a mistake.

Issues;

  1. There is now a substantial delay on “first run”, whereas I understand this is a “feature”, for me it’s a problem
  2. When running (after a period of time) as a user, I remain unconvinced the experience is “faster” … (I’m convinced it’s actually slower!)
  3. I’ve had to move from 512M VM’s to 1024M VM’s at double the cost, i.e. the memory footprint is much heavier for no perceived gain
    Since discovering this, I’ve been developing on CPython again, and when I get time, I will be moving my live app back to

CPython from Pypy. I’m also using Python3 as I’ve found some apparently trivial features of P3 very important to my design, something

I miss in Pypy. (for example, in terms of readability and maintenance, being able to use “return” instead of “returnValue” is actually VERY useful)

I read on the Pypy site and various forums that the performance boost does depend on what you’re doing, and that in some instances

it can actually be slower than CPython. Well, I spent half a day doing off the cuff / random performance comparisons, and tried a few

different versions of PyPy. Guess what?! For pretty much everything I do, CPython is quicker!! (this is primarily Mongo Database work,

Jinja2 template processing etc etc… i.e. Web application back-end.)

Maybe it’s just me - but worth bearing in mind that not everyone will get benefit from Pypy. I may of course be doing something wrong, but

the fact remains that for me, in a live app, it’s [not quicker | slower] and consumes more resource … one specific instance comes to mind

that’s easy to quantify; in my IDE I shell out and run “pylint” in the background during idle periods between keystrokes, and execution from

PyPy takes twice (or more) the time it takes when called from Python3.

i.e;

(pylint_stdout, pylint_stderr) = lint.py_run(tmp.name+’ -rn -fparseable’, return_std=True)

``

Hawkie is working on adding caching.

Ok, this sounds interesting … :slight_smile:

Btw: Crossbar.io supports HTTP/2 serving Web resources.

?!?!?! Wow, excellent, I missed that!!

(I asked about this 12 months ago, and Alex said there were no plans …)

Are there any docs on this, what do I have to do to my SSL connection to make it http2?

Combine this with the new Universal Transport allows you to run
WAMP/WebSocket, WAMP/RawSocket and Web resouces via HTTP/1.1 or 2 all on
the same port!

Yup, I do this! :slight_smile:

For public facing transport running in production, usually one will want
to run the universal transport on TLS.

I do this too! :slight_smile:

I would say this is a pretty neat feature set =)

Absolutely!

So … Flask or Falcon? :wink:

(https://i.needacandidate.today/ , HTTPS [Letsencrypt] / WAMP, all on one port … 100% Crossbar)

Gareth.

0 Likes

#8

Lot to respond to here but I will say that my experience with PyPy is something of the opposite from yours. I have a heavy Django system that does see some serious performance benefit from PyPy, especially when we throw high load at it.

There are two things to note with PyPy:

  1. It’s a tracing JITing VM which means that the initial run-time performance of something may be slower than CPython because the tracing only kicks in after a given section of code has been executed at least 1000 times (I’m basing this on a PyConZA talk given by one of the PyPy devs last year). Once the tracing has kicked and and compiled the section to native asm then the odds are it will be faster than CPython. This makes benchmarking PyPy vs CPython a little tricky if you don’t take this into account. Your example of running PyLint is problematic because you’re not running PyLink continuously in some kind of watch mode, you’re running it and it’s finishing and then you’re running it again and the tracing information from the previous run (If it got to the point of generating any) was lost.
  2. The PyPy version of Python3 is far less optimised than the Python 2 version. It wasn’t totally clear if you were using the Python 2 version of PyPy or the Python 3 version?
    In most case, a Python application that does a lot of work in pure Python code (as opposed to spending most of its time calling out to C extensions) will be faster in PyPy than CPython over a long run-time. Quick runs of small pieces of code or code which spends most of its time in C extensions will probably not benefit from PyPy that much.

With Crossbar you are liable to see some nice gains with PyPy over a long run period as the WAMP router and the processes it spawns are likely to run for a good amount of time and be executed often enough to get converted by tracing into native asm. Of course, if you’re stopping and starting Crossbar every minute because of changing code you won’t see this.

With regards to Flask vs. Falcon, I’ve only used Flask but Falcon looks nice. In terms of functionality they both look rather similar. I have read some criticism of the way Flask is implemented which could explain why Falcon is faster, as Falcon seems to do less “magic” (Which is a good thing imho).

0 Likes

#9

Hi,

Thanks for that … interesting … Ok, I run in a CI/CD environment so when new stuff is happening on the live server, it’s more than likely there will be an update at least once a day … so in this environment it’s quite likely that a lot of the code will never hit 1000 usages … so will never be optimised … which would explain a lot.

Also, I do very little “data processing”, most of the heavy lifting is done by external libraries which I’m going to assume contain “C”. It sounds like there are two downsides here, firstly Pypy will be slower calling these libraries, and second, the code Pypy “can” optimise isn’t doing “that” much in terms of CPU usage.

I was under the impression the pypy3 stuff was incomplete, but I may be misinformed, from pypy.org; (I was using pypy2)

PyPy implements the Python language version 2.7.10.

And;

Pypy to support Python3; Thanks to our donors, we have raised 45% of the total so far

??

With Crossbar you are liable to see some nice gains with PyPy over a long run period as the WAMP router and the processes it spawns are likely to run for a good amount of time and be executed often enough to get converted by tracing into native asm. Of course, if you’re stopping and starting Crossbar every minute because of changing code you won’t see this.

Mmm, I’m wondering if I should be using Pypy for Crossbar, then CPython for Microservices … but then I don’t really want to be using two different Pythons … :frowning:

With regards to Flask vs. Falcon, I’ve only used Flask but Falcon looks nice. In terms of functionality they both look rather similar. I have read some criticism of the way Flask is implemented which could explain why Falcon is faster, as Falcon seems to do less “magic” (Which is a good thing imho).

Ok, in which case I think I’ll stick with Falcon for development and see how it goes …

Is there any way to see (in pypy), at run-time, whether the code that’s being used has been optimised or not?

Or even better, is there a way to immediately “force” optimisation?

0 Likes

#10

Hi,

PyPy is faster than CPython by a factor 2-50x depending on code.

Startup time is slower, because of the JIT needing to warmup, but that is irrelevant both for Crossbar.io and microservices.

Calling into C will be fast (JIT optimizable) and low-overhead only if the interfacing is done using cffi, not cpyext.

Memory consumption - it depends, but can be lower than CPy.

PyPy will soon get 3.3+ support.

Cheers,
/Tobias

···

Am 26.08.2016 um 10:37 schrieb Gareth Bult:

Hi,

Thanks for that ... interesting .. Ok, I run in a CI/CD environment so when
new stuff is happening on the live server, it's more than likely there will
be an update at least once a day .. so in this environment it's quite
likely that a lot of the code will never hit 1000 usages .. so will never
be optimised .. which would explain a lot.

Also, I do very little "data processing", most of the heavy lifting is done
by external libraries which I'm going to assume contain "C". It sounds like
there are two downsides here, firstly Pypy will be slower calling these
libraries, and second, the code Pypy "can" optimise isn't doing "that" much
in terms of CPU usage.

I was under the impression the pypy3 stuff was incomplete, but I may be
misinformed, from pypy.org; (I was using pypy2)

PyPy implements the Python language version 2.7.10.

And;

*Pypy to support Python3; Thanks to our donors, we have raised 45% of the

total so far*

??

With Crossbar you are liable to see some nice gains with PyPy over a long

run period as the WAMP router and the processes it spawns are likely to run
for a good amount of time and be executed often enough to get converted by
tracing into native asm. Of course, if you're stopping and starting
Crossbar every minute because of changing code you won't see this.

Mmm, I'm wondering if I should be using Pypy for Crossbar, then CPython for
Microservices .. but then I don't really want to be using two different
Pythons .. :frowning:

With regards to Flask vs. Falcon, I've only used Flask but Falcon looks
nice. In terms of functionality they both look rather similar. I have read
some criticism of the way Flask is implemented which could explain why
Falcon is faster, as Falcon seems to do less "magic" (Which is a good thing
imho).

Ok, in which case I think I'll stick with Falcon for development and see
how it goes ...

Is there any way to see (in pypy), at run-time, whether the code that's
being used has been optimised or not?
Or even better, is there a way to immediately "force" optimisation?

0 Likes

#11

Hi,

PyPy is faster than CPython by a factor 2-50x depending on code.

Absolutely … but just to expand a little;

PyPy is faster than CPython by a factor 2-50x depending on code, except when it’s not (!)

Also, anyone looking at Pypy as an option should probably read this first;

http://stackoverflow.com/questions/18946662/why-shouldnt-i-use-pypy-over-cpython-if-pypy-is-6-3-times-faster

I found a number of interesting comments that based on my experience with Pypy I would agree with from the perspective of visible results;

Julian, it’s worth noting that the PyPy folks have been focusing a lot of effort on improving the runtimes of that particular benchmark suite for years now. To some degree it seems that they are “overfitting” their optimizations to this set of benchmarks and, in my experience, aside from purely numerical computations (which are better off in Fortran or C99 anyway), I’ve never gotten PyPy to be more than ~2X faster than CPython. – Alex Rubinsteyn Sep 22 '13 at 19:32

And also;

I’ve been using Pypy a lot. It tends to work very well. However, while Pypy is quite a bit faster for many CPU-heavy workloads, it’s actually slower for the I/O-heavy workloads I’ve thrown at it. For example, I wrote a deduplicating backup program called backshift. For an initial backup, which does lots of file chunking, pypy is great. But for subsequent backups which are mostly just updating timestamps, CPython is faster. – dstromberg Oct 24 '14 at 20:16

Startup time is slower, because of the JIT needing to warmup, but that
is irrelevant both for Crossbar.io and microservices.

Mmm …for an interactive / real-time program, where the speed of the UI is dependent on the speed of the
Microservices, when the initial slowdown is as noticeable as I’m seeing, this is actually an issue. When I initially upgraded

for example, this happened;

user; “- it’s gone slower since the upgrade, what’s up?!”
me; “I upgraded to Pypy which is faster”
user; “no it’s a lot slower … can we put it back?”

  • although I would agree it’s probably not so much of an issue for Crossbar itself …

Calling into C will be fast (JIT optimizable) and low-overhead only if
the interfacing is done using cffi, not cpyext.

I’ll pass on this (!) , all I can say is that I’m using PyMongo and txMongo, and other things like pylint … I’m in the process of

upgrading everything to txMongo, but other than that my options re; third party libraries are limited … and I wouldn’t like to

have to dump a superior library just because it had the ‘wrong’ binary interface …

Memory consumption - it depends, but can be lower than CPy.

Mmm, I’m generally finding that not only is it a lot more, but it increases over time … which is very bad. Trying to provision a

server where the speed and memory consumption are inconsistent and vary over time is a nightmare.

PyPy will soon get 3.3+ support.

:slight_smile:

I do like the idea of Pypy, to the extent I’ve spent a lot of time porting stuff to it, and I have live Crossbar services running on it.

However, based on my experiences thus far, all the new stuff I’m working on is and will be for now, CPython …
(but hopefully, as the code has run under both CPython and Pypy, switching back and fore shouldn’t be too much of a problem in future)

Gareth.

0 Likes

#12

Hi,

PyPy is faster than CPython by a factor 2-50x depending on code.

Absolutely ... but just to expand a little;

PyPy is faster than CPython by a factor 2-50x depending on code, *except
when it's not* (!)

Also, anyone looking at Pypy as an option should probably read this first;
http://stackoverflow.com/questions/18946662/why-shouldnt-i-use-pypy-over-cpython-if-pypy-is-6-3-times-faster

This post is misleading and partially wrong.

I found a number of interesting comments that based on my experience with
Pypy I would agree with from the perspective of visible results;

Julian, it's worth noting that the PyPy folks have been focusing a lot of

effort on improving the runtimes of that particular benchmark suite for
years now. To some degree it seems that they are "overfitting" their
optimizations to this set of benchmarks and, in my experience, aside from
purely numerical computations (which are better off in Fortran or C99
anyway), I've never gotten PyPy to be more than ~2X faster than CPython.
– Alex Rubinsteyn <http://stackoverflow.com/users/481326/alex-rubinsteyn>
Sep 22 '13 at 19:32

Yeah, I have measured it too .. my results are different, and these are the results I trust.

<http://stackoverflow.com/questions/18946662/why-shouldnt-i-use-pypy-over-cpython-if-pypy-is-6-3-times-faster#comment27980827_18947663>

And also;

I've been using Pypy a lot. It tends to work very well. However, while Pypy

is quite a bit faster for many CPU-heavy workloads, it's actually slower
for the I/O-heavy workloads I've thrown at it. For example, I wrote a

This is also wrong.

deduplicating backup program called backshift. For an initial backup, which
does lots of file chunking, pypy is great. But for subsequent backups which
are mostly just updating timestamps, CPython is faster. – dstromberg
<http://stackoverflow.com/users/1084684/dstromberg> Oct 24 '14 at 20:16
<http://stackoverflow.com/questions/18946662/why-shouldnt-i-use-pypy-over-cpython-if-pypy-is-6-3-times-faster#comment41732818_18946662>

..

Startup time is slower, because of the JIT needing to warmup, but that

is irrelevant both for Crossbar.io and microservices.

Mmm ...for an interactive / real-time program, where the speed of the UI is
dependent on the speed of the
Microservices, when the initial slowdown is as noticeable as I'm seeing,
this is actually an issue. When I initially upgraded

Then you are doing it wrong. A microservice should not be started per request (or call or whatever), but long running ..

for example, this happened;

user; "- it's gone slower since the upgrade, what's up?!"

me; "I upgraded to Pypy which is faster"
user; "no it's a lot slower ... can we put it back?"

- although I would agree it's probably not so much of an issue for
Crossbar itself ..

Calling into C will be fast (JIT optimizable) and low-overhead only if
the interfacing is done using cffi, not cpyext.

I'll pass on this (!) , all I can say is that I'm using PyMongo and
txMongo, and other things like pylint .. I'm in the process of
upgrading everything to txMongo, but other than that my options re; third
party libraries are limited .. and I wouldn't like to
have to dump a superior library just because it had the 'wrong' binary
interface ...

Memory consumption - it depends, but can be lower than CPy.

Mmm, I'm generally finding that not only is it a lot more, but it increases
over time ... which is very bad. Trying to provision a

No, wrong. PyPy has a generational, incremental GC which is much better than the ref counted one in CPython

Anyway, whatever;)

Tobias

···

Am 26.08.2016 um 13:43 schrieb Gareth Bult:

server where the speed and memory consumption are inconsistent and vary
over time is a nightmare.

PyPy will soon get 3.3+ support.

:slight_smile:

I do like the idea of Pypy, to the extent I've spent a lot of time porting
stuff to it, and I have live Crossbar services running on it.
However, based on my experiences thus far, all the new stuff I'm working on
is and will be for now, CPython ...
(but hopefully, as the code has run under both CPython and Pypy, switching
back and fore shouldn't be too much of a problem in future)

Gareth.

0 Likes

#13

Ok, I guess I’m doing something wrong then … I’ll go back and see if I can generate some stand-alone examples, it sounds like something I’m doing “inside” Crossbar/MS apps isn’t playing nice with Pypy …

0 Likes

#14

Ok, I’m trying to nail down 1 example issue, if anyone can point me in the right direction I would be very grateful. I need to call “pylint” in real time, so timing is an issue, to be realistic the maximum run-time should be less than 1 second per iteration. I’ve converted what I’m using into a small file that I’m calling “benchmark.py”. My test is to ensure I have the right version of pylint installed for a given version of python, then to run;

benchmark.py

from pylint import epylint as lint
import timeit

def test(log=False):
(pylint_stdout, pylint_stderr) = lint.py_run(‘benchmark.py -rn -fparseable’, return_std=True)
if not log: return
print(“Err>”,pylint_stderr.getvalue().split("\n"))
print(“Out>”,pylint_stdout.getvalue().split("\n"))

test(True) # check the output and make sure it’s “real”
timer=timeit.timeit(test,number=20)
print(“Time taken=”,timer)

``

These are the results I’m getting for 20 iterations of the various versions of Python at my disposal;

CPython2: Time taken = 8.780210971832275
CPython3: Time taken = 13.967931449995376
pypy2: Time taken = 21.180224895477295
pypy3: Time taken = 27.235675559029914 (this is using the latest ‘alpha’ which is @3.5)

``

Apparently there is a caching issue with “pylint” so it “must” be run in a sub-process, and this seems to be the recommended way to do it.

I’m stuck … can anyone see what’s going wrong?

My guess is that “pylint” is mis-behaving in some way (?) or doing something in a way Pypy really doesn’t like (?)

0 Likes

#15

The problem is that pylint is being run as a sub-process. This means it will never get properly optimised by PyPy unless you tell it to lint something so huge that it takes long enough for the tracing to optimise execution. Even then, any tracing information will be thrown away and not have an impact on the next run. Furthermore, because of the sub-process you are also also seeing the worst possible performance of JIT warm-up.

Unfortunately you can throw a switch to force PyPy to always generate asm because that would be effectively turn it into an AOT compiler. Tracing and generating native asm is an expensive process which is why PyPy only does it for code that has been executed often (and can thus be reasonably assumed to be likely to be executed even more often in the future).

You should investigate this caching issue in pylint and determine if there is a way to fix it for your usage. Otherwise I guess you may be stuck with using CPython for running pylint :-/

I won’t answer the other stuff from earlier because Tobias seems to have covered it nicely :slight_smile:

···

On Friday, 26 August 2016 19:44:48 UTC+2, Gareth Bult wrote:

Ok, I’m trying to nail down 1 example issue, if anyone can point me in the right direction I would be very grateful. I need to call “pylint” in real time, so timing is an issue, to be realistic the maximum run-time should be less than 1 second per iteration. I’ve converted what I’m using into a small file that I’m calling “benchmark.py”. My test is to ensure I have the right version of pylint installed for a given version of python, then to run;

benchmark.py

from pylint import epylint as lint
import timeit

def test(log=False):
(pylint_stdout, pylint_stderr) = lint.py_run(‘benchmark.py -rn -fparseable’, return_std=True)
if not log: return
print(“Err>”,pylint_stderr.getvalue().split("\n"))
print(“Out>”,pylint_stdout.getvalue().split("\n"))

test(True) # check the output and make sure it’s “real”
timer=timeit.timeit(test,number=20)
print(“Time taken=”,timer)

``

These are the results I’m getting for 20 iterations of the various versions of Python at my disposal;

CPython2: Time taken = 8.780210971832275
CPython3: Time taken = 13.967931449995376
pypy2: Time taken = 21.180224895477295
pypy3: Time taken = 27.235675559029914 (this is using the latest ‘alpha’ which is @3.5)

``

Apparently there is a caching issue with “pylint” so it “must” be run in a sub-process, and this seems to be the recommended way to do it.

I’m stuck … can anyone see what’s going wrong?

My guess is that “pylint” is mis-behaving in some way (?) or doing something in a way Pypy really doesn’t like (?)

0 Likes

#16

Ok, I have another little test.

I was a bit quick off the mark initially as I only compared pypy3 to python3, and pypy3 obviously has some issues atm, but then it’s still ‘alpha’.

Extending the test (below) to python2 and pypy2, pypy comes out 1.5x faster than CPython.

import txmongo
from twisted.internet.defer import inlineCallbacks
from twisted.internet import reactor

tx = txmongo.MongoConnection()
db = tx.ion

@inlineCallbacks
def get_some_stuff():

iterations = 0
while iterations < 100:
modes = {}
nodes = yield db.nodes.find()
for node in nodes:
mode = node.get(‘mode’,‘none’)
if mode in modes:
modes[mode] += 1
else:
modes[mode] = 1

print “{}%\r”.format(iterations)
iterations += 1

print
total = 0
for mode in modes:
count = modes[mode]
print “{}> {}”.format(mode,count)
total += count
print “Total of {} files.”.format(total)
reactor.stop()

reactor.callLater(0,get_some_stuff)
reactor.run()

``

pypy2: 18s
python2: 28s
python3: 29s
pypy3: 40s

``

Interestingly, if I change the test a little and replace the find with .find(limit=10) and increase iterations to 10,000 i.e. increase the amount of work the DB is asked to do, it looks like this;

pypy2: 6s
python2: 7s
python3: 7s
pypy3: 12s

``

Indeed if one plays with the numbers, for 999 iterations of the latter version of the test, CPython2 is 2x faster than Pypy.

Would I be right in thinking for “unoptimised code”, CPython is actually faster than Pypy?

In this particular instance, I seem to need around 5000 iterations before the total accumulated CPU usage is less for Pypy … (?)

This would explain why my application seems slower as it’s quite possible I’m not getting near this number of iterations for most of the codebase before the nightly restart.

I’m wondering if there is any way to force pypy to optimise “sooner”, or if it can be “pre-compiled” … will have to do some reading …

0 Likes