[Question] Quicken the App.create() process time

[Kyria]

I have a little question (not a bug) about the App.create() process.

As I dev, i need to often restart my app, which causes my app to instantiate each time pyswagger, and with 4 or more endpoint, it begins to be a significant time to wait.

By curiosity, I've tested bravado to compare both (which don't work the same I believe) and it looks like in average pyswagger App.create() is 2 to 3 times slower than bravado SwaggerClient.from_url()... (tested multiple time on the same server (Xeon, 16GB RAM, 250Mbps), using timeit)

Any idea why ? Is it possible to quicken this in any way ? I know a possible solution is to cache the App.create() result, as I already do this, but is there any other way ?

Thanks !

[mission-liao]

@Kyria I think the way pyswagger loading an OpenAPI would be slow because of these steps:

• generating a description layer to represent the JSON/Yaml document as python objects.
• perform preparation/patching by traversing the tree composed of those python objects.

If you can provide a sample document for me, maybe I can try to generate a breakdown for further investigation.

[Kyria]

You can try with this one swagger.json (it's basically a copy/paste from the one I use).

What I tested (using the url above, so at least you know what I did):

from pyswagger import App
from bravado.client import SwaggerClient
import timeit

bra = "SwaggerClient.from_url('https://gist.githubusercontent.com/Kyria/5e00fa2aac2e23eaecbcb307a4b57230/raw/38f4b5b288c49cb0ac84e5b228b5cc61629887e9/swagger.json')"
psw = "App.create('https://gist.githubusercontent.com/Kyria/5e00fa2aac2e23eaecbcb307a4b57230/raw/38f4b5b288c49cb0ac84e5b228b5cc61629887e9/swagger.json')"

# test on calling these 2 lines 10 times 
timeit.timeit(bra, "from __main__ import SwaggerClient", number=10)
# returns around 14-18 sec

timeit.timeit(psw, "from __main__ import App", number=10)
# returns around 50-55 sec

[mission-liao]

@Kyria Thanks for sharing samples. I run profiling against your sample code (remove bravado part, of course) and repeat App.create 10 times. The total time is 147s. Here is the breakdown of some significant calls:

cumulative seconds	filename:lineno(function)
17.837	base.py:238(get_private_name)
25.107	merge.py:18(_merge)
75.301	scan.py:6(default_tree_traversal)
129.006	scan.py:104(scan)
22.970	utils.py:231(jp_compose)

In short:

• The part to support $ref in PathItem takes time merge.py:18(_merge)
• The way to encapsulate object property seems not necessary base.py:238(get_private_name)
• The way that pyswagger modulizes the code to patch object is also a time consuming part
  • scan.py:6(default_tree_traversal)
  • scan.py:104(scan)
• I found that in pyswagger, sometimes we split a JSON point and compose it back, this part might be reduced utils.py:231(jp_compose)

I'm in the middle of migrating pyswagger to support OpenAPI 3.0.0 spec (refer to this issue). Therefore the story would be different if I want to tune the performance for loading such spec. I think your sample would be a great reference for me to tune performance after the migration is done. However, it's would be redundant to tune performance for pyswagger at this moment.

[Kyria]

I understand you won't do it right now, the goal of this issue was mostly to make you aware of that performance with some specs and to look for possible solutions (if you had any).

Thanks !