dchevell
ProcessPoolExecutor raise Exception
currently using thread pool testing on local machine (windows) :
app.config['EXECUTOR_TYPE'] = 'thread' app.config['EXECUTOR_MAX_WORKERS'] = 4
but facing issue while using process pool executor :
app.config['EXECUTOR_TYPE'] = 'process' app.config['EXECUTOR_MAX_WORKERS'] = 4
// extention.py from flask_marshmallow import Marshmallow from flask_sqlalchemy import SQLAlchemy from flask_executor import Executor from requests_futures.sessions import FuturesSession
db = SQLAlchemy() ma = Marshmallow() executor = Executor() session = FuturesSession(executor = executor )
//run.py from flask import Flask from extention import jwt, db, executor from logging.handlers import RotatingFileHandler
def create_app(): app = Flask(name) app.config['EXECUTOR_TYPE'] = 'process' app.config['EXECUTOR_MAX_WORKERS'] = 1 app.config['SQLALCHEMY_DATABASE_URI'] = "mysql+mysqlconnector://XXXXXX:XXXXX@XXXXX:3306/admin_server_anypay" #SQL Server app.config['SQLALCHEMY_POOL_RECYCLE'] = 280 app.config['SQLALCHEMY_POOL_SIZE'] = 20 app.config['SQLALCHEMY_MAX_OVERFLOW'] = 20 app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False app.config['SECRET_KEY'] = 'XXX' app.config['JWT_SECRET_KEY'] = 'XXX' app.config['JWT_ACCESS_TOKEN_EXPIRES'] = False app.config['JWT_HEADER_NAME'] = 'Signature' app.config['JWT_HEADER_TYPE'] = '' app.config['PROPAGATE_EXCEPTIONS'] = True configure_extensions(app) register_blueprints(app) return app
def configure_extensions(app): db.init_app(app) with app.app_context(): db.Model.metadata.reflect(db.engine)
def register_blueprints(app): with app.app_context(): import superview app.register_blueprint(superview.blueprint) executor.init_app(app)
if name == "main": app = create_app() app.run(host='localhost', debug=True, port=1234)
rocess Process-1:
Traceback (most recent call last):
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self.kwargs)
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\concurrent\futures\process.py", line 169, in process_worker
call_item = call_queue.get(block=True)
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\multiprocessing\queues.py", line 113, in get
return ForkingPickler.loads(res)
File "C:\devdive\Myapp\cmrestless\resources\resapp.py", line 18, in
The documentation doesn't make this clear enough, but right now there's no supported way in Flask to serialise an app / request context so that it can be used inside a different process. Flask provides mechanisms to copy request context to another thread, and I wrote another one to copy an application context, but serializing is different - there are a lot of things inside an application context that can't be serialised.
For now I would say use a ThreadPoolExecutor for this, but I'm working on a way to allow a "limited" app/request context to be serialised for this kind of use case. I'll leave this open so I can post any findings or updates.
@dchevell Very glad to have come across your module after a morning of learning and eventually discovering that Flask can't support Process Pool Execution.
Re your comment above, do you have any further thoughts / update on this?
I can post a separate issue if you'd prefer.
Huge thanks
If this is still an issue for anyone, please retest under 1.0 with the latest changes to how application contexts are pushed, and feel free to reopen or open a new issue.