backend.ai icon indicating copy to clipboard operation
backend.ai copied to clipboard

refactor: Split the session & kernel tables with ORM migration

Open fregataa opened this issue 3 years ago • 0 comments

resolves #325

1. Add Session DB table.

  • This table is used to schedule and manage compute sessions.
  • It seems to have many columns that overlaps with kernel DB table. We need to normalize those tables according to our spec.

2. Migrate to SQLAlchemy ORM partially

  • Migrate manager's scheduler codes from SQLAlchemy Core to ORM and use SQLAlchemy 2.0 query styles. We still are able to use Core tables and SQLAlchemy 1.x query styles.

3. Change compute session scheduler to use session table and APIs.

  • We have managed compute session with kernel table which cause blocking issues -e.g) cannot implement session tag table- and make us confused. From now, data of scheduling jobs such as enqueuing, preparing and destroy compute session will be inserted to session table. And we can handle those scheduling job data by APIs implemented in sessions.py.

4. Optimize DB query

  • Thanks to the new session table, we are able to optimize some DB query statements especially which are related to Manager's scheduler.
  • But there are a few obstacles since SQLAlchemy does not fully support Asynchronous tasks -e.g) Unable lazy loading- and usage of SQLAlchemy's orm.Session is not concrete -e.g) boundary of orm.Session is arbitrary. We can optimize such tasks much more.

Plus

  • We need to refactor this scheduler codes since this migrated code followed some legacy codes. e.g) it assumes that there is no heterogeneous kernels in one session such as no different images or etc, Updating status of compute sessions and kernels process is not clear yet. Need to specify the steps and interactions between session and kernel through updating status.
  • some of manager API still use SQLAlchemy Core based queries. We can update the codes.

TODO

  1. Normalize Session DB table and Kernel DB table. Mainly drop duplicated and legacy columns.
  2. Optimize the usage of SQLAlchemy Session.

Question

fregataa avatar Jul 22 '22 08:07 fregataa