Convert/Port Flask code to FastAPI

[lasryariel]

• Parent Issue: #771

Overview

The goal of this task is to migrate the existing Flask codebase to FastAPI to improve performance, flexibility, and development speed. FastAPI offers features such as asynchronous request handling, Pydantic for data validation, and better dependency injection, which will enhance our current implementation. This migration will also ensure that our project uses more modern and efficient frameworks.

Action Items

• [x] Review Existing Code: Identify the Flask code that needs to be ported to FastAPI. John is available for assistance in reviewing the codebase.
• [x] Implement Pydantic and Dependency Injection: Replace or update existing data validation mechanisms with Pydantic models. Use FastAPI's dependency injection features to streamline the service layer.
• [x] Evaluate Conversion Benefits: Assess which parts of the codebase would benefit from rewriting and which can be ported directly without modification.
• [x] Implement Configuration Injection: Ensure that configuration handling is updated to follow FastAPI's methods, including environment variable management and settings injection.
• [ ] API Error Design: Design consistent API errors for easy consumption by the front-end. Note: It will have a basic shape with message, code, status.
  • (can be converted to it's own issue)
• [x] Update Alembic Migration: Prune the old migration scripts using the instructions at Building an Up to Date Database from Scratch. Create a new Alembic migration environment.
• [x] Create Test Users: In development environment, pre-populate Cognito and DB with test users.
• [x] Update Docker file configuration: Ensure the Docker file can create a Docker image containing the FastAPI implementation and that a container can successfully run the API.
• [x] Update GitHub Action workflows: Update the workflows so that the new, migrated tests are run during a push to GitHub (only, not trying to deploy).
• [ ] Flask Deprecation: Confirm that all necessary Flask components have been successfully replaced and we are ready to deprecate Flask entirely.

Resources/Instructions

• Focus on improving code clarity and maintainability.
• FastAPI documentation for reference.
• SQLAlchemy documentation for understanding database handling and migration steps.
• Pydantic documentation for data validation guidance.

[paulespinosa]

Flask auth and users endpoints

These were converted as part of #788

• [X] /auth/confirm: Changed to /auth/signup/confirm

  • [X] get
  • [X] post - This method has been removed

• [X] /auth/confirmInvite: Changed to /auth/confirm-invite

  • [X] get

• [X] /auth/forgot_password: Changed to /auth/forgot-password.

  • [X] post

• [X] /auth/forgot_password/confirm: Changed to /auth/forgot-password/confirm.

  • [X] post

• [ ] /auth/google: Deferred until we get another Cognito/incubator account

  • [ ] get

• [ ] /auth/google/sign_in: Deferred until we get another Cognito/incubator account

  • [ ] post

• [ ] /auth/google/sign_up: Deferred until we get another Cognito/incubator account

  • [ ] post

• [x] /auth/invite

  • [x] post

• [x] /auth/new_password: Changed to /auth/new-password.

  • [x] post

• [X] /auth/private: Changed to /auth/secret.

  • [X] get

• [X] /auth/refresh

  • [X] get

• [x] /auth/resend_confirmation_code

  • [x] post

• [X] /auth/session

  • [X] get

• [X] /auth/signin

  • [X] post

• [X] /auth/signout

  • [X] post

• [X] /auth/signup/coordinator: Changed to /auth/signup with role provided in POST body.

  • [X] post

• [X] /auth/signup/host: Changed to /auth/signup with role provided in POST body.

  • [X] post

• [X] /auth/user: Changed to /users/current implemented in file modules/access/users_controller.py.

  • [X] get

• [X] /users/{userId}: Implemented in modules/access/users_controller.py.

  • [X] delete

Coordinators endpoint

• [x] /coordinator/dashboard/all
  • [x] get

Hosts endpoint

• [x] /host
  • [x] get

Forms endpoints focuses on Intake Profile

The Forms API has become a submodule of Intake Profile.

• [X] /forms: Not migrated because the Forms editor feature will be not be implemented.
  • [X] post: Not migrated because the Forms editor feature will be not be implemented.
• [X] /forms/{form_id}: Changed to /intake-profile/form/{form_id}. Serves a JSON representation of the form given an ID.
  • [X] get
• [X] /responses/{form_id}: Changed to /intake-profile/{intake_profile_id}. Just moved the code over into FastAPI but not made ready to be used by the frontend.
  • [X] get
  • [X] put

The Service Providers name has changed to Housing Orgs

These were converted as part of #788. The code is located in modules/tenant_housing_orgs.

• [X] /serviceProviders: Changed to /housing-orgs
  • [X] get
  • [X] post
• [X] /serviceProviders/{providerId}: Changed to /housing-orgs/{housing_org_id}.
  • [X] delete
  • [X] get
  • [X] put

Health check endpoints

• [x] /health
  • [x] get

[erikguntner]

Thanks @paulespinosa this is super helpful.

[paulespinosa]

In the FastAPI migration, code has been organized according to "workflow capability" rather than "technical function." It represents the current model used to represent a Host Home Program workflow.

As we learn more, the organization of code and the choices described below will change.

Code Organization

The directories of the front-end code and the back-end code have changed as follows:

	Before	After
The React Front-end	/app	/frontend
The FastAPI Back-end	N/A	/backend
The old Flask Back-end	/api	/flask-api

The new FastAPI Back-end

Under API code is now located in the /backend/app directory. Under this directory, the code is organized as follows:

• /backend/app/core
• /backend/app/modules

The modules/ Python package (i.e. directory), contains sub-packages (sub-directories) for the "business functions." Each of the sub-directories below contain their own controllers, models, schemas, and other related code used to implement their responsibilities.

• access - The code in this directory is responsible for sign-up, sign-in, integration with the third party Identity Provider AWS Cognito, maintaining the API's user and roles. All things related to identity, authentication, and authorization.
• intake_profile - The code in this directory is responsible for Intake Profiles.
• onboarding - The code in this directory is responsible for Onboarding Guests and Hosts.
• relaionship_management - The code in this directory is responsible for Relationship Management.
• tenant_housing_orgs - The code in this directory is responsible for maintaining information about the Housing Organization and it's Host Home Program.

Poetry

The back-end API now uses the Python package and dependency management tool poetry https://python-poetry.org/.

The project dependencies are specified in /backend/pyproject.toml.

The poetry tool reads pyproject.toml to create virtual environments, install dependencies, and lock dependencies.

DataAccessLayer and SQLAlchemy Models

In the Flask-based API, api/openapi_server/models/database.py contained all SQLAlchemy models and a class called DataAccessLayer.

The DataAccessLayer class was not migrated. The SQLAlchemy Session is now dependency injected into path operation functions by declaring a parameter with db_session: DbSessionDep. (The name of the parameter can be any name but the type must be DBSessionDep.) For example:

# FastAPI-based API SQLAlchemy Session dependency injection
@router.get("/{housing_org_id}")
def get_housing_org(housing_org_id: int, db_session: DbSessionDep) -> schemas.HousingOrg | None:

In the FastAPI-based API - The SQLAlchemy models are moved to their related packages under modules/. For example, the SQLAlchemy model class User(Base) is now located in modules/access/models.py.

SQLAlchemy models are defined by importing Base from the core.db module. For example:

from app.core.db import Base

class User(Base):
    __tablename__ = "user"
   # ...

Migrating Models to SQLAlchemy 2.0

In the FastAPI-based API, SQLAlchemy models have been updated to using the 2.0 style declarative mappings following the steps documented in the Migrating an Existing Mapping section of the SQLAlchemy 2.0 Migration Guide.

For example, the new HousingOrgs model uses the mapped_column, Mapped type, and Annotated to create a reusable type:

intpk = Annotated[int, mapped_column(primary_key=True)]

class HousingOrg(Base):
    __tablename__ = "housing_orgs"

    housing_org_id: Mapped[intpk]
    org_name: Mapped[str] = mapped_column(String, nullable=False, unique=True)
    programs: Mapped[List["HousingProgram"]] = relationship(
        back_populates="housing_org")

Data Schemas

Data schemas represent the shape of the data that come into and go out of the API via the HTTP endpoints (a.k.a path operation functions).

In the Flask-based API, api/openapi_server/models/schema.py contained all of the data schemas. These data schemas were based off of the marshmallow library and, with the help of the marshmallow_sqlalchemy library, allowed direct conversion of SQLAlchemy models to marshmallow data schemas.

In the FastAPI-based API, the data schemas have been moved to their related packages under modules/. For example, the Flask-based API data schema class RoleSchema(SQLAlchemyAutoSchema) has been moved to the FastAPI-based API data schema modules/access/schemas.py as class RoleBase(BaseModel).

In the FastAPI-based API, marshallow is not used. pydantic is used to define the data schemas. It has the built-in ability to transform SQLAlchemy models to data schemas automatically by defining model_config = ConfigDict(from_attributes=True) in the class that defines the data schema. For example:

# FastAPI-based API data schema
class RoleBase(BaseModel):
    id: int
    type: UserRoleEnum

    model_config = ConfigDict(from_attributes=True)

CRUD and Repositories

In the Flask-based API, database access was performed directly via the SQLAlchemy Session in a controller or indirectly through a class that roughly implemented the Repository pattern.

In the FastAPI-based API, database access is performed either in a crud.py file or by using a class that implements the Repository pattern. These files are located in their related packages under modules/. For example, the modules/tenant_housing_orgs package has the file crud.py containing code used for CRUD (Create, Read, Update, Delete) operations for Housing Orgs.

For simple CRUD-like operations on a SQLAlchemy model, use a CRUD file to define the operations. For more advanced use of domain models, use of the Repository pattern is a consideration.

In either case, transactions and commits are maintained by the caller. For example, the Housing Orgs controller below maintains the database transaction. The transaction automatically commits the changes.

@router.post("/",
             status_code=status.HTTP_201_CREATED,
             response_model=schemas.HousingOrg)
def create_housing_org(
        housing_org: schemas.HousingOrg,
        request: Request,
        session: DbSessionDep) -> Any:

    with session.begin():
        db_org = crud.read_housing_org_by_name(session, housing_org.org_name)
        if db_org:
            redirect_url = request.url_for('get_housing_org',
                                           **{'housing_org_id': db_org.housing_org_id})
            return RedirectResponse(url=redirect_url,
                                    status_code=status.HTTP_303_SEE_OTHER)

        new_housing_org = models.HousingOrg(org_name=housing_org.org_name)
        crud.create_housing_org(session, new_housing_org)

    session.refresh(new_housing_org) 

Controllers

In the Flask-based API, all of the controllers were located in api/openapi_server/controllers/.

In the FastAPI-base API, the controllers have been moved to their related packages under 'modules/'. For example, the Flask-based API api/openapi_server/controllers/auth_controller.py has been moved to the FastAPI-based API under modules/access/auth_controller.py.

Endpoints (a.k.a. path operation functions) are defined in the "controller" files using the FastAPI decorators. For example:

router = APIRouter()

@router.get("<endpoint path>")
@router.post("<endpoint path>")
@router.put("<endpoint path>")
@router.delete("<endpoint path>")

Dependency Injection

The FastAPI-based API uses FastAPI's dependency injection system. The dependencies are defined in modules/deps.py. FastAPI automatically injects dependencies when they are used in the parameters of path operation functions. For example:

# FastAPI-based API SQLAlchemy Session dependency injection
# DbSessionDep is defined in modules/deps.py
@router.get("/{housing_org_id}")
def get_housing_org(housing_org_id: int, db_session: DbSessionDep) -> schemas.HousingOrg | None:

Routing

The top-level router to the /api path is defined in main.py. The routes under this path are defined in modules/router.py. It defines the routes to each of the modules under modules/. FastAPI automatically finds all routers declared and used in controllers.

API Settings

In the Flask-based API, the API configuration settings were defined in api/openapi_server/configs/.

In the FastAPI-based API, the API configuration settings are located in core/config.py. It uses pydantic-settings to read environment variables or the .env file.

The Settings are available to path operation functions via dependency injection. The SettingsDep dependency is defined in modules/deps.py.

Database

In the FastAPI-based API, the SQLAlchemy database engine and session code is defined in core/db.py. Most interaction with SQLAlchemy Session or Engine will be provided via dependency injection to a controller's path operation function.

Testing

In the FastAPI-based API, tests have the sub-directories:

• e2e for end-to-end testing
• integration for integration testing
• unit for unit testing features of each module

pytest fixtures are similar to FastAPI dependency injection system. They look the same but are written slightly different, so be aware the differences. The fixtures are defined in tests/conftest.py. A notable integration test fixture is the client fixture which is a TestClient that can be used to make calls to the HUU API.

@pytest.fixture
def client(session_factory) -> TestClient:

    def override_db_session():
        try:
            session = session_factory()
            yield session
        finally:
            session.close()

    main_api.dependency_overrides[db_session] = override_db_session
    main_api.dependency_overrides[get_cognito_client] = lambda: None

    return TestClient(main_api)

An example use of the TestClient is as follows. Note the name of the test function's parameter is the name of the fixture. pytest will automatically pass the TestClient to the test function when written this way:

def test_signin_with_fake_credentials(client):
    response = client.post(PATH + '/signin',
                           json={
                               'email': '[email protected]',
                               'password': '_pp#FXo;h$i~'
                           })

    body = response.json()
    assert response.status_code == 400, body
    assert body["detail"]["code"] == "UserNotFoundException", body

To mock AWS Cognito, moto is used to mock

Alembic

Alembic is a SQLAlchemy database migration tool. It is used to update database schemas in a existing environment whenever a change is deployed.

During the migration, the existing migration scripts have been deleted. This was done since all existing/older environments will be created from zero again. This includes the incubator environment.

If you have an existing Postgres container volume or SQLite database, then they will need to be deleted. The PostgreSQL container volume can be deleted using the command:

docker volume rm homeuniteus_db-data 

Docker

In the migrated codebase, docker-compose.yml has been updated to contain the following containers:

• db has the PostgreSQL server running.
• motoserver has the moto server running. This is a mocked version of AWS.
• pgadmin has the pgAdmin4 server running. This allows developers to query the PostgreSQL server from the browser.
• backend has the API server running. It runs the startup scripts that pre-populate the database and moto server with test user accounts.
• frontend has a nginx server running serving a built version of the frontend.

The db and motoserver docker containers are now required (loosely speaking) to be running during development.

docker compose up -d --build pgadmin motoserver

The convenience container, pgadmin, transitively runs the db container.

The design of the Docker environment is pictured below.

GitHub Actions

[johnwroge]

Thanks Paul. This is very helpful!

[erikguntner]

Thanks @paulespinosa this is awesome info!