cloud-sql-python-connector icon indicating copy to clipboard operation
cloud-sql-python-connector copied to clipboard

Support/document connection pooling for async drivers

Open jackwotherspoon opened this issue 3 years ago • 0 comments

The Cloud SQL Python Connector recommends using our library with connection pooling (usually through SQLAlchemy library)

However, currently SQLAlchemy's create_async_engine method does not allow the use of an asynchronous creator argument #8215

This makes it very difficult to use connection pooling with the Cloud SQL Python Connector's async drivers. Currently there is a workaround that is both confusing and not practical to recommend to our users:

import asyncio
from sqlalchemy import text
from sqlalchemy.ext.asyncio import create_async_engine
from sqlalchemy.util import await_only
from google.cloud.sql.connector import Connector

async def async_creator():
    loop = asyncio.get_running_loop()
    async with Connector(loop=loop) as connector:
        conn = await connector.connect_async(
            "project:region:instance", # Cloud SQL instance connection name"
            "asyncpg",
            user="my-db-user",
            password="my-db-password",
            db="my-db-name",
        )
        return conn

async def async_main():
    def adapted_creator():
        dbapi = engine.dialect.dbapi
        from sqlalchemy.dialects.postgresql.asyncpg import (
            AsyncAdapt_asyncpg_connection,
)
        return AsyncAdapt_asyncpg_connection(
            dbapi,
            await_only(async_creator()),
            prepared_statement_cache_size=100,
        )

    # create async connection pool with wrapped creator
    engine = create_async_engine(
        "postgresql+asyncpg://",
        echo=True,
        creator=adapted_creator,
    )
    
    # use connection pooling with Cloud SQL Python Connector
    async with engine.connect() as conn:
        query = await conn.execute(text("SELECT * from ratings"))
        results = query.fetchall()
        for row in results:
            print(row)


asyncio.run(async_main())

For this reason, we should look at the possibility of supporting and returning native connection pools from the Cloud SQL Python Connector for async drivers. (ex. asyncpg.create_pool)

This will allow users to have the benefits of connection pooling without the need for the confusing SQLAlchemy workaround.

I would suggest a connector.create_pool method or something that has the exact same interface as connector.connect_async:

    # intialize Connector object using 'create_async_connector'
    connector = await create_async_connector()

    # create connection pool to Cloud SQL database
    conn: asyncpg.Pool = await connector.create_pool(
        "project:region:instance", # Cloud SQL instance connection name
        "asyncpg",
        user="my-db-user",
        password="my-db-pass",
        db="my-db-name",
        # ... additional database driver or connection pool args 
    )

jackwotherspoon avatar Aug 09 '22 15:08 jackwotherspoon