teable icon indicating copy to clipboard operation
teable copied to clipboard
verified publisher icon teableio

S3 storage not working on private bucket

Open mz0in opened this issue 1 year ago • 4 comments

in the recent update when enabling BACKEND_STORAGE_PROVIDER=s3 the app won't work

mz0in avatar Sep 19 '24 12:09 mz0in

Any more details?

boris-w avatar Sep 20 '24 03:09 boris-w

configuring the S3 , it was working before , and when the new PR plugin arrived it stoped working I thought it was a docker issue after looking into it t was because of this : BACKEND_STORAGE_PROVIDER when configuring it to : BACKEND_STORAGE_PROVIDER=local the app work on the configured port , when I switch it to s3 , the container is running but the url won't work here is the log snap :

2024/09/20 03:52PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/comment/:tableId/:recordId/:commentId/reaction, DELETE} route 2024/09/20 03:52PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/comment/:tableId/:recordId/:commentId/reaction, PATCH} route

  • info Sourcemaps generation have been disabled through NEXT_BUILD_ENV_SOURCEMAPS
  • info Sentry enabled for this build 2024/09/20 03:52PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=OfficialPluginInitService spanId=9211dedb0e6bf6c0 traceId=5b8bad8f4c489233ccf81db65204fda5 msg=Creating official plugin: Chart 2024/09/20 03:52PM 50 pid=7 hostname=0ec94d8f5e08 name=teable context=MailerService msg=Transporter is ready ▲ Next.js 14.1.4
    • Local: http://localhost:3002
    • Network: http://0.0.0.0:3002 ✓ Ready in 346ms

2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/oauth/client/:clientId/secret, POST} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/oauth/client/:clientId/secret/:secretId, DELETE} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/oauth/client/:clientId/revoke-access, POST} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/oauth/client/:clientId/revoke-token, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/oauth/client/authorized/list, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RoutesResolver msg=OAuthServerController {/api/oauth}: 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/oauth/authorize, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/oauth/access_token, POST} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/oauth/decision, POST} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/oauth/decision/:transactionId, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RoutesResolver msg=TrashController {/api/trash}: 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/trash, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/trash/items, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/trash/restore/:trashId, POST} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/trash/reset-items, DELETE} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RoutesResolver msg=PluginController {/api/plugin}: 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/plugin, POST} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/plugin, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/plugin/:pluginId, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/plugin/:pluginId/regenerate-secret, POST} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/plugin/:pluginId, PUT} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/plugin/:pluginId, DELETE} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/plugin/center/list, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/plugin/:pluginId/submit, PATCH} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/plugin/:pluginId/token, POST} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/plugin/:pluginId/refreshToken, POST} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/plugin/:pluginId/authCode, POST} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RoutesResolver msg=DashboardController {/api/base/:baseId/dashboard}: 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/base/:baseId/dashboard, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/base/:baseId/dashboard/:id, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/base/:baseId/dashboard, POST} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/base/:baseId/dashboard/:id/rename, PATCH} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/base/:baseId/dashboard/:id/layout, PATCH} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/base/:baseId/dashboard/:id, DELETE} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/base/:baseId/dashboard/:id/plugin, POST} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/base/:baseId/dashboard/:id/plugin/:pluginInstallId, DELETE} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/base/:baseId/dashboard/:id/plugin/:pluginInstallId/rename, PATCH} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/base/:baseId/dashboard/:id/plugin/:pluginInstallId/update-storage, PATCH} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/base/:baseId/dashboard/:id/plugin/:pluginInstallId, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RoutesResolver msg=CommentOpenApiController {/api/comment/:tableId}: 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/comment/:tableId/:recordId/count, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/comment/:tableId/count, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/comment/:tableId/:recordId/attachment/:path, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/comment/:tableId/:recordId/subscribe, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/comment/:tableId/:recordId/subscribe, POST} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/comment/:tableId/:recordId/subscribe, DELETE} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/comment/:tableId/:recordId/list, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/comment/:tableId/:recordId/create, POST} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/comment/:tableId/:recordId/:commentId, GET} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/comment/:tableId/:recordId/:commentId, PATCH} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/comment/:tableId/:recordId/:commentId, DELETE} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/comment/:tableId/:recordId/:commentId/reaction, DELETE} route 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=RouterExplorer msg=Mapped {/api/comment/:tableId/:recordId/:commentId/reaction, PATCH} route

  • info Sourcemaps generation have been disabled through NEXT_BUILD_ENV_SOURCEMAPS
  • info Sentry enabled for this build 2024/09/20 03:57PM 30 pid=7 hostname=0ec94d8f5e08 name=teable context=OfficialPluginInitService spanId=d12f252caa3a4a0c traceId=4a8446632d757b9b1b4ff0f98ed84971 msg=Creating official plugin: Chart 2024/09/20 03:57PM 50 pid=7 hostname=0ec94d8f5e08 name=teable context=MailerService msg=Transporter is ready

mz0in avatar Sep 20 '24 13:09 mz0in

I tried configuring S3 for docker startup, and it works fine. Could you let me take a look at the S3 configuration? I suspect that calling the upload method uploadFileWithPath didn’t succeed.

boris-w avatar Sep 24 '24 03:09 boris-w

I am using GCS buckets with S3 interoperability and am encountering an issue with private bucket. My application can upload files to the public bucket but not the private one. I have verified all permissions. The uploads work as expected when using local storage instead of s3. I am unsure how to obtain the relevant logs, but if you can instruct me, I will gladly provide them.

docker-compose.yaml: 
services:
  teable:
    container_name: teable
    image: ghcr.io/teableio/teable:latest
    restart: always
    ports:
      - '3000:3000'
    volumes:
      - teable-data:/app/.assets:rw
    env_file:
      - stack.env # Changed environment file to stack.env for Portainer compatibility.
    environment:
      - NEXT_ENV_IMAGES_ALL_REMOTE=true
    networks:
      - teable
    depends_on:
      teable-db-migrate:
        condition: service_completed_successfully
      teable-cache:
        condition: service_healthy
    healthcheck:
      test: ['CMD', 'curl', '-f', 'http://localhost:3000/health']
      start_period: 5s
      interval: 5s
      timeout: 3s
      retries: 3

  teable-db-migrate:
    container_name: teable-db-migrate
    image: ghcr.io/teableio/teable-db-migrate:latest
    environment:
      - PRISMA_DATABASE_URL=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}
    networks:
      - teable

  teable-cache:
    container_name: teable-cache
    image: redis:7.2.4
    restart: always
    expose:
      - '6379'
    volumes:
      - teable-cache:/data:rw
    networks:
      - teable
    command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD}
    healthcheck:
      test: ['CMD', 'redis-cli', '--raw', 'incr', 'ping']
      interval: 10s
      timeout: 3s
      retries: 3

networks:
  teable:
    name: teable-network

volumes:
  teable-data: {}
  teable-cache: {}


stack.env:
# replace the default password
POSTGRES_PASSWORD=replace_this_password
REDIS_PASSWORD=replace_this_password
SECRET_KEY=replace_this_secret_key

# replace the following with a publicly accessible address
PUBLIC_ORIGIN=
TIMEZONE=Asia/Kolkata

# ---------------------

# Postgres
POSTGRES_HOST=  # Cloud SQL
POSTGRES_PORT=5432
POSTGRES_DB=teable
POSTGRES_USER=postgres

# Redis
REDIS_HOST=teable-cache
REDIS_PORT=6379
REDIS_DB=0

# App
PRISMA_DATABASE_URL=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}
BACKEND_CACHE_PROVIDER=redis
BACKEND_CACHE_REDIS_URI=redis://default:${REDIS_PASSWORD}@${REDIS_HOST}:${REDIS_PORT}/${REDIS_DB}

# Object Storage
BACKEND_STORAGE_PROVIDER=s3
BACKEND_STORAGE_S3_REGION=
BACKEND_STORAGE_S3_ENDPOINT=https://storage.googleapis.com
BACKEND_STORAGE_S3_ACCESS_KEY=
BACKEND_STORAGE_S3_SECRET_KEY=
# public bucket
BACKEND_STORAGE_PUBLIC_BUCKET=
# private bucket
BACKEND_STORAGE_PRIVATE_BUCKET=

aquiveal avatar Dec 16 '24 08:12 aquiveal

I am using GCS buckets with S3 interoperability and am encountering an issue with private bucket. My application can upload files to the public bucket but not the private one. I have verified all permissions. The uploads work as expected when using local storage instead of s3. I am unsure how to obtain the relevant logs, but if you can instruct me, I will gladly provide them.我正在使用具有 S3 互操作性的 GCS 存储桶，但遇到了私有存储桶的问题。我的应用程序可以将文件上传到公共存储桶，但无法上传到私有存储桶。我已验证所有权限。使用本地存储而非 S3 存储时，上传工作正常。我不确定如何获取相关日志，但如果您能指导我，我很乐意提供。

docker-compose.yaml: 
services:
  teable:
    container_name: teable
    image: ghcr.io/teableio/teable:latest
    restart: always
    ports:
      - '3000:3000'
    volumes:
      - teable-data:/app/.assets:rw
    env_file:
      - stack.env # Changed environment file to stack.env for Portainer compatibility.
    environment:
      - NEXT_ENV_IMAGES_ALL_REMOTE=true
    networks:
      - teable
    depends_on:
      teable-db-migrate:
        condition: service_completed_successfully
      teable-cache:
        condition: service_healthy
    healthcheck:
      test: ['CMD', 'curl', '-f', 'http://localhost:3000/health']
      start_period: 5s
      interval: 5s
      timeout: 3s
      retries: 3

  teable-db-migrate:
    container_name: teable-db-migrate
    image: ghcr.io/teableio/teable-db-migrate:latest
    environment:
      - PRISMA_DATABASE_URL=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}
    networks:
      - teable

  teable-cache:
    container_name: teable-cache
    image: redis:7.2.4
    restart: always
    expose:
      - '6379'
    volumes:
      - teable-cache:/data:rw
    networks:
      - teable
    command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD}
    healthcheck:
      test: ['CMD', 'redis-cli', '--raw', 'incr', 'ping']
      interval: 10s
      timeout: 3s
      retries: 3

networks:
  teable:
    name: teable-network

volumes:
  teable-data: {}
  teable-cache: {}

stack.env:
# replace the default password
POSTGRES_PASSWORD=replace_this_password
REDIS_PASSWORD=replace_this_password
SECRET_KEY=replace_this_secret_key

# replace the following with a publicly accessible address
PUBLIC_ORIGIN=
TIMEZONE=Asia/Kolkata

# ---------------------

# Postgres
POSTGRES_HOST=  # Cloud SQL
POSTGRES_PORT=5432
POSTGRES_DB=teable
POSTGRES_USER=postgres

# Redis
REDIS_HOST=teable-cache
REDIS_PORT=6379
REDIS_DB=0

# App
PRISMA_DATABASE_URL=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}
BACKEND_CACHE_PROVIDER=redis
BACKEND_CACHE_REDIS_URI=redis://default:${REDIS_PASSWORD}@${REDIS_HOST}:${REDIS_PORT}/${REDIS_DB}

# Object Storage
BACKEND_STORAGE_PROVIDER=s3
BACKEND_STORAGE_S3_REGION=
BACKEND_STORAGE_S3_ENDPOINT=https://storage.googleapis.com
BACKEND_STORAGE_S3_ACCESS_KEY=
BACKEND_STORAGE_S3_SECRET_KEY=
# public bucket
BACKEND_STORAGE_PUBLIC_BUCKET=
# private bucket
BACKEND_STORAGE_PRIVATE_BUCKET=

This appears to be a CORS (Cross-Origin Resource Sharing) configuration issue in your GCS bucket. You'll need to configure CORS settings in your GCS bucket to allow cross-origin requests from your application domain.

boris-w avatar Apr 30 '25 11:04 boris-w