cloudstack icon indicating copy to clipboard operation
cloudstack copied to clipboard
verified publisher icon apache

Trying to register a template with a long URL results in error

Open fabiomatavelli-f3 opened this issue 2 years ago • 5 comments

ISSUE TYPE
  • Bug Report
COMPONENT NAME
UI, API
CLOUDSTACK VERSION
4.18.1.0
CONFIGURATION

N/A

OS / ENVIRONMENT

Linux Ubuntu 22.04

SUMMARY

Trying to register a template with an S3 signed url (or any other really long URL) fails.

STEPS TO REPRODUCE

In the Cloudstack UI:

  1. Go to Images > Templates
  2. Try to register a template from URL
  3. Use a long URL for the template (around 400~500 characters)
  4. Cloudstack will fail:

Screenshot 2024-02-16 at 11 04 26

EXPECTED RESULTS

Cloudstack to register the template without any problem.

ACTUAL RESULTS

Cloudstack fails with the following error:

com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.cj.jdbc.ClientPreparedStatement: INSERT INTO vm_template (vm_template.id, vm_template.format, vm_template.unique_name, vm_template.name, vm_template.public, vm_template.featured, vm_template.typ
e, vm_template.url, vm_template.hvm, vm_template.bits, vm_template.created, vm_template.account_id, vm_template.checksum, vm_template.display_text, vm_template.enable_password, vm_template.guest_os_id, vm_template.bootable, vm_template.prepopulate, vm_template.cross_zon
es, vm_template.hypervisor_type, vm_template.extractable, vm_template.source_template_id, vm_template.state, vm_template.template_tag, vm_template.uuid, vm_template.sort_key, vm_template.enable_sshkey, vm_template.size, vm_template.update_count, vm_template.updated, vm_
template.dynamically_scalable, vm_template.direct_download, vm_template.parent_template_id, vm_template.deploy_as_is, vm_template.user_data_id, vm_template.user_data_link_policy) VALUES (19344, 'QCOW2', x'31393334342D322D30333162623330652D326166332D336331632D396538362D6
13963393264313235373930', x'7562756E7475323030342D766D2D6B38732D32303234303231353131353434322D333332633961372D666D61746176656C6C692D636F6D70726573732D766D2D696D616765', 0, 0, 'USER', x'68747470733A2F2F6275636B65742E767063652D30333364633236656634316636373565302D396975307
2716A302E73332E65752D776573742D322E767063652E616D617A6F6E6177732E636F6D2F666F726D332D696D6167652D7061636B65722D65752D776573742D322F6F75747075742F766D2F666D61746176656C6C692D636F6D70726573732D766D2D696D6167652F7562756E7475323030342D766D2D6B38732D3230323430323135313135343
4322D333332633961372D666D61746176656C6C692D636F6D70726573732D766D2D696D6167652E71636F77322E677A3F582D416D7A2D416C676F726974686D3D415753342D484D41432D53484132353626582D416D7A2D43726564656E7469616C3D414B49415547514234325257485545504C4C4945253246323032343032313525324665752
D776573742D322532467333253246617773345F7265717565737426582D416D7A2D446174653D3230323430323135543138303332325A26582D416D7A2D457870697265733D39303026582D416D7A2D5369676E6564486561646572733D686F737426582D416D7A2D5369676E61747572653D35643039643562646562663738366536323435626
339303836613938376331613364613332316462313866313266326638303166656463316164633933643233', 0, 64, '2024-02-15 18:04:08', 2, null, x'7562756E7475323030342D766D2D6B38732D32303234303231353131353434322D333332633961372D666D61746176656C6C692D636F6D70726573732D766D2D696D616765'
, 0, 335, 1, 0, 1, 'KVM', 1, null, 'Active', null, x'31356530303839652D353735342D346361322D393264652D323666396233396634623963', 0, 0, null, 0, null, 0, 0, null, 0, null, null)
        at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1488)
        at com.cloud.storage.dao.VMTemplateDaoImpl.addTemplateToZone(VMTemplateDaoImpl.java:467)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy108.addTemplateToZone(Unknown Source)
        at com.cloud.template.TemplateAdapterBase.persistTemplate(TemplateAdapterBase.java:417)
        at com.cloud.template.HypervisorTemplateAdapter.create(HypervisorTemplateAdapter.java:255)
        at com.cloud.template.TemplateManagerImpl.registerTemplate(TemplateManagerImpl.java:355)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy222.registerTemplate(Unknown Source)
        at org.apache.cloudstack.api.command.user.template.RegisterTemplateCmd.execute(RegisterTemplateCmd.java:315)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:782)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:603)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:347)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:154)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:151)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:105)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Data too long for column 'url' at row 1
        at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:104)
        at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
        at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1092)
        ... 86 more

fabiomatavelli-f3 avatar Feb 16 '24 11:02 fabiomatavelli-f3

Thanks for opening your first issue here! Be sure to follow the issue template!

boring-cyborg[bot] avatar Feb 16 '24 11:02 boring-cyborg[bot]

so would changing this to 2048 be enough? an url should not be a blob or such, but some beyond 255 seems reasonable.

@JoaoJandre @sureshanaparti and all others, as this is a 4.18.1 bug (but a DB change), what version do you think we need to implement this for .

DaanHoogland avatar Feb 16 '24 14:02 DaanHoogland

indeed @DaanHoogland , I think even 1024 should be enough. Problem with S3 signed URLs is that they have the tokens/signature in the URL.

fabiomatavelli-f3 avatar Feb 16 '24 14:02 fabiomatavelli-f3

yeah @fabiomatavelli-f3 , I got the problem and the fix is easy but the question is in which version we could fix this. a dirty hack for you would just be ALTER TABLE cloud.vm_template MODIFY COLUMN url VARCHAR(1024), but you will possibly run into issues on upgrades, but this is the statement we would put in any of the upcoming upgrade files.

DaanHoogland avatar Feb 16 '24 15:02 DaanHoogland

yeah @fabiomatavelli-f3 , I got the problem and the fix is easy but the question is in which version we could fix this. a dirty hack for you would just be ALTER TABLE cloud.vm_template MODIFY COLUMN url VARCHAR(1024), but you will possibly run into issues on upgrades, but this is the statement we would put in any of the upcoming upgrade files.

@DaanHoogland the SQL can be executed several times. for example

mysql> ALTER TABLE cloud.vm_template MODIFY COLUMN url VARCHAR(1024) DEFAULT NULL COMMENT 'the url where the template exists externally';
Query OK, 0 rows affected (0.10 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> mysql> ALTER TABLE cloud.vm_template MODIFY COLUMN url VARCHAR(1024) DEFAULT NUENT 'the url where the template exists externally';
Query OK, 0 rows affected (0.14 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ALTER TABLE cloud.vm_template MODIFY COLUMN url VARCHAR(1024) DEFAULT NULL COMMENT 'the url where the template exists externally';
Query OK, 0 rows affected (0.14 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ALTER TABLE cloud.vm_template MODIFY COLUMN url VARCHAR(1024) DEFAULT NULL COMMENT 'the url where the template exists externally';
Query OK, 0 rows affected (0.10 sec)
Records: 0  Duplicates: 0  Warnings: 0

I think it can be put into 4.19.0->4.19.1 upgrade

weizhouapache avatar Feb 16 '24 20:02 weizhouapache

Closing this, as the fix PR got merged.

sureshanaparti avatar Apr 23 '24 07:04 sureshanaparti