Ceph RGW Object Storage couldn't be added (Invalid credentials or URL)

[hanisirfan]

ISSUE TYPE

• Bug Report
• Improvement Request
• Enhancement Request
• Feature Idea
• Documentation Report
• Other

COMPONENT NAME

Infrastructure > Object Store

CLOUDSTACK VERSION

4.20.0.0

CONFIGURATION

OS / ENVIRONMENT

Ceph version: 19.2.0 squid (stable)

SUMMARY

STEPS TO REPRODUCE

1. Create a Object Gateway user on Ceph Dashboard. The user is added under a Tenant.
2. Try to add object store using the Ceph provider.

EXPECTED RESULTS

Being able to add a Ceph object store to my infrastructure.

ACTUAL RESULTS

Failed with error:

Request failed. (530)
Failed to add object store: Error while initializing Ceph RGW Object Store. Invalid credentials or URL

ERROR LOG

2024-12-05 05:54:03,154 DEBUG [c.c.a.ApiServlet] (qtp2038105753-397:[ctx-d6792894]) (logid:de1e0941) ===START===  10.0.113.1 -- GET  name=Ceph+Test+1&provider=Ceph&url=https:%2F%2F<URL_HERE>%2F&command=addObjectStoragePool&response=json&sessionkey=_IgdzVMijOi6635kzbViqZtkR2A
2024-12-05 05:54:03,154 DEBUG [c.c.a.ApiServlet] (qtp2038105753-397:[ctx-d6792894]) (logid:de1e0941) Two factor authentication is already verified for the user 2, so skipping
2024-12-05 05:54:03,161 DEBUG [c.c.a.ApiServer] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"c2a81095-5469-11ef-8cfa-005056a0391c"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-12-05 05:54:03,163 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-12-05 05:54:03,163 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-12-05 05:54:03,176 INFO  [o.a.c.s.d.l.CephObjectStoreLifeCycleImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Attempting to connect to Ceph RGW at https://<URL_HERE>/ with access key <ACCESS_KEY_HERE>
2024-12-05 05:54:03,199 DEBUG [o.a.c.s.d.l.CephObjectStoreLifeCycleImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Error while initializing Ceph RGW Object Store: Cannot invoke "java.util.List.stream()" because the return value of "org.twonote.rgwadmin4j.impl.RgwAdminImpl.listBucketInfo()" is null
2024-12-05 05:54:03,199 DEBUG [c.c.s.StorageManagerImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Failed to add object store: Error while initializing Ceph RGW Object Store. Invalid credentials or URL java.lang.RuntimeException: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
        at org.apache.cloudstack.storage.datastore.lifecycle.CephObjectStoreLifeCycleImpl.initialize(CephObjectStoreLifeCycleImpl.java:88)
        at com.cloud.storage.StorageManagerImpl.discoverObjectStore(StorageManagerImpl.java:4140)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy130.discoverObjectStore(Unknown Source)
        at org.apache.cloudstack.api.command.admin.storage.AddObjectStoragePoolCmd.execute(AddObjectStoragePoolCmd.java:115)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:831)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:652)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:354)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:157)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:154)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:108)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:840)

2024-12-05 05:54:03,200 WARN  [o.a.c.m.w.WebhookServiceImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Skipping delivering event [ID: null, description: {"event":"OBJECT.STORE.CREATE","status":"Completed"}] to any webhook as account ID is missing
2024-12-05 05:54:03,202 WARN  [o.a.c.f.e.EventDistributorImpl] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Failed to publish event [category: ActionEvent, type: OBJECT.STORE.CREATE] on bus webhookEventBus
2024-12-05 05:54:03,213 ERROR [o.a.c.a.c.a.s.AddObjectStoragePoolCmd] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Exception: com.cloud.utils.exception.CloudRuntimeException: Failed to add object store: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
        at com.cloud.storage.StorageManagerImpl.discoverObjectStore(StorageManagerImpl.java:4145)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy130.discoverObjectStore(Unknown Source)
        at org.apache.cloudstack.api.command.admin.storage.AddObjectStoragePoolCmd.execute(AddObjectStoragePoolCmd.java:115)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:831)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:652)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:354)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:157)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:154)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:108)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.RuntimeException: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
        at org.apache.cloudstack.storage.datastore.lifecycle.CephObjectStoreLifeCycleImpl.initialize(CephObjectStoreLifeCycleImpl.java:88)
        at com.cloud.storage.StorageManagerImpl.discoverObjectStore(StorageManagerImpl.java:4140)
        ... 67 more

2024-12-05 05:54:03,213 INFO  [c.c.a.ApiServer] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) Failed to add object store: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
2024-12-05 05:54:03,214 INFO  [c.c.a.ApiServlet] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) (userId=2 accountId=2 sessionId=node0a1m8rqhb8vwl17sdi3x5937fp3) 10.0.113.1 -- GET name=Ceph+Test+1&provider=Ceph&url=https:%2F%2F<URL_HERE>%2F&command=addObjectStoragePool&response=json&sessionkey=_IgdzVMijOi6635kzbViqZtkR2A 530 Failed to add object store: Error while initializing Ceph RGW Object Store. Invalid credentials or URL
2024-12-05 05:54:03,214 DEBUG [c.c.a.ApiServlet] (qtp2038105753-397:[ctx-d6792894, ctx-4ac36717]) (logid:de1e0941) ===END===  10.0.113.1 -- GET  name=Ceph+Test+1&provider=Ceph&url=https:%2F%2F<URL_HERE>%2F&command=addObjectStoragePool&response=json&sessionkey=_IgdzVMijOi6635kzbViqZtkR2A

It might be something related to org.twonote.rgwadmin4j.impl.RgwAdminImpl.listBucketInfo(). Note that I'm inserting the base URL of my Ceph RGW endpoint. The endpoint should support both path and virtual host based URL (tested with WinSCP). No bucket created at all for the user.

[hanisirfan]

Just created a new RGW user using the Dashboard, but this time it's not under a tenant. Still the same error 😞

[abh1sar]

Just to cover all bases @hanisirfan can you please verify these :

1. url contains the correct radosgw port?
2. the user should be a System User.

[hanisirfan]

Just to cover all bases @hanisirfan can you please verify these :

1. url contains the correct radosgw port?

2. the user should be a System User.

I can add the Ceph RGW as object store now after I changed the user to System User. But when creating a bucket, I've hit with this error:

Failed to create bucket.
Failed to create bucket with name: cs-bucket1. The specified location-constraint is not valid (Service: Amazon S3; Status Code: 400; Error Code: InvalidLocationConstraint; Request ID: tx00000e9e4bf6e2a1861d8-00675d101d-124525-my-cx1; S3 Extended Request ID: 124525-my-cx1-my; Proxy: null)

PS: I created the user under a tenant.

[abh1sar]

@hanisirfan can you please try by creating the user not under the tenant.

[hanisirfan]

@hanisirfan can you please try by creating the user not under the tenant.

Created a system user that's not under a tenant (you can see the Tenant field is empty). Able to add it as Object Storage in CloudStack.

Tried to create a simple bucket but failed with the same error as before.

Failed to create bucket with name: bucket123. The specified location-constraint is not valid (Service: Amazon S3; Status Code: 400; Error Code: InvalidLocationConstraint; Request ID: tx000003165e926edc5103d-006764d195-181522-my-cx1; S3 Extended Request ID: 181522-my-cx1-my; Proxy: null)

[abh1sar]

@hanisirfan do you see any more info about the error in the management-server.log?
Can you verify if a new user has been created for the cloudstack account?
Are you able to create a bucket from other user accounts using another object store client?

[hanisirfan]

@abh1sar

I can verify that a Ceph RGW user has been created from Ceph Dashboard and it can create bucket and upload files using another S3 client, in my case, WinSCP. Both virtual host and path connection type works.

Here is the log from my latest:

2025-01-10 03:58:31,540 DEBUG [c.c.a.ApiServlet] (qtp2038105753-111314:[ctx-97070350]) (logid:8889d81c) ===START===  10.0.113.1 -- GET  name=bucket1&objectstorageid=2a25093c-a5ac-434c-bc86-63650a31e452&command=createBucket&response=json&sessionkey=STschzfMKzM-ClGn248Ikm6q8e4
2025-01-10 03:58:31,540 DEBUG [c.c.a.ApiServlet] (qtp2038105753-111314:[ctx-97070350]) (logid:8889d81c) Two factor authentication is already verified for the user 2, so skipping
2025-01-10 03:58:31,544 DEBUG [c.c.a.ApiServer] (qtp2038105753-111314:[ctx-97070350, ctx-2f8c511e]) (logid:8889d81c) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"c2a81095-5469-11ef-8cfa-005056a0391c"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-01-10 03:58:31,545 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp2038105753-111314:[ctx-97070350, ctx-2f8c511e]) (logid:8889d81c) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-01-10 03:58:31,545 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp2038105753-111314:[ctx-97070350, ctx-2f8c511e]) (logid:8889d81c) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-01-10 03:58:31,550 DEBUG [o.a.c.s.d.d.CephObjectStoreDriverImpl] (qtp2038105753-111314:[ctx-97070350, ctx-2f8c511e]) (logid:8889d81c) Attempting to create Ceph RGW user for account admin with UUID c2a81095-5469-11ef-8cfa-005056a0391c
2025-01-10 03:58:31,588 INFO  [o.a.c.s.d.d.CephObjectStoreDriverImpl] (qtp2038105753-111314:[ctx-97070350, ctx-2f8c511e]) (logid:8889d81c) User already exists in Ceph RGW: c2a81095-5469-11ef-8cfa-005056a0391c
2025-01-10 03:58:31,670 DEBUG [c.c.a.ApiServer] (qtp2038105753-111314:[ctx-97070350, ctx-2f8c511e]) (logid:8889d81c) Retrieved cmdEventType from job info: BUCKET.CREATE
2025-01-10 03:58:31,671 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-75:[ctx-29b1d038, job-2637]) (logid:cab09a2f) Add job-2637 into job monitoring
2025-01-10 03:58:31,674 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-75:[ctx-29b1d038, job-2637]) (logid:bd6c8395) Executing AsyncJobVO: {id:2637, userId: 2, accountId: 2, instanceType: Bucket, instanceId: 8, cmd: org.apache.cloudstack.api.command.user.bucket.CreateBucketCmd, cmdInfo: {"objectstorageid":"2a25093c-a5ac-434c-bc86-63650a31e452","response":"json","ctxUserId":"2","sessionkey":"STschzfMKzM-ClGn248Ikm6q8e4","name":"bucket1","httpmethod":"GET","ctxStartEventId":"33697","id":"8","ctxDetails":"{\"interface org.apache.cloudstack.storage.object.ObjectStore\":\"2a25093c-a5ac-434c-bc86-63650a31e452\",\"interface org.apache.cloudstack.storage.object.Bucket\":\"f6c04da4-347c-421b-8df9-8b15be1f6c74\"}","ctxAccountId":"2","uuid":"f6c04da4-347c-421b-8df9-8b15be1f6c74","cmdEventType":"BUCKET.CREATE"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345050723509, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2025-01-10 03:58:31,674 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp2038105753-111314:[ctx-97070350, ctx-2f8c511e]) (logid:8889d81c) submit async job-2637, details: AsyncJobVO: {id:2637, userId: 2, accountId: 2, instanceType: Bucket, instanceId: 8, cmd: org.apache.cloudstack.api.command.user.bucket.CreateBucketCmd, cmdInfo: {"objectstorageid":"2a25093c-a5ac-434c-bc86-63650a31e452","response":"json","ctxUserId":"2","sessionkey":"STschzfMKzM-ClGn248Ikm6q8e4","name":"bucket1","httpmethod":"GET","ctxStartEventId":"33697","id":"8","ctxDetails":"{\"interface org.apache.cloudstack.storage.object.ObjectStore\":\"2a25093c-a5ac-434c-bc86-63650a31e452\",\"interface org.apache.cloudstack.storage.object.Bucket\":\"f6c04da4-347c-421b-8df9-8b15be1f6c74\"}","ctxAccountId":"2","uuid":"f6c04da4-347c-421b-8df9-8b15be1f6c74","cmdEventType":"BUCKET.CREATE"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345050723509, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2025-01-10 03:58:31,675 INFO  [c.c.a.ApiServlet] (qtp2038105753-111314:[ctx-97070350, ctx-2f8c511e]) (logid:8889d81c) (userId=2 accountId=2 sessionId=node0lc2ensl6j4111p8hvhgma7q5631) 10.0.113.1 -- GET name=bucket1&objectstorageid=2a25093c-a5ac-434c-bc86-63650a31e452&command=createBucket&response=json&sessionkey=STschzfMKzM-ClGn248Ikm6q8e4 200 {"createbucketresponse":{"id":"f6c04da4-347c-421b-8df9-8b15be1f6c74","jobid":"bd6c8395-6624-47ef-ad42-99f5781f034a"}}
2025-01-10 03:58:31,675 DEBUG [c.c.a.ApiServlet] (qtp2038105753-111314:[ctx-97070350, ctx-2f8c511e]) (logid:8889d81c) ===END===  10.0.113.1 -- GET  name=bucket1&objectstorageid=2a25093c-a5ac-434c-bc86-63650a31e452&command=createBucket&response=json&sessionkey=STschzfMKzM-ClGn248Ikm6q8e4
2025-01-10 03:58:31,710 DEBUG [o.a.c.s.o.BucketApiServiceImpl] (API-Job-Executor-75:[ctx-29b1d038, job-2637, ctx-9834dac5]) (logid:bd6c8395) Failed to create bucket with name: bucket1 com.cloud.utils.exception.CloudRuntimeException: The specified location-constraint is not valid (Service: Amazon S3; Status Code: 400; Error Code: InvalidLocationConstraint; Request ID: tx00000fa3f15f49e7c8aa5-0067809ae7-124525-my-cx1; S3 Extended Request ID: 124525-my-cx1-my; Proxy: null)
        at org.apache.cloudstack.storage.datastore.driver.CephObjectStoreDriverImpl.createBucket(CephObjectStoreDriverImpl.java:120)
        at org.apache.cloudstack.storage.object.store.ObjectStoreImpl.createBucket(ObjectStoreImpl.java:107)
        at org.apache.cloudstack.storage.object.BucketApiServiceImpl.createBucket(BucketApiServiceImpl.java:147)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy434.createBucket(Unknown Source)
        at org.apache.cloudstack.api.command.user.bucket.CreateBucketCmd.execute(CreateBucketCmd.java:188)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified location-constraint is not valid (Service: Amazon S3; Status Code: 400; Error Code: InvalidLocationConstraint; Request ID: tx00000fa3f15f49e7c8aa5-0067809ae7-124525-my-cx1; S3 Extended Request ID: 124525-my-cx1-my; Proxy: null), S3 Extended Request ID: 124525-my-cx1-my
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
        at com.amazonaws.services.s3.AmazonS3Client.createBucket(AmazonS3Client.java:1131)
        at com.amazonaws.services.s3.AmazonS3Client.createBucket(AmazonS3Client.java:1059)
        at org.apache.cloudstack.storage.datastore.driver.CephObjectStoreDriverImpl.createBucket(CephObjectStoreDriverImpl.java:109)
        ... 32 more

2025-01-10 03:58:31,714 WARN  [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-75:[ctx-29b1d038, job-2637, ctx-9834dac5]) (logid:bd6c8395) Skipping delivering event [ID: null, description: {"details":"Bucket Id: f6c04da4-347c-421b-8df9-8b15be1f6c74","event":"BUCKET.CREATE","status":"Completed"}] to any webhook as account ID is missing
2025-01-10 03:58:31,714 WARN  [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-75:[ctx-29b1d038, job-2637, ctx-9834dac5]) (logid:bd6c8395) Failed to publish event [category: ActionEvent, type: BUCKET.CREATE] on bus webhookEventBus
2025-01-10 03:58:31,722 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-75:[ctx-29b1d038, job-2637]) (logid:bd6c8395) Complete async job-2637, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create bucket with name: bucket1. The specified location-constraint is not valid (Service: Amazon S3; Status Code: 400; Error Code: InvalidLocationConstraint; Request ID: tx00000fa3f15f49e7c8aa5-0067809ae7-124525-my-cx1; S3 Extended Request ID: 124525-my-cx1-my; Proxy: null)"}
2025-01-10 03:58:31,723 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-75:[ctx-29b1d038, job-2637]) (logid:bd6c8395) Publish async job-2637 complete on message bus
2025-01-10 03:58:31,723 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-75:[ctx-29b1d038, job-2637]) (logid:bd6c8395) Wake up jobs related to job-2637
2025-01-10 03:58:31,723 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-75:[ctx-29b1d038, job-2637]) (logid:bd6c8395) Update db status for job-2637
2025-01-10 03:58:31,724 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-75:[ctx-29b1d038, job-2637]) (logid:bd6c8395) Wake up jobs joined with job-2637 and disjoin all subjobs created from job- 2637
2025-01-10 03:58:31,729 DEBUG [c.c.a.ApiServer] (API-Job-Executor-75:[ctx-29b1d038, job-2637]) (logid:bd6c8395) Retrieved cmdEventType from job info: BUCKET.CREATE
2025-01-10 03:58:31,730 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-75:[ctx-29b1d038, job-2637]) (logid:bd6c8395) Done executing org.apache.cloudstack.api.command.user.bucket.CreateBucketCmd for job-2637
2025-01-10 03:58:31,730 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-75:[ctx-29b1d038, job-2637]) (logid:bd6c8395) Remove job-2637 from job monitoring

The error on the CloudStack UI.

Failed to create bucket with name: bucket1. The specified location-constraint is not valid (Service: Amazon S3; Status Code: 400; Error Code: InvalidLocationConstraint; Request ID: tx00000fa3f15f49e7c8aa5-0067809ae7-124525-my-cx1; S3 Extended Request ID: 124525-my-cx1-my; Proxy: null)

[hanisirfan]

Might the issue is similar with this?

localstack/localstack#8000

[hanisirfan]

@abh1sar I don't know if I read your previous question wrongly, but from what I understand, you asked whether I created a Ceph RGW user and operations can work with other S3 client. My answer was yes.

An addition to this, I just looked at my Ceph Dashboard and there is an unknown user. Is this something that is related to CloudStack Ceph implementation?

[abh1sar]

@hanisirfan
Cloudstack creates a user in ceph for every account in cloudstack that creates a bucket. This is a cloudstack created account.
So seems like this error is seen only by this user and not the CEPH RGW user.

[hanisirfan]

@abh1sar I'm sorry but I don't understand your explanation. Can you please elaborate? I'm kinda new to this. 🙏