Storage pool response improvements #10740

sureshanaparti · 2025-04-16T15:43:33Z

Description

This PR has the following storage pool response improvements (addresses #8731):

adds details parameter to the response
adds capacitybytes parameter to the response, in sync with the capacityiops response parameter and createStoragePool cmd request parameter / db column in storage_pool table. (existing disksizetotal parameter in the storage pool response can be deprecated)
persists the capacityBytes and capacityIops in the storage_pool_details table while creating storage pool as well, for consistency - as these are updated with during update storage pool

capacitybytes in the storage_pool table is updated during the following events:

host connect/re-connect, using ModifyStoragePoolAnswer (from the capacityBytes in StoragePoolInfo)
storage stats collector when the capacityBytes of the pool is not same as the one collected
update storage pool cmd (and this is updated/sync-ed again by storage stats collector)

and update storage pool cmd persists the capacityBytes and capacityIops in the storage_pool_details table, and these values are not updated/synced (so, for consistency, create storage pool also persists the capacityBytes and capacityIops in the storage_pool_details table).

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Tested create storage pool (NFS) with capacitybytes, capacityiops and some details.

Create storage pool (capacitybytes synced with the details from ModifyStoragePoolAnswer, detail parameter doesn't change) =>

(cmk) 🐱 > create storagepool scope=cluster zoneid=f3cc4c59-ceef-4471-aadc-83eea0bada6c podid=0d4a256f-214c-4700-b8f2-ced24959375b clusterid=e3e8fcfd-f9a9-43e5-a569-3965f8097067 name=pri4 provider=DefaultPrimary url=nfs://10.0.32.4/acs/primary/ref-trl-8357-k-Mol8-suresh-anaparti/pri4 capacitybytes=107374182400 capacityiops=10000 tags=pri3 details[0].key1=value1 details[0].key2=value2 details[0].key3=value3
{
  "storagepool": {
    "capacitybytes": 2197949513728,
    "capacityiops": 10000,
    "clusterid": "e3e8fcfd-f9a9-43e5-a569-3965f8097067",
    "clustername": "p1-c1",
    "created": "2025-04-16T19:24:55+0000",
    "details": {
      "capacityBytes": "107374182400",
      "capacityIops": "10000",
      "key1": "value1",
      "key2": "value2",
      "key3": "value3"
    },
    "disksizeallocated": 0,
    "disksizetotal": 2197949513728,
    "hasannotations": false,
    "hypervisor": "KVM",
    "id": "5de042fd-8f29-3b0e-a98d-f351524da85f",
    "ipaddress": "10.0.32.4",
    "istagarule": false,
    "managed": false,
    "name": "pri4",
    "overprovisionfactor": "2.0",
    "path": "/acs/primary/ref-trl-8357-k-Mol8-suresh-anaparti/pri4",
    "podid": "0d4a256f-214c-4700-b8f2-ced24959375b",
    "podname": "Pod1",
    "provider": "DefaultPrimary",
    "scope": "CLUSTER",
    "state": "Up",
    "tags": "pri3",
    "type": "NetworkFilesystem",
    "zoneid": "f3cc4c59-ceef-4471-aadc-83eea0bada6c",
    "zonename": "ref-trl-8357-k-Mol8-suresh-anaparti"
  }
}

List storage pool =>

(cmk) 🐱 > list storagepools id=5de042fd-8f29-3b0e-a98d-f351524da85f 
{
  "count": 1,
  "storagepool": [
    {
      "capacitybytes": 2197949513728,
      "capacityiops": 10000,
      "clusterid": "e3e8fcfd-f9a9-43e5-a569-3965f8097067",
      "clustername": "p1-c1",
      "created": "2025-04-16T19:24:55+0000",
      "details": {
        "capacityBytes": "107374182400",
        "capacityIops": "10000",
        "key1": "value1",
        "key2": "value2",
        "key3": "value3"
      },
      "disksizeallocated": 0,
      "disksizetotal": 2197949513728,
      "hasannotations": false,
      "hypervisor": "KVM",
      "id": "5de042fd-8f29-3b0e-a98d-f351524da85f",
      "ipaddress": "10.0.32.4",
      "istagarule": false,
      "managed": false,
      "name": "pri4",
      "overprovisionfactor": "2.0",
      "path": "/acs/primary/ref-trl-8357-k-Mol8-suresh-anaparti/pri4",
      "podid": "0d4a256f-214c-4700-b8f2-ced24959375b",
      "podname": "Pod1",
      "provider": "DefaultPrimary",
      "scope": "CLUSTER",
      "state": "Up",
      "storagecapabilities": {
        "VOLUME_SNAPSHOT_QUIESCEVM": "false"
      },
      "tags": "pri3",
      "type": "NetworkFilesystem",
      "zoneid": "f3cc4c59-ceef-4471-aadc-83eea0bada6c",
      "zonename": "ref-trl-8357-k-Mol8-suresh-anaparti"
    }
  ]
}

Update storage pool (capacitybytes and capacityiops, actual and detail parameters updated) =>

(cmk) 🐱 > update storagepool id=5de042fd-8f29-3b0e-a98d-f351524da85f capacitybytes=107374182400 capacityiops=20000
{
  "storagepool": {
    "capacitybytes": 107374182400,
    "capacityiops": 20000,
    "clusterid": "e3e8fcfd-f9a9-43e5-a569-3965f8097067",
    "clustername": "p1-c1",
    "created": "2025-04-16T19:24:55+0000",
    "details": {
      "capacityBytes": "107374182400",
      "capacityIops": "20000",
      "key1": "value1",
      "key2": "value2",
      "key3": "value3"
    },
    "disksizeallocated": 0,
    "disksizetotal": 107374182400,
    "disksizeused": 1765468536832,
    "hasannotations": false,
    "hypervisor": "KVM",
    "id": "5de042fd-8f29-3b0e-a98d-f351524da85f",
    "ipaddress": "10.0.32.4",
    "istagarule": false,
    "managed": false,
    "name": "pri4",
    "overprovisionfactor": "2.0",
    "path": "/acs/primary/ref-trl-8357-k-Mol8-suresh-anaparti/pri4",
    "podid": "0d4a256f-214c-4700-b8f2-ced24959375b",
    "podname": "Pod1",
    "provider": "DefaultPrimary",
    "scope": "CLUSTER",
    "state": "Up",
    "tags": "pri3",
    "type": "NetworkFilesystem",
    "zoneid": "f3cc4c59-ceef-4471-aadc-83eea0bada6c",
    "zonename": "ref-trl-8357-k-Mol8-suresh-anaparti"
  }
}

List storage pool (capacitybytes doesn't sync-ed with stats collector yet) =>

(cmk) 🐱 > list storagepools id=5de042fd-8f29-3b0e-a98d-f351524da85f 
{
  "count": 1,
  "storagepool": [
    {
      "capacitybytes": 107374182400,
      "capacityiops": 20000,
      "clusterid": "e3e8fcfd-f9a9-43e5-a569-3965f8097067",
      "clustername": "p1-c1",
      "created": "2025-04-16T19:24:55+0000",
      "details": {
        "capacityBytes": "107374182400",
        "capacityIops": "20000",
        "key1": "value1",
        "key2": "value2",
        "key3": "value3"
      },
      "disksizeallocated": 0,
      "disksizetotal": 107374182400,
      "disksizeused": 1765468536832,
      "hasannotations": false,
      "hypervisor": "KVM",
      "id": "5de042fd-8f29-3b0e-a98d-f351524da85f",
      "ipaddress": "10.0.32.4",
      "istagarule": false,
      "managed": false,
      "name": "pri4",
      "overprovisionfactor": "2.0",
      "path": "/acs/primary/ref-trl-8357-k-Mol8-suresh-anaparti/pri4",
      "podid": "0d4a256f-214c-4700-b8f2-ced24959375b",
      "podname": "Pod1",
      "provider": "DefaultPrimary",
      "scope": "CLUSTER",
      "state": "Up",
      "storagecapabilities": {
        "VOLUME_SNAPSHOT_QUIESCEVM": "false"
      },
      "tags": "pri3",
      "type": "NetworkFilesystem",
      "zoneid": "f3cc4c59-ceef-4471-aadc-83eea0bada6c",
      "zonename": "ref-trl-8357-k-Mol8-suresh-anaparti"
    }
  ]
}

List storage pool (capacitybytes sync-ed with stats collector, detail parameter doesn't change) =>

(cmk) 🐱 > list storagepools id=5de042fd-8f29-3b0e-a98d-f351524da85f 
{
  "count": 1,
  "storagepool": [
    {
      "capacitybytes": 2197949513728,
      "capacityiops": 20000,
      "clusterid": "e3e8fcfd-f9a9-43e5-a569-3965f8097067",
      "clustername": "p1-c1",
      "created": "2025-04-16T19:24:55+0000",
      "details": {
        "capacityBytes": "107374182400",
        "capacityIops": "20000",
        "key1": "value1",
        "key2": "value2",
        "key3": "value3"
      },
      "disksizeallocated": 0,
      "disksizetotal": 2197949513728,
      "disksizeused": 1765912084480,
      "hasannotations": false,
      "hypervisor": "KVM",
      "id": "5de042fd-8f29-3b0e-a98d-f351524da85f",
      "ipaddress": "10.0.32.4",
      "istagarule": false,
      "managed": false,
      "name": "pri4",
      "overprovisionfactor": "2.0",
      "path": "/acs/primary/ref-trl-8357-k-Mol8-suresh-anaparti/pri4",
      "podid": "0d4a256f-214c-4700-b8f2-ced24959375b",
      "podname": "Pod1",
      "provider": "DefaultPrimary",
      "scope": "CLUSTER",
      "state": "Up",
      "storagecapabilities": {
        "VOLUME_SNAPSHOT_QUIESCEVM": "false"
      },
      "tags": "pri3",
      "type": "NetworkFilesystem",
      "zoneid": "f3cc4c59-ceef-4471-aadc-83eea0bada6c",
      "zonename": "ref-trl-8357-k-Mol8-suresh-anaparti"
    }
  ]
}

DB =>

mysql> SELECT * FROM cloud.storage_pool WHERE name = 'pri4'\G;
*************************** 1. row ***************************
                   id: 4
                 name: pri4
                 uuid: 5de042fd-8f29-3b0e-a98d-f351524da85f
            pool_type: NetworkFilesystem
                 port: 2049
       data_center_id: 1
               pod_id: 1
           cluster_id: 1
           used_bytes: 1765912084480
       capacity_bytes: 2197949513728
         host_address: 10.0.32.4
            user_info: NULL
                 path: /acs/primary/ref-trl-8357-k-Mol8-suresh-anaparti/pri4
              created: 2025-04-16 19:24:55
              removed: NULL
          update_time: 2025-04-16 19:26:47
               status: Up
storage_provider_name: DefaultPrimary
                scope: CLUSTER
           hypervisor: KVM
              managed: 0
        capacity_iops: 20000
               parent: 0
1 row in set (0.00 sec)

mysql> SELECT * FROM cloud.storage_pool_details WHERE pool_id = 4;
+----+---------+---------------+--------------+---------+
| id | pool_id | name          | value        | display |
+----+---------+---------------+--------------+---------+
| 14 |       4 | key1          | value1       |       1 |
| 15 |       4 | capacityIops  | 20000        |       1 |
| 16 |       4 | key2          | value2       |       1 |
| 17 |       4 | key3          | value3       |       1 |
| 18 |       4 | capacityBytes | 107374182400 |       1 |
+----+---------+---------------+--------------+---------+
5 rows in set (0.00 sec)

How did you try to break this feature and the system with this change?

… update capacityBytes and capacityIops if applicable while creating storage pool

…th the capacityiops response parameter and createStoragePool cmd request parameter (existing disksizetotal parameter in the storage pool response can be deprecated)

sureshanaparti · 2025-04-16T15:45:57Z

@blueorangutan package

blueorangutan · 2025-04-16T15:48:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2025-04-16T15:52:24Z

Codecov Report

Attention: Patch coverage is 7.57576% with 61 lines in your changes missing coverage. Please review.

Project coverage is 15.16%. Comparing base (f13cf59) to head (f150493).
Report is 4 commits behind head on 4.19.

Files with missing lines	Patch %	Lines
...ava/com/cloud/upgrade/dao/Upgrade41920to41930.java	15.38%	22 Missing ⚠️
...cycle/CloudStackPrimaryDataStoreLifeCycleImpl.java	0.00%	18 Missing ⚠️
...e/cloudstack/api/response/StoragePoolResponse.java	0.00%	12 Missing ⚠️
.../storage/datastore/db/PrimaryDataStoreDaoImpl.java	0.00%	4 Missing ⚠️
...om/cloud/api/query/dao/StoragePoolJoinDaoImpl.java	0.00%	4 Missing ⚠️
...orage/volume/datastore/PrimaryDataStoreHelper.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.19   #10740      +/-   ##
============================================
- Coverage     15.17%   15.16%   -0.01%     
- Complexity    11332    11334       +2     
============================================
  Files          5415     5416       +1     
  Lines        474893   474976      +83     
  Branches      57920    57935      +15     
============================================
+ Hits          72046    72051       +5     
- Misses       394792   394871      +79     
+ Partials       8055     8054       -1

Flag	Coverage Δ
uitests	`4.28% <ø> (ø)`
unittests	`15.89% <7.57%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blueorangutan · 2025-04-16T17:05:12Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13079

…s table while creating storage pool as well, for consistency - as these are updated with during update storage pool

sureshanaparti · 2025-04-17T11:24:00Z

@blueorangutan package

blueorangutan · 2025-04-17T11:26:05Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-04-17T12:42:27Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13094

sureshanaparti · 2025-04-17T13:26:13Z

@blueorangutan test

blueorangutan · 2025-04-17T13:28:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-04-18T03:40:47Z

[SF] Trillian test result (tid-13037)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 48226 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10740-t13037-kvm-ol8.zip
Smoke tests completed. 133 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

sureshanaparti added 3 commits April 16, 2025 16:23

Return details of the storage pool in the response including url, and…

6adf4d3

… update capacityBytes and capacityIops if applicable while creating storage pool

Added capacitybytes parameter to the storage pool response in sync wi…

333a395

…th the capacityiops response parameter and createStoragePool cmd request parameter (existing disksizetotal parameter in the storage pool response can be deprecated)

Don't keep url in details

d676cef

boring-cyborg bot added component:api component:primary-storage component:storage labels Apr 16, 2025

sureshanaparti linked an issue Apr 16, 2025 that may be closed by this pull request

listStoragePools return all fields #8731

Open

Persist the capacityBytes and capacityIops in the storage_pool_detail…

f150493

…s table while creating storage pool as well, for consistency - as these are updated with during update storage pool

DaanHoogland added component:upgrade status:needs-testing labels Apr 17, 2025

sureshanaparti added this to the 4.19.3 milestone Apr 17, 2025

sureshanaparti marked this pull request as ready for review April 17, 2025 11:24

sureshanaparti mentioned this pull request Apr 17, 2025

listStoragePools return all fields #8731

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Storage pool response improvements #10740

Storage pool response improvements #10740

sureshanaparti commented Apr 16, 2025 •

edited

Loading

sureshanaparti commented Apr 16, 2025

blueorangutan commented Apr 16, 2025

codecov bot commented Apr 16, 2025 •

edited

Loading

blueorangutan commented Apr 16, 2025

sureshanaparti commented Apr 17, 2025

blueorangutan commented Apr 17, 2025

blueorangutan commented Apr 17, 2025

sureshanaparti commented Apr 17, 2025

blueorangutan commented Apr 17, 2025

blueorangutan commented Apr 18, 2025

Storage pool response improvements #10740

Are you sure you want to change the base?

Storage pool response improvements #10740

Conversation

sureshanaparti commented Apr 16, 2025 • edited Loading

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

sureshanaparti commented Apr 16, 2025

blueorangutan commented Apr 16, 2025

codecov bot commented Apr 16, 2025 • edited Loading

Codecov Report

blueorangutan commented Apr 16, 2025

sureshanaparti commented Apr 17, 2025

blueorangutan commented Apr 17, 2025

blueorangutan commented Apr 17, 2025

sureshanaparti commented Apr 17, 2025

blueorangutan commented Apr 17, 2025

blueorangutan commented Apr 18, 2025

sureshanaparti commented Apr 16, 2025 •

edited

Loading

codecov bot commented Apr 16, 2025 •

edited

Loading