Create new Instance from VM backup #10140

abh1sar · 2024-12-25T05:12:50Z

Doc PR: apache/cloudstack-documentation#474

Description

This PR adds the ability to create a new Instance from a VM backup for Dummy, NAS and Veeam backup Providers.
This will still work if the original Instance used to create the backup was expunged.

New API

createVmFromBackup extends deployVIrtualMachine
new argument : backupId
User can just specify the backupId and a new instance will be created by getting the VM and volumes' configuration stored in backup_details table.
User can also choose to configure the details themselves using the parameters present in deployVIrtualMachine cmd

UI

New action button in backups list view - 'Create Instance from Backup'
User can choose to directly create the VM or configure VM parameters before creating it.

DB changes

New backup_details table
to store vm config data such as templateUuid, serviceOfferingUuid, hypervisor, networkUuids
to store volumes config data such as diskOfferingUuids, size, miniops and maxiops for custom disk offerings.

Other Changes

Added new parameter Map datadisksdetails to deployVirtualMachine api to create multiple data volumes at the time of instance creation.

Plugins related changes

NAS and Veeam plugin changes to restore backup to a new VM.

Other Miscellaneous changes

Add Name and description column to backup
Show Interval type in UI for backups
Support cephFS for NAS backup provider
Add Backup Storage Capacity Tracking and Alerts Infrastructure
Backup Storage (used and available) is shown on the Capacity dashboard and tracks the alerts if supported by the plugin
(Currently only NAS BnR plugin supports it)
Add Object Storage Capacity Tracking and Alerts Infrastructure
Object Storage (allocated and available) is shown on the Capacity dashboard and tracked for alerts.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2024-12-25T05:20:17Z

Codecov Report

Attention: Patch coverage is 33.05589% with 1126 lines in your changes missing coverage. Please review.

Project coverage is 16.45%. Comparing base (f275c28) to head (e2fa0be).
Report is 50 commits behind head on main.

Files with missing lines	Patch %	Lines
...loudstack/api/command/user/vm/BaseDeployVMCmd.java	15.80%	289 Missing and 36 partials ⚠️
.../src/main/java/com/cloud/vm/UserVmManagerImpl.java	30.34%	133 Missing and 30 partials ⚠️
...rg/apache/cloudstack/backup/BackupManagerImpl.java	75.17%	39 Missing and 30 partials ⚠️
...ack/api/command/user/vm/CreateVMFromBackupCmd.java	6.45%	58 Missing ⚠️
...rg/apache/cloudstack/backup/dao/BackupDaoImpl.java	37.33%	41 Missing and 6 partials ⚠️
...a/org/apache/cloudstack/backup/BackupDetailVO.java	18.91%	29 Missing and 1 partial ⚠️
...rg/apache/cloudstack/backup/NASBackupProvider.java	50.00%	27 Missing and 3 partials ⚠️
.../apache/cloudstack/backup/VeeamBackupProvider.java	6.25%	30 Missing ⚠️
...che/cloudstack/backup/NetworkerBackupProvider.java	0.00%	25 Missing ⚠️
...loudstack/backup/GetBackupStorageStatsCommand.java	18.51%	22 Missing ⚠️
... and 38 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #10140      +/-   ##
============================================
+ Coverage     16.30%   16.45%   +0.14%     
- Complexity    13447    13608     +161     
============================================
  Files          5675     5688      +13     
  Lines        499249   501615    +2366     
  Branches      60377    60780     +403     
============================================
+ Hits          81415    82548    +1133     
- Misses       408761   409824    +1063     
- Partials       9073     9243     +170

Flag	Coverage Δ
uitests	`3.86% <ø> (-0.14%)`	⬇️
unittests	`17.36% <33.05%> (+0.19%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

leo79901 · 2024-12-31T06:52:13Z

Yes, we need this !

…d data volumes before restore.

abh1sar · 2025-01-06T12:00:02Z

@blueorangutan package

blueorangutan · 2025-01-06T12:02:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-01-06T13:19:25Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11996

github-actions · 2025-01-08T14:37:51Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

engine/schema/src/main/resources/META-INF/db/schema-42010to42100.sql

…p_details.

abh1sar · 2025-04-10T21:19:33Z

@blueorangutan package

blueorangutan · 2025-04-10T21:20:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-04-10T22:34:12Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13013

…kupCmd

abh1sar · 2025-04-14T07:18:24Z

@blueorangutan package

blueorangutan · 2025-04-14T07:20:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-04-14T08:44:37Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13040

abh1sar · 2025-04-14T08:54:50Z

@blueorangutan test

blueorangutan · 2025-04-14T08:56:04Z

@abh1sar a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-04-15T00:55:07Z

[SF] Trillian test result (tid-12978)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 54966 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10140-t12978-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestClusterDRS>:setup	`Error`	0.00	test_cluster_drs.py

weizhouapache · 2025-04-16T06:21:26Z

overall lgtm

note about the setDeviceId

for vm created from template, the data volume uses diskofferingId , so deviceId=1 is correct
for vm created from ISO, the root volume uses diskofferingId , deviceId should be 0 I think.
for vm created with multiple data disks, it uses datadiskofferinglist, but deviceId of volumes are not set in dataDiskOfferingInfo.

is deviceId important for this feature ? cc @abh1sar

abh1sar · 2025-04-16T16:57:48Z

Thanks @weizhouapache

for vm created from ISO, the root volume uses diskofferingId , deviceId should be 0 I think.

I tested this. The root volume is being assigned deviceId 0 for both template and iso.

for vm created with multiple data disks, it uses datadiskofferinglist, but deviceId of volumes are not set in dataDiskOfferingInfo.

deviceIds are set here

    public List<DiskOfferingInfo> getDataDiskOfferingListFromBackup(Backup backup) {
        ...
       diskOfferingInfoList.add(new DiskOfferingInfo(diskOffering, size, minIops, maxIops, deviceId));

Please let me know if this sounds ok.

weizhouapache · 2025-04-17T08:42:06Z

Thanks @weizhouapache

for vm created from ISO, the root volume uses diskofferingId , deviceId should be 0 I think.

I tested this. The root volume is being assigned deviceId 0 for both template and iso.

for vm created with multiple data disks, it uses datadiskofferinglist, but deviceId of volumes are not set in dataDiskOfferingInfo.

deviceIds are set here
    public List<DiskOfferingInfo> getDataDiskOfferingListFromBackup(Backup backup) {
        ...
       diskOfferingInfoList.add(new DiskOfferingInfo(diskOffering, size, minIops, maxIops, deviceId));
Please let me know if this sounds ok.

thanks @abh1sar
good to know it

About the deviceId of datadiskoffering

dataDiskOfferingInfo.setDeviceId(1L);

should it be 0 if the vm is created from ISO ?
this is not a valid question if we do not support backup actions for vm from ISO.

abh1sar · 2025-04-18T04:00:34Z

About the deviceId of datadiskoffering
dataDiskOfferingInfo.setDeviceId(1L);
should it be 0 if the vm is created from ISO ? this is not a valid question if we do not support backup actions for vm from ISO.

@weizhouapache For VM from ISOs createVirtualMachineFromScratch is called where deviceId remains null in rootDiskOfferingInfo and in allocateRawVolume the deviceId for volume is set to 0L.

        if (deviceId != null) {
            vol.setDeviceId(deviceId);
        } else if (type.equals(Type.ROOT)) {
            vol.setDeviceId(0l);

abh1sar · 2025-04-23T11:42:33Z

@blueorangutan package

blueorangutan · 2025-04-23T11:44:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

abh1sar · 2025-04-23T13:05:17Z

@blueorangutan package

blueorangutan · 2025-04-23T13:06:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-04-23T13:07:07Z

Packaging result [SF]: ✔️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13151

blueorangutan · 2025-04-23T14:35:08Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13152

abh1sar added 2 commits December 25, 2024 08:24

Initial api and service layer changes

35ed72c

Nas backup provider changes

e094e14

boring-cyborg bot added component:api component:backup component:kvm labels Dec 25, 2024

Add hypervisor, service_offering and template to backups table.

27feeda

abh1sar added 3 commits January 1, 2025 17:15

Create backup_details table and add details related to vm.

f98da37

Initial UI form

04a2deb

Save networkIds and diskOfferingIds in backup_details. Create require…

d3439ad

…d data volumes before restore.

boring-cyborg bot added component:storage component:UI labels Jan 3, 2025

Restore backup to VM plugin change for VeeamBackupProvider

753d80f

abh1sar added 3 commits January 7, 2025 02:50

UI form updates

72c943f

Create instance from backup using just the backupid

7ae078e

Remove zone and iso from DeployVMFromBackup.vue

31f89df

github-actions bot added the status:has-conflicts label Jan 8, 2025

abh1sar added 3 commits January 9, 2025 20:57

Create multiple data volumes using the deployVirtualMachine Api

7e3f083

Create multiple data volumes for createVMFromBackup api and ui

f46fad8

Merge remote-tracking branch 'upstream/main' into instance

d941809

boring-cyborg bot added component:compute component:kubernetes component:orchestration labels Jan 9, 2025

github-actions bot removed the status:has-conflicts label Jan 9, 2025

Fix build failure after merge

10f7beb

weizhouapache reviewed Apr 10, 2025

View reviewed changes

engine/schema/src/main/resources/META-INF/db/schema-42010to42100.sql Outdated Show resolved Hide resolved

Fixed deploydb issue by using TEXT instead of varchar(65536) in backu…

f8a20a4

…p_details.

abh1sar added 5 commits April 11, 2025 05:48

Handle create VM from backup for VMs created using ISO.

17ecc5a

Minor bug fix in UserVmManagerImpl

6b098ce

Fix for UT failures

8803c78

Fix build failures

57b0f2c

BaseDeployVMCmd as the base class for DeployVmCmd and CreateVmFromBac…

1075a55

…kupCmd

apache deleted a comment from blueorangutan Apr 16, 2025

Fix Expunge generic error

d2fb239

Fix Backup Storage capacity and NAS backup without mount options

2d89a8f

Dummy BnR plugin integration test

e2fa0be

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create new Instance from VM backup #10140

Create new Instance from VM backup #10140

abh1sar commented Dec 25, 2024 •

edited

Loading

codecov bot commented Dec 25, 2024 •

edited

Loading

leo79901 commented Dec 31, 2024

abh1sar commented Jan 6, 2025

blueorangutan commented Jan 6, 2025

blueorangutan commented Jan 6, 2025

github-actions bot commented Jan 8, 2025

abh1sar commented Apr 10, 2025

blueorangutan commented Apr 10, 2025

blueorangutan commented Apr 10, 2025

abh1sar commented Apr 14, 2025

blueorangutan commented Apr 14, 2025

blueorangutan commented Apr 14, 2025

abh1sar commented Apr 14, 2025

blueorangutan commented Apr 14, 2025

blueorangutan commented Apr 15, 2025

weizhouapache commented Apr 16, 2025

abh1sar commented Apr 16, 2025

weizhouapache commented Apr 17, 2025

abh1sar commented Apr 18, 2025 •

edited

Loading

abh1sar commented Apr 23, 2025

blueorangutan commented Apr 23, 2025

abh1sar commented Apr 23, 2025

blueorangutan commented Apr 23, 2025

blueorangutan commented Apr 23, 2025

blueorangutan commented Apr 23, 2025

Create new Instance from VM backup #10140

Are you sure you want to change the base?

Create new Instance from VM backup #10140

Conversation

abh1sar commented Dec 25, 2024 • edited Loading

Description

Other Miscellaneous changes

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov bot commented Dec 25, 2024 • edited Loading

Codecov Report

leo79901 commented Dec 31, 2024

abh1sar commented Jan 6, 2025

blueorangutan commented Jan 6, 2025

blueorangutan commented Jan 6, 2025

github-actions bot commented Jan 8, 2025

abh1sar commented Apr 10, 2025

blueorangutan commented Apr 10, 2025

blueorangutan commented Apr 10, 2025

abh1sar commented Apr 14, 2025

blueorangutan commented Apr 14, 2025

blueorangutan commented Apr 14, 2025

abh1sar commented Apr 14, 2025

blueorangutan commented Apr 14, 2025

blueorangutan commented Apr 15, 2025

weizhouapache commented Apr 16, 2025

abh1sar commented Apr 16, 2025

weizhouapache commented Apr 17, 2025

abh1sar commented Apr 18, 2025 • edited Loading

abh1sar commented Apr 23, 2025

blueorangutan commented Apr 23, 2025

abh1sar commented Apr 23, 2025

blueorangutan commented Apr 23, 2025

blueorangutan commented Apr 23, 2025

blueorangutan commented Apr 23, 2025

abh1sar commented Dec 25, 2024 •

edited

Loading

codecov bot commented Dec 25, 2024 •

edited

Loading

abh1sar commented Apr 18, 2025 •

edited

Loading