Tag Archives: Veeam

Veeam Jobs fail after adding new ESXi hosts

When you add new VMware Hosts to a VMware vCenter Cluster, please remember to do a rescan of vCenter in Veeam. Veeam now rescans the complete environment and adds the new ESXi hosts to its database so everything will work correctly.

My problem was after I placed new VMware ESXi hosts in the VMware vCenter cluster, the backup was failing on every VM that was placed on the new ESXi hosts. Veeam had detected the new VMware ESXi hosts, so I thought everything was good.

My Error message was: “Failed to create processing task for VM [vm name] Error: Host ‘ESXi host’ with type ‘ESX’ and parentId ‘e2afc39e-f38c-466b-9431-cd1b7aedc48b’ not found.”

A quick google action on this error message resulted in a Veeam Forum post. See http://forums.veeam.com/vmware-vsphere-f24/problem-with-backups-after-adding-new-host-to-vcenter-t18512.html

User v.Eremin is saying in his post:

As long as VMs are added to backup console via vCenter, backup server should be vMotion aware and you shouldn’t encounter any issues after VM migration. However, it seems that vCenter added to backup console didn’t know about the new host and about the fact that certain VMs had been migrated to it, thus, backup jobs failed. So, now, when the rescan is done, everything should be ok, even without manual re-adding of VMs.

This gave me the tip to first do a rescan of vCenter to see if this solved my problem. And this worked for me, thanks Veeam Forums!

Veeam Backup & Replication 8 Update 2 released

Yesterday Veeam launched update 2 for the Veeam Backup & Replication 8 application. With this update Veeam is fully compatible with VMware vSphere 6.

There are the release notes.

Engine

  • Job performance improvements. A number of optimizations focused on reducing the time jobs spend performing auxiliary tasks should significantly reduce the overall job execution time for incremental runs of backup and Backup Copy jobs.
  • Slow backup storage optimization. Target data mover now caches metadata from backup files, instead of requesting it from backup files residing multiple times over the course of the job run. This significantly improves performance of jobs targeting storage with poor random I/O performance (such as EMC Data Domain), while reducing the overall storage load. Important notes:
    • The cache is only enabled when using a 64-bit OS on backup repository (or gateway server), and increases RAM consumption of each job by 2GB on average, depending on backup chain size. Using non-default backup block size changes cache RAM requirements proportionally.
    • Using the default “Automatic selection” gateway server setting for Shared folder, EMC DataDomain or HP StoreOnce based backup repositories disables the cache.
    • Disabling built-in deduplication disables the cache.
  • 64-bit Linux data mover. For increased scalability, a 64-bit data mover will now be used on 64-bit Linux backup repositories with an OS kernel version of 2.6.18 or later.
  • Linux data mover update. To enable new functionality, jobs will leverage the new version of the data mover on Linux servers with kernel version 2.6 or later. For compatibility with existing servers, jobs will use a legacy data mover when OS kernel version is earlier than 2.6 or cannot be detected.
  • Direct data mover communication. When both data movers are running on the same server (e.g., when backing up to a local storage attached to a backup proxy server, etc.), they will now exchange data through shared memory. This should improve data transfer performance of local backup jobs currently reporting Network as the bottleneck, or when you see high load on some of the backup proxy server NICs when the data was supposed to stay local to the server.
  • vPower NFS performance. Increased scalability of vPower NFS server by significantly reducing CPU usage associated with guest I/O operations using small data blocks
  • VeeamZIP performance. Improved VeeamZIP performance when using a Veeam backup repository as the target.
  • Replication from backup file enhancements. Replication from backup files no longer requires matching block size between the backup file and replica job settings

VMware

  • vSphere 6 support. Added support for ESXi 6.0 and vCenter Server 6.0 (including vCenter Server Appliance).
  • vCloud Director 5.6 support. Added support for backup and restore of vCloud Director 5.6 virtual machines (VMs) and vApps.
  • VMware Virtual Volumes (VVols) support. VMs residing on virtual volumes can be processed in Virtual Appliance (Hot Add) and Network (NBD) processing modes. Hot add processing mode requires that all proxy VM disks are located on the same virtual volume with the processed VM.
  • VMware Virtual SAN (VSAN) 2.0 support. VMs residing on VSAN can be backed up in Virtual Appliance (Hot Add) and Network (NBD) processing modes. For VMs with existing snapshots, hot add processing requires that all proxy VM disks are located on the same VSAN datastore.
  • Storage Policy-Based Management (SPBM) policy backup and restore. Storage policy associations are backed up and restored for each virtual disk upon full VM restore. This eliminates the manual process, which directly impacts recovery time. SPBM policies are important to restore because “out of policy” VMs can impact availability of either the restored VM itself or other VMs sharing the same storage.
  • Support for Fault Tolerant VMs (FT VMs) for backups and replicas. VMs with Fault Tolerance enabled can now be backed up and replicated. Legacy Fault Tolerant VMs must have Fault Tolerance re-enabled after vSphere 6 upgrade using vSphere Web Client to enable this capability.
  • vSphere 6 tags integration. vSphere 6 introduces new APIs for programmatic access and management of vSphere tags. With vSphere 6 tag support, you can continue building new advanced backup policies based on tags, even after you upgrade to vSphere 6. Due to new unique tag IDs in vSphere 6, be sure to review your job setup after you upgrade to vCenter Server 6.0.
  • Cross-vCenter vMotion awareness. Added support for cross-vCenter vMotion to the Quick Migration functionality. Now, when migrating a VM to another vCenter Server, the associated entries on backup or replication jobs will be updated automatically to keep the VM protected
  • Quick Migration to VVols. Enables moving VMs to VVols when using vMotion is not an option due to unreliable or slow network links, in scenarios when vMotion is not supported, or due to lack of VMware licensing. This functionality can help perform full migrations to new vSphere 6 clusters built from the ground up without inheriting new design problems from previous clusters.
  • Hot-Add transport mode of SATA virtual disks. On ESXi 6.0, SATA virtual disks can now be processed in Virtual Appliance (Hot Add) processing mode, which was previously limited to SCSI virtual disks only.
  • vSphere tags priority. vSphere tags priority has been raised one level above containers priority to ensure that jobs behave in line with customers’ expectations when job’s inclusion and/or exclusion lists use both containers and tags.
  • Disable CBT reset. The workaround for VMware Changed Block Tracking (CBT) corruption issue at 128GB boundaries (VMware KB2090639) can now be disabled by customers with ESXi hosts patched against this issue. To disable automatic CBT reset upon virtual disk size change, create theResetCBTOnDiskResize (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set it to 0.

Storage integration

  • Granular rescan settings. Added ability to control which storage LUNs and volumes should be rescanned periodically for new VMs. These settings can be accessed directly from the Add Storage wizard while registering the storage, or by right-clicking the already added storage in the Storage Infrastructure tab.
  • NetApp integration enhancements. The following NetApp-specific enhancements have been added:
    • Added support for triggering updates of non-qtree and vFiler-based SnapMirror and SnapVault
    • Added support for backing up VMs from NetApp MetroCluster in the failed over state
    • Preferred Network settings of Global Network Traffic Rules dialog now also used to pick preferred storage adapters for data transfer. And in cases when multiple adapters use the same subnet, NetAppOrderedIPList (STRING) registry value underHKLM\SOFTWARE\Veeam\Veeam Backup and Replication can now be created to specify preferred adapters’ IP addresses separated by a semicolon.

Hyper-V

  • SMB3 restore performance. Improved performance of full VM restore to SMB3 shares with caching disabled (such as in Nutanix).
  • Hyper-V Integration Services status check. To prevent situations with some VMs unexpectedly going into the Saved State during a volume snapshot creation, jobs will now check that Hyper-V Integration Services (HIS) are running and reachable, waiting for HIS to respond for up to 10 minutes before failing the VM processing.
  • Replica VM adjustment. Maximum RAM value of the replica VM is now automatically reduced according to the target host capabilities.

File-Level Recovery

  • Native 4K disk support. Windows File-Level Recovery now supports mounting native 4K virtual disks from backups out of the box, without requiring a user to switch to the mount engine via the corresponding registry value.
  • Linux ACL preservation. Multi-OS File-Level Recovery now preserves Linux ACL when restoring a file via Restore and Copy To operations. This requires that both backed up and destination VM runs Linux with kernel version 2.6 or later.
  • Linux host selection. Multi-OS File-Level Recovery now allows choice of a target Linux host to restore guest files to without having to add it to the managed servers first.
  • Restored files logging. Multi-OS File-Level Recovery now logs restored files in its session log, similar to the way Windows File-Level Recovery performs its restore activity logging.
  • SNMP traps. Both Windows and Multi-OS File-Level Recovery can now trigger SNMP notifications. To enable such notifications, create theEnableRestoreSNMPTraps (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set it to 1.

Veeam Explorer for Microsoft SQL Server

  • Improved restore performance. Database restore performance has been improved by up to 3x.
  • Increased scalability. Reduced load on SQL Server hosting Veeam® Backup & Replication™ database in scenarios when transaction logs are being backed up from multiple SQL servers on a frequent schedule.
  • Log truncation retry. In cases when log truncation fails under the specified guest processing account, it will be retried from the LOCAL SYSTEM account, which often carries the necessary privileges to perform this operation.
  • SQL database restore. Added a warning when restoring a SQL database to a different location where the database with the same name already exists when using the Enterprise Manager web UI
  • Improved issues reporting. Backup job now detects and reports more issues around transaction log processing, such as failure to truncate logs. This may result in the appearance of a new warning after the update, simply because existing issues were not previously reported.

 

Veeam Cloud Connect

  • Multiple subscriptions support. Added ability to add the same service provider multiple times using different credentials.
  • Reconnect performance improvements. Reconnecting to a service provider should now occur much faster when a cloud provider has multiple Cloud Gateways deployed.
  • Enhancements for service providers. The following are enhancements relevant to Veeam Cloud Connect service providers only:
    • Added ability to granularly purge global cache from WAN accelerator for the specific tenant.
    • When deleting a tenant, global cache belonging to the tenant is now deleted automatically.
    • Added ability to query the amount of protected VMs for the given tenant through RESTful API using the vmCount property.
    • Service Providers node of the management tree is always shown when Cloud Connect Service Provider license is installed, without requiring the service provider to register at least one hypervisor host first. This enables service providers to more easily client consoles to user for testing or manipulating client backups.
    • Enhanced debug logging to include initiator information for when the connection is terminated.

 

Tape

  • Job performance. File to Tape and Backup to Tape job performance was further improved.
  • Synthesized full backup enhancements. Synthesized full backup functionality has been reworked for better reliability, performance and support for very large backup files.
  • Third-party tape handling. Added support for detecting and reusing tapes written by third-party tape backup vendors.

User Interface

  • Manual job chain execution. By popular demand, when manually starting a job that has other jobs chained to it via “After this job” scheduling option, you will now be offered a choice between starting the selected job alone, or starting the entire chain.
  • Disable network traffic rules. Both traffic throttling and encryption can now be temporarily disabled without requiring you to delete the corresponding rule completely, which is useful for troubleshooting or for temporarily removing bandwidth consumption restrictions.
  • Improved guest credentials test. Guest processing credential test will now additional verify the presence of mlocate, which is required for Linux guest file system indexing.

Setup

  • Silent install. Starting with Update 2, updates will fully support automated silent install, including triggering update of remote backup infrastructure components. This should significantly reduce the time required to update Veeam deployments with a large number of backup servers.

PowerShell

  • Support in Free Edition. Start-VBRZip cmdlet is now enabled in Veeam Backup™ Free Edition, allowing users to schedule periodic VeeamZIP™ backups using external task schedulers, such as Windows Task Scheduler. This should make Veeam Backup Free Edition a viable alternative to ghettoVCB and other similar script-based free backup solutions.

Integration with Veeam Endpoint Backup FREE

This update enables you to take full advantage of Veeam backup repositories as additional target locations for your Veeam Endpoint Backup jobs, with backups for each endpoint being stored in a separate folder named after the endpoint.

To assign end-user permissions to individual Veeam backup repositories, use the new repository properties dialog that will appear in the repository’s shortcut menu upon first connection attempt from Veeam Endpoint Backup™ FREE to the Veeam Backup & Replication server. Before that, this dialog can be accessed by Ctrl-right-clicking the backup repository.

Global network traffic rules defined in Veeam Backup & Replication apply to endpoint backup jobs using Veeam backup repository as the target, allowing you to ensure that concurrent backups from multiple endpoints do not impact your available bandwidth, or that endpoint backup network traffic is encrypted.

Using a Veeam backup repository as the target for your endpoint backup jobs enables the following additional functionality:

  • Centralized monitoring. Perform basic monitoring and management for all incoming endpoint backup jobs, including email notifications about endpoint backup status.
  • Offsite protection. Get your endpoint backups off site to disk-based storage, tape or your Veeam Cloud Connect service provider with Backup Copy and Backup to Tape jobs.
  • Backup encryption. You can choose to optionally encrypt your endpoint backups stored in Veeam backup repositories. All standard Veeam encryption features are supported for endpoint backups, including password loss protection.
  • Application item-level recovery. Restore application items from backups of server machines with Veeam Explorers™ for Microsoft Active Directory, Exchange, SharePoint and SQL Server.
  • Disk export. Export physical disk content from backup into VMDK/VHD/VHDX virtual disk files

To obtain this update, please click here (you need to be logged in to download the update).

 

 

 


Veeam Backup & Replication v8 Patch 1 released

Last week Veeam released the first patch voor Veeam Backup & Replication v8. So what is new and fixed?

These are the release notes.

New Features and Enhancements

General 

  • Retention policy for replication jobs is now processed concurrently for multiple VMs in the job, as opposed to one by one, reducing the overall job run time.
  • Added email notification about File to Tape and Backup to Tape jobs waiting for user interaction (for example, when a tape medium needs to be inserted by the user)
  • VMware Tools quiescence should now log an informational event when VMware Tools are outdated, as opposed to a warning event.
  • Independent virtual disks which are explicitly excluded from processing in the job settings should no longer log a warning during the first time when the given VM is being backed up.
  • Virtual Lab proxy appliance was upgraded to virtual hardware version 7 to allow for it to manage more networks out of box.
  • Jobs should no longer appear to “hang doing nothing” in some circumstances, with the duration not ticking for any event in the job log.

Cloud Connect

  • Cloud backup repository size limit of 63TB has been removed

Linux support enhancements

  • Added support for file level recovery from XFS volumes (default file system in Red Hat Enterprise Linux and CentOS distributions starting from version 7.0)
  • Added support for file level recovery from Btfrs volumes (default file system in SUSE Linux Enterprise Server starting from version 12.0).
  • Added support for file level recovery from VMs with more than 10 LVMs.
  • Improved Linux guest file system indexing performance for incremental backup job runs.
  • FTP access mode of the file level recovery helper appliance now also displays friendly LVM volume names introduced in v8.
  • To enable identity verification of the remote server, SSH key fingerprint is now displayed to a user when registering the new Linux server. Accepted SSH keys are stored in the configuration database to protect from MITM attack.

 

Performance enhancements

  • Added experimental support for direct data movers communication when both are running on the same server (for example, when backing up to a local storage on backup proxy server). If your local backup jobs report Network as the bottleneck, and you see high load on some backup proxy server NICs when the data was supposed to stay local to the server, you are likely to benefit from this behavior modifier. To enable alternative data exchange path, create the DataMoverLocalFastPath (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set it to the following values:

    0: Default behavior (no optimizations)

    1: Data exchange through TCP socket on the loopback interface (faster)

    2: Data exchange through shared memory (fastest)

EMC Data Domain integration 

  • Synthetic full backup creation and transformation performance has been improved significantly.

 

HP 3PAR StoreServ integration

  • Remote Copy snapshots are now automatically excluded from rescan, both speeding up the rescan process and reducing storage load.

 

NetApp integration

  • Added support for NetApp Data ONTAP 8.3.
  • Added support for HA Pair 7-mode configuration.
  • Added support for vFiler DR configuration.

 

Resolved Issues

 

General

  • Restored incremental backup job performance back to v7 levels by removing unnecessary metadata queries from previous backup files.
  • Multiple issues when parallel processing is disabled (high CPU usage on backup server, wrong task processing order, reversed incremental backup jobs with disk excluded failing to access backup files).
  • Automatic network traffic encryption (based on public IP address presence) is enabled even when both data movers are running on the same computer, reducing job’s processing performance.
  • Quick Backup operation triggers jobs chained to the job it uses.
  • Loss password recovery option does not function correctly when Enterprise Manager has multiple backup server registered.
  • Under rare circumstances, backup job with encryption enabled may fail with the “Cannot resize block data is encrypted” error.
  • When multiple network traffic rules having the same source and target ranges are set up, changing settings to one of the rules also updates all other rules.

 

vSphere

  • In vSphere 5.5, enabling virtual disk updates redirection to the VMFS datastore in Instant VM Recovery settings, and then migrating a recovered VM to the same datastore with Storage vMotion causes data loss of data generated by the running VM.
  • Full VM restore process puts all virtual disks to the datastore selected for VM configuration file,ignoring virtual disks placement settings specified in the wizard.
  • In certain environments, jobs fail to process VMs in Direct SAN Access mode with the “Failed to create processing task for VM Error: Disk not found” error.
  • Dismounting processed virtual disks from backup proxy server takes longer than expected in the Virtual Appliance (hot add) processing mode.
  • Rescan of vCenter server containing hosts without HBA and SCSI adaptors fails with the “Object Reference not set to an instance of an object” error.
  • Replication jobs to a cluster fail with the “The operation is not allowed in the current state” error if chosen cluster host is in the maintenance mode.
  • Under rare circumstances, jobs with Backup I/O Control enabled may occasionally log the “Operation is RetrievePropertiesOperation” related errors.
  • Enabling Backup I/O Control limits snapshot removal tasks to one per datastore regardless of datastore latency levels.
  • VM Copy jobs always fail over to the Network (NBD) processing mode.
  • VMs with virtual disks having the same names but with different capitalization, and located on the same datastore cannot be backed up.
  • Backing up vCloud Director vApp to a CIFS share may fail with the “Object Reference not set to an instance of an object” error.
  • Snapshot Hunter ignores backup window specified in the corresponding job.
  • vCloud Director VMs created from linked clone template and with a user snapshot present incorrectly trigger Snapshot Hunter.

 

Hyper-V

  • CBT does not track changes correctly on VHDX disks files larger than 2 TB.
  • CBT may return incorrect changed blocks information for differential disks, when parent and child disks have the same name.
  • Large amount of unmapped VHDX blocks may cause the job to failover to full scan incremental backup with the “Failed to update unaccounted changes for disk. Change tracking is disabled” error.
  • Existing Hyper-V backup and replication jobs processing VM with SCSI disks start to consume x2 space on target storage after upgrade to v8.
  • Removing a node from a cluster does not remove the corresponding node from the cluster on the Backup Infrastructure tab of the management tree until the next periodic infrastructure rescan (in up to 4 hours).
  • Replication from backup fails with the “Virtual Hard Disk file not found[/id” error if backup job that created this backup had one or more disks excluded from processing.
  • Under rare circumstances, legacy replicas using non-default block size may fail with the “[i]FIB block points to block located outside the patched FIB” or “Block offset [] does not match block ID []” error messages.
  • Offhost backup from Windows Server 2012 R2 fails with the “Exception of type ‘Veeam.Backup.AgentProvider.AgentClosedException’ was thrown” error when processing a VM that can only be backed up in Saved or Crash-Consistent states, and native Hyper-V quiescence is enabled in the job settings.
  • If a Hyper-V cluster node goes offline during backup from shared volume snapshot, and does not recover until the Veeam job fails, CSV volume backup infrastructure resource will not be released by the scheduler. As the result, other jobs waiting to process VMs from the same volume may remain hanging in the “Resource not ready: Snapshot” state.
  • Replication job fails with the “Cannot assign the specified number of processors for virtual machine” error for VMs with core count larger than the one of the target Hyper-V host. Legacy replication job works fine in the same circumstances, however VM failover is not possible.

 

Application-aware processing

  • VeeamGuestHelper process crashes when backing up Windows 2003 SP2 VM running Oracle.
  • Application-aware processing hangs for extended time when processing a Windows VM containing one or more Oracle databases in the suspended state.
  • Certain advanced guest processing settings lead to VM processing failing with the “Unknown indexing mode ‘None’ ” error.

Multi-OS File Level Recovery

  • Restore to original location does not work in certain LVM configurations.

 

Enterprise Manager

  • Under certain circumstances, Enterprise Manager data collection may fail with the “Cannot insert the value NULL into column ‘guest_os’ ” error.
  • Restore scope rebuild operation tries to reach out to the hosts already removed from the corresponding backup server, resulting in login errors.

 

Built-in WAN Acceleration

  • WAN accelerators fail to detect and populate global cache Windows Server 2012 R2 and Windows 8.1 OS data, which may result in reduced data reduction ratios.

 

Cloud Connect

  • The Decompress backup data blocks before storing backup repository option is ignored for Cloud Connect backend repositories.
  • Backup Copy jobs with encryption enabled fail with the “Attempted to compress encrypted data-block” error if backend repository has the Decompress backup data blocks before storing option enabled.
  • Veeam Backup Service crashes with the “An existing connection was forcibly closed by the remote host” if network connection fails immediately upon establishing the initial connection to the Cloud Gateway.
  • Cloud Connect quotas are rounded incorrectly in the service provider’s interface.
  • Tenant statistics data such as data sent/received is only shown for the last 24 hours.

 

Veeam Explorer for Active Directory

  • “This implementation is not part of the Windows Platform FIPS validated cryptographic algorithms” error is displayed if Use FIPS compliant algorithms for encryption, hashing, and signing Group Policy setting is enabled.
  • Attempting to open single label domain’s AD database fails with the “No writable domain found” error.
  • Attempting to open third level domain’s AD database fails with the “No writable domain found” error.
  • Under certain circumstances, object restore may fail with the “Unable to cast object of type ‘System.String’ to type ‘System.Byte[]’ “error.

 

Veeam Explorer for Exchange

  • Process of exporting email messages cannot be stopped.
  • Copying or saving attachments from multiple different emails always copies or saves attachment from the first email for which this operation was performed.
  • Opening public folder’s mailbox fails with the “Unable to find wastebasket” error.

 

Veeam Explorer for SQL Server

  • Restoring to SQL instance with case sensitive collation enabled fails with the “The multi-part identifier “a.database_id” could not be bound” error.

Backup from Storage Snapshots 

  • Backup jobs from storage snapshots on iSCSI SAN take too long to initialize when large amount of iSCSI targets are configured on the backup proxy server.
  • Under certain circumstances, backup jobs from storage snapshots on FC SAN may fail with the “Unable to find volume with Node WWN and LUN ID” error.

 

HP 3PAR integration

  • Rescan of 3PAR array with empty host sets fails with the “Object reference not set to an instance of an object” error.

 

NetApp integration

  • Under certain circumstances, storage rescan process may leave temporary storage snapshots behind.

 

Tape

  • Backup to Tape jobs may appear to run slower after upgrading to v8 because of copying unnecessary backup files (VRB) to tape when source backup job uses reversed incremental backup mode.
  • On standalone drives, tape jobs configured to eject tape upon completion fail the “Object reference not set to an instance of an object” error after 10 minutes of waiting for the user to insert tape.
  • Tape job configured to use full and incremental media pools hosted by the different standalone tape drives fails when processing incremental backup with the “Value cannot be null” error. Tapes are formatted with the block size of tape medium, as opposed to the default block size of the tape drive.
  • Tape job configured to use full and incremental media pools hosted by the different tape libraries fails while waiting for an incremental pool’s tape with the “Tape not exchanged” error
  • Tape jobs fail with the “GetTapeHeader failed” error on IBM TS3100 tape library with Path failover feature disabled.

 

PowerShell

  • Objects obtained with Find-VBRViFolder cmdlet have incorrect Path attribute.
  • Get-VBRTapeJob cmdlet fails to obtain job object for tape jobs with incremental processing disabled.
  • Get-VBRTapeLibrary cmdlet fails with the Standard Edition license.
  • All built-in WAN accelerator management cmdlets have been made available in all paid product editions

 

To obtain the patch, please click here


Veeam Backup & Replication setup or upgrade not starting

A couple of weeks ago I tried to do an upgrade of the Veeam Backup & Replication 6.1 to v7.0. I mounted the fresh downloaded ISO of Veeam v7.0. The splash screen showed, and when I clicked on the Install / Upgrade options nothing happened.


This server is running Windows 2008R2 Service Pack 1, and is also the VMware vCenter server for the ESXi environment.

Because this server is a vCenter Server and purely a management server this server didn’t had all the Windows Updates installed that are available today. So my first guess was to update the complete server. After updating this server and a fresh reboot, I started the installation again. No success, still the same behavior. Splash screen disappears and reappears again without starting the Veeam installation.

There are also no errors in the event logs of the server. Always nice to trouble shoot…

When installing the Microsoft System CLR Types for SQL components there is an error. 


So there was something wrong with the Windows environment, but everything was working like it should. I already checked all the perquisites, and everything was installed. I didn’t know where to look for, so after some time I contacted Veeam support. They checked the environment and reinstalled the .Net Framework. After this installation the Veeam installer was working correctly. Problem was a corrupted or damaged .Net framework installation.

Lessons learned:

  • Veeam Support is awesome, they contacted me very quickly and solved my problem within an hour or so.
  • When running into strange behavior with the installer, try to repair or reinstall the .Net Framework it might solve your problems.

New free tool: Veeam Task Manager for Hyper-V

Today Veeam released a new free tool for your Hyper-V environment called “Veeam Task Manager for Hyper-V”

Veeam Task Manager for Hyper-V is a portable, standalone performance monitoring tool that’s FREE!

Improve troubleshooting in your Hyper-V environment by seeing what Windows Task Manager doesn’t show you. Veeam Task Manager for Hyper-V displays a real-time view of both host and VM compute resource consumption.

Get the critical visibility you need.

This lightweight tool is portable so you can run it from any USB device for emergency troubleshooting. No installation or integration needed!


More information:

Download the FREE Veeam Task Manager today

How to stop your Veeam backup job when it is stuck

A couple of weeks ago I ran into an issue that one of my Veeam backup jobs was stuck in a stopping state. After waiting for a long time I started searching for a solution to kill this backup job.

I found a KB article at the Veeam Knowledgebase that is worth mentioning. This is KB1727 “How to forcibly stop jobs that are stuck in ‘stopping’ status”

Solution:

The problem may occur because a process or task that the job is waiting for has not been completed. Such a task can be snapshot removal. Therefore, as the first step it is recommended to check the vSphere Client to see if there is a snapshot removal process pending/working on the vSphere side.

Make sure no important jobs/restores are running because the following steps will force them to stop:

  1. Close the Veeam Backup & Replication console, and open services.msc on the Veeam Backup server,
  2. Stop the Veeam Backup Service.
  3. Open the Task Manager and kill all Veeam.Backup.Manager and VeeamAgent.exe processes.
  4. Restart the Veeam Backup Service to release file locks held by either SQL or the OS.
  5. Remove snapshots from VMs (if they have not been already removed).

Note. If you use the Virtual Appliance (HOTADD) mode, before removing the snapshots make sure there are no stuck disks on the Veeam Backup server or one of the backup proxies. Otherwise, the snapshots can be orphaned.

 

Nice to know…


Veeam Backup & Replication 7.0 R2 update

 

Last week Veeam released there long awaited update to fix the support for VMware vSphere 5.5 and Windows 2012R2.

New Features

VMware 

  • vSphere 5.5 support, including support for 62TB virtual disks and virtual hardware v10 virtual machines.
  • vCloud Director 5.5 support.
  • Support for Windows Server 2012 R2 and Windows 8.1 as guest virtual machines (VMs).
  • Added ability to limit maximum amount of active VM snapshots per datastore to prevent it from being overfilled with snapshot deltas. The default value of 4 active snapshots can be controlled with MaxSnapshotsPerDatastore (REG_DWORD) registry key.

Microsoft

  • Windows Server 2012 R2 Hyper-V and free Hyper-V Server 2012 R2 support, including support for Generation 2 virtual machines.
  • System Center 2012 R2 Virtual Machine Manager (VMM) support.
  • Support for Windows Server 2012 R2 and Windows 8.1 as guest virtual machines (VMs).
  • Support for the installation of Veeam Backup & Replication and its components on Windows Server 2012 R2 and Windows 8.1.

Built-in WAN acceleration

  • Increased data processing performance up to 50% with hard drive based cache, and up to 3 times with SSD based cache. Multi-core CPU on source WAN accelerator is recommended to take full advantage of the enhanced data processing engine.

Replication

  • Added ability for source and target proxy servers to reconnect and resume replication when network connection between source and target site drops for a short period of time.

Tape

  • Added support for a number of enterprise-class tape libraries with partitioning functionality that allows presenting multiple tape library partitions to the same host.
  • Import/export slot interaction has been redesigned to add support for a number of IBM and Oracle tape libraries.

Application-aware processing 

  • Added ability for application-aware processing logic to detect passive Microsoft Exchange DAG database present on the VM, and process it accordingly.
  • Added support for Exchange CCR clusters.

User interface 

  • User interface should now remember size and positions off the main window, as well as all panels and columns.

Resolved Issues

Engine

  • Importing backup fails with the “Failed to load backup meta from file. Unexpected end of file has occurred” if VBM file size exceeds 10MB.
  • Deleted VM retention incorrectly applies to not deleted VMs when full backup transform duration exceeds the deleted VM retention period (14 days by default).
  • Removed unintentional compression (and further uncompression) of data transferred between source and target data movers when both are running on the same server, and compression is disabled in the job settings.

VMware 

  • Replication jobs should now automatically handle the situations when source VM disk size changes, instead of failing with the “Declared and actual sizes of the disk are different” error.
  • Replication jobs using Windows 2003 hot add source proxy fails with the “Client error: Failed to send command” error.
  • Very long running replication jobs may fail with the “Failed to process [srcSaveSignature]” error.
  • Re-IP addressing process may unintentionally update some reparse points found in the guest file system, which under certain circumstances may make them unusable.
  • vCenter Server version is not updated after vCenter update on the vSphere Web Plug-in configuration page of Enterprise Manager web UI.
  • vSphere Web Client plug-in shows “Access denied” error for all users that have never logged into Enterprise Manager.
  • Added ability to configure timeouts for networkless (VIX) operations via vixOperationTimeoutSec (REG_DWORD) and vixExecutionTimeoutSec (REG_DWORD) registry values in seconds.
  • Added ability to configure timeouts for Instant VM Recovery operations via IrMountLeaseTimeOut (REG_DWORD) and IrVcdMountLeaseTimeOut (REG_DWORD) registry values in minutes. The new, increased defaults are 30 and 45 minutes respectively.

Hyper-V 

  • Under certain scenarios of dynamic VHD virtual disk size increase, changed block tracking information can be reset, resulting in incremental job run having to read the entire VHD to determine the incremental changes.
  • Performing compact operation on VHD virtual disk can reset changed block tracking information.
  • Instant VM Recovery fails with the “Connection with Hyper-V mount agent was lost” error for VMs with dynamic VHD disk size not being a multiple of 256MB.
  • Failure to rescan any host makes rescan of the remaining cluster hosts to fail with the following error: “Failed to execute WMI Query
  • Instant VM Recovery fails with the “Failed to prepare VHD file for mounting” when target Hyper-V host’s storage uses native 4KB sector size disks.
  • Application-aware processing of Windows Server 2012 VM running on Windows Server 2008 R2 Hyper-V may hang for 10 minutes.
  • Removed the single VSS snapshot per cluster limit. As the results, all jobs should now correctly use the max concurrent snapshot per volume setting.
  • Replication job may fail during the free disk space check with the “Failed to extract volume mount point by the specified path” error, depending on how destination folder was selected in the replication job wizard.
  • Veeam jobs may cause non-VM owner Hyper-V nodes to log the following events to the system event log: “(eventid = 8194): “Unexpected error querying for the IVssWriterCallback interface. hr = 0x80070005, Access is denied
  • Changed block tracking functionality does not function on cluster disks.

Application-aware processing 

  • Exchange 2003 running in VM restored from Veeam backup may rename its mailbox databases after each subsequent reboot.

Backup Copy 

  • Retry cycles cause incorrect processing duration shown in the Backup Copy job report.
  • Backup Copy job reports show the inflated traffic savings ratios due to being calculated off the incorrect parameter.
  • Certain Backup Copy jobs failures result in Backup Copy job report not send immediately, but rather at the end of the copy interval.
  • A number of additional enhancements to the Backup Copy job report.

File level recovery 

  • Copy To fails on files with long paths with the following error: “The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248
  • Mount cache files should now be cleaned up from %AppData% folder automatically.

Guest indexing 

  • Guest file system catalog data processing performance was accelerated a few times in cases when the catalog contains very large amount of restore points.
  • Catalog service may take a very long time to start when there is a large data upload queue present.

1-Click VM Restore 

  • File restore operator scope settings should no longer affect ability to perform VM restore in certain scenarios with overlapping permissions.

Built-in WAN Acceleration 

  • Backup Copy job fails with the “Unable to initialize remote target session connection: all aliases have failed” after source VM disk has been resized.
  • Large Backup Copy jobs may fail with the “ERR |Invalid length of the binary data” error.
  • WAN accelerator resource is not released correctly if Backup Copy job did not copy any data during the copy interval, preventing the following copy intervals from obtaining access to WAN accelerators.
  • Under certain circumstances, the Backup Copy job health check may fail with the “Access is denied” error.

Backup from Storage Snapshots 

  • Added ability to perform multiple VM snapshot removal concurrently (according to the global concurrency setting) when multiple VMs are backed up from the same LUN by the same job.
  • HP StoreServ (3PAR) integrated jobs may fail with the “Invalid input: string length exceeds limit” error depending on the snapshot name length.

Tape 

  • Drive to library matching code does not process some symbols in the serial numbers, resulting in matching problems with certain tape libraries (for example, QUADStor VTL).
  • Added ability to disable tape device rescan completely with the DisableTapeSubsystem (REG_DWORD) registry key. This can help in situations when the same tape devices is shared between multiple backup solutions.
  • In some cases, stopping tape jobs can cause backup session not finalized correctly, resulting in future attempts to write to the same tape failing with the “Tape positioning” error.
  • The Do not protect data (cyclic overwrite tapes as required) option does not work correctly for standalone drives.
  • Tapes with multiple partitions cannot be correctly formatted.
  • Tape jobs fail with the “Tape changer alert: The library was unable to read the bar code on a cartridge” error upon receiving the corresponding alert from the tape library.
  • Certain tape device configurations cause “There is no media in the drive” error to be logged incorrectly when the media is in fact present in the drive.
  • Under rare circumstances, writing to the first tape in the media set may fail with the following error: “The INSERT statement conflicted with the FOREIGN KEY constraint “FK_Tape.file_parts_file_versions”

File Copy 

  • Linux symbolic links are not copied by file management operations and file copy jobs.

SureBackup 

  • SureBackup job may incorrectly use vCloud Director virtual machine backup when Application Group is configured by adding VM from vSphere infrastructure.
  • SureBackup does not update port group correctly when both production and virtual lab networks are located on the same Distributed Virtual Switch (DVS).

Veeam Explorer for Microsoft Exchange 

  • Launching Veeam Explorer for Exchange from Backup Browser may fail with the following error: “Error: JetError -546, JET_errLogSectorSizeMismatch, the log file sector size does not match the current volume’s sector size

Veeam Explorer for Microsoft SharePoint

  • Using localized SQL server for staging fails with the “File is a Directory” error.
  • Attachments lost during recovery when attachment path contains Unicode symbols, even though they can be seen in the user interface.

Known Issues

General 

  • Network traffic verification does not function when the Use multiple upload streams per job option is disabled.

vSphere 5.5 

  • E1000E vNIC adaptors has been found to crash ESXi host under heavy load. Avoid using them until the patch is available from VMware. For more information, please refer to VMware KB2059053.
  • Virtual appliance (hot add) processing mode does not support SATA disks. This is by design of VMware hot add functionality, which requires SCSI adaptors.
  • vSAN beta is not supported.

Hyper-V 2012 R2 

  • Offhost backup from deduplicated volume fails if Data Deduplication feature is disabled on backup proxy server, or incompatible with Windows Server deduplication version used on backed up volume.
  • Shared VHDX virtual disks backup is not supported.

Windows Server 2012 R2 

  • In Windows Server 2012 R2 environments with Windows Server deduplication feature in use, be sure to install Veeam Backup & Replication on Windows Server 2012 R2, and enable Data Deduplication feature. Windows Server 2012 deduplication driver is not compatible with Windows Server 2012 R2 deduplication, causing file level recovery errors with false data corruption reports.

R2 update is available for download starting November 13, 2013 (KB1831)

This update is cumulative and includes all fixes from Patch #1 below.

Veeam released Patch 1 for Veeam Backup & Replication v7.

Veeam has released Patch 1 for Veeam Backup & Replication v7.

Mind this is NOT a patch to make Version 7 compatible with VMware vSphere 5.5! vSphere 5.5 is not yet supported by Veeam Backup & Replication v7.

For more information see Veeam KB1823.

So what is fixed in this patch.

Engine

  • Under certain circumstances, backup to CIFS target may fail with the following errors: “The process cannot access the file because it is being used by another process”, “Failed to create or open file \\server\share\file.vbk“.
  • In reversed incremental backup mode, retrying previously failed backup of newly added virtual disk makes the created, and the following restore points for this virtual disk invalid.
  • Linux version of standalone backup extraction tool fails to restore v7 backups with the following error: “Declared number of bank pages (XXXXX) is exceed limit (1024)”.
  • Network throttling rules are not applied if the “Use multiple upload streams option” is disabled.
  • Under certain circumstances, the reported processing rate may be much lower than actual with parallel processing enabled.
  • Added ability to manually configure more than 100 ports to be used for data movers. This may be needed when running more than 100 jobs concurrently.
  • Deleted VM retention option is now disabled by default for the newly created jobs, and its logic has been enhanced to better handle job failures.

VMware

  • Jobs fail to remove VM snapshots if vCenter server is added to Managed Servers in B&R user interface both as a Windows server, and as a vCenter server.
  • Replication jobs do not use multiple upload streams for the first processed disk of each VM. This leads to either reduced performance, or network throttling rules not applying during processing of the first disk.
  • Maximum concurrent snapshot removal settings are ignored, limiting VM snapshot removal to one at a time within a job with parallel processing enabled.
  • The source VM snapshot is now first to be removed during in replication jobs. This helps to reduce the time VM runs off snapshot.
  • Powered on source VMs are now shut down automatically to streamline self-service full VM restores.

Hyper-V

  • Instant VM Recovery fails when dynamic VHD disk size is not a multiple of 256 MB. Such disks cannot be created with Hyper-V management tools, but can appear as a result of P2V/V2V conversion, or produced by 3rd party tools.
  • Instant VM Recovery fails when target Hyper-V host’s storage uses native 4KB sector size disks.
  • Under certain circumstances, Instant VM Recovery may fail with the following error: “CVeeamFCTRegionDynamicVHD::RequestProcess: Failed to read needed portion of data, Details: IDriverRequest::DataSet: Too much data”
  • Adding a new virtual disk to the original VM makes the created, and the following replica restore point for the given virtual disk invalid.
  • Changing virtual disk exclusion settings on the existing replication job makes failover fail with the “Object reference not set to an instance of an object”.
  • Failure to rescan any host also fails the remaining hosts rescan with the following error: “Failed to execute WMI Query”
  • Restoring VHDX into the root of CSV folder makes the restored VM fail to start with the following error: “General access denied error”
  • SureBackup testing of VMs with SCVMM-specific network settings may fail with the following error: “Failed to import virtual machine VMNAME configuration”
  • Application-aware processing for guest OS without Microsoft VSS support is now skipped automatically regardless of the job’s settings.

Application-aware processing

  • Backing up all Active Directory domain controllers at the same time (which is something parallel processing made possible to achieve) may lead to DFS-R SYSVOL replication failures.
  • Changed Microsoft SharePoint farm topology collection failure event from warning to informational.

Backup Copy

  • Multiple enhancements to Backup Copy job’s email notifications and statistics.
  • Added ability to disable creating weekly full backups for GFS (Grandfather-Father-Son) retention policy.

Guest indexing

  • Temporary files produced by guest file system indexing process are not correctly removed from AppData\Local\Temp.

HP Storage Snapshots Integration

  • Initiating Active Full backup on the job with existing backups renders the created and the following backups in the backup chain unrecoverable.
  • Incremental replication runs fail with the “The method or operation is not implemented” error.
  • VM snapshots should now be created concurrently in scenarios when the job setup allows to backup multiple VMs from the same LUN. This decreases the time VMs will run off snapshot, and improves overall job performance.

SureBackup

  • Under certain circumstances, parsing route.exe output may fail. As a result, virtual lab creation process and SureBackup jobs may “hang” indefinitely.

Tape

  • Changer alerts may be incorrectly interpreted as drive alerts.
  • Tape drives appear online even when the corresponding tape library is offline.
  • When the “Continue media set” option is enabled, but the last tape is offline, the job fails with the following error: “Error Tape not exchanged”.
  • Restoring VM from tape fails with the “The directory name is invalid” error if the source VM backup job’s name contains invalid NTFS symbols (\|/*:?<>).
  • Eject media once the job finishes” option now applies to both full and incremental job runs.
  • Added support for ejecting media once the job finishes to standalone drives. Previously, attempting to use this option with standalone drives resulted in the “Automated library not found” error.
  • Tape job email report now includes barcodes and names of all tapes that were used in the corresponding job session.

User interface

  • Deleting a single imported backup that is located in the root folder of the backup repository deletes the whole repository instead.
  • Improved Microsoft Exchange server autodiscovery logic in restore to original location wizard of Veeam Explorer for Microsoft Exchange.
  • Veeam Explorer for Microsoft SharePoint crashes when desktop background changes (such as when switching to the new RDP session).

Other

  • PuTTY has been updated to the more recent version for security reasons.
  • Configuration backup fails with the “Value cannot be null” error if SMTP credentials in the email notification settings are not specified.

To obtain the patch please download it from: Download


Veeam announces latest features v7

Last week Veeam announces the latest new big features in there new to release Veeam Backup & Replication v7 backup application.

This release is a big one with tons of new features and fixes. Veeam started in February with announcing the first new big feature and continued to do this every month. This is a summary of all the big new features with some extra information.

01. vCloud Director Support.

Veeam Reveals Enhanced Backup and Recovery for vCloud Director: Using the vCD API, Veeam will display the vCD infrastructure directly in Veeam Backup & Replication™, backup all vApp metadata and attributes, restore vApps and VMs directly to vCD, and support restore of fast-provisioned VMs. 

Continue reading

Solving Veeam Error: Host with uuid was not found

Today I was upgrading an VMware ESX 4.1 host to VMware ESXi 5.1. I did the update with the VMware Update Manager. After a successful upgrade I tried to backup a VM with Veeam Backup & Replication 6.5. This resulted in an error. See screenshot.


How to solve this problem. I found this easy solution in the Veeam Support Forums.

Go to the license screen in Veeam Backup & Replication. Revoke the license of the ESXi host, and start the backup again. If all went well, the ESXi host gets a new Veeam license assigned and the Veeam backup is successful.

Problem solved!!

Extra information can be found: