Category Archives: Veeam

Veeam Jobs fail after adding new ESXi hosts

When you add new VMware Hosts to a VMware vCenter Cluster, please remember to do a rescan of vCenter in Veeam. Veeam now rescans the complete environment and adds the new ESXi hosts to its database so everything will work correctly.

My problem was after I placed new VMware ESXi hosts in the VMware vCenter cluster, the backup was failing on every VM that was placed on the new ESXi hosts. Veeam had detected the new VMware ESXi hosts, so I thought everything was good.

My Error message was: “Failed to create processing task for VM [vm name] Error: Host ‘ESXi host’ with type ‘ESX’ and parentId ‘e2afc39e-f38c-466b-9431-cd1b7aedc48b’ not found.”

A quick google action on this error message resulted in a Veeam Forum post. See http://forums.veeam.com/vmware-vsphere-f24/problem-with-backups-after-adding-new-host-to-vcenter-t18512.html

User v.Eremin is saying in his post:

As long as VMs are added to backup console via vCenter, backup server should be vMotion aware and you shouldn’t encounter any issues after VM migration. However, it seems that vCenter added to backup console didn’t know about the new host and about the fact that certain VMs had been migrated to it, thus, backup jobs failed. So, now, when the rescan is done, everything should be ok, even without manual re-adding of VMs.

This gave me the tip to first do a rescan of vCenter to see if this solved my problem. And this worked for me, thanks Veeam Forums!

Veeam Backup & Replication 8 Update 2 released

Leave a reply

Yesterday Veeam launched update 2 for the Veeam Backup & Replication 8 application. With this update Veeam is fully compatible with VMware vSphere 6.

There are the release notes.

Engine

Job performance improvements. A number of optimizations focused on reducing the time jobs spend performing auxiliary tasks should significantly reduce the overall job execution time for incremental runs of backup and Backup Copy jobs.
Slow backup storage optimization. Target data mover now caches metadata from backup files, instead of requesting it from backup files residing multiple times over the course of the job run. This significantly improves performance of jobs targeting storage with poor random I/O performance (such as EMC Data Domain), while reducing the overall storage load. Important notes:
- The cache is only enabled when using a 64-bit OS on backup repository (or gateway server), and increases RAM consumption of each job by 2GB on average, depending on backup chain size. Using non-default backup block size changes cache RAM requirements proportionally.
- Using the default “Automatic selection” gateway server setting for Shared folder, EMC DataDomain or HP StoreOnce based backup repositories disables the cache.
- Disabling built-in deduplication disables the cache.
64-bit Linux data mover. For increased scalability, a 64-bit data mover will now be used on 64-bit Linux backup repositories with an OS kernel version of 2.6.18 or later.
Linux data mover update. To enable new functionality, jobs will leverage the new version of the data mover on Linux servers with kernel version 2.6 or later. For compatibility with existing servers, jobs will use a legacy data mover when OS kernel version is earlier than 2.6 or cannot be detected.
Direct data mover communication. When both data movers are running on the same server (e.g., when backing up to a local storage attached to a backup proxy server, etc.), they will now exchange data through shared memory. This should improve data transfer performance of local backup jobs currently reporting Network as the bottleneck, or when you see high load on some of the backup proxy server NICs when the data was supposed to stay local to the server.
vPower NFS performance. Increased scalability of vPower NFS server by significantly reducing CPU usage associated with guest I/O operations using small data blocks
VeeamZIP performance. Improved VeeamZIP performance when using a Veeam backup repository as the target.
Replication from backup file enhancements. Replication from backup files no longer requires matching block size between the backup file and replica job settings

VMware

vSphere 6 support. Added support for ESXi 6.0 and vCenter Server 6.0 (including vCenter Server Appliance).
vCloud Director 5.6 support. Added support for backup and restore of vCloud Director 5.6 virtual machines (VMs) and vApps.
VMware Virtual Volumes (VVols) support. VMs residing on virtual volumes can be processed in Virtual Appliance (Hot Add) and Network (NBD) processing modes. Hot add processing mode requires that all proxy VM disks are located on the same virtual volume with the processed VM.
VMware Virtual SAN (VSAN) 2.0 support. VMs residing on VSAN can be backed up in Virtual Appliance (Hot Add) and Network (NBD) processing modes. For VMs with existing snapshots, hot add processing requires that all proxy VM disks are located on the same VSAN datastore.
Storage Policy-Based Management (SPBM) policy backup and restore. Storage policy associations are backed up and restored for each virtual disk upon full VM restore. This eliminates the manual process, which directly impacts recovery time. SPBM policies are important to restore because “out of policy” VMs can impact availability of either the restored VM itself or other VMs sharing the same storage.
Support for Fault Tolerant VMs (FT VMs) for backups and replicas. VMs with Fault Tolerance enabled can now be backed up and replicated. Legacy Fault Tolerant VMs must have Fault Tolerance re-enabled after vSphere 6 upgrade using vSphere Web Client to enable this capability.
vSphere 6 tags integration. vSphere 6 introduces new APIs for programmatic access and management of vSphere tags. With vSphere 6 tag support, you can continue building new advanced backup policies based on tags, even after you upgrade to vSphere 6. Due to new unique tag IDs in vSphere 6, be sure to review your job setup after you upgrade to vCenter Server 6.0.
Cross-vCenter vMotion awareness. Added support for cross-vCenter vMotion to the Quick Migration functionality. Now, when migrating a VM to another vCenter Server, the associated entries on backup or replication jobs will be updated automatically to keep the VM protected
Quick Migration to VVols. Enables moving VMs to VVols when using vMotion is not an option due to unreliable or slow network links, in scenarios when vMotion is not supported, or due to lack of VMware licensing. This functionality can help perform full migrations to new vSphere 6 clusters built from the ground up without inheriting new design problems from previous clusters.
Hot-Add transport mode of SATA virtual disks. On ESXi 6.0, SATA virtual disks can now be processed in Virtual Appliance (Hot Add) processing mode, which was previously limited to SCSI virtual disks only.
vSphere tags priority. vSphere tags priority has been raised one level above containers priority to ensure that jobs behave in line with customers’ expectations when job’s inclusion and/or exclusion lists use both containers and tags.
Disable CBT reset. The workaround for VMware Changed Block Tracking (CBT) corruption issue at 128GB boundaries (VMware KB2090639) can now be disabled by customers with ESXi hosts patched against this issue. To disable automatic CBT reset upon virtual disk size change, create theResetCBTOnDiskResize (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set it to 0.

Storage integration

Granular rescan settings. Added ability to control which storage LUNs and volumes should be rescanned periodically for new VMs. These settings can be accessed directly from the Add Storage wizard while registering the storage, or by right-clicking the already added storage in the Storage Infrastructure tab.
NetApp integration enhancements. The following NetApp-specific enhancements have been added:
- Added support for triggering updates of non-qtree and vFiler-based SnapMirror and SnapVault
- Added support for backing up VMs from NetApp MetroCluster in the failed over state
- Preferred Network settings of Global Network Traffic Rules dialog now also used to pick preferred storage adapters for data transfer. And in cases when multiple adapters use the same subnet, NetAppOrderedIPList (STRING) registry value underHKLM\SOFTWARE\Veeam\Veeam Backup and Replication can now be created to specify preferred adapters’ IP addresses separated by a semicolon.

Hyper-V

SMB3 restore performance. Improved performance of full VM restore to SMB3 shares with caching disabled (such as in Nutanix).
Hyper-V Integration Services status check. To prevent situations with some VMs unexpectedly going into the Saved State during a volume snapshot creation, jobs will now check that Hyper-V Integration Services (HIS) are running and reachable, waiting for HIS to respond for up to 10 minutes before failing the VM processing.
Replica VM adjustment. Maximum RAM value of the replica VM is now automatically reduced according to the target host capabilities.

File-Level Recovery

Native 4K disk support. Windows File-Level Recovery now supports mounting native 4K virtual disks from backups out of the box, without requiring a user to switch to the mount engine via the corresponding registry value.
Linux ACL preservation. Multi-OS File-Level Recovery now preserves Linux ACL when restoring a file via Restore and Copy To operations. This requires that both backed up and destination VM runs Linux with kernel version 2.6 or later.
Linux host selection. Multi-OS File-Level Recovery now allows choice of a target Linux host to restore guest files to without having to add it to the managed servers first.
Restored files logging. Multi-OS File-Level Recovery now logs restored files in its session log, similar to the way Windows File-Level Recovery performs its restore activity logging.
SNMP traps. Both Windows and Multi-OS File-Level Recovery can now trigger SNMP notifications. To enable such notifications, create theEnableRestoreSNMPTraps (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set it to 1.

Veeam Explorer for Microsoft SQL Server

Improved restore performance. Database restore performance has been improved by up to 3x.
Increased scalability. Reduced load on SQL Server hosting Veeam® Backup & Replication™ database in scenarios when transaction logs are being backed up from multiple SQL servers on a frequent schedule.
Log truncation retry. In cases when log truncation fails under the specified guest processing account, it will be retried from the LOCAL SYSTEM account, which often carries the necessary privileges to perform this operation.
SQL database restore. Added a warning when restoring a SQL database to a different location where the database with the same name already exists when using the Enterprise Manager web UI
Improved issues reporting. Backup job now detects and reports more issues around transaction log processing, such as failure to truncate logs. This may result in the appearance of a new warning after the update, simply because existing issues were not previously reported.

Veeam Cloud Connect

Multiple subscriptions support. Added ability to add the same service provider multiple times using different credentials.
Reconnect performance improvements. Reconnecting to a service provider should now occur much faster when a cloud provider has multiple Cloud Gateways deployed.
Enhancements for service providers. The following are enhancements relevant to Veeam Cloud Connect service providers only:
- Added ability to granularly purge global cache from WAN accelerator for the specific tenant.
- When deleting a tenant, global cache belonging to the tenant is now deleted automatically.
- Added ability to query the amount of protected VMs for the given tenant through RESTful API using the vmCount property.
- Service Providers node of the management tree is always shown when Cloud Connect Service Provider license is installed, without requiring the service provider to register at least one hypervisor host first. This enables service providers to more easily client consoles to user for testing or manipulating client backups.
- Enhanced debug logging to include initiator information for when the connection is terminated.

Tape

Job performance. File to Tape and Backup to Tape job performance was further improved.
Synthesized full backup enhancements. Synthesized full backup functionality has been reworked for better reliability, performance and support for very large backup files.
Third-party tape handling. Added support for detecting and reusing tapes written by third-party tape backup vendors.

User Interface

Manual job chain execution. By popular demand, when manually starting a job that has other jobs chained to it via “After this job” scheduling option, you will now be offered a choice between starting the selected job alone, or starting the entire chain.
Disable network traffic rules. Both traffic throttling and encryption can now be temporarily disabled without requiring you to delete the corresponding rule completely, which is useful for troubleshooting or for temporarily removing bandwidth consumption restrictions.
Improved guest credentials test. Guest processing credential test will now additional verify the presence of mlocate, which is required for Linux guest file system indexing.

Setup

Silent install. Starting with Update 2, updates will fully support automated silent install, including triggering update of remote backup infrastructure components. This should significantly reduce the time required to update Veeam deployments with a large number of backup servers.

PowerShell

Support in Free Edition. Start-VBRZip cmdlet is now enabled in Veeam Backup™ Free Edition, allowing users to schedule periodic VeeamZIP™ backups using external task schedulers, such as Windows Task Scheduler. This should make Veeam Backup Free Edition a viable alternative to ghettoVCB and other similar script-based free backup solutions.

Integration with Veeam Endpoint Backup FREE

This update enables you to take full advantage of Veeam backup repositories as additional target locations for your Veeam Endpoint Backup jobs, with backups for each endpoint being stored in a separate folder named after the endpoint.

To assign end-user permissions to individual Veeam backup repositories, use the new repository properties dialog that will appear in the repository’s shortcut menu upon first connection attempt from Veeam Endpoint Backup™ FREE to the Veeam Backup & Replication server. Before that, this dialog can be accessed by Ctrl-right-clicking the backup repository.

Global network traffic rules defined in Veeam Backup & Replication apply to endpoint backup jobs using Veeam backup repository as the target, allowing you to ensure that concurrent backups from multiple endpoints do not impact your available bandwidth, or that endpoint backup network traffic is encrypted.

Using a Veeam backup repository as the target for your endpoint backup jobs enables the following additional functionality:

Centralized monitoring. Perform basic monitoring and management for all incoming endpoint backup jobs, including email notifications about endpoint backup status.
Offsite protection. Get your endpoint backups off site to disk-based storage, tape or your Veeam Cloud Connect service provider with Backup Copy and Backup to Tape jobs.
Backup encryption. You can choose to optionally encrypt your endpoint backups stored in Veeam backup repositories. All standard Veeam encryption features are supported for endpoint backups, including password loss protection.
Application item-level recovery. Restore application items from backups of server machines with Veeam Explorers™ for Microsoft Active Directory, Exchange, SharePoint and SQL Server.
Disk export. Export physical disk content from backup into VMDK/VHD/VHDX virtual disk files

To obtain this update, please click here (you need to be logged in to download the update).

Veeam Backup & Replication v8 Patch 1 released

Leave a reply

Last week Veeam released the first patch voor Veeam Backup & Replication v8. So what is new and fixed?

These are the release notes.

New Features and Enhancements

General

Retention policy for replication jobs is now processed concurrently for multiple VMs in the job, as opposed to one by one, reducing the overall job run time.
Added email notification about File to Tape and Backup to Tape jobs waiting for user interaction (for example, when a tape medium needs to be inserted by the user)
VMware Tools quiescence should now log an informational event when VMware Tools are outdated, as opposed to a warning event.
Independent virtual disks which are explicitly excluded from processing in the job settings should no longer log a warning during the first time when the given VM is being backed up.
Virtual Lab proxy appliance was upgraded to virtual hardware version 7 to allow for it to manage more networks out of box.
Jobs should no longer appear to “hang doing nothing” in some circumstances, with the duration not ticking for any event in the job log.

Cloud Connect

Cloud backup repository size limit of 63TB has been removed

Linux support enhancements

Added support for file level recovery from XFS volumes (default file system in Red Hat Enterprise Linux and CentOS distributions starting from version 7.0)
Added support for file level recovery from Btfrs volumes (default file system in SUSE Linux Enterprise Server starting from version 12.0).
Added support for file level recovery from VMs with more than 10 LVMs.
Improved Linux guest file system indexing performance for incremental backup job runs.
FTP access mode of the file level recovery helper appliance now also displays friendly LVM volume names introduced in v8.
To enable identity verification of the remote server, SSH key fingerprint is now displayed to a user when registering the new Linux server. Accepted SSH keys are stored in the configuration database to protect from MITM attack.

Performance enhancements

Added experimental support for direct data movers communication when both are running on the same server (for example, when backing up to a local storage on backup proxy server). If your local backup jobs report Network as the bottleneck, and you see high load on some backup proxy server NICs when the data was supposed to stay local to the server, you are likely to benefit from this behavior modifier. To enable alternative data exchange path, create the DataMoverLocalFastPath (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set it to the following values:

0: Default behavior (no optimizations)

1: Data exchange through TCP socket on the loopback interface (faster)

2: Data exchange through shared memory (fastest)

EMC Data Domain integration

Synthetic full backup creation and transformation performance has been improved significantly.

HP 3PAR StoreServ integration

Remote Copy snapshots are now automatically excluded from rescan, both speeding up the rescan process and reducing storage load.

NetApp integration

Added support for NetApp Data ONTAP 8.3.
Added support for HA Pair 7-mode configuration.
Added support for vFiler DR configuration.

Resolved Issues

General

Restored incremental backup job performance back to v7 levels by removing unnecessary metadata queries from previous backup files.
Multiple issues when parallel processing is disabled (high CPU usage on backup server, wrong task processing order, reversed incremental backup jobs with disk excluded failing to access backup files).
Automatic network traffic encryption (based on public IP address presence) is enabled even when both data movers are running on the same computer, reducing job’s processing performance.
Quick Backup operation triggers jobs chained to the job it uses.
Loss password recovery option does not function correctly when Enterprise Manager has multiple backup server registered.
Under rare circumstances, backup job with encryption enabled may fail with the “Cannot resize block data is encrypted” error.
When multiple network traffic rules having the same source and target ranges are set up, changing settings to one of the rules also updates all other rules.

vSphere

In vSphere 5.5, enabling virtual disk updates redirection to the VMFS datastore in Instant VM Recovery settings, and then migrating a recovered VM to the same datastore with Storage vMotion causes data loss of data generated by the running VM.
Full VM restore process puts all virtual disks to the datastore selected for VM configuration file,ignoring virtual disks placement settings specified in the wizard.
In certain environments, jobs fail to process VMs in Direct SAN Access mode with the “Failed to create processing task for VM Error: Disk not found” error.
Dismounting processed virtual disks from backup proxy server takes longer than expected in the Virtual Appliance (hot add) processing mode.
Rescan of vCenter server containing hosts without HBA and SCSI adaptors fails with the “Object Reference not set to an instance of an object” error.
Replication jobs to a cluster fail with the “The operation is not allowed in the current state” error if chosen cluster host is in the maintenance mode.
Under rare circumstances, jobs with Backup I/O Control enabled may occasionally log the “Operation is RetrievePropertiesOperation” related errors.
Enabling Backup I/O Control limits snapshot removal tasks to one per datastore regardless of datastore latency levels.
VM Copy jobs always fail over to the Network (NBD) processing mode.
VMs with virtual disks having the same names but with different capitalization, and located on the same datastore cannot be backed up.
Backing up vCloud Director vApp to a CIFS share may fail with the “Object Reference not set to an instance of an object” error.
Snapshot Hunter ignores backup window specified in the corresponding job.
vCloud Director VMs created from linked clone template and with a user snapshot present incorrectly trigger Snapshot Hunter.

Hyper-V

CBT does not track changes correctly on VHDX disks files larger than 2 TB.
CBT may return incorrect changed blocks information for differential disks, when parent and child disks have the same name.
Large amount of unmapped VHDX blocks may cause the job to failover to full scan incremental backup with the “Failed to update unaccounted changes for disk. Change tracking is disabled” error.
Existing Hyper-V backup and replication jobs processing VM with SCSI disks start to consume x2 space on target storage after upgrade to v8.
Removing a node from a cluster does not remove the corresponding node from the cluster on the Backup Infrastructure tab of the management tree until the next periodic infrastructure rescan (in up to 4 hours).
Replication from backup fails with the “Virtual Hard Disk file not found[/id” error if backup job that created this backup had one or more disks excluded from processing.
Under rare circumstances, legacy replicas using non-default block size may fail with the “[i]FIB block points to block located outside the patched FIB” or “Block offset [] does not match block ID []” error messages.
Offhost backup from Windows Server 2012 R2 fails with the “Exception of type ‘Veeam.Backup.AgentProvider.AgentClosedException’ was thrown” error when processing a VM that can only be backed up in Saved or Crash-Consistent states, and native Hyper-V quiescence is enabled in the job settings.
If a Hyper-V cluster node goes offline during backup from shared volume snapshot, and does not recover until the Veeam job fails, CSV volume backup infrastructure resource will not be released by the scheduler. As the result, other jobs waiting to process VMs from the same volume may remain hanging in the “Resource not ready: Snapshot” state.
Replication job fails with the “Cannot assign the specified number of processors for virtual machine” error for VMs with core count larger than the one of the target Hyper-V host. Legacy replication job works fine in the same circumstances, however VM failover is not possible.

Application-aware processing

VeeamGuestHelper process crashes when backing up Windows 2003 SP2 VM running Oracle.
Application-aware processing hangs for extended time when processing a Windows VM containing one or more Oracle databases in the suspended state.
Certain advanced guest processing settings lead to VM processing failing with the “Unknown indexing mode ‘None’ ” error.

Multi-OS File Level Recovery

Restore to original location does not work in certain LVM configurations.

Enterprise Manager

Under certain circumstances, Enterprise Manager data collection may fail with the “Cannot insert the value NULL into column ‘guest_os’ ” error.
Restore scope rebuild operation tries to reach out to the hosts already removed from the corresponding backup server, resulting in login errors.

Built-in WAN Acceleration

WAN accelerators fail to detect and populate global cache Windows Server 2012 R2 and Windows 8.1 OS data, which may result in reduced data reduction ratios.

Cloud Connect

The Decompress backup data blocks before storing backup repository option is ignored for Cloud Connect backend repositories.
Backup Copy jobs with encryption enabled fail with the “Attempted to compress encrypted data-block” error if backend repository has the Decompress backup data blocks before storing option enabled.
Veeam Backup Service crashes with the “An existing connection was forcibly closed by the remote host” if network connection fails immediately upon establishing the initial connection to the Cloud Gateway.
Cloud Connect quotas are rounded incorrectly in the service provider’s interface.
Tenant statistics data such as data sent/received is only shown for the last 24 hours.

Veeam Explorer for Active Directory

“This implementation is not part of the Windows Platform FIPS validated cryptographic algorithms” error is displayed if Use FIPS compliant algorithms for encryption, hashing, and signing Group Policy setting is enabled.
Attempting to open single label domain’s AD database fails with the “No writable domain found” error.
Attempting to open third level domain’s AD database fails with the “No writable domain found” error.
Under certain circumstances, object restore may fail with the “Unable to cast object of type ‘System.String’ to type ‘System.Byte[]’ “error.

Veeam Explorer for Exchange

Process of exporting email messages cannot be stopped.
Copying or saving attachments from multiple different emails always copies or saves attachment from the first email for which this operation was performed.
Opening public folder’s mailbox fails with the “Unable to find wastebasket” error.

Veeam Explorer for SQL Server

Restoring to SQL instance with case sensitive collation enabled fails with the “The multi-part identifier “a.database_id” could not be bound” error.

Backup from Storage Snapshots

Backup jobs from storage snapshots on iSCSI SAN take too long to initialize when large amount of iSCSI targets are configured on the backup proxy server.
Under certain circumstances, backup jobs from storage snapshots on FC SAN may fail with the “Unable to find volume with Node WWN and LUN ID” error.

HP 3PAR integration

Rescan of 3PAR array with empty host sets fails with the “Object reference not set to an instance of an object” error.

NetApp integration

Under certain circumstances, storage rescan process may leave temporary storage snapshots behind.

Tape

Backup to Tape jobs may appear to run slower after upgrading to v8 because of copying unnecessary backup files (VRB) to tape when source backup job uses reversed incremental backup mode.
On standalone drives, tape jobs configured to eject tape upon completion fail the “Object reference not set to an instance of an object” error after 10 minutes of waiting for the user to insert tape.
Tape job configured to use full and incremental media pools hosted by the different standalone tape drives fails when processing incremental backup with the “Value cannot be null” error. Tapes are formatted with the block size of tape medium, as opposed to the default block size of the tape drive.
Tape job configured to use full and incremental media pools hosted by the different tape libraries fails while waiting for an incremental pool’s tape with the “Tape not exchanged” error
Tape jobs fail with the “GetTapeHeader failed” error on IBM TS3100 tape library with Path failover feature disabled.

PowerShell

Objects obtained with Find-VBRViFolder cmdlet have incorrect Path attribute.
Get-VBRTapeJob cmdlet fails to obtain job object for tape jobs with incremental processing disabled.
Get-VBRTapeLibrary cmdlet fails with the Standard Edition license.
All built-in WAN accelerator management cmdlets have been made available in all paid product editions

To obtain the patch, please click here

Veeam VMCE7 to VMCE8 upgrade path

1 Reply

A couple of months ago Veeam launched there VMCE certification. This certification is valid for the Veeam Backup & Replication V7 version. Now that Veeam is almost launching their new flagship version 8 all the VMCE certified engineers should upgrade their certification.

Veeam released a document that describes the possible upgrade paths. For more information see document: VMCE7 to VMCE8: Upgrade Rules

There can be two scenarios:

A student with no existing VMCE certificate (a new student).
A student with the VMCE 7 certification.

New students

When the VMCE 8 program is launched, any student eager to get VMCE-certified but who have not completed the VMCE 7 certification, will need to:

Attend the VMCE 8 classroom through Instructor-Led Training (ILT) or Video-on-Demand Learning (VODL) at one of the available Veeam Authorized Education Centers (VMAEC). Note: VODL is available in North America only.
Pass the VMCE 8 proctored exam at an available Pearson Vue test center.

These students may exceptionally be authorized to follow the free online “VMCE 8 – What’s New” course on Veeam.com, and then validate their VMCE certification with a VMCE 8 proctored exam at an available Pearson Vue test center.

VMCE 7 Certified Engineers

Students who complete the VMCE 7 certification program will be entitled, with a specific restricted access, to the free online “VMCE 8 – What’s New” course on the Veeam.com website.

These students will then need to pass the VMCE 8 proctored exam at an available Pearson Vue test center to upgrade their certificate from VMCE 7 to VMCE 8. This must occur in the allowable certification maintenance time according to the certification rules below.

Students who attend the VMCE 7 ILT or the VODL, but who do not complete the VMCE 7 exam, will exceptionally be authorized to follow the free online “VMCE 8 – What’s New” course on the Veeam.com website, and then validate their VMCE certification with a VMCE 8 proctored exam at an available Pearson Vue test center.

Certification Maintenance: Reminder from the VMCE Program Guide

All VMCEs are required to stay current on knowledge and certifications related to Veeam Software. This includes re-certification on all new major releases of Veeam Software. VMCEs will have 120 days, after the release of a major software update and the release of associated upgrade courses and exams, to take the course and pass the exam necessary to upgrade the certification level.

If VMCEs fail to upgrade in the allotted time, there will be another 90 days under which they must take the full course and pass the full exam. After this time, if any VMCEs have still not been upgraded, the certification will expire and the engineer will no longer be eligible for the benefits of the VMCE program.
Source: VMCE7 to VMCE8: Upgrade Rules

More information about the VMCE Certification, see the Veeam Website.

Veeam Backup & Replication setup or upgrade not starting

Leave a reply

A couple of weeks ago I tried to do an upgrade of the Veeam Backup & Replication 6.1 to v7.0. I mounted the fresh downloaded ISO of Veeam v7.0. The splash screen showed, and when I clicked on the Install / Upgrade options nothing happened.

This server is running Windows 2008R2 Service Pack 1, and is also the VMware vCenter server for the ESXi environment.

Because this server is a vCenter Server and purely a management server this server didn’t had all the Windows Updates installed that are available today. So my first guess was to update the complete server. After updating this server and a fresh reboot, I started the installation again. No success, still the same behavior. Splash screen disappears and reappears again without starting the Veeam installation.

There are also no errors in the event logs of the server. Always nice to trouble shoot…

When installing the Microsoft System CLR Types for SQL components there is an error.

So there was something wrong with the Windows environment, but everything was working like it should. I already checked all the perquisites, and everything was installed. I didn’t know where to look for, so after some time I contacted Veeam support. They checked the environment and reinstalled the .Net Framework. After this installation the Veeam installer was working correctly. Problem was a corrupted or damaged .Net framework installation.

Lessons learned:

Veeam Support is awesome, they contacted me very quickly and solved my problem within an hour or so.
When running into strange behavior with the installer, try to repair or reinstall the .Net Framework it might solve your problems.

New free tool: Veeam Task Manager for Hyper-V

Leave a reply

Today Veeam released a new free tool for your Hyper-V environment called “Veeam Task Manager for Hyper-V”

Veeam Task Manager for Hyper-V is a portable, standalone performance monitoring tool that’s FREE!

Improve troubleshooting in your Hyper-V environment by seeing what Windows Task Manager doesn’t show you. Veeam Task Manager for Hyper-V displays a real-time view of both host and VM compute resource consumption.

Get the critical visibility you need.

This lightweight tool is portable so you can run it from any USB device for emergency troubleshooting. No installation or integration needed!

More information:

Download the FREE Veeam Task Manager today

Veeam Backup & Replication v7 Patch 4 released

Leave a reply

I noticed on twitter that Veeam released Patch 4 for Veeam B&R v7. So what is fixed?

New Features and Enhancements

VMware Virtual SAN (VSAN)

In addition to adding basic support (as provided by other vendors), the intelligent load-balancing engine was enhanced to account for VSAN specifics. As the result, for each VM the job will pick backup proxy running on VSAN cluster node with most of the virtual disks’ data available locally. This significantly reduces backup traffic on VSAN cluster network, resulting in minimum possible impact on production environment from backup activities.

Microsoft SQL Server 2014

Added support for Microsoft SQL Server 2014 both as the protected guest workload (including application-aware processing functionality), and the back-end database for backup and Enterprise Manager servers.

License key auto update

Added automated license key update option to the License Information dialog. With auto-update enabled, the product will check Veeam licensing server periodically for an updated license key, and download and install the key automatically as soon as it becomes available. This feature is particularly useful to the Service Providers and subscription-based customers, and it removes the need to download and install the license key manually each time when the license extension is purchased.

Backup Copy

The maximum allowed amount of restore points in the Backup Copy job has been increased to 999.
Backup Copy will now resume the transfer after network connection drop to a Linux-based backup repository.
Backup Copy jobs should no longer report errors during the days when source backup jobs are not scheduled to run – for example, during the weekend.

Hyper-V

Added support for certain Hardware VSS Providers that previously could not be detected by the storage rescan process, and as such could not be used by the jobs.
Jobs will now retry failed snapshot creation when another shadow copy of the same volume is already in progress, instead of immediately failing to process a VM.

Resolved Issues

General

Under certain circumstances, the target data mover may crash with the “Cannot allocate memory for an array” error while processing a backup job with WAN storage optimization setting.
Increased maximum amount of open files for Linux data mover to improve backup reliability in cases with a very large amount of concurrent tasks hitting the same repository.

vSphere

If the same ESX(i) host is added to Managed Servers both as a part of the vCenter, and as a standalone host, the jobs fail to remove VM snapshots correctly.
Replication job does not truncate Microsoft Exchange transaction logs when the corresponding setting is set to “Truncate logs on successful backup only”.
Setting EnableSameHostHotaddMode registry value (introduced in Patch #3) to 1 causes replication jobs to fail with the “Object reference not set to an instance of an object” error.
If you clone replica VM with vSphere, and start the replication job, the target VM will be selected randomly between the two VMs.

Hyper-V

Issues introduced with Windows Server 2012 R2 Update (April 2014 rollup), see KB1863 for more info.
Very large amount of parallel tasks may cause in significant delays (up to a few minutes) between assigning required backup infrastructure resources to the job, and the job actually starting to leverage those resources.
Replication jobs do not truncate Microsoft Exchange transaction logs regardless of the corresponding settings state.
Processing of virtual machines with long dash symbol in the file name fails with the “The system cannot find the path specified.\n\nFailed to open file [C:\ClusterStorage\FOLDER\GUID.xml] in readonly mode. (System.Runtime.InteropServices.COMException)” error.
Parallel snapshots of SMB3 share may fail with the “Failed to add volume to the VSS snapshot set. Another shadow copy creation is already in progress. Wait a few moments and try again” error.
Restoring a VM to localized Hyper-V cluster creates duplicate cluster resources remaining in the Starting state in Failover Cluster Manager.

Application-aware processing

During full VM restore and SureBackup job, Domain Controllers may spend longer than required in the DSRM mode (up to 10 minutes – until the corresponding timeout expires), thus unnecessarily delaying DC recovery.
Increased timeouts for network-less application aware processing of vSphere guests to improve the chance of processing completing successfully.

Backup Copy

Backup Copy configured with multiple backup jobs as the source does not include VMs which are excluded in one source backup job, but included in another.
Health Check process continuously fails with the “Timed out waiting for the backup repository resource to become available for backup integrity check” error during the periods when the Backup Copy job is restricted from using network bandwidth.
Backup Copy is incorrectly marked as failed despite completing successfully when source or target WAN accelerator becomes unavailable (for example, due to the network issue and reboot) during the period when Backup Copy job is waiting for its availability.

Backup from Storage Snapshots

Disabling failover to network processing mode on a backup proxy causes intelligent load balancing not to assign the backup proxy to jobs running with storage snapshots integration enabled.
Under certain circumstances, backup from 3PAR volumes may fail with the “LUN is already taken for set ESXs” error.

Configuration backup

The presence of null byte in tape changer serial number causes configuration backup to fail.

File Copy

Certain files are skipped with the File Copy job with the “Cannot copy a symbolic link of type Veeam.Backup.Core.CLocalFileItem into a directory of type Veeam.Backup.Core.CRemoteWinFolder” error, however the job reports success. For example, this may happen when copying files from volumes with Windows Server 2012 deduplication enabled.
File Copy job fails when attempting with the “Client error: boost::filesystem::remove: Access is denied” error when attempting to overwrite read-only file on the destination Windows server. The remaining files are not copied.

Native File Level Recovery

Starting file level recovery from the Restore wizard fails with the “Object reference not set to an instance of an object” error when SAN snapshots is picked as the recovery source. Starting the same process from SAN infrastructure tab does not have this issue.
In rare scenarios, file level recovery is unable to parse LDM contents of the virtual disk. As the result, Backup Browser opens empty, without any volumes mounted.
Restoring to original location for volumes which are mounted to a folder (as opposed to a drive letter) fails when such volumes reside on GPT partition or dynamic disk.

Multi-OS FLR

Folder permissions, group and owner IDs are not preserved when restoring a folder with “Preserve permissions and ownership” check box selected.
Multi-OS FLR fails with the “The file specified is not a virtual disk” error on replica VMs with VMDK larger than 2TB in size.

Tape

Depending on where the file is placed on the tape, restore process may not calculate the file size correctly, resulting in corrupted file restored from good backup.
File to Tape job fails to process files with full path of exactly 260 symbols long with the “The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters” error. Files with full path longer or shorter than 260 symbols are processed normally.
Restoring a very large file from tape may fail with the “Value was either too large or too small for an Int64” error.
Restoring a VM from tape may fail with the “Unable to read tape metadata” error.

VMs and Templates View of the virtual infrastructure Browse dialog, and Category View on the Virtual Machines tab do not show vSphere vApps.
Sending email notifications fails with the “Key not valid for use in specified state” error if SMTP credentials were entered by a user account other than Veeam Backup Service account.

Veeam Explorer for Microsoft Exchange

Opening mailbox database may occasionally fail with the “The process cannot access the file ‘C:\PATH\ese.dll’ because it is being used by another process” error.
Mounting mailbox database or transaction log volumes fails with the “Exchange logs cannot be replayed, block size mismatch” error from 512-byte emulation (512e) disks.
In rare scenarios, opening a mailbox from German localized Microsoft Exchange deployment fails with the “JetError -1404, JET_errIndexNotFound, No such index” error.

Veeam Explorer for Microsoft SharePoint

SharePoint restore wizard fails with the “Object reference not set to an instance of an object” error if actual SharePoint configuration does not match the one collected with application-aware processing logic during the backup job.

WAN Acceleration

Increased source WAN accelerator digests retention to 2 months to avoid digests recalculation every week.

Pilot Patch #4 (build 7.0.0.870)

The generally available version of Patch #4 (build 7.0.0.871) also addresses the following issue reported during pilot patch deployments:

Tape inventory fails on certain tape drives with the following error: “Tape fatal error. No more data is on the tape”

This patch also contains all fixes from Patch 1, R2 update and Patch 3

More Information:

Prior to installing this patch please reboot the Veeam server to clear any locks on the Veeam services and when the reboot is done, please stop all the Veeam services and apply the patch.

When the patch is installed manually, start the Veeam services and open Veeam, and allow it to update all the components.

To obtain the patch please download it from Download

Win a trip around the world with Veeam

Leave a reply

Veeam is about to get its 100,000^th customer and is launching an interactive contest for a chance to win a trip around the world and other prizes (Google Glass, iPad and Microsoft Surface).

To participate, visitors need to register and predict the location of Veeam’s 100,000^th customer on the interactive map. The closer you are to the right spot, the better chance you have to win the trip around the world and other prizes.

We currently have a live pre-registration page. The main contest will start next week.

Join in on the fun! We hope you and your readers will participate and celebrate this great achievement with us.

Guess the location here: http://world.veeam.com/

For more information: http://world.veeam.com/veeam_tc_2014.pdf

Veeam Backup & Replication v7 Patch 3 released

Leave a reply

I noticed on twitter that Veeam released Patch 3 for Veeam B&R v7. So what is fixed?

Engine

Significantly reduced load on SQL Server hosting the configuration database, and improved performance of various job activities and user interface operations.

vSphere

Improved vCenter Server hierarchy enumeration logic to speed up vCenter Server registration process in a few times.
Intelligent load balancing can now be configured to give preference to backup proxy located on the same host using the EnableSameHostHotaddMode (DWORD) registry value.
Replication jobs should now fail immediately upon encountering fatal error, rather than wasting time performing additional operations that are no longer relevant.

Hyper-V

Added timeouts on all snapshot operations, including snapshot creation, deletion, import to off-host proxy, and reading snapshot data to prevent conditions with stuck snapshots due to environmental issues.
Maximum volume snapshot life time is now customizable using the hvSnapshotLifeTimeHour (DWORD) registry value. Default value is 72 hours, but customization may be necessary to allow full backup of very large VMs on slow backup hardware.
Added ability to configure SCVMM communication timeouts using the SCVMMConnectionTimeoutMinutes (DWORD)

1-Click FLR

Support for downloading files larger than 4GB.

Built-in WAN acceleration

Data processing performance has been further improved by up to 25% comparing to R2 update.

Multi-OS File level recovery

Added support for file names with Unicode characters.

Tape

Support for backing up files with file names longer than 260 characters, and directory names longer than 248 characters.
Support for devices that do not implement sending error in response to attempt of writing to write protected tape.
Support for devices that do not implement sending the end of media event, such as certain IBM tape drives. Capacity for such drives is now controlled by the job.
Support for devices that do not implement drive enumeration in the drivers correctly. Such tape devices are now additionally queried for drives with SCSI commands.

User interface

Column sizes and sorting rules will now persist for each user in all grids and views.
Disabling GFS retention in Backup Copy job now offers to remove the existing full backup along.
Backup file repair process is now reported with the dedicated job status.

Rotated media

This update adds full support for rotated media with all job types and backup modes removes the need of using additional scripting. The new functionality is controlled by the following registry values:

ForceCreateMissingVBK (DWORD)

Enables support for rotated drives. With this functionality enabled, if any backup file from the latest full backup chain is missing (such as when the existing hard drive is replaced by another one), jobs will start the new backup chain and create the new full backup (instead of failing out).

Set to 1 to enable support for rotated drives.

ForceDeleteBackupFiles (DWORD)

Enables automatic cleanup of the backup repository from ALL existing files if any backup file from the latest full backup chain is missing (such as when the existing hard drive is replaced by the hard drive brought back from offsite location, and containing some older backup files).

Set to 3 to make the job delete the entire contents of the backup job’s folder only.

Set to 5 to make the job delete the entire contents of the root backup repository folder, thus potentially deleting any files belonging to other jobs along. Note that if backup repository points at the volume’s root folder, the entire volume’s contents is erased.

Resolved Issues

Engine

Job may fail VM processing if backup proxy has a very large amount of NICs with the “A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond” error. To work around this, increase the timeout by creating the newly introduced ConnectByIPsTimeoutSec (DWORD) registry value.
Network throttling rule does not apply if some job is already running at the time when the throttling period starts for the given rule.
In some scenarios, target data mover for CIFS-based backup repository starts on the backup server, rather than on the specified proxying server.
Summary.xml growing beyond 16385 bytes results in the following job runs fail with the “Read summary xml error” error.
Running too many parallel task on a single proxy or repository may cause the “Cannot find connections pool by id” error.
Stopping the job on transform day causes resource leak.

vSphere

Processing of VM with virtual disk larget than 2TB fails with the “Requested value ‘seSparse’ was not found. (System.ArgumentException)” error if there is an existing VM snapshot present.
Failback of VM with virtual disk larget than 2TB fails with the “Requested value ‘seSparse’ was not found. (System.ArgumentException)“.
Network remapping functionality of replication job does not function correctly if performed within a single DVS (Distributed Virtual Switch), or when source and target VM are connected to more than one DVS.
VeeamReplicaSummary growing beyond 16385 bytes results in the following job runs fail with the following error: “Failed to create replica vm disk“.
Full VM restore of a VM containing virtual disks with the same names creates a new VM with incorrect configuration if the original VM also had one or more snapshots present at the time of backup. Attempting to start the restored VM results in “Cannot open the disk or one of the snapshot disks it depends on. The system cannot find the file specified” vSphere error.
All jobs now detect and refuse to process VMs with virtual disks disk size not being a multiple of 1KB to prevent issues at restore (“Client error: Declared and actual sizes of the disk are different“) or Backup Copy (“Inline data validation failed“). Such disks cannot be created with vSphere user interfaces, and are usually a product of P2V conversions or similar activities.
Backing up VMs with non-default workingDir parameter from vCenter Server 4.X fails with the “Sequence contains no matching element” error.

Hyper-V

Changed Block Tracking fails on dynamic VHD disk aligned to 1024 bytes with the “Failed to update unaccounted changes for disk. Change tracking is disabled” error.
Replication job with seeding enabled fails with the “Client error: Client either not connected to server or connection is broken”
Added workaround to prevent CSV cache from leaking during backup activities (Microsoft issue).
Under certain circumstances, replication to a cluster may fail with the “The process cannot access the file because it is being used by another process” error.
Stopping a job at certain “bad” time results in hardware VSS snapshot remaining on the storage with the “Failed to close current snapshot” error logged.
Volume rescan fails with the “Server ‘This server’ does not exists or access denied. Host discovery failed. The RPC server is unavailable” error when the product is installed on the Hyper-V host.
Selecting the Run server on this side option on the source host makes replication jobs fail.
Replication job statistics may show incorrect Read and Transferred values upon network disconnect at the end of incremental job run.
Swap exclusion notification is missing from the VM processing log.
Removed case sensitivity when obtaining host names from SCVMM server for better matching reliability with hosts already added to the configuration.

Application-aware processing

Application-aware processing breaks DFSR replication of SYSVOL when all domain controllers are backed up at the same time due to the parallel processing.
Transaction logs are truncated when the VM processing fails due to loss of connectivity with backup proxy even when the job is set to truncating logs on successful backup only.
Improved in-guest VSS snapshot handling to increase backup reliability for Windows Server 2008

Backup Copy

Health check process does not respect concurrent tasks limit and I/O throttling rules on backup repository.
After certain failures health check process may start logging failure sessions continuously.
Health check process fails with the “Failed to process method {Signature.FullRecheckBackup}: There is no FIB [10disk_1-flat.vmdk] in the specified restore point” error if the Backup Copy job is mapped on backup file containing VM backups with excluded disks.

File level recovery

Windows File Level Recovery fails to mount volumes with missing or incorrect volume lengths. The issue is typically observed on virtual disks created via the P2V process.
Restoring a file to the original location with Multi-OS File Level Recovery fails with the “Wasn’t able to find source file item [(/media/dm-0/root/test, /root/test)], skipping overwriting” error if the original file still exists.

Guest indexing

Catalog service may leak memory when processing very large amounts of guest indexes.

Built-in WAN Acceleration

Under rare circumstances, Backup Copy job to Linux-based repository may experience data mover process SegFault crash with Back-trace containing IP 0x848deff.
If target WAN accelerator runs out of disk space during digests generation, the job starts failing repeatedly with the following error: “An existing connection was forcibly closed by the remote host –tr: Cannot write data to the socket”

Backup from Storage Snapshots

Processing of VM on assigned to backup proxy with multiple FC adapters may fail with “Server with WWN is not found on HP 3PAR host” if some FC adapters do not have access to production 3PAR storage.
In certain HP 3PAR storage configuration, VM processing may fail with the “Unable to find volume with Node WWN and LUN ID” error.
Processing of VM with workingDir parameter set to a different HP 3PAR or HP StoreVirtual datastore fails with the “Unable to find matching device for LVID” error.
HP StoreVirtual snapshots may fail to be deleted with the “Failed to connect to host” error due to reaching the limit of SSH connections to StoreVirtual storage after certain time.

Tape

Under certain circumstances, the tape backup job may fail with the “Violation of PRIMARY KEY constraint ‘PK_Tape.file_versions’. Cannot insert duplicate key in object ‘dbo.Tape.file_versions’” error.
Restoring a file split across multiple tapes may fail with the “Block offset in source file differs from the current write position in target file” error.
Very long read or write operation may fail on certain tape libraries (such as IBM TS3100) with the following error: “The requested resource is in use“.
Tape jobs using standalone drives hand on second task with the “Tape medium was used in backup session, NOT VALID” error.
Tape media is not automatically erased in standalone tape drives when “Do not protect” option is enabled on a media pool.
Returning tape mediums without barcode from Offline after re-inventorization results in “Tape requires return to original, but it is already filled” error if the original slot is occupied.
VBRCatalog\Tapes\Media\ folder is not properly cleaned up upon various tape media operations, resulting in thrash accumulation.
Changing a tape may result in the “Subquery returned more than 1 value” error.
Corrected statuses for a number of tape drive alerts according to their real severity.

Configuration backup

Configuration backup fails if a tape changer connected to the backup server has null byte in its serial number with the “‘.’, hexadecimal value 0x00, is an invalid character” error.
Configuration backup of very large configuration database fails with the “Compressed or Uncompressed size, or offset exceeds the maximum value. Consider setting the UseZip64WhenSaving property on the ZipFile” error.
Under certain circumstances, configuration backup restore fails with the “String or binary data would be truncated. The statement has been terminated” error

Enterprise Manager

The required SureBackup job does not start and stop immediately after Virtual Lab request is approved or dismissed in the Enterprise Manager.
Adding a domain user to portal roles results in the “The specified directory service attribute or value does not exist” warning.
“Keep only last” setting of the Event history accepts unsupported values, resulting in Options dialog on attached backup servers fail to with value not valid errors.

User Interface

One of the configuration database tables is never cleaned up, which may cause user interface sluggishness after using the product for extended time.
Sorting by Total Size and Free Space columns does not work correctly in Files > VMware vSphere > Datastores view.
Sorting by Type, Size or Modified columns in Files > VMware vSphere > Datastores makes the user interface hang.

This patch also contains all fixes from Patch 1 and R2 update

More Information:

Prior to installing this patch please reboot the Veeam server to clear any locks on the Veeam services and when the reboot is done, please stop all the Veeam services and apply the patch.

When the patch is installed manually, start the Veeam services and open Veeam, and allow it to update all the components.

To obtain the patch please download it from: Download

How to stop your Veeam backup job when it is stuck

Leave a reply

A couple of weeks ago I ran into an issue that one of my Veeam backup jobs was stuck in a stopping state. After waiting for a long time I started searching for a solution to kill this backup job.

I found a KB article at the Veeam Knowledgebase that is worth mentioning. This is KB1727 “How to forcibly stop jobs that are stuck in ‘stopping’ status”

Solution:

The problem may occur because a process or task that the job is waiting for has not been completed. Such a task can be snapshot removal. Therefore, as the first step it is recommended to check the vSphere Client to see if there is a snapshot removal process pending/working on the vSphere side.

Make sure no important jobs/restores are running because the following steps will force them to stop:

Close the Veeam Backup & Replication console, and open services.msc on the Veeam Backup server,
Stop the Veeam Backup Service.
Open the Task Manager and kill all Veeam.Backup.Manager and VeeamAgent.exe processes.
Restart the Veeam Backup Service to release file locks held by either SQL or the OS.
Remove snapshots from VMs (if they have not been already removed).

Note. If you use the Virtual Appliance (HOTADD) mode, before removing the snapshots make sure there are no stuck disks on the Veeam Backup server or one of the backup proxies. Otherwise, the snapshots can be orphaned.

Nice to know…