Author Archives: Marco

About Marco

Marco works for ViaData as a Senior Technical Consultant. He has over 15 years experience as a system engineer and consultant, specialized in virtualization. VMware VCP4, VCP5-DC & VCP5-DT. VMware vExpert 2013, 2014,2015 & 2016. Microsoft MCSE & MCITP Enterprise Administrator. Veeam VMSP, VMTSP & VMCE.

Veeam Backup & Replication v8 Patch 1 released

Leave a reply

Last week Veeam released the first patch voor Veeam Backup & Replication v8. So what is new and fixed?

These are the release notes.

New Features and Enhancements

General 

  • Retention policy for replication jobs is now processed concurrently for multiple VMs in the job, as opposed to one by one, reducing the overall job run time.
  • Added email notification about File to Tape and Backup to Tape jobs waiting for user interaction (for example, when a tape medium needs to be inserted by the user)
  • VMware Tools quiescence should now log an informational event when VMware Tools are outdated, as opposed to a warning event.
  • Independent virtual disks which are explicitly excluded from processing in the job settings should no longer log a warning during the first time when the given VM is being backed up.
  • Virtual Lab proxy appliance was upgraded to virtual hardware version 7 to allow for it to manage more networks out of box.
  • Jobs should no longer appear to “hang doing nothing” in some circumstances, with the duration not ticking for any event in the job log.

Cloud Connect

  • Cloud backup repository size limit of 63TB has been removed

Linux support enhancements

  • Added support for file level recovery from XFS volumes (default file system in Red Hat Enterprise Linux and CentOS distributions starting from version 7.0)
  • Added support for file level recovery from Btfrs volumes (default file system in SUSE Linux Enterprise Server starting from version 12.0).
  • Added support for file level recovery from VMs with more than 10 LVMs.
  • Improved Linux guest file system indexing performance for incremental backup job runs.
  • FTP access mode of the file level recovery helper appliance now also displays friendly LVM volume names introduced in v8.
  • To enable identity verification of the remote server, SSH key fingerprint is now displayed to a user when registering the new Linux server. Accepted SSH keys are stored in the configuration database to protect from MITM attack.

 

Performance enhancements

  • Added experimental support for direct data movers communication when both are running on the same server (for example, when backing up to a local storage on backup proxy server). If your local backup jobs report Network as the bottleneck, and you see high load on some backup proxy server NICs when the data was supposed to stay local to the server, you are likely to benefit from this behavior modifier. To enable alternative data exchange path, create the DataMoverLocalFastPath (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set it to the following values:

    0: Default behavior (no optimizations)

    1: Data exchange through TCP socket on the loopback interface (faster)

    2: Data exchange through shared memory (fastest)

EMC Data Domain integration 

  • Synthetic full backup creation and transformation performance has been improved significantly.

 

HP 3PAR StoreServ integration

  • Remote Copy snapshots are now automatically excluded from rescan, both speeding up the rescan process and reducing storage load.

 

NetApp integration

  • Added support for NetApp Data ONTAP 8.3.
  • Added support for HA Pair 7-mode configuration.
  • Added support for vFiler DR configuration.

 

Resolved Issues

 

General

  • Restored incremental backup job performance back to v7 levels by removing unnecessary metadata queries from previous backup files.
  • Multiple issues when parallel processing is disabled (high CPU usage on backup server, wrong task processing order, reversed incremental backup jobs with disk excluded failing to access backup files).
  • Automatic network traffic encryption (based on public IP address presence) is enabled even when both data movers are running on the same computer, reducing job’s processing performance.
  • Quick Backup operation triggers jobs chained to the job it uses.
  • Loss password recovery option does not function correctly when Enterprise Manager has multiple backup server registered.
  • Under rare circumstances, backup job with encryption enabled may fail with the “Cannot resize block data is encrypted” error.
  • When multiple network traffic rules having the same source and target ranges are set up, changing settings to one of the rules also updates all other rules.

 

vSphere

  • In vSphere 5.5, enabling virtual disk updates redirection to the VMFS datastore in Instant VM Recovery settings, and then migrating a recovered VM to the same datastore with Storage vMotion causes data loss of data generated by the running VM.
  • Full VM restore process puts all virtual disks to the datastore selected for VM configuration file,ignoring virtual disks placement settings specified in the wizard.
  • In certain environments, jobs fail to process VMs in Direct SAN Access mode with the “Failed to create processing task for VM Error: Disk not found” error.
  • Dismounting processed virtual disks from backup proxy server takes longer than expected in the Virtual Appliance (hot add) processing mode.
  • Rescan of vCenter server containing hosts without HBA and SCSI adaptors fails with the “Object Reference not set to an instance of an object” error.
  • Replication jobs to a cluster fail with the “The operation is not allowed in the current state” error if chosen cluster host is in the maintenance mode.
  • Under rare circumstances, jobs with Backup I/O Control enabled may occasionally log the “Operation is RetrievePropertiesOperation” related errors.
  • Enabling Backup I/O Control limits snapshot removal tasks to one per datastore regardless of datastore latency levels.
  • VM Copy jobs always fail over to the Network (NBD) processing mode.
  • VMs with virtual disks having the same names but with different capitalization, and located on the same datastore cannot be backed up.
  • Backing up vCloud Director vApp to a CIFS share may fail with the “Object Reference not set to an instance of an object” error.
  • Snapshot Hunter ignores backup window specified in the corresponding job.
  • vCloud Director VMs created from linked clone template and with a user snapshot present incorrectly trigger Snapshot Hunter.

 

Hyper-V

  • CBT does not track changes correctly on VHDX disks files larger than 2 TB.
  • CBT may return incorrect changed blocks information for differential disks, when parent and child disks have the same name.
  • Large amount of unmapped VHDX blocks may cause the job to failover to full scan incremental backup with the “Failed to update unaccounted changes for disk. Change tracking is disabled” error.
  • Existing Hyper-V backup and replication jobs processing VM with SCSI disks start to consume x2 space on target storage after upgrade to v8.
  • Removing a node from a cluster does not remove the corresponding node from the cluster on the Backup Infrastructure tab of the management tree until the next periodic infrastructure rescan (in up to 4 hours).
  • Replication from backup fails with the “Virtual Hard Disk file not found[/id” error if backup job that created this backup had one or more disks excluded from processing.
  • Under rare circumstances, legacy replicas using non-default block size may fail with the “[i]FIB block points to block located outside the patched FIB” or “Block offset [] does not match block ID []” error messages.
  • Offhost backup from Windows Server 2012 R2 fails with the “Exception of type ‘Veeam.Backup.AgentProvider.AgentClosedException’ was thrown” error when processing a VM that can only be backed up in Saved or Crash-Consistent states, and native Hyper-V quiescence is enabled in the job settings.
  • If a Hyper-V cluster node goes offline during backup from shared volume snapshot, and does not recover until the Veeam job fails, CSV volume backup infrastructure resource will not be released by the scheduler. As the result, other jobs waiting to process VMs from the same volume may remain hanging in the “Resource not ready: Snapshot” state.
  • Replication job fails with the “Cannot assign the specified number of processors for virtual machine” error for VMs with core count larger than the one of the target Hyper-V host. Legacy replication job works fine in the same circumstances, however VM failover is not possible.

 

Application-aware processing

  • VeeamGuestHelper process crashes when backing up Windows 2003 SP2 VM running Oracle.
  • Application-aware processing hangs for extended time when processing a Windows VM containing one or more Oracle databases in the suspended state.
  • Certain advanced guest processing settings lead to VM processing failing with the “Unknown indexing mode ‘None’ ” error.

Multi-OS File Level Recovery

  • Restore to original location does not work in certain LVM configurations.

 

Enterprise Manager

  • Under certain circumstances, Enterprise Manager data collection may fail with the “Cannot insert the value NULL into column ‘guest_os’ ” error.
  • Restore scope rebuild operation tries to reach out to the hosts already removed from the corresponding backup server, resulting in login errors.

 

Built-in WAN Acceleration

  • WAN accelerators fail to detect and populate global cache Windows Server 2012 R2 and Windows 8.1 OS data, which may result in reduced data reduction ratios.

 

Cloud Connect

  • The Decompress backup data blocks before storing backup repository option is ignored for Cloud Connect backend repositories.
  • Backup Copy jobs with encryption enabled fail with the “Attempted to compress encrypted data-block” error if backend repository has the Decompress backup data blocks before storing option enabled.
  • Veeam Backup Service crashes with the “An existing connection was forcibly closed by the remote host” if network connection fails immediately upon establishing the initial connection to the Cloud Gateway.
  • Cloud Connect quotas are rounded incorrectly in the service provider’s interface.
  • Tenant statistics data such as data sent/received is only shown for the last 24 hours.

 

Veeam Explorer for Active Directory

  • “This implementation is not part of the Windows Platform FIPS validated cryptographic algorithms” error is displayed if Use FIPS compliant algorithms for encryption, hashing, and signing Group Policy setting is enabled.
  • Attempting to open single label domain’s AD database fails with the “No writable domain found” error.
  • Attempting to open third level domain’s AD database fails with the “No writable domain found” error.
  • Under certain circumstances, object restore may fail with the “Unable to cast object of type ‘System.String’ to type ‘System.Byte[]’ “error.

 

Veeam Explorer for Exchange

  • Process of exporting email messages cannot be stopped.
  • Copying or saving attachments from multiple different emails always copies or saves attachment from the first email for which this operation was performed.
  • Opening public folder’s mailbox fails with the “Unable to find wastebasket” error.

 

Veeam Explorer for SQL Server

  • Restoring to SQL instance with case sensitive collation enabled fails with the “The multi-part identifier “a.database_id” could not be bound” error.

Backup from Storage Snapshots 

  • Backup jobs from storage snapshots on iSCSI SAN take too long to initialize when large amount of iSCSI targets are configured on the backup proxy server.
  • Under certain circumstances, backup jobs from storage snapshots on FC SAN may fail with the “Unable to find volume with Node WWN and LUN ID” error.

 

HP 3PAR integration

  • Rescan of 3PAR array with empty host sets fails with the “Object reference not set to an instance of an object” error.

 

NetApp integration

  • Under certain circumstances, storage rescan process may leave temporary storage snapshots behind.

 

Tape

  • Backup to Tape jobs may appear to run slower after upgrading to v8 because of copying unnecessary backup files (VRB) to tape when source backup job uses reversed incremental backup mode.
  • On standalone drives, tape jobs configured to eject tape upon completion fail the “Object reference not set to an instance of an object” error after 10 minutes of waiting for the user to insert tape.
  • Tape job configured to use full and incremental media pools hosted by the different standalone tape drives fails when processing incremental backup with the “Value cannot be null” error. Tapes are formatted with the block size of tape medium, as opposed to the default block size of the tape drive.
  • Tape job configured to use full and incremental media pools hosted by the different tape libraries fails while waiting for an incremental pool’s tape with the “Tape not exchanged” error
  • Tape jobs fail with the “GetTapeHeader failed” error on IBM TS3100 tape library with Path failover feature disabled.

 

PowerShell

  • Objects obtained with Find-VBRViFolder cmdlet have incorrect Path attribute.
  • Get-VBRTapeJob cmdlet fails to obtain job object for tape jobs with incremental processing disabled.
  • Get-VBRTapeLibrary cmdlet fails with the Standard Edition license.
  • All built-in WAN accelerator management cmdlets have been made available in all paid product editions

 

To obtain the patch, please click here


VMware Template unable to Convert to Virtual Machine

1 Reply

Sometimes the following will happen:

You have a couple of templates and want to update them, you right click the VMware Template and the option to convert to Virtual Machine is grayed out. In this case it was a VMware vSphere ESXi 5.5U2 environment.

Also the Rename and other options are grayed out. How to solve this issue?

Try to deploy a virtual machine from this template. After you successful deployment the options are available again. Strange, but in my case it works..

How to solve the ‘Cache 1’ Backup Power Source (Capacitors) Status = ‘Charging’. Warning

1 Reply

One of our HP Storevirtual P4530 server was giving the following Warning.

‘Cache 1’ Backup Power Source (Capacitors) Status = ‘Charging’. WARNING (E01020401)

When this error occurs the performance of the Storevirtual Storage environment is terrible. Very high latency’s with read and write.

This was the second time in a few months’ time. The first time HP replaced the Cache Battery. It could be the same problem, so I contacted HP support. It wasn’t, they told me the following.

With a Firmware earlier than 5.42 of the RAID controller there is an issue with the charging and discharging of the cache battery. When the warning occurs the Cache Battery is in an error state and therefor disabled. The HP Storevirtual P4530 is using a HP P420i RAID controller. For more information see HP Document id: c024258375

To check if the Cache Battery is disabled create a Support Log bundle of the Storage node. Then check the ADUReport and look for the Cache Disable Code. See screenshots.

Advice from HP Support is to upgrade to a supported level. In my case 5.42 for the P420i RAID controller.

To verify which firmware is supported and which isn’t see the HP Storevirtual Storage Hardware Firmware matrix. This document can be found at the HP SPOCK see link. If you have any questions, please contact HP Support. I’m not an HP employee….

Thanks HP Support for this fix.

Solve the “System logs on ESXi Host are stored on non-persistent storage” message

5 Replies

Recently I had to install a few VMware ESXi 5.5 host on removable SD cards. After the installation finishes and you connect the first time with the vSphere Client, a message is displayed.


This means that the log files are written on the SD card. After a reboot the log files disappear. This could be a problem when trying to trouble shoot a problem and the ESXi host has rebooted.

So how to solve this configuration issue. See VMware KB2032823 for instructions. This is very easy and highly advisable.

Verify the Location of System Logs in the vSphere Client

  1. In the vSphere Client, select the host in the inventory panel.
  2. Click the Configuration tab and click Advanced Settings under Software.
  3. Make sure that Syslog.global.logDir points to a persistent location.
    The directory should be specified as [datastorename] path_to_file where the path is relative to the datastore. For example, [datastore1] /systemlogs.
  4. If the Syslog.global.logDir field is empty or explicitly points to a scratch partition, make sure that the field ScratchConfig.CurrentScratchLocation shows a location on persistent storage.

Verify the Location of System Logs in the vSphere Web Client

  1. Browse to the host in the vSphere Web Client navigator.
  2. On the Manage tab click Settings.
  3. Under System, click Advanced System Settings.
  4. Make sure that Syslog.global.logDir points to a persistent location.
  5. If the field Syslog.global.logDir is empty or points to a scratch partition, make sure that the field ScratchConfig.CurrentScratchLocation shows a location on persistent storage.

    Note: You must reboot the host for the changes to take effect.

Veeam VMCE7 to VMCE8 upgrade path

1 Reply

A couple of months ago Veeam launched there VMCE certification. This certification is valid for the Veeam Backup & Replication V7 version. Now that Veeam is almost launching their new flagship version 8 all the VMCE certified engineers should upgrade their certification.

Veeam released a document that describes the possible upgrade paths. For more information see document: VMCE7 to VMCE8: Upgrade Rules

There can be two scenarios: 

  • A student with no existing VMCE certificate (a new student).
  • A student with the VMCE 7 certification.

 

New students

When the VMCE 8 program is launched, any student eager to get VMCE-certified but who have not completed the VMCE 7 certification, will need to:

  • Attend the VMCE 8 classroom through Instructor-Led Training (ILT) or Video-on-Demand Learning (VODL) at one of the available Veeam Authorized Education Centers (VMAEC). Note: VODL is available in North America only.
  • Pass the VMCE 8 proctored exam at an available Pearson Vue test center.

These students may exceptionally be authorized to follow the free online “VMCE 8 – What’s New” course on Veeam.com, and then validate their VMCE certification with a VMCE 8 proctored exam at an available Pearson Vue test center.

VMCE 7 Certified Engineers

Students who complete the VMCE 7 certification program will be entitled, with a specific restricted access, to the free online “VMCE 8 – What’s New” course on the Veeam.com website.

These students will then need to pass the VMCE 8 proctored exam at an available Pearson Vue test center to upgrade their certificate from VMCE 7 to VMCE 8. This must occur in the allowable certification maintenance time according to the certification rules below.

Students who attend the VMCE 7 ILT or the VODL, but who do not complete the VMCE 7 exam, will exceptionally be authorized to follow the free online “VMCE 8 – What’s New” course on the Veeam.com website, and then validate their VMCE certification with a VMCE 8 proctored exam at an available Pearson Vue test center.

Certification Maintenance: Reminder from the VMCE Program Guide

All VMCEs are required to stay current on knowledge and certifications related to Veeam Software. This includes re-certification on all new major releases of Veeam Software. VMCEs will have 120 days, after the release of a major software update and the release of associated upgrade courses and exams, to take the course and pass the exam necessary to upgrade the certification level.

If VMCEs fail to upgrade in the allotted time, there will be another 90 days under which they must take the full course and pass the full exam. After this time, if any VMCEs have still not been upgraded, the certification will expire and the engineer will no longer be eligible for the benefits of the VMCE program.
Source: VMCE7 to VMCE8: Upgrade Rules

More information about the VMCE Certification, see the Veeam Website.

Solve the Flex-common-lib-x.x.x.swf fail to load error 2032

Leave a reply

When connecting to the vSphere web client with Firefox the following error could occur. RSL https://vcenterserver:9443/vsphere-client/locales/rsl/flex-common-lib-5.5.0.swf failed to load. Error #2032

This happened to me after an upgrade to the latest version of VMware vCenter.


A quick search on google resulted in a solution. Clear your cache from your browser. In my case Firefox. Reload the site, and problem solved.

VCAP5-DCA Objective 9.1 – Install ESXi hosts with custom settings

Leave a reply

Knowledge

  • Identify custom installation options
  • Identify ESXi Image Builder requirements

Skills and Abilities 

  • Create/Edit Image Profiles
  • Install/uninstall custom drivers
  • Configure advanced boot loader options
  • Configure kernel options
  • Given a scenario, determine when to customize a configuration

Continue reading →

Unable to install VMware tools. Error 1606.

Leave a reply

Today I had to virtualize several Windows 2008 R2 Terminal server. After a successful conversion I ran into a problem when installing the VMware Tools.

Error 1606. Could not access network location [servername]\[sharename]\desktop\programs\startup\.


This error is caused by a Group Policies that was redirecting the start menu to another server (also for the domain administrator and local administrator). Because I started the VM in an isolated environment it was unable to check the free disc space of the network share and gave me this error. This error is discussed in VMware KB1017886.

In my case there were two directories redirected to a server that was not reachable for this server. I also checked the Current User settings, but there was nothing directing to the unreachable server.



To solve the error change both register keys to their original settings. 

  • The “Common Startup” key should be “%ProgramData%\Microsoft\Windows\Start Menu\Programs\Startup”
  • The “Common Programs” key should be “%ProgramData%\Microsoft\Windows\Start Menu\Programs”

The original setting of all the keys. 


Restart the VMware Tools installation and now everything should work without any problems.

More information:

Veeam Backup & Replication setup or upgrade not starting

Leave a reply

A couple of weeks ago I tried to do an upgrade of the Veeam Backup & Replication 6.1 to v7.0. I mounted the fresh downloaded ISO of Veeam v7.0. The splash screen showed, and when I clicked on the Install / Upgrade options nothing happened.


This server is running Windows 2008R2 Service Pack 1, and is also the VMware vCenter server for the ESXi environment.

Because this server is a vCenter Server and purely a management server this server didn’t had all the Windows Updates installed that are available today. So my first guess was to update the complete server. After updating this server and a fresh reboot, I started the installation again. No success, still the same behavior. Splash screen disappears and reappears again without starting the Veeam installation.

There are also no errors in the event logs of the server. Always nice to trouble shoot…

When installing the Microsoft System CLR Types for SQL components there is an error. 


So there was something wrong with the Windows environment, but everything was working like it should. I already checked all the perquisites, and everything was installed. I didn’t know where to look for, so after some time I contacted Veeam support. They checked the environment and reinstalled the .Net Framework. After this installation the Veeam installer was working correctly. Problem was a corrupted or damaged .Net framework installation.

Lessons learned:

  • Veeam Support is awesome, they contacted me very quickly and solved my problem within an hour or so.
  • When running into strange behavior with the installer, try to repair or reinstall the .Net Framework it might solve your problems.