Author Archives: Marco

About Marco

Marco works for ViaData as a Senior Technical Consultant. He has over 15 years experience as a system engineer and consultant, specialized in virtualization. VMware VCP4, VCP5-DC & VCP5-DT. VMware vExpert 2013, 2014,2015 & 2016. Microsoft MCSE & MCITP Enterprise Administrator. Veeam VMSP, VMTSP & VMCE.

New free tool: Veeam Task Manager for Hyper-V

Today Veeam released a new free tool for your Hyper-V environment called “Veeam Task Manager for Hyper-V”

Veeam Task Manager for Hyper-V is a portable, standalone performance monitoring tool that’s FREE!

Improve troubleshooting in your Hyper-V environment by seeing what Windows Task Manager doesn’t show you. Veeam Task Manager for Hyper-V displays a real-time view of both host and VM compute resource consumption.

Get the critical visibility you need.

This lightweight tool is portable so you can run it from any USB device for emergency troubleshooting. No installation or integration needed!


More information:

Download the FREE Veeam Task Manager today

Veeam Backup & Replication v7 Patch 4 released

I noticed on twitter that Veeam released Patch 4 for Veeam B&R v7. So what is fixed?

New Features and Enhancements

VMware Virtual SAN (VSAN) 

  • In addition to adding basic support (as provided by other vendors), the intelligent load-balancing engine was enhanced to account for VSAN specifics. As the result, for each VM the job will pick backup proxy running on VSAN cluster node with most of the virtual disks’ data available locally. This significantly reduces backup traffic on VSAN cluster network, resulting in minimum possible impact on production environment from backup activities.

Microsoft SQL Server 2014

  • Added support for Microsoft SQL Server 2014 both as the protected guest workload (including application-aware processing functionality), and the back-end database for backup and Enterprise Manager servers.

License key auto update

  • Added automated license key update option to the License Information dialog. With auto-update enabled, the product will check Veeam licensing server periodically for an updated license key, and download and install the key automatically as soon as it becomes available. This feature is particularly useful to the Service Providers and subscription-based customers, and it removes the need to download and install the license key manually each time when the license extension is purchased.

Backup Copy

  • The maximum allowed amount of restore points in the Backup Copy job has been increased to 999.
  • Backup Copy will now resume the transfer after network connection drop to a Linux-based backup repository.
  • Backup Copy jobs should no longer report errors during the days when source backup jobs are not scheduled to run – for example, during the weekend.

Hyper-V

  • Added support for certain Hardware VSS Providers that previously could not be detected by the storage rescan process, and as such could not be used by the jobs.
  • Jobs will now retry failed snapshot creation when another shadow copy of the same volume is already in progress, instead of immediately failing to process a VM.

Resolved Issues

General 

  • Under certain circumstances, the target data mover may crash with the “Cannot allocate memory for an array” error while processing a backup job with WAN storage optimization setting.
  • Increased maximum amount of open files for Linux data mover to improve backup reliability in cases with a very large amount of concurrent tasks hitting the same repository.

vSphere

  • If the same ESX(i) host is added to Managed Servers both as a part of the vCenter, and as a standalone host, the jobs fail to remove VM snapshots correctly.
  • Replication job does not truncate Microsoft Exchange transaction logs when the corresponding setting is set to “Truncate logs on successful backup only”.
  • Setting EnableSameHostHotaddMode registry value (introduced in Patch #3) to 1 causes replication jobs to fail with the “Object reference not set to an instance of an object” error.
  • If you clone replica VM with vSphere, and start the replication job, the target VM will be selected randomly between the two VMs.

Hyper-V

  • Issues introduced with Windows Server 2012 R2 Update (April 2014 rollup), see KB1863 for more info.
  • Very large amount of parallel tasks may cause in significant delays (up to a few minutes) between assigning required backup infrastructure resources to the job, and the job actually starting to leverage those resources.
  • Replication jobs do not truncate Microsoft Exchange transaction logs regardless of the corresponding settings state.
  • Processing of virtual machines with long dash symbol in the file name fails with the “The system cannot find the path specified.\n\nFailed to open file [C:\ClusterStorage\FOLDER\GUID.xml] in readonly mode. (System.Runtime.InteropServices.COMException)” error.
  • Parallel snapshots of SMB3 share may fail with the “Failed to add volume to the VSS snapshot set. Another shadow copy creation is already in progress. Wait a few moments and try again” error.
  • Restoring a VM to localized Hyper-V cluster creates duplicate cluster resources remaining in the Starting state in Failover Cluster Manager.

Application-aware processing

  • During full VM restore and SureBackup job, Domain Controllers may spend longer than required in the DSRM mode (up to 10 minutes – until the corresponding timeout expires), thus unnecessarily delaying DC recovery.
  • Increased timeouts for network-less application aware processing of vSphere guests to improve the chance of processing completing successfully.

Backup Copy

  • Backup Copy configured with multiple backup jobs as the source does not include VMs which are excluded in one source backup job, but included in another.
  • Health Check process continuously fails with the “Timed out waiting for the backup repository resource to become available for backup integrity check” error during the periods when the Backup Copy job is restricted from using network bandwidth.
  • Backup Copy is incorrectly marked as failed despite completing successfully when source or target WAN accelerator becomes unavailable (for example, due to the network issue and reboot) during the period when Backup Copy job is waiting for its availability.

Backup from Storage Snapshots

  • Disabling failover to network processing mode on a backup proxy causes intelligent load balancing not to assign the backup proxy to jobs running with storage snapshots integration enabled.
  • Under certain circumstances, backup from 3PAR volumes may fail with the “LUN is already taken for set ESXs” error.

Configuration backup

  • The presence of null byte in tape changer serial number causes configuration backup to fail.

File Copy

  • Certain files are skipped with the File Copy job with the “Cannot copy a symbolic link of type Veeam.Backup.Core.CLocalFileItem into a directory of type Veeam.Backup.Core.CRemoteWinFolder” error, however the job reports success. For example, this may happen when copying files from volumes with Windows Server 2012 deduplication enabled.
  • File Copy job fails when attempting with the “Client error: boost::filesystem::remove: Access is denied” error when attempting to overwrite read-only file on the destination Windows server. The remaining files are not copied.

Native File Level Recovery

  • Starting file level recovery from the Restore wizard fails with the “Object reference not set to an instance of an object” error when SAN snapshots is picked as the recovery source. Starting the same process from SAN infrastructure tab does not have this issue.
  • In rare scenarios, file level recovery is unable to parse LDM contents of the virtual disk. As the result, Backup Browser opens empty, without any volumes mounted.
  • Restoring to original location for volumes which are mounted to a folder (as opposed to a drive letter) fails when such volumes reside on GPT partition or dynamic disk.

Multi-OS FLR

  • Folder permissions, group and owner IDs are not preserved when restoring a folder with “Preserve permissions and ownership” check box selected.
  • Multi-OS FLR fails with the “The file specified is not a virtual disk” error on replica VMs with VMDK larger than 2TB in size.

Tape

  • Depending on where the file is placed on the tape, restore process may not calculate the file size correctly, resulting in corrupted file restored from good backup.
  • File to Tape job fails to process files with full path of exactly 260 symbols long with the “The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters” error. Files with full path longer or shorter than 260 symbols are processed normally.
  • Restoring a very large file from tape may fail with the “Value was either too large or too small for an Int64” error.
  • Restoring a VM from tape may fail with the “Unable to read tape metadata” error.

UI

  • VMs and Templates View of the virtual infrastructure Browse dialog, and Category View on the Virtual Machines tab do not show vSphere vApps.
  • Sending email notifications fails with the “Key not valid for use in specified state” error if SMTP credentials were entered by a user account other than Veeam Backup Service account.

Veeam Explorer for Microsoft Exchange

  • Opening mailbox database may occasionally fail with the “The process cannot access the file ‘C:\PATH\ese.dll’ because it is being used by another process” error.
  • Mounting mailbox database or transaction log volumes fails with the “Exchange logs cannot be replayed, block size mismatch” error from 512-byte emulation (512e) disks.
  • In rare scenarios, opening a mailbox from German localized Microsoft Exchange deployment fails with the “JetError -1404, JET_errIndexNotFound, No such index” error.

Veeam Explorer for Microsoft SharePoint

  • SharePoint restore wizard fails with the “Object reference not set to an instance of an object” error if actual SharePoint configuration does not match the one collected with application-aware processing logic during the backup job.

WAN Acceleration

  • Increased source WAN accelerator digests retention to 2 months to avoid digests recalculation every week.

Pilot Patch #4 (build 7.0.0.870)

The generally available version of Patch #4 (build 7.0.0.871) also addresses the following issue reported during pilot patch deployments:

  • Tape inventory fails on certain tape drives with the following error: “Tape fatal error. No more data is on the tape”

This patch also contains all fixes from Patch 1, R2 update and Patch 3

More Information:

Prior to installing this patch please reboot the Veeam server to clear any locks on the Veeam services and when the reboot is done, please stop all the Veeam services and apply the patch.

When the patch is installed manually, start the Veeam services and open Veeam, and allow it to update all the components.

To obtain the patch please download it from Download

VCAP5-DCA Objective 8.2 – Administer vSphere Using the vSphere Management Assistant

Knowledge

  • Identify vMA prerequisites
  • Identify vMA specific commands
  • Determine when vMA is needed

Skills and Abilities 

  • Install and configure vMA
  • Add/Remove target servers
  • Perform updates to the vMA
  • Use vmkfstools to manage VMFS datastores
  • Use vmware-cmd to manage VMs
  • Use esxcli to manage ESXi Host configurations
  • Troubleshoot common vMA errors and conditions

    Continue reading

Solving the com.vmware.sps.fault.QsConnectionException error

After en APD (All Path Down) in our hosting environment I had the following error when I tried to do something with the ESXi datastores.


The VMware environment was working correctly.
I started searching the VMware Knowledge base and found KB2064174. This KB describes exactly my problem.

Symptoms 

  • Selecting/browsing a datastore, the VMware vCenter Server 5.1.x displays the error message:

com.vmware.sps.fault.qsconnectionException

  • Selecting the Storage Views of any datastore displays the error message:

cannot initiate storage service

  • vMotion or an offline migration of a virtual machines fails with the below error message:

pbm.fault.PBMFault.summary

Resolution

To resolve this issue, ensure the VMware inventory service and the VMware profile Driven Storage Service are running.

  1. Log in as an administrator to the server that is running the vCenter Server.
  2. Click Start > Run.
  3. Type services.msc
  4. From the list of services, check the status of the VMware inventory service and the VMware profile Driven Storage Service.
  5. If any of these services are not running, right-click on the service and select Start. (A restart of the services may be applicable as well).

Note: If both of the services are not running, start the Inventory service first and the Profile Driven storage service second.

For VMware vCenter Appliance 5.x

Restart virtual center server through the web interface.

  1. Log in to the vCenter Server Appliance’s VAMI (Virtual Appliance Management Interface) at http://<IP/hostname of VC>:5480
  2. Select the Summary tab under vCenter Server.
  3. Click Stop beside vCenter Server to stop the process.
  4. Click Start beside vCenter Server to start the process

In my case all the services where running, so I restarted the VMware vCenter Inventory services. After this restart the error message disappeared when browsing the datastores.

vExpert 2014 Announcement

Today, April 1ste the VMware vExperts for 2014 are announced. And YES!!! I am one of them for the second time. And no this is not a joke!!

For some prove, the complete list of all the vExperts can be found here. This year there are 754 awarded with the vExpert 2014 title.


Each of these vExperts have demonstrated significant contributions to the community and a willingness to share their expertise with others.

I would like to thank John Troyer, Corey Romero and the rest of the VMware Social & Community Team for making this possible.

Congratulations to all the other 2014 vExperts.

Win a trip around the world with Veeam

Veeam is about to get its 100,000th customer and is launching an interactive contest for a chance to win a trip around the world and other prizes (Google Glass, iPad and Microsoft Surface).

To participate, visitors need to register and predict the location of Veeam’s 100,000th customer on the interactive map. The closer you are to the right spot, the better chance you have to win the trip around the world and other prizes.

We currently have a live pre-registration page. The main contest will start next week.

Join in on the fun! We hope you and your readers will participate and celebrate this great achievement with us.

Guess the location here: http://world.veeam.com/

For more information: http://world.veeam.com/veeam_tc_2014.pdf

VCAP5-DCA Objective 8.1 – Execute VMware Cmdlets and Customize Scripts Using PowerCLI

Knowledge

  • Identify vSphere PowerCLI requirements
  • Identify Cmdlet concepts
  • Identify environment variables usage

Skills and Abilities 

  • Install and configure vSphere PowerCLI
  • Install and configure Update Manager PowerShell Library
  • Use basic and advanced Cmdlets to manage VMs and ESXi Hosts
  • Use Web Service Access Cmdlets
  • Use Datastore and Inventory Providers
  • Given a sample script, modify the script to perform a given action
    Continue reading

Veeam Backup & Replication v7 Patch 3 released

I noticed on twitter that Veeam released Patch 3 for Veeam B&R v7. So what is fixed?

Engine

  • Significantly reduced load on SQL Server hosting the configuration database, and improved performance of various job activities and user interface operations.

vSphere

  • Improved vCenter Server hierarchy enumeration logic to speed up vCenter Server registration process in a few times.
  • Intelligent load balancing can now be configured to give preference to backup proxy located on the same host using the EnableSameHostHotaddMode (DWORD) registry value.
  • Replication jobs should now fail immediately upon encountering fatal error, rather than wasting time performing additional operations that are no longer relevant.

Hyper-V

  • Added timeouts on all snapshot operations, including snapshot creation, deletion, import to off-host proxy, and reading snapshot data to prevent conditions with stuck snapshots due to environmental issues.
  • Maximum volume snapshot life time is now customizable using the hvSnapshotLifeTimeHour (DWORD) registry value. Default value is 72 hours, but customization may be necessary to allow full backup of very large VMs on slow backup hardware.
  • Added ability to configure SCVMM communication timeouts using the SCVMMConnectionTimeoutMinutes (DWORD)

1-Click FLR

  • Support for downloading files larger than 4GB.

Built-in WAN acceleration

  • Data processing performance has been further improved by up to 25% comparing to R2 update.

Multi-OS File level recovery

  • Added support for file names with Unicode characters.

Tape

  • Support for backing up files with file names longer than 260 characters, and directory names longer than 248 characters.
  • Support for devices that do not implement sending error in response to attempt of writing to write protected tape.
  • Support for devices that do not implement sending the end of media event, such as certain IBM tape drives. Capacity for such drives is now controlled by the job.
  • Support for devices that do not implement drive enumeration in the drivers correctly. Such tape devices are now additionally queried for drives with SCSI commands.

User interface

  • Column sizes and sorting rules will now persist for each user in all grids and views.
  • Disabling GFS retention in Backup Copy job now offers to remove the existing full backup along.
  • Backup file repair process is now reported with the dedicated job status.

Rotated media

This update adds full support for rotated media with all job types and backup modes removes the need of using additional scripting. The new functionality is controlled by the following registry values:

ForceCreateMissingVBK (DWORD)

Enables support for rotated drives. With this functionality enabled, if any backup file from the latest full backup chain is missing (such as when the existing hard drive is replaced by another one), jobs will start the new backup chain and create the new full backup (instead of failing out).

Set to 1 to enable support for rotated drives.

ForceDeleteBackupFiles (DWORD)

Enables automatic cleanup of the backup repository from ALL existing files if any backup file from the latest full backup chain is missing (such as when the existing hard drive is replaced by the hard drive brought back from offsite location, and containing some older backup files).

Set to 3 to make the job delete the entire contents of the backup job’s folder only.

Set to 5 to make the job delete the entire contents of the root backup repository folder, thus potentially deleting any files belonging to other jobs along. Note that if backup repository points at the volume’s root folder, the entire volume’s contents is erased.

Resolved Issues

Engine

  • Job may fail VM processing if backup proxy has a very large amount of NICs with the “A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond” error. To work around this, increase the timeout by creating the newly introduced ConnectByIPsTimeoutSec (DWORD) registry value.
  • Network throttling rule does not apply if some job is already running at the time when the throttling period starts for the given rule.
  • In some scenarios, target data mover for CIFS-based backup repository starts on the backup server, rather than on the specified proxying server.
  • Summary.xml growing beyond 16385 bytes results in the following job runs fail with the “Read summary xml error” error.
  • Running too many parallel task on a single proxy or repository may cause the “Cannot find connections pool by id” error.
  • Stopping the job on transform day causes resource leak.

vSphere

  • Processing of VM with virtual disk larget than 2TB fails with the “Requested value ‘seSparse’ was not found. (System.ArgumentException)” error if there is an existing VM snapshot present.
  • Failback of VM with virtual disk larget than 2TB fails with the “Requested value ‘seSparse’ was not found. (System.ArgumentException)“.
  • Network remapping functionality of replication job does not function correctly if performed within a single DVS (Distributed Virtual Switch), or when source and target VM are connected to more than one DVS.
  • VeeamReplicaSummary growing beyond 16385 bytes results in the following job runs fail with the following error: “Failed to create replica vm disk“.
  • Full VM restore of a VM containing virtual disks with the same names creates a new VM with incorrect configuration if the original VM also had one or more snapshots present at the time of backup. Attempting to start the restored VM results in “Cannot open the disk or one of the snapshot disks it depends on. The system cannot find the file specified” vSphere error.
  • All jobs now detect and refuse to process VMs with virtual disks disk size not being a multiple of 1KB to prevent issues at restore (“Client error: Declared and actual sizes of the disk are different“) or Backup Copy (“Inline data validation failed“). Such disks cannot be created with vSphere user interfaces, and are usually a product of P2V conversions or similar activities.
  • Backing up VMs with non-default workingDir parameter from vCenter Server 4.X fails with the “Sequence contains no matching element” error.

Hyper-V

  • Changed Block Tracking fails on dynamic VHD disk aligned to 1024 bytes with the “Failed to update unaccounted changes for disk. Change tracking is disabled” error.
  • Replication job with seeding enabled fails with the “Client error: Client either not connected to server or connection is broken
  • Added workaround to prevent CSV cache from leaking during backup activities (Microsoft issue).
  • Under certain circumstances, replication to a cluster may fail with the “The process cannot access the file because it is being used by another process” error.
  • Stopping a job at certain “bad” time results in hardware VSS snapshot remaining on the storage with the “Failed to close current snapshot” error logged.
  • Volume rescan fails with the “Server ‘This server’ does not exists or access denied. Host discovery failed. The RPC server is unavailable” error when the product is installed on the Hyper-V host.
  • Selecting the Run server on this side option on the source host makes replication jobs fail.
  • Replication job statistics may show incorrect Read and Transferred values upon network disconnect at the end of incremental job run.
  • Swap exclusion notification is missing from the VM processing log.
  • Removed case sensitivity when obtaining host names from SCVMM server for better matching reliability with hosts already added to the configuration.

Application-aware processing

  • Application-aware processing breaks DFSR replication of SYSVOL when all domain controllers are backed up at the same time due to the parallel processing.
  • Transaction logs are truncated when the VM processing fails due to loss of connectivity with backup proxy even when the job is set to truncating logs on successful backup only.
  • Improved in-guest VSS snapshot handling to increase backup reliability for Windows Server 2008

Backup Copy

  • Health check process does not respect concurrent tasks limit and I/O throttling rules on backup repository.
  • After certain failures health check process may start logging failure sessions continuously.
  • Health check process fails with the “Failed to process method {Signature.FullRecheckBackup}: There is no FIB [10disk_1-flat.vmdk] in the specified restore point” error if the Backup Copy job is mapped on backup file containing VM backups with excluded disks.

File level recovery

  • Windows File Level Recovery fails to mount volumes with missing or incorrect volume lengths. The issue is typically observed on virtual disks created via the P2V process.
  • Restoring a file to the original location with Multi-OS File Level Recovery fails with the “Wasn’t able to find source file item [(/media/dm-0/root/test, /root/test)], skipping overwriting” error if the original file still exists.

Guest indexing

  • Catalog service may leak memory when processing very large amounts of guest indexes.

Built-in WAN Acceleration

  • Under rare circumstances, Backup Copy job to Linux-based repository may experience data mover process SegFault crash with Back-trace containing IP 0x848deff.
  • If target WAN accelerator runs out of disk space during digests generation, the job starts failing repeatedly with the following error: “An existing connection was forcibly closed by the remote host –tr: Cannot write data to the socket

Backup from Storage Snapshots

  • Processing of VM on assigned to backup proxy with multiple FC adapters may fail with “Server with WWN is not found on HP 3PAR host” if some FC adapters do not have access to production 3PAR storage.
  • In certain HP 3PAR storage configuration, VM processing may fail with the “Unable to find volume with Node WWN and LUN ID” error.
  • Processing of VM with workingDir parameter set to a different HP 3PAR or HP StoreVirtual datastore fails with the “Unable to find matching device for LVID” error.
  • HP StoreVirtual snapshots may fail to be deleted with the “Failed to connect to host” error due to reaching the limit of SSH connections to StoreVirtual storage after certain time.

Tape

  • Under certain circumstances, the tape backup job may fail with the “Violation of PRIMARY KEY constraint ‘PK_Tape.file_versions’. Cannot insert duplicate key in object ‘dbo.Tape.file_versions’” error.
  • Restoring a file split across multiple tapes may fail with the “Block offset in source file differs from the current write position in target file” error.
  • Very long read or write operation may fail on certain tape libraries (such as IBM TS3100) with the following error: “The requested resource is in use“.
  • Tape jobs using standalone drives hand on second task with the “Tape medium was used in backup session, NOT VALID” error.
  • Tape media is not automatically erased in standalone tape drives when “Do not protect” option is enabled on a media pool.
  • Returning tape mediums without barcode from Offline after re-inventorization results in “Tape requires return to original, but it is already filled” error if the original slot is occupied.
  • VBRCatalog\Tapes\Media\ folder is not properly cleaned up upon various tape media operations, resulting in thrash accumulation.
  • Changing a tape may result in the “Subquery returned more than 1 value” error.
  • Corrected statuses for a number of tape drive alerts according to their real severity.

Configuration backup

  • Configuration backup fails if a tape changer connected to the backup server has null byte in its serial number with the “‘.’, hexadecimal value 0x00, is an invalid character” error.
  • Configuration backup of very large configuration database fails with the “Compressed or Uncompressed size, or offset exceeds the maximum value. Consider setting the UseZip64WhenSaving property on the ZipFile” error.
  • Under certain circumstances, configuration backup restore fails with the “String or binary data would be truncated. The statement has been terminated” error

Enterprise Manager

  • The required SureBackup job does not start and stop immediately after Virtual Lab request is approved or dismissed in the Enterprise Manager.
  • Adding a domain user to portal roles results in the “The specified directory service attribute or value does not exist” warning.
  • “Keep only last” setting of the Event history accepts unsupported values, resulting in Options dialog on attached backup servers fail to with value not valid errors.

User Interface

  • One of the configuration database tables is never cleaned up, which may cause user interface sluggishness after using the product for extended time.
  • Sorting by Total Size and Free Space columns does not work correctly in Files > VMware vSphere > Datastores view.
  • Sorting by Type, Size or Modified columns in Files > VMware vSphere > Datastores makes the user interface hang.

This patch also contains all fixes from Patch 1 and R2 update

More Information:

Prior to installing this patch please reboot the Veeam server to clear any locks on the Veeam services and when the reboot is done, please stop all the Veeam services and apply the patch.

When the patch is installed manually, start the Veeam services and open Veeam, and allow it to update all the components.

To obtain the patch please download it from: Download


VCAP5-DCA Objective 7.1 – Secure ESXi Hosts

Knowledge

  • Identify configuration files related to network security
  • Identify virtual switch security characteristics

Skills and Abilities

  • Add/Edit Remove users/groups on an ESXi host
  • Customize SSH settings for increased security
  • Enable/Disable certificate checking
  • Generate ESXi host certificates
  • Enable ESXi lockdown mode
  • Replace default certificate with CA-signed certificate
  • Configure SSL timeouts
  • Configure vSphere Authentication Proxy
  • Enable strong passwords and configure password policies
  • Identify methods for hardening virtual machines
  • Analyze logs for security-related messages
  • Manage Active Directory integration
    Continue reading