Veeam SQL Server backup – A VSS critical writer has failed. Writer name: [SqlServerWriter]/ Cannot open backup device

Getting errors when trying to backup SQL server with veeam

In Job.backup.logs getting

<01> Error    Failed to create snapshot: Backup job failed.
[01.12.2021 08:50:06] <01> Error    Cannot create a shadow copy of the volumes containing writer's data.
[01.12.2021 08:50:06] <01> Error    A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{0984f742-a29b-4d5c-ac1e-b18489e72da3}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4]. (System.Exception)
[01.12.2021 08:50:06] <01> Error       at Veeam.EndPoint.SysUtils.CEpSnapshotHolder.WaitForSnapshotCreation()
[01.12.2021 08:50:06] <01> Error       at Veeam.EndPoint.CEndPointAutoSnapshot.CreatePreparedFreezedSnapshot(CVolumeToSnapshot[] volumesToSnapshot, String[] excludedFolders, EVssBackupType vssBackupType, Boolean disableBcdUpdate)
[01.12.2021 08:50:06] <01> Error       at Veeam.EndPoint.CEndPointAutoSnapshot.CreateFreezedSnapshot(CVolumeToSnapshot[] volumesToSnapshot, String[] excludedFolders, Boolean disableBcdUpdate, IEpAutoSnapshotCreationLogger logger)
[01.12.2021 08:50:06] <01> Error       at Veeam.EndPoint.Sources.CVssSnapshotCreator.CreateSnapshot(CShadowVolumesLayout disksLayout, CLiveVolumeInfo[] volumes, CVolumesList nonShadowVolumes, CPartitionInfo[] partitionsToBackup, IDbExcludesProvider dbExcludesProvider, CBackupCreationUsnProvider backupCreationUsnProvider, Boolean dummySnapshot, CShadowVolumesLayout& shadowVolumesLayout)

Applications events in Event Viewer:

Sqllib error: OLEDB Error encountered calling ICommandText::Execute. hr = 0x80040e14. SQLSTATE: 42000, Native Error: 3013
Error state: 1, Severity: 16
Source: Microsoft SQL Server Native Client 11.0
Error message: BACKUP DATABASE is terminating abnormally.
SQLSTATE: 42000, Native Error: 3201
Error state: 7, Severity: 16
Source: Microsoft SQL Server Native Client 11.0
Error message: Cannot open backup device '{44972488-CB50-4FC6-A564-F4960056B1FA}1172'. Operating system error 0x80070002(The system cannot find the file specified.).

vssadmin list writers shows SQL VSS writers as “Retriable error

In SQL Server management studio, after executing

SELECT COUNT(*) FROM sys.databases where database_id not in ( 1,2,3,4 )

Got output of over 1000 databases.

Google search revealed that changing max worker threads Option in SQL server can fix the issue.

My machine has 4 CPUs so tried changing value from 0 to 512 didn’t help

So i changed Maximum worker threads to 3000 and backup was successful

The reason is that there are not enough free threads while creating the volume shadow copy with large number of databases.

Advertisement

Your computer can’t connect to the remote computer because the Remote Desktop Gateway server address is unreachable or incorrect – RemoteApps, Windows server 2012R2

We have Remote Desktop apps hosted on Windows server 2012 R2, for past few years. Everything worked fine for years, then suddenly, we started to get following error:

“Your computer can’t connect to the remote computer because the Remote Desktop Gateway server address is unreachable or incorrect. Type a valid Remote Desktop Gateway server address”

Same fix applied to “This computer can’t connect to remote computer, Try connecting again.”

Opened https://remote.example.com/RDWEB, type credentials, remote apps are shown, then after clicking on application, again prompted for password and getting error above.Restarted RDP gateway, recreated Remote session no help. The strange things is that all works fine from the LAN, but not from the internet. Even stranger things is that Remote Application can be launched from mobile phone outside company network.

After some googling, found a “Workaround”:

In Remote properties of RDP Gateway, server, unchecked “Allow connections only from computers Running Remote Desktop with Network Layer Authentication (recommended) “

Second workaround is to specify RDP as Security layer:

Computer Configuration > Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Security

Required use of specific security layer for remote (RDP) connections, security layer: RDP

For permanent fix, try regenerating RDP self-signed certificate:

First, remove old cert from Remote Desktop-Certificate folder then restart Remote Desktop Configuration service

net stop SessionEnv
net start SessionEnv

In case cert was not generated, you’ll get following error:

The RD Session Host Server has failed to create a new self signed certificate to be used for RD Session Host Server authentication on SSL connections. The relevant status code was Access is denied.

Make a copy of C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys folder and delete content of MachineKeys folder, then set NTFS Permissions to Full control to local administrator group to the same folder:

Properites-security-Advanced-Replace all child object permissions\

Restart Remote desktop service, check self-signed SSL cert was generated, then RDP should work.

PowerShell – remove blank/empty rows from CSV file

Input.CSV file was like this:

Entity;Type
user1;Account
;
User2;Account
;
;
user3;Account
;

I wanted to remove all empty/blank lines from csv file

Get-Content "C:\input.csv" | Where { $_.Replace(";","") -ne "" } | Out-File C:\output.csv

output.csv

Entity;Type
user1;Account
User2;Account
user3;Account

PowerShell – Check if port is open

Script will check if particular port is opened or not

Function Check-Port{
[cmdletbinding()]
Param (
[parameter(Mandatory=$true)]
[string]$ipaddress,
[string]$port
)
# End of Parameters
Process{
  $Date = Get-Date
  $MyDate =  " " + $Date.Day + "." + $Date.Month + "." + $Date.Year + " " + $date.Hour + ":" + $date.Minute
  
  Try{
     $connection = $null
     $connection = New-Object System.Net.Sockets.TcpClient($ipaddress, $port)
  }

  Catch{
       Write-Warning $Error[0]
     }
  if ($connection.Connected){
     Write-Host "Port $port is open - $MyDate" -ForegroundColor Green
    }
   }
 }

Usage example and output

Set-AdfsSslCertificate Access denied

Recently i renewed SSL certificate in ADFS console, but one more step remained. Point ADFS to the new certificate. Quick google search discovered what command needs to be used on ADFS server:Set-AdfsSslCertificate -Thumbprint ac9XXXXXXXXXXXXXX

Thumbrint of the cert can be found in cert properties

I Run command as administrator (domain admin user) but faced following error:

PS0317: One or more of AD FS servers returned errors during execution of command 'Set-AdfsSslCertificate'. Error information: PS0316: AD FS Server:
'localhost', Error: 'Connecting to remote server localhost failed with the following error message : Access is denied. For more information, see the
about_Remote_Troubleshooting Help topic.'.

Enter-PSSession localhost command alone gave same error, but this one worked:

Enter-PSSession localhost -Credential $(Get-Credential)

Id Name            ComputerName    ComputerType    State         ConfigurationName     Availability
 -- ----            ------------    ------------    -----         -----------------     ------------
 19 WinRM19         localhost       RemoteMachine   Opened        Microsoft.PowerShell     Available

So tried again by modifying above command:

Invoke-Command -ComputerName localhost -Credential $(Get-Credential) -ScriptBlock {Set-AdfsSslCertificate -Thumbprint ac9XXXXXXXXXXXXXX}

But again same error.

Created new local admin account-no help.

One way to overcome this issue is to Run Set-AdfsSslCertificate command as built-in local Administrator account on ADFS machine.

Second way is to remove user from Protected group

Invoke-AdfsFarmBehaviorLevelRaise – The WinRM client sent a request to an HTTP server

After upgrading from ADFS 3.0 to 4.0 next step was to upgrade Farm behavior level (FBH) from server 2012 to server 2019 on primary server in the farm,and i faced following error

Invoke-AdfsFarmBehaviorLevelRaise : Database upgrade cannot be performed on adfs.example.com. Error: Connecting to remote server 
adfs.example.com failed with the following error message : The WinRM client sent a request to an HTTP server and got a response saying the 
requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol. For more 
information, see the about_Remote_Troubleshooting Help topic..
At line:1 char:1
+ Invoke-AdfsFarmBehaviorLevelRaise

I checked current WinRM settings:

winrm enumerate winrm/config/listener

Listener [Source="GPO"]
    Address = *
    Transport = HTTP
    Port = 5985
    Hostname
    Enabled = true
    URLPrefix = wsman
    CertificateThumbprint
    ListeningOn = 127.0.0.1, 192.21.21.19

It looked good, but after some googling turned out that issue is that IPv6 address also needs to be specified in winrm output,because on server IPv6 is turned on in TCP/IP properties.

Easiest solutions would be to simply untick Internet protocol Version 6, but in case IPv6 needs to be configured, the easiest way is to configure via GPO

Computer configuration/Windows settings/Administrative templates/Windows Components/Windows Remote Management (WinRM)/WinRM Service/Allow Remote server Management through WinRM. Specify asterix (*) for IPv6 filter

After applying GPO i was able to raise ADFS FLB

winrm enumerate winrm/config/listener
Listener [Source="GPO"]
    Address = *
    Transport = HTTP
    Port = 5985
    Hostname
    Enabled = true
    URLPrefix = wsman
    CertificateThumbprint
    ListeningOn = 127.0.0.1, 192.21.21.19, ::1, fe80::5cb5:74ea:9e1d:1b0c%14

Removing old domain from NTFS and shared permissions after Domain migration using Subinacl tool

A subinacl is Windows command line utility used to manage Share and NTFS permissions.

I used AMDT tool to migrate users, groups,shares and computer objects from one domain to another, during computer migration, i choose to retain old and to add new permissions so users from both domains can access to shares during migration process.

In my case source domain is source.local and destination domain is destination.local and i had shared and NTFS permsissions from both domains.

 

We first need to export shared and NTFS permissions to file

cd "C:\Program Files (x86)\Windows Resource Kits\Tools"
Subinacl /noverbose /output=C:\NTFSPermissions.txt /subdirectories "C:\SHARE\*.*"

Subinacl /noverbose /output=C:\SharedPermissions.txt /share "\\FS\SHARE"

# or Subinacl /outputlog="c:\outputlog.txt"  /Subdirectories C:\SHARE*.*  /changedomain=source=destination

Then remove entries for old domain (source.local)

subinacl /playfile C:\NTFSPermissions.txt /replacestringonoutput=source=destination

subinacl /playfile C:\SharedPermissions.txt/replacestringonoutput=source=destination

Adding new domain ACE’s entries to shared and NTFS permissions

In case we didn’t use AMDT to migrate fileserver computer, ie, we just copied shared folder together with permissions

ROBOCOPY "\\source\sharelocation" "\\destination\sharelocation" /MIR /SEC /LOG:location:\nameoflogfile.

Only ACE entries from source (old) domain is present

In that case, we can use /migratetodomain subinacl switch to add ACE entries of new domain to shared and NTFS permissions.

# add ACE entries to root folder only
Subinacl /outputlog="c:\outputlog.txt"  /subdirectories "C:\SHARE"  /migratetodomain=source=destination

# add ACE entries to subfolder/files

Subinacl /outputlog="c:\outputlog.txt"  /subdirectories "C:\SHARE\*.*"  /migratetodomain=source=destination

Windows Certification Authority (CA) – Request client SSL certificate on behalf of another user

In order to issue SSL certificate on behalf of another user we first need Enrollment Agent Certificate,in order to create it, we need to copy Enrollment Agent certificate template.

Creating and issuing of Enrollment Agent Certificate

In Certification authority console, under Certificates templates, right click on Certificate templates-Manage

Right click “Enrollment Agent”-Duplicate template

Under Cryptography tab make sure Microsoft Enhanced Cryptography Provider V1.O is selected,minimum key size:2048

Under Request handling, check “Allow private key to be exported”

Under security tab make sure Authenticated users have rights “Read” and “Enroll”.

Right click Certificate Templates-New-Certificate Template to Issue

Select template and click OK.

Configuring certificate template for enrollment

On certificate which should be enrolled to users, right click-Properties-Issuance requirements tab

This number of authorized signatures:1

Policy type required in signature:Application policy

Application policy:Certificate request agent

Enrollment of Enrollment Agent Certificate

On your workstation machine open Certificate console,Expand Personal,right click on Certificates-All tasks-Request New Certificate

Click Next twice and select certificate we just issued-Enroll

Issuing certificate for other user

On your workstation machine open Certificate console,Expand Personal,right click on Certificates-All tasks-Advanced operations-Enroll of behalf of

In signing certificate page click browse-certificate should pop-up automatically

Select certificate you want to issue to user.

When asked for user click browse, in Location specify domain and type user for which you need to issue certificate. You can issue cert only for one user.

Enrollment agents

Enrollment agents are users who can issue certificates for other users.

On CA machine, open Certification Authority console, right click on machine name-Properties-Enrollment agents.

Here you can specify who can issue specific templates.

Powershell script for issuing multiple SSL certificates

This script issues certificated for multiple enabled AD users (search filter can be modified)

function GenerateSSLCert {
    [CmdletBinding()]
    param (
     
    [Parameter(Mandatory=$true)][string]$userName
    )
      
    $PKCS10 = New-Object -ComObject X509Enrollment.CX509CertificateRequestPkcs10
    # Certificate template name for issuing to users
    $PKCS10.InitializeFromTemplateName(0x1,"GP") 
    $PKCS10.Encode()
    $pkcs7 = New-Object -ComObject X509enrollment.CX509CertificateRequestPkcs7
    $pkcs7.InitializeFromInnerRequest($pkcs10)
    $pkcs7.RequesterName = "test\$userName"
    $signer = New-Object -ComObject X509Enrollment.CSignerCertificate
    # Thumbrint of Enrollment Agent certificate
    $signer.Initialize(0,0,0xc,"xxxxxxxxxxxxxxxxxxxxxxxx")
    $pkcs7.SignerCertificate = $signer
    $Request = New-Object -ComObject X509Enrollment.CX509Enrollment
    $Request.InitializeFromRequest($pkcs7)
    $Request.Enroll()
}
   

Get-ADUser -SearchBase "OU=Users,DC=test,DC=local" -filter * | Where { $_.DistinguishedName -notmatch  "OU=DisabledUsers|OU=OpenVPNUsers|OU=ServiceUsers" -and $_.Enabled -eq $True } | ForEach-Object {
 
   Try{
      $user = $_.SamAccountName
      GenerateSSLCert $user 
   }
   Catch{
      Write-Host $_.Exception.Message
      Write-Host "SSL certicate not created for user:$user"
   }
 }

PaloAlto firewall – Remote Access VPN with Two-Factor Authentication

In order for user to be able to make VPN with global protect, SSL certificate needs to be installed on machine, also, username specified in certificate subject must exist on On-premise domain controller.

In this example certificates are issued from internal Certificate authority.

Generating certificate for PaloAlto firewall

We need SSL to issue certificate assigned to public DNS name of firewall.

Device-certificates-device certificates-generate

Click the checkbox next to the Certificate Name or any whitespace on that line to select it and click export certificate

File with csr extension will be created, copy that file to Certification authority machine, open cmd and run certreq -submit -attrib "CertificateTemplate:template_name" where “template_name” is the name of Certification template.

You’ll be prompted for csr file

Then select certification authority

Certificate will be created with cer extension.

Copy certificate to machine on which you created certificate request. Now

click on Device-certificates-device certificates on PaloAlto web interface, click import in bottom panel.

In the Import Certificate dialog, type the name of the pending certificate. It must match exactly the name we used when created certificate request

Importing CA root certificate

On Certificate authority open MMC-Local computer-Trusted root Certification Authority-Certificates-right click on CA certificate-all tasks-export-select Base-64 encoded

Save file with cer.extension

On PaloAlto Device-certificates-device certificates-import

Creating LDAP profile

Specify Domain controllers for user authentication

Device-server profile-Add-type:active-directory, specify Domain controllers,base DN for user search and service account and password for searching AD.

Creating Authentication profile

Device-Authentication profile-Add

Specify Login Active directory Attribute, select LDAP as type and select LDAP profile created in previous step

Creating Certificate profile

Device-certificate manager-Certificate profile-Add

In Username field select Subject Alt and select Email or UserPrincipal name, this means that PaloAlto get username from SSL certificate and it will be automatically populated in GlobalProtect username field.

In Advanced tab select Security group for VPN users. Users who need VPN need to be members of this group.

Creating SSL/TLS profiles

This profile defines which SSL certificate will be used and SSL version

Device-Certificate management-SSL/TLS Service profile-Add

Specify PaloAlto SSL certificate we generated and uploaded in one of previous steps and minimal TLS version.

GlobalProtect portal configuration

Network-GlobalProtect-Portals-Add

On general page select publicy accessible interface and IP address

In Authentication tab specify SSL/TLS Service profile we created in previous step,Certificate profile and click Add

Specify authentication profile

In Agent tab,uder Agent,click Add

In Authentication tab,For Client Certificate select Local and select PaloAlto FW certificate

In External tab specify DNS name of PaloAlto firewall

Click OK, now under Agent tab-Under Trusted Root CA click Add

Network-GlobalProtect-Gateways-Add and add CA root certificate we created earlier.

Configuring GlobalProtect Gateway

Network-GlobalProtect-Portals-Gateways-Add

Specify PaloAlto firewall publicly accessible interface and IP address

Under Authentication tab select SSL/TLS Service profile and certificate profile and click Add

Select Authentication profile, set name and click OK

Under Agent tab specify tunnel settings

Under Client setting specify IP pools and Access routes

Under Network Services specify primary and secondary DNS servers

Commit changes

Generating user certificates

Now we need to create and issue SSL certificates for users who will connect to VPN using global protect

On Issuing certificate authority, in Certification authority console, right click on Certificate templates-Manage

Certification Templates console will launch, right click on template User-duplicate template.

In tab Subject Name make sure settings are as in picture bellow

In General tab select validity period, once done,click OK.

In security tab, make sure Domain users have rights to Enroll and Read.

Once all is set click OK, right click Certificate Templates-New-Certificate Template to Issue

Select template and click OK.

Installing SSL certificate on user computers

To use VPN, install GlobalProtect software,.

Open Local computer certificate store (start-run-certlm.msc).Expand Personal,right click on Certificates-All tasks-Request New certificate,select Certificate template and install certificate

Now you should be able to initiate VPN connection

Rsync on Windows

Rsync, which stands for “remote sync”, is a remote and local file synchronization tool. It uses an algorithm that minimizes the amount of data copied by only moving the portions of files that have changed.It’s native linux command, but thanks to Cygwin, we can use it on Windows too.

Installing Cygwin on Windows

Cygwin is a Unix-like environment and command-line interface for Microsoft Windows.It’s a repository of open source software compiled with this dll. In other words, it’s package manager of Linux command line tools which can be run on Windows.

Download cygwin and run installation, in View select Full, in searchbox type rsync, and select rsync under Net category, check Src, and choose version under New

Do the same for openssh (Windows 10 and Server 2019 have this package shipped)

Click next to install both packages.

By default, cygwin is installed on C:\cygwin64 folder, tools are in bin folder

Creating SSH keys (keypair) on Windows

In order to copy files from Windows to Linux we need to authenticate on Linux box, either using credentials or ssh keys. I find ssh more convenient authentication type, so we need to create private and public key on Windows and copy public key to linux machine.

Creating ssh keys

Open Cygwin

Create keys

ssh-keygen

Copy content of C:\cygwin64\home\Administrator.ssh\id_rsa.pub to linux box (/root/.ssh/authorized_keys file)

In cygwin terminal type

ssh-copy-id

Test access from Windows to linux. In this example 192.168.1.11 is IP of Linux machine, userame root

ssh root@192.168.1.11

Copy files from Windows to Linux

In this example we’ll copy content of C:\inetpub folder to /data folder on Linux box

Again, from cygwin terminal type

rsync -avm //localhost/c$/inetpub/ root@192.168.1.11:/data

Simple copy usually is not enough, we need to set ownership and permission on the destination.

There are three basic file system permissions, or modes, to files and directories:

• read (r)
• write
• execute (x)

Each mode can be applied to these classes:

• user (u)
• group (g)
• other (o)

The user is the account that owns the file. The group that owns the file may have other accounts on the system as members. The remaining class, other (sometimes referred to as world), means all other accounts on the system.

To see all permissions on files and folder use following command: ls -l /folder

Following command sets read (r) for user (u),and read (r),write and execute (x) for group (g ) and set ownership to user root and group root.

rsync -avm --chmod=u=r --chown=root:root //localhost/c$/inetpub/ root@192.168.1.11:/data

ls -l /data/
total 0
dr--rwx--- 3 root root 19 Jun 22 10:22 custerr
dr-------- 4 root root 64 Jun 22 10:24 history
dr--rwx--- 3 root root 22 Jun 22 10:22 temp
dr--rwx--- 2 root root 44 Jun 22 10:22 wwwroot

Following command will give user (u) and group (g) read (r) and execute (x) permission on destination folder, and set ownership to user root and group root.

rsync -avm --chmod=ug=rx --chown=root:root //localhost/c$/inetpub/ root@192.168.1.11:/data

ls -l /data/
total 0
dr-xr-x--- 3 root root 19 Jun 22 10:22 custerr
dr-xr-x--- 4 root root 64 Jun 22 10:24 history
dr-xr-x--- 3 root root 22 Jun 22 10:22 temp
dr-xr-x--- 2 root root 44 Jun 22 10:22 wwwroot

Following command will set read (r) and write permissions to user (u),group(g) and all (o) group, ownership to user root and password root

rsync -avm --chmod=ugo=rw --chown=root:root //localhost/c$/inetpub/ root@192.168.1.11:/data

ls -l /data/
total 0
drw-rw-rw- 3 root root 19 Jun 22 10:22 custerr
drw-rw-rw- 4 root root 64 Jun 22 10:24 history
drw-rw-rw- 3 root root 22 Jun 22 10:22 temp
drw-rw-rw- 2 root root 44 Jun 22 10:22 wwwroot

To avoid copying permissions to destination, it’s enough to omit -a switch in rsync command. To copy empty directories use –rWH switches and omit -m switch.

In order to execute above command from command prompt/Powershell, add C:\cygwin64\bin to system path environmental variable