Taryn Pratt - Pivots and other SQL fun

Testing Backups and Offloading CheckDB

Mon, 19 Apr 2021 06:00:00 +0000

While every DBA knows they need to backup all their databases, not all may realize the importance of testing those backups. Performing backups is pointless, if you’re unable to restore them.

I wanted to restore our backups on a regular basis, so I set up a process to test them automatically, on a nightly basis.

Background

Early on when I started working on the SQL Servers at Stack Overflow, we were taking daily backups. We had a handful of databases that were being restored for other processes, but the majority weren’t actively tested to ensure the backups were good. Since you never want to be in a situation where you need to restore a database and find it doesn’t work, my goal was to create a process to automatically restore our backups to a separate server, and then run DBCC CHECKDB on it.

Now, you might be wondering why I’d run CHECKDB on the backup after I restore it and asking yourself “aren’t you supposed to run CHECKDB on the SQL Server you take a backup on?”

Yes, technically many people suggest you do that, but we don’t, because while we take full backups on the primary replica in our Always On Availability Groups, we don’t have maintenance windows, and running CHECKDB on a very busy production SQL Server like we have for Stack Overflow can cause issues (ask me how I know). We can’t easily run a full CHECKDB in production, so since I was going to be restoring the backups to make sure they worked, I might as well run CHECKDB on it to verify everything.

Automating the Testing

Before I jumped into creating a new process there were some questions I needed to answer for this project:

What databases do we want or need to test the backups for?
Where can we perform this testing? Do we have a server available for use?
How often do we want to test the backups? Nightly? Weekly? What would the cadence be for this process?

The answer to the first question was pretty clear, since we can’t easily run DBCC CHECKDB in production basically, any database that was in an Always On Availability Group was on the list to have the backup restored and tested.

Next, I needed a place to perform the testing. Thankfully, we had a separate server where we were already restoring some backups, so it was the perfect candidate for this. The only issue was lack of disk space for some of the large databases. We resolved the lack of space issue by purchasing a couple of new NVMEs which gave us a 14TB drive to use for the restores.

Lastly, it was figuring out how often we wanted to test our restores. Since we take full backups nightly, we decided we would restore all databases daily, then run DBCC CHECKDB against it. Is that a little much? Maybe, but it also gives us confidence that our backups are working.

Now that I had the answer to the questions above, it was time to create a process that would do the following:

Automatically add or remove databases that need testing
Restore the our backups on a separate server
Run DBCC CHECKDB on the backup after being restored

Maintaining a List of Databases to Test

I knew the initial list of databases we wanted to test, but what if that list changed? What would happen if we added new databases, or removed databases, and no longer had backups? I didn’t want the nightly process to fail, and even though I typically know when we add or remove databases, I didn’t want to hand-hold and edit the list manually, I wanted it to be dynamic.

To do this, I had to save the list of databases to restore. I did this by first creating a table in the master database on the server we were using for backup and restore testing. The table structure is:

1
2
3
4
5
6
7
8


CREATE TABLE [dbo].DatabasesToRestore(
 [Id] [int] IDENTITY(1,1) NOT NULL,
 [Name] [sysname] NOT NULL,
 [AvailabilityGroup] nvarchar(100) NULL,
 IsActive bit NOT NULL DEFAULT(1)
 CONSTRAINT [PK_DatabasesToRestore] PRIMARY KEY CLUSTERED ([Id] ASC)
) ON [PRIMARY]
GO

This table stores basic information about each database, including name, but also the name of the AvailabilityGroup that the database is in. I purposely stored the AG name because of the way we save our backups. Our backup directory structure uses the Availability Group name instead of the name of the SQL Server in the file path because it allows the backup locations to stay consistent no matter what server is the primary. The file share for backups uses this directory structure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


|--- server
|--- Backups
|--- SQL
|--- AG-name1
|--- Database1
--- database1_20210101.bak
--- database1_20210102.bak
|--- Database2
--- database2_20210101.bak
|--- AG-name2
|--- Database1
|--- Database2

By storing the name of the Availability Group for each database, it allowed me to dynamically create the path to the backup for testing. It also made the process to find additions and deletions for each availability group more efficient, so I didn’t have to manually adjust the list of databases that would need nightly testing. All that was needed was the initial population of the databases to restore, and then the stored procedure would keep the list up to date. The stored procedure that I wrote is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85


CREATE OR ALTER Procedure [dbo].[spUpdateDatabasesToRestore]
AS
BEGIN
 DECLARE @fileSearch nvarchar(500) = '',
 @backupPath NVARCHAR(500) = '',
 @error nvarchar(600);

 DECLARE @ag_name nvarchar(100)

 DECLARE ag_cursor CURSOR FOR
 SELECT DISTINCT AvailabilityGroup
 FROM dbo.DatabasesToRestore
 OPEN ag_cursor
 FETCH NEXT FROM ag_cursor INTO @ag_name
 WHILE @@FETCH_STATUS = 0
 BEGIN

 PRINT 'Getting databases for AG: '+ @ag_name

 -- go through the list of availability groups and their backup path
 -- update the list of databases to restore
 -- adding new ones and deleting old ones that are no longer active databases

 SET @backupPath = concat('\\fileserver\Backups\SQL\', @ag_name, '\')
 SET @fileSearch = 'DIR /S /b/a-d/od/t:c ' + @backupPath;

 DECLARE @files table (ID int IDENTITY, FileName varchar(max), DBName sysname null, FileDate date null)

 INSERT INTO @files (FileName)
 EXEC master.sys.xp_cmdshell @fileSearch

 UPDATE @files
 SET DBName = Left(replace(FileName, @backupPath, ''), charindex('\', replace(FileName, @backupPath, ''))-1),
 FileDate =
 cast(substring(right(replace(FileName, @backuppath, ''), len(replace(FileName, @backuppath, ''))
 -charindex('\', replace(FileName, @backuppath, ''))),
 charindex('_FULL_', right(replace(FileName, @backuppath, ''),
 len(replace(FileName, @backuppath, ''))-charindex('\', replace(FileName, @backuppath, '')))) + 6, 8) as date)
 WHERE FileName like '%.bak%'

 PRINT 'inserting new databases for AG: '+ @ag_name

 -- Insert new databases into the Databases table
 INSERT INTO DatabasesToRestore (Name, AvailabilityGroup)
 SELECT DBName, AGName = @ag_name
 FROM
 (
 SELECT DBName,
 rn = row_number() over(partition by DBName order by FileDate desc)
 FROM @files
 WHERE FileDate >= GETDATE()-7
 AND FileName LIKE '%'+@ag_name+'%'
 ) f
 WHERE rn = 1
 AND NOT EXISTS (SELECT 1
 FROM dbo.DatabasesToRestore d
 WHERE f.DBName = d.Name
 AND d.AvailabilityGroup = @ag_name
 AND d.IsActive = 1)

 PRINT 'Removing any old databases for AG: '+ @ag_name

 -- mark any Databases that are no longer producing backups as IsActive = 0
 UPDATE d
 SET d.IsActive = 0
 FROM dbo.DatabasesToRestore d
 LEFT JOIN
 (
 SELECT DISTINCT DBName, AGName = @ag_name
 FROM @files
 WHERE FileDate >= GETDATE()-7
 AND DBName IS NOT NULL
 AND FileName LIKE '%'+@ag_name+'%'
 ) f
 ON d.Name = f.DBName
 AND d.AvailabilityGroup = f.AGName
 WHERE d.IsActive = 1
 AND d.AvailabilityGroup = @ag_name
 AND f.DBName IS NULL

 FETCH NEXT FROM ag_cursor INTO @ag_name
 END
 CLOSE ag_cursor
 DEALLOCATE ag_cursor
END

This procedure uses xp_cmdshell to get the list of files in the backup directory, then adds new databases that are missing from the current list, and removes databases that haven’t had a new .bak file in the last seven days. Finally, we have a scheduled job that executes this stored procedure nightly to keep the table of DatabasesToRestore up to date.

Restoring Database Backups

Once I had a list of the databases to restore, and a way to maintain that list on its own, it was time to get the process in place to restore the backups for testing nightly.

In addition to restoring the backups, I also wanted to keep a record of what was being restored, as a reference, in the event there were issues or failures with the nightly job. To do this, I created another table which I put in the master database.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


CREATE TABLE dbo.DatabaseRestoreLog
(
 [ID] [int] IDENTITY(1,1) NOT NULL,
 [DatabaseId] [int] NOT NULL,
 [RestoredName] [sysname] NULL,
 [RestoredDate] [date] NOT NULL,
 [RestoreStartTime] datetime NULL,
 [RestoreEndTime] datetime NULL,
 [DBCCStartTime] datetime NULL,
 [DBCCEndTime] datetime NULL,
 [PassedDBCCChecks] [bit] NOT NULL DEFAULT (0)
 CONSTRAINT [PK_DatabaseRestoreLog] PRIMARY KEY CLUSTERED ([ID] ASC),
 CONSTRAINT [FK_DatabaseRestoreLog_DatabasesToRestore]
 FOREIGN KEY ([DatabaseId]) REFERENCES dbo.DatabasesToRestore ([Id])
);

This table allows the job to capture critical details, like how long the restore took, it also tracked whether or not the backup passed the DBCC checks (more on that later).

As I mentioned earlier, we were restoring some databases in another process, so instead of inventing something from scratch, I decided to modify the existing stored procedure and expand it for the restore testing. Again, this procedure utilizes xp_cmdshell and loops through all of the databases in the DatabasesToRestore table and looks for the oldest databases that haven’t been restored recently, creates the path to the backup, and then grabs the most recent .bak file to restore for testing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201


CREATE OR ALTER Procedure dbo.spBackupRestoreTesting
 @TimeLimit int = 43200, -- the amount of time we want to limit the restores - the default is 12 hours
 @exec BIT = 0
AS
BEGIN
 DECLARE @fileSearch nvarchar(500) = '',
 @backupFile nvarchar(500),
 @data_dir nvarchar(100) = 'F:\DBRestore\',
 @cmd nvarchar(max),
 @dropCmd nvarchar(500),
 @RestoredDBName sysname,
 @backupPath NVARCHAR(500) = '',
 @error nvarchar(600);

 -- Set the time we started the restore process so we know when we want to exit
 -- if it hits the TimeLimit, we'll stop for the day
 DECLARE @StartTime datetime
 SET @StartTime = GETDATE()

 DECLARE @fileList Table (rowNumber int identity(1,1), backupFile nvarchar(255), backupPath nvarchar(500));

 DECLARE @FileListTable Table (
 [LogicalName] nvarchar(128),
 [PhysicalName] nvarchar(260),
 [Type] char(1),
 [FileGroupName] nvarchar(128),
 [Size] numeric(20,0),
 [MaxSize] numeric(20,0),
 [FileId] bigint,
 [CreateLSN] numeric(25,0),
 [DropLSN] numeric(25,0),
 [UniqueId] uniqueidentifier,
 [ReadOnlyLSN] numeric(25,0),
 [ReadWriteLSN] numeric(25,0),
 [BackupSizeInBytes] bigint,
 [SourceBlockSize] int,
 [FileGroupId] int,
 [LogGroupGUID] uniqueidentifier,
 [DifferentialBaseLSN] numeric(25,0),
 [DifferentialBaseGUID] uniqueidentifier,
 [IsReadOnly] bit,
 [IsPresent] bit,
 [TDEThumbprint] varbinary(32),
 [SnapshotUrl] nvarchar(360));

 DECLARE @database_id int
 DECLARE @database_name sysname
 DECLARE @ag_name nvarchar(100)
 DECLARE @last_restore_date date

 -- loop through the list of databases
 -- restore them, then run DBCC checks on them
 WHILE GETDATE() < DATEADD(ss,@TimeLimit,@StartTime)
 BEGIN
 -- grab the oldest databases that haven't been restored recently
 SELECT TOP 1 @database_id = d.Id,
 @database_name = d.Name,
 @ag_name = d.AvailabilityGroup,
 @last_restore_date = drl.LastRestoredDate
 FROM DatabasesToRestore d
 LEFT JOIN
 (
 -- get the last date each database was restored
 SELECT
 DatabaseId,
 LastRestoredDate = MAX(RestoredDate)
 FROM DatabaseRestoreLog
 GROUP BY DatabaseId
 ) drl
 ON d.Id = drl.DatabaseId
 WHERE d.IsActive = 1
 AND (drl.LastRestoredDate <> CAST(GETDATE() AS DATE)
 OR drl.LastRestoredDate IS NULL)
 ORDER BY drl.LastRestoredDate

 IF @@ROWCOUNT = 0
 BEGIN
 BREAK
 END

 PRINT CONCAT('Restoring : ', @database_name, ' in AG ', @ag_name, ' last restored date: ', @last_restore_date)

 SET @backupPath = CONCAT('\\fileserver\Backups\SQL\', @ag_name, '\', @database_name, '\')
 PRINT @backupPath

 SET @fileSearch = 'DIR *.bak /b /O:D ' + @backupPath;

 INSERT INTO @fileList(backupFile)
 EXEC master.sys.xp_cmdshell @fileSearch;
 UPDATE @fileList Set backupPath = @backupPath;

 SELECT Top 1 @backupFile = backupFile, @backupPath = backupPath
 FROM @fileList
 WHERE backupFile Like @database_name + '%'
 ORDER BY backupFile DESC

 IF @backupFile Is Not Null
 BEGIN
 DECLARE @fullPath nvarchar(500) = @backupPath + @backupFile;
 PRINT 'Backup File Found: ' + @fullPath;

 INSERT INTO @FileListTable
 EXEC('Restore FileListOnly From Disk = ''' + @fullPath + '''');

 SET @RestoredDBName = REPLACE(@backupFile, '.bak', '')

 -- insert into the log the Database we're restoring
 INSERT INTO dbo.[DatabaseRestoreLog] (DatabaseId, RestoredName, RestoredDate, RestoreStartTime)
 VALUES (@database_id, @RestoredDBName, GETDATE(), GETDATE());

 SET @cmd = 'Restore Database [' + @RestoredDBName + '] ' + char(13) + ' From Disk = ''' + @fullPath + ''' With File = 1, ' + char(13);
 SELECT @cmd = @cmd + ' Move N''' + LogicalName + ''' To N''' + @data_dir + LogicalName + '_' + @RestoredDBName + '.' +
 REVERSE(SUBSTRING(REVERSE(PhysicalName), 0, CHARINDEX('.', REVERSE(PhysicalName),0))) + ''', ' + char(13)
 FROM @FileListTable

 SET @cmd = @cmd + ' NoUnload, Stats = 5;' + char(13);
 SET @cmd = @cmd + 'Alter Database [' + @RestoredDBName + '] Set Recovery Simple With No_Wait;'

 -- if for some reason the database with the restored name exists, drop it first
 IF EXISTS (SELECT 1
 FROM sys.databases
 WHERE Name = @RestoredDBName)
 BEGIN
 PRINT 'Dropping existing [' + @RestoredDBName +']';
 SET @dropCmd = 'Alter Database [' + @RestoredDBName + '] Set Single_User With Rollback Immediate; Drop Database [' + @RestoredDBName + '];';
 PRINT(@dropCmd);

 IF @exec = 1
 EXEC(@dropCmd);
 END

 -- if it doesn't exist, then restore the database
 PRINT 'Restoring [' + @database_name + '] from ' + @backupFile;
 PRINT(@cmd);
 IF @exec = 1
 EXEC(@cmd);

 -- make sure the DB exists before trying to run DBCC CHECKDB on it
 IF EXISTS (SELECT 1 FROM sys.databases WHERE Name = @RestoredDBName)
 BEGIN
 -- When the restore is complete, update the Log to reflect the end time
 UPDATE dbo.DatabaseRestoreLog
 SET RestoreEndTime = GETDATE(),
 DBCCStartTime = GETDATE()
 WHERE DatabaseId = @database_id
 AND RestoredName = @RestoredDBName

 EXEC dbo.spRunDBCCChecks @dbName = @RestoredDBName, @exec = @exec;

 -- When the DBCC Check is complete, update the Log to reflect the end time
 UPDATE dbo.DatabaseRestoreLog
 SET DBCCEndTime = GETDATE()
 WHERE DatabaseId = @database_id
 AND RestoredName = @RestoredDBName
 END
 ELSE
 -- if the restore failed, then raise an error that it failed and send an emails
 BEGIN
 SET @error = 'Restore of database ''' + @RestoredDBName + ''' failed. Check the log on the server.' ;
 -- send an email alert to that the DBCC CHECKDB failed for the restored item
 EXEC msdb.dbo.sp_send_dbmail @profile_name = 'Mail', @recipients = 'useyourown@email.com', @subject = @error, @body = @error;
 RAISERROR(@error, 18, 1);
 END

 END
 ELSE
 BEGIN
 SET @error = 'No Recent backup was found in ''' + @backupPath + '''';
 RAISERROR(@error, 18, 1);
 END

 -- delete the info on the one we just restored
 DELETE FROM @fileList
 DELETE FROM @FileListTable
 END

 -- once we're done, let's clear out the DBCC_History_Log table a bit, cause it grows fast
 DELETE
 FROM dbo.DBCC_History_Log
 WHERE DBCCCheckDate < GETDATE() - 30;

 -- send an email to sql-alerts with a count of what was done and what passed
 DECLARE @TotalDatabasesRestored int
 DECLARE @TotalPassedCheck int

 SELECT
 @TotalDatabasesRestored = count(*),
 @TotalPassedCheck = count(case when PassedDBCCChecks = 1 then DatabaseId end)
 FROM dbo.DatabaseRestoreLog
 WHERE RestoredDate = CAST(GETDATE() as date)

 DECLARE @subject nvarchar(100) = CONCAT('Database restore results for ', cast(getdate() as date))
 DECLARE @body nvarchar(500) = CONCAT('Total Databases Restored: ', @TotalDatabasesRestored, 'Total Databases Passing DBCC Check:', @TotalPassedCheck)

 EXEC msdb.dbo.sp_send_dbmail @profile_name = 'Mail',
 @recipients = 'useyourown@email.com',
 @subject = @subject,
 @body = @body;


END

Once the database is restored we perform a DBCC CHECKDB on it to see if there are any issues.

Since we restore hundreds of databases daily, I set a limit on the amount of time we perform restore testing each day. We currently limit testing to 12 hours a day, but if need be, we could easily adjust this to 18 hours or some other value. I attempted to make this process flexible to account for adding or removing databases.

Running DBCC CheckDB

Outside of restoring our backups, the last piece is the most important…running a CHECKDB against the database since we do not do so in production.

If you look closely at the previous stored procedure, you won’t see a direct call to DBCC CHECKDB. That’s because I created a separate stored procedure to make that call. But before we get to that stored procedure, there is one more table that I created for this process. Unless you are actively watching the CHECKDB run, you’re going to want to capture the results so you can see any issues that were found. In order to do this, I created one more table to capture the lengthy results:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


CREATE TABLE dbo.DBCC_History_Log
(
 [ID] [int] IDENTITY(1,1) NOT NULL CONSTRAINT PK_DBCC_History_Log PRIMARY KEY,
 [DatabaseName] [sysname] NULL,
 [DBCCCheckDate] [date] NOT NULL,
 [Error] [int] NULL,
 [Level] [int] NULL,
 [State] [int] NULL,
 [MessageText] [varchar](7000) NULL,
 [RepairLevel] nvarchar(500) NULL,
 [Status] [int] NULL,
 [DbId] [int] NULL,
 [DbFragId] [int] NULL,
 [ObjectId] [int] NULL,
 [IndexId] [int] NULL,
 [PartitionID] [bigint] NULL,
 [AllocUnitID] [bigint] NULL,
 [RidDbId] [int] NULL,
 [RidPruId] [int] NULL,
 [File] [int] NULL,
 [Page] [int] NULL,
 [Slot] [int] NULL,
 [RefDbId] [int] NULL,
 [RefPruId] [int] NULL,
 [RefFile] [int] NULL,
 [RefPage] [int] NULL,
 [RefSlot] [int] NULL,
 [Allocation] [int] NULL,
 [TimeStamp] [datetime] NULL CONSTRAINT [DF_dbcc_history_TimeStamp] DEFAULT (GETDATE())
);

You might be wondering why I’d want to save this info? Since this is a nightly process, I just want it to run in the background and only notify me if something is wrong in the CHECKDB. I also don’t want to have to rerun CHECKDB if there’s an issue. Essentially, I only need to review the data in the table to see the error, and investigate from there.

This table contains the standard output from a CHECKDB, but also includes a few additional columns including the name of the database being tested and the date that the check was performed. The table is populated via the final stored procedure in this process:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81


CREATE OR ALTER Procedure dbo.spRunDBCCChecks
 @dbName sysname,
 @exec BIT = 0
AS
BEGIN
 DECLARE @dropCmd nvarchar(500),
 @error nvarchar(600),
 @MailMsg nvarchar(500);

 -- DBCC CheckDB will output results that we want to capture to verify if there are errors
 IF OBJECT_ID('tempdb..#tempdbcccheck') IS NOT NULL
 DROP TABLE #tempdbcccheck

 CREATE TABLE #tempdbcccheck
 (
 [Error] [int] NULL,
 [Level] [int] NULL,
 [State] [int] NULL,
 [MessageText] [varchar](7000) NULL,
 [RepairLevel] nvarchar(500) NULL,
 [Status] [int] NULL,
 [DbId] [int] NULL,
 [DbFragId] [int] NULL,
 [ObjectId] [int] NULL,
 [IndexId] [int] NULL,
 [PartitionID] [bigint] NULL,
 [AllocUnitID] [bigint] NULL,
 [RidDbId] [int] NULL,
 [RidPruId] [int] NULL,
 [File] [int] NULL,
 [Page] [int] NULL,
 [Slot] [int] NULL,
 [RefDbId] [int] NULL,
 [RefPruId] [int] NULL,
 [RefFile] [int] NULL,
 [RefPage] [int] NULL,
 [RefSlot] [int] NULL,
 [Allocation] [int] NULL
 );

 PRINT 'Starting DBCC CHECKDB on ' + @dbName

 INSERT INTO #tempdbcccheck
 EXEC ('DBCC CHECKDB(''' + @dbName + ''') with tableresults');

 -- insert the results in the history log table
 INSERT INTO [dbo].[DBCC_History_Log] (DatabaseName, DBCCCheckDate, Error, [Level], [State], MessageText, RepairLevel, [Status], DbId, DbFragId,
 ObjectId, IndexId, PartitionID, AllocUnitID, RidDbId, RidPruId, [File], Page, Slot, RefDbId, RefPruId, RefFile, RefPage, RefSlot, Allocation)
 SELECT @dbName, GETDATE(), t.*
 FROM #tempdbcccheck t

 -- if the there were no errors in the database check, then we can safely drop the DB
 IF NOT EXISTS (SELECT 1
 FROM dbo.DBCC_History_Log
 WHERE DatabaseName = @dbName
 AND DBCCCheckDate = cast(getdate() as date)
 AND [Level] > 10 )
 BEGIN
 UPDATE [dbo].[DatabaseRestoreLog]
 SET [PassedDBCCChecks] = 1
 WHERE [RestoredName] = @dbName
 AND [RestoredDate] = cast(getdate() as date)

 PRINT 'Clean DBCCCheck - dropping database [' + @dbName +']';
 SET @dropCmd = 'Alter Database [' + @dbName + '] Set Single_User With Rollback Immediate; Drop Database [' + @dbName + '];';
 PRINT(@dropCmd);

 IF @exec = 1
 BEGIN
 EXEC(@dropCmd);
 END
 END
 ELSE
 BEGIN
 SET @error = 'DBCC CHECKDB generated errors for database ''' + @dbName + ''' not dropping for further review.' ;
 -- send an email alert to sql-alerts that the DBCC CHECKDB failed for the restored item
 EXEC msdb.dbo.sp_send_dbmail @profile_name = 'Mail', @recipients = 'useyourown@email.com', @subject = @error, @body = @error;
 RAISERROR(@error, 18, 1);
 END

END

In this stored procedure I do a couple of things, run CHECKDB and then validate that the CHECKDB was successful. If it wasn’t successful, then we don’t drop the database, and we send an email alert stating that something needs to be investigated. If everything is ok with the CHECKDB, then we drop the database and it will either move to the next database to restore, or it will exit the job if we have hit the 12 hour time limit.

To get this entire process done, I have two SQL Agent jobs running - one to update the list of the DatabasesToRestore via spUpdateDatabasesToRestore and one that executes the Backup and Restore Testing procedure (spBackupRestoreTesting).

I’ve added all these scripts to GitHub, feel free to peruse them that way or tell me how terrible they are.

Final Thoughts

Currently, we’re restoring 398 databases daily and running the DBCC CHECKDB on each one of them. Does that seem like a lot? Probably, but this process has, in fact, caught issues with some of our databases.

If the CHECKDB finds an issue, it doesn’t drop the database (leaving it for investigation), it sends an email alert. This gives me ample time to figure out what has gone wrong and fix the issue on the production database.

Will this capture issues on the production server that don’t exist in the backup? No, but it does give us confidence in our backups, and has found issues with databases that needed to be fixed - like data corruption. Ideally, you’d be able to run CHECKDB in production, but when you can’t, you need to get creative. This process does two things - tests our backups daily to ensure they’re working, and runs the CHECKDB to make sure there is no corruption in the backup (which would bite us if we ever needed a backup).

How do you run CHECKDB? Do you run it on your production servers or do you run it on a separate server?

Fighting with Deadlocks

Fri, 09 Apr 2021 04:00:00 +0000

If you follow me on Twitter, then you might have noticed that I’ve been fighting a lot of fires lately. Between high CPU (several times 1, 2), blocking queries, a slow failover, and deadlocks there have been a ton of things that needed attention. Not all of these issues are interesting to people, but one of them might be…the deadlocks. I’m going to go into details about what deadlocks we hit, why we hit them, and how we resolved them.

Background

A few years ago, we launched Stack Overflow for Teams, which allows you to have a private version of Stack Overflow to share knowledge. I haven’t written about the infrastructure of Teams, but my colleague, Dean Ward, gave a presentation about how Stack Overflow for Teams was built - it’s good, go watch it.

Here’s a brief overview of what we have on the SQL Server side - we have a 3 server cluster for Teams (2 in NY, 1 in CO) that are using Always On Availability Groups. We use schemas to keep teams separated from each other, and we put 10,000 schemas in each database, so we have multiple databases with 10,000 schemas in each one.

Managing Schemas

When we started Stack Overflow for Teams, we manually provisioned each new team (yeah, I know) - this meant someone ran a script to create the login, user, schema, tables, and then granted the necessary permissions for the schema user. Even though we joked about it, manually provisioning new teams wasn’t scalable and didn’t last long, so we came up with a process to pre-provision everything needed for a new team to start.

We never wanted to run into a race condition or incur the performance overhead with setting up a new team, so we pre-provisioned a set number of schemas that would be ready when someone signed up. The pre-provisioned teams would be already set up with

Login
User
Schema
Permissions
Tables, Views, and any other database objects

Initially, we had about 100 schemas ready to go, and then had a scheduled job that would check periodically to refresh our provisioned bucket and replenish it back to full. We realized very quickly that 100 schemas wasn’t enough because we constantly hit issues with the logins not being available on the secondary replicas, so we decided to increase the provisioned pool to 500. For the last several years, a pool of 500 schemas worked great, until it didn’t.

The Launch of Free Teams

As expected, with the announcement that Stack Overflow for Teams is free for up to 50 users, we saw an incredible spike in sign-ups. Along with the spike in sign-ups, I started to receive a huge increase in alerts about deadlocks on the primary SQL Server for Teams. In the two weeks it took us to resolve the deadlocks, we hit at least 200 deadlocks.

What Is A Deadlock?

A deadlock occurs when two processes want access to the same resource. The first one has a lock on a resource, and the second process wants the same access, or vice versa. Since the processes are competing for the same resource, the result is a situation where one process needs to terminate so the other can continue.

Time to Investigate

Most people who manage a highly transactional database are probably familiar with deadlocks, or have seen them pop up every now and then. We had them in the past when provisioning schemas, but it was long ago and we thought the issue was over.

But, after the launch of free teams, I was receiving a deadlock email alert every few hours from SQL Sentry. While these email alerts provided some details into the issue, I needed to dig in further to figure out what exactly was happening, and try to resolve it. Thankfully, I have a handful of tools at my disposal to help investigate what we were seeing.

Between SQL Sentry and Brent Ozar’s sp_BlitzLock, I was able to get more details on each deadlock, including what the victim was, what type of locks we were hitting, and the processes involved in each deadlock.

The query text of the victim was pretty much same each time. As you can see, it wasn’t very helpful, as the victim was always when another process was just trying to set the transaction level (or at least that’s all the text we ever captured):

The other query involved in the deadlock was also basically same each time:

1

GRANT SELECT, INSERT, UPDATE, DELETE, ALTER ON SCHEMA::[channel29630] TO [channel29630]

And the deadlock graph for each one was always the same, but it provided a critical piece of information that we needed - the lock type:

The dreaded Sch-M lock. I knew about the evils of this lock from Michael J Swart, but we had never faced it to this level before.

Every deadlock we hit was due to the permission GRANT during provisioning. Even if the victim query was on a completely different schema, the Sch-M lock taken by the permission grant always won.

Resolving the Deadlocks

Now that we identified the cause of the deadlock, it was time to figure out how to stop them. We brainstormed several ideas on how to fix it.

The scheduled job to replenish the pool of pre-provisioned teams runs every 5 minutes, so first we thought what if we spaced out that process? That wouldn’t work because all it would do is punt the problem to the new timeframe, which might potentially make it worse since we’d need to provision more schemas at a time. So, instead of the 5 - 15 we were provisioning every 5 minutes, we might need to create 50 or more if we waited an hour.

We also thought about provisioning a larger number of schemas, and then replenishing the pool nightly when we have less traffic, but again realized that it was just punting the problem - not actually fixing it.

Finally, we discussed pre-provisioning an entire database. That might work.

As I mentioned earlier, we put 10,000 schemas in a database, so the thinking was, if we set up a new database and provisioned the whole database with what we needed to start a team, then we no longer would hit the deadlocks by granting the permissions every 5 minutes. By setting up all the permissions ahead of time, the scheduled job would still need the Sch-M lock, but it would be taking it on a database not yet in use, hopefully removing its impact and the deadlocks.

The plan was to change the order in which we provisioned the new teams. We would pre-provision a new database with 10,000 logins, users, schemas, and their permissions, and would still keep our current bucket of 500 provisioned schemas that have the tables, views, and other database objects (the ones ready to go when a new team is created). Once we got close to using all of the schemas in a database, the pre-provision process would kick off on the next database and set up the next 10,000 schemas, removing the chances of getting deadlocks.

Dean Ward spent several days refactoring the code for the provisioning and when we pushed it out, it was time to create all the schemas and see if the deadlocks went away.

It worked.

We resolved the deadlock by refactoring the process so we weren’t taking the Sch-M lock for the permission grant every 5 minutes. By granting all the permissions ahead of time, we no longer have to fight with other processes and we stopped getting deadlocks.

Final Thoughts

Deadlocks are “fun” and can be complicated. This one was interesting because I had never seen a deadlock victim on setting the transaction isolation level, but once I saw the Sch-M lock it was clear we needed to do some refactoring of our process. Sch-M locks are definitely not fun, and if you need to take that lock on a heavily used database you might need to look at refactoring or even performing the work off hours.

Side Note

Oh, and while we did resolve the deadlocks, we also introduced an issue that none of us thought about until we started getting exceptions. The new refactored pre-provisioning process creates the login, user and schema, but when we move that schema to a provisioned state and we create the tables and database objects in the schema, we also change the password for the login. Our process to sync logins across our replicas didn’t account for those password changes which resulted in login failures, so we had to quickly resolve that issue after the change.

And by the way, if you’re interested, go sign up for a team, it’s free and it won’t generate a deadlock.

Syncing Logins Between Availability Group Replicas

Fri, 18 Dec 2020 04:00:00 +0000

I mentioned this in a few previous posts, but for for those who may have missed it or forgotten, here’s a quick refresher - we use Always On Availability Groups at Stack Overflow on all of our main production servers running the network of public Q&A sites, Jobs, and Stack Overflow for Teams. It’s a great way to implement disaster recovery for a SQL Server environment.

Always On Availability Groups can support up to nine availability replicas, and while we don’t use anywhere near that many replicas in each of our clusters, we do have 2 replicas per cluster (3 servers total), with the replicas being used as a readable secondary.

Since we use readable secondaries in our environments, the application needs to connect to both the primary and the secondary servers with the same login. The catch is, logins don’t automatically sync across replicas. If the logins don’t sync, the application won’t connect to a secondary, which results in login failures.

Copying Logins with a T-SQL Script

Before you jump the gun and start criticizing my method, know that I’m aware there are other ways to do this, including using dbatools, but this method has been around since the early days of Stack Overflow and the Stack Exchange network (with some modifications) and it works for our needs for now.

We don’t create new databases very often, but on Stack Overflow for Teams we use schemas to keep customers separated from each other - i.e. every team has a login and user to allow it to query their data. While we pre-provision a set number of schemas, tables, and other database objects for teams, we want the logins to sync on a frequent basis to avoid login failures on the secondaries. We have a job in place which syncs the logins via a SQL Server Agent job every 10 minutes for Stack Overflow for Teams. (We have a similar job for the public Q&A sites that runs nightly.)

The job needs to do several things:

Get a list of the existing logins on the primary
Create the logins on replica
Drop any logins no longer being used i.e. teams that are deleted

Gather Existing Logins

The first thing the job does is, create a temp table called @logins to store all the details needed from the primary. The table is pretty basic:

1
2
3
4
5
6
7


DECLARE @logins TABLE
(
 SID varbinary(256),
 Login sysname,
 SQL varchar (1024),
 DefaultDB sysname
);

The table is populated by connecting to the primary server using OPENQUERY (Eeek a linked server)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


SELECT *
FROM OPENQUERY([SQL-AG], '
 SELECT p.sid sid_varbinary, p.name, p.default_database_name,
 sl.password_hash pwd_varbinary,
 CASE sl.is_policy_checked WHEN 1 THEN ''ON'' WHEN 0 THEN ''OFF'' ELSE NULL END is_policy_checked,
 CASE sl.is_expiration_checked WHEN 1 THEN ''ON'' WHEN 0 THEN ''OFF'' ELSE NULL END is_expiration_checked
 FROM sys.server_principals p
 JOIN sys.syslogins l ON l.name = p.name
 JOIN sys.sql_logins sl ON l.name = sl.name
 WHERE p.type = ''S''
 AND p.name <> ''sa''
 AND l.denylogin = 0
 AND l.hasaccess = 1
 AND p.is_disabled = 0
 ORDER BY p.name');

and executing sp_hexadecimal to get the SID and password which are then used to create a dynamic SQL string which is stored in the temp table.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


EXEC sp_hexadecimal @PWD_varbinary, @PWD_string OUT
EXEC sp_hexadecimal @SID_varbinary, @SID_string OUT

SET @sqlString = 'CREATE LOGIN ' + QUOTENAME(@name) + ' WITH PASSWORD = ' + @PWD_string
 + ' HASHED, SID = ' + @SID_string + ', DEFAULT_DATABASE = [' + @defaultdb + ']'
 + ', CHECK_POLICY = ' + @is_policy_checked
 + ', CHECK_EXPIRATION = ' + @is_expiration_checked

INSERT INTO @logins (SID, Login, SQL, DefaultDB)
VALUES (@SID_varbinary, @name, @sqlString, @defaultdb);

Create Logins on Replicas

Now that the table has all the logins from the primary, it’s time to create them locally. While it’s possible that we have only one login to sync, it’s more likely we have multiple. Keeping this in mind, the script loops through the temp table and executes the dynamic SQL for each login that doesn’t already exist on the replica.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


DECLARE @sql varchar (1024), @db sysname, @loginname sysname;
DECLARE login_cursor CURSOR FOR
 SELECT SQL, DefaultDB, Login
 FROM @logins
 WHERE SID NOT IN (SELECT sid
 FROM sys.server_principals)
 AND EXISTS (SELECT 1
 FROM sys.databases
 WHERE name = DefaultDB)
 AND Login NOT IN (SELECT name
 FROM sys.server_principals
 WHERE name = Login
 And type = 'S')
OPEN login_cursor

-- add new logins
FETCH NEXT FROM login_cursor INTO @sql, @db, @loginname
WHILE @@fetch_status = 0
BEGIN
 EXEC(@sql);
 FETCH NEXT FROM login_cursor
 INTO @sql, @db, @loginname
END
CLOSE login_cursor
DEALLOCATE login_cursor

Drop Unused Logins on Replicas

On the public Q&A sites, like Stack Overflow, we don’t need to drop logins, but after running Stack Overflow for Teams for several months, we realized very quickly that we need to purge logins no longer being used. The last step of the job goes through the existing logins on the secondaries and deletes any that don’t exist on the primary. Using the original @logins temp table I can get a list of the logins to drop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


SELECT p.sid, p.name, p.default_database_name
FROM sys.server_principals p
JOIN sys.syslogins l ON l.name = p.name
JOIN sys.sql_logins sl ON l.name = sl.name
WHERE p.type = 'S'
 AND p.name <> 'sa'
 AND l.denylogin = 0
 AND l.hasaccess = 1
 AND p.is_disabled = 0
 AND NOT EXISTS (SELECT 1
 FROM @logins cl
 WHERE p.sid = cl.sid
 AND p.name = cl.Login
 AND p.default_database_name = cl.DefaultDB)
ORDER BY p.name

then it creates a dynamic SQL string which gets executed cleaning up the unused logins.

1
2
3


SET @drop_sqlString = 'DROP LOGIN ' + QUOTENAME(@name) + '; '
--print @drop_sqlString
EXEC(@drop_sqlString);

Putting the Pieces Together

Both the create and delete of the logins are done in various loops, in other words cursors. The full script is below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116


DECLARE @name sysname,
 @PWD_varbinary varbinary (256),
 @PWD_string varchar (514),
 @SID_varbinary varbinary (85),
 @SID_string varchar (514),
 @sqlString varchar (1024),
 @is_policy_checked varchar (3),
 @is_expiration_checked varchar (3),
 @defaultdb sysname;

DECLARE @logins TABLE
(
 SID varbinary(256),
 Login sysname,
 SQL varchar (1024),
 DefaultDB sysname
);

DECLARE login_cursor CURSOR FOR
 SELECT *
 FROM OPENQUERY([SQL-AG], '
 SELECT p.sid, p.name, p.default_database_name,
 sl.password_hash pwd_varbinary,
 CASE sl.is_policy_checked WHEN 1 THEN ''ON'' WHEN 0 THEN ''OFF'' ELSE NULL END is_policy_checked,
 CASE sl.is_expiration_checked WHEN 1 THEN ''ON'' WHEN 0 THEN ''OFF'' ELSE NULL END is_expiration_checked
 FROM sys.server_principals p
 JOIN sys.syslogins l ON l.name = p.name
 JOIN sys.sql_logins sl ON l.name = sl.name
 WHERE p.type = ''S''
 AND p.name <> ''sa''
 AND l.denylogin = 0
 AND l.hasaccess = 1
 AND p.is_disabled = 0
 ORDER BY p.name')
OPEN login_cursor

FETCH NEXT FROM login_cursor
 INTO @SID_varbinary, @name, @defaultdb, @PWD_varbinary, @is_policy_checked, @is_expiration_checked
WHILE @@fetch_status = 0
BEGIN
 EXEC sp_hexadecimal @PWD_varbinary, @PWD_string OUT
 EXEC sp_hexadecimal @SID_varbinary, @SID_string OUT

 SET @sqlString = 'CREATE LOGIN ' + QUOTENAME(@name) + ' WITH PASSWORD = ' + @PWD_string
 + ' HASHED, SID = ' + @SID_string + ', DEFAULT_DATABASE = [' + @defaultdb + ']'
 + ', CHECK_POLICY = ' + @is_policy_checked
 + ', CHECK_EXPIRATION = ' + @is_expiration_checked

 INSERT INTO @logins (SID, Login, SQL, DefaultDB)
 VALUES (@SID_varbinary, @name, @sqlString, @defaultdb);

 FETCH NEXT FROM login_cursor
 INTO @SID_varbinary, @name, @defaultdb, @PWD_varbinary, @is_policy_checked, @is_expiration_checked
END
CLOSE login_cursor
DEALLOCATE login_cursor

DECLARE @sql varchar (1024), @db sysname, @loginname sysname;
DECLARE login_cursor CURSOR FOR
 SELECT SQL, DefaultDB, Login
 FROM @logins
 WHERE SID NOT IN (SELECT sid
 FROM sys.server_principals)
 AND EXISTS (SELECT 1
 FROM sys.databases
 WHERE name = DefaultDB)
 AND Login NOT IN (SELECT name
 FROM sys.server_principals
 WHERE name = Login
 And type = 'S')
OPEN login_cursor

-- add new logins
FETCH NEXT FROM login_cursor INTO @sql, @db, @loginname
WHILE @@fetch_status = 0
BEGIN
 EXEC(@sql);
 --print @sql
 FETCH NEXT FROM login_cursor
 INTO @sql, @db, @loginname
END
CLOSE login_cursor
DEALLOCATE login_cursor

-- drop logins that are no longer being used
DECLARE @drop_sqlString nvarchar(max) = '';
DECLARE drop_login_cursor CURSOR FOR
 -- logins on current server that don't exist on the primary
 SELECT p.sid, p.name, p.default_database_name
 FROM sys.server_principals p
 JOIN sys.syslogins l ON l.name = p.name
 JOIN sys.sql_logins sl ON l.name = sl.name
 WHERE p.type = 'S'
 AND p.name <> 'sa'
 AND l.denylogin = 0
 AND l.hasaccess = 1
 AND p.is_disabled = 0
 AND NOT EXISTS (SELECT 1
 FROM @logins cl
 WHERE p.sid = cl.sid
 AND p.name = cl.Login
 AND p.default_database_name = cl.DefaultDB)
 ORDER BY p.name
OPEN drop_login_cursor
FETCH NEXT FROM drop_login_cursor INTO @SID_varbinary, @name, @defaultdb
WHILE @@fetch_status = 0
BEGIN

 SET @drop_sqlString = 'DROP LOGIN ' + QUOTENAME(@name) + '; '
 --print @drop_sqlString
 EXEC(@drop_sqlString);

 FETCH NEXT FROM drop_login_cursor INTO @SID_varbinary, @name, @defaultdb
END
CLOSE drop_login_cursor
DEALLOCATE drop_login_cursor

As I said, this might not be the way you would do this, but this works for us and keeps the logins synced across all of our replicas which minimizes login failures. I’ve uploaded the full script to GitHub, if you’re interested in it.

Deploying My SQL Server Maintenance Scripts

Fri, 16 Oct 2020 04:00:00 +0000

In my previous post, I listed out the tools I use with SQL Server. Some of the tools are SQL scripts that need to be deployed to each server. If you have 1-2 SQL Servers, manually deploying scripts might not be bad, but ideally you don’t want to manually deploy anything, so I wrote a little script that allow me to install the SQL scripts in my toolbox to any environment.

Background

When I started as the DBA at Stack Overflow, our SQL Servers were kind of a mess (sorry to those who maintained them before me, but they were). Many of the servers hadn’t been patched in a long time, and we had different versions of various SQL scripts like, sp_whoisactive and Brent Ozar’s First Reponder Kit, everywhere. Since the SQL Servers were now my responsibility I wanted everything in order. I decided that one of my first tasks would be to create a simple process to deploy the same scripts everwhere.

My Little Deploying Scripts Project

When I started the project, I had already decided what maintenance tools I wanted to install across our environments - these are most of the SQL scripts in my toolbox - I just needed to figure out how I was going to get them on our 30 servers at the same time.

After several years away from SQL Server and PowerShell, I needed to get my feet wet again, so I decided to write a very basic process that would do the following:

Put the .sql files somewhere accessible
Get the list of servers to install on
Loop through each server and deploy all the .sql files

Storing the SQL Scripts

I decided to put the .sql files in a centralized location that all of the servers would be able to access. This was easy enough because we already had infrastructure in place that could be used for this purpose. All I needed to do was pick a spot and throw all the files in a directory.

Once the files were in the location, I would just loop through them and execute each one. Step 1 done.

Gathering the List of SQL Servers

Step 2 in the project was to get a list of all the SQL Servers I wanted the tools on. Easy enough, right? But then I needed a way to parse that list in PowerShell. I wasn’t exact sure how I wanted to do this.

I looked at some of the other tools being used at Stack Overflow for ideas. The SRE team uses puppet to deploy to our infrastructure, and puppet uses JSON files to store key-value pairs. I wondered if I could do the same for this tiny little project.

After several failed attempts at getting the JSON right, I was able to use the following format to store the info about the SQL Servers I wanted to deploy to:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


{
"ScriptDirectory":"\\\\put\\your\\directory\\name\\here\\",
"servers":[
{
"name":"sqlserver01"
,"type":"zone1_servers"
},
{
"name":"sqlserver02"
,"type":"zone3_servers"
},
{
"name":"sqlserver03"
,"type":"zone4_servers"
},
{
"name":"protectedsqlserver01"
,"type":"zone2_servers"
}
]
}

In the config.json file, I’m storing the ScriptDirectory which is the path I put all of the .sql files to install. Next, it contains an array of the list of servers. Within the array, I’m storing the name of the server and then the type of the server. The name is pretty self-explanatory, just include each server name and you’re done.

The type might seem slightly odd since these are all SQL Servers, but there was a reason I needed this. We have some servers protected by different firewalls, for example the servers that hold the Stack Overflow and the Stack Exchange network of databases are in a totally different zone, than the servers for Stack Overflow for Teams. My PowerShell script would need to be able to distinguish between the multiple firewalls, so I used this as the identifier.

Now that I had a way to get all the servers to push to Step 2 was done.

My Little PowerShell Script

The last part of the project was to put all the pieces together and write the PowerShell script to deploy everything automatically. I wrote the script to accept a couple of parameters:

$DeploySecure - this parameter is required and is used to direct the deployment to the specific firewalled areas I mentioned above
$ServerGroup - this is an optional parameter and will let you deploy to specific sets of servers based on their type in the config

Keep in mind this script is about 3 years old, and I know it could be written much better than it is.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57


<#
.SYNOPSIS
 Script for deploying SQL Server maintenance scripts across environments

.EXAMPLE
 # for zone 1 environments
 .\Deploy-MaintenanceScripts.ps1 -DeploySecure $false

 # for zone 2 environments
 .\Deploy-MaintenanceScripts.ps1 -DeploySecure $true
#>
[CmdletBinding()] #See http://technet.microsoft.com/en-us/library/hh847884(v=wps.620).aspx for CmdletBinding common parameters
param(
[parameter(Mandatory = $true)]
[bool]$DeploySecure = $false,
[parameter(Mandatory = $false)]
[string]$ServerGroup
)
$Config = Get-Content .\Config.json | ConvertFrom-Json
if ($DeploySecure -eq $false) {
$Servers = $Config.servers | Where-Object {$_.type -ne "zone2_servers"}
Write-Host "Deploying to non-zone 2 environments."
}
else {
$Servers = $Config.servers | Where-Object {$_.type -eq "zone2_servers"}
Write-Host "Deploying to zone 2 environments."
}
if ($ServerGroup){
$Servers = $Config.servers | Where-Object {$_.type -eq $ServerGroup}
Write-Host "Deploying servers to $ServerGroup"
}
$ScriptDirectory = $Config.ScriptDirectory
If(-not (Test-Path $ScriptDirectory)){
Write-Error "$ScriptDirectory cannot be accessed!" -ErrorAction Stop
}
$scripts = Get-ChildItem $ScriptDirectory | Where-Object {$_.Extension -eq ".sql"}
$script_count = (Get-ChildItem $ScriptDirectory | Measure-Object).Count
$str_script = if($script_count -eq 1) {'script'} else {'scripts'}
Write-Host "Attempting to deploy $script_count $str_script to each server."
foreach ($server in $Servers) {
Write-Host "Deploying to" $server.name
foreach ($script in $scripts) {
Write-Host "Installing" $script
Invoke-Sqlcmd -ServerInstance $server.Name -InputFile $script.FullName -DisableVariables
}
}

Step 3 done.

As you can see, it’s a pretty basic script. It grabs the list of servers from the config.json file, goes to the ScriptDirectory to loop through each file, and executes it on each SQL Server. I’m sure there are plenty of other ways to do this, but it works and it only needs to be run when there are changes to the maintenance scripts I want to deploy which isn’t often.

The script is flexible for my needs. If I want to execute it for my restricted firewalled area, I just need to run:

1

.\Deploy-MaintenanceScripts.ps1 -DeploySecure $true

And it will install the scripts on every server in zone2_servers.

If I want to deploy to a specific set of servers, I can run:

1

.\Deploy-MaintenanceScripts.ps1 -DeploySecure $false -$ServerGroup zone1_servers

And it will deploy to all servers with zone1_servers as their type.

I’ve uploaded the code to a repo in GitHub if anyone would find it useful.

Next Steps

Since I rarely need to make changes to the scripts, I don’t have this currently setup in our CI/CD to deploy automatically when I push changes to GitHub. I could, and might at some point, then I wouldn’t have to manually deploy via an automated script.

While this works for me, what do you use to deploy maintenance scripts to all your SQL Servers? Do you have a homegrown process or a 3rd party tool?

My SQL Server Toolbox

Thu, 15 Oct 2020 04:00:00 +0000

I get asked a lot about the tools I use at Stack Overflow to monitor and work with our SQL Servers. I figured it might be helpful to others to make the list public. I’ll also do my best to keep it updated as things change.

The current list of both free and 3rd party paid tools is below:

Free Tools

Opserver - Stack Exchange’s Monitoring System - I pretty much live in our instances of Opserver because it gives me a one-stop shop to see the health of all of our SQL Servers
Brent Ozar’s First Reponder Kit. This is filled with lots of goodies to help you understand what’s going on with your server. Currently we install these scripts from the kit:
- sp_Blitz.sql
- sp_BlitzCache.sql
- sp_BlitzFirst.sql
- sp_BlitzIndex.sql
- sp_BlitzLock.sql
- sp_BlitzQueryStore.sql
Adam Machanic’s sp_whoisactive - this is similar to executing sp_who2 but provides a more detailed look at what is happening on your server
sp_hexadecimal and sp_help_revlogin to transfer logins between instances of SQL Server
Ola Hallengren’s SQL Server Maintenance Solution for Backups, Integrity Checks, and Index and Statistics Maintenance. We don’t use the entire package of scripts, but we currently use:
- DatabaseIntegrityCheck.sql
- IndexOptimize.sql
- CommandExecute.sql
- CommandLog.sql
- DatabaseBackup.sql - we use a modified version of this script everywhere
dbatools PowerShell modules
Erik Darling’s SQL Server Troubleshooting Scripts - additional tools to help you identify and troubleshoot problems with your SQL Servers
Columnstore Maintenance Scripts from a few places:
- Joe Obbish’s cci_maintenance
- Nike Neugebauer’s Columnstore Indexes Scripts Library

3rd Party Tools

SentryOne Plan Explorer - this is a great tool to review execution plans. I find it far easier to review execution plans in Plan Explorer than in SQL Server Management Studio
SentryOne SQL Sentry monitors all of our SQL Servers and provides excellent details on performance and insight into specific queries that might be running during various CPU spikes or other periods of concern
ApexSQL SQL script generation - there are times we need to rebase our SQL migrations and this tool is a great way to script out all objects and data

All the SQL scripts get deployed via process that I wrote. I’ll share how I do that in my next post.

Migrating a 40TB SQL Server Database

Tue, 28 Jul 2020 06:00:38 +0000

Initially, I wasn’t sure whether to write about this migration project, but when I randomly asked if people would be interested, the response was overwhelming. This was a long, kind of boring, very repetitive, and at times incredibly frustrating project, but I learned a lot, and maybe someone else will learn from this too. There may be far better ways to move this amount of data. In the path I went down, there was a huge amount of juggling that had to take place (I’ll explain that later). Thankfully, I’m a decent juggler.

While most people are probably interested in what we do with our primary SQL Servers (the servers that run Stack Overflow and the Stack Exchange network of sites), this post isn’t about those servers. This post is about what I called a miscellaneous server in a previous post - more specifically, it’s about the migration of our traffic log data.

Background

Our HAProxy1 logs aka traffic logs, are currently stored on two SQL Servers, one in New York, and one in Colorado. While we store a minimal summary of the traffic data, we have a lot of it. At the beginning of 2019, we had about 4.5 years of data, totaling about 38TB. The database was initially designed to have a single table for each day. This meant that at the start of 2019 we had approximately 1600 tables in a single database with multiple database files (due to the 16TB size limit of data files). Each table had a clustered columnstore index, which held anywhere from 100-400 million rows in it.

I had to move the data from their existing daily tables into a new structure — one table for each month. This needed to be completed on both the NY and CO servers, and the data was stored on spinny drives (Eeek). No matter what, it was going to be painful and slow.

Oh, and to make things even more complicated, this had to be done with minimal free disk space on the existing servers. The servers had a 44TB spinny drive partition, and we were using 36-38TB of it, depending on the server. This meant I’d be following these steps throughout the process:

migrate data to the new format
delete old data
shrink old database files…yeah, I know, but I had no choice
repeat over, and over, and over again

I fully expected the process to take months, and it did — after hitting delays in getting new servers the entire project took about 11 months.

1 Note: If you’re interested, my colleague, Nick Craver (b | t), wrote extensively about our monitoring on his blog and gave an overview of our HAProxy usage.

Things to Remember

Before I get started there are a few things I need to mention which impacted the project:

I work remote 100% of the time, so I would not be able to run any of the processes from my local computer. Everything needed to be executed on a machine that was constantly connected to the network, so there wouldn’t be any VPN failures.
I needed to run this on two separate machines. I was doing this on SQL servers in two different datacenters, I needed a machine in the same location, so I wouldn’t have to deal with slowness over the network.
We have jump boxes that we use for various things at Stack Overflow, and we have one in NY and in CO, which were perfect to run the migration
The old database was still being used in a live production environment, meaning that as I was moving data, we were constantly adding a new table daily. In other words, I had a moving target
The databases are in simple recovery so we don’t have to deal with transaction logs. Our backup is the original source log files and the pair of SQL Servers - NY and CO - are copies of each other.

The bullets above were pain points throughout the project. Being disconnected from the VPN meant I’d have to reconnect to monitor the migration progress. Getting booted from a jumpbox, meant that whatever process was running would need to be kicked off again, after some clean-up. Since we were still inserting new data I was chasing the target, the longer it took, the more data I’d be moving in the end.

Why, oh why would you do this?

I like to be miserable. Kidding.

There were lots of reasons this needed to be done, one being tech debt. We realized that the original daily table structure wasn’t ideal. If we needed to query something over several days or months, it was terrible — lots of UNION ALLs or loops to crawl through days or even months at a time was slow.

Why not delete or purge some of the data?

As I mentioned, we were dealing with very little free disk space on both servers. This data is primarily used by the developers for investigating issues and by the data team for analysis. We already had purged some data, but for the data team - more data is better.

Instead of purging, the plan was to eventually get new hardware to replace them, but we weren’t exactly sure when that would happen. The goal was to migrate the data to the new format, then when we got new servers it would be as simple as moving the drives to get the new database in place (or so we thought).

So It Begins

Each server had the following drives and approximate free space:

a 230GB C: drive for Windows - obviously we wouldn’t put data files here
a 3.64TB NVMe D: drive - containing tempdb, one data file, and the log file for the existing HAProxyLogs database - it was approximately 85% full
a 44TB E: drive filled with spinny disks - with the remaining 3 data files for the HAProxyLogs database - this was anywhere from 85-90% full

There was not a lot of room to move terabytes of data to a new database on the same server.

Since new servers were on down the road, I asked about the possibility of getting more disk space. Having more space would give me some breathing room to not be so crunched during the process. After some research, it was decided that we could fit a couple of additional NVMe SSDs into both servers. We only had PCIe slots available in the server, so we ended up using U.2 NVMe drives on PCIe adapters to get us some space. The SSDs gave us a brand spanking new F: drive that was 14TB of free space. While I couldn’t move everything to the new drive, it gave me plenty of space to work with, at least, for a little bit.

Now that we had free space to use, it was time to set up the new database and begin the migration. I wrote the script to create the new database:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


CREATE DATABASE [TrafficLogs] CONTAINMENT = NONE
ON PRIMARY
( NAME = N'TrafficLogs_Current',
 FILENAME = N'F:\Data\TrafficLogs_Current.mdf',
 SIZE = 102400000KB,
 FILEGROWTH = 5120000KB
),
FILEGROUP [TrafficLogs_Archive]
(
 NAME = N'TrafficLogs_Archive1',
 FILENAME = N'E:\Data\TrafficLogs_Archive1.ndf' ,
 SIZE = 102400000KB,
 FILEGROWTH = 5120000KB
),
(
 NAME = N'TrafficLogs_Archive2',
 FILENAME = N'E:\Data\TrafficLogs_Archive2.ndf',
 SIZE = 102400000KB , FILEGROWTH = 5120000KB
),
(
 NAME = N'TrafficLogs_Archive3',
 FILENAME = N'E:\Data\TrafficLogs_Archive3.ndf',
 SIZE = 102400000KB,
 FILEGROWTH = 5120000KB
)
LOG ON
(
 NAME = N'TrafficLogs_log',
 FILENAME = N'F:\Data\TrafficLogs_log.ldf',
 SIZE = 5120000KB,
 MAXSIZE = 2048GB,
 FILEGROWTH = 102400000KB
);

The database was going to have all of the old historical tables (i.e. the 40TBs) stored on the TrafficLogs_Archive filegroup, and we’d use the PRIMARY and TrafficLogs_Current for newer data being added.

You’ll notice that the TrafficLogs_Archive filegroup went on our already very full E: drive, instead of the new F: drive - more on that mistake later.

Moving All The Data

We had a database, so it was time to start the migration. To be clear, I was actually taking over a process that was started, stopped, and then punted from several years earlier. The project was on the backlog of things to be done long before I became the DBA at Stack Overflow. Back then, everyone realized what a time consuming project this would be, and since we didn’t have the resources to devote to it, the can continuously got kicked down the road. As a result, more and more tables were added making the overall task a lot larger.

Since this had been a previously abandoned project, there were already some scripts written, which meant I wasn’t totally starting from scratch. I was handed a couple of scripts to:

Create all of the new monthly tables

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69


-- written by Nick Craver
Declare @month datetime = '2015-08-01';
Declare @endmonth datetime = '2021-01-01'

WHILE @month < @endmonth
BEGIN

Set NoCount On;
Declare @prevMonth datetime = DateAdd(Month, -1, @month);
Declare @nextMonth datetime = DateAdd(Month, 1, @month);
Declare @monthTable sysname
 = 'Logs_' + Cast(DatePart(Year, @month) as varchar)
 + '_' + Right('0' + Cast(DatePart(Month, @month) as varchar), 2);

Begin Try
 If Object_Id(@monthTable, 'U') Is Not Null
 Begin
 Declare @error nvarchar(400)
 = 'Month ' + Convert(varchar(10), @month, 120)
 + ' has already been moved to ' + @monthTable + ', aborting.';
 Throw 501337, @error, 1;
 Return;
 End

 -- Table Creation
 Declare @tableTemplate nvarchar(4000) = '
 Create Table {Name} (
 [CreationDate] datetime Not Null,
 <insert all the columns>,
 Constraint CK_{Name}_Low Check (CreationDate >= ''{LowerDate}''),
 Constraint CK_{Name}_High Check (CreationDate < ''{UpperDate}'')
 ) On {Filegroup};

 Create Clustered Columnstore Index CCI_{Name}
 On {Name} With (Data_Compression = {Compression}) On {Filegroup};';

 -- Constraints exist for metadata swap
 Declare @table nvarchar(4000) = @tableTemplate;
 Set @table = Replace(@table, '{Name}', @monthTable);
 Set @table = Replace(@table, '{Filegroup}', 'Logs_Archive');
 Set @table = Replace(@table, '{LowerDate}', Convert(varchar(20), @month, 120));
 Set @table = Replace(@table, '{UpperDate}', Convert(varchar(20), @nextMonth, 120));
 Set @table = Replace(@table, '{Compression}', 'ColumnStore_Archive');
 Print @table;
 Exec sp_executesql @table;

 Declare @moveSql nvarchar(4000)
 = 'Create Clustered Columnstore Index CCI_{Name}
 On {Name} With (Drop_Existing = On, Data_Compression = Columnstore_Archive) On Logs_Archive;';
 Set @moveSql = Replace(@moveSql, '{Name}', @monthTable);
 Print @moveSql;
 Exec sp_executesql @moveSql;

End Try
Begin Catch
 Select Error_Number() ErrorNumber,
 Error_Severity() ErrorSeverity,
 Error_State() ErrorState,
 Error_Procedure() ErrorProcedure,
 Error_Line() ErrorLine,
 Error_Message() ErrorMessage;
 Throw;
End Catch

set @month = dateadd(month, 1, @month)

END
GO
 

A LINQPad script to loop through each day, starting with the earliest, and insert the data into the new table

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115


-- written by Nick Craver
<Query Kind="Program">
<NuGetReference>Dapper</NuGetReference>
<Namespace>Dapper</Namespace>
</Query>
void Main()
{
MoveDate(new DateTime(2015, 08, 1));
DateTime date = new DateTime(2015, 08, 1);
while (date < DateTime.UtcNow)
{
MoveDate(date);
date = date.AddDays(1);
}
}
static readonly List<string> cols = new List<string> { "<col list>" };
public void MoveDate(DateTime date)
{
var tableName = GetTableName(date);
var destTable = GetDestTableName(date);
$"Attempting to migrate {date:yyyy-MM-dd} f
 from {tableName} to {destTable}".Dump($"{date:yyyy-MM-dd}");
using (var conn = GetConn())
{
int rowCount;
try
{
rowCount = conn.QuerySingle<int>($"Select Count(*)
 From HAProxyLogs.dbo.{tableName};");
}
catch (SqlException e)
{
(" Error migrating: " + e.Message).Dump();
return;
}
$" Summary for {date:yyyy-MM-dd}".Dump();
$" {rowCount:n0} row(s) in {tableName}".Dump();
var pb = new Util.ProgressBar($"{tableName} (0/{rowCount})");
pb.Dump(tableName + " copy");
Func<int> GetDestRowCount = ()
=> conn.QuerySingle<int>($"Select Count(*)
 From {destTable}
 Where CreationDate >= @date
 And CreationDate < @date + 1;", new { date });
Action<long, int> UpdatePB = (copied, total) =>
{
pb.Fraction = (double)copied / total;
pb.Caption = $"{tableName} ({copied}/{total})";
};
var destRowCount = GetDestRowCount();
$" {destRowCount:n0} row(s) in {destTable}".Dump();
if (destRowCount > 0)
{
$"Rows found in destiation table - aborting!".Dump();
return;
}
var reader = conn.ExecuteReader($"Select {string.Join(",", cols)}
 From HAProxyLogs.dbo.{tableName};");

using (SqlConnection dest = GetConn())
{
dest.Open();
using (SqlBulkCopy bc = new SqlBulkCopy(dest))
{
bc.BulkCopyTimeout = 5*60;
bc.BatchSize = 1048576;
bc.NotifyAfter = 100000;
bc.DestinationTableName = GetDestTableName(date);
bc.SqlRowsCopied += (e, s) => UpdatePB(s.RowsCopied, rowCount);
foreach (var c in cols)
{
bc.ColumnMappings.Add(c, c);
}
try
{
bc.WriteToServer(reader);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
reader.Close();
}
}
var destFinalCount = GetDestRowCount();
}
}
}
public SqlConnection GetConn() =>
new SqlConnection(new SqlConnectionStringBuilder
{
DataSource = "servername",
InitialCatalog = "TrafficLogs",
IntegratedSecurity = true
}.ToString());
public string GetTableName(DateTime dt) => $"Log_{dt:yyyy_MM_dd}";
public string GetDestTableName(DateTime dt) => $"HAProxyLogs_{dt:yyyy_MM}";

Ideally, this was going to be as simple as kicking off the LINQPad script, and letting it run in the background looping through all the tables. I thought I wouldn’t have to worry about it, but things are never really that easy.

I started the migration in Colorado, on January 14, 2019 (yes, I know the exact date). Pulling millions of rows of data from spinny drives, and then inserting it back into new tables on those same spinny drives was slow. By day 2, knew this was going to be painful. My original plan was to run the migration in Colorado first, then in New York. However, once I saw how slow each table was taking, I decided to run them simultaneously. About a week later, I kicked off the New York data migration.

Data Migration Issues

Early on, I hit various issues that I needed to work through. Some of these were easy fixes, others not so much.

Script Timeouts

The first issue was with the C# script. The script periodically timed-out when querying the database. After adjusting the settings, I crossed my fingers and hoped it would continue to work, which it did, for the most part.

Random Script Errors

Next, I ran into an issue, where the script would continue to the next table, aka day, if there was an error. That was terrible because we wanted the data to be inserted in date order, for the clustered columnstore index. If the script errored and moved on to the next day, cleanup would need to be done for the day that didn’t finish. At that point, I’d have to start the process over for that particular day.

You might be wondering what I mean by “cleanup”.

Let me take a quick step back and explain the old process. The original table structure aka the daily tables were rowstore with a generic identity (ID) column to make each row unique. After 14 days, we would drop the primary key from the table, and add a clustered columnstore index. Even though the conversion to a clustered columnstore occurred, the ID column would remain.

In the new table, we dropped the ID column. Dropping this column meant, trying to figure out the uniqueness of the rows inserted before a failure would be incredibly difficult. It was far easier, in the event of a failure, to just delete everything from the table for that day, and everything after that day (if it moved to another table), and restart the process.

Data Validation

I realized I needed to validate that the total number of rows inserted into the new table matched the total number of rows from the old daily table. Sounds easy, right? Well, not necessarily.

Sure, the original tables were per day, but there were cases where instead of containing data from date 00:00-23:59 it would potentially include some rows from date+1. This was a known issue (due to latency of logs arriving from HAProxy), and I was told the totals didn’t have to match perfectly - if we lost some rows, it was fine. The thing is, when you’re responsible for moving data from one system to another, you really want things to match…perfectly.

Because of the stragglers from a different day, I couldn’t just query the total rows from the old table and verify that the new table contained the same count. I had to come up with a way to query the count by day. I did this by adding a column to the new table called OriginalLogTable and then populating it with the name of the old table, for example Log_2019_07_01. Having this column allowed me to compare the count of rows in the old table, with the count in the new table by grouping by the OriginalLogTable. This solved my problem because when I finished a month, I could easily verify that each day matched between the old and new tables, using the script below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110


if object_id('tempdb..#NewTableDetails') is not null
 drop table #NewTableDetails

create table #NewTableDetails
(
 TrafficLogOrigTable varchar(50),
 TrafficLogTotalRows bigint,
 OriginalLogDate date
)

if object_id('tempdb..#LogDetails') is not null
 drop table #LogDetails

create table #LogDetails (LogDate date, OriginalTotalRows bigint)

declare @sql nvarchar(max) = ''
declare @startdate datetime = '2019-10-01'
declare @enddate datetime = '2019-11-01'

if object_id('tempdb..#dates') is not null
 drop table #dates;

CREATE TABLE #dates
(
 [Date] DATE PRIMARY KEY,
 FirstOfMonth AS CONVERT(DATE, DATEADD(MONTH, DATEDIFF(MONTH, 0, [Date]), 0)),
 OldTableName as concat('Log_', year([date]), '_', right('0' + rtrim(month([date])), 2), '_', right('0' + rtrim(day([date])), 2)),
 NewTableName as concat('HAProxyLogs_', year([date]), '_', right('0' + rtrim(month([date])), 2))
);

INSERT #dates([Date])
SELECT d
FROM
(
 SELECT d = DATEADD(DAY, rn - 1, @StartDate)
 FROM
 (
 SELECT TOP (DATEDIFF(DAY, @StartDate, @enddate))
 rn = ROW_NUMBER() OVER (ORDER BY s1.[object_id])
 FROM sys.all_objects AS s1
 CROSS JOIN sys.all_objects AS s2
 ORDER BY s1.[object_id]
 ) AS x
) AS y;

-- for each new table name get the count of rows, get list of OldTables
-- Imported and then get the counts for each one
declare @newtablename nvarchar(100)
declare @date datetime
declare newtable_cursor cursor for
 select distinct NewTableName
 from #dates
 where [date] < @enddate
open newtable_cursor
fetch next from newtable_cursor into @newtablename

while @@FETCH_STATUS = 0
begin
 set @sql = 'insert into #NewTableDetails (TrafficLogOrigTable, TrafficLogTotalRows)
 select
 TrafficLogOrigTable = OriginalLogTable,
 TrafficLogTotalRows = count(*)
 from TrafficLogs.dbo.[' + @newtablename +']
 group by OriginalLogTable ';

 exec sp_executesql @sql

 FETCH NEXT FROM newtable_cursor INTO @newtablename
end
CLOSE newtable_cursor
DEALLOCATE newtable_cursor

update #NewTableDetails
set OriginalLogDate
 = Cast(Replace(Replace(TrafficLogOrigTable, 'Log_', ''), '_', '-') as date)

declare @oldTableName nvarchar(100)
declare @oldtabledate date
declare table_cursor cursor for
 select OriginalLogDate, TrafficLogOrigTable
 from #NewTableDetails
open table_cursor
fetch next from table_cursor into @oldtabledate, @oldTableName

while @@FETCH_STATUS = 0
begin
 set @sql = 'insert into #LogDetails (LogDate, OriginalTotalRows)
 select
 LogDate = '''+ convert(varchar(10), @oldtabledate, 23) +''',
 OriginalTotalRows = count(*)
 from HAProxyLogs.dbo.[' + @oldTableName +']';
 exec sp_executesql @sql


 FETCH NEXT FROM table_cursor INTO @oldtabledate, @oldTableName
end
CLOSE table_cursor
DEALLOCATE table_cursor

select
 nt.OriginalLogDate,
 nt.TrafficLogOrigTable,
 nt.TrafficLogTotalRows,
 ot.OriginalTotalRows,
 ot.LogDate,
 IsMatch = case when nt.TrafficLogTotalRows = ot.OriginalTotalRows then 'Y' else 'N' end
from #NewTableDetails nt
full join #LogDetails ot
 on nt.OriginalLogDate = ot.LogDate
order by nt.OriginalLogDate

Column Changes

I also discovered that we added new columns after the original daily tables were designed. This meant that over the course of the data migration the table structure would change. I adjusted the LinqPad script so it would add the new columns when needed based on date, this would make sure it continued to work and we wouldn’t skip new columns.

There were many more issues which involved debugging the script, but eventually I had a C# script that fixed all of the above problems. The new script looked a little like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124


<Query Kind="Program">
<NuGetReference>Dapper</NuGetReference>
<Namespace>Dapper</Namespace>
</Query>
void Main()
{
DateTime date = new DateTime(2016, 07, 26);
while (date < DateTime.UtcNow)
{
MoveDate(date);
date = date.AddDays(1);
}
}
static readonly List<string> cols = new List<string> { "<col list>" };
public void MoveDate(DateTime date)
{
var tableName = GetTableName(date);
var destTable = GetDestTableName(date);
$"Attempting to migrate {date:yyyy-MM-dd} from {tableName}
 to {destTable}".Dump($"{date:yyyy-MM-dd}");
if (date >= new DateTime(2017, 01, 12))
{
cols.Add("new col1");
cols.Add("new col2");
}
using (var conn = GetConn())
{
int rowCount;
try
{
rowCount = conn.QuerySingle<int>($"Select Count(*)
 From HAProxyLogs.dbo.{tableName};");
}
catch (SqlException e)
{
(" Error migrating: " + e.Message).Dump();
return;
}
$" Summary for {date:yyyy-MM-dd}".Dump();
$" {rowCount:n0} row(s) in {tableName}".Dump();
var pb = new Util.ProgressBar($"{tableName} (0/{rowCount})");
pb.Dump(tableName + " copy");
Func<int> GetDestRowCount = ()
=> conn.QuerySingle<int>($"Select Count(*)
 From {destTable}
 Where CreationDate >= @date
 And OriginalLogTable = '{tableName}';", new { date });
Action<long, int> UpdatePB = (copied, total) =>
{
pb.Fraction = (double)copied / total;
pb.Caption = $"{tableName} ({copied}/{total})";
};
var destRowCount = GetDestRowCount();
$"There are {destRowCount:n0} row(s) for table: {destTable}".Dump();
if (destRowCount > 0)
{
$"Rows found in destination table - aborting!".Dump();
return;
}
var sqlString = $"Select {string.Join(",", cols)}, OriginalLogTable = '{tableName}'
 From HAProxyLogs.dbo.{tableName};";
 var reader = conn.ExecuteReader(sqlString);
using (SqlConnection dest = GetConn())
{
dest.Open();
using (SqlBulkCopy bc = new SqlBulkCopy(dest))
{
bc.BulkCopyTimeout = 15*60;
bc.BatchSize = 1048576;
bc.NotifyAfter = 100000;
bc.DestinationTableName = GetDestTableName(date);
bc.SqlRowsCopied += (e, s) => UpdatePB(s.RowsCopied, rowCount);
foreach (var c in cols)
{
bc.ColumnMappings.Add(c, c);
}
try
{
bc.ColumnMappings.Add("OriginalLogTable", "OriginalLogTable");
bc.WriteToServer(reader);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
reader.Close();
}
}
// try updating stats after full dump to see if that stops time-outs
 var statsString
= $"waitfor delay '00:00:05';
 update statistics dbo.[{destTable}] [{destTable}_OriginalLogTable];";
conn.Query(statsString);
var destFinalCount = GetDestRowCount();
$"After migration there are {destFinalCount:n0} row(s) in {destTable}".Dump();
}
}
}
public SqlConnection GetConn() =>
new SqlConnection("Data Source=servername;Initial Catalog=TrafficLogs;Trusted_Connection=True;Connection Timeout = 1500;");
public string GetTableName(DateTime dt) => $"Log_{dt:yyyy_MM_dd}";
public string GetDestTableName(DateTime dt) => $"HAProxyLogs_{dt:yyyy_MM}";

It’s So Slow

We all knew this was going to be a slow process, but I don’t think anyone realized how slow. I did everything I could think of to try and speed up the script.

It took about 2-2.5 hours to insert each old table into the new table. By the middle of February, I had:

1332 tables left to migrate in New York
and 782 tables left in Colorado

At the rate it was going, I knew it was going to take months to complete.

I went back to the drawing board to figure out some way to move the data faster. Obviously, I wanted to follow the same steps - insert old data, verify the row count matches, move to the next day, repeat - I used the C# script as my model - I mean it worked, it was just a little slow. I tried a bunch of different things, none were blazingly fast:

an SSIS package - sorry SSIS fans, this was even slower and I won’t embarrass myself by sharing it
a SQL script using dynamic SQL - this was also pretty slow

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106


declare @startDate datetime = '2015-11-01'
declare @endDate datetime = '2015-12-01'

declare @OldTableName varchar(100)
declare @NewTableName varchar(100)

declare @cols varchar(max) = 'comma, separated, col, list'
declare @originalrowcount int
declare @newrowcount int
declare @finalrowcount int
declare @rowcountsql nvarchar(max)

declare @insertsql nvarchar(max)

declare @statssql nvarchar(max)

while @startDate < @endDate
begin

set @OldTableName
 = concat('Log_', year(@startDate), '_'
 , right('0' + rtrim(month(@startDate)), 2), '_'
 , right('0' + rtrim(day(@startDate)), 2))
set @NewTableName
 = concat('HAProxyLogs_', year(@startDate), '_'
 , right('0' + rtrim(month(@startDate)), 2))

print concat('Attempting to migrate ', convert(varchar(10), @startDate, 120)
 , ' from ', @OldTableName, ' to ', @NewTableName)
print ' '

if @startDate >= '2017-01-12'
 begin
 if (charindex('new col1', @cols) = 0)
 begin
 set @cols = concat(@cols, ', new col1')
 end

 if (charindex('new col2', @cols) = 0)
 begin
 set @cols = concat(@cols, ', new col2')
 end
 end

set @rowcountsql = concat('select @cnt = count(*) from HAProxyLogs.dbo.[', @OldTableName, ']');
exec sp_executesql @rowcountsql, N'@cnt int output', @cnt = @originalrowcount output;

/*
print concat('Summary for ', convert(varchar(10), @startDate, 120)
 , ' ', @originalrowcount, ' rows in ', @OldTableName);
print ' '
print concat('Starting copy of ', @OldTableName);
print ' '
*/

-- check if the new table has any rows in it
set @rowcountsql
 = concat('select @cnt = count(*) from ', @NewTableName
 , ' where CreationDate >= ''', convert(varchar(10), @startDate, 120)
 , ''' and OriginalLogTable = ''', @OldTableName, ''';')
--print @rowcountsql
exec sp_executesql @rowcountsql, N'@cnt int output', @cnt = @newrowcount output;

print concat('There are ', @newrowcount, ' row(s) for table: ', @NewTableName);
print ' '

If @newrowcount > 0
 begin
 print 'Rows found in the destination table - aborting'
 print ' '
 break;
 end

set @insertsql
 = concat('insert into TrafficLogs.dbo.', @NewTableName, '(', @cols, ', OriginalLogTable)
 select ', @cols, ', OriginalLogTable = ''', @OldTableName, '''
 from HAProxyLogs.dbo.', @OldTableName);
print @insertsql
print ' '
exec sp_executesql @insertsql;

set @statssql
 = concat('update statistics TrafficLogs.dbo.', @NewTableName
 , ' ', @NewTableName, '_OriginalLogTable');
print @statssql
print ' '
exec sp_executesql @statssql

set @rowcountsql
 = concat('select @cnt = count(*) from ', @NewTableName
 , ' where CreationDate >= ''', convert(varchar(10), @startDate, 120)
 , ''' and OriginalLogTable = ''', @OldTableName, ''';')
exec sp_executesql @rowcountsql, N'@cnt int output', @cnt = @finalrowcount output;

print concat('After migration there are ', @finalrowcount, ' row(s) in the ', @NewTableName);

If @originalrowcount <> @finalrowcount
 begin
 print 'final row does not match original stopping operation';
 print ' '
 break;
 end

set @startDate = dateadd(day, 1, @startDate)

END

a PowerShell version - slow but seemed to be more stable than the original C# script

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162


<#
 .SYNOPSIS
 Traffic Log Data Migration

 .EXAMPLE
 .\DataMigration.ps1 -ServerName name -StartDate 2019-01-01 -EndDate 2019-02-01
#>
[CmdletBinding()] #See http://technet.microsoft.com/en-us/library/hh847884(v=wps.620).aspx for CmdletBinding common parameters
param(
[parameter(Mandatory = $true)]
[string]$ServerName,
[parameter(Mandatory = $true)]
[DateTime]$StartDate,
[parameter(Mandatory = $true)]
[DateTime]$EndDate
)
# taken from https://blog.netnerds.net/2015/05/getting-total-number-of-rows-copied-in-sqlbulkcopy-using-powershell/
$source = 'namespace System.Data.SqlClient
{
 using Reflection;
 public static class SqlBulkCopyExtension
 {
 const String _rowsCopiedFieldName = "_rowsCopied";
 static FieldInfo _rowsCopiedField = null;
 public static int RowsCopiedCount(this SqlBulkCopy bulkCopy)
 {
 if (_rowsCopiedField == null) _rowsCopiedField
 = typeof(SqlBulkCopy).GetField(_rowsCopiedFieldName, BindingFlags.NonPublic
 | BindingFlags.GetField | BindingFlags.Instance);
 return (int)_rowsCopiedField.GetValue(bulkCopy);
 }
 }
}
'
Add-Type -ReferencedAssemblies 'System.Data.dll' -TypeDefinition $source
$null = [Reflection.Assembly]::LoadWithPartialName("System.Data")
Function ConnectionString([string] $ServerName, [string] $DbName)
{
"Data Source=$ServerName;Initial Catalog=TrafficLogs;Trusted_Connection=True;Connection Timeout = 2000;"
}
Function GetOriginalTableRowCount($sqlConnection, $table){
$sqlConnection.open();
$OriginalRowCountCmd = "select OriginalCount = count(*) from " + $table;
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand($OriginalRowCountCmd, $sqlConnection)
$SqlCmd.CommandTimeout = 0;
$row_count = [Int64] $SqlCmd.ExecuteScalar()
$sqlConnection.Close();
return $row_count
}
Function GetNewTableRowCount($sqlConnection, $table){
$sqlConnection.open();
$RowCountCmd = "select NewCount = count(*) from " + $table
+ " where CreationDate >= '"+($i.ToString("yyyy-MM-dd"))
+"' and OriginalLogTable = 'Log_"+($i.ToString("yyyy_MM_dd"))+"'";
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand($RowCountCmd, $sqlConnection)
$SqlCmd.CommandTimeout = 0;
$row_count = [Int64] $SqlCmd.ExecuteScalar()
$sqlConnection.Close();
return $row_count
}
$cols = "col1", "col2", "col3", "...";
for ($i = $StartDate; $i -lt $EndDate; $i = $i.AddDays(1))
{
$MigrationStart = Get-Date
Write-Host "Migration started at: $MigrationStart"
$OldTableName = "HAProxyLogs.dbo.Log_"+($i.ToString("yyyy_MM_dd"))
$NewTableName = "TrafficLogs.dbo.HAProxyLogs_"+($i.ToString("yyyy_MM"))
Write-Host "Attempting to migrate "($i.ToString("yyyy_MM_dd"))" from " $OldTableName " to " $NewTableName;
# build the connection string and open it
$SrcConnStr = ConnectionString $ServerName
$SrcConn = New-Object System.Data.SqlClient.SQLConnection($SrcConnStr)
# whats the row count on the original table in HAProxyLogs
$originalRowCount = GetOriginalTableRowCount $SrcConn $OldTableName
Write-Host "Summary for " ($i.ToString("yyyy-MM-dd"))":
 There are "$originalRowCount" rows in table "$OldTableName
# make sure there is no data in the table currently for the day
$newRowCount = GetNewTableRowCount $SrcConn $NewTableName
if($newRowCount -gt 0){
Write-Host "Rows found in the destination table - aborting migration"
break;
}
else {
Write-Host "No data for the date in the destination table - continuing migration"
}
# select the data we need from the original table
$selectCmdText = "select "+ ($cols -join ', ')
+", OriginalLogTable = 'Log_"+($i.ToString("yyyy_MM_dd"))
+"' from "+$OldTableName;
$SqlCommand = New-Object system.Data.SqlClient.SqlCommand($selectCmdText, $SrcConn)
$SqlCommand.CommandTimeout = 0;
$SrcConn.Open()
[System.Data.SqlClient.SqlDataReader] $SqlReader = $SqlCommand.ExecuteReader()
$destConnection = ConnectionString $ServerName
$bulkCopy = New-Object Data.SqlClient.SqlBulkCopy($destConnection)
$bulkCopy.BatchSize = 1048576
$bulkCopy.BulkCopyTimeout = 0 # 20*60
$bulkCopy.DestinationTableName = $NewTableName
$bulkCopy.NotifyAfter = 500000
# taken from https://blog.netnerds.net/2015/05/getting-total-number-of-rows-copied-in-sqlbulkcopy-using-powershell/
$bulkCopy.Add_SqlRowscopied(
{Write-Host "$($args[1].RowsCopied)/$originalRowCount rows copied
 from $OldTableName to $NewTableName" })
foreach ($col in $cols){
$bulkCopy.ColumnMappings.Add($col,$col)
}
try {
Write-Host "Starting Migration"
$bulkCopy.ColumnMappings.Add("OriginalLogTable", "OriginalLogTable");
$bulkCopy.WriteToServer($SqlReader)
$total = [System.Data.SqlClient.SqlBulkCopyExtension]::RowsCopiedCount($bulkcopy)
Write-Host "$total total rows written"
}
catch {
$ex = $_.Exception
Write-Error $ex.Message
While($ex.InnerException)
{
$ex = $ex.InnerException
Write-Error $ex.Message
}
}
Finally
{
$SqlReader.Close()
$bulkCopy.Close()
$SrcConn.Close()
}
# once done update statistics on the new table
$statsCmdText = "update statistics "+$NewTableName
+" HAProxyLogs_"+($i.ToString("yyyy_MM"))+"_OriginalLogTable";
Invoke-DbaQuery -SqlInstance $ServerName -Query $statsCmdText;
Write-Host "After migration there are $total rows
 were written to $NewTableName table from $OldTableName";
if($originalRowCount -ne $total){
Write-Host "total rows migrated does not match. Stopping operation"
break;
}
$total = 0;
}

All of my options were slow.

I ended up using the PowerShell option because I could easily run multiple months at once. I had 3 PowerShell sessions going, which let me chew through 3 months at a time. It was far easier to run several months in PowerShell than with the C# script. Utilizing PowerShell allowed me to get through a lot more data in a shorter amount of time. Don’t get me wrong it was still slow - each session was moving about a month of tables in about a week of work - but at least I was moving more data and chasing the target slightly faster.

While the PowerShell script wasn’t perfect, and I still hit issues with timeouts and errors, it overall worked better. Now that I had a somewhat working process that didn’t need daily hand-holding, it was time to focus on some of the other problems with the migration.

Continuous Issues of No Free Disk Space

When your servers are already limited in free space and you have to move 40TB of data around on the same server, it becomes a little like whack-a-mole to get things moved. I had to get creative to free up space while continuing to move all the data to the new format.

Deleting and Shrinking

Yes, you read that right. Shrinking. Before I get to that, let me explain why I needed to do this.

We didn’t have new hardware yet and these were production SQL Servers - meaning the old database was still receiving new data and we were adding a new table everyday. As I was moving tables to the new database, we were adding tables with hundreds of millions of rows of data to the old database. This was all happening on the exact same server and drive with minimal free space.

Hence the need to juggle.

After moving each month, I would do a final validation, then it was time for some cleanup. This included deleting all of the daily tables for the month that were just migrated, as well as dropping the OriginalLogTable column from the monthly table in the new database. The daily tables were no longer needed because we had the data in the new table, so getting rid of duplicate data would allow me to gain back disk space…sort of.

I say sort of, because I was just moving data on the same drives. We changed formats from daily tables to monthly tables, and while we got better compression in the new tables, there weren’t huge gains in free space. We had the same number of rows of data stored in fewer tables, which meant less bits in the long run, but in order to gain back the space allocated to the old data files, I had to shrink them.

As I moved the data from the old files, I was increasing the size of the new data files. This was all being done on the E: drive that was 85-90% full, which I mentioned above. I was constantly trying to keep the used disk space in check.

It was quite a balancing act.

Each month of data was anywhere from 300GB to 1TB in size, and the 3 old data files were approximately 12TB each. At the end of each migration, I tried to shrink the old database files by the amount we just deleted. This took several attempts to get something that worked well.

Initially, I naively tried to run DBCC SHRINKFILE(FileName, TRUNCATEONLY) to free up space. As I expected, that did not work.

Next, I tried to run DBCC SHRINKFILE(FileName, CurrentFileSize - SizeOfDataJustDeleted) on each file. That didn’t work either. Well to be clear, it would have if we had more drive space on the D: drive where the log file was stored. The D: drive was already about 85% full with tempdb, the HAProxyLogs log file, and another datafile. When you run DBCC SHRINKFILE and let’s say you want to shrink a file by 300GB, then you need to have the ability to increase your database transaction log file by the amount you’re trying to shrink. Since we were already constrained on drive space where the t-log was stored, I wasn’t able to shrink the files by the amount I deleted because I kept running out of space on the D: drive.

I had a couple of options to free up space on the D: drive - move tempdb or move the other log/database files to our newly configured F: drive with 14TB of free space. I decided to move tempdb because it was as easy as executing the following command and restarting the SQL service.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


alter database tempdb
 modify file (NAME = tempdev, FILENAME = 'F:\Data\tempdev.mdf', SIZE = 20000MB);
alter database tempdb
 modify file (NAME = templog, FILENAME = 'F:\Data\templog.ldf', SIZE = 1000MB);
alter database tempdb
 modify file (NAME = tempdev2, FILENAME = 'F:\Data\tempdev2.mdf', SIZE = 20000MB);
alter database tempdb
 modify file (NAME = tempdev3, FILENAME = 'F:\Data\tempdev3.mdf', SIZE = 20000MB);
alter database tempdb
 modify file (NAME = tempdev4, FILENAME = 'F:\Data\tempdev4.mdf', SIZE = 1000MB);

By moving tempdb we had minimal downtime and I gained approximately 122GB in free space. While that gave me a little more room to shrink the old data files, it still wasn’t quite working as I wanted, so it was back to the drawing board.

Trying to shrink the files in huge chunks of 300GB or more was too much, it was slow and resulted in blocking of the inserts into the old database. I found a solution that seemed promising and would involve shrinking the files in much smaller bits in a loop, so I figured I’d try it. It took a little bit of poking to get the right number to shrink in every execution, but I eventually ended up using something similar to this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


declare @from int
declare @leap int
declare @to int
declare @datafile varchar(128)
declare @cmd varchar (512)
declare @starttime datetime
declare @endtime datetime

/*settings*/
set @from = 13533200 /*Current size in MB*/
set @to = 11600000 /*Goal size in MB*/
set @datafile = 'HAProxyLogs1' /*Datafile name*/
set @leap = 10000 /*Size of leaps in MB*/

while ((@from - @leap) > @to)
 begin

 set @starttime = getdate()

 print concat('started: ', @starttime)

 set @from = @from - @leap
 set @cmd = 'DBCC SHRINKFILE (' + @datafile +', ' + cast(@from as varchar(20)) + ')'
 print @cmd
 exec(@cmd)
 print '==> SHRINK SCRIPT - '+ cast ((@from-@to) as varchar (20)) + 'MB LEFT'

 set @endtime = getdate()

 print concat('ended: ', getdate(), ' took :' , datediff(minute, @starttime, @endtime))
 end

set @cmd = 'DBCC SHRINKFILE (' + @datafile +', ' + cast(@to as varchar(20)) + ')'
print @cmd
exec(@cmd)

I’d run the script against each file separately to fully recover the disk space from the month that was deleted. Again it was slow, but it worked.

I’m aware of all the issues with shrinking database files, and no, I wouldn’t normally do it. However, shrinking the files was necessary to move the project forward, as I had to get around the lack of disk space.

Just Use the New Drive

Now that I had a script to migrate the data relatively quickly (the PowerShell one), and had a script to shrink the old data files, you’d think I’d be all set to just run with this project, right?

Not so fast.

When I set up the new TrafficLogs database I put the data files for the archive (the old stuff being migrated) on the drive with the spinny disks. We didn’t think about using the new F: drive with 14TB of free SSD space to use, mainly because it wasn’t going to be enough for the entire migration. But after a few weeks, I decided to use that drive for a few reasons:

They were NVMe SSDs. Which meant writing to them was much faster than writing to the the spinny drives
It was about 14TB of free space, which let me move a huge amount of data without worrying about shrinking immediately. I could punt some of the whack-a-mole with the disk space.

Unfortunately, the only issue with doing this was once I filled the 14TB of disk space, I had to migrate the datafiles all back to the spinny drives. Which I did at the end of March 2019 after I ran out of drive space.

Was this ideal? No. Was it a pain? Yes. But again, it moved the entire project forward and I was still slowly getting all the data migrated over.

Drives, Please Don’t Die

In early May 2019, I was still plugging away at migrating data. At this point, I was back to where I started - copying data from the old database on the spinny drives, inserting it into the new database, deleting the old data, and then shrinking the files all on the same drives.

After months of doing this, I was seeing a noticeable slowdown in the process. The read/write delays on the drives had increased since January. The average delays of reads were over 100 milliseconds and about 80 milliseconds for writes.

The overall response time was terrible.

I was struggling with PAGEIOLATCH_SH waits as I was trying to insert data into the new tables.

We had been using the spinny drives for years, and I had been hitting them non-stop for about 4 months. I was very worried the drives were going to fail before I was done migrating all of the data.

Thankfully, they didn’t. I eventually was able to move all of the old data over to the new database. It only took 6 months to complete.

I was done, but not really. We still didn’t have the new servers which meant every table created would still need to be migrated until we had them, and had cutover to the new servers and database.

The moving target was still moving.

Finally, New Hardware

The plan was to get the new servers in June/July, but the purchase was delayed several months. After respeccing them to have all NVMe SSD storage, they finally arrived in late September. To make the final migration of the new hardware as easy as possible, from June through the end of September, I moved one table a day, keeping things up to date.

In early October, the new servers were ready for me to start installing SQL Server on them. The end was near.

After lots of fussing with the Windows Server install, I eventually got installed SQL Server 2017 on both of them. It was time for the next obstacle moving the gigantic databases over.

Backup and Restore Failure

The most obvious way to move a database from the old server to the new server is, of course, to take a backup and then restore it. So, that’s what I did, or I should say, tried to do. I say tried because when I took the backup, I set the script to write it out on the new server, but unfortunately, the backup took most of the space which meant I couldn’t restore it. It was a bit of a facepalm moment.

We’re a very lean shop when it comes to hardware, and this is by far our largest chunk of data we have to store, so there wasn’t a place anywhere on the network to store a 33-38TB full backup temporarily, while it was being restored to a server.

Time for more juggling.

Copying Tables…Again

Since I wasn’t able to take a backup and restore it, I was back to square one, trying to figure out how to quickly move this data. After spending almost the entire year moving tables one by one, the idea of having to do it all over again pretty much drove me to tears, but that’s what I tried next.

I would be moving from server to server within the same datacenter and rack, so I wanted to see how fast it would be to use various methods to bulk insert the data from the old database to the new one. I tried:

PowerShell method - took 36 hours to move one table with 4 billion rows
SSIS version - took 2.5 days to move the same table
OPENROWSET - took 16 hours to move 650 million rows

PowerShell was the winner when it came to speed, but Nick Craver wasn’t convinced. He decided to try it. He wrote a little LINQPad script and after running it he finally believed me that moving this much data sucked.

At this point, I was devastated because it looked like I was going to have to migrate everything for a second time.

Time to Try an Availability Group

I really, really, really did not want to move hundreds of tables again, so I threw out the idea of using an availability group to automatically seed the databases to the new servers. We already have AGs throughout our infrastructure, so I was familiar with using them, but I was unsure if it would successfully work for a database of this size.

Before starting, I pinged Sean Gallardy and asked if he thought it was possible to automatically seed a database of between 30-40TB - he guessed it’d be slow but possible, maybe taking 1-3 days. I was 100% ok with waiting 1-3 days, if it saved me months of moving tables again.

I decided to spin up the availability group on the Colorado server first, and if it was successful, I’d move to New York. In order to set this up, I would need to go through all these steps:

Set up new Windows Failover Cluster with the two servers
Take another backup, as I couldn’t use the first one because the database was in simple recovery instead of full. This meant waiting another 11.5 hours for the backup of the 33TB database in Colorado to finish
Set up the AG and fingers crossed let it seed
Initiate failover to the new server
Destroy AG and shutdown old server

After backup in Colorado finished, I realized that we might have a small problem when trying to autoseed. The docs explain:

Automatic seeding requires that the data and log file paths are the same on every SQL Server instance participating in the availability group

The old servers had the data on E: and F:, and the new servers were using D:. To be certain there were no issues, I quickly renamed the drive on the new server to E: and moved the files over using:

1
2
3
4
5
6
7


use [master]
​
alter database [TrafficLogs]
modify file(name= TrafficLogs_Current, filename = 'E:\Data\TrafficLogs_Current.mdf');
​
alter database [TrafficLogs]
modify file(name= TrafficLogs_log, filename = 'E:\Data\TrafficLogs_log.ldf');

By changing the assigned letter to the drive, I also had to move tempdb to the new drive, but once all that was done it was time to try setting up the availability group.

I spun up the temporary AG in Colorado, waited and watched SQL Server. I was constantly checking both DMVs sys.dm_hadr_automatic_seeding and sys.dm_hadr_physical_seeding_stats to see the status of the seeding process. I was concerned it would fail and we’d be back to the drawing board. After about 6 hours, it looked like we had seeded about 17TB of the database.

We also could see from our SignalFX monitoring of network traffic that something was happening. The SignalFX chart below shows we were averaging about 150 - 180 Mb/sec of traffic to a box that wasn’t being used by anything except for the database seeding.

By the next morning, the seeding had finished and we had a database with all the tables on the new Colorado server. Progress!!

Before initiating the failover, I ran a DBCC CHECKDB on the new server aka the secondary in the availability group to be sure the database was in a good state, and after 22 hours it completed with no issues reported.

It was time for the failover. I was a little nervous about doing it, but we were ecstatic, shocked, and impressed (all-in-one) that it worked.

While all of this work was happening in Colorado, I kicked off the process on the New York server since the database was larger, and I knew it was going to take a long time. After taking 41 hours to backup the 40TB database, I was ready to follow all the same steps on the New York server. I set up the cluster, availability group, seeded the database, ran a very long DBCC CHECKDB, and finished with a successful failover.

The hardest parts were done, but it wasn’t quite time to celebrate. I still had some final work to do.

Final Cleanup

We finally had the new servers in place with the databases, but we didn’t have the new traffic log data flowing to the new servers quite yet. The service that sends the traffic logs (Traffic Processing Service aka TPS) was still pointing to the old database. This was done purposely, to not interrupt any of the teams using the data. The idea was, once the servers were in place, we could side-by-side release the new version to push data in both places for a period of time (by sending syslog traffic from HAProxy to both servers). This would allow teams to port their processes to the new database and table structures in a gradual way.

The week after the failover, we released the new traffic processing service and I did a few more days of data migration from the old server. I also destroyed the availability group and the windows failover clusters. Once all that was done, it was finally time to celebrate.

Final Thoughts

During my time as the DBA at Stack Overflow, I’ve had the opportunity to work on various large projects (some documented here and here), but none have taken as long as this one. This took about 11 months with all of the moving, validating, deleting, and juggling space.

I honestly love taking on these big projects. Planning the entire thing, and taking it from start to finish is extremely gratifying. Even though I automated the majority of the work, this project was incredibly frustrating and I definitely don’t want to do it again anytime soon (unless I have plenty of drive space and don’t have to juggle things for 11 months). This project was challenging in many ways, and considering I had to do the same work on two servers it was doubly challenging. In the end, I was very relieved to successfully use an availability group to seed the databases to the new servers, and finally be done with moving all the things.

Have you ever had to move multiple databases of this size? Were you lucky enough to have extra disk space to get it done? I’d be curious how others would have solved the problem of moving about 40TB of data (twice) with minimal free disk space? If you have thoughts, let me know.

SQL Server 2019 Tuple Mover Behavior Change

Mon, 20 Jul 2020 04:00:00 +0000

This is a follow-up to my post about an issue with clustered columnstore, when upgrading from SQL Server 2017 to SQL Server 2019. After extensive testing and working with support, I wanted to share some information about a change in SQL Server 2019 that might impact others.

Overview

I suggest reading my other post first, it’ll only take a few minutes. I’ll wait…

However, if you really don’t want to read it, here’s a quick recap on the initial issue.

In early February 2020, a lot of data was deleted from some clustered columnstore indexes in our PRIZM database. Some of the tables were rebuilt, but 11 tables weren’t since we don’t have maintenance windows, and that would involve downtime. The rebuilds would happen once we upgraded to SQL Server 2019, to take advantage of the ability to rebuild those columnstore indexes online.

The day we upgraded to SQL Server 2019, there was rapid growth of the transaction logs for PRIZM. The only way to get the growth to stop, was to enable both trace flags 634 and 661.

Trace Flag	Definition
634	Disables the background columnstore compression task. SQL Server periodically runs the tuple mover background task that compresses columnstore index rowgroups with uncompressed data, one such rowgroup at a time.
661	Disables the ghost record removal process. For more information, see this Microsoft Support article.

After working with support, we were able to remove trace flag 661, but still had 634 in place, so more work needed to be done.

Enabling Another Trace Flag

Removing trace flag 634 from the production server was the next step, but every attempt, led to the rapid increase in the transaction logs and a spike in query timeouts on the server.

After passing lots of details to support, they requested we enable an undocumented trace flag - 10257 - to see if we still experienced the transaction log growth. Since this trace flag isn’t documented, the description we were given is:

10257 - disables the columnstore tuple mover background merge process. The tuple mover background merge runs a reorganize on a few rowgroups at a time with more conservative thresholds.

Time for more testing!!

I removed trace flag 634, and enabled trace flag 10257 to see how the transaction logs responded. The swap was done for about an hour and during this time the transaction logs didn’t have the rapid growth, but there still was an increase in query timeouts.

The rowgroup states also changed:

Rowgroup State	Before with TF 634 (only)	After with TF 10257 (only)
Compressed	9844	9927
Closed	261	194
Open	721	721
Tombstone	48	68

The change in the numbers showed the tuple mover was running, but with the background merge process disabled there wasn’t the rapid growth of the logs (which was a good thing). However, we had to deal with query timeouts when trace flag 634 was removed. Now what?

More Investigating

I wanted a backup from before the upgrade to do two things:

Check the state of the rowgroups before our initial upgrade. I was hoping to see how many were tombstones, open, closed, compressed?
Could I recreate the issue by upgrading a server from SQL Server 2017 to SQL Server 2019?

It had been a month since we upgraded our SQL Servers, which meant our backups were only available offsite. After a little fighting, I was finally able to pull it from storage.

I initially restored the database to one of our existing test servers with SQL Server 2019 on it. I wanted to see the state of the rowgroups on the tables that had not been rebuilt. Did we have hundreds of tombstones or not? Unfortunately, after restoring the database it didn’t show me what I was looking for. Based on a suggestion from Andy Mallon (b | t), I tried to restore with STANDBY to see if that made any difference…it didn’t. I was able to get stats, but it didn’t have the rowgroups in a tombstone state.

Next, I decided to restore the database to a server with SQL Server 2017 on it, but since I already upgraded all of our servers to 2019, one wasn’t readily available. Thankfully, I have the ability to spin up test servers as needed, so that’s what I did.

After restoring the database to the 2017 SQL Server, it had the following:

No ghost records
No TOMBSTONE rowgroups
666 OPEN rowgroups
0 closed rowgroups
The transaction log was approximately 19.5 GB

Hmmm, ok there was nothing too concerning in the restored database. No red flags or alarm bells. It was time to upgrade to SQL Server 2019 to see what happens.

Immediately after upgrading to SQL Server 2019, everything appeared ok. Then, I waited about 30 minutes to capture the stats again. After 30 minutes on 2019, we had:

24k ghost records
79 TOMBSTONE rowgroups
666 OPEN rowgroups
0 closed rowgroups
and a transaction log that is 43GB

Initially, I didn’t have either trace flag (634, or 661) enabled. Once the trace flags were enabled, the growth of the transaction logs stopped. Welp, I had reproduced the issue.

Support also asked me to grab the results of the following query when the database was still on SQL Server 2017, and again, after I upgraded to SQL Server 2019 to get the status of deleted rows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


select
 i.name,
 rg.object_id,
 rg.partition_number,
 rg.row_group_id,
 rg.delta_store_hobt_id,
 rg.state_desc,
 rg.total_rows,
 rg.deleted_rows,
 100.0*(ISNULL(rg.deleted_rows,0))/total_rows AS 'Fragmentation',
 rg.created_time,
 rg.closed_time
from sys.dm_db_column_store_row_group_physical_stats rg
inner join sys.indexes i
 on rg.object_id = i.object_id
 and rg.index_id = i.index_id
order by i.name

I passed this info along to them, and we were finally able to put it all together.

Putting the Pieces Together

Now that we had more data points, we could step back and look at the big picture, which allowed us to determine what caused the problem in the first place, and how to prevent it going forward.

The Root Cause

The deletions we did in early February were the start of the problem. Without having maintenance windows, we were unable to rebuild the tables, leaving rowgroups in a highly fragmented state with a lot of deleted rows - many of them had 100% deletion. We didn’t see an issue in SQL Server 2017, because the tuple mover only triggered when a rowgroup closed and the tuple mover moved it to a compressed state.

Behavior Change

In SQL Server 2019, the tuple mover has a helper called the background merge task. The docs have some details about it:

Starting with SQL Server 2019 (15.x), the tuple-mover is helped by a background merge task that automatically compresses smaller OPEN delta rowgroups that have existed for some time as determined by an internal threshold, or merges COMPRESSED rowgroups from where a large number of rows has been deleted. This improves the columnstore index quality over time.

If deleting large amounts of data from the columnstore index is required, consider splitting that operation into smaller delete batches over time, allowing the background merge task to handle the task of merging smaller rowgroups and improve index quality, eliminating the need to schedule index reorganization maintenance windows after data deletion.

In other words, the new background merge task has benefits, such as, helping the tuple mover clean up smaller rowgroups to improve the overall quality of the index. This task also automatically kicks in to merge compressed rowgroups where the number of deleted rows hits an internal threshold…kinda sounds a little like what we had.

The Log Growth Issue

We experienced rapid transaction log growth the day we upgraded to SQL Server 2019 because the background merge task was triggered by the deleted rows we had in various tables. The log growth was due to the merge task trying to free the space in the rowgroup.

Since we had a lot of rowgroups with deleted rows, the merge task was aggressively cleaning them up, which led to exponential growth of the logs. By enabling trace flag 634, we disabled the tuple mover process which stopped the logs from growing. If we wanted to keep the tuple mover enabled, then we’d need to use the new trace flag 10257 to disable the new background merge task.

Initially, the new background merge task was extremely aggressive, which is why we saw so many query timeouts. There were a lot of rowgroups that needed cleaning, so the process ran all the time, resulting in contention with our normal load on the server.

We never saw this behavior in any of our testing because we weren’t deleting data from our development environments.

The Resolution

In order to remove trace flag 634 from production, we had a couple of options:

Disable the trace flag and let the tuple mover finish its business
Rebuild the tables

We went with option 1.

I disabled the trace flag and monitored the transaction log which took several hours to finish. During which, we experienced minimal log growth. We also saw query timeouts, but these dropped significantly as the number of rowgroups being processed by the background merge task steadily dropped to zero. Eventually, we had no rowgroups that needed to be cleaned up, which meant no further log growth, and no additional query timeouts from the tuple mover background merge task.

We finally resolved this issue that lingered from the day we upgraded.

Avoiding this Issue

If you’re planning on moving to SQL Server 2019, and you have clustered columnstore indexes, and deleted data from them, then check the total number of deleted rows. If you have a lot, plan to perform maintenance before upgrading, or to utilize the trace flags to keep the tuple mover in check while cleaning the tables.

Don’t be like us, especially if you delete a lot of data from your clustered columnstore indexes.

A critical thing to remember with SQL Server 2019 is, if you need to delete data from your columnstore indexes, do it in smaller batches and trickle deletes. This allows the background merge task to perform the clean up slowly, and minimizes some of the pain we experienced when upgrading.

As a reminder, before enabling any trace flags on your server (especially in production) make sure you know what you’re doing, or, you’re working with support and they suggest using it. The new background merge task is there to assist the tuple mover in keeping columnstore indexes in a better state. In other words, ideally, you’ll leave it alone and let it do its job. If you’re performing maintenance on your columnstore indexes, the background merge task should not negatively impact you, and should actually help with index quality.

Final Thoughts

I tested SQL Server 2019 for months before we upgraded, and didn’t see this behavior at any point. But, as many of us know, upgrades are always fun. There is always a risk of issues when installing new versions. It doesn’t matter how much you test or plan, there’s always a possibility of something going wrong.

I hope this helps someone who is thinking about upgrading, uses clustered columnstore, and doesn’t have an easy way to perform maintenance in their production environment. Check those deleted rows, and test, test, and test again, if you’re planning on upgrading.

At some point soon, I’ll put pen to paper or fingers to keyboard and start writing about the upgrade to SQL Server 2019, along with what benefits we’ve gotten from it so far.

I also would like to personally thank Pedro Lopes (t), for helping with this issue and others we encountered after upgrading to SQL Server 2019, as well as reviewing this post.

SQL Server 2019 Aggressive Clustered Columnstore Cleanup

Fri, 24 Apr 2020 04:00:00 +0000

In late March 2020, we upgraded our production SQL Servers to SQL Server 2019 with CU3. After finishing the upgrade, we hit an issue with clustered columnstore that we hadn’t experienced in the previous version of SQL, SQL Server 2017. The issue also wasn’t something we encountered during our extensive testing on various servers in development, which dated back to September 2019. The problem has been mitigated, but I wanted to share our experience.

Background

We have one database in production, called PRIZM, which is made up exclusively of tables with clustered columnstore indexes. PRIZM is used for event tracking, and it backs our A/B testing, feature funnels, etc. in hopes of improving most of what we do. Because we use PRIZM everywhere, it’s one of our larger databases with lots of tables. At the time of the upgrade, the database was approximately 500GB with about 1000 tables.

There were various driving forces for it, but in early February we needed to purge data from PRIZM. Some of the reasons for doing so included:

disk space being utilized
general cleanup/removal of old events we no longer needed to maintain
ease of GDPR compliance processing

We, or I should say Marc Gravell, deleted approximately 8 billion rows across 912 tables in the database. By doing this, we gained back drive space and removed data we no longer needed to keep.

After performing the deletions, we did some of the recommended maintenance on columnstore indexes. While many of the smaller tables were rebuilt, a handful of tables (11 total) weren’t rebuilt due to their size. Since we don’t have maintenance windows, it would be high-impact to attempt these rebuilds on a very active production environment.

The plan was to perform these table rebuilds after upgrading to 2019. SQL Server 2019 added the ability to rebuild clustered columnstore indexes online and that feature was one of the main reasons we wanted to upgrade.

Upgrade Day

I’m not going to cover our full upgrade to 2019 - that will hopefully be a later post. At this point, let’s just assume everything went as planned and this is picking up after the main failovers of the servers that run Stack Overflow and the rest of the public network.

After we finished the failovers for our public production servers, there was some behind the scenes work that needed to be done. This included patching and upgrading the former primary servers, and a failover of our reporting servers. The reporting cluster is a member of several distributed availability groups. When we’re doing upgrades, we move the primary server from New York (NY-RPTSQL01) to Colorado (CO-RPTSQL01). Once all the upstream SQL Servers are upgraded, there’s one last failover to get the primary back to NY-RPTSQL01. I’m specifically mentioning these servers because they receive data from PRIZM, so they are impacted by any activity in that database.

Transaction Logs Exponentially Growing

As I was upgrading the former primaries, I noticed the transaction log files for PRIZM were growing. Initially, I just assumed it was normal and I went about finishing my other tasks.

After about an hour, the log had grown again, and was growing rapidly. Like really rapidly, which was unusual. The logs were growing about 10GB every 5 minutes. We have lower traffic on the weekend so the amount of data being inserted into the database should not have been anywhere near this level.

Normally large transaction logs might raise my eyebrows, but I wouldn’t be super concerned. However, I needed to perform the failover of the reporting cluster, and with the logs increasing at the rate they were, it was going to make that more difficult, if not impossible.

If you’re not familiar with availability groups, the transaction logs from the primary server need to be written to the secondary servers. This same thing happens with distributed availability groups, but at a larger scale. In this case the transaction logs for the databases, including PRIZM which is on our primary server for Stack Overflow, needed to be flushed to the reporting cluster.

As the logs were increasing in PRIZM, they needed to be written to the reporting servers and CO-RPTSQL01 had a huge log send queue, and NY-RPTSQL01 had a huge recovery queue. Both of those are defined in the Microsoft docs, but I’ve included the definitions below:

Log send queue - Amount of log records in the log files of the primary database, in kilobytes, that has not yet been sent to the secondary replica. This value is sent to the secondary replica from the primary replica. Queue size does not include FILESTREAM files that are sent to a secondary.
Recovery queue - Amount of log records in the log files of the secondary replica that has not yet been redone.

The recovery queue needs to be at zero to failover - when it’s zero it means the server is caught up to the primary. Since there were transaction logs that needed to be written to the secondary in the reporting cluster, NY-RPTSQL01, I couldn’t failover.

What on Earth is Happening?

When I first noticed the log send queue size, it was about 90GB. Within just a few hours it was at 320GB, and growing.

We knew normal traffic wasn’t causing the increase. We also knew there wasn’t a delete job that would have triggered the massive growth of transaction logs, especially at the rate we were seeing. Now it was time to try and figure out what was causing the rapid growth.

I looked at our instance of OpServer to see if it reported any queries running against PRIZM. Nothing.

I executed sp_whoisactive. Nothing.

From what I was seeing it appeared “something” was crawling through our CCI tables and processing “something”. Like all the hand-wavy “something was doing something”? Yeah me too. We didn’t know what was happening and I was frantically trying to get it to stop.

I then executed sp_who2 and saw some PRIZM related things as background/system tasks, so I quickly wrote a script that would let me filter only for PRIZM:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


DECLARE @Table TABLE(
 SPID INT,
 Status VARCHAR(MAX),
 LOGIN VARCHAR(MAX),
 HostName VARCHAR(MAX),
 BlkBy VARCHAR(MAX),
 DBName VARCHAR(MAX),
 Command VARCHAR(MAX),
 CPUTime INT,
 DiskIO INT,
 LastBatch VARCHAR(MAX),
 ProgramName VARCHAR(MAX),
 SPID_1 INT,
 REQUESTID INT
)
INSERT INTO @Table EXEC sp_who2
SELECT *
FROM @Table
WHERE DBName = 'PRIZM'

The output of sp_who2 was repeatedly showing GHOST CLEANUP and CREATE INDEX. Over and over and over again. To be clear, I’m not a clustered columnstore expert, I know enough to be able to maintain them as needed. I went to Twitter and mentioned what I was seeing. I was advised by @sqL_handLe to try trace flag 661 which disables the ghost record removal process, and by Joe Obbish via Erik Darling to enable trace flag 634 to disable the tuple mover background task.

Initially, we enabled trace flag 634, but the logs continued to grow. We disabled trace flag 634. Then we enabled trace flag 661, and the logs continued to grow, so we disabled it. Finally, we tried enabling both of the trace flags. The big jumps stopped, but we now had about 400GB of logs that needed to be written to the reporting cluster before we could perform the failover.

While the logs were exploding we wondered if whatever was happening might have been caused by the deletions we did in early February. But why would they be triggered by the upgrade to SQL Server 2019?

We figured we could just let whatever was running, keep running, and maybe it’d eventually stop. However remember I mentioned one of the reasons we deleted a bunch of data from PRIZM was due to disk space being utilized? Well, the t-log growth was rapidly taking that back. The morning of the upgrade we were at 83% being used, as the day went on we were approaching 90%. This became an urgent issue because the growth was not slowing down and honestly we were worried we’d run out of disk space if we just let it play out.

It was time to open a support ticket. Hopefully support could help us.

A brief side note, before we enabled the trace flags we were getting lots of exceptions across the network. Once we enabled them, the exceptions slowed. Coincidence?

Time to Investigate

The next morning we still had logs that were being written to the reporting cluster, 195GB to be exact. Yes I know that as ridiculously high still, but we have throughput issues between our two datacenters (that’s a whole different story). Due to the size of the recovery queue that meant no failover until that flushed.

We heard back from support, so it was time to investigate. They requested that we take a full dump file, which is pretty difficult since we have 1.5TB of memory on the Stack Overflow SQL Server and the C: drive isn’t large enough to capture that, so we went with a smaller filtered dump and 3 minidumps. That meant another public outage, but we needed the files for them.

When we took the dump files, we disabled both trace flags (634 and 661), during the 10 minutes the trace flags were off, the transaction logs increased another 25-30GB.

We’re Being Haunted by Ghosts

After providing the dump files, they requested that we check the status of the ghost records in the database using and provide the results:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


SELECT
 database_id,
 object_name(object_id) as Name,
 index_id,
 i.record_count,
 GHOST_RECORD_COUNT,
 Version_ghost_record_count,
 INDEX_TYPE_DESC,
 ALLOC_UNIT_TYPE_DESC
FROM sys.dm_db_index_physical_stats (DB_ID(N'PRIZM'), object_id('<TableName>'), NULL, NULL , 'DETAILED') i
ORDER BY GHOST_RECORD_COUNT desc

The results of the query were:

After running the query, I performed another test for support, which involved:

Taking a t-log backup
Removing both trace flags 634 and 661 allowing the logs to increase
Enabling the trace flags
Taking another log backup

Disabling the trace flags for just a few minutes resulted in the logs increasing rapidly, but in looking at the results from the query above, the total number of ghost records dropped.

I decided to test further to see if disabling the trace flags would fully remove the GHOST_RECORD_COUNT. Each time both trace flags were disabled, the total ghost records would drop and the t-logs would explode in size, but we weren’t sure which trace flag we should keep enabled, so it was time for yet another test.

I tried just disabling trace flag 661, which would allow the ghost record process to run. I wanted to see if the cleanup process would finish, and the logs would stop growing. Eventually, the total number of ghost records dropped to zero, and the logs seemed to stop growing. Problem partially solved.

Oh, Tuple Mover what are you doing?

Now that one of the trace flags was disabled, we had to deal with trace flag 634 which we did not have enabled on SQL Server 2017. I started querying the status of the rowgroups on each table and I found something that appeared odd. I ran the following query for a specific table in PRIZM:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


select
 i.name,
 rg.object_id,
 rg.partition_number,
 rg.row_group_id,
 rg.delta_store_hobt_id,
 rg.state_desc,
 rg.total_rows,
 rg.deleted_rows,
 100.0*(ISNULL(rg.deleted_rows,0))/total_rows AS 'Fragmentation',
 rg.created_time,
 rg.closed_time
from sys.dm_db_column_store_row_group_physical_stats rg
inner join sys.indexes i
on rg.object_id = i.object_id
and rg.index_id = i.index_id
order by i.name

The results showed a lot of rowgroups in a TOMBSTONE state:

This was curious to me because, in my limited experience with clustered columnstore, these older closed rowgroups in a TOMBSTONE state should have been cleaned up a long time ago, well before we upgraded to SQL Server 2019.

On April 15th, I ran a test disabling trace flag 634. As soon as the trace flag was disabled, we experienced several things:

transaction log growth
a drop in the total number of TOMBSTONE rowgroups
and, a dramatic increase in query timeouts on the server

I left the trace flag disabled for several hours, all the while monitoring the 3 things above.

About mid-day, I re-enabled the trace flag on the server. When I did that, the logs stopped growing, the total number of TOMBSTONE rowgroups stopping decreasing, and the flood of query timeouts we saw throughout the entire test slowed to a trickle. All of this information was passed along to support. They suggested that the TUPLE_MOVER doesn’t touch the TOMBSTONE rowgroups and that should only be handled by the ghost cleanup process, but that was not the behavior we were seeing. They requested another test, but this time they also wanted us to keep track of the OPEN and CLOSED rowgroups.

The next day, it was time to try this again. Prior to disabling trace flag 634, we had the following rowgroup states in the PRIZM database:

Total	State
581	TOMBSTONE
672	OPEN
167	CLOSED

I left the trace flag off for about 2 hours. While it was disabled, the transaction logs grew 136GB, and we also experienced significant CXPACKET waits across the server which resulted in a lot of queries timing out.

The total number of TOMBSTONE row groups dropped during that time, eventually dropping to ~13 rowgroups. Once the TOMBSTONE totals bottomed out, the number of CLOSED rowgroups started to drop, and the total TOMBSTONE rowgroups started to increase again. When the test was complete, the rowgroup state totals were:

Total	State
48	TOMBSTONE
672	OPEN
131	CLOSED

I could only run the test for about 2 hours due to the amount of exceptions we saw. The curious thing is that when we enable the trace flag again, the number of queries that timeout on significantly decreases and the amount of CXPACKET waits we see also drops.

Next Steps

We’re still working with support on this, but we need to do one of a two choices:

Disable trace flag 634 and let it just finish whatever it’s doing while monitoring the rowgroup states. Maybe it would eventually stop aggressively increasing the transaction logs, we’re not sure. We have not done this yet because having this backlog allows us to test other fixes, but it also just fixes the immediate issue, it doesn’t prevent us from hitting this issue again later if we need to perform another large delete in PRIZM.
Manually rebuild the 11 tables that were never rebuilt. I tried to do this online several times now that we’re on SQL Server 2019, but it results in blocking of data being inserted to PRIZM, which is a shame since we were hoping to use that feature. In order for us to rebuild, we’ll need to disable any of the processes that use PRIZM first, then rebuild each table, so that will be done at a later date.

We have some other issues in progress with SQL Server 2019, so I’m juggling those issues and this one. I’m hoping for a resolution soon on this, but before I forgot any of the details I wanted to share them.

Final Thoughts

We still have a lot of outstanding questions on this, the biggest one being, why did the upgrade to SQL Server 2019 trigger what appears to be a massive cleanup of these clustered columnstore indexes? We didn’t have either trace flag enabled on SQL Server 2017, so there shouldn’t have been anything preventing the clean-up of the ghost records or the TOMBSTONE rowgroups, so why did this happen immediately after we failed over to 2019? Is there something different about it being an in-place upgrade?

If you have tested SQL Server 2019 and you use clustered columnstore indexes, have you seen this behavior? If so, what did you do to fix the issue? Did you just enable the trace flags and leave them on?

And yes we were eventually able to failover the reporting cluster…I know you were worried. :)

A follow-up on the next steps has been posted here.

A gotcha when upgrading to SQL Server 2019

Fri, 21 Feb 2020 06:00:38 +0000

In my last post, I mentioned that I started the process of upgrading Stack Overflow to SQL Server 2019. This week I tackled our first production servers and after upgrading, we hit a small issue aka a gotcha because we were using an old system view. Below is a recap of what I encountered.

A Little Background

The servers I upgraded were the three SQL Servers that run Stack Overflow for Teams. For those not familiar with our setup, we use availability groups (AGs) across all of our main SQL Servers. This allows us to use the primary server for read/writes, and use the secondaries for a lot of the read-only traffic. Since we utilize both the primary and secondaries for read purposes, logins need to be the same across all of the servers in the AGs. In order to keep the logins in sync, we have a job that periodically runs on each server and creates new logins using dynamic SQL.

Upgrade Day

Since I previously upgraded our servers to SQL Server 2017, I knew what to expect as I moved through the cluster. Once I started with the secondary servers the databases would not be accessible until the failover was complete. After upgrading a secondary, I would see this if I tried to access a database:

This also meant that any of the jobs touching the databases would fail, so I disabled SQL Agent jobs to minimize noisy alerts to my email. After the upgrade and failover, I’d enable all the jobs and be on my merry way.

The upgrade of both secondary servers went totally as planned, without any issues. The evening rolled around and despite Nick Craver’s popcorn, the failover was successful, and we finally had SQL Server 2019 running in production.

I upgraded the last server in the cluster (the former primary), and did some other clean-up tasks, including enabling all the SQL Agent jobs, before calling it a night. Once the jobs were enabled I started getting emails that one was failing — the Login Replication job.

Something’s broken - time to investigate.

The Failing Job

I pulled up the history on the job and saw the error:

Message

Executed as user: <username>. Invalid value given for parameter PASSWORD. Specify a valid parameter value. [SQLSTATE 42000] (Error 15021). The step failed.

Sigh ok, something is really broken because this was working before we failed over.

The code for the login replication basically does the following via a cursor (yeah, I know, but it works…normally):

Select from the primary via OPENQUERY to query the logins and passwords
Using sp_hexadecimal convert the varbinary password to a string value
Create a string to be executed, i.e. dynamic SQL that runs a CREATE LOGIN

I have trimmed the code because it’s long, but the key parts query the login and password.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


SELECT *
FROM OPENQUERY([SQL-AG], '
 SELECT p.sid sid_varbinary, p.name, p.default_database_name,
 CAST(l.password AS varbinary (256)) pwd_varbinary,
 CASE sl.is_policy_checked WHEN 1 THEN ''ON'' WHEN 0 THEN ''OFF'' ELSE NULL END is_policy_checked,
 CASE sl.is_expiration_checked WHEN 1 THEN ''ON'' WHEN 0 THEN ''OFF'' ELSE NULL END is_expiration_checked
 FROM sys.server_principals p
 JOIN sys.syslogins l ON l.name = p.name
 JOIN sys.sql_logins sl ON l.name = sl.name
 WHERE p.type = ''S''
 AND p.name <> ''sa''
 AND l.denylogin = 0
 AND l.hasaccess = 1
 AND p.is_disabled = 0
 ORDER BY p.name');

The value of pwd_varbinary and sid are then used with sp_hexadecimal:

1
2


EXEC sp_hexadecimal @PWD_varbinary, @PWD_string OUT
EXEC sp_hexadecimal @SID_varbinary, @SID_string OUT

Finally, we concatenate them into a SQL string that we execute:

1
2
3
4
5


SET @sqlString = ''CREATE LOGIN '' + QUOTENAME(@name)
 + '' WITH PASSWORD = '' + @PWD_string + '' HASHED, SID = '' + @SID_string
 + '', DEFAULT_DATABASE = ['' + @defaultdb + '']''
 + '', CHECK_POLICY = '' + @is_policy_checked
 + '', CHECK_EXPIRATION = '' + @is_expiration_checked

When I ran the SELECT statement, I noticed the issue — the value of pwd_varbinary was null which obviously was wrong, and since we were passing a null as the password to sp_hexadecimal the job was failing.

Great, so now what?

The Fix

We couldn’t just stop replicating the logins across all servers, due to our usage of the secondaries, and we needed to do this automatically on a regular interval. We had to figure out a solution. Thankfully the fix was easy.

After seeing that the null value of the password was coming from sys.syslogins, and since we were already querying sys.sql_logins we could easily replace:

1

CAST(l.password AS varbinary (256)) pwd_varbinary,

from sys.syslogins with

1

sl.password_hash pwd_varbinary,

from sys.sql_logins (h/t to Nick Craver). With this one change to the query, the new SELECT statement was:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


SELECT *
FROM OPENQUERY([SQL-AG], '
 SELECT p.sid sid_varbinary, p.name, p.default_database_name,
 sl.password_hash pwd_varbinary,
 CASE sl.is_policy_checked WHEN 1 THEN ''ON'' WHEN 0 THEN ''OFF'' ELSE NULL END is_policy_checked,
 CASE sl.is_expiration_checked WHEN 1 THEN ''ON'' WHEN 0 THEN ''OFF'' ELSE NULL END is_expiration_checked
 FROM sys.server_principals p
 JOIN sys.syslogins l ON l.name = p.name
 JOIN sys.sql_logins sl ON l.name = sl.name
 WHERE p.type = ''S''
 AND p.name <> ''sa''
 AND l.denylogin = 0
 AND l.hasaccess = 1
 AND p.is_disabled = 0
 ORDER BY p.name');

Once it was updated, the job successfully ran, and our logins were replicating again without issue. Now, I really called it a night.

Ok, but what was the gotcha?

Remember I said that this was working fine on SQL Server 2017?

When we started looking for solutions, I looked up the Microsoft Docs for sys.syslogins and right at the top of the doc it says:

This SQL Server 2000 system table is included as a view for backward compatibility. We recommend that you use the current SQL Server system views instead. To find the equivalent system view or views, see Mapping System Tables to System Views (Transact-SQL). This feature will be removed in a future version of Microsoft SQL Server. Avoid using this feature in new development work, and plan to modify applications that currently use this feature.

The doc shows the description and value of password:

Column	Data type	Description
password	nvarchar(128)	Returns NULL.

Hmm, that’s weird because the password column contains an actual value in SQL Server 2017. Something obviously changed. Yes, we were risky because we were using an old view, but the important thing is the value in the password column changed, and it bit us.

In SQL Server 2017, there is still a value for password in sys.syslogins, but in SQL Server 2019 it is now null.

If this was mentioned somewhere, I missed it. Thankfully, this wasn’t a critical job otherwise it could have been more problematic. We don’t have the Login Replication job running in development, so we didn’t hit the issue until we moved to production.

Technically it’s on us because we were using an old system view and didn’t check for changes, but keep in mind if you’re using the sys.syslogins view anywhere, and rely on the password, you’ll need to make a code update before moving to SQL Server 2019.

If you’ve upgraded to SQL Server 2019, have you hit any of these issues yet?

Recovering Lost Linked Servers

Thu, 06 Feb 2020 06:00:38 +0000

Recently, I kicked off a project to start moving us to SQL Server 2019. During my initial review of our servers, I found quite a few (9 total) that were still running on Windows Server 2012 R2. This meant that I would need to upgrade the operating system and move us to SQL Server 2019. Having completed plenty of SQL Server upgrades, as well as operating system upgrades, I couldn’t possibly make a mistake, right? Wrong…I completely forgot to script out the linked servers on the server I upgraded this week. I screwed up and decided to write about how I went about fixing it.

My Plan

This week, I tackled the server running Stack Exchange Data Explorer (SEDE). In the past, when I have upgraded an operating system on a SQL Server I work off a list:

Take a backup of all system databases - if I’m reinstalling the same version of SQL Server, I could restore the master, model, and msdb databases to minimize some of my setup again
Script out any objects in the master that we need, this would be tables and stored procedures. We store objects for maintenance purposes in master…please don’t scream at me.
Generate creation scripts for all jobs
Export logins and users from the server
Make sure I have everything else I need scripted before kicking off a rebuild

Upgrade Day

Before starting the rebuild, I did one final update of SQL Server 2017 to bring it to the latest patch we were using, CU 18. This was to make sure it was fully up-to-date before I rebuilt the server.

You might be curious why I patched SQL Server 2017 when we were going to SQL Server 2019? I did this because the server that runs data explorer has a lot of specific permissions, and it also has about 350 databases. My initial goal was to restore the system databases to minimize the risk of missing a permission or something in the shuffle when setting up the new server. Once the system databases were restored, I’d upgrade to SQL Server 2019.

After patching SQL Server, I checked all the steps above, and thought I had everything I needed to recreate the server, so I kicked off the rebuild. About 30 minutes into the process, I had a total facepalm moment and realized I forgot to script the linked servers. It was too late to do anything about it at that time, but thankfully I had the system databases which would save me. Once Windows Server 2016 was done installing, it was time to install SQL Server 2017 and then attempt the restore of the system databases.

I say attempt to restore the system databases because while I’ve restored the master database before, it’s not fun, or at least, not my definition of fun. The biggest pain point I have is getting the single connection to the server. I know how to put the server in single user mode to restore the master database, but at Stack Overflow we have a lot of monitoring systems that are actively checking the server status. Unless I disabled all those logins, it’d be nearly impossible to get that single connection.

I opened up a command window and started going through all the steps to restore the master database. I executed:

net stop MSSQLSERVER
net start MSSQLSERVER /m

Then, I opened another command window and connected to the instance by using:

sqlcmd -s .\MSSQLSERVER

When I hit go, I received the dreaded error:

Login failed for user <xxx> Reason: Server is in single user mode. Only one administrator can connect at this time.

Boo. I tried connecting a few more times and each time I couldn’t get the single connection. I disabled some logins for our monitoring services and couldn’t connect to the instance because something was getting the single connection first. I may also have been hitting my head on my desk out of frustration. After fighting with it for far longer than I’m going to admit, I gave up. I knew I was going to just have to rebuild the server from my scripts, and attempt to recreate the linked servers from scratch. Knowing it was inevitable, I went ahead and proceeded with the upgrade to SQL Server 2019. When that was done, I executed all my scripts, got the databases back in place, and released data explorer back to the world.

The only things still missing were the linked servers. While I was happy to have moved the SQL upgrades along, I was very angry with myself for missing that script in the first place. The server had not received much love, and I didn’t have the linked server, which was created long before me, scripted out and uploaded to GitHub (yeah, I know that was also dumb). I setup the linked servers by hand but couldn’t get them to connect. After a very long day I knew I wouldn’t need the linked server until the weekend, so I left it to be fixed the next morning.

Recovering the Lost Linked Servers

The next day, I had to get the linked servers working. My attempts to create them by hand failed miserably. I had the system backups from before the upgrade, so I wanted to try to get the details out of them. I did several things that didn’t work, including restoring the master database to a different server, under a different name (i.e. master_old) and trying to query the linked server info.

Finally, I decided to create a new test server, install SQL Server 2017, and restore the master database to it. I setup a new build process to create a VM with a single disk drive, with just enough space for Windows and SQL to be installed on it. I installed SQL Server 2017 to the same CU I needed, and it was time to try the restore again. My thinking was, I could pretty much do anything to this SQL instance in order to get to the linked server data I wanted.

Using the same commands as above, I set the instance to single user mode, and then attempted to connect to the instance. No error message this time. I was in the instance as the sole user…success. Now that I was connected it was time to restore:

1

RESTORE DATABASE master FROM DISK = 'C:\Backups\master.bak' WITH REPLACE

I received no errors. It looked like everything worked. I opened up the SQL Server Configuration Manager, started the SQL service, and refreshed the page. The SQL service stopped. Uh oh, something is wrong. I went to the error log and popped it open. There was a giant error:

2020-02-06 16:42:24.74 spid9s Starting up database ‘model’.

2020-02-06 16:42:24.74 spid9s Error: 17204, Severity: 16, State: 1.

2020-02-06 16:42:24.74 spid9s FCB::Open failed: Could not open file C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\DATA\model.mdf for file number 1. OS error: 3(The system cannot find the path specified.).

2020-02-06 16:42:24.74 spid9s Database ‘model’ cannot be opened due to inaccessible files or insufficient memory or disk space. See the SQL Server errorlog for details.

2020-02-06 16:42:24.74 spid9s SQL Server shutdown has been initiated

2020-02-06 16:42:24.74 spid9s SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.

I saw the issue right away. The server was looking for the model database in the MSSQL12.MSSQLSERVER directory, but that didn’t exist any longer. The old server had it’s files in the directory for SQL Server 2014 aka MSSQL12, but on this new server everything is located in the directory for MSSQL14.MSSQLSERVER. As I mentioned, I didn’t really care what I did to this test machine, so I manually created the directory that it was looking for - C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\DATA and made sure that the SQL user account had access to the directory. Then I just copied and pasted the model database from MSSQL14.MSSQLSERVER and dropped it into the MSSQL12.MSSQLSERVER directory.

I attempted to start the SQL service, and again it failed. Back to the error log, I went. This time the error was:

2020-02-06 16:48:42.11 spid9s Clearing tempdb database.

2020-02-06 16:48:42.11 spid9s Error: 5123, Severity: 16, State: 1.

2020-02-06 16:48:42.11 spid9s CREATE FILE encountered operating system error 3(The system cannot find the path specified.) while attempting to open or create the physical file ‘D:\Data\tempdb.mdf’.

2020-02-06 16:48:42.12 spid9s Could not create tempdb. You may not have enough disk space available. Free additional disk space by deleting other files on the tempdb drive and then restart SQL Server. Check for additional errors in the operating system error log that may indicate why the tempdb files could not be initialized.

2020-02-06 16:48:42.12 spid9s SQL Server shutdown has been initiated

2020-02-06 16:48:42.12 spid9s SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.

There was some progress because it was a different error, right?

Now it’s complaining because I don’t have a D:\ drive on my test instance. I built the server with a single drive, but luckily I can easily fix that for a VM. I logged into our VM platform, quickly added a tiny new disk drive, then formatted it on my test server. Once the D:\ drive was available, I created D:\Data and granted permission to the SQL user account to access the directory.

It was time to go back to the SQL Server Configuration Manager to start the service. As I kicked the service to start, I kept thinking ‘please work, please work’. This time the service started and stayed running. I logged into SSMS, connected to the instance, and went looking for my Linked Servers.

Guess what? They were there. Somehow it worked. I was able to script out both linked servers and execute them where they belong. After some testing, I confirmed that the linked servers were working on the original server I built.

It’s quite possible that there are other ways to recover something like this. I’m also relieved this worked and I know I’m lucky to have the tools to be able to quickly spin up a new test SQL Server to hack away on. I learned a valuable lesson - verify I have everything that I need when rebuilding servers.

Oh, and by the way, yes I did upload those linked server creation scripts to GitHub.

The Perils of Querying SQL Server Replicas Under Load

Wed, 15 Jan 2020 06:00:38 +0000

Last week at Stack Overflow we had an internal hack-a-thon, or as we call it, a make-a-thon. I was on the bug-bashing team, which is the team that attempts to fix smallish bugs we haven’t gotten around to fixing, due to other time-constraints. I was tagged to investigate a bug about duplicate badges being awarded because it looked to possibly be an easy fix in SQL. At first glance it looked simple enough, but once I started digging in, I figured out very quickly it wouldn’t be.

A Little Background

If you’re not familiar with badges on Stack Overflow, they are awarded for performing actions on the site. For example, the badge in the bug report, Suffrage is awarded for voting 30 times in a day. Some of our badges can be awarded multiple times while others are awarded only once. The Suffrage badge is supposed to be a one time badge, which is why it is odd that someone had it twice.

Initial Investigation

Based on the bug report, I knew three pieces of information: 1) the badge, 2) who it was awarded to and 3) when it happened. I turned to the database to see if I could get any more info.

My first step was to see if there were other cases where the Suffrage badge was awarded twice.

1
2
3
4


SELECT count(*)
FROM Users2Badges
WHERE BadgeId = 804 -- Suffrage
HAVING COUNT(*) > 1

Nope. Only one user was lucky enough to get it twice. Then I decided to see if anything was unusual about the day it was awarded. When I say unusual, I mean, did we have any outages, or, was there anything that could have glitched when the badges were granted?

The badge in question was awarded on June 22nd 2018, so you might be wondering how in the world would I be able to track down outages, etc. from 1.5 years ago?

I’m glad you asked. When we throw certain exceptions on the site, they get sent to our internal chatrooms. All I had to do was go back to the transcript for that specific day, and do a quick scan to see if anything jumped out at me. Lo and behold something did. A whole lot of badge grant exceptions:

A Bit More Background

Around that time, June 2018, we were feeling the crunch of low free space on our primary SQL Server SSDs. We had < 10% free, which wasn’t great, so we investigated what, if anything, we could do without having to buy new drives. During that research, we came across our Log table, which exists in all of our databases. These tables are used to capture messages from the application. Seems harmless, right? Normally, yes, but we had been logging data into these tables for years and never deleted a single row for most log types. The Log tables were storing a ridiculous number of rows across ALL OF OUR DATABASES. We were going to gain a ton of space back by purging this old data. Here is just one database as an example:

StackExchange.Tor:

Database Size: 10.3 GB

Log Table Size: 10.12 GB

Since we needed to purge data from these tables across the entire network, we used our scheduler to do it. The scheduler would allow us to do this in batches, in order to not slam the SQL Servers. The goal was to chew through the deletions, but not cause blocking or timeouts on any other process. It was a game of cat and mouse with the servers as we attempted to figure out the correct batch size. We basically were going to execute this everywhere:

1
2
3
4


DELETE Top (50000)
FROM Log
WHERE LogEntryType = 30
 AND CreationDate < GETUTCDATE() - 60

We tried a 50k batch size which resulted in timeouts, similar to the screenshot above. We tried 25k which worked. We upped it again to 40k, and it failed. Finally, we dropped it back down to 25k and just let it delete as needed. Any time the scheduled deletions threw exceptions, we had a record of it in our chatroom. These messages in chat were critical to see what was happening when the duplicate badge was granted.

After looking through the chat transcripts, I had a far better understanding of what was taking place when the badge was awarded incorrectly. I also was pretty sure I knew the cause of the duplicate awards, I just needed to prove it.

Digging Deeper

Based on what I saw in the transcripts, my gut was telling me the issue had to do with the massive influx to the transaction logs making the log_send_queue_size sky-rocket. This resulted in a much larger amount of data that needed to be written to the secondary, in other words the redo_queue_size was also extremely large (both values are available by querying sys.dm_hadr_database_replica_states). As a result, we were reading dirty data when we awarded the badges. We award badges in this manner:

Query the readable secondary to verify who should get the badge — we use the secondary to help spread out our read workload
Award the badge on the primary

We grant badges every 5 minutes, so if the secondary server was overloaded in trying to write a lot of transactions due to another process, we might be reading stale data on the next execution. I had one example of this from the bug report, I just needed a few more to prove that was the case. Time to go back to the database.

I wrote up an ugly little query to get a list of all duplicate badges awarded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


;WITH IncorrectAwards as
(
 SELECT ub.UserId, ub.BadgeId, Total = count(*)
 FROM Users2Badges ub
 WHERE EXISTS (SELECT 1
 FROM Badges b
 WHERE b.Single = 1
 AND ub.BadgeId = b.Id)
 GROUP BY ub.UserId, ub.BadgeId
 HAVING count(*) > 1
)
SELECT *
FROM Users2Badges ub
WHERE EXISTS (SELECT 1
 FROM IncorrectAwards ia
 WHERE ub.UserId = ia.UserId
 and ub.BadgeId = ia.BadgeId)

Using the results and the exceptions in our chatroom, I was able to confirm that we had other instances where a huge process hit the servers, overloaded the transaction logs, and resulted in duplicate badges. And what do you know I found a bunch of them!

I didn’t need to look very far back in time either. Back in November 2019, we performed a massive reputation recalculation which involved awarding reputation to question askers across the entire network. Let’s just say we overwhelmed our servers just a little bit during this time and the redo of the transaction logs in our Availability Groups and Distributed Availability Groups was incredibly slow. Since we were hammering the secondaries with a lot of transactions, we unfortunately were reading staler and staler data when we awarded badges which lead to other duplicates.

Fixing the Issue…Or Not

Now that I discovered the issue, it should be an easy fix, right? Well, sort of. Yeah, I could easily delete the duplicate badges and move on — which of course fixes the initial bug report, but doesn’t solve the underlying issue.

What happens the next time we run something massive on the primary and we overload the transaction logs, creating a delay in the data on the secondary? The same thing would most likely happen. The problem is, we need to use the secondary to check to see who should get the badge, we can’t move that back to the primary. Honestly, we’re not exactly sure how we’re going to fix it yet. We have some ideas like maybe checking the size of the redo queue before we execute the badge grants and if it’s some undetermined size threshold we skip the next badge grant, or some other fix we haven’t thought of yet.

Final Thoughts

You’re probably wondering why I wrote this post since we didn’t even fix the bug? Well, mainly because we use our readable secondary a lot and I’m sure we aren’t the only ones. While I knew we always needed to be mindful of the redo of the transaction logs to our secondaries in the availability groups, I need to be more vigilant in watching the impact to hopefully prevent a lot of clean-up later on.

By the way, since this initial bug was caused by purging the Log tables due to low free drive space, I figured I’d let you know that purging all that old unneeded data brought us from 90%+ used to <50% used. It triggered some duplicate badges, but it saved us lots of money in buying new SSDs and has given us a few more years of breathing room.

Also if you’ve hit any issues like this and have suggestions on how to fix it, let me know. I’m super curious.

T-SQL Tuesday #122 - Impostor Syndrome

Tue, 14 Jan 2020 06:00:38 +0000

This month’s T-SQL Tuesday is hosted by Jon Shaulis, who has asked that we talk about impostor syndrome. This subject is close to my heart, as I have dealt with it throughout my career.

I didn’t study computer science or engineering in college. I never even took a computer class. I sort of just fell into working as a web developer because I had someone tell me they wouldn’t hire me due to lack of experience.

Early in my career, I was a web developer, an MS Access developer, and a C#/WinForms Software Engineer, and I never knew what I was doing in any of those roles. For all of them, I used books, blogs, and Stack Overflow to learn what I needed, in order to complete some of my day to day tasks. I did whatever it took to get the job done, and learned everything I could in the process. Each day I hoped people wouldn’t realize I didn’t really know what I was doing.

Eventually, I found my niche. I loved working with databases and writing SQL. I worked as a Database Developer/DBA for quite a few years and then took a break from it, but always knew I’d go back. After a two year hiatus, I went back as the DBA at Stack Overflow.

Once I returned as a DBA, I struggled daily with impostor syndrome. Imagine, after a two year break from SQL Server, returning to work on the servers that run one of the most visited websites in the world. And on top of that, working with people who know far more about our servers and SQL than I ever did. I completely felt out of my element and struggled significantly with impostor syndrome.

To be honest, I still struggle with it. I work with incredibly smart people. Far smarter than I am. I suggest things that are shot down. I propose both good and bad ideas that are sometimes accepted, and sometimes not.

How do I deal with it?

I continue to suggest things that should be changed. They may be right or wrong, but I still do it. We discuss the pros and cons of the suggestions, and each time I learn from it.

Additionally, I’m trying to write more. I know there are things people can learn from me. I’m not the most confident in my writing (insert more impostor syndrome), but I’m working to overcome that by contributing more to the SQL community where I can. I fully expect I’ll make mistakes along the way, but that’s the only way I’ll be able to learn and get past my impostor syndrome.

Hitting Parallel_Redo_Flow_Control waits with Availability Groups

Mon, 09 Dec 2019 06:00:38 +0000

In late June 2019, June 26th to be exact, we experienced an outage on Stack Overflow for about 11 minutes. It’s not unusual that we had an outage. They happen. Not often, but they do still happen. This one, however, was a little different because it was caused by a maintenance job that was running on our primary SQL Server for Stack Overflow.

The job that caused it was something I’d noticed about a month prior, but had stopped it before an actual outage occurred. Once corrected, I didn’t see any sign of it again, so I figured we were fine. I was wrong. This time around, the job triggered the same issue and unfortunately, took us down for longer than I would like.

You’re probably thinking — wait a minute, a maintenance job triggered an outage? What type of job did that?

Hang On, Please Explain Yourself

While I can’t be 100% sure of the trigger, I’m 99.9% sure, because the job was running before the outage, so the timing is right. After looking through our monitoring logs, everything pointed to the job being the cause, so yes, I’m confident it caused it.

We don’t have regular maintenance windows for any of our servers, so we run jobs throughout the week, and if possible, try to schedule them during low-usage times. In this case, the job was an index maintenance job.

Now, before you scream at me about running an index maintenance job, I’m not going to argue the pros and cons of using it or whether or not we should run it — we can do that at another time. For this post, just accept the fact that we were running a job to rebuild/reorganize indexes.

Some Background

We run SQL Server 2017 Enterprise Edition which gives us the ability to rebuild indexes online, but we never had a scheduled job for it. If an index needed to be rebuilt, we’d manually do it. Ideally, we would automate this and target the indexes suffering from the most fragmentation, rebuild it, and move on to the next one.

After I finished the OS upgrade of our main servers we were in a better place to try running an index optimize job on a regular basis. We waited until after the upgrade from Windows Server 2012 because of the improved throughput we would get with our secondaries. Once the upgrades were done, we knew we could try to run the job more often. (Side note: why we waited until after we did the upgrade to run the index job is a story for another time, but let’s just say it caused a different outage from running out of disk space.)

Using most of Ola Hallengren’s maintenance solution across our servers, I set up the Index Optimize job to look for specific fragmentation levels, and depending on the server, I spaced out the execution frequency. I needed to give enough time in between executions for the logs to transmit to our secondaries. On some servers, I scheduled the job to run hourly, and on others it was scheduled to run every two hours.

In addition to frequency, I also had to be mindful of how long they ran. For example, running the job on the server that holds the Stack Exchange network of databases (~375 databases), would take hours to run through all the indexes to be defragmented. For that server, the job was scheduled to look for indexes with X% fragmentation and would run for 5-10 minutes. This spacing prevented overloading the transaction logs in between log backups, and would help minimize what needed to be transported to each of our secondaries.

My thinking was, we’d be far more efficient biting off smaller chunks of changes 5-10 minutes at a time, versus running it for many, many hours, and generating tons of log data that needed to be pushed to the secondaries. Doing this in bits also allowed our remote secondaries in Colorado to keep up with the transaction log backup. If I ran the job for too long, Colorado would fall further and further behind while trying to write the transaction log, so it made far more sense to do this a little at a time.

Enough of the Background, How Did you Break Production?

About a month before the outage, I caught the issue and stopped it before we got to the level of the outage…well, that didn’t happen this time.

I’m going to take another step back and give a bit more background.

As mentioned in some of my other posts, we use SQL Server’s Always On Availability Groups. Most of our clusters have 3 servers - one primary, one local secondary, and one remote secondary. In our primary datacenter - we have two servers - the primary receives write and read traffic, while our local secondary receives a lot of the read-only traffic.

We offload a lot to our readable secondary (at some point, if there’s interest, I’ll provide an overview of what we do). The secondary, in both of our main clusters, is used for many of our scheduled jobs and many of these scheduled jobs run hourly from across our web tier.

Outage Timeline

The timeline of the events will give a bit of insight into what exactly happened when we went offline. Using our monitoring tools, I was able to determine the timeline as:

The index maintenance job ran on the primary server
The job rebuilt/reorganized an index against a specific table
The index rebuild was in progress of being replayed to the local secondary - the NY secondary server
As the change to the index was in the process of being written to the secondary, the scheduler triggered a job which needed to query the secondary and the table where the log changes being written to
The scheduler hit blocking when attempting to query the table on the secondary server because the index rebuild was still writing to it
The replay of the index rebuild was running just as the scheduler attempting to query the same table, resulting in a significant number of sessions being blocked
The secondary was overloaded with queries attempting to query the table (both the scheduled queries and normal traffic against that table) and we basically fell over due to blocked sessions resulting in connection pool exhaustion upstream

The outage was the result of a domino effect, but the starting domino was the index maintenance job.

The size of the index, and the criticality of the table also added to the issue, but the redo to the secondary while we were trying to query it pushed us over the edge and the server we couldn’t keep up. This resulted in hundreds of sessions blocked. Since nothing could query the required tables, we went offline.

As soon as we started to hit the blocking sessions, I received an alert from our instance of Sentry One SQL Sentry that there was blocking on the server. I quickly opened the email and saw that we were hitting PARALLEL_REDO_FLOW_CONTROL waits. The Microsoft Docs explain that this wait

Occurs when the main redo thread cannot dispatch more transaction log records when log cache array for dispatched transaction log is full.

Indicates that one or more parallel redo worker threads cannot keep up with main redo thread transaction log dispatching speed or are blocked by some resources such as other type of waits. When this wait occurs frequently, parallel redo worker threads does not function efficiently

It makes sense that this was our issue. We were overloading the server with the underlying change to the index. The transaction log was very large during this time, and was trying to write to the secondary server. At this exact same moment, we also needed to query that table for scheduled jobs. Since the table was unavailable for use, the application could not continue and we went offline.

Since I saw the behavior in the previous month, I knew what to do. I quickly connected to the secondary server and ran Adam Machanic’s sp_whoisactive to see all the sessions attempting to query, and see what was being blocked. Once I had that list, it was time to kill all the sessions. I needed to give a bit of time for the index rebuild to write to the server, and by killing the sessions it was just enough to let things settle down. Unfortunately, we were down for about 11 minutes, but between the monitoring from Sentry One and seeing the issue pop up the month before, I knew that I would give us some breathing room by killing spids. Once the sessions were killed (helping unblock application connection pools as well), the log was able to write to the server and we came back up.

While this explains how we fixed the outage, that’s not the point of this post.

Then, What is the Point of This?

If you run Always On Availability Groups, and you use a readable secondary, you need to be mindful of the issues with PARALLEL_REDO_FLOW_CONTROL. These waits resulted in enough blocking to take us offline.

Yes, I’m aware that there is a trace flag 3459 that we can turn on, but we don’t have that enabled for our Stack Overflow SQL Server. Most of the time, it doesn’t make sense for us to disable parallelism (due to scale), so we don’t.

In order to prevent this from happening again during the index maintenance, I moved the impacted database to a separate index maintenance job that runs only on the weekend, when we have lower traffic and less opportunities to collide with other jobs. Once I moved that database to a separate job, we stopped hitting blocking issues with PARALLEL_REDO_FLOW_CONTROL and that job.

This solution works for us, but your situation might be different. If you run Always On Availability Groups, and you utilize a readable secondary, be mindful of those PARALLEL_REDO_FLOW_CONTROL waits, since they can result in significant blocking for an application reading from the secondary. In our case, adjusting when we ran our job helped especially since we don’t use the Trace Flag 3459, but that trace flag might work for you. My recommendation would be to test different situations with your AGs and maintenance jobs to see what works.

Have you hit this same issue? Did you fix it using the trace flag or some other way?

Finally, My first PASS Summit

Mon, 02 Dec 2019 08:00:00 +0000

I’ve been working with SQL Server for a long time, and have always wanted to attend PASS. For one reason of another, with numerous employers, I hadn’t been able to go. The answer was always no. <insert sad face>

The biggest blocker was always the cost.

The conference is expensive.

Seattle is expensive.

It has always been impossible to get my employer to foot the bill for it.

This year I was incredibly lucky that my employer, Stack Overflow, paid for my trip to PASS Summit. I was thrilled to finally be able to attend. Since I was given the opportunity to go, I wanted to write out my thoughts as a PASS “First-Timer”, as well as some session highlights.

A bit about me

I’m an introvert. Walking into a place…by myself, where I know no one is incredibly difficult. It triggers lots of anxiety which makes the idea of attending a conference with thousands of people who I don’t know, insanely hard. Sure, I know people at PASS Summit because I have seen their faces online from following them on Twitter, and have interacted with them virtually, but I have never met most of them in person. In order to meet them, I need to actually meet them…like talk to them…which is a struggle. I was going to try my best to work past that and take in everything I could.

Scheduling Overload

The week before PASS, I sat down and filled out my schedule which was a bit challenging. There were so many sessions that I wanted to attend. My final schedule had a ton of sessions at the same time. I did this in the event I would change my mind last minute, so I’d have an alternate option. I knew I was overplanning, and was being overly ambitious, but I was going to push myself to hit as many sessions as I could.

First Timer Buddy Program

I saw mentions of the Buddy Program and wasn’t sure if I was going to sign up, but I’m glad I did. For those who don’t know, the Buddy program at PASS Summit pairs you with someone who has been to the conference before, and they offer suggestions on events and sessions to attend. If you’re lucky and if time allows, you’ll meet them and have a familiar face in the thousands of people. I was lucky to get paired with Matt Gordon. Matt reached out a couple of weeks before the Summit and suggested a couple of sessions, then the week of the conference we met and had lunch together as a group. It was great to meet other first-timers in a smaller setting.

WIT Reception

I signed up for the WIT Reception which took place on the Tuesday night before the official start of the conference. I loved this event. It was smaller than many of the other events at the conference, and it was a great way to meet people before the conference officially started, and gave me a chance to have some familiar faces to find in the crowds of people the rest of the week. Not to mention, I met some really awesome people at the event.

The WIT happy hour overlapped with the First Timers Welcome Reception. Since I was comfortable at the WIT event, I did not leave to attend the First Timer Event. Was that the right decision? I don’t know. I met some great people at the WIT event and didn’t want to leave, however, I know that if I went to the First Timer Event, I probably would have met other newbies, which might have been beneficial.

Day One

Performance Tuning Azure SQL Databases with David Maxwell

At this time, we only use physical SQL Servers in our datacenters. As a result, I don’t have a lot of hands-on experience with Azure SQL, so I figured I would take the opportunity to learn a bit while at PASS. I started with a how to performance tune Azure SQL databases. This session was really good to hear about some of the differences in on-prem and Azure SQL and how you tackle performance issues using Extended Events and Automating Tuning.

Improving Availability in SQL Server and Azure SQL Database with Accelerated Database Recovery and Resumable Operations with Pam Lahoud

This session showed off one of the new features of SQL Server 2019 that I’m excited to test in our environments, Accelerated Database Recovery. It allows the databases to recover faster from long-running transactions that were aborted and are rolling back.

Prior to SQL Server 2019, you might have to wait hours for a rollback of transaction on a long-running query before the server would recover, all the while you could have queries blocked, etc. With the new release, if there is a long running transaction that gets killed, then the database will be unblocked immediately. This is a cool new feature of SQL Server 2019 that has some overhead, but I’m excited to test it out.

The session also covered Resumable Index Operations which allows an index rebuild operation to be paused and then resumed or aborted at another time. Again, a cool feature I’m excited to play with.

Day Two

Keynote with Tarah Wheeler - Cybersecurity is Everyone’s Problem

There is no doubt this was one of my favorite sessions of the conference. Tarah Wheeler is an expert in cybersecurity and an author. She talked about the conflict between cybersecurity and data science - the struggle between keeping or purging data in the age of three internets - the European Union (EU and GDPR), China, and then the rest of the world. Since data is constantly growing, how do you satisfy the need to delete data under different regulatory requirements. The problem is huge and is going to keep compounding as we continue to store data. The question she posed in this slide captures how complicated the situation is.

The Keynote was incredible on every level and hands down, it was the best thing I attended all week.

Columnstore Indexes in 2017-2019 with Niko Neugebauer

I have read many of Niko’s blog posts on Clustered Columnstore, so getting the opportunity to see him talk about them in person was a no brainer.

Within just a few minutes, I thought, “hey that might be our problem.” The biggest lightbulb moment in this session was how dictionary size and memory pressure can break row groups in ways that are potentially unexpected. As soon as I heard this, I knew what I was going to investigate it when I was back in the office. This session was great because I had takeaways that I could possibly use or research immediately.

Azure SQL Database - Lessons Learned from the Trenches with Jose Manuel Jurado Diaz and Roberto Cavalcanti

I also loved this session. The speakers went through different scenarios that customers have hit while using Azure SQL Databases.

Seeing different tips and tricks to look for when performance tuning in that environment gave me a lot to think about for future work in the cloud. Things as simple as an implicit conversion can perform significantly different on Azure SQL Database once you hit an increase in data that an old cached plan can no longer handle.

Improving Columnstore Load Scalability on Large Servers with Joe Obbish

I had heard there were 180 slides for this session and was sure my head would be spinning from the amount of information. Instead of being overwhelmed by it, Joe did a great job of going over how to optimize loading millions of rows of data into a clustered columnstore table in just 6 minutes.

He went through each step in his optimization process, explaining what he looked for to tune in the next step. With each attempt the processing time dropped considerably.

Besides watching how to performance tune huge data loads into clustered columnstore tables, I learned several bits that I could take back and see about implementing in our environment. This was another top session for the week.

Day Three

Batch Execution Mode on Rowstore Indexes with Niko Neugebauer

This session included some great demos on the new feature in SQL Server 2019, Batch Execution Mode on traditional rowstore indexes. I’m not sure how much benefit we will get with this in our environment, but I’m curious to get SQL Server 2019 installed on more servers to see if we do.

Deep Dive into Blocking and Deadlocks Troubleshooting with Dmitri Korotkevitch

Blocking and Deadlocks every DBAs nemesis. At our scale, I fight with them a lot, so having the chance to attend a session about troubleshooting was a must. Even though I have experience with dealing with both blocking and deadlocks, it was a great overview on why they occur, and how best to troubleshoot them. This was a solid session to attend.

Final Thoughts

The week flew by. It was information overload in each session that I attended, but I’m so happy I finally had the chance to go. Even though I was out of my comfort zone, I met a ton of people and learned a lot while I was there.

I hope I’ll be able to attend again in the future…maybe even next year in Houston.

How Stack Overflow upgraded from Windows Server 2012

Thu, 18 Jul 2019 06:00:38 +0000

Warning: This post is long. While working through this massive server upgrade/migration process, tears were shed, many cuss words were said, along with a general feeling of frustration, which ultimately culminated into extreme happiness once the migration was completed. The scale and complexity of the implementation factor into the length of this post, and I’ll share my thought process on how this was executed, so here goes.

Last year, when we upgraded to SQL Server 2017 we didn’t make any changes to the operating system on our main production servers. They were Windows Server 2012 (not R2), and we knew that moving to another operating system would be painful because it would involve tearing down each of the clusters, rebuilding them, and potentially having extended downtime - which we really can’t have. It sounded too difficult at the time, so it was punted…again.

When I was mapping out projects for 2019, at the top of my list was to move from Windows Server 2012 and move to Windows Server 2019, because ‘hey, it’s 2019 let’s move away from 7 year old OS’. From the start, it was obvious this would be extremely complicated, but why not start the new year off with a crazy project that gets us in a position to move to SQL Server 2019. I had never done anything like this before, but in January, I started working out how we were going to upgrade our existing production SQL Servers from Windows 2012. And in this very long post I explain all my planning, testing, unexpected issues, and implementation of this move.

Reasoning

My first step was to identify the benefits of migrating. I would be spending a considerable amount of time on the project, so it was important to know what we would gain from the upgrade. There were two obvious wins 1) moving away from a 7 year old operating system, and 2) it also would allow us to move to SQL Server 2019 (yay… more upgrades). However, the biggest benefit we were hoping for, was an improvement with our availability group log transport throughput. Our current production clusters each have 3 nodes - 2 (primary and local secondary) in New York (actually New Jersey) and 1 (remote secondary) in Colorado - and we tend to see significant delays and non-synchronizing databases in Colorado. Upgrading from 2012 to Windows 2016+ would get us gains in performance, and hopefully reduce some of our syncing issues. For me, that would be a huge plus, as I wanted to avoid seeing stuff like this repeatedly during the week:

Phase 1: Lab Testing…All The Things

I’m incredibly lucky to have a lab environment to play with. At the start of this project, I had two lab Windows Server Failover Clusters (WSFC) - each with 2 nodes that were running Windows Server 2016, SQL Server 2017, and each cluster had availability groups (AG), as well as a distributed availability group (DAG) between the two clusters. Since I didn’t want to destroy these clusters for the test, I needed to create new servers for testing.

My goal was to replicate our production set-up on a much smaller scale. From there, I’d work through different scenarios to get to the final desired result. The 2012 production clusters looked like this:

Production had two WSFCs running Windows Server 2012, each with 3 nodes. Both clusters have at least one availability group, as well as one distributed availability group going to a reporting cluster. Once this project was complete, the new clusters would look the same, however, these would have a new OS, new cluster names, and when finished, the primary SQL Server in each availability group would be the current NY Secondary.

In order to properly test, I needed 3 servers running Windows Server 2012. Well, guess what? We didn’t even have a way to install Windows Server 2012, and no longer had an image of the software. This left me hunting for a copy of it. Eventually I got one, but then our deployment process needed to be setup to work with 7 year old software. Once all those bits were in place, I was able to spin up my 3 servers to test with. At this point, I had a new 2012 cluster with 3 nodes (2 in NY, 1 in CO). All were running SQL Server 2017 with 2 availability groups, one AG that was limited to this cluster, and a second AG that was modeled after one in a distributed availability group.

The Will This Even Work Test

Before I started breaking this new test cluster, we had the idea of creating another server with Windows Server 2019 to see if it would work with the new lab cluster - basically, a test to see if the data would sync. I spun up another new server; this time it was on a fancy new operating system, Windows Server 2019, with SQL Server 2017 and was all ready to start testing. The goal was to insert the 2019 server into the mix with 2012, so it would receive data from the old server cluster. I wanted it to look like this:

Considering I had no idea what I was doing, I tried many things to get this to work. I even tried things I knew wouldn’t work just to cross it off the list of things tested. Here’s a brief list of some of the things I tried:

Putting the 2019 server into the existing 2012 cluster - as expected, this fails due to the operating systems being different
Attempted to just add it to the existing AGs when not being in a cluster - this fails because it’s not in a cluster
Created a separate single node cluster for the 2019 server and attempted to add as a replica to the AGs - this fails as well

None of these things worked. I was beginning to wonder how we were going to do this. My next test was to create a new distributed availability group for each existing availability group, using that as a way to insert the 2019 server into the mix. I finally hit on something that worked for the single server. After creating a new DAG between the 2012 and 2019 cluster, I had data syncing between two clusters on different operating systems. I was ecstatic to get this to work with one server, but how would I do this with the 3 servers in a single cluster, with all the AGs and distributed AGs already in play?

The Mock-up Production Test

Once I knew I could have clusters with different operating systems, synchronizing data via distributed AGs, I needed to attempt this with the new 2012 lab cluster. Since it was already working as a stand-alone cluster with SQL Server running, I wanted to set up a test that was as close as possible to what we have in production. I needed a replica of our reporting cluster. A bell went off in my head, ‘Ding! Ding!’ I had an existing 2016 cluster in my original lab environment - that would be perfect to use for this.

My plan of attack for this test was to first setup a DAG between the 2012 cluster and the original lab cluster running on 2016. This would be similar to what we had in production at the time. Basically we’d have the following:

When that was setup and synchronizing, I had a decent, albeit, small version of our production setup. It was now time to start breaking things. My thinking was to start with the NY secondary server and perform the following steps:

Evict it from the existing 2012 cluster
Rebuild it with the Windows Server 2019
Create a new WSFC with one node
Install SQL Server 2017
Finally, create new distributed AGs from the old cluster to the new one to sync the databases with that AG

My next step was to do the same thing with the CO secondary in the 2012 cluster. The difference being, it could just be added as a node to the new WSFC, and as a replica to the AGs in the new cluster. At this point in the process, I’d have the old 2012 cluster with a single server sending data to two clusters - the mocked up 2016 reporting cluster and the new 2019 cluster. Visually it’d look like this:

Before upgrading the last 2012 server, I would need to perform a failover of the distributed AGs from the 2012 cluster to the new 2019 cluster. In looking at this, there was one glaring problem…the reporting cluster. If I performed a failover of the distributed AGs to the new 2019 cluster, the reporting cluster would stop getting data. I saw there were two options:

Perform the failover and let the reporting cluster fall out of sync until I could get it everything back in place
Move the reporting cluster and it’s distributed AGs to the receive data before I failed over to the new 2019 cluster and “hope” things just starting syncing again.

Either way, the databases on the reporting cluster were going to fall out of sync, so I chose the first option for the lab.

Now that the decision was made, it was easy peasy to finish the move in the lab. There was only one server left in the old cluster, so my steps were to failover the distributed AGs to the new 2019 cluster (yes, I tested failing back just in case), destroy the 2012 cluster, rebuild the server with Windows 2019, add it to the 2019 WSFC, install SQL Server, and add it as a replica to all the AGs. Yay, everything was done! What’s left was just a bit of clean-up of the reporting cluster distributed AGs, and then I was ready to move to production with the plan.

I spent the next week writing up all the steps to move to production. There were a lot of moving pieces for production. I had to move the following:

2 WSFC
6 servers with new OS, fresh SQL Server installs
5 availability groups with a total of about 385 databases
5 AGs means 5 temporary distributed AGs to help with the move

This also meant we needed new IP addresses for the clusters, AG listeners, as well as new names for the clusters, AGs, distributed AGs, and listeners. During my testing I discovered that you can’t use the same names for these objects, even when they’re on different servers, which resulted in a lot of legwork to get prepped for the move to production, but I was ready…or so I thought.

Phase 2: The Part Where I Broke Dev

Pretty much the entire month of January, I tested and worked through how this would be done in production. I originally targeted February to start the production servers. I was wrong.

During the final stages of review, a suggestion was made to test an upgrade to Windows Server 2019 on a few development servers, to ensure we wouldn’t hit any hiccups with the OS when it went to production. Our main development servers for Core Q&A have the same hardware and set-up as production, but don’t run any AGs - the servers only have copies of the databases for development. In testing the OS against these servers, we’d be able to perform some load testing, and I could make sure our deployment process of the OS would work.

I picked three servers to test against - two VMs and one physical server. We use Foreman to automatically (re)build servers, which made the process relatively painless to deploy and rebuild servers. Since Windows Server 2019 had never been deployed, it wasn’t setup in Foreman. This meant I had to upgrade the servers by hand, as an in-place upgrade - this was done on the two VMs. Aside from some minor issues, everything went according to plan. The OS deployed just fine, and I easily reinstalled SQL Server across the board. The servers were back up and running within a few hours.

Next, it was time to deploy Windows Server 2019 to a physical server. That’s when everything went off the rails.

As I mentioned, our main dev servers have similar hardware to our production servers, which means we have a drive for the OS, a data drive for SQL with NVMe/PCIe, and possibly a third drive with spinny disks. The physical server I tested the upgrade against had all three drives. I kicked off the rebuild, and about two hours into the process the dreaded blue screen of death hit:

Very quickly, we spotted the issue. The NVMe drivers didn’t work for Windows Server 2019. Oh crap. Thankfully, I have awesome teammates who jumped into a hangout and helped debug. After discussing, we attempted to upgrade the driver to the new one, and, of course, that bombed. Then one of the NVMe drives looked like it failed. After some poking, the “failed” drive came back online and we were able to get the server back to its 2016 state pre-upgrade attempt.

Now what?

We still wanted to move to 2019, but the new driver wasn’t working with our NVMe RAID, which meant time to contact both Dell and Intel. With their help, we might be able to get the correct drivers and software to work with our drives and the NVMe RAID. Eventually, we got the necessary bits and had the new drivers in place with software that recognized our RAID. Next, it was time to test another 2019 deployment. After two hours into rebuild #2, I was staring at a 45% completion:

I tried not to worry and went on to other tasks. At hour 6 of being stuck at 45%, I was concerned. Even though this deployment hadn’t finished, it was obviously stuck, so we decided to attempt another fresh rebuild…#3.

When rebuild 3 began, everything seemed to be going ok. Once up and rebooted, we were back to a 2016 Windows Server. WTH?!?! This time we also had a freshly formatted 2TB NVMe partition…meaning all of our SQL data on the NVMe drives was gone. We decided to try once more, this time disabling the PCIe slots in the BIOS and trying another fresh rebuild - this time #4.

Our confidence level in moving to Windows Server 2019 was quickly dropping, but we wanted to see if we could get it to work. Our 4th rebuild, finally got 2019 installed on the server, but we hit another issue. When the PCIe slots were re-enabled, we only had 2 disks show up in the RSTe manager and in disk management on the server, but everything was showing in the Device Manager.

Ugh, why are they not matching? There were so many weird problems with this server.

The next step was to send someone to the data center to physically unplug & drain the power from the box to see if the issues with the PCIe slots would sort itself out - we have done this in the past and it worked. Unfortunately, our trick of draining power didn’t work. We still only had two disks show up…sigh. That means time for another rebuild.

This time we were going to try to rebuild the server to 2016 with the old drivers for the SSDs to see if we could at least get it back to the state it was before the upgrade to 2019. The hope was that if we rolled back the driver, the PCIe slots might come back to life and the drives would miraculously work again. Off to rebuild #5.

After fighting with the server to use the older drivers, I finally got them installed and kicked off another rebuild. Unfortunately, this time we had a failure because the deployment failed to join the server to the domain and we couldn’t login to the server…seriously, the server was possessed.

Yay, another rebuild (#6) to try to get it on the domain, with Windows Server 2016, and old drivers so the SSDs work. Guess what? This failed again, due to issues joining the domain. Again, my teammates rescued me and got into a hangout to beat the server into submission and get it joined to the network. Once it was finally joined to the domain, there was another freaking issue.

It was getting ridiculous with this server. I found a way to hack it back into submission because at this point anything was worth trying. The OS finally installed, but we had two very dead SSDs. They were showing up as 0GB. We tried upgrading firmware…nothing worked. The drives were dead. It was time to open a ticket with Intel since the drives were still under warranty. It was now a waiting game, until Intel confirmed the drives were fried.

While we were waiting for new drives from Intel, my teammate, Mark Henderson, spent time fixing up our Foreman deployment to work with 2019 and to bypass the NVMe drives when installing an OS. You know what that means…more rebuilds. Actually 3 more - one to 2016, one to 2019, and then one back to 2016 for a total of 9 rebuilds of Windows (so far) on a single server.

You might be wondering why we went back to 2016 after we finally successfully installed 2019?

This was done because our confidence level in using 2019 with our SSDs was pretty much gone. Intel no longer makes the drivers public, and referred us to Dell, and then Dell sent us back to Intel - it was a back and forth game - which wasn’t a good position to be in for production. Combined with multiple failures and dead SSDs, we didn’t feel comfortable moving forward with Windows Server 2019 on our production servers. We couldn’t risk killing our SSDs due to the install. We knew that 2016 worked with them because Dev had been using it for years, and we’d get the benefits of moving off of Windows Server 2012, so we made the decision to skip Windows Server 2019 at this time.

About 2 weeks later, we finally had new SSDs. It was now time to get them installed and test the deployment process again. Here we go with rebuild #10. Time to hopefully get a clean install of Windows Server 2016 - I spoke too soon. The rebuild failed. No, I’m really not joking.

On top of using Foreman, we use Puppet to get our servers in a desired state - the failure this time appeared to be due to a Puppet issue, but there was, of course, nothing in the error log that pointed to the issue. We had yet another failure, and we wanted one clean install on this server, meaning no issues whatsoever. That meant another rebuild to see if we could get an install with zero errors. This server wasn’t quite ready to be nice, we hit various issues with the install process and it took 4 more times before we had a clean build of the server for a total of 14 times!! After all the issues, we finally had a dev server almost back to its original state - Windows Server 2016, now it just needed SQL Server and that’s super easy to install, right?

I moved on to installing SQL Server 2017 again, and since this server was terrible on every level we ran into more issues. I have a couple of PowerShell scripts that I use to install SQL Server which work all the time, except on this server. There were errors installing the SqlServer module, then there were really odd issues with SPNs for the server not being tied to the proper account and the server was associated to me - it really was cursed. After a ton of more issues (you really don’t want details on everything), we finally had a dev server back in place. All databases were restored, and things were working again.

Now what? Well, we had another dev server to test this against, this time our NY dev server. We decided to try a clean install against that server, but first we spent about a week trying to fix all of the weird issues we were hitting with the CO dev server…yes, that means more rebuilds. Once everything was working without issue, it was time to move to NY Dev. Our NY Dev server is critical path for pushes to production, so that rebuild involved a bit of juggling, but eventually I was able to rebuild it and yay, there were no issues with SSDs or really anything else that would be a blocker for moving to production.

By the time I finally got done with NY Dev, it was about 2 weeks into April, that means it was over two months of working through all the issues with the development deploys. I was totally ready to move this project along and touch production.

Phase 3: Time For Production

It was now into April, and I thought this was going to be finished in February, so you could say I was extremely anxious to finish this project and move on to other things. Since our development environment was finally done and it appeared that our deployment process was working, I revisited the plan I wrote in January for our production environment.

I opened the Google Doc that I initially wrote and started expanding it into a very detailed plan on the deployment process. I spent several days writing out each step for every server we would be moving around. I tried to include everything I could think of. The step for each server even had the code that would be executed on it. This was probably the most detailed plan I had ever written, I attempted to cover everything I would think of. Nick Craver, then added more details because the connection strings for each application would be changing due to moving to new listeners.

I was ready to go. We were ready to go. The runbook had about 35 pages of notes and code to execute in the entire plan.

Every server had a detailed list of steps including obvious things to more complicated pieces. Inside of the document, you could look up any server and find a list similar to this:

Server Name (Date)

Stop Transaction Log backups on primary
Remove read-only routing from the AGs (this is done on the primary) - including code snippet to run on the primary
Flush connections in the app
Disable full backups on primary
Remove server from the existing AGs on primary
Evict server from current cluster
Setup server in Foreman for 2016 deployment, kick off the rebuild
Make sure puppet is up and running after the rebuild
Once done rebuilding, add Failover Cluster role to the server (this forces a reboot)
Create new WSFC (only on the first server in the cluster)
Once the Cluster Object is created in AD, then move it, if needed, to the proper Cluster Objects OU
After cluster creation, verify the IP addresses are correct and edit them, as needed
Make sure the WFSC cluster object has permissions to create/delete computer objects in AD
Install Dell NVMe tools
Install SQL Server – if the tempdb is still located on D:\Data - delete all the files first
Turn on trace flags – restart SQL service
Execute scripts to add logins / users / stored procs / linked server
Create New AGs
For each AG create new temporary distributed AG aka the TAG (our temporary distributed availability group for the migration) - created on the current primary and then the secondary
At this point everything should be syncing to the newly built server, and it’s a single server on its own cluster
Re-create sql jobs
Disable backup jobs for user databases
Re-enable backup jobs on primary server in old cluster

While the steps were different for each server, this was basically what I needed to do. By the end of Wednesday, April 17th, 2019, if we were lucky I’d have moved 3 servers to the new clusters, and would have SQL Server installed with everything syncing as I tested. The goal by the end of Day 3 was to have things looks a bit like this:

I announced we were going to start on Monday, April 15th and finish the following week. I felt pretty confident that we were ready for production. There was absolutely no way we’d have issues like we did on the development servers, right? Wrong. Oh did we ever hit unexpected issues.

Day 1

The very early morning of April 15th rolled around, and I kicked off the plan for the very first server, NY-SQL03, the NY secondary in the SE cluster. Everything went according to plan, Windows installed, SQL Server installed, new cluster set-up - all 4 temporary distributed availability groups (aka our TAGs) were in place. We had all databases syncing and reporting green in Opserver at the end of the end. Day 1 done, so far, so good - one server down, 5 more to go.

Day 2

The day started just as early because if things went well, I was going to tackle two servers, NY-SQL01, the NY secondary in the SO cluster, as well as CO-SQL03, our Colorado remote secondary in the SE cluster. I kicked off the rebuild on both servers, and NY-SQL01 was back in working order within a few hours with everything syncing as we expected. The other server, CO-SQL03, was another story.

While CO-SQL03 was rebuilding, I started to notice some weirdness…yes, weirdness (that’s a technical term, right?) with the server I rebuilt the day before - NY-SQL03. As a quick reminder, this server was currently the only node in the new 2016 cluster. It had about 375 databases which were in 4 availability groups, and 4 distributed availability groups - the distributed AGs were being used to keep the data in sync between the old 2012 cluster and the new cluster. The breakdown of number of databases per AG was:

AG-NYOnly - 6 databases syncing via distributed AG - NYOnly_TAG
AG-Misc - 10 databases syncing via distributed AG - Misc_TAG
AG-Chat - 3 databases syncing via distributed AG - Chat_TAG
AG-SENetwork - 354 databases syncing via distributed AG - SENetwork_TAG

Now, that you’ve had a reminder of what we had on the server, let me explain what I meant by “weirdness”. One of the primary ways we monitor our servers is through Opserver. After hours of what appeared to be databases syncing without any issues I started to notice that we were hitting blocking when trying to get the status of the availability groups and databases when querying the DMVs as Opserver does. And not just a little bit of blocking. This was significant blocking on everything. We couldn’t get any details in SSMS on the databases or availability groups. If you attempted to open any details in SSMS (AGs, databases), it would lock your session and you’d have to kill it in Task Manager.

Everything we tried to get the state of the AGs was locking up, even directly querying the DMVs in SSMS. We were seeing tons of HADR_AG_MUTEX waits:

In addition to the locking, we noticed a lot of inconsistencies in what was being returned via the queries. One minute it looked like the availability groups were fine, the next it was in a not-synchronizing state. We realized that we couldn’t trust the state of the AGs on the server. After hours of looking at this and trying to debug, I decided to drop our temporary distributed AGs, the TAGs, to see if rebuilding them would work because nothing else was working at this point.

I started recreating some of the TAGs to see if things would sync, as I created each one, they would start to sync, but we were still hitting significant waits and locking.

We tried restarting the SQL Server service. We tried stopping the availability groups in the WSFC. We were seeing errors in the logs that the AGs were in a failed state, we couldn’t query the DMVs, we didn’t have any idea what was happening. It looked like the 2 distributed AGs that were small were working, even though checking their health was very slow.

It was time to try to recreate the SENetwork_TAG, the one with the 354 databases. Early on in this process, we stopped Transaction Log backups and Daily Backups, so we had a couple of options:

Autoseed the databases as you can with availability groups
Use the old backup of the SE Network databases (the one from before we started the migration process), and restore it to the NY-SQL03 server, letting it get caught up with the AG syncing
Take a new backup of the databases in the AG, restore it to NY-SQL03 again, and then join it to the AG to let things start syncing.

Considering the slow querying of the DMVs and the AG health, we went with number 3. I took a recent backup of the database from the primary server in the old cluster and restored it to the new server with NORECOVERY. The thinking was that this would prevent overloading the server by automatically trying to seed 354 the databases all at once. Once all the databases were restored, we would execute the following on each one:

1

ALTER DATABASE <database_name> SET HADR Availability Group = [AG-SENetwork];

Problem was, this was really slow. Doing this for each database was taking a couple of minutes per database and we have 354 of them. The waits we were hitting were downright unbelievable:

The amount of time to execute the command against each database exponentially increased as we were adding more and more databases. It was going to take hours to get through them all. It was super late in the day, and Nick and I had been debugging and working in a hangout for many, many hours. Nick quickly wrote a script to loop through all of the databases in the SENetwork_AG and execute the SQL above. This automated the process which allowed it to run overnight, and more importantly allowed us to step away from our desks after far too many hours. We hoped everything would be in the AGs in the morning, so we could debug some of the issues we were seeing and potentially move forward with the upgrades.

Days 3-7

Yes, that really says Days.

The script eventually finished and we had all databases in the AG, however, we were still seeing tons of waits. I was chatting with some of the users on our Database Administrators Stack Exchange site, and there were suggestions to disable parallel redo on the server, as well as other things. We tried everything we could think of, and still didn’t know if things were syncing because we couldn’t get an accurate read from the DMVs or get passed the waits. Since nothing was working, it was time to open a ticket with Microsoft. Our ticket was opened, the server was creating plenty of mini-dumps that went sent along with memory dumps, and now it was time to wait for help.

A quick aside: while waiting for things to get moving with Microsoft, we attempted to get CO-SQL03 back in a good state. We take local copy-only backups in Colorado, so we don’t have to deal with trying to move terabytes of datafiles from NY to CO in the event of an issue. All of that typically is fine, but due to the issues, we ended up being down for far longer than expected. This meant our backups in CO were very far behind the current state of things in production. In order to get CO-SQL03 in working order, we needed to copy all of the database backups from NY to CO, which was going to take a really, really long time. Good thing we weren’t in a hurry, huh? We kicked off a process to copy the current backups from NY and get them moved to CO. With Shane Madden and Nick Craver’s quick thinking, we copied all of the files in less than a day. Now that we had good backups, I restored all of the databases in NORECOVERY so they would be ready whenever NY-SQL03 box was back up and running.

Our Microsoft ticket was forwarded and we began working with one of our DBA.SE users who happens to work for Microsoft, Sean Gallardy. At this point, it was time to go back to the regularly scheduled painful upgrade process. Based on the memory dumps, one of the first things Sean suggested was to make sure that auto-seeding of the databases in the SENetwork_TAG was disabled, so we executed:

1

ALTER AVAILABILITY GROUP [SENetwork_TAG] MODIFY REPLICA ON 'NY-SQL03' WITH (SEEDING_MODE = MANUAL)

While this prevented the databases from automatically being seeded, it didn’t resolve our issue. We were still failing to sync databases on the new cluster, and we were dropping mini-dumps like crazy on NY-SQL03. It was back to the drawing board with ideas to fix all the things. Nick and I got onto a really long hangout with Sean (thanks Sean), to debug and work through various ways to fix the problem. Eventually, Sean suggested that we try the following:

Destroy the temporary distributed availability group (TAG) for the SENetwork, since that was our problem child
Destroy the availability group on the new cluster
Manually restore the databases to both NY-SQL03 and CO-SQL03 in a restoring state
Recreate the temporary distributed availability group for the SENetwork
Add each database to the availability groups on both NY-SQL03 and CO-SQL03 very slowly so they could sync

That’s what we did. Another really slow process, and when I say slow I mean 2-4 databases at a time - and we had 354 to add, so it was painfully slow, but it worked.

We were hitting a bug with SQL Server. The bug had to do with distributed availability groups that had more than 15 databases and, of course, we had that. Our distributed availability group contained 354 databases and it couldn’t handle that many, resulting in it timing out. We got a workaround by following the steps above were able to move forward.

By the end of the week we had two new clusters up and receiving data from the old 2012 clusters. It was now time to finish moving pieces around so we could perform the failover of the temporary distributed availability groups. Even though we lost several days working through bugs, we were in a better, more stable state and could plan the date of the failover - if everything else went according to plan, it would be in just a few days. To help visualize where we were at the end of this week, our servers were in this state:

After a pretty quiet weekend, only a few minor hiccups with data syncing, we were ready to pick up and go full steam ahead to wrap things up.

Day 8

Monday morning rolled around and there were 3 servers left and one huge failover to do. Surely, we had hit all of the bugs and it was going to be a clear path to the finish line, right?

I started the day by evicting CO-SQL01 from its old cluster and kicking off the rebuild process. Ugh, and I hit some issues. The automated process, Foreman, didn’t want to work without some poking and prodding to get it moving, then the server didn’t want to find the Windows Updates it needed, but eventually the rebuild finished. I was able to install SQL Server, restore backups in a NORECOVERY state, apply all the transaction logs to bring it up to current state, and just like that, we had syncing data.

We finally had 4 servers in the new clusters, and they were reporting green everywhere.

Day 9 - Failover Day

The big day was here. It was time to perform the failover of 5 temporary distributed availability groups, with hopefully very little downtime. I previously tested the whole process in the lab, but had never done this in production. And considering all the problems we hit up to now, I was feeling just a wee bit stressed out.

The morning of the failover, we spent time going over the steps that would be executed that night. We wanted to make sure that we had accounted for all of the application side changes that were needed as well. The complexity of this whole thing wasn’t limited to just failing over the SQL Servers. We also needed to rebuild and push out new versions of all of our applications and services due to changes to the connection strings.

You might be wondering why we needed to do that? All of our application connection strings point to global listeners and since we had new clusters with new AGs/distributed AGs - they all needed to be created with new listeners (IP Addresses) and new names. This means that prior to the failover, the applications were pointing to the old listeners. In order for them to work when we failed over, the apps needed to be rebuilt to production to point to the new listeners.

We also needed to be very deliberate about the order of things on the failover day.

The first thing we considered were exceptions. We have a database called NY.Exceptions which we use to capture errors that occur throughout the network. We needed to be sure it would still capture errors, in the event the failover failed. To do this, we removed the NY.Exceptions database from both the old and new AGs and made sure the database was writable on both the new and the old clusters. We also added both databases to our monitoring in Opserver to catch lingering applications pointing to the wrong place. This would let us collect errors, even if things went wrong with a failover.

Next, we had to think about how to push our application changes to production. We use Team City to deploy our applications, which has a database on SQL Server. We needed to be sure that Team City was available before we moved forward. As a result, the NYOnly_AG was the very first one to failover. We pushed a change to the connection string from SQL-NYOnly_AG to SQLAG-NYOnly and built out the application. Once that was done, it was time to do the distributed availability group failover.

Based on my testing and the Microsoft Docs performing a failover of a distributed availability group is a manual process. The code needs to be executed on the current primary server, validated, then more code has to be executed, then validated, and repeat until it’s all done. In my huge planning document, I wrote all the code to perform the failover steps. Now it was time to execute them. Since we were going to be failing over the temporary distributed availability group called, NYOnly_TAG we executed the following steps:

On the current global primary server in the DAG (NY-SQL04) change the DAG to SYNCHRONOUS_COMMIT

1
2
3
4
5


 ALTER AVAILABILITY GROUP [NYOnly_TAG]
 MODIFY AVAILABILITY GROUP ON
 'NYOnly_AG' WITH (AVAILABILITY_MODE = SYNCHRONOUS_COMMIT),
 'AG-NYOnly' WITH (AVAILABILITY_MODE = SYNCHRONOUS_COMMIT );
 

Verify the commit state is SYNCHRONIZED on the global primary (NY-SQL04):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


 SELECT ag.name
 , drs.database_id
 , drs.group_id
 , drs.replica_id
 , drs.synchronization_state_desc
 , drs.end_of_log_lsn
 FROM sys.dm_hadr_database_replica_states drs
 INNER JOIN sys.availability_groups ag
 ON drs.group_id = ag.group_id;
 

Once we’re in a SYNCHRONIZED state, set the global primary (NY-SQL04) to a SECONDARY role
1 2

ALTER AVAILABILITY GROUP NYOnly_TAG SET (ROLE = SECONDARY);

Now the AG is not available for write traffic. Test that we’re ready to failover by executing the following on both the former global primary (NY-SQL04) and the soon to be new global primary (NY-SQL03). This query verifies that the LSN is the same on both servers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


 SELECT ag.name,
 drs.database_id,
 drs.group_id,
 drs.replica_id,
 drs.synchronization_state_desc,
 drs.end_of_log_lsn
 FROM sys.dm_hadr_database_replica_states drs
 INNER JOIN sys.availability_groups ag
 ON drs.group_id = ag.group_id;
 

Perform the failover to the new global primary (NY-SQL03)

1
2

 ALTER AVAILABILITY GROUP NYOnly_TAG FORCE_FAILOVER_ALLOW_DATA_LOSS;

Change the state of the distributed availability group back to ASYNCHRONOUS_COMMIT this is done on the new global primary (NY-SQL03)

1
2
3
4
5


 ALTER AVAILABILITY GROUP NYOnly_TAG
 MODIFY AVAILABILITY GROUP ON
 'NYOnly_AG' WITH (AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT),
 'AG-NYOnly' WITH (AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT);
 

At this point, our first distributed availability group had failed over and things were writable again. Only 4 more to go!

We could capture exceptions and with Team City up an running, it was time to tackle the next DAG, Chat_TAG. We performed the same steps listed above, on that DAG. Chat was selected to be done next, because we use our chat platform internally, and having it accessible is very important. When I’m doing regular maintenance on the SQL Servers, the Chat_AG is one of the first AGs we failover, and that the pattern held for the DAG failover. After executing all the steps above, we successfully failed over Chat.

2 down, 3 to go! Time for the more critical stuff to be tackled.

To minimize impact for users, we did the failover during the latter part of the day. Next up in the failover process was the DAG that contained the Careers database. By doing this, it would mean the end of syncing to the reporting cluster. We knew it was going to happen eventually, and held off as long as we could. Our stakeholders knew this was part of this failover process. The goal was to be back online as soon as possible once the entire failover process was complete.

Before starting the failover, the old distributed availability group between the 2012 cluster and the reporting cluster was destroyed. This was done because we knew the syncing to the reporting cluster would stop when we failed over. There was no need to have this extra layer of complexity and strain on the NY-SQL04 server. After the HighAvailability_DAG was destroyed, we set the Jobs area of the site to read-only, replaced all the connections strings in the applications, and then rebuilt them. It was then time failover the new DAG, Misc_TAG. Again, everything failed over without any issues. The Careers database was writable again, we moved Jobs out of read-only, and we were now more than half-way done with the failover.

The final two failovers were for the StackOverflow_TAG and SENetwork_TAG distributed AGs. These meant that we were going to have a bit of downtime. We were also planning on doing them in rapid succession. These two distributed AGs contain all the databases that handle the entire network of sites, and we wanted to fail them over as fast as possible to reduce the downtime. We pushed all new connection strings, the apps were rebuilt, and we were ready. All sites were set to read-only and it was go time.

First up, was the StackOverflow_TAG, as this only had 5 databases, and we figured it’d be quicker than our problem DAG, SENetwork_TAG - aka the one with 354 databases. I destroyed the old distributed availability group going to the reporting cluster, and we began the process to failover this DAG. Thankfully we hit no issues with it.

We were in the home stretch for the failovers with only one left…the big one. We hit so many issues with this distributed availability group when trying to get it set up, we were concerned about what would happen when we failed over. Unable to wait any longer, I kicked off the steps above to start the failover. It moved to SYNCHRONOUS_COMMIT without any problems, the databases on NY-SQL04 all reported to be in a SYNCHRONIZED state, and I moved the server to be a SECONDARY role in the DAG.

So far so good. Now it was time to verify the LSNs between the two servers using:

1
2
3
4
5
6
7
8
9


SELECT ag.name,
 drs.database_id,
 drs.group_id,
 drs.replica_id,
 drs.synchronization_state_desc,
 drs.end_of_log_lsn
FROM sys.dm_hadr_database_replica_states drs
INNER JOIN sys.availability_groups ag
 ON drs.group_id = ag.group_id;

Executing that query on both servers (NY-SQL04 and NY-SQL03) and trying to verify that the LSNs were the same on 354 databases was not what I’d call an enjoyable process. We would verify, then re-verify and something would change, so we’d verify again, and again. It took some time to figure out if we were in a good state to failover. Our goal was to not lose any data, but trying to make sure we had a match against all the databases was pretty tough. Once we were reasonably confident, we made the call that it was time to finish the failover of this DAG, so I executed the commands to failover the SENetwork_TAG. Once the failover command was executed, the new primary (NY-SQL03) came up just fine, and was responsive for us to query it. We were done with all of the temporary distributed availability groups, and all of the sites went back to read/write. Failover done!

We thought we were in the clear, but the former primary (NY-SQL04) was very, very angry. After the failover from NY-SQL04, the server was completely unresponsive to anything. All the databases locked, CPU spiked, we were hitting HADR waits, and we didn’t get any sort of response for more than 25 minutes. Thankfully, we weren’t overly concerned because we had a new primary that was working, and this would be completely rebuilt to Windows Server 2016. We made the decision to just stop the SQL Server service until I got around to the rebuild.

All of the failovers were complete and everything was running on the new Windows Server 2016 clusters. Our servers looked like this:

We still had two servers in the old 2012 clusters. NY-SQL04 wasn’t functioning as a SQL Server because we shutdown the SQL service post failover, and NY-SQL02 was a member of the temporary distributed availability group we just failed over. Those servers were going to be rebuilt the next day, so our focus turned to the reporting cluster. That cluster is used internally for applications, and we needed to get it back up as soon as possible for use.

Initially, we thought that if we recreated the distributed availability groups with the reporting clusters as a member, things would just pick back up and sync. We executed the following to recreate the two distributed AGs with the new cluster and reporting cluster:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79


-- run this on NY-SQL03
CREATE AVAILABILITY GROUP [Misc_DAG]
 WITH (DISTRIBUTED)
 AVAILABILITY GROUP ON
 'AG-Misc' WITH
 (
 LISTENER_URL = '<listener>',
 AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT,
 FAILOVER_MODE = MANUAL,
 SEEDING_MODE = AUTOMATIC
 ),
 'HighAvailability_RAG' WITH
 (
 LISTENER_URL = '<listener>',
 AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT,
 FAILOVER_MODE = MANUAL,
 SEEDING_MODE = AUTOMATIC
 );
GO

-- run this on NY-RPTSQL01
ALTER AVAILABILITY GROUP [Misc_DAG]
 JOIN
 AVAILABILITY GROUP ON
 'AG-Misc' WITH
 (
 LISTENER_URL = '<listener>',
 AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT,
 FAILOVER_MODE = MANUAL,
 SEEDING_MODE = AUTOMATIC
 ),
 'HighAvailability_RAG' WITH
 (
 LISTENER_URL = '<listener>',
 AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT,
 FAILOVER_MODE = MANUAL,
 SEEDING_MODE = AUTOMATIC
 );
GO

-- run this on NY-SQL01
CREATE AVAILABILITY GROUP [StackOverflow_DAG]
 WITH (DISTRIBUTED)
 AVAILABILITY GROUP ON
 'AG-StackOverflow' WITH
 (
 LISTENER_URL = '<listener>',
 AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT,
 FAILOVER_MODE = MANUAL,
 SEEDING_MODE = AUTOMATIC
 ),
 'StackOverflow_RAG' WITH
 (
 LISTENER_URL = '<listener>',
 AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT,
 FAILOVER_MODE = MANUAL,
 SEEDING_MODE = AUTOMATIC
 );
GO

-- run this on NY-RPTSQL01
ALTER AVAILABILITY GROUP [StackOverflow_DAG]
 JOIN
 AVAILABILITY GROUP ON
 'AG-StackOverflow' WITH
 (
 LISTENER_URL = '<listener>',
 AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT,
 FAILOVER_MODE = MANUAL,
 SEEDING_MODE = AUTOMATIC
 ),
 'StackOverflow_RAG' WITH
 (
 LISTENER_URL = '<listener>',
 AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT,
 FAILOVER_MODE = MANUAL,
 SEEDING_MODE = AUTOMATIC
 );
GO 

After executing this and recreating the distributed AGs, we realized very quickly we were wrong about them starting to sync again. The reporting cluster wouldn’t sync. We tried kicking the databases with various commands, and even tried failing the reporting cluster over to the secondary server in Colorado and then failing back. Nothing was working. After checking the logs on the reporting cluster primary, NY-RPTSQL01, we found lots of errors:

Msg 3456, Level 21, State 1, Line 22

Could not redo log record (8278:53574:4), for transaction ID (0:8787310), on page (1:10082), allocation unit 72057594040811520, database ‘CareersAuth’ (database ID 23). Page: LSN = (8277:108566:2), allocation unit = 72057594040811520, type = 1. Log: OpCode = 6, context 1, PrevPageLSN: (8278:53568:2). Restore from a backup of the database, or repair the database.

Msg 3313, Level 21, State 1, Line 22

During redoing of a logged operation in database ‘CareersAuth’, an error occurred at log record ID (8278:53574:4). Typically, the specific failure is previously logged as an error in the operating system error log. Restore the database from a full backup, or repair the database.

Msg 3414, Level 21, State 4, Line 22

An error occurred during recovery, preventing the database ‘CareersAuth’ (23:0) from restarting. Diagnose the recovery errors and fix them, or restore from a known good backup. If errors are not corrected or expected, contact Technical Support.

Well, that stinks. It meant we’d have to rebuild the AGs on the Reporting Servers using a recent backup, and then try to get them to synchronize. We started restoring the databases that were impacted in a NORECOVERY state with a local copy so the restore process didn’t too long. Once the databases were restored, we executed the following on each database to get them syncing again:

1

Alter Database [database_name] Set HADR Availability Group = [distributed_AG_Name];

While this all worked great for the primary reporting server, NY-RPTSQL01, we were not as lucky with the secondary server in the reporting cluster, CO-RPTSQL01. We already suffer from throughput issues with CO-RPTSQL01 and all of the de-synchronizing put all of the databases in a state that couldn’t be recovered. Unfortunately, all the databases would also have to be rebuilt by restoring copies, then altering them to set them as a part of the availability group. The big problem was we didn’t have a local copy in Colorado to restore due to all the other server moves (we had turned off daily backups), so we were going to have to copy the files from NY to CO to then restore on CO-RPTSQL01. That might not be a big deal normally, but we have databases that are several TBs and the move was going to take a while. I decided to call Day 9 done, and pick that process up in the morning, along with other clean-up.

Day 10 - Finishing Touches

After a couple of hours of sleep, it was time to try to finish this up. The list of things left to do was small, but was still quite a bit of work. We had to:

Restore databases to CO-RPTSQL01 and get the AGs/distributed AGs syncing throughout the reporting cluster
Destroy the temporary distributed availability groups on the new 2016 clusters
Rebuild NY-SQL04 and NY-SQL02 - the last two servers in the old 2012 clusters
Insert the newly rebuilt servers into the new 2016 Windows Clusters, install SQL Server, and add them to the AGs
Turn on all t-log backups and daily full backups
Put read-only routing in place on all availability groups

The goal was to get through the entire list same day, but that would be too easy. We ran into an issue immediately. Remember that NY-RPTSQL01 we got synchronizing at the end of Day 9 – it was lying to us. It said it was syncing, but it really wasn’t. If you queried the StackOverflow database, it was several hours behind the current production database…uh oh, something was broken. We tried to restart the SQL Service to see if that unblocked it, it didn’t. Unfortunately, the only way we were able to get it syncing again was to:

drop, then restore the databases
manually apply the t-logs to bring it up to date from the current global primary
then after executing some SQL to kick the AGs, like magic, it would be synchronizing again

We did this for all the impacted databases on the reporting cluster primary, NY-RPTSQL01, and then still had to get CO-RPTSQL01 back online. Nick Craver jumped in and worked on the NY and CO reporting servers, while I started rebuilding the final two servers, NY-SQL04 and NY-SQL02. The rebuild of the final two servers went exactly as expected. Once it came time to put the servers into the availability groups, I took fresh backups from the primaries and restored those. I did this because we weren’t going to automatically seed an AG with 354 databases or a 3.5TB one (StackOverflow), and I wanted to manually apply as few T-logs to the databases as possible to bring it up to current time.

It may have taken all day, but eventually we fixed the reporting cluster and had all six of our main SQL Servers moved into the new Windows Server 2016 clusters. By the end of the day, Opserver was reporting green everywhere:

It was such a good feeling to have everything moved over. After months of planning, testing, and frustration from hitting issue after issue, plus a bug in SQL Server, it was incredible to see it all come to fruition.

Final Thoughts

Besides the upgrade to SQL Server 2017 from the year before, I had never even attempted something with this level of complexity or planning. This project involved a lot of juggling of servers, AGs/distributed AGs, application changes, and considering we don’t have maintenance windows at Stack Overflow, it was difficult to imagine how to pull this off without being down for an extended period of time. To know this entire thing was executed with only about 10-15 minutes of downtime for the public facing sites is pretty darn amazing, in my opinion.

I have some takeaways from this - most are common sense, but I need to repeat them for my own sanity:

Test, re-test, and then test some more. The initial plan was to jump from the lab to prod, the rebuild of our development environment was unexpected. I also didn’t think this upgrade would break our development environment, but by doing these intermediate steps we found several bugs in our deployment process which we might have hit in production.
Have a lab environment, if possible. Between creating new lab environments and using my existing lab environments, I was able to work through all the different scenarios I figured we would hit in production. Of course, I wasn’t able to test at our scale and find the SQL Server bug, but I felt confident the steps I planned would work for the failovers.
Rely on your team. While I’m the only official DBA at Stack Overflow, there were times when I was at a loss on how to fix some of our issues with Foreman, Puppet, and needed extra hands. I can’t thank my team enough for stepping in and jumping into hangouts with me when I was stuck or something broke.
Remember that no matter how much you plan, something is going to go sideways. We had a ~35 page planning document that I wrote and none of that mattered when we hit the bug with availability groups.
Try not to stress out over the things out of your control. I hit roadblock after roadblock during this project, and felt that I was letting my team down by not moving it forward faster. When you expect a project to be done in one month and it takes about four, it’s hard not to feel like you’re failing and be stressed out over it. Each issue (failed drives, bugs) moved us forward and we improved our process, even if it was incredibly slow. In the end, we were in a much better situation because we squashed bugs and moved to better software. Keeping perspective on the end-goal is something to remind yourself when in these types of situations.
Keep notes as you’re going through a deployment like this. Every day write yourself a recap of what transpired that day, so you remember what went well or what broke. Unfortunately, I didn’t do this during the production part and spent a lot of time going through chat transcripts trying to remember all the steps and roadblocks we hit to document in this post.

I’m hoping this is helpful to someone who needs to go through this type of upgrade. Now that this project is done, I guess it’s time I start thinking about upgrading to SQL Server 2019.

How we upgraded Stack Overflow to SQL Server 2017

Wed, 07 Nov 2018 19:00:38 +0000

This post has been rattling around in my head for months, and with SQL Server 2019 on the horizon, I figured I’d finally put my thoughts down, especially since I know we might hit some of the same issues when we upgrade.

A quick background, when I became the DBA at Stack Overflow, we had various versions of SQL Server in place — the majority were at some level of SQL Server 2016, each with a different CU or SP installed. We even had an instance of SQL Server 2012 still running. The bulk of our servers hadn’t been patched for 6-9 months, and we were just as behind in our Windows Updates…eeek. In a perfect world, all servers run the same version and CU. However, since we don’t have regular maintenance windows, tackling this would mean downtime - which meant it had to be a project not a one-off type of task.

When I kicked off this project, back in February 2018 (see I told you this has been rattling around in my head for months), we had ~30 servers, the breakdown of them was:

3 servers in the Stack Overflow cluster (2 in NY, 1 in CO)
3 servers in the Stack Exchange cluster (2 in NY, 1 in CO)
3 servers in a cluster for Internal Stuff - these power our internal applications (2 in NY, 1 in CO)
2 servers for Reporting Cluster - these power various reporting applications (1 in NY, 1 in CO)
the remaining servers have miscellaneous functions like - lab/test environments, development servers, traffic logs, and many others used by various teams (located in both NY and CO)

We have various availability groups (AGs) running on the Stack Overflow, Stack Exchange, Internal Development, and Reporting servers. Adding another layer of complexity, we also have availability groups that are a part of Distributed Availability Groups (DAGs). The main clusters (SO, SE, and Internal) have data that flows from a handful of databases downstream to the reporting servers. It looks a bit like this:

The Initial Plan

With so many servers to upgrade, I needed to figure out the best path to get them done in the least amount of time, with the least amount of downtime. I started out by basically creating a spreadsheet of metadata. This spreadsheet included all the server names, the current software version (including CU/SP), the date of the last Windows Update, the team to notify before starting the maintenance, and the date the upgrade was completed. I also added a rudimentary risk level from low to high, and chose the levels based on whether the server was public, internal, dev, etc. Using this risk level, I prioritized the order in which to upgrade each server. Since all the servers hadn’t been patched in months, the patching plan for the day of the upgrade was:

upgrade to SQL Server 2017
run Windows Updates
reboot as needed
apply SQL Server CU update
reboot and repeat as needed - mainly Windows Updates until everything was up to date

My goal was to complete all servers over the course of a month. That may sound like a long timeframe, but with only me touching them, it gave me wiggle room in the event there were issues. I started on the servers with the least amount of risk, this meant the Lab, Dev, and some of the miscellaneous servers. These aren’t customer facing, and are used internally, so if something went wrong we wouldn’t have public downtime. After patching ~15 servers I felt pretty confident that the upgrade of the rest would go fine…this was a mistake.

But I Followed The Plan

Moving to the servers in the AGs/DAGs, I knew I would follow this order for each cluster:

Remote Secondary (CO)
Local Secondary (NY)
Local Primary (NY)

In between the NY Secondary and the NY Primary I’d perform a failover to let me hit the last server in the cluster. I also decided that I would do the remaining servers in this order:

Reporting Cluster
Internal Dev Cluster
SO/SE - CO/NY secondaries
SO/SE - NY primaries after a failover

This allowed me to get the majority of the upgrades done during the week, then we’d perform a quick failover on a weekend, during lower traffic, and I’d knock out upgrading the last two critical servers. Based on this, the CO reporting server was up first.

All boxes on the patching plan were ticked for this server, and when it came back up after the final reboot, I checked our instance of OpServer and everything was green and syncing; I thought ‘Woot, everything is great.’ Since we use readable secondaries, I wanted to perform a touch test on the databases in the AGs and DAGs to make sure everything was working. I logged into SSMS, but when I attempted to query any database in a DAG they weren’t accessible. I couldn’t read from them. This was a giant ‘oh crap’ moment.

I frantically started checking the server to see if there were errors. There was nothing. Nothing in the logs, and no errors from the installation. From what I could tell, everything went as expected. Then why couldn’t I read from the databases? After doing a bit of research, I found it was because the secondary was a higher version from the primary. Initially I thought, ‘OK, I’ll upgrade the NY reporting server and everything would be fine’, but then I remembered these servers are downstream in the DAGs. Uh oh.

If I upgraded both reporting servers, which receive data from 3 DAGs (SO/SE/Int Dev), and those are all a lower version of SQL Server, both reporting servers would be totally out of commission until the upstream servers were all upgraded. All this would result in major downtime for reporting and wasn’t going to work.

I did research, pinged people on Database Administrators Stack Exchange, and poked my co-worker Nick Craver about it. From everything I heard and read, the behavior was expected, but incredibly difficult to deal with if you’re not upgrading everything at the same time, or in a short amount of time. I wanted to spread the upgrades out over a month to make them manageable, but when they go out of commission because of a upgrade, that timeframe went out the window.

All this left me in a state of confusion. How the heck am I going to upgrade these with basically zero downtime, and ensure they don’t become unreadable for days?

My plan blew up — I couldn’t do the upgrade in the original order and timeframe that I wanted.

Back to the Drawing Board

Since we couldn’t be without both reporting servers for days, that meant I needed to come up with a new plan/timeframe to get this done. It would also need buy-in, because at some point we’d have more downtime than initially expected.

Microsoft recommends upgrading servers in this order:

Upgrade remote secondaries
Upgrade local secondaries
Manual failover of the primary to the local secondary
Upgrade previous primary
Failback to previous primary

As you can see, that’s what I was following. The problem comes in when the server is in a Distributed Availability Group. I did the CO reporting server first, and ideally we’d do the NY version next, but since these are downstream in 3 DAGs, both NY and CO would be completely unreadable until the primaries upstream in the DAGs are upgraded. This means we automatically have reduced service with the servers, I can’t move to the CO secondary in the Internal servers because it’ll become unreadable like the reporting servers. As I move outward, the reduced service grows. So now what am I going to do?

Thankfully, I have a lab environment to test with. If you don’t have one, make one. I have 4 VMs that I set up to mimic our AG/DAG environment. This allowed me to test the upgrade order of the servers, perform failovers, verify the AGs/DAGs came back online, etc. After multiple failed attempts at getting the order right, I finally figured out that I would follow the same order and pattern that I was using, just not at the cluster level - it would be based on location instead.

The new plan would be:

Upgrade all local NY secondaries in a DAG/AG - doing this makes them completely unreadable
- the NY secondaries in the Internal, SE, SO clusters
Upgrade all remote CO secondaries in a DAG/AG - doing this makes them completely unreadable
- the CO reporting server was done, this left the CO servers in the Internal, SE, SE clusters
Perform a failover in a specific order
- Reporting Server - failover to CO - this makes the entire cluster unreadable until all upgrades are finished - failover non-DAG availability groups, then failover the DAGs - hope that when it shows it’s syncing it really is, since we won’t be able to query anything
- Internal Server - failover to NY secondary - failover DAGs first, verify it’s syncing to reporting, then failover AGs
- SE Cluster - failover to NY secondary - failover DAGs first, verify it’s syncing to reporting, then failover AGs
- SO Cluster - failover to NY secondary - failover DAG
Verify that everything is syncing or at least verify that everything appears to be syncing - I’m looking at you reporting servers
Upgrade the former primaries in NY - reporting, internal SE, and SO
Verify that all databases are syncing - most likely they aren’t, so running ALTER DATABASE [<name>] SET HADR RESUME; should resolve any syncing issues - again looking at you reporting servers
Perform any failovers to reinstate a primary. For us, this only applies to the reporting cluster. When we failover within NY, we just let the new primary stay primary until we need to failover again - this minimizes downtime for us.

You can see that this deviates from the recommended order by Microsoft because we did the local NY secondaries first. This was done in the event something catastrophic happened in NY, and we had to fail to CO. We knew CO was still up and running since we could query it. If we had upgraded CO and made it unreadable, we wouldn’t be able to query it and confirm things were syncing properly for an extended period of time. The thought of this made us uncomfortable.

I tested and retested this plan in the lab environment to get it down before attempting this on any of the critical servers. The only real issue was the reporting servers would be out of commission until the rest upgraded. This meant everything would have to be upgraded fast, so we did not have reporting unavailable for too long. Based on this, it was decided we’d do the local NY secondaries on a Friday, then perform the rest of the upgrades and failover on a Saturday - this meant downtime, so we had to give plenty of notice to the community before tackling it.

In the few days before the upgrade, I wrote up the playbook for the weekend. To minimize downtime and to get the remaining 11 servers upgraded quickly, there were 3 of us who were going to be doing the last upgrades, so I wanted everything documented. The playbook had every step outlined, who was doing what to which server, SQL code for each step (if needed), what was to be installed on each server, the order of failover of AGs…everything. This was my first major upgrade and I was a nervous wreck, I didn’t want anything to be missed or to go wrong, so I probably overplanned, but why not, right?

The weekend came and went, and we had no issues with the upgrades. When we failover our main SO/SE clusters, we typically put the sites in read-only mode, just to be safe that nothing crazy happens when SQL is failing over - for this upgrade our downtime/read-only time was < 10 minutes which was far less than we expected. If you’re super curious, you can watch a recording of the livestream, though I’m not sure why you’d want to sit through ~3 hours of us talking about upgrades.

In the end, I learned a bunch from this. I figured out how best to upgrade to new versions of SQL Server when Distributed Availability Groups are in play without making servers unreadable for too long. I also learned just how darn handy having a lab environment is, and that I super over-prepared/planned for the days of the actual work. I’m not sure if this will help anyone else in the future, but now I know to avoid these landmines when upgrading.

I also know there are probably plenty of other landmines I can hit when we move to SQL Server 2019.

Pivoting yet again - New blog and domain

Mon, 27 Aug 2018 15:00:00 +0000

Almost a year ago, in November 2017, I moved back into a more technical role at Stack Overflow and became a Database Administrator. During this time, I have wanted to relaunch my blog, as it’s been over three years since I really wrote anything (yes, I’m terrible).

Well, that time has finally come.

After spending a few weeks, looking at various options to replace Wordpress, I finally decided on using Hugo and Netlify. Hugo was incredibly easy to set up and Netlify deploys my changes automatically when I push to GitHub. Additionally, I moved away from onlybluefeet.com and am now using tarynpivots.com. This new URL seems fitting, as I love the PIVOT function in SQL Server, and I seem to change things a lot. I have no plans on moving away from being a DBA any time soon, so my plan is it to write about the SQL Server issues we deal with at Stack Overflow, as well other SQL things.

Ch Ch Ch Changes are afoot

Fri, 22 May 2015 22:49:01 +0000

Well it’s been an interesting couple months and there are quite a few changes going on in my life, but before I get into what’s coming up, I feel that I need to give a bit of an overview of my journey thus far.

I never thought about becoming a developer. When I was growing up, I loved school (yeah, I’m weird), I loved math and science (even weirder), I was going to be a doctor, work on genetics, or something similar. That was my plan even when I started college. Then there were some family things that happened and I was steered to law. I changed majors and got a degree in Justice Studies with the plan to go to law school. I eventually went to law school and after 1 semester realized I hated it. There was no way I was going to finish it, so I quit law school.

Problem was, I had no backup plan. I had a no clue what I was going to do. I bounced around a bit, took a class here and there trying to figure it out, but nothing sparked my interest.

I was working a customer service job, and there was an internal job posted for a Web Site Developer that required experience with HTML, CSS, and ColdFusion. I’d never done any of that before, but my feeling was “Heck I’m smart, I can learn it” so I applied it. The manager was nice enough to give me an interview and once he realized I had no experience, he said “I can see you want the job, but no. We need someone with some knowledge.” I completely understood, but I’m stubborn and I decided to prove him wrong. I went out that weekend, bought a few books, and started learning HTML, etc. Via the work grapevine, it got back to the manager of the Web Team that I did this, so he decided that my drive was enough to give me a chance and boom I got the job.

In one day, I went from not knowing what I was going to do, to becoming a developer. Post Web Site Developer, many of my jobs were the same way, I didn’t know the technology required, but I’d learn it. When I left the web job, I needed to know VBA and MS Access, that turned into C#, SQL Server, and Winforms. Each of these I learned on the job or on my own. The only difference was I stopped buying books and started using sites like Stack Overflow to learn.

As I said, I didn’t grow up wanting to be a developer. While I like being a developer, lately I’ve felt burned out with it. I’ve had this nagging feeling that I need to be doing something or working on something that I care about. That I feel passionate about. But I wasn’t sure what that was.

I started using Stack Overflow almost 5 years ago, when I needed to write C#. I had a book but I wasn’t able to solve some of my problems, so I’d either ask a question or search for answers. I loved the site immediately. I loved that I could get a solution to my specific problem. Yes, I had growing pains on the site, like so many others. For example, I wouldn’t give enough details when posting a question, etc. but I learned how the site worked, and I learned a lot from it. I began to feel comfortable answering questions in a few tags. I loved being able to help others and hopefully users would learn something from my answers.

So what does this have to do with changes?

I’ve been both a user and a moderator on Stack Overflow. But now I’m going to be working in a different capacity, I’ve been hired to be a Community Manager by Stack Exchange. I can’t express how excited I am for this. I love the sites, I love the product that they put out there, I love that people get benefit from them. Of course it’s a bit scary because it’s a career change, but the fact that I love what they do is a huge driving force behind my decision. To be able to work on something you care about makes a big difference. I can’t wait to see where these changes take me.

Using powershell to fix server space issues

Sun, 12 Apr 2015 18:30:08 +0000

A few weeks ago, we were running into severe disk space and memory issues on our development servers at work. Our set-up is a bit odd, we have 3 servers - one for the transactions, one for the web interface, and the final one for reporting. Using transactional replication we have databases that can exist on all 3 servers. Yes, it’s can be a real nightmare to maintain, but anyone who works with replication already knows this. Each development team gets its own version of production “in a box”. These copies are on VM slices with limited memory and disk size.

Our issues were happening on the web database and reporting servers. We have replication and sql jobs running, development teams testing, and a variety of other things hitting the servers pretty hard and no memory to process it, so we were running out of disk space. There were several days in a row that I noticed we were down to 20MB of space on our web database server. This lack of resources was causing replication to fail throughout our environment, which resulted in delays to our current sprint.

I’m the early bird on my team and every morning I was executing the following code to “clean-up” our environment via shrinking the log files. Eeeek, the dreaded shrinking of log files. I know you don’t want to do this really ever, but it’s a development environment and it’s how we handle our limited memory issues.

My first check to was see the size of the logs by running DBCC SQLPERF (LOGSPACE). This command gives the transaction log space for all of the databases on the server. Once I got the list I was able to target the databases that had excessively large log files in order to shrink them.

Next, I’d get the name of the log file to shrink via:

1
2
3
4


SELECT name
FROM sys.master_files
WHERE database_id = db_id()
 AND type = 1

Lastly, using the name of the log above I’d run DBCC SHRINKFILE to drop the size of the logs and restore a bit of memory on our box.

Doing this first thing in the morning for multiple databases on multiple servers for multiple days in a row was terrible. So, I decided I needed to automate this. Considering I learned a bit about Powershell from Mike Fal at SQL Saturday Phoenix, I thought “hey, let me take a stab at writing my first Powershell script to do this for me!!”

To be honest, I wasn’t exactly sure where to start so I asked Mike for a bit of help on looping through all databases on a server. He was nice enough to give me a starting script. Using this, I attempted to incorporate my code above into it with the purpose of shrinking all log files on the server. I’m sure there are much better ways to do this but here is my first Powershell script.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


Import-Module sqlps -DisableNameChecking
$dbs = Invoke-SqlCmd -ServerInstance 'servername' -Query "select name from sys.databases where database_id > 4"
foreach($db in $dbs){
$Log = Invoke-Sqlcmd -ServerInstance 'servername' -Database $db.name -Query "select name from sys.master\_files where database\_id = db_id() and type = 1"
$LogName = $Log.name
$query = "DBCC SHRINKFile($LogName, 1)"
Invoke-Sqlcmd -ServerInstance 'servername' -Database $db.name -Query $query
}

This does exactly what I need it to do. I can set it up on our environments to run nightly to shrink the logs so I no longer need to manually “fix” stuff every day. I’ve passed it along to our engineering services group to set this up in all of the development environments when they create them with new production copies.

The script is available on GitHub if anyone is interested in using it, or tweaking it, or criticizing it, etc.

SQL Saturday Adventures

Sun, 12 Apr 2015 16:50:33 +0000

A few weeks bit over a month ago, I took the plunge and attended my very first SQL Saturday, SQL Saturday #370 in Phoenix. I’ve never had a chance to go to any of the local user group events or another SQL Saturday due to scheduling issues, etc. but this year I reshuffled things so I could take part. I’m really glad that I did, I truly learned a lot of stuff and I also realized I know more than I probably give myself credit for. Here’s a few of the highlights from me day:

My first session was Understanding Parameter Sniffing by Benjamin Nevarez. We deal with parameter sniffing a lot at work, so it was something I had experience with. I also gave a Lunch & Learn presentation to co-workers on the topic, which made me curious to see what else I should have covered during my session. To my surprise, I knew most of what Benjamin covered - which was great. His session was listed as “Beginner” but it was nice to know almost everything he discussed.

During lunch, I attended the WIT Panel. For those who don’t know, WIT is for Women in Technology. I’ve rarely worked with female developers, so I went to this session to hear stories from the women in the same field. It was interesting to hear similar stories from the women on the panel, as well as from the audience. I found myself saying yeah, I’ve experienced that repeatedly during the session. Hearing about “Mr. Know-It-All” from multiple panelists made me laugh because we’ve all dealt with that guy. There are so many things I could say about the WIT Panel and being a woman in technology, I’ll have to make that a separate post. My takeaway from this was that my experiences are similar to many other women in tech, which I already knew, but it was nice to hear. I’d recommend this session to anyone attending a future SQL Saturday.

The last session I went to Powershell Tips and Tricks for SQL Server Administration by Mike Fal. I had never used Powershell and had limited exposure to it prior to this session, but I know Mike from DBA.SE and Stack Overflow so I wanted to attend his session and I’m glad that I did. I saw a few things that I would be able to use almost immediately at work with Powershell (I’ll be posting my lame attempts in another post).

I was truly impressed by out SQL Saturday that we had here in Phoenix. I’m glad I forced myself to go and I’ll definitely go to more events in the future.