SQL Server Query Store: 7 Practical DBA Troubleshooting Scenarios

SQL Server Query Store Troubleshooting: Practical DBA Scenarios and T-SQL Examples

A hands-on guide for SQL Server DBAs investigating slow queries, plan regressions, CPU pressure, missing-index warnings, and CLR execution time.

SQL Server Query Store Performance Tuning DBA Guide
Download the file- Query Store Scenarios

SQL Server Query Store captures query text, execution plans, and runtime performance statistics over time. It helps database administrators compare plans, investigate performance changes, and identify queries that deserve deeper analysis. Instead of relying only on the plan currently in cache, a DBA can use Query Store history to understand how a query behaved during different time periods.

Before you begin: Query Store must be enabled and collecting useful data for the database. The available history depends on its configuration, capture mode, cleanup settings, and workload. The examples below use Query Store catalog views and are intended as starting points for investigation—not as a substitute for reviewing the actual execution plan and workload.
In this article
  1. Investigate intermittent slow queries and parameter sniffing
  2. Compare performance before and after a database upgrade
  3. Find queries associated with high CPU usage
  4. Find query plans with missing-index warnings
  5. Identify queries with long compile times
  6. Review recent query performance and plan details
  7. Measure CLR time reported by Query Store
  8. Practical cautions and DBA checklist

1. Investigating Intermittent Slow Queries: Parameter Sniffing

Parameter sniffing occurs when SQL Server compiles a parameterized statement using parameter values available at compilation time. The resulting plan is cached and may later be reused for different values. If data distribution is skewed, one plan may not perform well for every parameter value.

For example, a table might contain one row for most IDs but thousands of rows for a particular ID. An index seek may work well for a selective value, while a scan or a different access strategy may be more efficient for a value that returns many rows. The correct choice depends on the query, indexes, statistics, row counts, and overall plan—not on a fixed rule that scans are always better for large results.

Example setup

Use only in a disposable test database. This demonstration creates a sample table and stored procedure. It drops an existing table with the same name, so do not run it in a production database.

-- TEST ENVIRONMENT ONLY: this drops the table if it exists.
DROP TABLE IF EXISTS dbo.large_table;
GO

CREATE TABLE dbo.large_table
(
    id   int,
    name varchar(30),
    c3   binary(2000) NULL
);
GO

SET NOCOUNT ON;

DECLARE @i int = 1;
WHILE @i < 10000
BEGIN
    INSERT INTO dbo.large_table (id, name)
    VALUES (@i, 'test');

    SET @i += 1;
END;
GO

-- Add many rows for one value to create skewed data.
INSERT INTO dbo.large_table (id, name)
VALUES (100001, 'test');
GO 10000

CREATE INDEX IX_large_table_id
ON dbo.large_table(id);
GO

CREATE OR ALTER PROCEDURE dbo.test_proc (@i int)
AS
BEGIN
    SET NOCOUNT ON;
    SELECT *
    FROM dbo.large_table
    WHERE id = @i;
END;
GO

Execute the procedure with different parameter values and compare the actual execution plans and elapsed time:

EXEC dbo.test_proc @i = 10;
EXEC dbo.test_proc @i = 100001;
GO

The first execution can influence the cached plan used by later executions. However, the exact plan selected depends on SQL Server version, compatibility level, statistics, optimizer decisions, and database configuration. Reproduce and measure the behavior in your own environment rather than assuming a particular plan will always appear.

Step 1: Find the query in Query Store

SELECT query_text_id, query_sql_text
FROM sys.query_store_query_text
WHERE query_sql_text LIKE '%large_table%';

Use the returned query_text_id to locate the associated query:

SELECT query_id, query_text_id, object_id
FROM sys.query_store_query
WHERE query_text_id = <query_text_id>;

Replace the placeholder with the actual ID returned by the first query.

Step 2: Compare plans for the query

SELECT
    q.query_id,
    OBJECT_NAME(q.object_id) AS containing_object,
    qt.query_sql_text,
    p.plan_id,
    p.is_forced_plan,
    p.last_compile_start_time,
    p.last_execution_time,
    TRY_CONVERT(xml, p.query_plan) AS query_plan_xml
FROM sys.query_store_query AS q
JOIN sys.query_store_query_text AS qt
    ON qt.query_text_id = q.query_text_id
JOIN sys.query_store_plan AS p
    ON p.query_id = q.query_id
WHERE qt.query_sql_text LIKE '%large_table%'
ORDER BY q.query_id, p.plan_id;

Compare the plans and their runtime history. Check logical reads, duration, CPU, execution counts, and the parameter values or cardinality assumptions visible in the plan when available. A plan that is good for one parameter may be poor for another, but multiple plans alone do not prove parameter sniffing.

Step 3: Force a known-good plan only after validation

EXEC sys.sp_query_store_force_plan
    @query_id = <query_id>,
    @plan_id  = <plan_id>;
Important disclaimer: Plan forcing must be validated and may not always make a query faster. A forced plan can become unsuitable as data distribution, indexes, statistics, or workload change. Test the candidate plan, monitor its impact, and have a rollback plan. Remove forcing if it causes a regression:
EXEC sys.sp_query_store_unforce_plan
    @query_id = <query_id>,
    @plan_id  = <plan_id>;

In SQL Server Management Studio, open the database's Query Store reports, review tracked or regressed queries, compare available plans, and use Force Plan only after verifying that the selected plan is consistently appropriate. UI labels vary by SSMS version.



2. Investigating Regressions After a Database Upgrade

Query Store can help identify queries whose performance changed after an upgrade, compatibility-level change, service pack, or cumulative update. The most useful comparison is between a workload baseline and the same workload after the change.

  1. Confirm Query Store is enabled and retaining sufficient history before the change.
  2. Run representative application workloads and record a baseline.
  3. Perform the planned upgrade or configuration change.
  4. Run the same or comparable workload after the change.
  5. Review Query Store's Regressed Queries report and compare plans and runtime statistics.
  6. Validate any proposed fix against representative workload and application requirements.

If Query Store was enabled only after the upgrade, it may not contain the earlier baseline. Also check whether the database compatibility level changed; an engine upgrade and a compatibility-level change are separate factors that can affect query optimization.

3. Finding Queries Associated with High CPU Usage

The following query ranks Query Store queries by estimated total CPU time over runtime-statistics intervals that overlap the last hour. Query Store stores runtime statistics in intervals, so the result is an interval-based approximation rather than a precise live-session measurement. CPU values in Query Store are reported in microseconds.

DECLARE @SinceUtc datetimeoffset =
    DATEADD(hour, -1, SYSUTCDATETIME());

WITH AggregatedCPU AS
(
    SELECT
        q.query_id,
        SUM(CONVERT(float, rs.count_executions)
            * rs.avg_cpu_time) AS total_cpu_time,
        COUNT(DISTINCT p.plan_id) AS number_of_plans
    FROM sys.query_store_query AS q
    JOIN sys.query_store_plan AS p
        ON p.query_id = q.query_id
    JOIN sys.query_store_runtime_stats AS rs
        ON rs.plan_id = p.plan_id
    JOIN sys.query_store_runtime_stats_interval AS rsi
        ON rsi.runtime_stats_interval_id =
           rs.runtime_stats_interval_id
    WHERE rsi.end_time >= @SinceUtc
      AND rs.execution_type_desc = 'Regular'
    GROUP BY q.query_id
)
SELECT TOP (25)
    q.query_id,
    OBJECT_NAME(q.object_id) AS containing_object,
    qt.query_sql_text,
    a.total_cpu_time,
    a.number_of_plans
FROM AggregatedCPU AS a
JOIN sys.query_store_query AS q
    ON q.query_id = a.query_id
JOIN sys.query_store_query_text AS qt
    ON qt.query_text_id = q.query_text_id
ORDER BY a.total_cpu_time DESC;

Use the results to prioritize investigation, then inspect the plans and query-level runtime statistics. High total CPU can be caused by a query that runs many times, a query that consumes substantial CPU per execution, or both. For current active requests, also inspect DMVs such as sys.dm_exec_requests; Query Store is primarily historical and aggregated.

Disk I/O and memory pressure

Query Store runtime statistics can help identify expensive queries, but Query Store alone does not provide a complete diagnosis of server-wide disk or memory pressure. For I/O-related query analysis, compare logical reads, physical reads where available in the relevant SQL Server version, execution count, and plan operators. For memory issues, inspect actual plans for memory grants and spills and use appropriate DMVs, Extended Events, and operating-system monitoring. Confirm the columns supported by your SQL Server version before adapting a query.



4. Finding Query Plans with Missing-Index Warnings

Execution plans may contain missing-index recommendations. The following example searches recent Query Store plans for a missing-index element and returns the estimated impact value recorded in the plan.

SELECT
    qt.query_sql_text,
    TRY_CONVERT(xml, p.query_plan).value(
      'declare default element namespace
       "http://schemas.microsoft.com/sqlserver/2004/07/showplan";
       (/ShowPlanXML/BatchSequence/Batch/Statements/StmtSimple/
        QueryPlan/MissingIndexes/MissingIndexGroup/@Impact)[1]',
      'decimal(18,4)') AS estimated_missing_index_impact,
    q.query_id,
    p.plan_id,
    rs.last_execution_time,
    rs.avg_duration,
    TRY_CONVERT(xml, p.query_plan) AS query_plan_xml
FROM sys.query_store_query_text AS qt
JOIN sys.query_store_query AS q
    ON q.query_text_id = qt.query_text_id
JOIN sys.query_store_plan AS p
    ON p.query_id = q.query_id
JOIN sys.query_store_runtime_stats AS rs
    ON rs.plan_id = p.plan_id
WHERE rs.last_execution_time >=
      DATEADD(hour, -1, SYSUTCDATETIME())
  AND TRY_CONVERT(xml, p.query_plan).exist(
      'declare default element namespace
       "http://schemas.microsoft.com/sqlserver/2004/07/showplan";
       /ShowPlanXML/BatchSequence/Batch/Statements/StmtSimple/
       QueryPlan/MissingIndexes/MissingIndexGroup/MissingIndex') = 1
ORDER BY rs.avg_duration DESC;
Do not create every suggested index automatically. Review equality, inequality, and included columns; existing indexes; write overhead; storage; and whether the query is important to the workload. Missing-index impact is an optimizer estimate, not a guaranteed performance improvement. This example can return multiple rows for a query because runtime statistics and plans may have multiple records.

5. Finding Queries with Long Compile Times

Compilation overhead can matter when queries compile frequently or have complex optimization requirements. The following query lists plans with high average compile duration among queries executed recently.

SELECT TOP (10)
    qt.query_sql_text,
    q.query_id,
    p.plan_id,
    p.avg_compile_duration,
    rs.last_execution_time,
    rs.avg_duration
FROM sys.query_store_query_text AS qt
JOIN sys.query_store_query AS q
    ON q.query_text_id = qt.query_text_id
JOIN sys.query_store_plan AS p
    ON p.query_id = q.query_id
JOIN sys.query_store_runtime_stats AS rs
    ON rs.plan_id = p.plan_id
WHERE rs.last_execution_time >=
      DATEADD(hour, -1, SYSUTCDATETIME())
ORDER BY p.avg_compile_duration DESC;

Interpretation: this is a screening query, not proof that compilation caused the application's delay. Query Store's compile-duration data is associated with plans and may not represent a compile event for every execution. Correlate the result with compilation frequency, recompilation events, query complexity, parameterization, and observed application latency. Confirm that the relevant columns are available in your SQL Server version.

6. Reviewing Recent Query Plans and Runtime Statistics

Use this query to review recent Query Store activity, including the query text, plan ID, execution timing, and runtime metrics available in the view.

SELECT TOP (100)
    qt.query_sql_text,
    q.query_id,
    p.plan_id,
    p.is_forced_plan,
    rs.last_execution_time,
    rs.count_executions,
    rs.avg_duration,
    rs.max_duration,
    rs.avg_cpu_time,
    rs.max_cpu_time,
    rs.avg_logical_io_reads,
    rs.max_logical_io_reads,
    TRY_CONVERT(xml, p.query_plan) AS query_plan_xml
FROM sys.query_store_query_text AS qt
JOIN sys.query_store_query AS q
    ON q.query_text_id = qt.query_text_id
JOIN sys.query_store_plan AS p
    ON p.query_id = q.query_id
JOIN sys.query_store_runtime_stats AS rs
    ON rs.plan_id = p.plan_id
ORDER BY rs.last_execution_time DESC;

Query Store view columns differ across SQL Server releases. If a column is not available in your version, remove it or consult the documentation for that version. Runtime statistics are stored by interval and may produce multiple rows per plan; aggregate them appropriately when comparing periods.

7. Measuring Time Reported for CLR Queries

Suppose an UPDATE statement calls a user-defined function that invokes a SQL CLR assembly. Query Store runtime statistics may expose CLR timing metrics for supported SQL Server versions. For example:

-- Example only: use the actual table and function in your environment.
UPDATE dbo.Account_tbl_ENC
SET AcctNumber_RNDL = dbo.TryEncrypt(AcctNumber, 'None');

To review CLR-related metrics recorded in Query Store:

SELECT TOP (100)
    qt.query_sql_text,
    q.query_id,
    p.plan_id,
    rs.last_execution_time,
    rs.avg_clr_time,
    rs.max_clr_time,
    rs.avg_cpu_time,
    rs.max_cpu_time,
    rs.avg_duration,
    rs.max_duration,
    TRY_CONVERT(xml, p.query_plan) AS query_plan_xml
FROM sys.query_store_query_text AS qt
JOIN sys.query_store_query AS q
    ON q.query_text_id = qt.query_text_id
JOIN sys.query_store_plan AS p
    ON p.query_id = q.query_id
JOIN sys.query_store_runtime_stats AS rs
    ON rs.plan_id = p.plan_id
ORDER BY rs.last_execution_time DESC;

Review the CLR time alongside total duration and CPU time. The values are Query Store runtime statistics, not a per-line profiler trace. If the CLR metrics are unavailable in your SQL Server version or environment, verify the supported schema and use suitable profiling or Extended Events methods for deeper analysis.

8. Practical DBA Checklist

  • Verify Query Store is enabled, collecting data, and retaining the period you need.
  • Use a representative time window and account for UTC timestamps in the examples.
  • Compare query text, plan IDs, execution counts, CPU, duration, and I/O—not only one metric.
  • Inspect actual execution plans and validate findings with application owners.
  • Test plan forcing and index changes in a safe environment before production use.
  • Capture a performance baseline before upgrades or compatibility-level changes.
  • Remember that Query Store aggregates historical data; use live DMVs and other monitoring tools for current blocking and active-request investigations.
  • Check the SQL Server version and Query Store feature availability before running scripts.

Conclusion

Query Store is a valuable tool for SQL Server performance troubleshooting. It helps DBAs investigate plan changes, find expensive queries, review missing-index warnings, examine compile-time metrics, and understand CLR timing where supported. The best results come from combining Query Store history with execution-plan analysis, live monitoring, workload knowledge, and controlled testing.

Do not force a plan or add an index solely because a query appears in a report. Confirm the root cause, measure the change, and monitor the workload after implementation.


Disclaimer: The scripts are examples for learning and troubleshooting. Validate them against your SQL Server version, schema, workload, permissions, and operational standards before production use. Performance outcomes are environment-dependent; no script, index recommendation, or forced plan is guaranteed to improve performance.

Fix SQL Server In-Place Upgrade Error: msodbcsql.msi Missing

Fix SQL Server In-Place Upgrade Error: msodbcsql.msi Missing

During a SQL Server in-place upgrade, SQL Server Setup may stop during the Upgrade Progress stage with the following Windows Installer error:

The feature you are trying to use is on a network resource that is unavailable.

The installer then asks for msodbcsql.msi. This can happen even when Microsoft ODBC Driver 17 for SQL Server is already installed.



Figure 1: SQL Server Setup pauses during Upgrade Progress and requests the missing msodbcsql.msi installation package.

The most common reason is that Windows Installer cannot locate the original installation source. For example, it may look for a temporary folder such as:

C:\Windows\TEMP\IXP000.TMP\

Windows Installer may retain a cached MSI package under:

C:\Windows\Installer\

This article explains how to identify the installed ODBC driver version, locate the cached MSI package, and troubleshoot the missing-installer prompt.

1. Check the Installed ODBC Driver Version

Open PowerShell as Administrator and run the following command. This checks the Windows uninstall registry without triggering the Windows Installer consistency checks associated with Win32_Product.

$paths = @(
    "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\*",
    "HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall\*"
)

Get-ItemProperty $paths -ErrorAction SilentlyContinue |
    Where-Object {
        $_.DisplayName -match "ODBC Driver \d+ for SQL Server"
    } |
    Select-Object DisplayName, DisplayVersion, InstallLocation

Review the output to confirm whether Microsoft ODBC Driver 17 for SQL Server is installed. The displayed version depends on the driver release installed on the server.

2. Locate Cached MSI Packages

Windows Installer may store cached MSI packages in C:\Windows\Installer\. Their filenames may be randomly generated, so do not expect every cached package to be named msodbcsql.msi.

Get-ChildItem "C:\Windows\Installer" -Filter "*.msi" -File |
    Select-Object FullName, Length, LastWriteTime

Do not select an arbitrary MSI based only on its filename or modification date.

3. Identify the Correct Cached MSI

Windows Installer product registration may contain the cached package path in the LocalPackage property. Run this PowerShell command to look for the registered ODBC Driver 17 product:

$installerPath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData"

Get-ChildItem $installerPath -ErrorAction SilentlyContinue |
    ForEach-Object {
        $productsPath = Join-Path $_.PSPath "Products"

        if (Test-Path $productsPath) {
            Get-ChildItem $productsPath -ErrorAction SilentlyContinue |
                ForEach-Object {
                    $installProperties = Join-Path $_.PSPath "InstallProperties"

                    if (Test-Path $installProperties) {
                        $product = Get-ItemProperty $installProperties `
                            -ErrorAction SilentlyContinue

                        if ($product.DisplayName -like "*ODBC Driver 17*") {
                            [PSCustomObject]@{
                                ProductName   = $product.DisplayName
                                Version       = $product.DisplayVersion
                                InstallSource = $product.InstallSource
                                LocalPackage  = $product.LocalPackage
                            }
                        }
                    }
                }
        }
    }

If the product registration is available, inspect the returned LocalPackage path and verify that the file exists.

Test-Path "C:\Windows\Installer\<cached-package>.msi"

Replace the placeholder with the actual path returned by the previous command. If no result is returned, do not assume that the driver is missing; the registration layout or installed package may differ.

4. Provide the MSI File to SQL Server Setup

  1. Return to the Windows Installer dialog.
  2. Click Browse.
  3. Select the verified cached MSI package if it is available and accepted by Windows Installer.
  4. Click OK and allow the installer to continue.
  5. Return to SQL Server Setup and check whether the upgrade progresses.

Important: Cached packages may have generated filenames. Do not rename or replace cached installer files without verifying the package identity and installer requirements. A cached MSI may not be accepted as a replacement source in every repair scenario.

5. Alternative Resolution: Download the Matching ODBC Driver

If the cached package is missing or rejected, obtain a compatible installer from Microsoft's official documentation:

Download Microsoft ODBC Driver for SQL Server

Before repairing or reinstalling, verify the installed driver version, Windows architecture, required installer package, and application dependencies.

Do not uninstall the existing driver as your first troubleshooting step. Other applications may depend on it, and removing it can introduce additional issues.

6. Validate the Driver Installation

After resolving the installer prompt, run the following PowerShell command to verify the registered driver version:

$paths = @(
    "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\*",
    "HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall\*"
)

Get-ItemProperty $paths -ErrorAction SilentlyContinue |
    Where-Object {
        $_.DisplayName -match "ODBC Driver \d+ for SQL Server"
    } |
    Select-Object DisplayName, DisplayVersion

You can also inspect registered ODBC drivers:

Get-OdbcDriver |
    Where-Object { $_.Name -like "*SQL Server*" } |
    Select-Object Name, Platform, Attribute

Resume or rerun SQL Server Setup as appropriate. Review the setup summary and logs for any remaining failures.

SQL Server Setup logs are typically located under:

C:\Program Files\Microsoft SQL Server\<version>\Setup Bootstrap\Log\

The exact directory depends on the SQL Server version and installation configuration.

7. Troubleshooting Summary

Symptom Recommended Action
Setup requests msodbcsql.msi Identify the installed ODBC driver version.
Original installation source is unavailable Check the Windows Installer cache and product registration.
Cached MSI cannot be identified Inspect the LocalPackage property in installer registration.
Cached package is missing or rejected Obtain a compatible installer from Microsoft.
Upgrade still fails Review SQL Server Setup logs for the next reported failure.

Conclusion

The msodbcsql.msi prompt during a SQL Server in-place upgrade does not necessarily mean that Microsoft ODBC Driver 17 is missing. Windows Installer may simply be unable to locate the original installation source.

The recommended troubleshooting sequence is to identify the installed driver, inspect installer registration, locate and verify the cached MSI, and use a compatible Microsoft installer if necessary.

Best practice: Before upgrading a production SQL Server instance, verify prerequisites, preserve required installation media, take appropriate backups, and ensure a tested recovery plan is available.

Disclaimer: Test the resolution in a non-production environment before applying it to production.

SQL Server 2016 to SQL Server 2022 Upgrade: Complete DBA Runbook with SSIS, SSRS Migration & Troubleshooting

SQL Server 2016 to SQL Server 2022 Upgrade: Complete DBA Runbook with SSIS, SSRS Migration & Troubleshooting

Introduction

Upgrading SQL Server in a real environment involves much more than running the installation wizard. DBAs must prepare backups, validate dependencies, manage application downtime, troubleshoot setup failures, and verify that databases, SQL Server Agent jobs, SSIS packages, and reporting services work correctly after the upgrade.

In this article, I will walk through an actual SQL Server 2016 to SQL Server 2022 upgrade scenario, including a Database Engine startup failure caused by a missing SSISDB database principal and a separate SSRS Evaluation edition issue.

This guide is intended for SQL Server DBAs who want a practical reference for planning, executing, troubleshooting, and validating an upgrade.

Important: The Database Engine upgrade and SSISDB recovery were completed in the documented environment. SSRS migration and several post-upgrade validations remained open at the time of documentation.

1. Environment Details

ComponentDetails
Source SQL ServerSQL Server 2016 Developer Edition
Target SQL ServerSQL Server 2022 Developer Edition
Source version13.3.6300.2
Target version16.0.1000.6 RTM
InstanceDefault instance (MSSQLSERVER)
Operating systemWindows Server 2016 Datacenter
SSISExisting SSISDB catalog
Source SSRSSQL Server Reporting Services 2017 Evaluation
Target SSRSSQL Server Reporting Services 2022 Developer
Upgrade methodIn-place Database Engine upgrade

Note: Developer Edition is intended for development and testing, not production workloads. Confirm the appropriate edition and licensing before applying this procedure to a production environment.

2. Pre-Upgrade Checklist

Before starting the upgrade, complete the following checks:

  • Obtain change approval and confirm the maintenance window.
  • Notify application owners about the expected downtime.
  • Take full backups of all critical user databases.
  • Back up system databases according to the recovery plan.
  • Back up SSISDB and its database master key.
  • Back up the ReportServer databases and SSRS encryption key.
  • Document SQL Server Agent jobs, schedules, operators, alerts, and proxies.
  • Record linked servers, credentials, endpoints, and service accounts.
  • Document SSIS packages, projects, environments, and connection managers.
  • Document SSRS reports, subscriptions, data sources, URLs, and configuration.
  • Confirm sufficient disk space for setup, backups, and database growth.
  • Verify that SQL Server 2022 and SSRS installation media are available.
  • Prepare a tested rollback or recovery plan.

Never begin an upgrade without a verified recovery strategy.

3. Back Up SSISDB

SSISDB contains the Integration Services catalog, including deployed projects, environments, parameters, and execution information.

BACKUP DATABASE [SSISDB]
TO DISK = 'L:\Backups\SSISDB_PreUpgrade_FULL.bak'
WITH INIT, COMPRESSION, CHECKSUM, STATS = 10;
GO

Validate the backup using an approved recovery procedure. Where appropriate, use RESTORE VERIFYONLY and perform a test restore in a separate recovery environment.

Back up the SSISDB database master key

USE [SSISDB];
GO

BACKUP MASTER KEY
TO FILE = 'L:\Backups\SSISDB_DMK.key'
ENCRYPTION BY PASSWORD = 'Strong-Unique-Password';
GO

Replace the example password with a strong, unique password before execution. Store the password and key backup securely and separately.

Verify database master key protection:

SELECT
    name,
    is_master_key_encrypted_by_server
FROM sys.databases
WHERE name = 'SSISDB';
GO

A value of 1 indicates that the database master key is encrypted by the SQL Server Service Master Key.

4. Back Up SSRS Databases and Encryption Key

Before changing the Reporting Services installation, protect its databases and configuration.

BACKUP DATABASE [ReportServer]
TO DISK = 'L:\Backups\ReportServer_PreUpgrade.bak'
WITH INIT, COMPRESSION, CHECKSUM, STATS = 10;
GO

BACKUP DATABASE [ReportServerTempDB]
TO DISK = 'L:\Backups\ReportServerTempDB_PreUpgrade.bak'
WITH INIT, COMPRESSION, CHECKSUM, STATS = 10;
GO

The actual environment also contained backups named ReportServer_952026.bak and ReportServerTempdb_952026.bak.

Use Report Server Configuration Manager to back up the SSRS encryption key. Record the service account, Web Service URL, Web Portal URL, subscriptions, stored credentials, and custom configuration.

The encryption key is particularly important because it protects encrypted Reporting Services content, including stored credentials.

5. Perform Pre-Upgrade Health Checks

Check the SQL Server version and edition:

SELECT
    @@VERSION AS SQLVersion,
    SERVERPROPERTY('ProductVersion') AS ProductVersion,
    SERVERPROPERTY('ProductLevel') AS ProductLevel,
    SERVERPROPERTY('Edition') AS Edition;
GO

Check database status:

SELECT
    name,
    state_desc,
    user_access_desc,
    recovery_model_desc
FROM sys.databases
ORDER BY name;
GO

Check SSISDB specifically:

SELECT
    name,
    state_desc,
    user_access_desc,
    recovery_model_desc
FROM sys.databases
WHERE name = 'SSISDB';
GO

Resolve existing database state, backup, or recovery issues before proceeding.

6. Handle the ODBC Driver Setup Prerequisite

During this upgrade, SQL Server Setup requested the source MSI package for ODBC Driver 17.

The installed driver version was 17.10.6.1, and the cached Windows Installer package was located at C:\Windows\Installer\d326ee9d.msi.

If Setup requests the original MSI source, copy the cached package to a temporary directory rather than modifying or deleting the Windows Installer cache.

New-Item -ItemType Directory `
    -Path "C:\Temp\ODBC17" -Force

Copy-Item `
    "C:\Windows\Installer\d326ee9d.msi" `
    "C:\Temp\ODBC17\msodbcsql.msi"

If Setup requests the folder containing msodbcsql.msi, provide the temporary directory. Confirm that the copied package is the correct installer before continuing.

7. Start the SQL Server 2016 to 2022 Upgrade

Run SQL Server 2022 Setup as an administrator.

  1. Select the upgrade option for an existing SQL Server installation.
  2. Select the existing MSSQLSERVER instance.
  3. Review the selected features and prerequisites.
  4. Review any warnings and resolve blocking issues.
  5. Start the upgrade during the approved maintenance window.
  6. Monitor SQL Server Setup logs and the SQL Server ERRORLOG.

During the upgrade, the Database Engine failed to start and Setup displayed:

Wait on the Database Engine recovery handle failed.

This message is a symptom rather than a complete diagnosis. The SQL Server ERRORLOG revealed the underlying failure.

8. Troubleshoot the ModuleSigner Failure

The SQL Server ERRORLOG showed that ISServer_upgrade.sql failed while executing against SSISDB.

The relevant error was:

Error: 15151
Cannot find the user 'ModuleSigner',
because it does not exist or you do not have permission.

The failed statement was:

GRANT SELECT
ON [catalog].[catalog_properties]
TO ModuleSigner;

The script-level upgrade then failed with error 912, followed by error 3417 indicating that the master database could not be recovered.

Further investigation showed that SSISDB recovery completed successfully and DBCC CHECKDB reported no errors. The missing ModuleSigner database principal was identified as the immediate cause of the upgrade-script failure.

Step 1: Start SQL Server with trace flag 902

For this specific upgrade-script failure, the documented recovery procedure temporarily started SQL Server with trace flag 902:

NET START MSSQLSERVER /T902

Trace flag 902 bypasses upgrade scripts during startup. It is a temporary troubleshooting measure, not a normal operating configuration. Use it only as part of a controlled recovery procedure after reviewing the ERRORLOG and applicable Microsoft guidance.

Step 2: Verify the SSIS signing certificate

USE [SSISDB];
GO

SELECT
    name,
    subject,
    expiry_date
FROM sys.certificates
WHERE name = 'MS_SQLISSigningCertificate';
GO

The certificate existed in this environment. Verify the existing certificate before taking any action. Do not recreate or replace a certificate that is already present.

Step 3: Check whether ModuleSigner exists

USE [SSISDB];
GO

SELECT
    name,
    type_desc,
    authentication_type_desc
FROM sys.database_principals
WHERE name = 'ModuleSigner';
GO

The query returned no rows, confirming that the database principal was missing.

Step 4: Recreate the certificate-mapped database user

Because the existing certificate was present, the documented remediation was:

USE [SSISDB];
GO

CREATE USER [ModuleSigner]
FOR CERTIFICATE [MS_SQLISSigningCertificate];
GO

This creates the database user mapped to the existing certificate. Do not create the user blindly in another environment; first confirm the exact failure, certificate, database state, and applicable Microsoft recovery instructions.

Verify the result:

USE [SSISDB];
GO

SELECT
    name,
    type_desc,
    authentication_type_desc
FROM sys.database_principals
WHERE name = 'ModuleSigner';
GO

The user should now appear in the results.

9. Verify SSIS Cleanup Objects and Restore Multi-User Access

The recovery procedure also checked the SSIS cleanup database user:

USE [SSISDB];
GO

SELECT
    name,
    type_desc,
    authentication_type_desc
FROM sys.database_principals
WHERE name = '##MS_SSISServerCleanupJobUser##';
GO

Then it checked the corresponding server login:

USE [master];
GO

SELECT
    name,
    type_desc
FROM sys.server_principals
WHERE name = '##MS_SSISServerCleanupJobLogin##';
GO

Both cleanup objects already existed in the documented environment, so no changes were required.

Next, check the SSISDB database state:

SELECT
    name,
    state_desc,
    user_access_desc
FROM sys.databases
WHERE name = 'SSISDB';
GO

SSISDB was ONLINE but in SINGLE_USER mode.

After confirming that the recovery procedure could safely restore normal access, the database was returned to multi-user mode:

USE [master];
GO

ALTER DATABASE [SSISDB]
SET MULTI_USER WITH ROLLBACK IMMEDIATE;
GO

Caution: WITH ROLLBACK IMMEDIATE terminates other database connections and rolls back their transactions. Use it only during an approved recovery window after assessing the impact.

10. Restart SQL Server Normally

After completing the remediation, stop and start SQL Server without trace flag 902.

NET STOP MSSQLSERVER
NET START MSSQLSERVER

Do not leave /T902 configured for normal startup. Confirm that SQL Server starts normally and that the upgrade scripts complete successfully.

11. Run SQL Server Setup Repair

Once the Database Engine could start normally, SQL Server Setup Repair was executed.

L:
cd \Backups\x64\setup
setup.exe /ACTION=REPAIR /INSTANCENAME=MSSQLSERVER

Repair is different from repeating the original upgrade operation. Confirm the installation media path, instance name, and Setup recommendations before executing it.

During Repair, Setup encountered a separate SQL Telemetry service startup error. The message indicated that the service could not start because it was disabled or had no enabled devices associated with it.

Check the service configuration:

sc query SQLTELEMETRY
sc qc SQLTELEMETRY

If the service is confirmed to be disabled and Setup requires it, review the organization's service policy and enable it only if approved.

sc config SQLTELEMETRY start= auto
net start SQLTELEMETRY

Retry Setup Repair after resolving the service issue. Do not make unrelated SSISDB changes to resolve a Windows service startup error.

12. Validate the Database Engine Upgrade

After the repair and restart, verify the SQL Server version and edition:

SELECT
    @@VERSION AS SQLVersion,
    SERVERPROPERTY('ProductVersion') AS ProductVersion,
    SERVERPROPERTY('ProductLevel') AS ProductLevel,
    SERVERPROPERTY('Edition') AS Edition;
GO

Check database status:

SELECT
    name,
    state_desc,
    user_access_desc,
    recovery_model_desc
FROM sys.databases
ORDER BY name;
GO

Check SSISDB:

SELECT
    name,
    state_desc,
    user_access_desc
FROM sys.databases
WHERE name = 'SSISDB';
GO

The documented result was:

  • SQL Server 2022 RTM.
  • Product version 16.0.1000.6.
  • Developer Edition, 64-bit.
  • SSISDB ONLINE.
  • SSISDB MULTI_USER.
  • The missing ModuleSigner principal was recreated.

These results confirmed that the Database Engine recovery was successful. They do not, by themselves, prove that every application, SSIS package, or reporting workload is functioning correctly.

13. Validate SSIS Packages and SQL Server Agent Jobs

After upgrading the Database Engine, validate the Integration Services catalog and dependent jobs.

For SSIS, verify:

  • SSISDB is accessible.
  • Folders, projects, environments, parameters, and references are present.
  • Representative packages execute successfully.
  • SQL Server Agent jobs that run SSIS packages succeed.
  • Package execution history and error messages are normal.
  • Connection managers and encrypted connection information work.
  • Required proxies and credentials are valid.

Check SQL Server Agent jobs:

SELECT
    name,
    enabled,
    date_created,
    date_modified
FROM msdb.dbo.sysjobs
ORDER BY name;
GO

Confirm that SQL Server Agent is running, critical jobs remain enabled, schedules are unchanged, and job owners and credentials are valid.

14. SSRS Upgrade: A Separate Issue

SQL Server Reporting Services 2017 and later are standalone products. Their installation and migration must be handled separately from the Database Engine upgrade.

In the documented environment, the existing SSRS installation was:

  • Product: Microsoft SQL Server Reporting Services.
  • Version: 14.0.8444.41957.
  • Edition: Evaluation.
  • SKU: SsrsEvaluation.

The SSRS 2022 installer displayed the following error:

Evaluation period has expired. Please upgrade to different edition before attempting to upgrade again.

This was not a Database Engine failure. The existing SSRS Evaluation installation had expired, and the installer blocked the upgrade.

Protect SSRS before making changes

Before attempting an edition change, removal, or reinstallation:

  1. Verify the ReportServer database backup.
  2. Back up the SSRS encryption key.
  3. Record the SSRS service account.
  4. Record the Web Service URL and Web Portal URL.
  5. Back up custom configuration files.
  6. Document subscriptions, data sources, schedules, and stored credentials.
  7. Confirm a tested recovery procedure.

Preferred path: Change the edition if supported

For a development or testing environment, evaluate SQL Server Reporting Services Developer Edition, subject to licensing requirements.

Check whether the installed SSRS setup supports an edition change. If the expired Evaluation installation prevents the change, do not delete the ReportServer databases or encryption key.

Alternative path: Migration or reinstallation

If an edition change is not possible, plan a controlled SSRS migration or reinstallation.

  1. Validate the ReportServer and encryption-key backups.
  2. Record the existing configuration and service dependencies.
  3. Follow the approved procedure to remove or replace the blocked installation.
  4. Install the appropriate SSRS version and edition.
  5. Configure the service account and URLs.
  6. Connect to or restore the ReportServer database using the documented migration procedure.
  7. Restore the encryption key.
  8. Restore applicable custom configuration.
  9. Validate reports, data sources, credentials, subscriptions, schedules, and portal access.

Do not improvise changes to the ReportServer databases or encryption keys. Resolve any encryption-key restoration issue before deleting or recreating report server databases.

Status in this upgrade: SSRS migration remained open and required completion and validation.

15. Post-Upgrade Validation Checklist

Database Engine

  • SQL Server starts normally.
  • SQL Server version and edition are correct.
  • All databases are online or in their expected operational states.
  • Application logins and connectivity work.
  • Read/write transactions and critical stored procedures succeed.

SSIS

  • SSISDB is online and in multi-user mode.
  • Projects, environments, parameters, and references are present.
  • Representative packages execute successfully.
  • SQL Server Agent jobs and schedules are validated.

SSRS

  • The Reporting Services service is running.
  • Web Service and Web Portal URLs work.
  • Reports and shared data sources work.
  • Stored credentials and subscriptions are validated.
  • Scheduled execution and email delivery work where configured.
  • Application users can access reports with the expected permissions.

Performance

  • Compare CPU, memory, I/O, and wait statistics against the baseline where available.
  • Review Query Store and execution-plan regressions.
  • Check index and statistics maintenance.
  • Validate tempdb configuration and file growth.
  • Confirm max server memory, MAXDOP, and cost threshold settings.
  • Monitor SQL Server Agent and SSIS execution durations.

Do not automatically change every database compatibility level immediately after the engine upgrade. Test application behavior first and plan compatibility-level changes separately.

16. SQL Server 2022 Patching

The environment documented in this runbook was running SQL Server 2022 RTM, build 16.0.1000.6.

Before considering the upgrade complete, review the current SQL Server 2022 cumulative update, its known issues, and the organization's approved patching policy.

SELECT
    SERVERPROPERTY('ProductVersion') AS ProductVersion,
    SERVERPROPERTY('ProductLevel') AS ProductLevel;
GO

Apply an approved cumulative update in a separate maintenance window after the upgrade has been validated.

For current build information, refer to Microsoft's SQL Server 2022 build documentation:

SQL Server 2022 build versions

17. Rollback and Recovery Considerations

An in-place SQL Server upgrade requires a tested recovery plan.

Do not attempt to downgrade SQL Server by simply installing SQL Server 2016 over SQL Server 2022.

If the upgraded engine is stable but an application issue exists, investigate the issue before initiating a full rollback.

If a complete rollback is required, follow the organization's documented restore or rebuild procedure using validated backups.

For SSISDB, preserve the database backup and database master key backup. For SSRS, preserve the ReportServer database backups and encryption key.

Never drop SSISDB or ReportServer databases as an initial troubleshooting step.

18. Final Status of the Documented Upgrade

ActivityStatus
Database Engine upgradeCompleted
SQL Server 2022 version verificationPassed
SSISDB online and multi-user validationPassed
ModuleSigner remediationCompleted
SSIS cleanup objectsVerified
SSIS package validationPending
SQL Server Agent job validationPending
SSRS Evaluation edition issueOpen
SSRS 2022 installation and configurationOpen
SSRS report and subscription validationOpen
Application connectivity validationPending
SQL Server cumulative updatePlanned

This distinction is important: a successful Database Engine upgrade does not automatically mean that SSIS, SSRS, and all application integrations have been fully validated.

Conclusion

This SQL Server 2016 to SQL Server 2022 upgrade demonstrated the importance of investigating the SQL Server ERRORLOG rather than relying only on the generic Setup failure message.

The Database Engine startup issue was traced to a missing ModuleSigner principal in SSISDB. After verifying the existing signing certificate, recreating the certificate-mapped user, restoring normal database access, restarting SQL Server, and running Setup Repair, the Database Engine was verified on SQL Server 2022.

The separate SSRS Evaluation expiration required its own migration or edition-change plan.

For DBAs, the key lessons are straightforward: prepare and verify backups, understand the dependencies between SQL Server components, use controlled recovery procedures, validate workloads after the upgrade, and keep unresolved tasks visible until they are completed.

I hope this practical runbook helps other SQL Server DBAs plan and troubleshoot their own upgrade projects.

More SQL Server DBA articles: https://sqldbanow.com/

References