Written by • December 30, 2020• 18:44• Database development, SQL Server, Statements, Tables

Use Cases for SQL Server MERGE Statement: Syncing Online and History Tables

HomeDatabase development, SQL Server, Statements, TablesUse Cases for SQL Server MERGE Statement: Syncing Online and History Tables

INTRODUCTION

The SQL Server MERGE statement is an incredibly useful tool for carrying out DML operations based on comparing two tables or two data sets. Usage of this statement is actually like performing multiple operations in a single statement.

This article will explore three use cases that border on ensuring data between an online table and a history table in sync.

Let’s describe the scenario in a few outlined statements which would be familiar to many DBAs or developers:

  1. The source table called Tran captures transactions happening in real-time on an ongoing basis.
  2. The agreed retention period for table Tran is one month. Tran must be purged at the end of every month.
  3. Daily, Tran’s data must be pushed to TranHistory during the “End of Day” process.
  4. The TranHistory table is a historical table aggregating transaction data over one year.
  5. All inserts and updates on the Tran table must reflect in the TranHistory table at the end of each day.

PREPARING THE SCENARIO

The scenario described above implies that records in the Tran table also exist in table TranHistory till they are purged monthly. Each day, there will be a few new records in the Tran table but NOT in the TranHistory table. We must find a way to insert these new rows.

First, let us prepare the tables in question (See Listing 1 and 2).

-- Listing 1: Create Tran Table
USE AU
GO
CREATE TABLE [Tran] (
  responseId int NOT NULL ,
  senderId varchar(15) ,
  msisdn varchar(15) ,
  [message] varbinary ,
  status smallint ,
  application varchar ,
  receivedTime timestamp NULL ,
  processedTime datetime2 NULL ,
  flag smallint ,
  requestDelivery smallint ,
  delivered smallint ,
  account varchar(20) ,
  srcTon smallint ,
  srcNpi smallint ,
  destTon smallint ,
  destNpi smallint ,
  errorCode smallint ,
  messageId varchar ,
  sequenceNo int ,
  retries smallint ,
  messagePriority int ,
  userId varchar(20) ,
  bulkId varchar(20) 
)

-- Listing 2: Create TranHistory Table
USE AU
GO
CREATE TABLE [TranHistory] (
  responseId int NOT NULL ,
  senderId varchar(15) ,
  msisdn varchar(15) ,
  [message] varchar(160) ,
  status smallint ,
  application varchar ,
  receivedTime datetime2 NULL ,
  processedTime datetime2 NULL ,
  flag smallint ,
  requestDelivery smallint ,
  delivered smallint ,
  account varchar(20) ,
  srcTon smallint ,
  srcNpi smallint ,
  destTon smallint ,
  destNpi smallint ,
  errorCode smallint ,
  messageId varchar ,
  sequenceNo int ,
  retries smallint ,
  messagePriority int ,
  userId varchar(20) ,
  bulkId varchar(20) ,
  archivedTime datetime2 NOT NULL ,
)

WHEN NOT MATCHED THEN INSERT

Using the code in Listing 3, we insert a few rows in the Tran table to get started. Then, we proceed to use a MERGE statement to move the data to the TranHistory table.

-- Listing 3: Insert Initial Set of Rows in Tran Table
USE [AU]
GO

INSERT INTO [dbo].[Tran]
     VALUES
           (8000,'0233456789','Wishing you a Happy New Year',1,'K','20201110 15:00:00','20201110 15:10:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(7777,'0233456789','The blessing of the Lord be with you',1,'K','20201110 08:00:00','20201110 08:10:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(7005,'0234876789','Happy Birthday to you',1,'K','20201110 09:00:00','20201110 09:20:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(9002,'0233456789','Merry Christmas',1,'K','20201110 07:00:00','20201110 07:15:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(6789,'0233467889','Buy our brand new cars for less than $8000',1,'K','20201110 14:00:00','20201110 14:20:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(7685,'0244556789','Happy New Month. God bless and increase you',1,'K','20201110 17:00:00','20201110 17:08:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(4983,'0229856789','Help is coming your way today!',1,'K','20201110 19:00:00','20201110 19:20:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(6879,'0239986789','Call us for your next relocation project',1,'K','20201110 19:15:00','20201110 19:20:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(4567,'0233456789','Hard Work Always Pays',1,'K','20201110 22:05:00','20201110 22:20:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(8890,'0244656733','Don''t wait to buy land, buy land and wait',1,'K','20201110 15:05:00','20201110 15:20:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(6789,'0233466734','We are relocating. Call us on 077788997',1,'K','20201110 18:02:00','20201110 18:17:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(9899,'0233456556','Watch out for our latest movie',1,'K','20201110 06:00:00','20201110 06:02:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(6789,'0233456338','We are here to make you happy',1,'K','20201110 12:16:00','20201110 12:20:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
GO

We could have done this in a much simpler way, but we want to show the MERGE statement syntax (See Listing 4):

-- Listing 4: Merge Records in Tran Table to TranHistory Table
MERGE INTO [TranHistory] a USING [Tran] b
ON a.responseId=b.responseID
WHEN NOT MATCHED BY TARGET THEN INSERT
([responseId],[senderId],[msisdn],[message],[status],[application],[receivedTime],[processedTime],[flag],[requestDelivery],[delivered],[account],[srcTon],[srcNpi],[destTon],[destNpi],[errorCode],[messageId],[sequenceNo],[retries],[messagePriority],[userId],[bulkId],[archivedTime])
 VALUES 
([responseId],[senderId],[msisdn],[message],[status],[application],[receivedTime],[processedTime],[flag],[requestDelivery],[delivered],[account],[srcTon],[srcNpi],[destTon],[destNpi],[errorCode],[messageId],[sequenceNo],[retries],[messagePriority],[userId],[bulkId],getdate());
GO

The MERGE statement says:

  1. Take the contents of the [Tran] table and compare them with the [TranHistory] table based on the responseId column.
  2. Insert those rows which you find in the source table but do not find in the target table (TranHistory).

As it is, Tran and TranHistory are at per. But suppose the following day, new rows are introduced in the Trans Table. We must also push them to the TransHistory table while retaining records in the Tran table till the month is over.

We again use the script in listing 4. This time, we add an OUTPUT clause to tell what was introduced (See Listing 5):

-- Listing 5: Merge Records in Tran Table to TranHistory Table (Add OUTPUT Clause)
USE AU
GO

MERGE INTO [TranHistory] a USING [Tran] b
ON a.responseId=b.responseID
WHEN NOT MATCHED BY TARGET THEN INSERT
([responseId],[senderId],[msisdn],[message],[status],[application],[receivedTime],[processedTime],[flag],[requestDelivery],[delivered],[account],[srcTon],[srcNpi],[destTon],[destNpi],[errorCode],[messageId],[sequenceNo],[retries],[messagePriority],[userId],[bulkId],[archivedTime])
 VALUES 
([responseId],[senderId],[msisdn],[message],[status],[application],[receivedTime],[processedTime],[flag],[requestDelivery],[delivered],[account],[srcTon],[srcNpi],[destTon],[destNpi],[errorCode],[messageId],[sequenceNo],[retries],[messagePriority],[userId],[bulkId],getdate())
OUTPUT deleted.*, $action, inserted.*;
GO

We can repeat this process after introducing additional rows to the Tran table (Listing 6 and 7), and we get similar behavior:

-- Listing 6: Insert Six New Rows in Tran Table
USE [AU]
GO

INSERT INTO [dbo].[Tran]
     VALUES
			(6879,'0239986789','Call us for your next relocation project',1,'K','20201110 19:15:00','20201110 19:20:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(4567,'0233456789','Hard Work Always Pays',1,'K','20201110 22:05:00','20201110 22:20:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(8890,'0244656733','Don''t wait to buy land, buy land and wait',1,'K','20201110 15:05:00','20201110 15:20:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(6789,'0233466734','We are relocating. Call us on 077788997',1,'K','20201110 18:02:00','20201110 18:17:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(9899,'0233456556','Watch out for our latest movie',1,'K','20201110 06:00:00','20201110 06:02:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(6789,'0233456338','We are here to make you happy',1,'K','20201110 12:16:00','20201110 12:20:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
GO
Six Additional Rows Added to Tran Table
Figure 1: Six Additional Rows Added to Tran Table
First Part of the OUTPUT Clause
Figure 2: First Part of the OUTPUT Clause
Second Part of the OUTPUT Clause
Figure 3: Second Part of the OUTPUT Clause
-- Listing 7: Insert Additional Three Rows in Tran Table
USE [AU]
GO

INSERT INTO [dbo].[Tran]
     VALUES
	(7789,'0233456433','Are you ready for your next level?',1,'K','20201110 14:35:00','20201110 14:40:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(8000,'0233457759','Hutchies Honey, another level of taste',1,'K','20201110 08:00:00','20201110 08:08:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(7777,'0233458909','Make sure you vote tomorrow',1,'K','20201110 09:45:00','20201110 09:50:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
			,(9890,'0233459994','Wishing you a Merry Christmas',1,'K','20201110 10:00:00','20201110 10:05:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
GO
No Rows Deleted
Figure 4: No Rows Deleted
Four Rows Inserted
Figure 5: Four Rows Inserted

WHEN MATCHED THEN UPDATE

Another case is an update to the live table (Tran) when we want to sync such updates to the TranHistory table. We create this scenario by updating rows in the Tran table using the script in Listing 8.

-- Listing 8: Update and Insert Rows in Tran Table
USE AU
GO

UPDATE [dbo].[Tran] SET account='JUNIPER' 
WHERE account='KAIROS';
GO

INSERT INTO [dbo].[Tran]
     VALUES
	(5578,'0233566933','Newest on the Block!',1,'K','20201110 14:35:00','20201110 14:40:00',1,1,1,'KAIROS',1,2,3,4,1,1,9789,2,1,'ROUTEMOBILE','9988776')
GO

Figure 6 shows that rows are updated in the destination table. We get this detail using the OUTPUT clause.

Rows Updated in TranHistory
Figure 6: Rows Updated in TranHistory

We must use the clause highlighted in Listing 9. The MERGE statement identified rows that match the JOIN condition and updates the data in the column specified (account). We can update multiple rows with this approach.

-- Listing 9: Sync Data in TranHistory by Updating Rows
USE AU
GO

MERGE INTO [TranHistory] a USING [Tran] b
ON a.responseId=b.responseID
WHEN MATCHED THEN UPDATE
SET a.account=b.account
WHEN NOT MATCHED BY TARGET THEN INSERT
([responseId],[senderId],[msisdn],[message],[status],[application],[receivedTime],[processedTime],[flag],[requestDelivery],[delivered],[account],[srcTon],[srcNpi],[destTon],[destNpi],[errorCode],[messageId],[sequenceNo],[retries],[messagePriority],[userId],[bulkId],[archivedTime])
 VALUES 
([responseId],[senderId],[msisdn],[message],[status],[application],[receivedTime],[processedTime],[flag],[requestDelivery],[delivered],[account],[srcTon],[srcNpi],[destTon],[destNpi],[errorCode],[messageId],[sequenceNo],[retries],[messagePriority],[userId],[bulkId],getdate())
OUTPUT deleted.*, $action, inserted.*;
GO

WHEN NOT MATCHED BY SOURCE THEN DELETE

One more scenario involves rows being deleted from the source table (Listing 10). When it happens, we must identify the rows that no longer exist in the source table and remove them from the destination table.

We accomplish this using the clause highlighted in Listing 10. Again, Figures 7 and 8 show the rows impacted by this MERGE statement.

-- Listing 10: Update and Insert Rows in Tran Table
USE AU
GO

DELETE FROM [dbo].[Tran] 
WHERE account='JUNIPER';
GO

-- Listing 11: Syncing Tables After Deleting from Tran Table
USE AU
GO

MERGE INTO [TranHistory] a USING [Tran] b
ON a.responseId=b.responseID
WHEN NOT MATCHED BY SOURCE THEN DELETE
WHEN MATCHED THEN UPDATE
SET a.account=b.account
WHEN NOT MATCHED BY TARGET THEN INSERT
([responseId],[senderId],[msisdn],[message],[status],[application],[receivedTime],[processedTime],[flag],[requestDelivery],[delivered],[account],[srcTon],[srcNpi],[destTon],[destNpi],[errorCode],[messageId],[sequenceNo],[retries],[messagePriority],[userId],[bulkId],[archivedTime])
 VALUES 
([responseId],[senderId],[msisdn],[message],[status],[application],[receivedTime],[processedTime],[flag],[requestDelivery],[delivered],[account],[srcTon],[srcNpi],[destTon],[destNpi],[errorCode],[messageId],[sequenceNo],[retries],[messagePriority],[userId],[bulkId],getdate())
OUTPUT deleted.*, $action, inserted.*;
GO

Figure 7 shows the rows that were deleted, and Figure 8 shows those that were updated. It is the power of the MERGE statement. We can carry out delete, insert, and update operations all in one statement.

Deleted Rows in TranHistory
Figure 7: Deleted Rows in TranHistory
Operation Identified as "DELETE," NO Rows Inserted
Figure 8: Operation Identified as “DELETE,” NO Rows Inserted

CONCLUSION

This article reviewed the MERGE statement usage to sync data between an online table and a history table while maintaining the desired retention required in both tables.

There are many other use cases for SQL Server MERGE statements not covered in this article, but they are explored in Microsoft documentation. Source data specified in the USING clause is not limited to tables. Result sets from explicit SELECT statements can be source data. Common table expressions can also be source data.

REFERENCES

MERGE in Transact-SQL

Tags: , , Last modified: September 18, 2021

About the Author /

Kenneth Igiri is an Enterprise Architect with eProcess International S.A., Ecobank Group's Shared Services Centre. Kenneth has over fifteen years' experience in Information Technology with nine of those years focused on SQL Server and Oracle databases. His interests include Database Performance, HADR, Cloud Computing, Data Architecture and Enterprise Architecture. Asides from work, Kenneth teaches at Children's Church, writes faith-based fiction and helps small businesses grow. You can connect with Kenneth via his blog https://kennethigiri.com, LinkedIn, or on Amazon.com.

← Previous Story
Five Different Methods To Start, Stop, And Restart SQL Server Services
→ Next Story
Fundamentals of Managing Datafiles in SQL Server

Related Posts

Сodingsight - SQL Server T-SQL CASE

Getting Started with the SQL Server T-SQL CASE Expression Statement

March 15, 2022• T-SQL

SQL Server CASE expression is very useful in returning an output based on some conditional matching criteria. We can state multiple...
Read More →
Codingsight - Primary Keys in SQL Server

SQL Server Primary Key

February 22, 2022• SQL Server

Across all major RDBMS products, Primary Key in SQL constraints has a vital role. They identify the records present in a table uniquely....
Read More →

How to Use SQL Server T-SQL Function SUM: 5 Use Cases

September 08, 2021• Database development, Statements

The T-SQL SUM function is one of the fundamental functions. Its purpose is to calculate the sum of all values in the specified column –...
Read More →
CodingSight - SQL order by

SQL Server ORDER BY Clause: The 5 Do’s and Don’ts to Sort Data Like a Pro

August 12, 2021• Database development, Statements

Ugly. That’s what unsorted data looks like. We make data easy for the eyes by sorting them. And that’s what SQL ORDER BY is for. Use...
Read More →

Comments are closed.

↑
Close