Range Lookup in SSIS

We all are aware how beneficial lookup tasks are within SQL Server Integration Services. They are simple to use and also perform very well.

However, for complex lookups you cannot use the simple lookup tasks. For this you need to use range lookups. To learn how to, follow this link : http://www.julian-kuiters.id.au/article.php/ssis-lookup-with-range .

Although it does handle really complex scenarios and avoids using complex correlation joins etc in the sql, the performance can be significantly slower. I had a package that processed around 30 millions rows in 3 hours...now takes around 12 hours to do the same number of rows. This is because to enable range lookup you have to turn on the memory restriction. This basically means SSIS will only pull back one row at a time. Which is actually what you want but in terms of a large dimension (at a guess a couple of million) this could pose an issue. So in general as long as you index your table properly it shouldn’t be too much of an issue however it can never be as quick as 'cached lookup' which is the standard lookup.

Published 17 February 2008 16:47 by Jay Uttam

Comments

# re: Range Lookup in SSIS@ 20 February 2008 12:21

I've found to get the best performance you need to use a script component and load your dataset into memory (usual caveats about available memory etc) and then you can use the binary search to find the value you want. This will be far quicker than having to make a database call for every unique value that comes along.

simonsabin

# re: Range Lookup in SSIS@ 27 January 2010 20:20

There is solution based on the third-party commercial CozyRoc SSIS+ library. CozyRoc has implemented data flow destination script, which creates memory-efficient range dictionary object. The dictionary object can then be used in CozyRoc Lookup Plus component. For more information and demonstration how to use the script, check here:

www.cozyroc.com/.../range-dictionary-destination

CozyRoc