LiveDataScript: Scripting data from a database into insert statements
30 July 08 09:50 AM | Christian | with no comments

We've partnered with a company called Live Software Solutions on a number of projects recently and they've just decide to release a great tool we've been using on a number of projects as a free download.

It allows you to generate a single script with insert statements from multiple sources and is useful for when you have data in one database that you want to write into another, for example you've got a developer database and you want to script out a subset of data that will be used on QA and production.

It's also great when you want to get data out of a table based on joins to another table because that's quite tricky to get setup any other way but with this you simply write your select statement and then script the selection out.

Check it out here: http://www.lss.co.uk/Products/LiveDataScript/

 

Regards,

Christian Bolton  - MCA: Database, MCM: SQL Server 2008
Database Architect
http://coeo.com - The SQL Server Experts

Filed under: ,
SQLBits: Advanced SQL Server Troubleshooting Workshop
24 July 08 01:11 PM | Christian | 1 comment(s)

We're running a full-day pre-conference workshop before SQLBITSIII in Hatfield, UK in September on Advanced SQL Server Troubleshooting.  Its a level 300/400 course.  See below for more details...

Advanced Troubleshooting Workshop for SQL Server 2005

This one day event will provide attendees with an understanding of the tools and techniques required to troubleshoot difficult and complex problems within SQL Server 2005.

Topics include:

  • Memory grants and cache usage
  • I/O Considerations and diagnosing problems
  • SQL Server waits and wait types
  • Troubleshooting latch waits
  • Query plan optimisation and execution

 

Perform data capture easily, and provide answers quickly with skills and knowledge gained through this training:

  • Capturing performance data with PSSDiag
  • Data analysis with SQL Nexus

     

For further information download the 1-page flyer.

This Advanced Troubleshooting training day provides great value for £200, and a 25% early bird discount is available, just register before August 15th and attend for only £150!

Places are limited and event registration is open now: click to Register.

This training event is delivered in partnership with SQL Bits III. The training day will be hosted in Hatfield, UK.

Should you have any questions about the training agenda, content, logistics etc please contact info@coeo.com

Regards,

Christian Bolton  - MCA: Database, MCM: SQL Server 2008
Database Architect
http://coeo.com - The SQL Server Experts

Customer Advisory Team - BizTalk Server Performance Optimization Guide
09 July 08 01:20 PM | Christian | with no comments

We've been doing some work with the BizTalk Customer Advisory Team at Microsoft to help develop official guidance for SQL Server in a BizTalk environment.  This has just been launched as part of the Microsoft BizTalk Server Performance Optimization Guide.

BizTalk has a strong depenency on SQL Server and massive performance gains can be achieved by configuring SQL Server properly for a BizTalk environment.  If you support SQL Server for BizTalk its a must read!

Regards,

Christian Bolton  - MCA: Database, MCM: SQL Server 2008
Database Architect
http://coeo.com - The SQL Server Experts

 

Connected Systems Division
July 8, 2008 – Microsoft BizTalk Server Performance Optimization Guide RTW                                                                                                           

Summary
The BizTalk Customer Advisory Team and BizTalk UE team are pleased to announce the first edition of the “Microsoft BizTalk Server Performance Optimization Guide”.

The Microsoft BizTalk Server Performance Optimization Guide is the second installment in a series of guides intended to provide easily accessible, hands-on guidance to our customer and partner community. This 228 page guide is available on MSDN, TechNet and as a separate DOCX or CHM download alongside the already available “Microsoft BizTalk Operations Guide” 

The guide is based on real-world experience from BizTalk CAT (Rangers), Premier Field Engineering, MCS and other customer engagements. This guide is intended to serve two purposes. Firstly, to provide regularly maintained prescriptive guidance & best practices on optimizing BizTalk Server performance for demanding production environments. Secondly, to provide a foundation for the development of PFE, MCS and Partner training and service offerings.

The key sections of the guide are:
• Getting Started: Provides an overview of the BizTalk Server functional components that can affect performance. It also describes the phases of a BizTalk Server performance assessment.
• Finding and Eliminating Bottlenecks: The Finding and Eliminating Bottlenecks section describes various types of performance bottlenecks as they relate to BizTalk Server solutions and information about how to resolve the bottlenecks.
• Automating Testing: Describes how to implement an automated build process and how to automate functional and load testing using Visual Studio Team System, BizUnit and Loadgen.
• Optimizing Performance: The Optimizing Performance section provides guidance for optimizing performance of specific components in a BizTalk Server environment

The target audience for this guide is Microsoft field, partner organizations, and customers who plan, deploy, and maintain mission critical BizTalk Server installations. The guide was created from the key learnings, processes and methodology that have been developed by the Rangers to effectively run Performance Labs for our customers.

The guide has been carefully reviewed and vetted by experts from the community of BizTalk Server, whom we gratefully acknowledge. We believe that the information presented here will help BizTalk Server users optimize their solutions.

Please post and forward to your blogs, customers, partners and colleagues in the field.

Full MSDN URL:        http://msdn.microsoft.com/en-us/library/cc558617.aspx 
Full TechNet URL:     http://technet.microsoft.com/en-us/library/cc558617.aspx


Customer Response
In the very short period this guide was available, without official announcements, we have seen 1950 downloads page views. In addition, there were over thousand downloads of the offline versions.

The guide has so far received a 5-star rating and received feedback such as:

“Just a quick note to say that this document is bloody excellent. Great job! Please pass my regards to the rest of the team who may have contributed into the content.”
“So I spent the morning puttering around this document and I have to say Bravo!  Awesome resource!  Great job and thanks…”

Acknowledgments
My personal thanks goes to the many people who played key roles in getting this guide published. This includes:

Authors
Ewan Fairweather, Microsoft
Rob Steel, Microsoft

Editors (UE)
Trace Young
Larry Franks
Cathy Baldwin
Monica Rush
Penni Johnson

Contributors
Paolo Salvatori, Microsoft
Ben Pearce, Microsoft

Reviewers
Stephan Pepersack, Microsoft
Justin Langford, Coeo
Kevin B. Smith, Barclays Capital
Christian Bolton, Coeo
Brian Gregor, Microsoft
Robert Hogg, Blackmarble
John Plummer, Microsoft
Niklas Engfelt, Microsoft
Everett Yang, Microsoft
Clint Huffman, Microsoft
Shane Creamer, Microsoft
Young Jun Hong, Microsoft
Guy Lau, Microsoft

Filed under: ,
New SQL Server 2008 wait types - PREEMPTIVE_* and FT_*
06 June 08 01:05 PM | Christian | with no comments

I’m a big fan of using SQL Server waits to troubleshoot performance problems so I was pleasantly surprised to see that there are 475 possible wait types in SQL Server 2008 as opposed to 230 in SQL Server 2005. 

The first interesting ones I’ve looked at are the new PREEMPTIVE wait types.  Any code that needs to execute outside SQL Server has to go outside the control of SQL Server’s co-operative (or non-preemptive) scheduler and will use the preemptive scheduling model used by the OS.

Typically these external executions would be very difficult to troubleshoot using wait types because they would either come under a single wait like OLEDB for example or wouldn’t be tracked at all like OS level functions.  If you’ve ever tried to troubleshoot a SQL Server issue that turned out to be latency talking to the domain controller you’ll be presently surprised with the PREEMPTIVE_OS wait types that will show authentication waits.

I was looking to see if there were any PREEMPTIVE waits for Full-Text Search as they just show up as OLDEDB waits in previous versions but now that FTS is “integrated” with SQL Server (iFTS) there is no dependency on mssearch so we can have native SQL Server waits.  There are 7 new FTS related wait types:

FT_COMPROWSET_RWLOCK
FT_IFTS_RWLOCK
FT_IFTSHC_MUTEX
FT_IFTSISM_MUTEX
FT_MASTER_MERGE
FT_METADATA_MUTEX
FT_RESTART_CRAWL

They haven’t made it into BOL yet (in Feb CTP anyway) so here is the rather long list of PREEMPTIVE wait types:

PREEMPTIVE_ABR
PREEMPTIVE_AUDIT_ACCESS_EVENTLOG
PREEMPTIVE_AUDIT_ACCESS_SECLOG
PREEMPTIVE_CLOSEBACKUPMEDIA
PREEMPTIVE_CLOSEBACKUPTAPE
PREEMPTIVE_CLOSEBACKUPVDIDEVICE
PREEMPTIVE_CLUSAPI_CLUSTERRESOURCECONTROL
PREEMPTIVE_COM_COCREATEINSTANCE
PREEMPTIVE_COM_COGETCLASSOBJECT
PREEMPTIVE_COM_CREATEACCESSOR
PREEMPTIVE_COM_DELETEROWS
PREEMPTIVE_COM_GETCOMMANDTEXT
PREEMPTIVE_COM_GETDATA
PREEMPTIVE_COM_GETNEXTROWS
PREEMPTIVE_COM_GETRESULT
PREEMPTIVE_COM_GETROWSBYBOOKMARK
PREEMPTIVE_COM_LBFLUSH
PREEMPTIVE_COM_LBLOCKREGION
PREEMPTIVE_COM_LBREADAT
PREEMPTIVE_COM_LBSETSIZE
PREEMPTIVE_COM_LBSTAT
PREEMPTIVE_COM_LBUNLOCKREGION
PREEMPTIVE_COM_LBWRITEAT
PREEMPTIVE_COM_QUERYINTERFACE
PREEMPTIVE_COM_RELEASE
PREEMPTIVE_COM_RELEASEACCESSOR
PREEMPTIVE_COM_RELEASEROWS
PREEMPTIVE_COM_RELEASESESSION
PREEMPTIVE_COM_RESTARTPOSITION
PREEMPTIVE_COM_SEQSTRMREAD
PREEMPTIVE_COM_SEQSTRMREADANDWRITE
PREEMPTIVE_COM_SETDATAFAILURE
PREEMPTIVE_COM_SETPARAMETERINFO
PREEMPTIVE_COM_SETPARAMETERPROPERTIES
PREEMPTIVE_COM_STRMLOCKREGION
PREEMPTIVE_COM_STRMSEEKANDREAD
PREEMPTIVE_COM_STRMSEEKANDWRITE
PREEMPTIVE_COM_STRMSETSIZE
PREEMPTIVE_COM_STRMSTAT
PREEMPTIVE_COM_STRMUNLOCKREGION
PREEMPTIVE_CONSOLEWRITE
PREEMPTIVE_CREATEPARAM
PREEMPTIVE_DEBUG
PREEMPTIVE_DFSADDLINK
PREEMPTIVE_DFSLINKEXISTCHECK
PREEMPTIVE_DFSLINKHEALTHCHECK
PREEMPTIVE_DFSREMOVELINK
PREEMPTIVE_DFSREMOVEROOT
PREEMPTIVE_DFSROOTFOLDERCHECK
PREEMPTIVE_DFSROOTINIT
PREEMPTIVE_DFSROOTSHARECHECK
PREEMPTIVE_DTC_ABORT
PREEMPTIVE_DTC_ABORTREQUESTDONE
PREEMPTIVE_DTC_BEGINTRANSACTION
PREEMPTIVE_DTC_COMMITREQUESTDONE
PREEMPTIVE_DTC_ENLIST
PREEMPTIVE_DTC_PREPAREREQUESTDONE
PREEMPTIVE_FILESIZEGET
PREEMPTIVE_FSAOLEDB_ABORTTRANSACTION
PREEMPTIVE_FSAOLEDB_COMMITTRANSACTION
PREEMPTIVE_FSAOLEDB_STARTTRANSACTION
PREEMPTIVE_FSRECOVER_UNCONDITIONALUNDO
PREEMPTIVE_GETRMINFO
PREEMPTIVE_LOCKMONITOR
PREEMPTIVE_MSS_RELEASE
PREEMPTIVE_ODBCOPS
PREEMPTIVE_OLE_UNINIT
PREEMPTIVE_OLEDB_ABORTORCOMMITTRAN
PREEMPTIVE_OLEDB_ABORTTRAN
PREEMPTIVE_OLEDB_GETDATASOURCE
PREEMPTIVE_OLEDB_GETLITERALINFO
PREEMPTIVE_OLEDB_GETPROPERTIES
PREEMPTIVE_OLEDB_GETPROPERTYINFO
PREEMPTIVE_OLEDB_GETSCHEMALOCK
PREEMPTIVE_OLEDB_JOINTRANSACTION
PREEMPTIVE_OLEDB_RELEASE
PREEMPTIVE_OLEDB_SETPROPERTIES
PREEMPTIVE_OLEDBOPS
PREEMPTIVE_OS_ACCEPTSECURITYCONTEXT
PREEMPTIVE_OS_ACQUIRECREDENTIALSHANDLE
PREEMPTIVE_OS_AUTHENTICATIONOPS
PREEMPTIVE_OS_AUTHORIZATIONOPS
PREEMPTIVE_OS_AUTHZGETINFORMATIONFROMCONTEXT
PREEMPTIVE_OS_AUTHZINITIALIZECONTEXTFROMSID
PREEMPTIVE_OS_AUTHZINITIALIZERESOURCEMANAGER
PREEMPTIVE_OS_BACKUPREAD
PREEMPTIVE_OS_CLOSEHANDLE
PREEMPTIVE_OS_CLUSTEROPS
PREEMPTIVE_OS_COMOPS
PREEMPTIVE_OS_COMPLETEAUTHTOKEN
PREEMPTIVE_OS_COPYFILE
PREEMPTIVE_OS_CREATEDIRECTORY
PREEMPTIVE_OS_CREATEFILE
PREEMPTIVE_OS_CRYPTACQUIRECONTEXT
PREEMPTIVE_OS_CRYPTIMPORTKEY
PREEMPTIVE_OS_CRYPTOPS
PREEMPTIVE_OS_DECRYPTMESSAGE
PREEMPTIVE_OS_DELETEFILE
PREEMPTIVE_OS_DELETESECURITYCONTEXT
PREEMPTIVE_OS_DEVICEIOCONTROL
PREEMPTIVE_OS_DEVICEOPS
PREEMPTIVE_OS_DIRSVC_NETWORKOPS
PREEMPTIVE_OS_DISCONNECTNAMEDPIPE
PREEMPTIVE_OS_DOMAINSERVICESOPS
PREEMPTIVE_OS_DSGETDCNAME
PREEMPTIVE_OS_DTCOPS
PREEMPTIVE_OS_ENCRYPTMESSAGE
PREEMPTIVE_OS_FILEOPS
PREEMPTIVE_OS_FINDFILE
PREEMPTIVE_OS_FLUSHFILEBUFFERS
PREEMPTIVE_OS_FORMATMESSAGE
PREEMPTIVE_OS_FREECREDENTIALSHANDLE
PREEMPTIVE_OS_FREELIBRARY
PREEMPTIVE_OS_GENERICOPS
PREEMPTIVE_OS_GETADDRINFO
PREEMPTIVE_OS_GETCOMPRESSEDFILESIZE
PREEMPTIVE_OS_GETDISKFREESPACE
PREEMPTIVE_OS_GETFILEATTRIBUTES
PREEMPTIVE_OS_GETFILESIZE
PREEMPTIVE_OS_GETLONGPATHNAME
PREEMPTIVE_OS_GETPROCADDRESS
PREEMPTIVE_OS_GETVOLUMENAMEFORVOLUMEMOUNTPOINT
PREEMPTIVE_OS_GETVOLUMEPATHNAME
PREEMPTIVE_OS_INITIALIZESECURITYCONTEXT
PREEMPTIVE_OS_LIBRARYOPS
PREEMPTIVE_OS_LOADLIBRARY
PREEMPTIVE_OS_LOGONUSER
PREEMPTIVE_OS_LOOKUPACCOUNTSID
PREEMPTIVE_OS_MESSAGEQUEUEOPS
PREEMPTIVE_OS_MOVEFILE
PREEMPTIVE_OS_NETGROUPGETUSERS
PREEMPTIVE_OS_NETLOCALGROUPGETMEMBERS
PREEMPTIVE_OS_NETUSERGETGROUPS
PREEMPTIVE_OS_NETUSERGETLOCALGROUPS
PREEMPTIVE_OS_NETUSERMODALSGET
PREEMPTIVE_OS_NETVALIDATEPASSWORDPOLICY
PREEMPTIVE_OS_NETVALIDATEPASSWORDPOLICYFREE
PREEMPTIVE_OS_OPENDIRECTORY
PREEMPTIVE_OS_PIPEOPS
PREEMPTIVE_OS_PROCESSOPS
PREEMPTIVE_OS_QUERYREGISTRY
PREEMPTIVE_OS_QUERYSECURITYCONTEXTTOKEN
PREEMPTIVE_OS_REMOVEDIRECTORY
PREEMPTIVE_OS_REPORTEVENT
PREEMPTIVE_OS_REVERTTOSELF
PREEMPTIVE_OS_RSFXDEVICEOPS
PREEMPTIVE_OS_SECURITYOPS
PREEMPTIVE_OS_SERVICEOPS
PREEMPTIVE_OS_SETENDOFFILE
PREEMPTIVE_OS_SETFILEPOINTER
PREEMPTIVE_OS_SETFILEVALIDDATA
PREEMPTIVE_OS_SETNAMEDSECURITYINFO
PREEMPTIVE_OS_SQLCLROPS
PREEMPTIVE_OS_SQMLAUNCH
PREEMPTIVE_OS_VERIFYSIGNATURE
PREEMPTIVE_OS_VSSOPS
PREEMPTIVE_OS_WAITFORSINGLEOBJECT
PREEMPTIVE_OS_WINSOCKOPS
PREEMPTIVE_OS_WRITEFILE
PREEMPTIVE_OS_WRITEFILEGATHER
PREEMPTIVE_OS_WSASETLASTERROR
PREEMPTIVE_REENLIST
PREEMPTIVE_RESIZELOG
PREEMPTIVE_ROLLFORWARDREDO
PREEMPTIVE_ROLLFORWARDUNDO
PREEMPTIVE_SB_STOPENDPOINT
PREEMPTIVE_SERVER_STARTUP
PREEMPTIVE_SETRMINFO
PREEMPTIVE_SHAREDMEM_GETDATA
PREEMPTIVE_SNIOPEN
PREEMPTIVE_SOSHOST
PREEMPTIVE_SOSTESTING
PREEMPTIVE_STARTRM
PREEMPTIVE_STREAMFCB_CHECKPOINT
PREEMPTIVE_STREAMFCB_RECOVER
PREEMPTIVE_STRESSDRIVER
PREEMPTIVE_TESTING
PREEMPTIVE_TRANSIMPORT
PREEMPTIVE_UNMARSHALPROPAGATIONTOKEN
PREEMPTIVE_VSS_CREATESNAPSHOT
PREEMPTIVE_VSS_CREATEVOLUMESNAPSHOT
PREEMPTIVE_XE_CALLBACKEXECUTE
PREEMPTIVE_XE_DISPATCHER
PREEMPTIVE_XE_ENGINEINIT
PREEMPTIVE_XE_GETTARGETSTATE
PREEMPTIVE_XE_SESSIONCOMMIT
PREEMPTIVE_XE_TARGETFINALIZE
PREEMPTIVE_XE_TARGETINIT
PREEMPTIVE_XE_TIMERRUN
PREEMPTIVE_XETESTING

Regards, 

Christian Bolton  MCA: Database
Database Architect
http://coeo.com - The SQL Server Experts

Microsoft Certified Master Certification just announced
06 June 08 09:05 AM | Christian | 2 comment(s)

Microsoft have just announced at TechEd 2008 a new tier certification called Microsoft Certified Master!

The Master certification came about because of the huge gap between the IT Pro certifications and the Microsoft Certified Architect programs so they decided to separate the training and exams from the review board on the product MCA courses and develop it into the Master certification.  The Master certification is now a pre-requisite for the product based MCA’s (currently Database,Messaging, Directory Services) but the best news for most people is that anyone can apply; you don't need to be from Microsoft or a Partner. A rough estimation of the target numbers for the certification’s are (this isn’t official info):
  • MCP                 - 1,000,000’s
  • ITPro                 - 100,000’s
  • Master NEW!     - 1,000’s
  • Architect            - 100’s 
All current MCA: Database and MCA: Messaging title holders will automatically receive the new certification (effectively, they’ve already done it). This is great news for engineers (particularly Microsoft PFE’s!) not interested in the consulting aspects required for MCA and provides better recognition than MCA: Database “Apprentice” as this level was previously known.  For those of you who’ve looked into the MCA: Database/Messaging requirements already (http://www.microsoft.com/learning/mcp/architect) here is a breakdown (subject to variation):-      
  • Microsoft Certified Master
    • 3 weeks of mandatory training
      • One 2-week block, one 1-week block (can choose to consume all together or not)
    • Core technical content / sessions
    • ‘Written’ exams
    • Qualification Lab Test
  • Microsoft Certified Architect (Technology Programs)
    • Messaging, Database, Directory
    • Pre-requisite = Certified Master (does not apply to Infrastructure and Solutions tracks)
       

All the training will be at Microsoft HQ in Redmond, Washington for the next year but following that they will be starting WORLDWIDE deliveries!

What will it cost?

  • Pricing
    • Package: $18,500
      • 3 weeks of training
      • 3 written tests
      • 1 Qualification Lab
    • Retakes
      • Written tests: $250 each
      • Lab Test: $1,500

Believe me, its a bargain.

Here is the "marketing" info from the attached flyer:

What is It?

The Microsoft® Certified Master Program takes the best in the IT industry and makes them even better. Whether you want to enhance and validate your advanced skills or take your career to the next level, achieving a Master Certification will help differentiate you from others in the competitive ranks of senior IT Professionals.

High Bar to Entry

Qualified program applicants are highly skilled technical experts within their respective disciplines and have several years of experience in designing, deploying, and managing solutions using Microsoft technology. Specific requirements vary by platform.

Exclusive Advanced Training from the top Experts

The Microsoft® Certified Master Program goes beyond any product training offered outside Microsoft today. Taught by instructors who are practiced experts within their fields of discipline, the Microsoft® Certified Master Program employs hands-on labs and testing to help ensure each student gains maximum value and insight from the course.

Instructors use their numerous years of experience to create 300- and 400-level classroom training and lab scenarios that provide a "learn-by-doing" training environment for students.

Product Certification from Microsoft

Certified graduates of the program earn recognition for their high level of product expertise and gain the finely tuned skills required to effectively design, implement, and optimize complex customer solutions.

Learn More

Find out how you can advance your career in technology as a Microsoft® Certified Master by visiting www.microsoft.com/learning/mcp/master  

 

Regards,

Christian Bolton  - MCA: Database, MCM: SQL Server 2008
Database Architect
http://coeo.com - The SQL Server Experts

Will you still want me, when I'm 64?
29 April 08 09:44 AM | Christian | with no comments

Thankyou to all those who attended my session on 64-bit SQL Server at the UK SQL Server User Group on 17th April 2008.  I've just got round to uploading the slidedeck for those of you that have been asking for it.  You can download it here: http://coeo.com/conferences.html.  For anyone interested that wasn't there here is a summary of the content:

  • 32-bit memory management
    • Virtual Address Space
    • memtoleave
    • /3GB
    • /PAE and AWE
  • 64-bit memory management
    • Virtual Address Space
  • History of current 64-bit platforms
  • What value for Max Server Memory setting on 64-bit?
  • AWE/Lock Pages in Memory with 64-bit
  • Pagefile size for 64-bit

I'm going to expand some of these points into separate blog postings for easier reference very soon.  Until then please feel free to comment here with anything I might have missed or alternative views.

 

Regards,

Christian Bolton  MCA: Database
Database Architect
http://coeo.com - The SQL Server Experts

 

My session at SQLBits II - What Are You Waiting For?
06 March 08 09:05 PM | Christian | with no comments

Thankyou to all those who attended my session "What Are You Waiting For?" last Saturday at SQLBits II in Birmingham.  There was some great feedback and constructive comments which I will incorporate when I deliver the session at the European SQLPASS conference next month in Germany (http://www.european-pass-conference.com/).  If any of you will be there be sure to come and say hello!

Congratulations to the two lucky guys who won copies of my new book, Professional SQL Server 2005 Performance Tuning published by Wrox.  I feel duty bound to inform those of you who weren't so lucky that its still available from all good bookstores (always wanted to say thatSmile ) and can be had at a particularly cheap high-value price (under £16) from amazon:

http://www.amazon.co.uk/gp/product/0470176393/ref=s9_asin_image_1_sims_c4_604241_264392_43915_51062_44380_41843_46601_36591-qvfp_g1?pf_rd_m=A3P5ROKL5A1OLE&pf_rd_s=center-1&pf_rd_r=046KG0PC5BMA2W18E4G1&pf_rd_t=101&pf_rd_p=139045791&pf_rd_i=468294

My presentation is attached to this post for those of you who missed it or just want to re-live the experience Big Smile

 

Regards,

Christian Bolton  MCA: Database
Database Architect
http://coeo.com - The SQL Server Experts
 

Microsoft Certified Architect: Database
14 February 08 10:05 PM | Christian | with no comments

I passed the final review board today to become the first Microsoft Certified Architect: Database outside of Microsoft.  Hooray!

"For architects who specialize in using Microsoft technology to provide expert solutions, Microsoft offers MCA training and certification in messaging (Microsoft Exchange Server) and database (Microsoft SQL Server focusing on OLTP (online transaction processing) disciplines." 

"To become a Microsoft Certified Technology Architect (referred to internally as "Ranger"), you must successfully participate in a four-week in-depth training program, successfully complete weekly written and lab-based exams, and pass a rigorous Review Board interview conducted by Microsoft experts and MCAs."

It used to be open to only Microsoft internal staff and Gold Partners but they've recently relaxed that to include Certified Partners as well.  It's very expensive ($25k+expenses) but I can heartily recommend it.

The four weeks training is in Redmond and delivered by the SQL Server product group which is amazing in itself but combined with 12 hour days 7 days a week with 20 or so other SQL Server experts from across the world is an experience I will never forget.  I've made friends and business contacts for life through the program and would encourage anyone that meets the entry requirements to take the opportunity if they are ever in a position to. 

You can read more about the program here: http://www.microsoft.com/learning/mcp/architect/database/default.mspx

 

Regards,

Christian Bolton  MCA: Database
Database Architect
http://coeo.com - The SQL Server Experts
 

 

Professional SQL Server 2005 Performance Tuning released
27 January 08 03:36 AM | Christian | with no comments

http://www.amazon.co.uk/gp/product/0470176393/ref=s9_asin_title_1?pf_rd_m=A3P5ROKL5A1OLE&pf_rd_s=center-1&pf_rd_r=1H851J8GCPJE25PHWM77&pf_rd_t=101&pf_rd_p=139045791&pf_rd_i=468294 

I've had a couple of e-mails from people in the US that have had their copies delivered today so Wiley/Wrox seem to be ahead of schedule:)  It was due for release in the US on 29th Jan and 8th Feb in the UK.

I'm particularly pleased with the chapter on Tuning and Hardware and OS in which I cover things like PAE, AWE, 64-bit, Hyper-Threading and disk sector alignment in-depth but in a way that I hope will be accessible to a wide audience and the chapter on SQL Server Wait Types which I'll be using as the basis of a session at the SQLBits conference in Birmingham, UK on 1st March.  More details here: http://www.sqlbits.com.  I'll hopefully have some copies of the book from Wiley to give away so come along if you're able to!

Regards,

Christian Bolton
Database Architect
http://coeo.com - The SQL Server Experts

 

 

 

Filed under:
SQL Server memtoleave, VAS and 64-bit
07 January 08 11:39 AM | Christian | 4 comment(s)

GrumpyOldDBA started a nice thread here: http://sqlblogcasts.com/blogs/grumpyolddba/archive/2008/01/06/configuring-windows-2003-x64-for-sql-server.aspx on 64-bit which I began to reply to but it got big enough for a full post.

This is a great topic to start a dialog on, I'm a great supporter of x64 as well having worked through lots of problems with PAE and AWE on the 32-bit platform in the past.

One of the common issues we used to see was with "MemToLeave" being too small or fragmented.  "MemToLeave" is virtual address space (VAS) that's left un-used when SQL Server starts so that external components called by SQL Server are saved some address space.  In the cases I saw, these components were always extended stored procs or linked servers.  The reason we need memtoleave is because the virtual address space on 32-bit is only 2GB which would all be taken by the buffer pool if we let it.  I think the memtoleave value settled at about 340MB after a few service packs of SQL 2000 although it could still be 256MB or a dynamic value in between.  The point is that its only a few hundred MB so it not unusual to run out although you can increase it if you get desperate by using the -g startup parameter.

In SQL 2000 you have to run the VMSTAT utility to get an accurate view of MemToLeave which is quite intrusive but in SQL Server 2005 you can get the information from DMV's.  This script will show you the largest available block of virtual address space (memtoleave) outside the buffer pool:

With VASummary(Size,Reserved,Free) AS
(SELECT
    Size = VaDump.Size,
    Reserved =  SUM(CASE(CONVERT(INT, VaDump.Base)^0)
    WHEN 0 THEN 0 ELSE 1 END),
    Free = SUM(CASE(CONVERT(INT, VaDump.Base)^0)
    WHEN 0 THEN 1 ELSE 0 END)
FROM
(
    SELECT  CONVERT(VARBINARY, SUM(region_size_in_bytes))
    AS Size, region_allocation_base_address AS Base
    FROM sys.dm_os_virtual_address_dump 
    WHERE region_allocation_base_address <> 0x0
    GROUP BY region_allocation_base_address 
 UNION  
    SELECT CONVERT(VARBINARY, region_size_in_bytes), region_allocation_base_address
    FROM sys.dm_os_virtual_address_dump
    WHERE region_allocation_base_address  = 0x0
)
AS VaDump
GROUP BY Size)


SELECT SUM(CONVERT(BIGINT,Size)*Free)/1024 AS [Total avail mem, KB] ,CAST(MAX(Size) AS BIGINT)/1024 AS [Max free size, KB] 
FROM VASummary 
WHERE Free <> 0

Whats really interesting is comparing the results between 32-bit and 64-bit:

My 32-bit laptop with SQL Server 2005 (32-bit obvously) and 2GB RAM:
Total avail mem, KB
340680
max free size, KB
120016

A 64-bit server with SQL Server 2005 x64 with 8GB RAM:
Total avail mem, KB
8574741364
max free size, KB
6703778112

My laptop has memtoleave of 120MB and the server has over 6TB. Beautiful.

Its not entirely glorious as it sounds though because you can still run out of VAS even on 64-bit if you run low on physical memory.  This is why its always recommend to have max server memory set on 64-bit SQL Server to prevent SQL Server taking too much.

Regards,

Christian Bolton
Database Architect
http://coeo.com - The SQL Server Experts

 

 

 

SQL Server: resource waits and signal waits
14 November 07 04:24 PM | Christian | 2 comment(s)

You won't find any resource waits that specifically measure CPU wait time, instead you measure “signal waits” which have been conveniently worked out for you in SQL Server 2005 and exposed in the sys.dm_os_wait_stats dmv.  Specifically, a signal wait is the time after a resource wait when the task has the resource it wanted and is just waiting to run on the CPU again.

For example, spid 55 starts a task that is now “running” on the CPU.  The task now needs to get some data so to keep an efficiently running system it will yield the processor to another task and wait in a “suspended” queue until it gets the data.  Time spent in this queue is what we know as a SQL Server “wait” and in the example the data is on disk so spid55 shows as waiting on a PAGEIOLATCH wait.  This wait represents a latch on a memory address to support a physical IO operation to get data for the address and is a measure of physical IO performance.

Now the task has the data it needs and wants some time on the CPU again.  The only problem is that other tasks want CPU time now as well and he has to join a queue called “runnable” while he waits for his turn.  The time he spends here is known as a signal wait.

The most accurate dmv for tracking SQL Server waits is sys.dm_os_waiting_tasks because it’s based at a task level and not a session level.  sys.dm_exec_requests does have wait information but because its at the session level you won’t see separate waits for parallel tasks or system tasks that can run without a session and therefore you might miss some key data.

sys.dm_os_waiting_tasks is great but the data is transient so it doesn’t keep any history.  To save you the bother of periodically writing this information somewhere else to correlate there is a dmv that already does it for you: sys.dm_os_wait_stats.  This will show you the combined wait times against each wait type of all tasks that have run since the last service restart.  It also exposes the signal wait time for each wait type so totalling those figures will give you a measurement of pure CPU waits within the instance.

Tom Davidson wrote a nice little script which shows the percentage of signal waits vs. resource waits on an instance:

Select signalWaitTimeMs=sum(signal_wait_time_ms)
    ,'%signal waits' = cast(100.0 * sum(signal_wait_time_ms) / sum (wait_time_ms) as numeric(20,2))
    ,resourceWaitTimeMs=sum(wait_time_ms - signal_wait_time_ms)
    ,'%resource waits'= cast(100.0 * sum(wait_time_ms - signal_wait_time_ms) / sum (wait_time_ms) as numeric(20,2))
from sys.dm_os_wait_stats

Its best run against data aggregated specifically within a busy period otherwise long periods of low activity will skew the results.  You could restart the SQL Server service to clear the historical data before the busy period but far more conveniently, you can manually flush the data by running DBCC sqlperf (‘sys.dm_os_wait_stats’, clear).  This is very handy  to get a good view of aggregated waits within a set period but if you’re doing it on someone else’s server make sure they don’t have monitoring tools that depend on the historical data before you flush it!

 

Regards,

Christian Bolton
Database Architect
http://coeo.com - The SQL Server Experts

 

What's the "cost"?
14 October 07 06:15 PM | Christian | 1 comment(s)

SQL Server uses a cost-based based optimizer which means that it will assign a "cost" to each execution plan and then choose the plan with the lowest cost.  You can see evidence of this when you look inside an execution plan and see things like "CPU cost" and "subtree cost". 

If you've been around SQL Server for a while you'll know that this cost doesn't translate into seconds or anything meaningful and is just an arbitrary number used to assign a value representing the resource cost.  You might even know that the "cost" value was benchmarked on a machine at Microsoft early in SQL Server's life (probably 7.0) but have you ever wondered what this historic supercomputer looked like?  Wonder no longer, I've attached a picture of it.  Have a look, it'll make you smile :)

I was on SQL Ranger training (now MCA: Database) in Redmond back in May and this question of cost came up at one of the sessions that Lubor Kollar was running.  For those of you that don't know Lubor he owned the development of the SQL Server optimizer for many years and now works in the Customer Advisory Team.  He explained that many years ago a guy worked for him called Nick and it was Nick's job to benchmark query times for the optimizer team.  Those benchmarks became the basis for the cost-based optimizer.

Lubor said that they looked into changing it but it had so many dependancies within SQL Server that it was prohibitively expensive to change.  Nothing would really be gained from changing it anyway as they'd be replacing what is now an arbitrary value for another arbitrary value.  So that is why we still have it today and why you'll see cost values with lots of decimal places; what took 1 sec on the benchmark machine runs a lot faster on your more modern machine.

Next time your looking at a cost in an execution plan spare thought for Nick and his historic machine, it might even make it all seem less complicated now you know its provenance!

 

Regards,

Christian

http://coeo.com

 

 

 

 

Disk Sector Alignment
09 October 07 07:30 PM | Christian | 10 comment(s)

I’ve just finished 3 chapters for an upcoming Wrox book called Professional SQL Server 2005 Performance Tuning: http://www.amazon.co.uk/Professional-Server-2005-Performance-Tuning/dp/0470176393/ref=sr_1_11/202-3946698-6401428?ie=UTF8&s=books&qid=1191700832&sr=8-11

I joined the author team late so my picture’s not on the cover pic yet.  I know its vain but I can’t wait J

Anyway, what I wanted to post on was a section I wrote on Disk Sector Alignment.  It seems to be one of those things that you've either been doing for years or you've never heard of.  When I left Microsoft some of the guys were trying to get something published “officially” but I don’t know how much interest they managed to get.  I've been meaning to blog on it for ages...

Sector Alignment

Sector Alignment is a little known performance optimization tip that documented for Exchange here: http://technet.microsoft.com/en-us/library/aa998219.aspx (but not very well described) and is equally valid for SQL Server.  There are a few blog postings that try to measure the performance difference for SQL Server but why it can cause a problem is not detailed very well. It should be performed on any NTFS partition that is created on a RAID array to “avoid misalignment with stripe units and caching which can cause performance degradation”. Let’s look in a bit more detail about what that actually means.

First of all, some disk terminology. What you’re interested in for the purpose of explaining sector alignment are sectors, tracks and blocks.

*        A sector is the smallest unit of storage space on disk and is typically 512 bytes.

*        A track is the circular path of sectors that sit around a single circumference of the disk. There are 63 sectors per track numbered 1-63 on each track.

*        A block is how you usually refer to sectors to make things easier for you. You start at the beginning of the disk and increment until the end of the disk. Blocks start from 0.

Partitions always start at the first sector on a track, so for the first partition that means Track 2 Sector 1 (because the MBR is on Track 1), which equates to Block number 63 (blocks start from 0). Misalignment occurs because the storage vendors define tracks differently. On an EMC Symmetrix, for example, a track is considered to be 64 blocks and the cache is also based on this.

A Symmetrix DMX RAID5 array uses a stripe size of 4 tracks, which totals 256 blocks. 64KB is the largest single write that Windows can make, so using the default partition location means that the first 64Kb write will be fine but the second will straddle 2 “stripes” causing both to be updated.

If you align the partition to 128 blocks (or 64KB as each block is 512bytes) then you don’t cross a track boundary and therefore issue the minimum number of I/Os.

There are 2 methods of aligning your partition and you’ll be pleased to know that it’s quite a simple process. Diskpar is a tool that’s been available for a while to enable you to align partitions but as of Windows 2003 SP1 the preferred method is Diskpart (note the extra ‘t’). The main difference between the two methods is that you specify the alignment value in blocks using Diskpar and in KB using Diskpart.

 

Regards,

 

Christian

http://coeo.com

 

 

 

Filed under:
How to corrupt a page in a specific index and then fix it
07 October 07 10:19 AM | Christian | 1 comment(s)

I was developing course materials for a workshop recently and I wanted to run a lab where you DBCC’d a database, found an error and then worked out how to fix it.  The only problem was deliberately corrupting a page in a specific index.  I did some investigating and thought I’d share my findings:

 

If you just need a broken database to play with then you can download one from Paul Randal’s old blog here: http://blogs.msdn.com/sqlserverstorageengine/archive/2007/04/17/example-corrupt-database-to-play-with.aspx

 

Tony Rogerson has a good blog here on how to corrupt data without using a Hex Editor:

http://sqlblogcasts.com/blogs/tonyrogerson/archive/2007/03/10/how-to-create-a-corrupt-database-using-bulk-insert-update-and-bcp-sql-server-as-a-hex-editor.aspx

 

My needs however where a bit different because I needed to corrupt a page in a specific non-clustered index in a database that my customer gave me to develop the workshop with.  I’ll use AdventureWorks to illustrate...

 

First of all you need a Hex Editor because you’re going to change some hex values in the mdf file of the database.  I used Hex Workshop which you can download from http://www.hexworkshop.com and use free for 35 days.  If you stop SQL Server and load AdventureWorks_Data.mdf into the editor you’ll see a stream of hex and no easy way to see what’s what.  Close the file and restart SQL Server.

 

Now change the PAGE_VERIFY option to NONE so that the checksum being different doesn’t intercept with an error before we can look at the DBCC results.

 

ALTER DATABASE AdventureWorks SET PAGE_VERIFY NONE

 

You’re going to be corrupting the AK_CustomerAddress_rowguid non-clustered index on the Sales.CustomerAddress table so the first thing we need is a page number that contains data for that index.  I found the solution on page 218 of Kalen Delaney’s Inside SQL Server 2005 Storage Engine book.

 

SELECT object_name(object_id) AS name, index_id,rows, type_desc AS page_type_desc, total_pages AS pages, first_page

FROM sys.partitions p JOIN sys.system_internals_allocation_units a

ON p.partition_id = a.container_id

WHERE object_id=object_id('sales.customeraddress')

 

This gives you:

 

name             index_id rows  page_type_desc pages first_page

---------------- -------- ----- -------------- ----- --------------

CustomerAddress         1 19220    IN_ROW_DATA   121 0xA02800000100

CustomerAddress         2 19220    IN_ROW_DATA    81 0x381200000100

 

There is only a clustered index and the non-clustered index on the table so the second row is the one you want.  Now back to Kalen’s book to translate the hex into a more familiar page reference:

 

create function [dbo].[covert_page_nums] (@page_num binary(6))

RETURNS varchar(11)

AS

BEGIN

 RETURN(convert(varchar(2), (convert(int, substring(@page_num, 6, 1)) * power(2, 8)) +

  (convert(int, substring(@page_num, 5, 1)))) + ':' +

    convert(varchar(11),

 (convert(int, substring(@page_num, 4, 1)) * power(2, 24)) +

 (convert(int, substring(@page_num, 3, 1)) * power(2, 16)) +

 (convert(int, substring(@page_num, 2, 1)) * power(2, 8)) +

 (convert(int, substring(@page_num, 1, 1)))) )

END

 

SELECT dbo.convert_page_nums(0x381200000100)

 

Gives you the starting page of the index as 1:4664

 

Now that you know where the index is you can use DBCC PAGE (described here: https://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx ) to see what’s on it.  You know from the previous scripts that the index covers 81 pages so add 1 to the starting page to put you into the index tree.

 

DBCC TRACEON (3604, -1)

GO

DBCC PAGE (AdventureWorks, 1, 4665, 1)

 

The important output here  is the content of Slot 0 (the first row on the page) and Slot 298 (the last row on the page):

 

Slot 0, Offset 0x60, Length 25, DumpStyle BYTE

 

Record Type = INDEX_RECORD           Record Attributes =                 

Memory Dump @0x61C4C060

 

00000000:   06852aa3 b42f26c8 4988b004 3267c944 †..*../&.I...2g.D        

00000010:   caae2b00 004e2f00 00†††††††††††††††††..+..N/..                

 

 

Slot 298, Offset 0x1d7a, Length 25, DumpStyle BYTE

 

Record Type = INDEX_RECORD           Record Attributes =                 

Memory Dump @0x61C4DD7A

 

00000000:   0613f0ce 88184df0 4d8b5508 3d90ba87 †......M.M.U.=...        

00000010:   e71e5800 008f3c00 00†††††††††††††††††..X...<..    

 

Now you can stop SQL Server and load the mdf file into the Hex Editor.

 

Search for 06852aa3.  The first result takes you to offset 38215776 (decimal) and the subsequent hex values exactly match what you know of the contents of the first row from the DBCC PAGE results.  Continuing the search reveals no other exact pattern matches.

 

Search for 0613f0ce.  The first result takes you to 38223226 (decimal) for another exact match and there are no other instances of value.

 

You now have the beginning and end of the page you want to corrupt and all that’s left to do is to overwrite the data in between with zero’s.  In Hex Workshop you can highlight the data and press <Ctrl-I> to overwrite it all.  Save the file and restart SQL Server.

 

Now run DBCC CHECKDB (AdventureWorks) and you’ll see:

 

Msg 8928, Level 16, State 1, Line 1

Object ID 725577623, index ID 2, partition ID 72057594048872448, alloc unit ID 72057594054311936 (type In-row data): Page (1:4665) could not be processed.  See other errors for details.

Msg 8941, Level 16, State 55, Line 1

Table error: Object ID 725577623, index ID 2, partition ID 72057594048872448, alloc unit ID 72057594054311936 (type In-row data), page (1:4665). Test (dbccAuditRecSucceeded == TRUE) failed. Slot 0, offset 0x60 is invalid.

 

Horray, mission accomplished ! J You can even back it up and add the restore into your demo/lab steps.

 

Imagine this was your production database, what would you do? Restore? DBCC CHECKDB REPAIR_REBUILD? They’d both work but they’re unnecessary overkill in this situation.  Find out what the index is first:

 

SELECT object_name(object_id)AS 'table',name,index_id,type_desc FROM sys.indexes

WHERE object_id = '725577623'

AND index_id = 2

 

Results

 

table             name                         index_id type_desc

----------------- ---------------------------- -------- ------------

CustomerAddress   AK_CustomerAddress_rowguid          2 NONCLUSTERED

 

Its a nonclustered index so we can rebuild it!

 

ALTER INDEX AK_CustomerAddress_rowguid ON Sales.CustomerAddress REBUILD

 

And now CHECKDB runs without error. Hooray, you’ve just saved the day!

 

 

 

 

Christian

 

 

Christian Bolton

http://coeo.com

Filed under: ,
My first post
06 October 07 08:47 PM | Christian | 2 comment(s)

Hello, my name is Christian Bolton and I’ve spent the last five years working as a Premier Field Engineer for SQL Server at Microsoft UK.  I’m now a Director and Database Architect for a Microsoft Partner called Coeo (http://coeo.com) who specialise in deep-dive SQL Server engagements.  My specialist areas are High-Availability, Scalability, and Performance Tuning.  I’ve finally got around to setting up a blog and I’m going to be posting on my experiences working with SQL Server at some of Microsoft’s largest customers as well some interesting work I’ve got lined up in my new role at Coeo.  I hope you find it useful!