Benjamin Nevarez Rotating Header Image


SQL Server on Linux Foreword by Kalen Delaney

I started working with SQL Server more than 30 years ago, when it was a Sybase product that ran on Unix (as well as on more than half a dozen other operating systems), and we did all our work on Unix machines. SQL Server didn’t run on Windows at that time, because there was no Windows operating system. But several years later, when Sybase partnered with Microsoft to port its database product onto PC-based operating systems, few people could foresee just how powerful and ubiquitous these PCs would become. PC is hardly about only personal computers these days.

I’ve seen a lot of changes with SQL Server over these three decades, and some of the most interesting ones for me are when the product seems to circle back and add features or internal behaviors that were originally part of the product but had once been discarded. Some of the original ideas were not that far off base after all. Having SQL Server return to its roots and become available on Linux (a Unix-based OS) in SQL Server 2017 almost seems like coming home.

Benjamin Nevarez has been working with Unix-based operating systems almost as long as I’ve been working with SQL Server. He was also excited to see SQL Server make an appearance on Linux. It didn’t take him long to decide to get his hands dirty and figure out how SQL Server professionals could get the maximum value out of the new OS. He wrote this book to make available to others all that he had learned.

I have known Ben for more than a dozen years, since he first started finding typos and other errors in my SQL Server 2005 books. We started a dialog, and I then asked him if he was interested in being a technical editor for some of my work. Through this technical collaboration, I have learned that when Ben sets out to learn something, he does it thoroughly. His attention to detail and passion for complete answers never cease to amaze me.

In this book, Ben tells you how to get started using SQL Server on Linux and how the database system actually works on the new platform. Chapters 1, 2, and 3 are particularly useful if you’re new to Linux but experienced with SQL Server. Although most SQL Server books wouldn’t go into operating system administration details, Chapter 3 does just that, to make the transition easier for people who have many years, if not decades, of experience with Windows. Of course, if you’re already proficient with Linux, but new to SQL Server, you can focus on the following chapters when Ben’s expertise with SQL Server shines through.

In Chapter 4, he tells you how SQL Server can be configured and blends SQL Server details with the Linux tools you need to access and control your SQL Server. Chapters 5 and 6 are very SQL Server focused. Chapter 5 provides some very detailed information about working with SQL Server queries, including how queries are optimized and processed, and how you can tune slow-running queries. Chapter 6 tells you all about some of the latest and greatest in SQL Server’s optimization techniques in the most recent versions of SQL Server. Finally, in Chapters 7 and 8, he provides coverage of two critical focus areas for a database administrator: managing availability and recoverability, and setting up security. These are critical topics for any DBA, and because they involve the relationship between the database engine and the operating system, it’s best to learn about them from someone who is an expert in both areas.

Although both Linux and SQL Server are huge topics and there is no way one book can provide everything you need to know about both technologies, Ben has done an awesome job of giving you exactly what you need to know, not only to get SQL Server running on the Linux operating system, but to have it performing well, while keeping your data safe and secure.

—Kalen Delaney

Poulsbo, Washington, March 2018

Why I decided to write a book about SQL Server on Linux

I started my IT career working with Unix applications and databases back in the early ’90s, and my first job ever was as a data processing manager for a small IT shop. Back then, I was running Unix System V Release 4 on an NCR system. With such big and expensive minicomputer systems, I was always wondering if I could have a Unix system on less expensive hardware, such as a PC, to learn and test without disrupting our shared test systems.

Then I read an article in a personal computing magazine about something called Linux. Nobody knew what Linux was back in those days. Very few people—mostly at universities—had even access to the Internet back then. So I downloaded Linux on four or five floppy disks, installed it on a PC, and started playing with it. It was a distribution called Slackware. It was amazing that I could finally experiment and test everything I wanted on my own personal Unix system.

I continued to work with Linux and all the popular Unix commercial implementations, including IBM AIX, HP-UX, Sun Solaris, and others, throughout the ’90s. For several years, people still didn’t know what Linux was. It was not an immediate success. But by the end of the ’90s I decided to specialize in SQL Server, and by doing that I left the Unix world behind.

So it looked like I was not going to touch a Unix system ever again. But one day in March 2016, Microsoft surprised the technology community by announcing that SQL Server would be available on Linux. When I first heard the news, I thought it would be cool to write a book about it. Because I was just finishing a book about SQL Server 2016, I decided to wait to see how the technology evolved and to take a break from writing. One day, as I was running while training for a marathon, I started thinking about the project again and decided it could be a great idea to write a book about SQL Server on Linux. Just after finishing my run, I went to my laptop and sent an e-mail to my contact at McGraw-Hill Education, who eventually connected me with Lisa McClain. Within a few days, I was now working on this new book project.

Let me tell you how I structured this book.

Chapter 1 shows you how to get SQL Server running on Linux as quickly as possible, so you can start using the technology, even though I haven’t covered all the details yet. The chapter covers how to install SQL Server on Red Hat Enterprise Linux, SUSE Linux Enterprise Server, and Ubuntu and how to configure an image of SQL Server on a Docker container. More details of the setup and configuration are included in Chapter 4.

Chapter 2 covers some SQL Server history with different operating systems and explains some of the details about how SQL Server on Linux works. This includes describing the interaction between SQL Server and the operating system, decisions regarding its architecture, and information about its software implementation, among other related topics. It also covers details about the SQL Operating System (SQLOS), the Drawbridge technology, and the SQL Platform Abstraction Layer (SQLPAL).

I include an entire chapter dedicated to Linux for the SQL Server professional. Chapter 3 covers all the basic Linux commands you need to get started, including managing files and directories and their permissions, along with a few more advanced topics, including system monitoring.

Chapter 4 covers SQL Server setup and configuration in a Linux environment, and it is divided into three main topics: using the mssql-conf utility to configure SQL Server, which is required in Linux environments; using Linux-specific kernel settings and operating system configurations; and using some traditional SQL Server configurations for both Windows and Linux installations.

After spending time learning how to set up and configure SQL Server, you’ll move to Chapter 5, which discusses how to use SQL Server to perform database operations. This chapter, in particular, covers query tuning and optimization topics, which are applicable both to Windows and Linux installations—and, in fact, to all the currently supported versions of the product.

Chapter 6 continues with query processing and covers the new features available in SQL Server 2017, such as adaptive query processing and automatic tuning.

Chapter 7 is about high-availability and disaster-recovery solutions for SQL Server on Linux and focuses on Always On availability groups. Availability groups on both Windows and Linux can be used in high-availability and disaster-recovery configurations and for migrations and upgrades, or even to scale out readable copies of one or more databases. The chapter also covers Pacemaker, a clustering solution available on Linux distributions.

Finally, I close the book with Chapter 8, which is about security. This chapter reviews security from a general point of view and includes details about some of the new security features in SQL Server, including Transparent Data Encryption, Always Encrypted, Row-Level Security, and Dynamic Data Masking.

High Performance SQL Server

High Performance SQL Server

I am very excited to announce that my new book, High Performance SQL Server, is now available on Amazon and, in this post, I include the Introduction of the book, which covers what this book is about and describes the content of each of its nine chapters.

I’ve been writing and presenting about query tuning and optimization for years. I even wrote a book on the topic called SQL Server 2014 Query Tuning and Optimization. Query tuning and optimization is extremely important to improve the performance of your databases and applications.

Equally important is having a well-designed and configured system in the first place. SQL Server default configuration can work fine for some applications, but mission critical and high-performance applications demand a thoughtful design and configuration. Well-written and tuned queries will not shine if a system is not properly configured. For example, queries will not use processor resources in an optimal way if a maximum degree of parallelism setting is not configured correctly. Database performance will suffer if a database is using the default file auto-growth settings or if the storage is not properly configured. A misconfigured  tempdb  may show contention on many busy systems. Even the query optimizer will struggle with a bad database design or badly written queries. These are just some common configuration problems out there in real production systems.

In addition, even when a well-designed application goes to production, performance tuning does not end there. Monitoring and troubleshooting are an extremely important part of an application and database life cycle since performance problems eventually will arise. Workloads may change, hopefully for the better (for example, an application having to deal with an unexpected increase on business transactions). Sometimes those changes will require a redesign, changes, and perhaps new configurations.

So this is, in fact, an iterative process, consisting of design and configuration, followed by implementation, monitoring, and troubleshooting, which again may lead to new designs or configurations, monitoring, and so on. In addition, collecting performance data, creating a baseline, and performing trend analysis is an important part of a production implementation, not only to troubleshoot problems but also to anticipate issues or understand future growth and additional requirements. It is essential to estimate and trend those changes proactively instead of waking up to a system suddenly having trouble in handling changing workloads or, even worse, to face a downtime that could have been avoided. There are several tools to help with this, and the great news is that SQL Server 2016 offers a very promising one called the Query Store.

I spent a good part of my daily job working on all these items so I decided to write a book about them. I wanted to cover everything you need to know about performance in SQL Server that does not require you to know about query tuning, work with execution plans, or “fight” the query optimizer. There are so many areas to cover and more are being added as new features and technologies appear on SQL Server such as In-Memory OLTP, columnstore indexes, and the aforementioned Query Store.

This book covers all currently supported versions of SQL Server with a major focus on SQL Server 2016. Although this is a performance book from the practical point of view, SQL Server internals are very important too. The best way to troubleshoot something is to know how it works and why things happen. Consequently, I focus on internals when required.

Finally, this book complements my query tuning and optimization book. If you are a database developer or a SQL Server professional who cares about query performance, you could benefit from both books. If you are a database administrator, a database architect, or even a system administrator, and you want to improve the performance of your system without query tuning, you can read only this book.

As mentioned earlier, understanding SQL Server internals is important to better optimize a system and troubleshoot database problems, so this book starts explaining how the SQL Server database engine works and covers everything happening in the system from the moment a connection is made to a database until a query is executed and the results are returned to the client. Chapter 1 includes such topics such as the Tabular Data Stream (TDS) and network protocols used by SQL Server, SQLOS, and the work performed by the SQL Server relational engine, focusing on query processing and the most common query operators.

Waits happen in an SQL Server instance all the time. Chapter 2 introduces the waits performance methodology, which can be used to troubleshoot performance problems, especially when other methods are not able to pinpoint a performance issue.

Chapter 3 covers the Query Store, a very promising query performance feature introduced with SQL Server 2016. The Query Store can help you to collect query and plan information along with their runtime statistics, which you can use to easily identify query-performance-related problems and even force an existing execution plan. The chapter closes by mentioning some related new features such as the Live Query Statistics and the SQL Server Management Studio plan comparison tool.

Chapter 4 explains a number of instance level configuration settings that can greatly impact the performance of your SQL Server implementation. As an interesting fact, it shows how some trace flags originally introduced to solve a particular problem are now implemented as SQL Server configuration defaults.

Chapter 5 covers  tempdb  configuration, which is especially important as such a database is shared between all the user and system databases on a SQL Server instance. Focus on the chapter is given to  tempdb  latch contention of allocation pages and  tempdb  disk spilling, a performance issue that occurs when not enough memory is available for some query processor operations.

In-memory technologies are introduced in Chapter 6, including In-Memory OLTP, which in SQL Server 2016 sees its second release, and columnstore indexes, now on its third mayor implementation. Both features suffered severe limitations with their original release so this chapter covers how these technologies work and what their current improvements are. The chapter ends with Operational Analytics, which combines both technologies to allow analytical queries to be executed in real-time in an OLTP system. In-memory technologies promise to be the future in relational database technologies.

Chapter 7 shows how proactively collecting and persisting performance information could be extremely beneficial to understand how a specific system works, to create a baseline, and to understand when performance is deviating from a desirable or expected behavior. The chapter also covers the most critical performance counters, dynamic management objects, and events, along with some of the tools used to display and collect such data.

Indexing, a required topic for database performance, is covered in Chapter 8. It explains how indexes work and why they are important on both OLTP and Data Warehouse environments. The chapter provides emphasis on using SQL Server tools to help create indexes such as the missing indexes feature and the more sophisticated Database Engine Tuning Advisor.

Finally, SQL Server storage is explained in Chapter 9. Disk has traditionally been the slowest part of a database system, but newer technologies such as flash-based storage offer great performance improvements and are becoming a de facto enterprise standard as their cost continues to decline. The chapter also indicates that storage optimization is not only about using the fastest disk possible but also minimizing its usage by implementing the methods covered in several chapters of the book, such as proper indexing or some query tuning techniques.

SQL Server 2014 Query Tuning & Optimization Book Code

Today I was finally able to finish extracting the code of my new book Microsoft SQL Server 2014 Query Tuning & Optimization after a few readers requested it recently but I had been busy with some traveling in the last few weeks. The book contains a large number of SQL queries, all of which are based on the AdventureWorks2012 database, except Chapter 9 which uses the AdventureWorksDW2012 database. All code has been tested on SQL Server 2014. Note that these sample databases are not included by default in your SQL Server installation, but can be downloaded from the CodePlex website.

To download the code click here Query Tuning and Optimization Code.

New Book: SQL Server 2014 Query Tuning & Optimization



It looks like my blog has been very quiet for a while once again. Yes, I’ve been busy working on some other SQL Server projects since last year. One of those projects was working on Kalen Delaney’s new book SQL Server 2012 Internals where I co-authored two chapters, “Special Databases” and “Query Execution”, which were originally written for previous versions of the book by Kalen and Craig Freedman, respectively. The book was finally released last November.

As soon as I finished my part on Kalen’s book, I immediately started working on my new book, SQL Server 2014 Query Tuning & Optimization, which has been available for pre-order at Amazon for a while now. This book covers everything you need to know about query tuning and optimization for SQL Server to help you design high-performing database applications. As a SQL Server 2014 book it obviously covers new features like Hekaton, the new cardinality estimator, and incremental statistics, among others, but most of the content can be used in previous versions of SQL Server as well. I am currently working on the last items of the book with the great team at McGraw-Hill Education and the help of the amazing Dave Ballantyne (@davebally) as technical reviewer and the book should be available soon.

Also, at the last PASS Summit in Charlotte I was invited by SQL Sentry to blog at their site, which has great content by some of the most amazing bloggers in the SQL Server community. Although I haven’t had much time to blog there as I would wanted, I have started with a few SQL Server 2014 articles that you can read here.

Finally, I have submitted sessions for SQLBits (first time ever submitted) and to the PASS Summit (which I have attended for 11 years so far), and just at the moment I am writing this I got the great news that I got two sessions selected for SQLBits so I am really looking forward to finally attend this conference in the UK this July. I am also planning to attend some other SQL Server events in the US, including our always amazing local SQLSaturdays in Orange County this April 26th, Sacramento on July 12th and San Diego, which is usually scheduled for September. Hope to see you at any of these SQL Server events.

Code from my book Inside the SQL Server Query Optimizer

Recently I’ve been requested the code of my book Inside the Server Query Optimizer so I am including it in this post. The book contains a large number of example SQL queries, all of which are based on the AdventureWorks database and Chapter 6 additionally uses the AdventureWorksDW database. All code has been tested on both SQL Server 2008 and SQL Server 2008 R2. Note that these sample databases are not included by default in your SQL Server installation, but can be downloaded from the CodePlex website.

Inside the SQL Server Query Optimizer code – InsideQueryOptimizerCode.txt

My book “Inside the SQL Server Query Optimizer” Now Available on Amazon

I just found today that my book, “Inside the SQL Server Query Optimizer”, is finally available on Amazon. You can find the description of the book on the Amazon page or a more detailed description chapter by chapter on one of my previous posts here. So I would like to use this post to thank the people who helped me or in some way influenced the content of the book.

Writing this book was a lot of fun, but also a lot of work; actually a lot more work than I originally expected. Fortunately I got help from several people. First of all, I would like to thank Chris Massey. Chris helped me as the technical editor of the book, and guided me through most of the writing process, doing absolutely outstanding work. Very special thanks also go out to Grant Fritchey who helped us as the technical reviewer, and went very much beyond just reviewing, as his comments provided invaluable feedback to improve the quality of this book. Thanks also go to Tony Davis for offering me this opportunity in the first place, and helping to get the project started.

Outside the Red Gate team, my deepest gratitude goes to Cesar Galindo-Legaria, Manager of the SQL Server Query Optimization team at Microsoft, for answering my endless list of questions about the Query Optimizer, and educating me through some of the information he has published, including numerous research papers and an excellent chapter of a SQL Server book. I had the opportunity to meet Tony, Cesar, and Grant at the PASS Summit back in 2009, when I first started this journey, and I hope to finally meet Chris at the same conference in October 2011.

Although Chris, Cesar, Grant and Tony have directly helped me to shape the content of this book, there’s also a host of other people who have indirectly influenced the book through their work, which helped me to learn about and better understand the SQL Server query processor. With that in mind, I would like to thank the authors who have discussed the query processor in some of the available SQL Server books, Microsoft white papers and blogs, including Kalen Delaney, Ken Henderson, Lubor Kollar, Craig Freedman, Conor Cunningham and Eric Hanson.

Research papers have provided me with an unlimited source of information, and helped me to understand the Query Optimizer at a significantly deeper level than ever before. So, thanks to all that amazing work by the many people in the research community including, in addition to Cesar, Goetz Graefe, Surajit Chaudhuri, Yannis Ioannidis, Vivek Narasayya, Pat Selinger, Florian Waas, and many, many more.

Finally, on the personal side, I would like to thank my parents, Guadalupe and Humberto, and my family: my wife Rocio, and my three sons, Diego, Benjamin and David. Thanks all for your unconditional support and patience.

My Book, “Inside the Query Optimizer”, available at the PASS Summit

My book, “Inside the SQL Server Query Optimizer”, is almost finished and we will have a conference edition of it available at the PASS Summit. The final version of the book, published by Red Gate books, will be available on Amazon by Christmas.

For more details on the contents, I am including the Preface of the book next.



The Query Optimizer has always been one of my favorite SQL Server topics, which is why I started blogging about it and submitting related presentations to PASS. And so it would have continued, except that, after several blog posts discussing the Query Optimizer, Red Gate invited me to write a book about it. This is that book.

I started learning about the Query Optimizer by reading the very few SQL Server books which discussed the topic, and most of them covered it only very briefly. Yet I pressed on, and later, while trying to learn more about the topic, I found an extremely rich source of information in the form of the many available research papers. It was hard to fully grasp them at the beginning, as academic papers can be difficult to read and understand, but soon I got used to them, and was all the more knowledgeable for it.

Having said that, I feel that I’m in a bit of a minority, and that many people still see the Query Optimizer just as a black box where a query is submitted and an amazing execution plan is returned. It is also seen as a very complex component, and rightly so. It definitely is a very complex component, perhaps the most complex in database management software, but there is still a lot of great information about the Query Optimizer that SQL Server professionals can benefit from.  

The Query Optimizer is the SQL Server component that tries to give you an optimal execution plan for your queries and, just as importantly, tries to find that execution plan as quickly as possible. A better understanding of what the Query Optimizer does behind the scenes can help you to improve the performance of your databases and applications, and this book explains the core concepts behind how the SQL Server Query Optimizer works. With this knowledge, you’ll be able to write better queries, provide the Query Optimizer with the information it needs to produce efficient execution plans, and troubleshoot the cases when the Query Optimizer is not giving you a good plan.

With that in mind, and in case it’s not obvious, the content of this book is intended for SQL Server professionals: database developers and administrators, data architects, and basically anybody who submits more than just trivial queries to SQL Server. Here’s a quick overview of what the book covers:

The first chapter, Introduction to Query Optimization, starts with an overview on how the SQL Server Query Optimizer works and introduces the concepts that will be covered in more detail in the rest of the book. A look into some of the challenges query optimizers still face today is covered next, along with a section on how to read and understand execution plans. The Chapter closes with a discussion of join ordering, traditionally one of the most complex problems in query optimization.

The second chapter talks about the Execution Engine, and describes it as a collection of physical operators that perform the functions of the query processor. It emphasizes how these operations, implemented by the Execution Engine, define the choices available to the Query Optimizer when building execution plans. This Chapter includes sections on data access operations, the concepts of sorting and hashing, aggregations, and joins, to conclude with a brief introduction to parallelism.

Chapter 3, Statistics and Cost Estimation, shows how the quality of the execution plans generated by the Query Optimizer is directly related to the accuracy of its cardinality and cost estimations. The Chapter describes Statistics objects in detail, and includes some sections on how statistics are created and maintained, as well as how they are used by the Query Optimizer. We’ll also take a look at how to detect cardinality estimation errors, which may cause the Query Optimizer to choose inefficient plans, together with some recommendations on how to avoid and fix these problems. Just to round off the subject, the chapter ends with and introduction to cost estimation.

Chapter 4, Index selection, shows how SQL Server can speed up your queries and dramatically improve the performance of your applications just by using the right indexes. The Chapter shows how SQL Server selects indexes, how you can provide better indexes, and how you can verify your execution plans to make sure these indexes are correctly used. We’ll talk about the Database Engine Tuning Advisor and the Missing Indexes feature, which will show how the Query Optimizer itself can provide you with index tuning recommendations.

Chapter 5, The Optimization Process, is the Chapter that goes right into the internals of the Query Optimizer and introduces the steps that it performs without you ever knowing. This covers everything from the moment a query is submitted to SQL Server until an execution plan is generated and is ready to be executed, including steps like parsing, binding, simplification, trivial plan and full optimization. Important components which are part of the Query Optimizer architecture, such as transformation rules and the memo structure, are also introduced.

Chapter 6, Additional Topics, includes a variety of subjects, starting with the basics of update operations and how they also need to be optimized just like any other query, so that they can be performed as quickly as possible. We’ll have an introduction to Data Warehousing and how SQL Server optimizes star queries, before launching into a detailed explanation of Parameter sniffing, along with some recommendations on how to avoid some problems presented by this behavior. Continuing with the topic of parameters, the Chapter concludes by discussing auto-parameterization and forced parameterization.

Chapter 7 describes Hints, and warns that, although hints are a powerful tool which allows you to take explicit control over the execution plan of a query, they need to be used with caution and only as a last resort when no other option is available. The chapter covers the most-used hints, and ends with a couple of sections on plan guides and the USE PLAN query hint.

Before we get started, please bear in mind that this book contains many undocumented SQL Server statements. These statements are provided only as a way to explore and understand the Query Optimizer and, as such, should not be used on a production environment. Use them wisely, and I hope you enjoy learning about this topic as much as I do.

Benjamin Nevarez

Welcome to my New Blog

I started blogging about SQL Server at SQLblog back in July, 2009 and it was a lot of fun, I enjoyed it a lot. Then later, after a series of blog posts about the Query Optimizer, I was invited to write an entire book about that same topic. But after a few months I realized that it was going to be hard to continue both blogging and writing chapters for a book, this in addition to my regular day job, so I decided to stop blogging for a little while.

Now that I have finished the last chapter of the book and I am working on the final chapter reviews, I decided to start blogging again, this time in this new blog.

Same as my old blog I plan to write about my topics of interest, like the relational engine, and basically anything related to SQL Server. Hopefully you find this blog interesting and useful.