I am having a lot of fun watching everyone get excited over SQL Server 2014’s Delayed Durability feature, mostly because I invented it back in 1986. At the time no one was particularly excited by the idea. It’s possible someone invented it before me, but I never found any evidence of that.
Not long after taking over as Project Leader for DEC’s Rdb product I was looking at ways to address one of its performance bottlenecks, the wait to flush the log on a commit. For those not schooled in database technology a key part of ensuring no data is lost on a system failure (a.k.a., durability) is to require that changes be written (forcibly if necessary) to the log file before you acknowledge a transaction has been committed. The log file is sequential, so writing to it is enormously faster than writing changes back to the database itself. But you still have to wait for the write to complete. The architecture of Rdb 1.x and 2.x did not allow for what is now known as Group Commit or a number of other techniques for speeding up commit. Further each database connection had its own log, so that even log writing typically required a seek (i.e., it was still random rather than serial) thus limiting throughput and typically imposing a 100+ms delay to commit. On heavily loaded systems I remember this climbing to 250ms or more. Since we couldn’t implement Group Commit in a minor release, I was thinking about other answers and had a revelation.
For many applications a business transaction (e.g., add a customer) is actually a series of database transactions. From the application perspective, the customer add is not complete until the final commit of the series of database transactions, and thus they already have (or could easily be written to have) recovery logic that deals with failures (e.g., system crashes) between those individual database transactions. In effect, the durability of those individual database transactions was optional, until the final one that completed the business transaction.
With this in mind I went and prototyped Delayed Durability as an option for Rdb transactions. On Rdb it was quite simple to implement and I literally had it working in one evening. But these were short turn-around releases, I was treading into another team’s area (the KODA storage engine), and there just wasn’t time to finish productizing it. So a couple of weeks later I pulled out the change and in Rdb 3.x we started working on other (app transparent) solutions to the synchronous commit performance problem.
Now jump forward to 1994 after I’ve joined Microsoft. There is somewhat of a battle going on between the team working on the first version of Microsoft Exchange (nee 4.0) and the JET Blue storage engine team over performance issues Exchange was having. Because I was new to Microsoft and had no biases I was asked to look into Exchange’s performance problems. That was quite the experience but I’ll limit this to just the relevant story. I learned that to send an email the client did a series (30-40 pops into mind as typical) of MAPI property set calls. And each one of those calls was turned into an individual database transaction. Which of course meant 30-40 synchronous log flushes per email message! No wonder they were having significant performance problems. While my major recommendation was that they find a way to group those property sets into a single transaction, I had another trick up my sleeve.
After confirming that Exchange was fully designed to handle system failures between those MAPI Property Set “transactions” I suggested to Ian Jose, the Development Lead for JET Blue, that he implement Delayed (I think I called it Deferred at the time) Durability. The next day he told me they’d added it, and so to the best of my knowledge the first product to ship with Delayed Durability was Exchange 4.0 in April 1996. A full decade after I first proposed the idea. Of course that wasn’t visible to anyone except through the greatly improved performance it provided. But still I was quite proud to see my almost trivial little child born.
With SQL Server 2014 shipping Delayed Durability as a user visible feature my little child is finally reaching maturity. It only took 28 years.
Update: My friend Yuri pointed out that Oracle implemented Asynchronous Commit in Oracle 10gR2 in 2006. So it only took 20 years, not 28, from my invention until the feature appeared in a commercial product.