The evidence is growing that Microsoft’s decision to provide developers little or no access to Windows RT systems prior to general availability is coming back to bite them.  In particular, there are reports that performance is sub par.  First we had reports that some third-party apps worked poorly on Surface, then that Office 2013 was sluggish, and now browser benchmarks show the Surface is slower than recent iPads and other devices.  And that is no surprise to anyone who has worked in the performance realm.  Indeed Windows President Steven Sinofsky’s rebuttal to complaints from a member of the Office 2013 team indirectly tells us what the problem is.  You can’t get good performance by doing work on anything less than real hardware and it is hard to tell who had access to any real hardware.  Third Party software developers didn’t have access to Windows RT systems at all, and it appears that Microsoft’s own teams didn’t have access to production quality systems.

The problem of software developers having real hardware for a sufficient period to tune their software has been with us since the beginning of the computer industry.  Hardware drove the industry at first, and the software guys were expected to keep up.  Remember you put vast amount of capital into building factories, buying parts inventory, stocking replacements at repair centers, and then training both manufacturing and support personnel.  When the hardware is ready to ship you can’t just leave systems sitting on the shipping dock waiting for the software as that has a material impact on financial results.  So software developers are pressured to take shortcuts to have their software ready to go when the hardware is ready.

An early example of why this is a problem comes from DEC, which in the days of the PDP-11 had numerous operating systems for it.  In order to make the problem of providing hardware to these operating system development groups tractable a hardware pool was created.  Operating System developers would then schedule time to use the hardware in the pool.  On one release of RSTS/E the developer had just a few hours with a particular piece of hardware and the release shipped claiming support for it but not actually working.  The debacle was so huge that from that point forward the RSTS/E group set a policy that they would support hardware only if they were provided with dedicated devices far enough in advance to allow for proper development, debugging, and testing.   The policy stuck, and represented a major rebalancing of power between hardware and software groups.

For a slightly more recent and directly applicable example we have the >8 processor support in Microsoft SQL Server 2000.  We’d intended to support 32-processor systems in SQL Server 2000 from the beginning and were working with the one piece of hardware available to us, the Unisys ES/7000.   Of course Unisys was developing the hardware at the same time, and the SQL Server lab only had an early prototype without many of the optimizations that Unisys was planning for the final product.  Poor initial performance was linked to the lack of those hardware optimizations, and eventually we were forced to delay 32-processor support until after the initial release.  Once the SQL Server team had access to a full production ES/7000 we discovered that, oops, it wasn’t the lack of optimizations in the prototype but rather that SQL Server needed considerable work to optimize algorithms for it (and similar machines like HP’s Superdome).  The immature hardware was masking the areas where SQL Server needed work.  These changes, almost sufficient for us to consider a separate minor release, were folded into SQL Server 2000 SP1.

The SQL Server experience, and many others by the way (e.g., I could give numerous VAX and Alpha examples not to mention other Windows examples), show that you (a) need to do performance optimization for any new system and (b) you need near-final hardware to do that.  But that does not appear to be the way Microsoft approached the Surface and Windows RT.

Even within Windows teams (that didn’t have known ARM-specific work) were originally told that they didn’t need to plan any work to support Windows on ARM (WoA).  Basically the build team would build everything for ARM and then let you know if they had any problems.  I don’t know if this was part of the efforts to keep WoA secret or a genuine belief by management that no significant work was required.  And I don’t know when or if this changed, though the change would have been after Windows 8 planning had wrapped up.  In any case, it appears that many (most?) teams within Windows would not have had dedicated (even prototype) ARM hardware at their disposal nor people scheduled to do any evaluation or optimization work.

So what of the Internet Explorer team?  And particularly the Chakra JavaScript Engine team?  They clearly had resources devoted to WoA work, but did they have real near-production hardware to use?  And if so, did they have it in time to do meaningful tuning before Windows 8 RTM?

I think that’s the problem for Microsoft Office as well.  Surprisingly late in the cycle someone mentioned to me that they didn’t yet know what the Office team was doing to support tablets, let alone WoA.  This suggests that Office kept this to a small team, meaning most of Office engineering did not have access to ARM hardware nor expertise or expectation of needing to do any WoA-related work.  And even the small team focusing on WoA might not have had access to near-production grade systems to perform performance evaluation and tuning on.

Of course there was no broad, or apparently even narrow, beta of WoA.  And no distribution of systems to developers so they could test their own apps.  So anything that might have caught performance issues in advance, and allowed time to resolve them before end-user availability, appears to have been forgone.  Microsoft basically bet that, other than things close to the metal like the HAL, boot path,  and drivers, that no tuning for WoA systems was required.  They were wrong.

The good news is that Microsoft has adopted an aggressive update posture for Windows 8 and Windows RT.  Apps distributed through the Windows Store can be updated on a weekly, or even more frequent, basis.  And Microsoft doesn’t seem shy about using Windows Update to update other Windows (or Microsoft) components.  So just as Microsoft responded quickly to the initial Office 2013 performance issue, we can expect they will address other Surface, Windows RT, and Windows 8 performance issues over the coming weeks and months.

Microsoft may have counted on its rapid update ability to mitigate their strategy of developing Windows RT and the Surface in the dark.  But first, and particularly in the case of the Surface, they have to worry about performance issues (real, or benchmark positioning) permanently damaging their new product family’s reputation.  Because once that happens it, and not reality, will dominate the conversation.