Performance of the Microsoft Surface and Windows RT

The evidence is growing that Microsoft’s decision to provide developers little or no access to Windows RT systems prior to general availability is coming back to bite them.  In particular, there are reports that performance is sub par.  First we had reports that some third-party apps worked poorly on Surface, then that Office 2013 was sluggish, and now browser benchmarks show the Surface is slower than recent iPads and other devices.  And that is no surprise to anyone who has worked in the performance realm.  Indeed Windows President Steven Sinofsky’s rebuttal to complaints from a member of the Office 2013 team indirectly tells us what the problem is.  You can’t get good performance by doing work on anything less than real hardware and it is hard to tell who had access to any real hardware.  Third Party software developers didn’t have access to Windows RT systems at all, and it appears that Microsoft’s own teams didn’t have access to production quality systems.

The problem of software developers having real hardware for a sufficient period to tune their software has been with us since the beginning of the computer industry.  Hardware drove the industry at first, and the software guys were expected to keep up.  Remember you put vast amount of capital into building factories, buying parts inventory, stocking replacements at repair centers, and then training both manufacturing and support personnel.  When the hardware is ready to ship you can’t just leave systems sitting on the shipping dock waiting for the software as that has a material impact on financial results.  So software developers are pressured to take shortcuts to have their software ready to go when the hardware is ready.

An early example of why this is a problem comes from DEC, which in the days of the PDP-11 had numerous operating systems for it.  In order to make the problem of providing hardware to these operating system development groups tractable a hardware pool was created.  Operating System developers would then schedule time to use the hardware in the pool.  On one release of RSTS/E the developer had just a few hours with a particular piece of hardware and the release shipped claiming support for it but not actually working.  The debacle was so huge that from that point forward the RSTS/E group set a policy that they would support hardware only if they were provided with dedicated devices far enough in advance to allow for proper development, debugging, and testing.   The policy stuck, and represented a major rebalancing of power between hardware and software groups.

For a slightly more recent and directly applicable example we have the >8 processor support in Microsoft SQL Server 2000.  We’d intended to support 32-processor systems in SQL Server 2000 from the beginning and were working with the one piece of hardware available to us, the Unisys ES/7000.   Of course Unisys was developing the hardware at the same time, and the SQL Server lab only had an early prototype without many of the optimizations that Unisys was planning for the final product.  Poor initial performance was linked to the lack of those hardware optimizations, and eventually we were forced to delay 32-processor support until after the initial release.  Once the SQL Server team had access to a full production ES/7000 we discovered that, oops, it wasn’t the lack of optimizations in the prototype but rather that SQL Server needed considerable work to optimize algorithms for it (and similar machines like HP’s Superdome).  The immature hardware was masking the areas where SQL Server needed work.  These changes, almost sufficient for us to consider a separate minor release, were folded into SQL Server 2000 SP1.

The SQL Server experience, and many others by the way (e.g., I could give numerous VAX and Alpha examples not to mention other Windows examples), show that you (a) need to do performance optimization for any new system and (b) you need near-final hardware to do that.  But that does not appear to be the way Microsoft approached the Surface and Windows RT.

Even within Windows teams (that didn’t have known ARM-specific work) were originally told that they didn’t need to plan any work to support Windows on ARM (WoA).  Basically the build team would build everything for ARM and then let you know if they had any problems.  I don’t know if this was part of the efforts to keep WoA secret or a genuine belief by management that no significant work was required.  And I don’t know when or if this changed, though the change would have been after Windows 8 planning had wrapped up.  In any case, it appears that many (most?) teams within Windows would not have had dedicated (even prototype) ARM hardware at their disposal nor people scheduled to do any evaluation or optimization work.

So what of the Internet Explorer team?  And particularly the Chakra JavaScript Engine team?  They clearly had resources devoted to WoA work, but did they have real near-production hardware to use?  And if so, did they have it in time to do meaningful tuning before Windows 8 RTM?

I think that’s the problem for Microsoft Office as well.  Surprisingly late in the cycle someone mentioned to me that they didn’t yet know what the Office team was doing to support tablets, let alone WoA.  This suggests that Office kept this to a small team, meaning most of Office engineering did not have access to ARM hardware nor expertise or expectation of needing to do any WoA-related work.  And even the small team focusing on WoA might not have had access to near-production grade systems to perform performance evaluation and tuning on.

Of course there was no broad, or apparently even narrow, beta of WoA.  And no distribution of systems to developers so they could test their own apps.  So anything that might have caught performance issues in advance, and allowed time to resolve them before end-user availability, appears to have been forgone.  Microsoft basically bet that, other than things close to the metal like the HAL, boot path,  and drivers, that no tuning for WoA systems was required.  They were wrong.

The good news is that Microsoft has adopted an aggressive update posture for Windows 8 and Windows RT.  Apps distributed through the Windows Store can be updated on a weekly, or even more frequent, basis.  And Microsoft doesn’t seem shy about using Windows Update to update other Windows (or Microsoft) components.  So just as Microsoft responded quickly to the initial Office 2013 performance issue, we can expect they will address other Surface, Windows RT, and Windows 8 performance issues over the coming weeks and months.

Microsoft may have counted on its rapid update ability to mitigate their strategy of developing Windows RT and the Surface in the dark.  But first, and particularly in the case of the Surface, they have to worry about performance issues (real, or benchmark positioning) permanently damaging their new product family’s reputation.  Because once that happens it, and not reality, will dominate the conversation.

 

About these ads
This entry was posted in Computer and Internet, Microsoft, Windows and tagged , , , , , . Bookmark the permalink.

9 Responses to Performance of the Microsoft Surface and Windows RT

  1. I’ve been using RT on Surface heavily in my business and I haven’t had any performance issues in IE, Office, or anything else really. Even with music playing in the background and multiple apps open.

  2. Tom says:

    But they didn’t even need near-final Surface hardware — it’s not like the touchscreen is going to change the performance characteristics of the Chakra Javascript engine.

    What they needed was a near-final SoC. Most of the performance differences are going to be caused by architecture — heck, even Atom performs *very* differently from other x86 chips. ARM is going to be even more of a change.

    They had ARM developer boards as far back as the March 2011 announcement of Windows-on-ARM. What happened to them?

    • halberenson says:

      How many? Were they representative of final hardware? At what point was Windows RT usable enough on them to really do work? What was the relative priority of improving Chakra for x86 vs. ARM? And on ARM, what was the priority of WP8 vs. Surface? Did the Chakra team even know about the Surface, or was most of their work done on a different SoC than it uses?

  3. dave says:

    Do you have any speculative thoughts on when the next wave of software updates might be released?

    • halberenson says:

      Sadly no. I think they are predisposed to ship updates quickly when something is a very real problem, but not when it is somewhat invented. So shipping performance improvements because users are really complaining about performance could be high on the list while shipping them because benchmark scores are low is not going to be a high priority.

      Right now I think the IE team is focused on getting IE10 on Windows 7 done, so that further complicates this particular set of potential updates.

  4. Jan Olbrecht says:

    Thank you for this informative post. I can only imagine the kind of hard decisions that had to be taken between opening the development process (both software and hardware in this case), time constraints in getting the Surface ready for launch and communicating all of this both within the company and to third parties. Your experience certainly helps to add details to this mental picture.

    For me personally, I can certainly see performance issues on some applications (the Kindle app comes to mind) that just shouldn’t exist. On the other hand I am pleasantly surprised with the performance of my Surface. Given that it is basically evenly powered as for example the Nexus 7, I and others have subjectively noted how much more responsive Windows RT seems to everyday interaction. Especially screen lag while scrolling is much better on the Surface than on other devices I have used so far (same for Windows Phone).

    Aside from tech enthusiasts, it seems to me that normal consumers will not really notice any appreciable performance differences. Some time remains until Christmas presents are opened.

    On the other hand, we should not forget that Microsoft (I think deliberately) chose to use the slowest of the current generation ARM processors in its tablet. This establishes a baseline for developers and things should start looking up, not only with newer generations of hardware but also with current hardware designs based on non-NVIDIA silicon.

    With this in mind, I think it’s just the right amount of pain I expected by being an early adopter.
    Regards,
    -Jan

    • halberenson says:

      If I weren’t following a gadzillion technology news sources I’d never have known about the performance things. In my normal use the Surface is performing quite nicely. And I suspect that is what other users are having as an experience too.

      I think most negative experiences will be from specific apps where the combination of slower hardware and lack of developer optimization for Windows RT systems comes together.

  5. Eugene says:

    Great analysis. Probably the first one on this topic that really make sense. Thank you.

    P.S.: I’m selling Metro-style Visio alternative on Windows Store so I keep my fingers crossed that MS will not port Visio to Windows RT anytime soon :)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s