Practical Screen Reader Comparison: A user-oriented approach
5 Screen Readers, 3 Operating Systems,
Birkir R. Gunnarsson and Hlynur M. Hreinsson
ICELANDIC NATIONAL INSTITUTE FOR THE BLIND, PARTIALLY SIGHTED AND DEAF BLIND (INIB)
3/31/2011
We work at the Icelandic National Institute for the Blind, Visually Impaired and Deaf blind (INIB). INIB is a government organization that serves approximately 1500 individuals in Iceland (a small island with a total population of 320,000). INIB´s role is to provide services to all individuals in Iceland with a diagnosed visual impairment, regardless of their age or location within the country. Our services range from providing appropriate glasses and other vision aids to preparing alternative formats for text books and other instructional material for visually impaired students. A major component of our services is providing appropriate Assistive Technology (hardware and software) as well as all required training in the use of Assistive Technology to our users. We serve approximately 150 active A.T. users of ages ranging from 5 to 90 with varying degrees of vision impairment and technology skills. Our job involves carrying out an assessment of our user’s technology needs and competencies as well as developing individualized appropriate Assistive Technology plans based on that assessment. The outcome of this process is then communicated to our manager, our education consultants and equipment funding committees. The process is similar to the U.S. IEP (Individual Education Program), but in our case the users’ age and A.T. abilities are much more varied than that of U.S. students in grades 1 through 12. Some of our users are university students, or working professionals, who need advanced functionality such as scripting and braille support for school work or to facilitate complex jobs, others are either very young or elderly individuals getting their first taste of reading email, Facebook, blogs and newspapers on the Internet.
Once our recommendations have been reviewed and the necessary equipment purchased, it is our job to carry out suitable follow up training. Often this requires face to face training with the user, but other times writing instruction manuals or recording podcasts will suffice. As our user´s skill set evolves we reevaluate and update their A.T. assessment and may update our A.T. equipment recommendations.
Until recently we enjoyed a relatively simple kind of life. We provided everyone who was qualified to receive a Screen Reading application with a copy of a certain Screen Reader that INIB had contracted for, regardless of the technology activities that user´s would engage in. This was an acceptable procedure at the time, since we had sufficient funding and the availability of Screen Readers for the Windows platform was rather limited, and virtually non-existent outside of Windows. All Screen Readers fell, broadly speaking, into the same price range and the functional differences between them were never studied in detail, and considered to be more of a user preference than major differences in functionality. But then two important things happened:
· New Screen Readers with varying functionalities and price tags started to appear on the market, both for Windows as well as other operating systems
· We started to experience budget restrictions and our management encouraged us to find more cost effective solutions without negatively affecting our user’s experience.
We were excited to meet that challenge head on, and set out to work on the problem.
We started out by realizing that individuals who simply want to use the computer for email, web browsing and possibly some instant messaging had no interest in the advanced scripting, flexibility and Braille support offered by the traditional Screen Readers. Many of our users, especially those who lose sight later in life, do not know Braille or are in the process of learning it. Other users, especially those in university or corporate settings, have fairly specific needs, such as being able to work with certain types of software applications used throughout their organization. We realized we needed to develop an A.T. evaluation process that we could combine with an assessment of our user´s needs and skills to be able to find the A.T. set up that was optimal for them. We also realized that our users situations could force them, and therefore us, to work with a certain software configuration, regardless of what we felt was the most accessible software set up for them, and we needed to be aware of the possibilities offered by all the Screen Reading solutions on the market, so that we could help our users out in whatever situation they may find themselves in.
This can be illustrated by mentioning a few hypothetical situations:
1. A user’s school or employer decides to save money and switch over to Unix/Linux based systems entirely. What effect will this have on our hypothetical user and what would be the biggest accessibility obstacles?
2. A user wants to buy him or herself a Apple computer and use it with VoiceOver. Whereas we certainly have restrictions on whom we can support and how, we believe that people should be able to choose whatever technology they desire and our job is to train our users to get the most out of that technology.
3. A school is using a certain type of software, such as Open Office, and we need to be able to decide which Screen Reader works best with that environment. Of course we cannot evaluate every single piece of software on the market and what Screen Reader works best in every scenario, but we need to be familiar with the most commonly used software.
4. We need to understand the individual strengths of different Screen Readers. For instance, a user who needs to work on a computer that does not have a Screen Reader preinstalled can utilize Non-Visual Desktop Access (NVDA) from a USB stick, without having to install video intercept technology. Therefore, even if NVDA does not have the extensive Braille support and scripting that Jaws or Hal may have, it is the most appropriate software for this situation. An additional benefit of Open Source or built-in screen reading technology is that we can get users up and running instantly without having to wait for funding or equipment committee decisions. We can always install a different Screen Reader later on. We realized that the key to an effective A.T. evaluation process was to compare Screen Reader functionalities in more detail than we had previously and also that we could no longer ignore emerging solutions from the Open Source community as well as the efforts that Apple have been putting into their accessibility solutions. In order to gain better understanding of the Screen Reader selections we started by doing a considerable amount of Google research, trying to gage how different Screen Readers performed and what they were capable of. We soon came to the conclusion that information we got from software user groups and developer´s web sites, though useful, was inconsistent and fragmented, and could not give us the unbiased and direct comparison we wanted. After securing a week of uninterrupted testing time, an office, and generous amounts of coffee, we settled in and got down to the nitty gritty details of comparing the Screen Readers ourselves.
After some deliberation on what it meant to compare Screen Readers we settled on defining specific software tasks that we felt, based on our experience, would be commonly required by the majority of our users.
Examples of such tasks include:
· Changing the speed, language and customize the punctuation feedback in a Screen Reader (those would be three separate tasks, but all fall in the same category).
· Have Screen Reader announce a misspelled word in a word processor, and work with the word processor dictionary to suggest an alternative, and correct the misspelled word.
· To allow a user to scroll through all messages in his Inbox and announce all the flags associated with each message (read or unread, replied, has attachments and priority).
· Can Screen Reader announce all non-empty cells in a spread sheet and distinguish between cells with comments, formulas and other objects.
We realized, as demonstrated above, that a computer task is generally achieved through the interaction of three different pieces of software: the Screen Reader in use, the operating system it is running on and an application designed to perform the task.
Therefore any comparison between Screen Readers, inevitably, involved comparing the most accessible software profile (combination of Screen Reader, operating system and third party application) for each task. This adds a whole new dimension to the comparison, because even if Screen Readers are not cross platform, and so the operating system is a constant, there is often a vast selection of software applications for performing most tasks, such as managing email or for word processing, each with different levels of accessibility, keyboard support and popularity among users. We decided to go with applications that the user was likely to encounter in a professional or academic environment and avoid applications designed specifically for accessibility; after all we believe users generally have the greatest level of success if they are able to work with the same software as their colleagues. Based on our philosophy, we looked for the most accessible mainstream application for each of the tasks we tested, both based on our experience as well as input from other users. Occasionally we tested with two or more applications where we felt it necessary. For every task and every software profile we documented not only its accessibility score but also what steps (key strokes or other actions) we took to achieve the task. Whereas it definitely slowed us down significantly, we managed to create a basic user guide for all of the Screen Readers we tested, something we think will be a valuable tool for creating beginner user guides and podcasts, or we can use as a part of a face to face training and evaluation tool. Finally we had to make some assumptions about the "typical user". For this test, and based on our experience with Icelandic A.T. users, we decided that users are unlikely to customize their Screen Readers by searching for add-on scripts or perform complex custom configuration on the applications or operating system to increase accessibility. Therefore, even if a certain task can be made easier by finding a custom install script created by others or modifying advanced configuration options, we decided that such a task would be too complicated for most users and did not take that into account when rating the performance of our Screen Readers. We realize that this somewhat negates the flexibility and power of certain software environments, especially Linux, but we also feel that our results indicate that slightly better documentation and automation of installs can go a long way to allow some Screen Readers to score a lot higher in our tests. We want to look at this world through the eyes of the average user, or as close as we can come to that. In future we may certainly expand our tests and give more weight to mailing list support and script availability.
The following table shows information about the Screen Readers we tested:
JAWS for Windows v.11 |
HAL v.11.5 |
NVDA v.2010.1 |
VoiceOver |
Orca – Gnome 2.3 |
For historical reasons and benchmarking purposes, we had to include Jaws and Hal. We have years of experience working with both of those, and feel they set the standard for new Screen Reading solutions. For our third Windows Screen Reader we wanted to test a different, Open Source solution with a significant user base. We decided to test NVDA for several reasons:
· NVDA is developed largely by blind programmers
· It has a rapidly expanding user base (according to the 2010 WebAIM survey it´s user base increased by 300% between October 2009 and December 2010).
· It is portable and can be run from a USB stick on any Windows computer
· Unlike Screen Readers who hook into the Windows display drivers, NVDA can be run as a second Screen Reader without clashing with other Screen Readers.
· In our past experiments with it, we had been very impressed, so much so that we translated NVDA into Icelandic, partly in our personal time.
There are a few notable omissions from this list. Window Eyes from GW Micro is a popular option that should be included, but it is too similar in price range and functionality to Hal and Jaws for our managers to consider it for the relatively small Icelandic market at this time. We tested System Access and SaToGo from Serotek last year and were definitely impressed with some parts of it, it performed very well with Microsoft Word, for instance, but we chose NVDA over System Access for the reasons previously listed.
We hope our next set of tests will give us the opportunity to include both of these Screen Readers as well as the Open Source Thunder Screen Reader and the Cobra Screen Reader from Baum. If anyone reading this report has experience of using any of these applications and would have time or interest in helping us evaluate these, please get in touch with us. No Screen Reader study would be complete without evaluating Apple´s Voiceover, especially with the development efforts and updates that Apple has put into it as part of the Snow Leopard release. The novelty of having a built-in, fully functioning screen reading solution as part of the operating system is certainly something we applaud, but we wondered how effective it would turn out to be for the tasks our users carry out on a daily basis. For that reason Voiceover was an obvious candidate for the fourth spot on our test list. Finally, we wanted to get an idea of how effective the Orca Screen Reader would be, installed on an Ubuntu Linux system, without any customization. Certainly we know that traditionally the Linux world has belonged to "nerdy" users who thrive on modifying and customizing their applications, and we realize that one of Linux’s main strengths is its flexibility and giving the user the power to alter the configuration and even the application source code. But it seems that Linux is slowly moving into the mainstream and we wanted to see if Screen Reader development was following along and if Ubuntu Linux with Orca was becoming a viable solution for our less tech savvy users, especially those users who could not afford expensive hardware. We considered trying Orca out as part of the Vinux distribution, but decided against it for two reasons. Firstly it went against our philosophy of testing with mainstream software that is not specifically customized for accessibility, and secondly the latest version of Vinux at the time of our tests was not user-friendly. The downloadable file format of Vinux 2.2 was very obscure and could not be unzipped using any common file compression software. No average user would have the technical knowledge, time and patience to go through the process of downloading, decompressing and installing Vinux. Since we conducted this study Vinux installation has been made more user friendly, and we would certainly consider testing it as well as the Sue Screen Reader for Linux next time around. Vinux or Sue users who are interested in this study are encouraged to contact us.
It should be noted here that we are in favor of any Screen Reading or A.T. solution that gives people new possibilities and more choices. We have been Hal and Jaws users (and trainers) for years, but have only recently started exploring other options as they have become available.
In order to minimize any possible prejudice we spent proportionately much more time on the Screen Readers we were less familiar with, and enlisted a couple of power users of VoiceOver and Orca to help us out with the tasks we defined, both by providing tips and tricks for the use of the Screen Readers themselves and also by recommending the most accessible applications. We realize that this is a work in progress, so any comments and corrections are most welcome (see contact information at the end of this report).
We split our tests into seven major categories, based on the tasks our users most frequently want to be able to perform. A full list of all the tasks can be found in the appendix. The test setup within each category is simple. We define a specific task (can user perform task x) and assign it a score based on its complexity or usefulness. Then we try to perform this task with the most accessible software profile (Screen Reader, operating system and, if applicable, mainstream software application). Then we rate the Screen Reader based on that performance in half-point increments with the minimum score of 0 and maximum score of the pre-assigned score for the task. For instance, if we assign two points for Screen Reader enabling user to detect a spelling mistake in a Word processor and using a dictionary to suggest a replacement, we would give one point if the misspelled word is announced, half a point if the dictionary is accessible and half a point if we can replace the misspelled word with the appropriate dictionary suggestion. If misspelled word is not announced, but the dictionary is accessible, we may give half a point, though we feel in this scenario the accessibility of the dictionary is fairly useless, since user has to hunt down misspelled words himself. We deducted points if we felt the actions needed to achieve the task were too complex or if there were other accessibility factors that interrupted the user. We will mention this fact where appropriate, but the accessibility score of the software profile was based on the most accessible application used.
For each category we start with a score sheet with the name of the Screen Reader, number of points scored and total number of available points. If Jaws, for instance scored 20 for access to program X but the total available points is 30 we write it out as Jaws 20/30. At the end of these seven subcategories we will summarize the outcome and provide additional commentary. We will wrap the report up with general thoughts on Screen Reader accessibility and how this project could be expanded and improved.
What we tested:
· Can a different TTS engine be installed by user?
· Can the Screen Reader be configured to start up with the O.S.?
· Can the Screen Reader be started from the keyboard?
· Can reading preferences and accuracy be customized? (Document reading, by word, spelling etc.)
· Can the user navigate the screen without moving the system cursor?
· What is the level of Screen Reader braille support?
Hal: 15/15
Jaws 14.5/15
VoiceOver 12/15
NVDA 8/15
Orca 8/15
It should not come as a surprise to anyone to see Hal and Jaws score high on this test, given their history and market dominance. Hal scored perfectly but we deducted half a point for Jaws because it does not install with a configured startup hotkey (For example Control, Alt and N will start up NVDA). We realize this is a very minor flaw, but for a user who starts up Windows without Jaws but then needs to invoke it, this is a major headache and we feel this is an important and easy way for a new user to activate his Screen Reader. NVDA only scores 8 points, mostly because it has very limited screen navigation functionality and braille support. We did not manage to get BRLTTY for Windows 7 to work and without it NVDA only supports a handful of braille displays (Focus, Alva and Handytech displays). The support for these displays is even somewhat limited. For instance, the mouse routing functionality of the displays is not supported. This stems, at least in part, from the fact that NVDA does not use display driver hooking, which has some benefits such as ease of installation, but it also is a serious drawback for a power braille user and those who need to find customized ways to interact with applications where direct keyboard interaction is not possible.
We were excited to get to test out the VoiceOver software on the Apple O.S. and impressed with how easy it was to fire it up. However we could only give VoiceOver 12 out of 15 points because we did not find a way to use the mouse routing capabilities of the braille displays, and we were unable to navigate the computer screen without moving the system cursor. We almost had to give Apple points for the TTS engine selection and nearly suffered health damage from laughing at the more innovative synths such as „Bad News“ and others. Of course the selection of TTS voices for the Apple O.S. is different from the Windows variety, as SAPI 5 voices do not run on Apple, but we found the Apple supplied English voices very clear and concise though Alex tends to get a little muddled at higher speeds.
Orca scored 8 points. We did not get braille support to work out of the box or with a few simple experiments, as previously described we value ease of installation and configuration highly for these tests. We used the free and open source eSpeak TTS engine with Orca and we like it. We assume that the selection of more human sounding voices for the Linux operating system is limited, which could be an inconvenience for inexperienced users.
What we tested:
· Can you navigate to all files and folders?
· Can software be added and removed?
· Does the Screen Reader read all system and context menus in the O.S.?
· Can the Screen Reader read system information such as time/date and battery status?
· Can the Screen Reader read error messages and tool tips?
· Can the Screen Reader be used in terminal mode (Linux shell for VoiceOver/Orca, DOS for Windows)?
Hal 13/13
Jaws 13/13
NVDA 12/13
VoiceOver 10.5/13
Orca 10.5/13
Hal and Jaws get full points here, but NVDA loses one point because it doesn´t work very well in the terminal window. NVDA reads the command typed but does not read the results that are displayed on the screen. With Hal and Jaws the user can use the arrow up key to navigate through the list of information the command produced. With NVDA the arrow up key only reads the command which was typed, not the result. We gave VoiceOver 10.5/13 points because selecting a certain drive or folder triggered a pop up window that we had to interact with and close, and once we were out of it we had to start all over again. We realize that once user's sign up for the recommended Apple services, create the necessary accounts and download the necessary software these pop up´s will go away, but we found it extremely distracting. We also deducted half a point because we could not read the date and time with VoiceOver without activating that functionality specifically. Orca lost 2 1/2 points because it did not read tool tips, did not function well in a terminal window and did not copy text from dialog boxes. Overall, as one should expect, the Screen Readers did a very adequate job of displaying all basic O.S. information to the user.
We decided that the type of applications used for basic editing, calculations and email communication logically belonged together in an "Office" category. Therefore corresponding to the products offered by the Microsoft Office suite, and supplemented by other offerings such as Apple's iWork and Open Office for all platforms. Since the access to Open Office on the Windows platform is not particularly good we had to settle for Microsoft Office. On the Apple system we, unfortunately, did not have access to Microsoft Office 2008 so we did our experiments with the iWork 09 suite of products, but stuck with Open Office (version 3.2.1) on the Linux platform. We used Word, Excel, Outlook and PowerPoint in the Office environment and corresponding applications for Open Office and iWork for other platforms. We tested Mozilla Thunderbird, both on Windows with the Windows Screen Readers, as well as with Orca on a Linux machine. For Apple we tested Apple Mail and the rest of the iWork family of products. Further explanations and comparison notes will be given in relevant sections. From our discussions with Apple users we gather that the accessibility of Office for Mac is not significantly different from the accessibility to their iWork suite of applications.
· Jaws 28/30
· Hal 27,5/30
· NVDA 21/30
· VoiceOver 14,5/30
· Orca 14/30
We used:
· MS Word 2007 for Windows (JAWS, HAL and NVDA)
· iWork Pages for Apple (VoiceOver)
· Open Office Writer for Linux (Orca)
· Hal 8/8
· Jaws 7/8
· NVDA 6/8
· VoiceOver 5/8
· Orca 5/8
Things we looked for:
· Does the Screen Reader indicate link´s?
· Is it possible to navigate accurately and consistently through tables?
· Can the Screen Reader be configured to announce formatting changes in text (underline/italics/bold)?
· Can the Screen Reader be configured to announce misspelled words and can it read dictionary suggestions and replacements?
Surprisingly Jaws lost one point here because it is possible, but extremely complicated, to configure it to indicate formatting in Word 2007. We did informal tests on Hal and Jaws in 2009 and we are very pleased with Hal's progress in table reading. Last year it was unreliable but in the latest version we tested in 2010 it announced coordinates and text perfectly. NVDA does not announce links in a word document and its table navigation is unreliable. When navigating a table, initially table coordinates are read correctly. However this is not the case when you start moving around the table. Then NVDA starts reading coordinates incorrectly. Here we should note that NVDA read tables correctly and consistently in Open Office Writer for Windows, but did not read the dictionary there. We could not navigate tables with VoiceOver on iWork Pages, and it didn´t indicate if a text was a link. Orca does not announce links in OO Writer documents and it is not possible to use the built in dictionary, therefore it loses 2 1/2 points.
We used:
· MS Excel 2007 for Windows (JAWS, HAL and NVDA)
· iWork Numbers for Apple O.S. 10.6 (VoiceOver)
· Open Office Calc for Ubuntu Linux (Orca)
· Jaws 9/10
· Hal 7.5/10
· NVDA 4/10
· Orca 1.5/10
· VoiceOver 1/10
Things we looked for:
· Does the Screen Reader read coordinates and content of a cell?
· Does the Screen Reader indicate if a cell contains formula or a comment, if so, can they be read?
· Can the Screen Reader be set to monitor cells and announce when their content is updated?
· Can the Screen Reader detect the first non-empty cell in a row or column? (these cells often contain column or row titles)
· Can the Screen Reader detect the presence of objects in a spread sheet? (pictures, clickable buttons and other graphics, often custom scripts inside a work sheet are evoked by clicking a button within the spread sheet)
As the scores table above shows, there are two choices for Screen Reader users who need to work with spread sheet applications on a regular basis, and those are Jaws and Hal. Hal has improved significantly since our tests in 2009; NVDA can read the very basic information in a spreadsheet but falls short in anything more complex or demanding such as reading the cells comment or a formula. It is understandable that expensive and high end Screen Readers with a long development history do better than other applications. Spread sheets are not "popular" in the sense of a high percentage of blind users working with them, and they present a long list of challenges on how to best and most efficiently communicate information to the user in a manner that he or she can utilize. But it is important to keep in mind how vital it is for a blind person in any type of analyst role to be able to do calculations, generate lists and graphs.
We used:
· Mozilla Thunderbird or Outlook 2007 on Windows 7 (HAL, JAWS, NVDA)
· Apple Mail on Apple 10.6 (VoiceOver)
· Mozilla Thunderbird on Ubuntu Linux (Orca)
Hal 10/10
Jaws 10/10
NVDA 10/10
VoiceOver 6.5/10
Orca 6/10
What we looked for:
· Can the user browse through all items in a given mail folder?
· Does the Screen Reader announce sender, subject, importance (if applicable) and flags (replied, read/unread, and whether mail has attachments)?
· When a mail item is opened, can all fields be navigated to(sender, subject, date etc.)?
· Can a mail be composed, spell checked, attachments be inserted and sent?
· Does the Screen Reader work with the mail clients address book (read choices, auto complete etc.)?
· Can the mail application's calendar be used to create a meeting invite or appointment?
Hal, Jaws and NVDA announced all this information perfectly in Outlook 2007, and very well in Thunderbird (though Jaws did not indicate the presence of attachments in Thunderbird). Also only Importance that is created in Thunderbird messages is detected in the Thunderbird client (Importance cannot be specified in Outlook and picked up in the Thunderbird client), but that is not a Screen Reader limitation. VoiceOver presents challenges. One of the drawbacks is that VO marks messages as "read" as soon as the cursor goes over it in the Inbox. Reading the various flags for the mail items is not automated and has to be done with a lot of key strokes. Unfortunately Orca did not perform as well as we had hoped, when browsing through the mail folder Orca did not read message information and add to that there were several other issues we encountered.
We used:
· MS PowerPoint 2007 on Windows 7 (JAWS, Hal and NVDA)
· iWork KeyNote 09 on Apple O.S. 10.6 (VoiceOver)
· Open Office Impress on Ubuntu Linux (Orca)
· Hal 2/2
· Jaws 2/2
· Orca 2/2
· NVDA 1.5/2
· VoiceOver 1.5/2
What we looked for:
· Can one browse through all slides in a presentation?
· Is it possible to read the text on a slide (without exporting it to a different application)?
We did not find VoiceOver very intuitive and it sometimes did not read the first sentence on the slid. Orca does very well with OO Impress on Ubuntu and scores as well as NVDA, HAL and JAWS.
We used:
· Internet Explorer 8 on Windows 7 (JAWS, Hal and NVDA)
· Safari on Apple O.S. 10.6 (VoiceOver)
· Firefox 3.5 on Ubuntu Linux (NVDA and Orca)
Jaws 17/17
Hal 16/17
NVDA 15.5/17
VoiceOver 12.5/17
Orca 9/17
What we looked for:
· Can one navigate by headings, buttons; edit fields, frames, landmarks and check boxes using designated keyboard keys or key combinations?
· Can accessible Flash objects be navigated and interacted with?
· Can a user select from an HTML list (both a "normal" list as well as one with onSelect script attached to it that automatically selects the item with focus)?
· Is there a key to jump past group of links or objects on a page?
· Can all links on a page be put into a list box for quick navigation?
· Can a radio button be selected and its value read?
· Can a simple form on a web page be filled in and submitted?
The Internet's importance is growing, while its complexity is growing as well. Therefore we find this a very important field and realize we must greatly expand this functionality test in future. For this test we have to assume that the designers of web pages do their homework and make use of HTML elements that can be utilized by Screen Reader users to quickly and accurately navigate through a web page. We assigned double points to some of the 13 things we looked at so the total available number of points was 17. In future special attention needs to be given to Flash and AJAX objects, something we did not test specifically this time around. Here we must keep in mind that browsers prior to IE 8 and Firefox 3.5 do not support landmarks and so it is important to look at the combination of the web browser and the Screen Reader when web browsing strengths and weaknesses of Screen Readers is evaluated. We also tested Screen Reader behavior in "Jump menus", these are menus that select the first object to get focus, so the first item in the list will always be chosen, unless the Screen Reader allows the user to interact with a copy of the list in a buffer and then passes the list with the selected item through to the browser application and the JavaScript function. One such menu test can be found here:
http://webaim.org/techniques/javascript/eventhandlers#onchange
Jaws scored 17 out of 17 possible points. Hal scored 16 (did not recognize landmarks); NVDA did not recognize frames and had difficulties with Flash objects so we gave it 14.5 points. VoiceOver with Safari does not recognize landmarks and does not allow the user to interact with Flash objects and we did not manage to jump between check boxes on a page (this can be an invaluable way to navigate a page quickly, for instance the HTML only interface version of www.gmail.com). We had a hard time browsing with VoiceOver and Safari since we could never use one keyboard key or shortcut to jump between HTML elements on a page, but always had to go through the accessibility router to find the element we wanted to navigate to. We suspect there may be undocumented shortcut ways to do this or users get more used to it, so we decided not to subtract more points from VoiceOver this time around. Finally, Orca only scored 9 points. Flash did not work, frame navigation did not work, one cannot get a list of all links into a list box, and jumping between check boxes or buttons did not work either. Overall we definitely found that the applications do pretty well with allowing users to browse the Internet. We would recommend any of the Windows Screen Readers but reserve a bit of judgment on the Apple approach to web browsing, noting that there may be revolutionary interface innovations in the near future if some of the touchpad features found in iOS are ported to their desktops or laptops. We found navigating the web with Firefox and Orca slow and cumbersome and not up to our expectations, though we certainly know there is a lot of work going into improved user experience with Orca.
We used:
· Windows Live Messenger on Windows 7 (JAWS, HAL and NVDA)
· Adium on Apple O.S. 10.6 (VoiceOver)
· Pigeon on Ubuntu Linux (Orca)
Orca 6/6
Hal 6/6
Jaws 6/6
NVDA 4/6
VoiceOver 3.5/6
What we looked for:
· Does the Screen Reader alert the user of an incoming IM request eventhough he is not inside the chat application?
· Can the message history be reviewed and copied within the IM application?
· Is it possible to send and receive attachments with the IM application?
After trying applications ourselves as well as listening to recommendations we tried Windows Live Messenger and Miranda for Windows, Pigeon for Linux and Adium for the OS X. In our test cases Hal, Jaws and Orca all scored perfectly. We did experiment with Miranda in Windows but did not get the Screen Readers to announce when users sign in and out of the user list, nor to alert the user of an incoming message, unless the user was inside the active chat window. Unless the user can be alerted about incoming messages whilst in a different window, we find it hard to give many points to said software. VoiceOver with Adium alerts user of an incoming message, but we did not get it to announce from whom it is. Also we tried to browse through conversation history with VoiceOver and Adium and it is a slow process, not to mention that a new incoming message puts the focus back to the top of the conversation window, requiring the user to start all over again. We find the same issues with incoming attachments and movements in the contact list (any incoming file or changes in the contact list move the VoiceOver focus to the top of the Adium chat window). We found it practically impossible to maintain an online conversation using Adium with VoiceOver, but we are aware of the fact that other users have managed to do so more successfully and, perhaps, the problem lies with some simple settings or other modifications, but if so the VoiceOver "Help" should document these.
We used:
· Skype 4.2 on Windows 7 (JAWS, Hal and NVDA)
· Skype for OS X on Apple (VoiceOver)
· Skype plug in for Pigeon on Ubuntu Linux (Orca)
Hal 6/6
Orca 6/6
Jaws 5/6
NVDA 4/6
VoiceOver 3/6
What we looked for:
· Is it possible to review the contact list, along with each contact's current status?
· Is it possible to read the incoming caller's name before answering?
· Is it possible to use the Skype chat window?
· Is it possible to search for, and add, a new contact?
We feel that Skype´s popularity warrants its own category for these tests. It is noticeable that Orca performs very well, as well as Hal, getting all 6 available points. Jaws loses a point because we did not find a way to have Jaws announce the name of the caller when a call comes in. NVDA does not manage to access the Skype chat window and does not read the name of the incoming caller. VoiceOver does not read the name of the incoming caller, does not read the chat window and cannot announce the online status of contacts in the Skype contact list, and therefore only scores 3 out of a possible 6 points.
We used:
· Adobe Reader 9
Jaws 5/5
NVDA 5/5
Hal 2/5
VoiceOver 2/5
Orca 0/5
What we looked for:
· Can the text in accessible (tagged) PDF documents be read?
· Can the Screen Reader detect links in a PDF document?
· Can accessible PDF forms be filled in (can the Screen Reader interact with edit fields and are these announced correctly if correctly labeled)?
We used a couple of PDF documents (see Appendix) for testing purposes. We tried to read the text in a document and tested filling in an accessible form, whether keyboard navigation (with the TAB key) worked and if labels were correctly announced.
Jaws and NVDA scored perfect on the test. Hal only scored 2 out of 5 points as links and other elements were not recognized and it was nearly impossible to fill in a PDF form. VoiceOver had the same level of functionality, could read the text in a PDF document but could not interact with it or recognize specially marked up parts of it (links, headings etc.). We did not get Orca to do anything with a PDF document. We find it vital for any user that is active in information sharing, be it on the Internet or through their job in an office environment, to be able to read and interact with PDF documents, due to their popularity. They are, essentially, a dominating file format in the current world of information sharing, and not being able to work with them can be extremely detrimental to a blind user.
The following table will show the score for each Screen Reader, both absolute points scored and also the grade as a percentage (out of a 100).
Screen Reader |
Points |
Percentage |
JAWS |
85.5 |
93% |
HAL |
85.5 |
93% |
NVDA |
69 |
75% |
VoiceOver |
58 |
63% |
Orca |
54 |
59% |
Hal and Jaws both score 93%. NVDA loses a lot of points for its limited braille support and lack of ability to explore the screen without moving the cursor (in other words, lack of virtual cursor). This has its advantages as NVDA does not hook up to the Windows display drivers, which allows it to be run from a USB key and without admin privileges. It still ends up with a score of 75%.
VoiceOver has a 63% passing grade, mostly failing in office type applications and for a rather confusing and, to us, unintuitive web navigation features. Finally, Orca has a score of 59%, lack of intuitive user help in places and problems working with office applications are the largest drawbacks, but it performs extremely well with the Pigeon messenger to enable instant messaging and Skype. These grades are only meant to give an idea of the user experience, not to be an ultimate measuring stick for what is the best application. In some cases the lack of accessibility features has to do with the lack of accessible third party applications. For instance at the time of our tests, there was no accessible PDF reader on Linux. It is more useful to think of these categories independently and decide what is important to your user. If the user requires Braille support, Hal and Jaws are still the best choices, NVDA and Voiceover have their strengths, and we should be able to get BRLTTY to work, thereby providing Braille support for Orca. If users work with a lot of Office applications, they should go with one of the established Windows Screen Readers. If the user is not interested in Braille and primarily uses Internet, email and instant messaging, NVDA is a viable solution. All the Screen Readers we tested do a decent job of allowing users to explore their computers, open files and interact with them, add and remove programs and do other basic tasks. Other factors matter, such as the ability to create special scripts or map files for custom applications users need to run. Dolphin Computer Access has been extremely accommodating in creating custom map files for our users, at no cost, which has enabled them to work with otherwise inaccessible applications. Freedom Scientific offers extensive scripting support for Jaws, but does not offer any customization services, which means either one has to learn to work with the scripts or hire someone who is capable of writing custom scripts for troublesome applications. We also cannot ignore the fact that users have their preferences. The scores are roughly in line with our expectations. It is unreasonable to assume that free and Open Source Screen Reading can catch up to more established and expensive Screen Reader technology in such a short time.
Finally, we know there is a lot of Orca development and it has always been more of a system for pros and for users who like to customize and script computers. A user may get considerable mileage, if given the time to customize and script and get information on complex settings that need to be changed for maximum accessibility. But this is not what we were after for this testing, since we want easy set up and functionality for the average user. We do not find that Orca has become a very strong mainstream alternative, though it performed fantastically with some applications, such as Pigeon. We will keep an eye on Orca and Gnome development in the future. We find that both Orca and VoiceOver would benefit significantly from improved help documentation. We were particularly impressed with JAWS´s help documentation.
We were a bit disappointed with the VoiceOver experience. Probably our expectations were a bit high, but with all the hype out there, we expected some game changing level of accessibility in VoiceOver on the Apple desktop. It is much superior to the built in accessibility in Windows, but we feel it is not sufficiently good to be the only Screen Reader a student, or professional, needs. We are pleasantly surprised with the quality, stability and reliability of NVDA. For users who do not require a braille display, or work extensively with Office applications, NVDA may actually be the only thing they'll need. We look forward to retesting with NVDA in 2011. Dolphin definitely did very well and we were pleasantly surprised to see significant improvements in a year from a major Screen Reading vendor, though their lack of PDF file support was a bit perplexing. Jaws was more of a disappointment for us. We feel recent and costly upgrades to the Screen Reader have brought very little new functionality that is important to our everyday user.
This research project actually started out as a one-day set of tests in 2009, involving Window Eyes, Hal, Jaws, NVDA and System Access. The project was not organized enough to warrant any type of international reporting, but we felt it could be the seed for a much larger and better organized project. Similarly, we feel that we can take this to another level. Once we have fully established test categories and framework for evaluation, we hope that other users may be able to step up and get involved by carrying out tests of other Screen Readers we have not managed to cover as of yet or provide corrections or input to our data. We also hope to create a web site around this project and put our report and comparison up there for all users who are interested. It could be a genuine Wikipedia style resource for blind Screen Reader and Screen Magnifier users. We believe that similar comparison studies could be implemented for Braille PDA´s, Screen Magnifiers and Screen Reading solutions for mobile phones, to name a few areas where choices are many and even more confusing than the Screen Reader market.
Finally we hope to be able to expand the project, add categories as well as updating our task list. At the very least we'd like to make this an annual comparison. We see great value in that for ourselves, and we hope that we could conduct studies in such a way that our colleagues at similar organizations may benefit from this work. We hope that manufacturers take note and look to these tests as benchmarks and take them into consideration when updating their software. We have already communicated some of our findings to Screen Reader vendors and we have seen significant improvements, for instance, the PDF handling of Hal 12 (which now is called Supernova) is on par with that of Jaws and NVDA. We communicated the PDF issues to Dolphin during and after our testing. Whether that had any actual effect on their decisions or development cycle we can’t say, but until proven wrong, we’d like to hope it helped bring the issue into focus. We have communicated concerns to other Screen Reader developers as well and will continue to do so when we find issues that we believe are important to our users.
Finally, we would like to extend our sincere gratitude to those individuals who provided valuable input to our research. First off we´d like to thank Dave Hunt for his assistance with the Orca Screen Reader. Second, Malthe Jensen provided valuable feedback concerning VoiceOver. Thirdly we like to thank the Apple dealership in Iceland for providing hardware for the tests. And last but not least, a special thanks to our manager, Huld Magnusdottir, who shared our vision and provided us the opportunity to conduct these tests. Any inaccuracies in this report are our responsibility and we encourage suggestions for improvement.
Birkir R. Gunnarsson birkir@midstod.is or birkir.gunnarsson@gmail.com
Hlynur M. Hreinsson hlynur@midstod.is or hm.hreinsson@gmail.com