This report was done on the latest usability testing from Baltimore this past week.  I don’t think that there were any real surprises, but it sure underlined how much work we have to do to make Drupal more intuitive and accessible for the average user.  These are my notes from that presentation.

“I won't release Drupal 7 until I crossed off at least 90% of the problems they identified” – Dries

The conductors of the testing included:

• bojhan
• beeradb
• add1sun
• beccascollan
• ultimateboy
• sumitk
• quicksketch
• catch

The presentation started with the following ideas:

• Just because we are finding things, doesn’t mean we’re fixing them.  This was clear because so many of the same issues that were reported upon in Boston, still existed in the testing this year.
• This was a less formal test than the last test and included both beginner and Intermediate testers and had remote sessions.

The testing was done at the University of Baltimore Usability Lab.  It included:

• Eye tracking systems
• One way mirrors

The lights had to be turned off and the facility had to be quiet while testing was going on.  The room that the had the conductors of the tests was pretty small.

The Participants included folks with 94 years of collective web experience:

• freelance developer
• systems consultant
• lead programmer
• lead UI developer
• student
• product director
• professor in web design
• computer consultant
• internet application developer
• designer and developer
• admin assistant – blogger

• Many experience levels – many were familiar with Drupal

Intermediate Test Plan – they had built at least one drupal site – They needed to:

1. Install Drupal (but they didn’t need to mess around with setting up the database or mess around with the settings.php file)
2. create content
3. add a link to the nav menu
4. create a content type
5. setup roles and permissions
6. categorize content
7. enable a module
8. add <img> to filter html
9. enable search
10. set up a trigger for new content notifications

Beginner Test Plan – Never used Drupal but know some about the Web

1. Install Drupal
2. post sample content
3. add a link to the nav menu
4. add a block to the right sidebar
5. categorize content
6. change the colour of the site

The Approach

• test and observe
• analyze and put the observations on stick notes
• organize and prioritize the notes
• record and share

Issues were written on post it notes and was put up on the wall.  The test conductors began to see patterns and themes emerging.  They then started categorizing the kinds of problems.  They purposefully wanted to keep the issues found out of the issue queue.

Several Drupal 7.0 patches were tested:

1. added vertical tabs
2. text format widget patch
3. Remove post settings admin screen
4. Finer control of the parent menu select box
5. Password Checker

Findings – Things that worked

1. Drupal Installation – everybody could do it.  Keep in mind that they users didn’t need to set up the database or futz with the settings.php file
2. Password Checker – worked well
3. Input formats renamed to text formats

Still, the stick notes of good things comprised a total of five notes.  The stick notes of bad observations covered more than one wall

Issues Included

1. How do I create content?  People didn’t see the create content link.  One guy even said “I would call the link create content”.  He still didn’t see the link.  It was painful to watch.
2. “WTF am I looking at?  WTF do users see?”  The welcome page doesn’t tell you where you are.  Folks not sure what they are looking at.  The first experience of new users is confusion.
3. Navigation and admin and menus as blocks – the split between the two is strange.  There is a mixture of admin and user information in the menu.  The user name appears as a title, and the users were confused.  “What would you do if you were at home at this point?”  “I would get something to eat.”  “It lied to me, it says I control my site’s menu.”
4. Node Orphanage
   1. Nodes are orphaned by default unless they are promoted to the front page.
   2. Menus and taxonomy are optional and misunderstood
   3. Even if they find the content in admin/content/node it “isn’t part of the site”.

The group found NEW ISSUES

1. Search
   1. when you enable the search module the box appears magically in the theme
   2. users enabled the search block before they saw the one already created
   3. users click “re-index” to re-index their site.  Even when they know about cron
2. Menus
   1. users aren’t clear who can see the nav menu
   2. the relationship between menus and blocks is not made clear
   3. menu item creation is ordered path first, then title, unlike most other forms in Drupal where title or name is first
   4. parent item selection - “create a main menu item” is a valid menu parent.
3. Text Formats
   1. task was to add the <img> tag to an input filter.
   2. folks had trouble.  Click on the configure  and it brings you to an edit page.  Hit configure and went to an edit page.
4. Taxonomy – Quotes included:  “Too many levels of abstraction” and “Taxonomy, that’s a funny word”.  “I think of biology and deer”.
1. there was a wall of text –“That’s more than I want to read right now.”
2. expected categories or tags but no config vocabs
3. the example tags vocab with a corresponding example of a structured vocab

This test found 128 specific issues.  At the rate we’re fixing things it will be something like 2013 before 90% of the bugs are squashed.

www.drupalusability.org has all the tests, videos, and findings.