Card sorting method

Card sorting is a widely used method for uncovering how your users think and adapting your information architecture accordingly.

The method might be a solution for you if you want to organise content, label a group of similar items and/or determine how understandable and relevant your content is.

While card sorting doesn't provide you with the final structure of your website, the results will provide many insights that you can use throughout the information architecture design phase.

There are three types of card sorting: open, closed and hybrid. All three can be run externally (remotely) or internally (face to face with users).

• open card sorting: users sort cards into categories of their choosing, and come up with a label for each category
• closed card sorting: users sort cards into categories that have been pre-defined by you
• hybrid card sorting: users sort cards into categories pre-defined by you, and can create their own categories as well

A card sorting exercise can either be moderated or unmoderated.

Moderated: a researcher is present (either in person or remotely) during the card sorting session to take note of users' comments and behaviour. If a user questions where to put certain cards, or change where a card goes, the researcher is on hand to discuss with them why a certain card might belong in more than one category or why a card should change categories.

Unmoderated: this type of card sorting session is usually done with the help of an online tool. Sessions can take place asynchronously or simultaneously. The online tool allows researchers to provide prompts when necessary and appropriate to guide participants through the activity. An online tool can provide an excellent analysis of the final results, correlating card groupings, and statistical patterns.

• if you don’t have an existing structure yet, this is a good time to start quickly thinking about how you are going to structure your content in a way that accommodates the way most of your future users think
• if you have an existing structure, card sorting may be a way to challenge your original architecture. Perhaps you see a need to test your users’ needs again? Or you noticed that one section is leading to poorer results than others? Or maybe you have some new content but find it hard to know where to place it in the existing structure? All those questions suggest that you might need to perform a quick card sorting exercise

If you already have an existing structure, decide which content you need to assess. Write each piece of content on individual cards. Ask representative users from your target group to sort those cards into groups that make sense to them. This is an opportunity for you to spot recurring groups, issues, disparities and general comments about your content.

How to prepare a card sorting exercise

1. Define your audience

             a. Facts-based

                    ■     retrieve target audience profiles from online surveys

     2. Number of participants

• 25 participants per user category/persona

     3. Selecting the content

Card sorting is more relevant and powerful where combined with existing task analysis.

            a. Prioritise the content you want to investigate.

                    ■     Facts-based

• top tasks survey
• task mapping results
• analytics (including visited pages in combination with bounced results)
• social media
• trends and keywords
• input from real users (survey + post-test questionnaire)
• content inventory

             b. Choose the top tasks, pages and/or content.

                    ■     tasks should be representative of the site as a whole or the site's product areas

            c. Try to select only those items at the same level in the hierarchy.

            d. How many cards?

                    ■     between 30 and 50 cards (always fewer than 100 otherwise participants may lose interest)

                    ■     more than 50 cards might be too complex for remote testing

                    ■     where you’re combining both approaches, be sure to apply the same rules

   4. Making the cards

           a) Cards can contain:

                     ■     a simple title

                     ■     a generic description

                     ■     an example of real content

                     ■     a functionality description

                     ■     an image, drawing, photograph, symbol, …

         b) Labelling the cards: content vs. tasks

                    ■     if you are using text as a label, try to phrase it into a type of task

• e.g. turn "Funding" (= content)  into "Check your eligibility for grant ‘X’”. This is particularly helpful for reverse sort (see Treejack Testing).
• tasks should be realistic and credible

                   ■    if possible, try to avoid repeating terms – it might encourage users to group all cards where the term appears e.g., do not write:

"Find out how to check your eligibility"
"Find out how to check available grants", etc.

"Find information about DG X"
"Find information about Commissioner X ", etc.

"Find financial support for SMEs"
"Know the legal procedure to establish an SME in Europe", etc.

 5. Deciding the type of card sorting you will run

1. 1. It all depends on the current state of the project.

6. Combine moderated vs. unmoderated sessions

1. Local tests: 5 users

■  this allows you to have a better understanding of your users’ ways of thinking and specific pain points

■     you can fit a 5 users test into a single day

2. Remote tests: 20 users.

■    this helps you to test a larger number of users at once

7. How many sessions?

a. Several sessions only if

■     the number of elements to be sorted is too high

■     you want to sort several levels within the tree

8. Duration of the session?

a) The amount of time required to perform a sort can vary considerably from person to person, but is largely dependent on the number of items to be sorted:

■     in local testing: count 1 hour for a 50-cards session

■     in remote testing: the session shouldn't last more than 30 minutes

• ideally, combine the remote card sorting with the local sessions
•  in a local test
  • explain the session
  • ask the participants to fill in a pre-test questionnaire
  • ask the participants to think aloud while performing the exercise
  •  …
  •   set a maximum number of categories
    • at first, leave the user to create as many groups as they would like
      • in a local test, this allows you to understand their way of thinking
•  if you consider there are too many categories, ask the user to reduce the number of categories to a maximum of 10
  • this number still gives enough freedom to the user while allowing you to fit all the categories into your navigation bar (if any)

The following tools can help you run a remote card sorting session and analyse the results:

• Optimal Sort
• uzCardSort
• UsabilityTest
• SimpleCardSort
• UXSort

Please note that these tools are commercial software. If you use them, you will need to obtain assurance from the provider that they are GDPR-compliant and consult your DPC.

Quantitative analysis

1. Propagation rate

○     look at the percentage of times a specific card has been put in the same category

■     ≥ 75%: the card is strongly associated with that group

■     ≥25%, < 75%: the card is weakly associated with that group. It’s worth investigating further why

■     < 25%: the card is very scattered across groups

• • • this might indicate that it should not have a physical place in the overall architecture. In some cases perhaps it should be persistent, (e.g., search bar)

2. Co-occurrence matrix

○     this allows you to identify how frequently users associate themes. If two cards regularly appear together, chances are that they cover a similar theme or function

■     ≥ 75%: valid association

■     < 75%: not valid association

3. Dendograms

○     a dendogram is a tree structure visualisation automatically generated from the output of the co-occurrence matrix

○     the aim is to minimise the number of individual cards that don’t fit  well together. While there are some objective techniques to fine-tune the different categories, you will always need to use your best judgment too

Qualitative analysis (between objective and opinion-based)

1. Category overlaps

○     gather all categories and try to find some common themes emerging. Groups that appear only rarely are removed during the iterative selections

○     in an open sort: this can help to build standardised categories

2. Category coherence - emerging patterns across cards

○     look at similar groups made by different participants and try to see if the same cards often appeared in those categories

3. Individual cards - emerging patterns across categories

○     look at each individual card in order to see all the groups to which it was assigned. Then, look for the most-used categories. This helps to identify individual cards that are particularly ambiguous

4. Users’ comments

○     during your observations and with the post-test questionnaire in hand, try to take your users’ comments into account when thinking about how you structure your content.

Any personal data that is processed as part of any task relating to the development of the European Commission's web presence must be done so in compliance with Regulation (EU) 2018/1725 of the European Parliament and of the Council of 23 October 2018. 

See data protection for more info.

• http://www.usability.gov/how-to-and-tools/methods/card-sorting.html
• http://www.nngroup.com/articles/card-sorting-how-many-users-to-test/
• https://gibbon.co/UXdesign/ux-card-sorting/10-things-to-know-about-card-sorting-2
• http://www.measuringusability.com/blog/card-sorting-ia.php
• http://boxesandarrows.com/card-sorting-a-definitive-guide/
• Barrère G., & Mazzone E., (2012). Card sorting - ne perdez plus vos utilisateurs!  Eyrolles, Paris, 2012. ISBN 978-2-212-13448-3

Need further assistance on this topic? Please contact the team in charge of Europa Domain Management (EU Login required).