There are three components to what we do with the CUTGroup: UX testing (“user experience testing,” also called “usability testing,” a technique used in user-centered interaction design to evaluate a product by testing it on users); digital skills (which we define broadly as the human ability to get things done on computers); and community engagement (which, in our context, is defined as a process of building relationships for the purpose of collectively making lives better through technology).
One of the most important aspects of our work is that it does each of these three separate things pretty well, but none of them really well. The only thing we do really well is the CUTGroup itself.
We struggle at Smart Chicago with how prescriptive we should be with our program. Clearly, any kind of user testing is helpful to the technology developers. The teaching and learning of digital skills is a worthwhile act, regardless of context. And any time civic hackers can get with community members—in any setting, for any purpose—that’s a good thing.
As you’ll see in the Methods chapter and in the examples of tests we’ve conducted, different CUTGroup components take on higher or lower levels of importance, depending on the particular nuances of the need for any given project.
Sometimes, the app isn’t made yet, and we’re testing the relative value of a concept. In others, we’re testing a mature website that has some user interface issues, and just want to get bug reports. Since we make a lot of technology, and we are engaged with audiences and experts all the time, sometimes we just want to have a structured meeting that helps us think fresh and re-engage with some of the people who matter most.
But we’ve settled on the belief that the melding of these three components is what makes the CUTGroup the CUTGroup—its essence. It would therefore be impossible to say that you’re running a CUTGroup program near you without being devoted to building these components in some semblance of equality and strength.
So let’s take a look at each component, talk about how they fit together, and lay down some markers around what we think are the minimum elements for a viable CUTGroup.
UX testing
We have been careful to design the CUTGroup as legitimate UX testing. By that we mean the developers who work with us get specific, actionable feedback from relevantly situated users. We can obtain both quantitative and qualitative info, discovering how dozens of people answer the same question and drill into a deep conversation about any particular feature. We’ve found glaring bugs, discovered unique insights, and helped people plan new product releases.
We stray from UX test design principles, however, in a very key way: by requiring the developer to participate in the test. We deprecate this and other more social-science aspects of classic UX testing (never coaching the user, for instance) because it’s not conducive to the other two components.
There is an odd dynamic here, however. At Smart Chicago, we have a broad mission to help build the civic innovation sector of the technology industry.
By this, we mean to deliberately situate the work of civic hackers firmly in the broader market for consumer-focused technology.
By cutting against the grain of UX testing methods designed to align a software product with the needs and desires of people who might buy it, we risk keeping the sector in an immature state of development. We justify this by seeing it as a temporary condition of the field. There is currently so little engagement with regular users, and very little tradition around user testing, that we’ve developed this hybrid to account for that.
The minimum UX testing element that must be included in a CUTGroup program is the delivery of concrete, actionable direction to the developer that has been generated directly from testers who have been deliberately chosen for the test.
Digital skills
This is the component for which there is probably less precedence in the work of a typical civic hacking aficionado. But it’s a really important one to our work at Smart Chicago, which was formed out of conversations about the digital divide.
A CUTGroup test typically consists of a mix of 10-20 people of widely varying digital skills, all learning and teaching each other, in the same room, out in plain public view. It becomes something of a salon. People get to know each other, laugh, eat candy, and pick up digital skills that they didn’t expect to get when they first signed up. This goes for the developers, too—they are learning how to conduct UX tests and to make their sites better.
The more professional-grade digital-skills teachers would want to formalize this learning into a structure that allows the tester to move along a continuum of learning toward an established goal. We gave up on that idea because it would cost too much money (we can’t pay people a $20 Visa gift card every time they show up for a class), and because it would defeat the informal but focused nature of a test.
So the minimum digital-skills element that must be included in a CUTGroup program is simple: a spirit of learning and a commitment that all of the testing happen in the same room, with all of the messiness that goes along with that.
Community engagement
This is the component that has surprised us the most, and the one for which we did the least amount of planning.
We always knew we wanted to conduct the tests in public computer centers and mainly in libraries. We love the vibe there and, to us, these are the most accessible and open places we could imagine. There are so many structural elements of the library—their wide geographic range, onsite wifi, community rooms, open architecture, and accessibility compliance—that our program slots in well when we center it there.
But we were surprised to see how much people like the CUTGroup. We always ask the same last two questions on every CUTGroup test form. One is quantitative (“Did you like this CUTGroup test?”) and the other is qualitative (“Anything else to add?”). Out of the hundreds of testers, one has answered “no” the the first question, and we’ve received dozens of positive comments about the experience.
People often have to be shepherded out of the meeting room as the lights go off in the library. We all have a really good time, and sometimes we’re able to talk about difficult things (like how to get to school in an era of school closings) in very flat, constructive ways. The CUTGroup is a joy. To anyone who has ever done civic engagement (which can be a slog), you understand that this is a big deal.
From the start, we’ve been completely committed to drawing together as many people as we can from every part of town and all sorts of backgrounds into a single set of people on a common mission.
We differ from the practice of community organizers, however, because we don’t drill down into specific geographic areas or subject matters, and we don’t attempt to achieve specific social or policy outcomes. However, this leads to trust among our users because we stay focused on what we said we were interested in when they signed up: We give them money, and they give us feedback on technology.
In order to make the CUTGroup representative of the community, and to serve these larger goals around development, the minimum community engagement element that must be included in a CUTGroup program is the recruitment of a large and diverse tester base.
This map shows the wide geographic (and, by extension socio-economic) range of people who came to a test of OpenStreetMap. Some people came 20 miles on a January evening when it was 10 degrees Fahrenheit with light snow.
Developer Fernando Diaz tests his food inspection site with three residents in the Kelly Library.