My Time in the People and Robots Lab

Posted on by Curt 7 min read

I started my Master’s in Computer Science at UW-Madison in the fall of 2018 and graduated in the spring of 2021. I spent most of it in the People and Robots Lab under Dr. Bilge Mutlu, studying how people and collaborative robots (cobots) actually work together on the factory floor. Most of my work was building the robot user interfaces and experiment systems that let us ask those questions, and then running the studies around them. The projects from those years each have their own post. This one is the map, and a bit of reflection.

A lot of the lab’s work traced back to one piece of fieldwork, an ethnography of cobots in industry.1 The lab interviewed the managers, engineers, operators, and instructors who live with these systems. My advisor had been working in this space for years and steered me toward it. An epistemic network analysis of those interviews surfaced the concepts that mattered to practitioners,2 and those concepts seeded most of what I worked on. Authr, CoFrame, and the interface aids in the attention study all cite them.

Nomenclature: Epistemic Network Analysis (ENA) is a technique from quantitative ethnography that models the connections between ideas in a body of discourse as a network, so you can see which concepts people bring up together and compare those networks across groups. It comes out of David Williamson Shaffer’s group at UW-Madison. The tooling can be found at epistemicnetwork.org.

The robots

I worked with three arms, each taught me something.

The Kinova Mico came first. That is where I learned MoveIt and basic ROS. It was slow and janky, but I learned how to control an arm.

The UR5 is where I got familiar with the UR stack and the Robotiq gripper, went further into ROS, and played with the RelaxedIK solver. The block-finding work started there.

The UR3e became my robot. The studies ran on it, and between studies it was my playground for real-time motion control. It was also the physical side of CoFrame, the way the UR5 had been for Authr, and it was center stage in the AR UX exploration.

There was a Franka Emika Panda on the bench too. It is in the photo at the top, next to the Kinova and the UR3e. My part of the Panda was lab maintenance. Get the hardware set up, which is what I am good at, and get the basic control software running. Once that was done it was not my robot anymore. Other grad students focusing on polishing and peg-in-hole teleop tasks worked with it.

And two Kuris. Mayfield Robotics gave the lab a pair for free, and we let them roam the floor. There were other robots in the lab, but I did not work with them.

The work

My projects were each a different angle on the same question. How do you make a human-robot team work well. Three of them have their own notes post.

The fourth we lovingly called the ONET study. When the COVID lockdown made in-person human studies hard to run (hard to pay people to willingly show up, really), we leaned on existing data instead. Working with colleagues in Human Factors and Optimization, we used the O*NET catalog of work activities as a test bed for where collaborative robots actually add value across manual work. My piece was rating cobot capability for each activity, using a decision flowchart a colleague and I built and grounding it in a sizable literature review of cobots at the current state of industry. Those ratings fed the allocation algorithm at the heart of the study.3

The side projects

Some of the work never made it into a paper but it was still pretty fun.

  • The robot needed to know where the blocks were before any study could run, so I built it a vision system. A mundane problem and an interesting engineering exercise.
  • One Friday I found a WebGL fluid simulation and turned the arm’s motion into fluid art over a weekend. It became a quality metric for planner output by accident.
  • There was an AR UX exploration on the UR3e that COVID killed. It was an exploration of Microsoft HoloLens for physically situating CoFrame.
  • Táltos-oid, the sixth finger, was a course project rather than lab work. But I built it while I was enmeshed in the lab.
  • CS 570, the applied HCI course, had me iterating on the UR’s Polyscope software, imagining what could be better.
A high-fidelity mockup of a reworked Polyscope program screen: a node list of actions, primitives, and control flow down the left, a program tree with drop targets in the middle, and a parameterization pane on the right.
The CS 570 high-fidelity mockup of a reworked Polyscope program screen.

Keeping the lab running

Beyond the formal projects, I did a fair amount to keep the lab running. I maintained a centralized robot description and configuration repository. I wrote documentation and bringup code for the UR3e and the Microsoft HoloLens. I developed and patched the Robotiq gripper ROS driver for our setups, including the serial-tunnel support the UR3e post complains about. It was also just expected that you would help colleagues with their user studies, technical development, and paper writing, which I did often (all good learning exercises).

Looking back

One thing that stuck with me from those years is agency. I put the Táltos-oid finger in a plastic box during debugging because I had decided it was trying to get away from me. The Kinova’s joints and linkages read as an agent in a way the UR arms do not. The Kuris made their way around the lab as they saw fit. Each of those robots sits somewhere on a dial from a fully predictable tool to a thing acting entirely on its own, and I have been thinking about that dial ever since. The Táltos-oid retrospective goes into it.

What I found, regarding profession, is that my interests lie in application over theory. I am not interested in being a mathematician who happens to think about compute problems. I enjoy tackling engineering problems using mathematics, science, and practical knowledge to build a real system that interacts with real users, with all of the messy bits that happen when a system is physically situated. The lab was full of those messy bits.

I also want to make a shout out to the colleagues and fellow researchers I met while in grad school. These are connections I cherish, and when the opportunity arises I try to reconnect with them, say when we are both passing through the same city. Grad school trauma bonding is a thing.

Thanks for reading. Stay tuned and keep learning.

Footnotes

  1. A. Sauppé and B. Mutlu, “The Social Impact of a Robot Co-Worker in Industrial Settings,” Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ‘15), pp. 3613-3622, 2015. doi:10.1145/2702123.2702181 ↩

  2. J. E. Michaelis, A. Siebert-Evenstone, D. W. Shaffer, and B. Mutlu, “Collaborative or Simply Uncaged? Understanding Human-Cobot Interactions in Automation,” Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ‘20), pp. 1-12, 2020. doi:10.1145/3313831.3376547 ↩

  3. L. Liu, A. J. Schoen, C. Henrichs, J. Li, B. Mutlu, Y. Zhang, and R. G. Radwin, “Human Robot Collaboration for Enhancing Work Activities,” Human Factors, vol. 66, no. 1, pp. 158-179, 2024. doi:10.1177/00187208221077722 ↩

Posted in Grad School Tagged Human-Robot Interaction, Cobots, UR3e, UR5, Kinova