Listen to our podcast episode or read the transcript below featuring Dr. Jen Collinger and Dr. Rob Gaunt, biomedical engineers, as we talk about how to design a robot that you can control just with your thoughts.
You know, I was really inspired to go into this field as soon as I discovered that it even existed.
You have the opportunity to dream new things and as long as you can convince a few other people that those ideas are worth it, you can have a shot at actually trying those things out.
There’s a spot in your brain where when you touch the tip of your index finger, those neurons light up, they become active.
Tanya: Hi everyone! Welcome to Ask-a-Scientist, a Science Journal for Kids podcast where we explore what it’s like to be a scientific researcher. I am Tanya Dimitrova and I’m here with my co-host, Dr. Miranda Wilson.
Miranda: Hi there!
T: Today we have two guests, Dr Jennifer Collinger and Dr Robert Gaunt, are biomedical engineers. They do research at the University of Pittsburgh. Jen, Rob, and their team of colleagues have been working on a machine that sounds just like science fiction: a brain-computer interface that enables people to control a robotic arm just using their brain.
Wow! Imagine that! A person who is paralyzed due to a spinal injury, for instance, can instruct the robotic arm to pick up an object by just thinking about it. And not only that, the person can feel back the pressure when the robotic arm picks up the object.
That is amazing. And it can seriously enhance the quality of life for people with tetraplegia and other disabilities.
The research about this robotic arm has been published in the journal Science and here at Science Journal for Kids. We recently adapted it for school students.
Today, we will talk with Jen and Rob about their work, but we will also get to know, at least a little bit, the people behind the professional scientists. Hi, Jen. Hi, Rob. Welcome to the podcast.
Rob: Great to be here.
J: Yeah, thanks for having us.
M: Excellent. Your current research is clearly at the cutting edge of science and technology. But we’re going to start from the time you were students yourselves. And we’re actually going to start with a question from a student.
Tracy: Hello, my name is Tracy and I’m a student in year 12. What subjects did you take in high school? Were they all sort of science engineering-based?
J: Sure. Yeah, I did take a lot of math and science classes in high school, in particular physics class. Our teacher really showed us how you could apply the principles of physics to the movement of the human body. And that really piqued my interest just because I played a lot of sports in high school. And so I started to think about how you could apply science and engineering to more medically focused problems. And that helped motivate my career.
M: What about you, Rob?
R: Yeah, I did all the sciences, you know, all the ones you would imagine: math, chemistry, physics, biology. But I did a lot of other things in high school as well. I, you know, I played sports. And I also, you know, did a lot of music as well as played the piano and musical theater and those sorts of things. And I think all of those different things all contribute in some way to what I was ultimately able to do.
M: That sounds awesome. It sounds like you were both always interested in STEM in at least high school. What about before that?
J: I probably didn’t have a word for it before high school. You know, but I was always participating in science fairs and sort of anything that was, you know, hands-on where I kind of got to dig in deep into a project. Those were the types of things that I would gravitate towards.
R: And I always, I liked taking things apart. You know, I was that kid that was pulling toys apart, trying to figure out how they work. And, you know, built my own car and repaired my own car, tore the engine down. So I was always interested in how things work, taking them apart and putting them back together.
T: We’re speaking about toy cars, not real-life cars, right?
R: I did that to my that my first car as well. I tore it down to to a pile of parts in my in my parents’ garage and then rebuilt it.
M: What kind of car was it?
R: It was a Chevy S10, a little pickup truck.
T: Were you able to drive it afterwards?
R: I did, and there weren’t too many pieces left over. I guess they were all unimportant ones anyways. [Laughter]
T: So let’s talk about your time studying at university. Jen, you did both your bachelor’s in bioengineering and your PhD at the University of Pittsburgh, where you still are doing research. Ever thought about giving another university a chance?
J: You know, I’ve just really enjoyed my time here and so I keep going. I think throughout my time it sort of evolved, you know, what my focus has been. I actually never intended to go to the University of Pittsburgh because it was close to where I went to high school, but I took a tour of a research lab here where they were studying the way that tendons are loaded and if you could design surgeries to be more effective and prevent re-injury. And I just thought that was a really neat application of the math and science that I’d been studying in high school.
And so I did end up applying to the bioengineering program here at Pitt and, as an undergrad, worked in that same research lab that I had gone on a tour of when I was in high school.
You know, as I was thinking about graduate schoolI certainly expanded my search, but there was an opportunity to work here and shift a little bit more into the field of rehabilitation. And so for my PhD work, I was still doing biomechanics and studying the way that people move. But I was working with people with spinal cord injury who propel their manual wheelchairs. And we were looking at whether the way that they propel their chair with their hands and arms can actually potentially lead to injury of their shoulder, which could be devastating for somebody who relies on their upper limbs for so many functions. And so we were able to identify more effective propulsion techniques that could be taught to hopefully reduce injury. So that sort of got me into the field of rehabilitation.
And then again, another opportunity presented itself to really pivot again a little bit. So now instead of trying to prevent injuries, the goal was really to use neural technology to restore upper limb function. And so my interest just sort of evolved throughout that path and the University of Pittsburgh was a great place for me to keep doing that.
T: Rob, you are from Canada, right?
R: That’s correct.
T: And you first studied mechanical engineering and then you did a PhD in biomedical engineering at the University of Alberta. So I was wondering, did you decide to take that route because engineering just mechanical things was just too boring?
R: I wouldn’t say that. I think much like Jen described for a lot of people, and certainly for me, you know, the more you get into something, you get into a field, the more you learn about it, the more opportunities become apparent to you, things you didn’t know existed. And, you know, now you become aware of. And so it’s really difficult when you’re young to make decisions, especially in research about what you ultimately want to do, because a lot of things, you just don’t know that they exist at all.
You know, and so I did mechanical engineering because I was interested in how things worked and mechanical things. And I think that was a good choice. I spent some time, a few years actually working as a mechanical engineer, but it became clear to me that really my interest was less say in the robots themselves and more on how you might actually control those robots. How might you connect a person to a robot to be able to allow them to control it?
So that’s really where my interest started switching to bioengineering. And you know, the term that we often refer to ourselves now as, as neural engineers, didn’t really exist so much back in those those days. And so a PhD in biomedical engineering was also something that made sense at the time, although I did something that was a little bit different for my PhD, but it did set me up well for what I’m doing today.
T: We’re about to get to the brain-controlled robots, but first I wanted to ask you something a little bit different. We are always interested to know how professional scientists like yourselves manage to balance all aspects of your work life – like doing research, teaching, writing – with your personal lives, spending time with family, taking care of your health, having fun sometimes. So is it possible and how do you do it?
R: Sure it’s you know it’s absolutely possible. I think like many careers, you end up making decisions about how busy you want to be. And certainly this is a busy career. I spend a lot of time working. But I’ve got a great family and we do a lot of things together and you know a few weeks ago we or a couple of months ago now we took two weeks off and went to go through the national parks out in Utah and so you just make these decisions that are, you know, that seem right at the time.
So you know it’s… If you let the work consume you, certainly there is enough of it to do that it could take all your time. And so this really becomes sort of decisions that you make for yourself about how to balance and make sure that you’re putting the right amount of time into the things that are important for you in your life.
T: Right. And I guess that’s true for any professional field, not just professional scientists.
R: You know, the work will never be finished. That’s one thing I came to understand a long time ago. There will always be more of it to do and it will never be finished. And so sometimes you just need to take a break and do something else.
J: You know, one thing I think about an academic career that is really nice is that it does come with a lot of flexibility in different aspects. So to some level, we get to choose the types of problems that we’re working on. And so when you’re working on things that you feel are important to you, that makes the work time go a lot faster. And so even though there is a lot of work to do, we have some control over exactly how we’re spending that time.
And, in terms of balancing with outside of work life, I am married, I have two kids, they’re seven and 10 years old now. And so I think we’re, you know, as my kids get older, I think finding that balance has become a little bit easier, for sure. And so, it’s nice to be able to flex my schedule a little bit if I want to go watch their baseball game or football game, you know, we can kind of take advantage of that. And then, like Rob said, the work is never done. So, we find the time to do that. But there is that flexibility, which I think is really nice.
T: I wonder if each of you could describe for us what a day in your life is like, a typical day. Obviously, every day is different and you just said you have a lot of flexibility in building your program. But what is a day of the life of a professional researcher like?
R: Well, I think, you know, there’s a lot of meetings, there’s a lot of administrative responsibilities, but that, those are always usually focused around moving projects forward. And so things are very project-focused. So I certainly spend time in project meetings.
I also spend a lot of time talking to students. You know, how they’re doing in their experiments, what they were up to, if they were working towards conference deadlines, you know, planning those sorts of things out with them. And so a lot of it is one-on-one meetings with people or larger group meetings with the students and engineers and other staff that we have here in our group. So, that takes on average probably at least half my day every day.
And then I do still spend some time in the lab watching experiments and helping out. And even though I can’t usually, you know, run them myself much anymore, the students are a lot better at all of that stuff than I am. I do like to go in and see what’s happening and at least stay a little bit connected to it.
Apart from that, there’s always writing, working on grants, and other sort of future planning. Like Jen said, there’s a lot of control over what you do. And so that’s actually one of the really fun parts about this job is that you have the opportunity to dream new things. And as long as you can convince a few other people that those ideas are worth it, you can have a shot at actually trying those things out.
J: So most days, I get my kids on the school bus and then head into the office. And yeah, a big chunk of the day is really spent talking to people on the team.
So the work that we do really requires teamwork and people with all different types of expertise. So clinicians, technicians, engineers, we’ve got students, scientists, who are all working together to make, you know, push this science forward. And so we spend a lot of time trying to talk about what the goals are and troubleshooting and just, you know, trying to help their careers progress as well.
And then of course, I get to observe the experiments, see how things are going with each of our study participants who really are also like part of our team, the work that we do. People come work with us for years at a time and they’re in the lab three days a week, typically. And so there’s a lot of experiments that are happening. And it’s really nice to get to know them as well as part of the team.
You know, outside of those things, definitely writing grants and writing papers, trying to continue this work and share what we’ve learned.
T: I think a lot of listeners are coming to realize that scientific research is actually a very team-oriented effort and very few things are done by individual researchers, right? Everyone works as a team. Like you said, Jen, you spend most of your day talking to people.
R: You’re absolutely right. I mean, some days I wish I could carve out one hour to be by myself and work on some of my, you know, some of the things that I need to do. But it really is a team sport. And that’s actually one of the great, fun parts about it.
One of the really inspiring and motivating parts of the job is actually talking to and working with students who are just getting into this, who see their careers out in front of them, teaching them and learning from them as well.
M: So let’s move on to some more specifics about your research. This robotic arm and brain-computer communication channel you’ve created is absolutely amazing. Your paper was published in the journal Science in 2021. Your research has also been published all over the news and has been covered by all the major media channels.
It’s not surprising considering how advanced your research is. I mean, being able to control a robotic arm just with your brain and being able to feel a response sounds almost like something out of a science fiction novel.
But your research wasn’t always so complex. The first iteration of your initial brain-computer interface just relied on vision. And the person using it could control the robotic arm based only on what they could see. But you found that that system was slow compared to the movements of an able-bodied person. So your next iteration included tactile feedback, which really improved the speed of the movements.
So can one of you tell us why that tactile feedback is so important for using these prosthetic arms so effectively?
R: Sure. And I think the right place to start to answer this question, at least briefly, is to think about how important sensation, touch, you know, is for us in our own lives as we move. And we don’t think about it a lot. We think a lot about vision and hearing and taste. Touch, maybe a little bit less.
But I’ll tell you that there are very occasionally people who lose the sense of touch. Or who lose that sense of where our limbs are in space. And so we are filled with sensors, our skin is filled with thousands of sensors, just like our muscles. And if that gets lost – if you lose the ability to feel things – it’s almost like you are paralyzed. Even though you can move your muscles, you can control your muscles normally, strength is normal, it’s very difficult to make effective movements.
Something that many people might be familiar with is if you’ve ever gone to the dentist and you’ve had your tooth frozen or something like that, you know how difficult it is to have a drink of water after that. Water will be dribbling down your lips. You can’t move if you can’t feel, basically.
And so that’s kind of the idea behind it. And although vision is an extremely powerful sense, and we use it to move all the time, we did have this expectation that without that sense of touch, people’s ability to move and control things effectively and rapidly and skillfully, like we do, would be impoverished.
And so that’s why we went down the pathway of trying to add at least some limited sense of artificial touch back into the system. It’s because we know how important that is for us, for able-bodied people in normal movements and behavior.
T: We have a few more questions from students specifically about your research.
Adyant: Hi, I’m Adyant and I’m a 7th grader from California. My first question is which parts of the brain typically receive and interpret information from when the hand touches something? And how is this information communicated?
R: There’s a part of our brain called the somatosensory cortex and it’s right beside the motor cortex. And so these are the two parts of the brain that we target.
In the somatosensory cortex is that part of the brain where there’s basically a map of our bodies that responds when we touch things. And so everybody’s got this part. The map there is always the same. We can find it using imaging. We can put people in an MRI machine and ask them to do tasks and kind of find out where that is. And then that’s where we go and implant our electrodes.
And then with tiny pulses of electricity to that part of the brain, they can experience something like a sense of artificial touch.
T: How do you implant electrodes?
J: Yeah, so the first thing we need to do is figure out where to place them. So both the motor and the somatosensory cortex are quite large relative to the size of the electrodes that we are implanting. So we have the person go in an MRI scanner and measure a functional MRI while they attempt to perform different movements of their hands and fingers. So when they’re doing these attempted movements, even though their spinal cord injury might prevent them from actually moving, we can get fairly detailed maps of where this area in the sensory cortex is that had been previously connected to their hands.
And so we, our neurosurgeon on the team, then kind of plans the location of where these electrodes will be implanted. And we can place them based on these functional maps.
And what it turns out is that even 10 years after spinal cord injury, after this person has never felt a sensation in their own hand, we can stimulate that part of the brain that had previously been receiving that input. And it generates a sensation that feels like it comes from their own hand.
T: We have another question from Polina.
Polina: Hi, my name is Polina and I’m a volunteer at Science Journal for Kids and Teens and today I have a few questions for you. How do biomedical engineers work with other healthcare professionals such as doctors and researchers?
J: Our team is definitely multidisciplinary. So Rob and I are both engineers by training. And so we’re doing a lot of the implementation of the system. So how to extract signals from the motor cortex in order to control a robot, or how to stimulate through electrodes in order to convey that sensory feedback.
But we work closely, obviously, with a neurosurgeon who does the implantation of the electrodes themselves. We’ve worked with occupational and physical therapists to help design the tasks that we’re going to use to evaluate performance. We work closely with rehab physiatrists to manage the care of the study participants who have spinal cord injury.
And again, to just get feedback on, you know, what our goals should be in terms of trying to restore function for people with spinal cord injury. And obviously, the participants themselves are the best experts who can really tell us what they want from these types of devices.
T: We have one question from Tracy.
Tracy: How do you think this technology will develop in the future? Will it give rise to sci-fi scenarios like becoming a cyborg?
T: And even more generally, could you discuss some of the potential ethical concerns that may arise as this technology continues to advance and become more widely adopted?
R: Well, so maybe I’ll say something about what I think the possibilities might be in the future and let Jen talk about some of the ethical considerations that are really important for our research. I think in the very big picture, it’s certainly easy to get excited by the sci-fi possibilities of a brain-computer interface. Certainly, I’m attracted to that, and it’s exciting to think about. I think the important thing to say here is that the extent that we’re able to use brain-computer interfaces to do anything useful at all really is based on what we know about how the brain operates.
And so all the work that we do here is really based on fundamentally the work, you know, dozens and a hundred years of history in neuroscience, of people trying to understand what it is the brain does and how it operates. And we put those things into practice, and discover new problems which we solve. So the extent to which we can imagine doing things in the future really depends on what we know about how the brain works.
Fundamentally, there’s no real reason why we couldn’t build machines that can interface with our brain to do almost anything that you or I do or think normally. But there are all sorts of practical challenges, really, really difficult engineering problems that would need to be solved to do it. You know, the brain has got 80 billion neurons in it and we put a couple of hundred electrodes in so we are vastly under-sampling what is going on in the brain. And closing that gap is a really, really huge problem. There’s a lot of work to do. But it certainly is fun to think about what those possibilities might be in the future.
J: Yeah, and that leads us to think about ethical considerations, particularly for future devices. Because right now, even though while we’re only sampling from, you know, a few hundred neurons, you could imagine if you get more and more information, and algorithms become more and more advanced, that you could learn more information about a person from their brain activity.
And so privacy and how to protect this data, make sure that it’s not used in a way that could be used to discriminate against anyone, and that they retain ownership over their own brain data is definitely a big topic in the field. And how do we actually make sure that that happens?
On the flip side, we are developing technology that we think is going to be helpful for people with motor impairments. And so we want to make sure that it is accessible to everyone. So we don’t want it to be so expensive or specialized that people can’t access it when it really could help them increase their function or quality of life.
And so I think those are two different sides of the challenge. One is where you want to be very cautious and make sure we’re protecting privacy. And on the other side, if we have technology that can actually benefit people, how do we get that out to market so that they can start using it?
T: Right. And actually, one of the students asked specifically about that. He asked…
A: How much money does a bi-directional brain-computer interface cost and how can we work to make them more accessible and affordable?
T: I mean, I don’t know if there is a price tag on it. It’s not something you can buy in the supermarket, obviously, but…
J: Yeah, I think right now it’s difficult to put a price on these devices because they’re really only being used as part of academic research studies and early feasibility clinical trials. So there’s certainly the cost of the devices themselves, but really it’s teams of people who are still trying to figure out the science that is needed to make the devices work as well as possible.
So I think moving forward, it really comes down to kind of scaling this up into a clinical product that can be more widely distributed, right? As you have more of these devices, the cost will naturally come down. And I think it’s just going to have to be a priority of the community, especially early on, because it will be fairly small patient populations that are targeted to try to ensure equitable distribution of the technology.
M: We always like to end our interviews with a fun pie-in-the-sky question. So we know a million dollars doesn’t go as far as it used to, especially in the field of robotics. But let’s say you had a million dollars. What is one burning research question that you would like to try and answer?
R: The answer to that question would be a lot easier for me to come up with if it was $100 million, not $1 million.
M: Okay, let’s go with that, then!
R: Yeah, you’re right. These things, doing this type of work is expensive. It does cost a lot of money. But most of that’s actually going towards paying people – scientists and students to actually figure this out. But alright, let’s get back to the point, alright. And I’ll scale it up to $100 million.
There’s a thing I would like, and then there’s the thing I would want to discover. The thing I really want is for these devices to be able to be made fully implantable and wireless. So we didn’t really talk about the details about that at all right now. But right now there’s some parts of our systems that actually come out of the skin. And while that works very well for now, it’s clearly not where we would want any of these things to be in the future.
So let’s make that problem go away. And maybe with $100 million, we could do that. And that would be great.
And then what I would want to do is, there’s really this question about when you or I feel things, we have this, you know, very rich representation: we can feel all kinds of things, we can discriminate incredibly fine textures, we can, you know,… Our sense of touch is really exquisite.
But when we electrically stimulate in the brain? At best it’s this very crude sensation. It’s really not very much like this rich sense of texture and feeling that we can get.
And I’d like to understand, one, why that is. Why is it that we are somewhat limited in the sensations we could create? And then, could we design systems, what would we need to do? How would we need to redesign the system to create a more natural sense of touch and then, ultimately, hopefully, show that that allows people to do things that they weren’t able to do before.
So that’s kind of the sort of question that motivates me that I’d like that I will keep working on and certainly would give a really solid shot at for $100 million.
J: Yeah, I think like everyone in this field, I would really love to have a fully implanted device that will last forever that gives us access to even more information from the brain than we have access to now. And so I think with that starting point, you can really go after some of the more challenging scientific questions.
And my focus is mostly on the motor control side. So how do we get information out of the brain in order to control something?
And so I’d say there’s maybe two main areas that I’d really like to continue to push forward. One is just trying to make our ability to decode those patterns of activity much more robust and stable. So even though things tend to work well every day, performance can be variable. And this could be due to changes in the participants’ internal state, their level of fatigue, any pain that they’re experiencing. And we really need to be able to adapt to that and make the device work all the time. A big challenge of assistive technology in general is that if it doesn’t work every time and have easy setup, it’s just never going to be used. And so I think it’s really important that neural interfaces meet that goal as well.
The other area is that we’ve been doing a lot of control of robotic arms because we’re interested in studying how to use brain activity to control movement. But ultimately, people with spinal cord injury don’t necessarily want to use a robotic arm as an assistive device. They would really like to reanimate their own limbs. And so I think, you know, partnering with other people who are already trying to investigate these solutions of stimulating muscles and nerves and spinal cord, using exoskeletons to be able to reanimate the limb. I think, you know, pushing on that to actually deliver on a device that people would want to use would be another thing that I would invest that money in.
T: Well, we’d like to thank you both for sharing your time and insights with our listeners. We all learned a lot about how the brain works, how robots work, what professional researcher’s life is. So thank you so much for being on the podcast today.
J: Thanks for having us.
R: Thanks very much.
M: Did you know that you can directly read one of Jen’s and Rob’s scientific papers stripped from its complex scientific jargon and made understandable to readers as young as fourth grade in school? The link is in the show notes, or you can just Google its title: Can a robotic arm be controlled by the brain?
Or go directly to sciencejournalforkids.org and search for “robot”.
T: That’s all for today. This podcast was produced with help from our research assistants Natalia Torres Bejar and students Adyan Bhavsar, Polina Simonenko, and Tracy Kwan Nguyen; sound engineer Maria Mikhailova; and hosts Miranda Wilson and me, Tanya Dimitrova.
Thank you for listening. Subscribe to this podcast to receive notifications about the next episode of Science Journal for Kids, Ask a Scientist.
Till next time!
You can read the student-friendly article by Rob and Jen “Can a robotic arm be controlled by the brain?” at https://www.sciencejournalforkids.org/articles/can-a-robotic-arm-be-controlled-by-the-brain/ in two different reading levels.
Learn how to build a simple robot at home from our Lesson Idea video:
Get the latest updates on Jen and Rob’s research at Rehab Neural Engineering Labs at the University of Pittsburgh and watch videos with the robotic arm at https://www.rnel.pitt.edu/research/neuroprosthetics/intracortical-brain-computer-interfaces
Their original paper was published in the journal Science.