What is TeleDrive?
The COVID-19 pandemic left FIRST Tech Challenge competitions, scrimmages, and team driver practice in the dark. Because of local restrictions, many teams use online resources like Zoom to communicate and start preparing for the next season. However, there still isn’t a solution to being able to practice driving the robot as well as coordinating competitions and scrimmages – all remotely… until now! Welcome to TeleDrive.
TeleDrive is a simple, yet powerful tool that allows teams to control their robot from the comfort of their homes. Using this, competitions can be run remotely and teams can drive their robot without having to worry about personal health and safety. Our program uses simple commands to bring low-latency and high-performance control, just as if controlling your robot on the field.
How does TeleDrive work?
TeleDrive has two main parts: sending the movement commands and receiving the live view of the robot moving on the field. By utilizing the Internet’s great power, all of these happen with less than 25-millisecond latency. How do we do it? Great question.
Sending Gamepad Commands to the Robot:
The first and main part of TeleDrive is the transmission of gamepad commands from the sender (each of the drivers’ laptops) to the robot, using the Internet. Each part of this is written such that drivers can control the robot with minimal latency. TeleDrive is built upon UDP, an internet protocol that allows data to be sent by connecting to another IP address and port. This allows for sub-15 millisecond control. Unlike TCP, if data is not received, it wouldn’t wait to resend the data which results in extremely low latency. In this setup, there are three computers required: the two driver-controlled computers (on the left side) and the “middle-man” computer (on the right side). On each of the computers – the driver controller computers, and the “middle-man” computer – a script is running in the background to either listen or send data. On the driver controller computers, the script will send to the specific IP address of the “middle-man” computer. The “middle-man” computer, which is listening on a UDP socket, immediately relays the data to the robot controller (or Control Hub) as it is also connected to the robot controller’s IP address. To easily convert these gamepad controls, we have created a Java class that reads the data sent and sets the “gamepad1” and “gamepad2” variables accordingly. Since this code extends the “OpMode” class, you can just extend our class to get the full functionality of your TeleOp code with UDP without changing any of the preexisting OpMode code.
Receiving Extremely Low-Latency Live Stream of the Robot:
The second part of TeleDrive is being able to see how your robot is moving on the field without having to deal with the latency of the image being sent back. With the low latency transmission of commands from the driver controllers all the way to the robot, remotely, it has very little latency and we wanted the live camera feed to be the same. To do so, we use an app called Parsec which is able to send the live stream to multiple computers connected to the host (laptop on the left side). Parsec is an application that uses UDP, the same protocol as sending the commands, but has one key difference which allows ping times under 25 milliseconds: they didn’t build on top of any wrappers since they wanted full hardware control and the ability to manipulate every element of the stream to reduce latency as much as possible. Parsec acts as a screen sharing in our case where multiple driver controller computers become the clients for the host computer and receive the live stream from the host. From this, the drivers are able to watch their robot move while they click buttons on their gamepads all within 1/40 of a second!
TeleDrive in Real Life
Check out the TeleDrive Demo!
The Future of TeleDrive
Robot Controller App Proposal:
Our current system for sending the robot movement commands through the Internet is very simple. But that isn’t enough. We believe that if the Robot Controller phone or Control Hub were to be connected to the Internet, there should be the ability to send commands to init, start, and stop the OpModes remotely as well. Currently, one person has to be near the robot to start and stop the OpMode when ready and the use of the “middle-man” computer will then be redundant, simplifying the flow of commands from the driver to the robot. Essentially, if the Robot Controller app is changed so that it can receive commands via WiFi and through the Internet, it would simplify the whole flow while reducing latency even more. With these changes to the Robot Controller app, it would be super-simple for teams to setup TeleDrive for their TeleOp code and get ready for remote competitions, scrimmages, and even driver practice.
Proposal to FIRST:
Because of the pandemic, we did not have a concrete solution for holding competitions and we believe that TeleDrive is made for just that. Since TeleDrive only needs one computer per driver at their remote locations and one computer to receive and relay the commands to the robot, TeleDrive makes it very simple to host competitions without having the drivers attend in person. Not only that, sending the video stream back is very simple – all the computer needs is Parsec, the application used for live streaming with low-latency, and Internet connection. We feel that TeleDrive brings a solution to the table in solving the problem of hosting competitions remotely during these challenging times of COVID-19. In fact, the only people who need to be at the competition itself would be the Referees, Competition Coordinators, and Judges. All teams would simply have to drop off their robot beforehand and connect to the Robot Controller directly through WiFi, as mentioned in the previous section.
I exchanged a few emails with REV proposing something like this. Maybe you should contact them for the “final mile” solution of making the REV sync up with the WiFi instead of a phone’s WiFi direct.
Thank you for the advice! We’re currently working on integrating it with the existing FTC app software!
Amazing tech. I’m part of the volunteer squad for FTC engineering. Would you be willing to “meet” with another team working on something similar? They would like to trade and combine both to make a more complete whole???
Wow! I’m part of the volunteer squad for FTC engineering. Would you be willing to “meet” with another team working on something similar? They would like to trade and combine both to make a more complete whole???
Hello Mr. Hall,
I have recently emailed you at khall@firstinspires.org. Looking forward to talking more!
Very impressive! Thanks for your hard work!
Thank you, Sandy!
My name is Jonathan Weiland and I’m the director of FIRST Tech challenge in Illinois. I was hoping to contact Mihir. I came across your teledrive project and I’m trying to use that in our state to help kids who can’t compete this year. I’m also working with a team of students to develop a robot simulator for the FIRST Tech challenge. Below is a flow chart/ diagram of what we hope to do and some examples of what have already been created by people on our team.
FTC virtual robot simulator diagram https://drive.google.com/file/d/1hdJ5vRX_s30YqjTQJAZNER7bOabn0tkb/view
Following are a couple of videos of what we are trying to accomplish, very briefly we want to enable FIRST robotics development and eventually contests to happen online through a virtual field based on Unity and ROS.
1. https://www.youtube.com/watch?v=A66TlzYuKSo&t=73s
2. https://www.youtube.com/watch?v=gFCLU5iWXp0
I was hoping that we could chat or zoom if you’re interested in finding out about our project and maybe even joining .
Hello Mr. Weiland,
I have recently emailed you at jwftcil@gmail.com. Looking forward to taking more!
try this email jonathan weiland .
I never saw your first email
Are you able to have one driver driving in-person and one driver driving remotely?
Yes, one driver can drive through TeleDrive and one can drive with the wired controller to the Driver Station phone. Since our TeleDrive Java class uses the gamepads, you can use it wired or you can also use it over the Internet.
This is so awesome. Is the host code TeleDrive_Host and TeleDrive_Driver also open source? Our team would like to learn this works.
Thank you! Yes, we are going to open the code for teams very soon.
Cool innovation, nicely done. Documenting some prerequisites and assumptions will help set expectations for teams that want to try this out. For example, this approach requires one to extend the Internet connection point (public IP address and port) to the PC/Mac that is acting as the relay host. In a home environment this is fairly easy. In school environments I think this will be more difficult because students/coaches won’t typically have access to the school’s Internet connection point to set up the port translation and allow the incoming traffic through the firewall. With respect to the latency, it sounds like you have some pretty awesome Internet service capabilities at your location. Others might not see that kind of performance if they are dealing with lower quality ISP services. I mention these things because a local coach contact me to help set this up, and I’m having to adjust the coach’s expectations as I do so.
Thank you, Mark, for your feedback. We are working on an improved version where everything is in one place and easier to use.
Hi Mark,
I am excited to share with you TeleDrive 2.0!
This all-in-one app that not only allows you to drive your robot remotely but also includes the capability for low latency video feed with group audio call. All of this with end-to-end encryption!
If you used TeleDrive 1.0 then you may remember having to use Parsec for video, Discord for audio, and the TeleDrive executables for gamepad. This version simplifies the setup with ease of use.
Also no need to set up port forwarding on your router anymore, so it will work great in school environments!
Get the Mac/Windows Apps Here: https://github.com/innov8rz-ftc-team-11039/TeleDrive-2.0/releases/tag/v1.0
GitHub Repository link for installation and setup: https://github.com/innov8rz-ftc-team-11039/TeleDrive-2.0