Work

Machine Learning on iOS

An experiment in running road recognition right on a phone, to test whether a phone could serve as the brain of a self-driving system.

  • Taoyuan Vocational HS (NTUT-affiliated)
  • HS Year 3
  • 2023
  • Solo project
On the left, a phone showing a road with two lane lines; in the middle, an arrow pointing right; on the right, a phone showing the detected lane lines.

How It Started

October is when the TIRT vision self-driving car contest kicks off, so I teamed up with last year’s teammates again as a three-person crew to get ready. Last year we found that the Raspberry Pi just didn’t have enough horsepower. While trying to fix that, I had a lightbulb moment: I’d learned Swift over the summer, so why not use a phone as the main board? Phones have high-res cameras, powerful processors, precise 9-axis IMUs, and a lot more going for them.

So I started digging into how to actually pull it off, and found that DeepLabV3, Google’s semantic segmentation framework, was a great fit for recognizing roads. From there, I set out to figure out how to get a DeepLabV3 model running inside an iOS app!

How I Built It

  1. Hand-labeled the lane lines with an annotation tool
  2. Trained a model on the labeled images using Python TensorFlow and the DeepLab V3 framework
  3. Converted the TensorFlow model into a Core ML model with the Core ML conversion tools
  4. Loaded and initialized the model in Swift
  5. Loaded an image and stored it in a UIImage variable
  6. Converted the UIImage into a MultiArray of shape [1×512×512×3]
  7. Created a prediction request with the Core ML model using the array-format image
  8. Took the output array, converted it back to a UIImage, and displayed it
  9. Ran a simple for loop to mark out the lane lines

What I Learned

Running machine learning in an iOS app turned out way better than I expected. A model trained on just 180 images already beat my expectations. If the organizers open up the course for practice later, we could grab real track data and train on more images, and it should work pretty well.

The one small problem was that we kind of overestimated the phone’s processor. We assumed 60+ fps would be a breeze, but in reality we only got around 12 fps. Then we tried shrinking the images from 512×512 down to 128×128 and got close to 70 fps. Road recognition doesn’t need super high resolution anyway, so we went with that!

Since road recognition worked so much better than expected, I brought back the self-driving campus car idea I’d casually tossed around back in my first year of high school and pitched it to a few classmates again. A few of them who were into fixing bikes got really excited about it too. So to make sure this plan didn’t end up as just talk, I got everyone to chip in right away, and once we’d pooled NT$14,000 I bought a used six-seater surrey bike and parked it in our department building. Hopefully one day we’ll be the only high school with a self-driving car!