PinPointr
This past academic year, I had the pleasure of being a faculty advisor for a group of four Computer Science students completing their 4th year capstone project. How did this arrangement come to be? I was approached early in the fall term by Victor Timpau, who was enrolled in the Introduction to GIS course that I was teaching. Prior to this year, Victor twice participated in the annual ECCE App Challenge (2016 (pitch video https://vimeo.com/157817386) and 2017 (pitch video: https://youtu.be/b1f7HI3CrWg). Victor expressed interest in completing his capstone using elements of GIS. One other of his teammates, Matthew Kipp, also participated in the ECCE App Challenge in 2017 (pitch video: https://youtu.be/uN4NAquFan8). Through their brainstorming efforts they came up with Pinpointr, a mobile app based on machine learning and location to identify different maintenance issues on McMaster’s campus. Upon completion of their impressive app, I asked the group to compose a blog post for the ECCE site. Here is what Brandon Ronald, one of the members of the team had to say about their app, Pinpointr!
With a rapidly-evolving infrastructure and student population of over 30,000 and growing, various challenges have emerged in maintaining McMaster’s beautiful campus. Pinpointr is an application designed with the aim of solving these challenges in mind. Developed by a team of 4th year McMaster Computer Science students (Matthew Kipp, Sean McKay, Victor Timpau, and myself), the app seeks to provide an end-to-end reporting system for the unsightly and occasionally unsafe issues that occur on campus. These issues range from mild inconveniences such as overflowing garbage cans or empty hand sanitizers, to high-priority issues such as broken furniture or tripping hazards like ice on stairs.
The Pinpointr application system evolved from a simple realization: when a campus maintenance issue (CMI) arises, students or other non-maintenance university staff members are likely to be the first to notice. Furthermore, with the vast majority of modern-day students carrying smartphones on a regular basis, there is significant potential in developing a mobile application to crowd-source data regarding campus maintenance issues directly from the student body.
Though the idea itself is simple, its implementation has not been without significant challenges. In order to be useful for both front-end users (students, faculty members, etc.) and back-end users (facility services staff), the front-end application must be accessible and simple, while providing a sufficient level of useful data to the back-end regarding the issue being reported. This is where the Pinpointr system seeks to combine recent technologies in order to turn a complicated, time-consuming process such as filling out a lengthy form describing an issue, into a user-friendly, familiar one. Specifically, we aimed to create an application that makes the process of reporting a CMI as simple as the process of taking a photo and sending it to a friend. What follows is a discussion of the main components of Pinpointr:
Pinpointr – The Mobile App
In order for a user report of a CMI to be useful, it needs to provide two fundamental pieces of information: what the problem is, and where it is located. The challenge arises in developing a mobile application that can not only obtain this data, but do so while keeping the reporting process simple for the user. This is where the combination of machine learning-assisted computer vision (the what) and geolocation technology (the where) come into play.
Machine Learning
When the user takes the photo with the Pinpointr application, the image is instantly fed through a convolutional neural network. This neural network – trained on thousands of photos taken on campus during the development process – will then use the image to categorize what type of maintenance issue(s) it contains via image tags. In order to generate these image tags, Pinpointr uses an Object Detection algorithm based on Microsoft’s Computer Vision, with processing hosted on Microsoft’s Azure servers.
Geolocation
While the neural network works at identifying what the image contains, the photo’s geolocation is used to tell where it was taken. The photo’s location is then referenced against a shapefile dataset of campus containing the outlines of all of the buildings and significant outdoor areas on campus. Using a ray-casting/collision detection algorithm, the application then attempts to locate the building, room number, or physical site of the CMI.
User Verification
Once the user has verified the image’s tag(s) and location generated by the app, with a simple button tap they can then send the CMI report, containing the image and all relevant data to the back end of the application, used by facility services. An email is sent to McMaster’s facility services team, allowing them to review the report enter the CMI information into their own system. The CMI also appears as a pin on the Pinpointr website.
PinPointr – The Web App
The Pinpointr web application provides a map-oriented view of all recently reported issues, which are displayed on the map according to their geolocation. The site is built on Leaflet, which is a lightweight set of tools for using ArcGIS services. For those unfamiliar with Leaflet, it
is the leading open-source JavaScript library for mobile-friendly interactive maps. The Leaflet plugins work with several ArcGIS Service types including Esri basemaps and feature services, as well as tiled map, dynamic map and image services. In Pinpointr, we make use of the Esri basemap centered on McMaster’s campus. We have come to learn that the data provided for the campus area of the basemap came from our faculty advisor Patrick DeLuca, who continues to update the layer with the help from the folks at the Esri Canada Community Map Program whenever new buildings crop up on campus.
Future Potential
Pinpointr is a project with vast potential for future development. For example, geolocation accuracy has potential to continue to improve with technology. Newer mobile operating systems such as Android Pie (Android 9.0) allow for geolocation calculation via Round Trip Time (RTT). RTT, in combination with newer WiFi technology, uses wireless access points to calculate the user’s location accurate within 1-2 metres. This will allow Pinpointr to locate the building, floor, and room number of a CMI with greater precision, leading to increased ease of use.
On Pinpointr’s web application, adding Leaflet’s Heatmap plugin will allow the site to effectively display what kinds of CMI’s are most commonly reported, and where they occur. This data can potentially allow the facilities services team to improve efficiency through effective resource allocation.
Furthermore, deeper integration with McMaster’s existing systems for reporting campus issues will help simplify the full process of reporting issues and having them resolved. There is also potential of having custodial staff adopt this application as the standard for reporting over paper forms.
Lastly, the machine learning model behind Pinpointr’s image analysis algorithm has the potential to learn and improve upon itself over time. With users and staff continuously verifying and correcting image tags predicted by the model, the model’s training dataset continues to grow, allowing for image tag predictions to improve in accuracy over time.
In sum, Pinpointr’s current system provides a simple way for students and staff to report maintenance issues on campus. With wider user adoption and further development, it has the potential to become the standard technology behind maintaining McMaster’s campus, and perhaps beyond.