This spotlight post showcases the work that Tianming Wei has been involved over the past year related to a series of NSERC grants at the University of British Columbia and the University of Victoria in collaboration with Esri Canada. He became an ECCE student associate at UVic earlier this year after participating in the App Challenge, and will continue to collaborate with the Education and Research group at Esri Canada in relation to this project.

Canadian urban populations are increasingly aware of climate change and generally support mitigation strategies.  However, smart city planning and adaptation requires extensive knowledge of urban design. Cities are challenged in their response often due to a lack of public understanding of how urban form impacts energy use and emissions.  We propose to design, deploy and evaluate digital media tools that support collaborative exploration of urban design scenarios, including participants from remote locations.  Key elements of this collaborative platform include hands-on interactive tabletop displays and companion immersive visualization tools to support evidence-based understanding of energy usage and emissions.  Related urban design factors include affordability, housing diversity, neighborhood walkability, and transit access. Web-enabled geographic information systems (GIS) and computer-aided design (CAD) information will be integrated into a responsive and interactive displays.  These will include touch-tables, augmented with projected, hand-held, and head-worn displays customized to the workflow of public engagement workshops that employ participatory design approaches.

We are building on our previous work with an urban design application, UD Co-Spaces, a multi-user digital touch tabletop platform developed for collaborative exploration of the proper mix of residential, commercial, and public spaces in urban environments with real-time feedback on key sustainability indicators. The tabletop is augmented with auxiliary 3D overview wall displays and iPad viewpoint control.  UD Co-Spaces (Figure 1) has been deployed in community workshops by two cities Victoria and Vancouver in BC.

Figure 1: UD Co-Spaces system architecture.  An urban design is developed on the touch table, with metadata using Google earth as a map service, a backend database of prefabricated buildings (ElementDB), and support for a handheld controller (PDA/iPad) and a 3D viewer.

Importantly, visualization tools enable people to immerse themselves in their created virtual world while seeing and understanding the data-enabled implications of decisions related to energy usage, emissions, affordability, housing diversity, neighbourhood walkability, and transit access (Figure 2). Urban design professionals increasingly rely on advanced software tools, but those are often designed for individual work, do not adequately support rich collaboration, and are not accessible by novice members of the public. The UD Co-Spaces platform was built to accompany community oriented sessions in which people from various backgrounds can partake in the town-planning experience, in a shared physical location. Planning the built environment is complex and multi-faceted. Public engagement is essential when addressing traditional issues of land usage and provision of public services, but global issues such as climate change are increasingly part of the process. The urban built environment, including buildings, energy systems, and transportation, accounts for up to 80% of Canada’s greenhouse gas (GHG) emissions.

Our proposal is to research new services for urban design platforms to allow for groups to participate remotely — ideally from several locations simultaneously — through the use of cloud-based serverless architectures and augmented reality.  We plan to work closely with supporting organizations to develop novel digital media support for workflows that are important for public consultation when using a powerful GIS and leading edge digital media technology.  The general objective of this research project is to design, deploy and evaluate visualization and tooling to support data analytics in this urban design for smart cities context. Specifically, we aim to visualize urban form alongside urban sustainability metrics. Current digital solutions focus more on expert knowledge and less on public engagement.

Figure 2: The User Interfaces in UD Co-Spaces.

This research project will be done in collaboration with Esri to evaluate new software tools for an optimized experience of using geospatial map services for both two-dimensional and three-dimensional visualizations during urban design processes.  In order to innovate in terms of services for this class of applications, we will continue to build on UD Co-Spaces as an initial tabletop collaborative platform.  We will deploy and analyze cloud-based micro-services to replace and augment multiple components in the platform.  These will work with current services deployed in the original implementation, including a resource hosting service from an Apache server, a central message relay service from a Rabbit-MQ server, and a geographical map service from remote Esri servers.

We plan to evaluate the platform in terms of microbenchmarks for performance to determine the efficacy of a cloud-based approach in the scenario where multiple users are remotely connected to the platform simultaneously, sharing and editing their workspaces collaboratively. With this design, the platform can be setup on a cloud-based network and the micro-services can be created on demand from anywhere in the cloud rather than on a single local server, as was the case with the original architecture. The local server needed to be powerful enough to support all the services, and maintained throughout updates and failure modes that were often problematic to debug.

In order to support future immersive visualizations in Virtual and Augmented Reality with devices such as the HTC Vive, Oculus Rift, and Microsoft Hololens, the services must also be able to selectively increase the capability of streaming a large quantity data in close to real-time.  The challenge here will be to get the right mix of local, cloud, and preprocessed data to serve the application needs.

Services we plan to focus on include a range of local, cloud-based, and hybrid combinations of the two.  The following sections outline some of the services we plan to focus on, including: multi-touch, multi-layer, distributed use cases, scaling smart buildings, AR/VR support, and an easy to deploy and debug framework for the entire application.

Multi-touch service

The previous platform utilized a deprecated touch framework API, which was discontinued in 2008.  This API does not support the multi-touch feature (Figure 3) on top of the overlay of Esri basemaps and thus needs to be replaced with a newer service that can support the native touch in Microsoft Windows 10 and works well with the ArcGIS Runtime SDK.  An additional challenge here will be extend this functionality to an equivalent within AR/VR environments.

Figure 3: Multi-touch Support on UD Co-Spaces Platform.

Multi-layer service

Taking the advantage of ArcGIS Runtime SDK, it would be possible to create multiple layers on top of the basemap for different design tasks (Figure 4) while being able to temporarily disable unused layers, e.g., building footprints, study areas, topological informatic layers and so on.  The new service should also support displaying external third party GIS data, such as customized 3D models for buildings, cloud-based datasets for existing architecture and territory, and real-time information for public transportation.

Figure 4: Multi-layer Support within UD Co-Spaces Platform.

Distributed use-case service

In the scenario where multiple touch tables could be physically in the same room, the current platform provides models for design patterns and workspaces that can be saved, shared, and duplicated.  Projections on large wall displays also help when layers are evaluated for cross comparison. In future scenarios with remote deployment on multiple touch tables that are not physically in the same location, the system requires cloud-based deployment so that multiple micro-services can run within the distributed system. This could also help to support multiple virtual touch tables that are created with AR/VR.

Smart building scale service

This service would be integrated in the new platform to support dynamic scaling of pre-designed buildings. Scaling would need to change not only the size but the proportional energy footprint of smart buildings. This service will require extensive consultation with professional urban planners, and the service will rely on both pattern recognition and machine learning.  The balance of whether this is local or cloud-based or a combination will be part of the research associated with this service.

AR Support in the Microsoft Hololens

Based on the objective that the new platform be collaborative, we are actively exploring the ways in which AR (Figure 5) could be used to aid the design process by providing an immersive experience for urban planning that can be used alongside the touch tables and projectors. The visualizations could leverage the support of message and behavior monitoring through Rabbit-MQ, and content sharing from different design tasks and workspaces on multiple touch tables (both physical and virtual).

Figure 5: UD Co-Spaces Platform in VR with the Microsoft Hololens for Display

Tool support for deploying and debugging

Deploying this system is nontrivial, and given the opportunity to dynamically optimize by adjusting local/cloud-based services, we are planning tool support.  The interface will assist urban designers to recognize when services are started (Apache Server, Rabbit-MQ, case database, web-based widgets), their dependencies, and the current deployment context. Within this tool we will explore a re-bundling, possibly under Microsoft Visual Studio (Enterprise Version), so that multiple services could be installed and deployed and monitored more easily.  Additionally, visual debugging at an inter-service granularity will also be facilitated through this interface.