JMA Wireless, Amazon Web Services (AWS) and Crown Castle have completed a private LTE network deployment for Carnegie Mellon University using Citizens Broadband Radio Service (CBRS) spectrum.

The CBRS network took less than three months to construct and commission, and first went live this summer in June. JMA and AWS teamed up to design and help build the network, which uses JMA’s XRAN virtualized RAN software with a Druid Software core running on AWS Snowball Edge. JMA outdoor CBRS radios were installed on campus alongside the vendor’s directional CBRS antennas deployed by Crown Castle.

https://bit.ly/3kVLE3n

JMA Wireless (JMA) announced today the completion of a new private wireless network for Carnegie Mellon University (CMU) that will use Amazon Web Services, Inc. (AWS) to power ongoing research and innovation in CMU’s Living Edge Labs. https://bit.ly/3DJwZR4

In edge computing, network latency is key. Edge-native applications often benefit from the lowest latency and the least latency variation possible. For many applications, network round trip latencies less than 50ms are necessary to provide acceptable user experience. Some applications requiring near real-time control or extremely fast response need even less than 10ms. While we’ve been able to reach these levels in controlled wired and Wi-Fi environments, the performance of commercial 4G LTE networks has been one of the biggest challenges for the Living Edge Lab here at Carnegie Mellon University in Pittsburgh.

In 2017 to support our edge computing research, we built our first private LTE network. Working with Crown Castle and DT, we created a small 4G LTE network using an experimental Band 42 3.5GHz license. The network had three outdoor sites and provided limited coverage near the Carnegie Mellon University campus. While we were able to get better latencies than from commercial carriers, the number and types of commercially available devices for that frequency band, limited coverage, and limited support for the equipment made it difficult to use for much of our edge-native application research. Despite these challenges, the initial testbed provided significant learnings into wireless network architectures and requirements for edge computing.

Fast forward to late-2020. By this time, unlicensed GAA CBRS spectrum was available and commercial 4G LTE network and user equipment for this frequency band was increasingly available. We began a plan with our partners to upgrade the existing network to a modern, supportable network in the CBRS band, covering a broader geographic area with lower latency and greater throughput. Crown Castle, JMA Wireless and Amazon Web Services stepped up to redesign, sponsor and support the building of the network. In mid-June 2021, we collectively brought the full network into service and have begun using it for our research.

The network (shown below) consists of four outdoor cell sites using JMA CBRS Cell Hubs and directional antennas providing coverage for much of the outdoor space on the main Carnegie Mellon campus, in a public park known as Schenley Plaza near campus, and in an urban shopping district on Walnut Street. JMA deployed their 4G/5G virtualized radio access network (VRAN) on an off-the-shelf Dell/Intel server and worked with AWS to deploy the wireless evolved packet core (EPC) on AWS Snowball. The VRAN and EPC are in our Living Edge Lab data center and connected to the radios over the Crown Castle fiber network. The entire network was constructed and commissioned in less than three months.

Figure 1 – Network Overview

We’re getting great coverage from the network in the areas we’d hoped for; mean latencies are in the 30-35ms range with a 5-8ms standard deviation. Throughput is running 60-80 Mbps download and 15-25 Mbps upload. We’ve also been able to benchmark using our OpenRTiST application and are getting consistently 75ms round trip times and 26 frames per second, not quite as good as Wi-Fi but better than commercial 4G LTE networks – especially when those commercial networks use remote interexchange points (See Figure 2).

Figure 2 – Latency Measurements

Going forward, we are planning to tune the network to reduce the latency even further. One key effort will be upgrading the VRAN and EPC to 5G but we’re also looking at other options. And, we have plans for the network beyond latency reduction. Since we’re big believers in open source, we’re working with Facebook Connectivity to add Magma to the network. We’re considering adding more sites including some indoors. And, of course, we’re expanding our research, student projects and industry collaborations on the network.

I’ll conclude with a special thank you to our industry partners, Crown Castle, JMA Wireless and Amazon Web Services -- who supported the creation of this network – they are true partners in every sense of the word. Also, a shoutout to Druid Software and Federated Wireless who provided the wireless core and SAS service, respectively. It couldn’t have happened without them all. We learned a lot from this experience and we’re happy to share our learnings. If you have questions, feel free to reach out at info@openedgecomputing.org.

Like so many of us, I recently needed to search online for a Covid 19 vaccine appointment. I found a site that did a particularly good job of identifying locations with open appointment slots near me and enthusiastically began a frantic cycle of refreshing, entering my information and requesting the appointments. However, I soon learned that not only were others just a bit faster than I at snagging the golden tickets, but the system’s data update speed lagged the actual bookings of the appointments. I missed appointment after appointment. While I was frustrated with my lack of success, I didn’t give up – I just adapted my re-attempt rate to something more aligned to the real availability of new appointments. Rather than refresh every 30 seconds, I’d wait 5 minutes then 10 then 20 to retry.  In short, the system taught me to slow my pace to its speed.

This pacing effect in response to system delays is both intuitive and common. And, it works both ways. In the early days of both Windows and Netflix, it was quite common to see a spinning icon that indicated that the system was slow. We learned to wait or to try something else. But, now, as these systems have gotten better, we have less patience for delays – we learned a faster pace. There is a long history of research on the impacts of system delay on human responses. As technologies and applications evolve, our understanding of these impacts also evolves.

Some recent work by Manuel Olguin Muñoz, Roberta Klatzky, Junjue Wang, Padmanabhan Pillai, Mahadev Satyanarayanan, and James Gross at the Carnegie Mellon University Living Edge Lab and the KTH Royal Institute of Technology in Stockholm looked at these effects in the context of wearable cognitive assistance (WCA) applications, a class of augmented reality applications that I’ve discussed in previous blogs. These applications focus on giving real time automated instructions to a user performing a task, say, playing ping pong or assembling a Lego structure. In this context, there is continual feedback and response between the user and the system as the user performs the task. Our team was interested in these questions:

• Do WCA users change their behavior in response to system delays?
• Are their behavioral changes due to cognitive and/or emotional effects?
• Do they show any physiological responses to system delays?
• Do users’ personality characteristics impact their response?

They conducted a study with 40 undergraduate students interacting with a Lego assembly WCA application that was modified to introduce controlled system delays. The students’ responses were measured with application traces, biometric sensors and video captured from each session.

What they found was a clear indication of the pacing effect. When the system was slow giving responses, users slowed down their own execution of the task. When the system sped up, they sped up. However, it took some time for users to increase their pace after the system accelerated. While the team hypothesized that these changes would be, at least in part, due to frustration, anger or other emotions, they did not see any measurable indication of this in users’ heart rate, movement, EEG or galvanic skin response. They concluded that the pacing effect is primarily driven by a cognitive adjustment to the pace – just as I did in response to the slow updating of the Covid vaccine site.

They also looked into whether there was any correlation between user pacing response and user personality traits as measured by the Big Five Inventory of Personality and the Immersive Tendencies Questionnaire. They found that the traits of neuroticism, focus and involvement play a role explaining the differences between individual users. In this study, neuroticism, although not in the same sense as the common conception of that term, was the trait most correlated with user pacing. Here, it is more about the student’s personal tolerance for delay.

These results provide some insights for the designers of augmented reality applications – especially WCA applications:

• System slowdowns can lead to substantially longer user task completion times. This effect can cause greater than expected system resource consumption. Applying more resources to keep the pace may paradoxically result in use of fewer resources overall.
• And, given users’ slowed return to pace after the system recovers, it can be more valuable to use resources to keep the pace than to apply additional resources when system delays occur.
• On a multi-user system, a resource allocation bias toward fast users may be more efficient than a “fair” allocation of resources between fast and slow users. It allows the fast users to complete their tasks quickly and leave the system.
• It may be possible to maximize user satisfaction and resource efficiency by adjusting resource allocation in favor of those users known to be sensitive to system delays.
• Previous research has also shown that when delay occurs is also important. In WCA applications, delay matters most when the user has finished a step and is waiting for the next instruction.

As people use new application classes like augmented reality, our understanding of human-computer interactions expands to include new ways of interacting. And, because edge-native applications are very often highly interactive, we believe that the growth of the edge will be a fertile ground for new learnings. We welcome feedback and new ideas in this space – there is still much to discover. For more information, see the following references. By the way, I’m happy to say that my persistence paid off and I am now fully vaccinated!

FOR MORE INFORMATION

1. Olguín Muñoz M, Klatzky R, Wang J, Pillai P, Satyanarayanan M, Gross J (2021) Impact of delayed response on wearable cognitive assistance. PLoS ONE 16(3): e0248690. https://doi.org/10.1371/journal.pone.0248690
2. Ha K, Chen Z, Hu W, Richter W, Pillai P, Satyanarayanan M. Towards Wearable Cognitive Assistance.
   In: Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services. MobiSys’14. New York, NY, USA: ACM; 2014. p. 68–81. Available from: http://doi.acm.org/10.1145/2594368.2594383
3. Junjue Wang, Ziqiang Feng, Shilpa George, Roger Iyengar, Padmanabhan Pillai, and Mahadev Satyanarayanan. 2019. Towards scalable edge-native applications. In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing (SEC '19). Association for Computing Machinery, New York, NY, USA, 152–165. DOI: https://doi.org/10.1145/3318216.3363308
4. Jim Dabrowski, Ethan V. Munson, 40 years of searching for the best computer system response time, Interacting with Computers, Volume 23, Issue 5, September 2011, Pages 555–564, https://doi.org/10.1016/j.intcom.2011.05.008
5. Wearable Cognitive Assistance Blogs:

• https://www.openedgecomputing.org/when-the-trainer-cant-be-there/
• https://www.openedgecomputing.org/putting-it-all-together-with-cognitive-assistance/

OpenScout is an edge-native application designed for automated situational awareness. The idea behind OpenScout was to build a pipeline that would support automated object detection and facial recognition. This kind of situational awareness is crucial in domains such as disaster recovery and military operations where connection over the WAN to the cloud may be disrupted or temporarily disconnected.

Based upon the Gabriel platform, OpenScout performs object detection and facial recognition on the image frames sent by Android clients in the field. The information extracted from these images is then propagated into an ELK (elasticsearch-logstash-kibana) stack where further analysis and visualization can occur. The client is also returned this information for debugging purposes. Images with the bounding boxes of the objects and faces that were detected can optionally be stored on the server for further evaluation.

Just like other Gabriel applications, OpenScout clients connect to the core Gabriel container. The Gabriel server passes the image frames to both the object detection and face recognition engines. Currently we support two different methods for face recognition: OpenFace which was developed at CMU and is opensource; and Microsoft’s Face Cognitive service. The results from both the engines are then pushed into an ELK stack where other users can query, analyze, and visualize the data captured.

The object detection engine is based on Tensorflow and the included DNN is SSD Resnet 50 based upon the COCO dataset. Though OpenScout is packaged with SSD-Resnet 50, any Tensorflow model can be used. This includes any other DNN from the Tensorflow model zoo or a custom trained model.

For face recognition, we have included a default training set which includes three celebrities: Dwayne Johnson, Saorise Ronan, and Hugh Jackman. New faces can be trained in two ways: either ad-hoc from the Android client, or by manually adding training images to the face engine docker container before runtime.

After being processed by the cognitive engines, data is then pushed into ELK, allowing for visualizations like the one shown above. On the left is a pie chart that shows the distribution of the types of objects that have been detected. The map on the right shows the number of detections by location, leveraging the GPS coordinates sent by the device with each frame.

We envision additional cognitive engines in the future that can run in parallel to the object detection and face recognition algorithms. One can imagine having an image classifier that attempts to classify the national origin of military equipment or garb, based upon colors, flags, or other insignia. Or perhaps running OCR on images to automatically extract textual information observed in the field.

A video demonstration of OpenScout can be seen below, where we recognize celebrity faces, detect some household objects, and train the face recognition engine on the fly to recognize a new face.

OpenScout is open-source (Apache License 2.0). The source code is available on Github while the backend Docker image can be pulled from Docker Hub and the Android client can be downloaded from the Google PlayStore.

A discussion with Mahadev Satyanarayanan, the Carnegie Group University professor of computer science

In November 2020, the Deloitte Cloud Institute invited Mahadev Satyanarayanan, the Carnegie Group University Professor of Computer Science at Carnegie Mellon University, as the first Deloitte Cloud Institute Fellow.¹ Dr. Satyanarayanan (Satya as he’s popularly known) has made pioneering contributions to advance edge computing, mobile computing, distributed systems, and the Internet of Things (IoT) research focused on performance, scalability, availability, and trust challenges in computing systems.

We examine the present and future of distributed edge-cloud architectures in an exclusive discussion with Satya; David Linthicum, chief cloud strategy officer, Deloitte Consulting LLP; and Myke Miller, dean, Deloitte Cloud Institute. Diana Kearns-Manolatos from the Deloitte Center for Integrated Research moderated this discussion. But first, a quick foundation on edge computing and five key insights from the discussion.

See the rest of the article at Deloitte Insights.

Source: https://www.internetexchangemap.com/

In the 1980s, there were only a handful of places in the public internet where internet service providers could connect with the public internet to pass data between each other and with their users. These places, known as internet exchange points, were located at the intersections of major fiber optic routes . As shown in the 2021 map above, the number of these locations has grown but remain centralized in areas of high population and fiber density. In addition, carriers have built private peering points to directly deliver traffic to one another. I’ll call internet exchange and private peering points collectively interexchange points (IXP). In a highly centralized cloud computing world, decisions on where to place IXPs were driven by economies of scale, fiber access, and the locations of the large cloud service providers. The time it took for a packet to move from one end of the network to another, i.e., end-to-end latency, was secondary. Because of this, latency-sensitive edge-native applications will not be able to deliver acceptable user experiences on today’s networks.

Client-Cloudlet Path

In edge computing, there are many scenarios where users and edge computing nodes (aka cloudlets) need to interact even though they may be connected to different carrier networks. For example, data from a multiplayer game cloudlet will pass through one or more IXPs to reach players on different networks. If the IXP is in Phoenix when the players and cloudlet are in Pittsburgh, end-to-end latency can grow from 10s to 100s of milliseconds – rendering that game unplayable. Distant IXPs can be catastrophic to many edge-native applications.

In early 2020, the Open Edge Computing (OEC) Initiative established a workstream to investigate this IXP placement challenge. Using the Carnegie Mellon University Living Edge Lab and the Interdigital AdvantEDGE Mobile Edge Emulation Platform, we evaluated the impacts of different IXP locations on the performance of the OpenRTiST edge-native application. Some results of this work are shown in the diagram at left.

The work of this workstream is outlined in the OEC Whitepaper, “How Close to the Edge? -- Edge Computing and Carrier Interexchange”.

The business success of edge computing depends on addressing limitations imposed by the industry’s legacy approach to carrier interconnect. These limitations render many edge-native applications unusable in the many scenarios.   Our results show that viable edge computing requires:

• Regardless of IXP location, edge computing “cloudlets” must be located in the same metro/region as the application users.
• Once metro cloudlets are deployed, IXPs must be established within the metro area and networks engineered to prevent user to cloudlet data paths outside of the metro.
• Within the metro area, the marginal performance benefit to moving IXPs closer to the user (e.g., to the cell tower) is small and may not justify cost.
• Metro third-party neutral host IXPs will provide equivalent performance to direct carrier-to-carrier IXPs with potentially lower complexity.
• While application performance is the main criteria for IXP placement decisions, other requirements like lawful intercept and data geofencing also need to be considered.

Given long planning and implementation cycles, we strongly recommend that carriers start immediately to enable low-latency edge computing by deploying metro based IXPs with other carriers.

For more information on this work, see the OEC Whitepaper, “How Close to the Edge? -- Edge Computing and Carrier Interexchange” or drop us a note at info@openedgecomputing.org. The inspiration for this work was the Tomasz Gerszberg LinkedIn blog from December 2019. A special thank you to InterDigital, Vodafone and VaporIO for their support of this workstream.

This research was supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001117C0051 and by the National Science Foundation (NSF) under grant number CNS-1518865 and the NSF Graduate Research Fellowship under grant numbers DGE1252522 and DGE1745016. Additional support was provided by Intel, Vodafone, Deutsche Telekom, Crown Castle, InterDigital, Seagate, Microsoft, VMware and the Conklin Kistler family fund. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the view(s) of their employers or the above funding sources.

References

1.     M. Satyanarayanan. “The Emergence of Edge Computing”. In: Computer 50.1 (2017), pp. 30–39.

2.     The Open Edge Computing Initiative. https://openedgecomputing.org/

3.     M. Satyanarayanan et al. “Cloudlets: at the leading edge of mobile-cloud convergence”. In: 6th International Conference on Mobile Computing, Applications and Services. 2014, pp. 1–9.

4.     Tomasz Gerszberg. Shared Edge Experience. https://www.linkedin.com/pulse/shared-edge-experience-tomasz-gerszberg

5.     M. Satyanarayanan et al. “The Seminal Role of Edge-Native Applications”. In: 2019 IEEE International Conference on Edge Computing (EDGE). 2019, pp. 33–40.

6. Carnegie Mellon University. The Living Edge Lab. https://openedgecomputing. org/living-edge-lab/ and https://www.cmu.edu/scs/edgecomputing/index.html

7.     Michel Roy, Kevin Di Lallo, and Robert Gazda. AdvantEDGE: A Mobile Edge Emulation Platform (MEEP). https://github.com/InterDigitalInc/AdvantEDGE.

8.     S. George et al. “OpenRTiST: End-to-End Benchmarking for Edge Computing”. In: IEEE Pervasive Computing (2020), pp. 1–9. DOI: 10.1109/MPRV.2020.3028781.

IMAGE CREDIT: Pavan Trikutam -- Unsplashed

In the mid-1980s, Bell Labs conducted a human-factors study to understand how telephone users conceptualized the telephone network. Within the switching engineering division, we thought of the network as a complex collection of switching, transmission and operation systems interconnected by a web of copper and fiber “outside plant”. But our customers, who only used the network to make phone calls, had a much simpler mental model – two telephones attached to an opaque cloud. Those two phones were nebulously connected to each other by dialing digits or speaking to an operator who resided somewhere in that cloud. The network itself was unimportant to users and, in their minds, had little impact on the quality of their experience. And, even if they wanted to know more, information about the network was not available to them. But this infrastructure opaqueness was not a major problem because Bell Labs engineers understood and could design for user experience metrics like dial tone wait times and voice quality.

In today’s world of mobile device applications accessing cloud and edge computing resources over mobile networks, infrastructure opaqueness persists but is compounded by application opaqueness. Old-fashioned telephone calling applications were limited and well understood. Today’s mobile applications like gaming, video conferencing and video surveillances are unbounded in number and diversity. They rely on multi-generational, multi-operator networks connected to multiple cloud and edge computing service providers. Mobile network designers and mobile application developers are faced with significant gaps in their knowledge and information about each other’s world just as those 1980’s telephone users knew little about the telephone network.

But, the user’s quality of experience depends on assuring acceptable application performance in the face of opaque and diverse network and compute performance. We see this firsthand when our video streaming service begins to “spin” waiting for download over a congested network. Content Delivery Networks, the first widespread use of edge computing, were overlaid on existing networks to address the spin problem. Standards like MPEG-DASH and HLS emerged to detect and mitigate bandwidth limitations in video delivery applications. General purpose edge computing is enabling applications well beyond video delivery but new tools and approaches will be needed to enable application developers to design for diverse network and computing environments and for network operators to design for the many applications that their networks must support.

At the Living Edge Lab, we have been working on approaches that use benchmarking and simulation of edge-native applications in networks to understand the interplay between applications and their environments. This work starts from the premise that application experience, quantitatively measured at the user device, is key to understanding the value and limitations of the infrastructure that supports that application. Our work uses an instrumented version of the OpenRTIST edge-native application to collect key application experience metrics like framerate and end-to-end (E2E) latency in different network and computing environments. The first work, by Shilpa George, Thomas Eiszler, Roger Iyengar, Haithem Turki, Ziqiang Feng and Junjue Wang from Carnegie Mellon University and Padmanabhan Pillai from Intel Labs, measured the E2E application latency in a variety of physical edge and cloud-based environments and WiFi and LTE networks. As shown at left, this work found substantial differences in E2E latency depending on what infrastructure the application ran on.

The second work by me, Roger Iyengar and Michel Roy from InterDigital, extended the first work by running a series of simulations on an environment built around the AdvantEDGE network emulator. These simulations varied the location and characteristics of users, edge nodes and network interconnect points while the application ran. We also looked at the impacts of network handoff, cell type and mobility on application performance. As an example, the figure below shows the impact on OpenRTIST round trip time and framerate at different simulated carrier interconnect points.

So far, this work focuses on a single user and a single, specific application but we do have some key learnings.

• For applications with high levels of user interactivity, low latency networking to “backend” computing is critical. In many of our experiments, connection to cloud infrastructure outside of our metro area caused unacceptable E2E latency of >150ms.
• Carrier interconnect points also caused similar E2E latency challenges. Without local metro carrier interconnect, application sessions from an LTE mobile device to an edge node on local wired carrier connected through an interconnect point outside the metro area resulting in the same 150ms+ E2E latency.
• When the network path stays within the metro area, E2E latency begins to be driven as much by application compute latency as from network latency. This effect is, of course, highly application specific. But it suggests that attention to the CPU and GPU compute resources available from edge nodes is important for experience acceptability.

Going forward, we’ll focus on diversifying the number and types of applications under study to include multi-user interactive, IOT and edge analytics applications. We also plan to test in more real and simulated network environments to further validate and extend our learnings.

This work shows the value of using application benchmarking and system simulation to better understand what’s behind the network and computing curtain imposed by infrastructure opaqueness. These are not the only tools developers will need to design and characterize their applications but they can provide key insights into application acceptability in the face of widely varying environments.

For more information on our work in this area, see the following.

REFERENCES

1. S. George et al., "OpenRTiST: End-to-End Benchmarking for Edge Computing," in IEEE Pervasive Computing, vol. 19, no. 4, pp. 10-18, 1 Oct.-Dec. 2020, doi: 10.1109/MPRV.2020.3028781.
2. J. Blakley et al., “Simulating Edge Computing Environments to Optimize Application Experience”, Carnegie Mellon University Technical Report, CMU-CS-20-135, November 2020.