Accessories 2021: Low-latency Performance Technology – Adventures in Time and Space

Although technology that allows a musician in one location to perform with another somewhere else has been around for some time, interest in it is relatively new. The emergence of low-latency networked performance software from the university research departments that developed it is due in no small part to Covid-19. Musicians across the globe were suddenly unable to play with (or for) anyone who lived outside their household. Orchestras found themselves broken into a hundred pieces, decades-long chamber partnerships were cast asunder and students were prevented from having the kinds of in-person creative exchanges that define the conservatoire learning process.

Zoom, Skype and others like them quickly became a lifeline, but they have their limitations. As anyone who has attempted to use such platforms for music making or instrumental teaching will know, one of these is the delay between the sound being produced and its arrival in the ears of the person at the other end of the line. Eighteen months after the first Covid cases were detected, the majority of people have resigned themselves to the slight but crucial delay that tests the resolve of any teacher (or student) and makes real-time synchronised playing all but impossible. (And there are other problems besides – many platforms will intervene to ‘fix’ a decrescendo by boosting the sound levels, for example.)

‘I was categorically opposed to students suddenly using backing tracks, and things like virtual choirs and ensembles,’ says Ian Howell, director of the Voice and Sound Analysis Laboratory at the New England Conservatory (NEC). ‘If we want to become virtual performers then let’s do that properly, but that isn’t what people were trying to achieve.’ Like all further education institutions on the US east coast, the NEC was forced to close its doors to staff and students in March last year. As prospects of a speedy return to normality began to recede, Howell says, there was a growing sense of fatalism among some of the college’s staff and students. ‘I don’t want to sound too dramatic but we were looking at the music education industry and we had a bunch of people whose artistic traditions were defined by collaborating with people in real time. They were so overwhelmed that they were willing just to give that up.’ In response, Howell and his colleagues began to explore combinations of hardware and software that would allow a teacher and student, or a number of students, to play perfectly in time without leaving the safety of their homes.

While the NEC was tasked with building a system from the ground up, not all organisations entered the pandemic without low-latency technology in place. The Miami-based elite orchestral training academy New World Symphony (NWS) already had 15 years of experience in this area under its belt, thanks to its role in the development of LoLa, a low-latency networked performance transmission system created by the Conservatorio di Musica ‘Giuseppe Tartini’ in Trieste, Italy, and a consortium of Italian research institutions. Perhaps the gold standard in low-latency technology, LoLa allows performers in three different locations to exchange audio and video streams without a perceptible delay and is currently used by more than a hundred universities and music colleges worldwide.

As NWS director of new media and distance education Justin Trieger explains, however, LoLa has onerous hardware requirements that stretch into tens of thousands of pounds, even before the problem issue of network speed is taken into account. ‘Something common to much low-latency tech is the lack of data compression,’ he says. ‘So you need a connection with enough bandwidth and sufficiently little inherent latency to cope with streaming huge amounts of uncompressed audio and video.’ Such connections are not available from commercial internet providers, but only from private research networks with access to services such as Internet2, a specialised fibre-optic network restricted to US research institutions. Elsewhere, networks including DFN in Germany and Janet in the UK offer comparable connection speeds but are similarly off-limits to the general public (and institutions without the money to pay for them).

Even LoLa only requires this kind of ultra high-speed network in one direction, however – something that has allowed NWS to incorporate low-latency tech into a number of different projects. ‘Typically our uses of LoLa fall into a few categories,’ says Trieger. ‘There are group instruction contexts, like a sectional masterclass, where a guest artist would be connected by low-latency tech to a group of students at NWS. Another might be a living composer sitting in on a rehearsal of their music. It has also been extremely useful in our community engagement work, with kindergarten and school students around Miami.’ But while the pandemic has been behind growing interest in low-latency technology, it prevented NWS’s set-up from being used at all. ‘An Internet2 connection isn’t the kind of thing you can carry around with you,’ says Trieger. ‘So without access to our facility in Miami Beach we were unable to use LoLa in any of the projects where it usually features.’

In a 2017 interview, violinist Pinchas Zukerman described LoLa as ‘the saviour of the profession’ on account of its ability to transcend physical distance. But as the experience of NWS makes clear, although LoLa would have been enormously useful during an extended period of travel bans and social distancing, the system was not designed with a pandemic in mind. Despite its connections to east coast research institutions including Harvard and Tufts universities, NEC does not have access to a research-grade internet network and so did not have access to LoLa. As Howell explains, this may have been to their advantage: ‘I found that the people who knew the most about what low-latency tech could do were in academic posts and tended to use only the highest-quality software that relied on things like Internet2,’ he says. ‘The fact we weren’t using low-latency tech before the pandemic meant we weren’t tied into something that required access to a building we couldn’t get into. We took the view that we were starting from zero, so just finding out more about how low latency worked and how it could help us was the primary challenge.’

Despite their lack of experience, Howell and his colleagues published their initial findings just two weeks after the US east coast locked down. ‘It was important to work quickly,’ he says. ‘It was a time when administrators were having to make sweeping decisions that had a real impact on people’s lives but there was no knowledge there to back them up.’ In stark contrast to LoLa, low-latency software suitable for use with ordinary domestic internet connections tends to be free and open-source, which means that anyone can access it and adapt its inner workings to their needs. Customising the software is unlikely to appeal to any but the most technically minded, but at least exploring different solutions does not come with a large price tag attached. ‘Deciding between different forms of low-latency tech is about trading quality for speed,’ Howell explains. ‘We settled on a German program called SoundJack, which hit the right balance for us and meant we were able to restart workshop-type interactions like performance classes and masterclasses with guest teachers.’

Michael Dessen, a performer, composer and professor at the University of California, Irvine, has written a number of helpful articles that give detailed explanations of what the available low-latency software does and how it works (bit.ly/3a4QgPQ). He quotes research that suggests the practical latency limit in one direction for two people playing in tight synchronisation is 30 milliseconds, the same time it takes sound to travel around 9 metres through the air. The NEC’s success is evidence that it is perfectly possible to achieve latency below this level between two computers connected to ordinary network connections. But how?

Not all of the delay experienced by the people at either end of a connection depends on the program used or the speed of the internet connection. For that reason, as well as free software, musicians wanting to experiment themselves will also need one or two straightforward pieces of hardware. Before the sound data can be sent across the network, it must first be processed into ‘packets’, a job usually done by a computer’s built-in sound card. To reduce latency at this stage, Dessen suggests connecting an external audio interface instead. These can be picked up for around £100 and do a faster job than most sound cards of packaging up sound data and unpacking it in the other direction. Despite its retro feel, an Ethernet cable is another essential piece of kit due to the greater inherent latency of wireless internet connections.

Once these practical anti-delay precautions have been taken, the next step is choosing the right software. There is no shortage of options, and anyone seeking a more detailed explanation of the (often very technical) differences between them could do no better than to head to Dessen’s website. While NEC adopted SoundJack, Trieger suggests anyone new to the concept gives JackTrip a try. Other programs include Jamulus, Jamkazam, Quacktrip and Quaxtrip, though, as Trieger cautions, all may take a little getting used to. ‘Because they tend to run in the command line of an operating system and don’t have user-friendly graphical interfaces, these programs require a level of technical facility in the person using them,’ he explains. ‘It’s a fast-growing and fast-changing environment, though, so there’s every chance that may soon change.’

Until it does, do musicians unfamiliar with computer programming stand much of a chance? Howell thinks they do: ‘It took me a while to become proficient with the software but there are plenty of resources out there to help,’ he says. ‘Rolling out the technology across the school, I was supported by a very forward-thinking admin department which helped to assist the students in acquiring the bits and bobs they would need.’ In the case of many low-latency software packages, one such item is a Raspberry Pi – a small, inexpensive and relatively easily programmed device that was first created as a way of encouraging non-experts to explore computing. This little computer can be connected directly to a monitor or computer screen and allows users to run software designed specifically for the pared-down Raspberry Pi interface.

Together with a case and power supply, a Raspberry Pi costs around £50 in total, while an SD card – required to install software packages like JackTrip and SoundJack – costs less than £10. ‘I put together an army of graduate students,’ Howell continues. ‘They helped build the Raspberry Pis and get everyone set up using them.’ Those without a battalion of technical support may be reassured to know that Raspberry Pis with JackTrip software pre-loaded on to them have recently become available. Although this is as close to plug-and-play as any adventures in low-latency tech are likely to get for the moment, Trieger’s optimism about the future of the software applies here too. ‘A pre-loaded Raspberry Pi would have been unthinkable just a year ago,’ he says. ‘Things are really only moving in one direction.’

Most software a musician might be tempted to try for themselves works in a similar fashion: once everything is connected, users must register and create a server – or enter someone else’s. ‘A server is like a virtual room,’ Dessen writes. ‘Everyone connected to that server will hear each other in real time.’ If all has gone to plan, the hardware adjustments and the intervention of software like JackTrip and SoundJack to accelerate the transmission of sound data between users’ individual connections will have overcome enough of the system’s inherent latency to bring it below the 30-millisecond perceptibility threshold. Users will hear each other as if they are separated by a few metres, not hundreds of miles.

As parts of the world begin to emerge from the worst of the pandemic, it’s clear that some of the changes that Covid-19 brought about are here to stay. Perhaps chief among these is a growing sense that many activities once tied to a particular location – an office, for example, or a concert hall – can take place just about anywhere with an internet connection. As low-latency performance technology becomes easier to use, will it herald an exodus from the teaching studio and rehearsal room? ‘Even with data travelling at close to the speed of light, there are still limitations,’ Howell says. ‘The further you’re sending the information, the more consequential the distance becomes. I’ve used a low-latency connection from my office in Boston to someone in San Francisco; we both had high-quality fibre-optic internet and the latency was equivalent to standing around 15 metres apart. Anything beyond that and it starts to get a little trickier.’

Trieger agrees that the technology isn’t a magic bullet for problems of physical distance and geographical separation. ‘At this stage, I think everyone agrees that in-person interactions are always best,’ he says. ‘Of course, low-latency solutions are enormously beneficial in situations when people aren’t able to travel to a particular location, or when getting everyone in the room would be unfeasibly expensive. It helps to establish a sense of trust if everyone has met in person at least once beforehand, however.’ From an educational perspective, Howell adds that there can be no replacement for the buzz of a music college and the serendipitous meetings that happen there. ‘But if we can improve efficiency then we can improve the education students receive,’ he says. ‘You might be able to afford another private lesson if you aren’t having to get to them by train twice a month. Or maybe you can just get another couple of hours’ sleep that morning. Like anything that enables laziness, I think low-latency tech is here to stay.