Post by Luke Skarth-Hayley (2018 Cohort)
Link to Paper: https://dl.acm.org/citation.cfm?id=3356638
Before joining the Horizon CDT, I obtained a MSc Computer Science at the University of Nottingham. My MSc dissertation was nominated for the best dissertation of the year award. Alas, another, very good, dissertation received the prize. Still, my dissertation was good. I had unfinished business with my supervisor, Dr Julie Greensmith. We were going to publish a paper from it. Then, everything went a bit pear-shaped for a year.
A year later I joined Horizon. I met with Dr Greensmith again. We renewed our plans to get a paper out of my dissertation. Then I received an email about the Audio Mostly conference, “a conference on interaction with sound.” They were looking for papers and demos involving sonification and auditory interfaces. Perfect.
So, I should explain my MSc dissertation at this point!
During my MSc I took the Mixed Reality Technologies module. As part of the module I came to understand more about Human Computer Interaction (HCI). I also gained an interest in alternative interfaces and presentations for data. Adrian Hazzard’s work on locative soundtracks [3] was of particular inspiration. Thinking about sound and music, combined with alternative modes of presenting data, led me to sonification. Sonification is the quantifiable and reproducible mapping of data to sound. You put the same data in, you should get the same sound out. Think of it as the equal of making a graph: you expect the graph to look the same with the same data.
Some initial ideas I had were too complex to be achievable in a MSc dissertation. I intend to investigate those in future work. For the dissertation I built a sonification system for network state and intrusions. I designed a markup language for musical notation that would output MIDI messages in response to network state. Any music or audio software that supports MIDI can use these messages to generate musical sonifications.
My interest has evolved to working with data sources to trigger musical notes and phrases, to “compose” music from the interpreted data. The markup language enables a simple, static version of this. I have since found the idea of collaborating with pseudo-random systems that create reproducible outcomes from data to be an interesting approach for improvised performances of music and other creative works.
In the future I want to explore a general sonification interface that allows musicians and other creative artists to dynamically compose their works with any time-series or streaming data source. There is much interesting work in communicating the “invisible” effects of data and algorithms via creative channels. This is particularly relevant to the Horizon themes of “My Life in Data” and “Creating Our Lives in Data”.
There were several avenues by which I could have submitted my work to Audio Mostly. There were calls for papers, music, demos, workshops and posters. In the end I worked with Dr Greensmith to submit a paper and a demo paper on two different aspects of my research. The paper discussed the system, with a focus on the network analysis aspects. This included the potential to work with more advanced intrusion detection software. The demo paper took the form of an extended abstract discussing the sonification and musical composition aspects, and a demo presentation of the software at the conference. The demo paper also discussed the future possibilities of co-authoring and collaboration with sonification systems for musical composition.
The primary paper was co-authored by Dr Greensmith and me. The demo paper I wrote by myself after we wrote the full paper. Dr Greensmith’s input and experience were vital in helping me navigate my first academic conference submissions. We drank much coffee, hammered out draft after draft in LaTeX, and submitted in time for the deadline.
And then, there was waiting. For months. I learned more about the peer review process. People told me about reviewer number two. But my advice is to realise peer review is, as the name suggests, from other academics. Your peers are committing uncompensated time to examine your work. It takes time. Deadlines and dates will slip. This is fine. And if your work is rejected, do not take it personally. Take a break from it if you need, let the pain lessen. Then see what guidance and useful comments are in the feedback for future revisions or work.
When the reviewer comments came back, they were positive. Even reviewer number two was kind! Well. For one of the papers. The demo paper. Unfortunately, the longer paper was rejected. But there was some great feedback and future improvements to be made.
But the demo paper was accepted! There were a couple of minor edits recommended. There was also some pleasing interest in the compositional opportunities of working with a sonification system. There were also suggestions of papers to read to help round out the supporting theory. These were very helpful, and I made the appropriate edits to ensure the paper was “camera ready”. Wrestling with LaTeX formatting and all the other parts of getting a paper ready for publishing was… interesting! Make sure you familiarise yourself with any guidance from the conference or journal to which you submit. Ask if you are unsure. You will still make mistakes, and that’s okay, but be aware it can be tricky.
Then I had to make sure my demo software was ready and stable. It had been some time. External software libraries had updated. The Python programming language had updated. Everything had updated. And some of the music software I used to generate audio was discontinued. Fortunately, my software still worked. But I did have to re-compose the musical element somewhat. Be prepared. Especially when it comes to demoing software. Give yourself plenty of time to test it and ensure it functions as required.
A poster requires some visual competency, but then you can stand by it and hope someone takes an interest. A paper is a bit more nerve-wracking, even after you have had it accepted. You must stand in front of a room full of academics and present it. Then you get questioned.
A demo is quite different to either of them. Demos are great because something exists: a piece of software, hardware, an artefact. Something tangible. You can show the thing to explain the thing. This is a great opportunity. But there is also the potential for live demos failing, for software issues, hardware failures, or poor reception. Check your requirements, make sure you have what you need. Make sure everything works. Hope that people like it. In the case of sound-related work, make sure you have headphones.
As for my demo, it was generally well-received. There were some obvious caveats around limitations in the system, and the need for longitudinal user acceptance testing and in-the-wild deployments, along with potential fatigue caused by auditory displays in long-term use. However, that was mitigated by the relative ease with which users might edit the markup and change the sounds in the Digital Audio Workstation (DAW) used to generate the actual audio output.
There were also some interesting discussions around the idea of blending live coding [2] composition approaches with sonification – not soley of network data – to explore time-series data as a compositional element, and also to express aspects of data through composition, or being able to express rapid or long-term data at an “human-scale” listenability. I hope to have further contact with interested parties to explore this aspect further.
Explaining Siren Songs, Wearing A Familiar Jumper
I also spoke to some interested industry members about the use of sonification and auditory display for technicians during musical and audio-visual performances, and how they might be able to monitor system and device states “at-a-listen” alongside monitoring the audio of the performances themselves, integrating system state into an audio-focused workflow. Again, something to discuss with the interested parties in the future.
But what does this demo paper have to do with my PhD? One could argue not much! But my MSc was what inspired me to join Horizon and the MRL. Learning about Human Computer Interaction as an academic discipline. Encountering sonification and other ways of interacting with computing and data. These inspired me. These are things I seek to investigate as part of my research.
It has also been useful to revisit my previous work as I find the focus and direction for my PhD research. I am interested in how we interact with data, including our personal data, and the algorithms and systems that surround it. I am keen to explore implicit interfaces and interactions. How we engage with data and systems in ways that are less obvious, more obfuscated and hidden. With BBC R&D as my industry partner I am looking at how our engagement with media experiences generates data. I plan to explore how that data might be fed back into the media to enable responsive content. This effort seeks to explore the space between “lean back” television and “lean forward” interactivity. As part of this, implicit interactions with time-series data sources and other external data that affect media experiences in reproducible ways – and can be “composed with” by content creators – akin to the sonification process in the Siren Songs demo paper, is something I am keen to explore.
In practical terms it was also useful for me to understand how to format and submit work for a conference without the worry of it being “live” work from my PhD. There is value in comprehending a process ahead of time, without staking all your efforts in one place. Then, if you get rejected, if you fail, it impacts you less. This is a good tactic for calibrating your expectations. Particularly when you return to work you did before, and you are able to make a cold, hard assessment about it. My dissertation work is good, but I can see so many ways it can be better in future. I have already developed my thinking since I did it, and that is encouraging.
There are also good lessons in submitting work when it comes to time management. I now know what it takes to rewrite work for a specific audience or conference. I know the processes for using specific LaTeX templates that conferences ask for. I know how EasyChair, a conference management system, works as someone submitting a paper. It is not as easy as the name suggests. I recommend signing up for an account and understanding any submission system well ahead of your first submission. For me, time management includes planning ahead. Familiarising myself with a process or system ahead of time. I do recommend trying the same, where you can, to ease the cognitive burden for yourself.
Returning to my MSc work did prompt some interesting thoughts. For a start, with time one can see where to make improvements in writing up work, and very much so in writing software. There is also a pleasure in realising where your thinking and research have changed and evolved. This demo paper also aided me in confirming interests that are relevant to my PhD and the broad areas I would like to explore in my research.
To take one example: I still have an interest in sound and music, both as research topics and for personal projects. Whilst my focus is on my PhD, I am keen to return to sound either as part of my research, or as a side project. I am fascinated by the work of Algomech and the Algorave scene [1,4,5], and live coding [2]. There is an interesting niche to mine in collaborating with the sonification of data to express that data as part of a creative performance. I plan to experiment with this myself, or perhaps others may be inspired to do so.
Beyond this, it was useful to attend Audio Mostly 2019 and gain insight into the current state of the art in audio. To see research on game engines and games, and spatial audio for immersive entertainment. These are already prompting ideas for my PhD research.
Finally, it was great to engage with a very positive research community. Audio Mostly is a great conference to attend. I spoke to many interesting researchers and have made some useful contacts. I hope to collaborate with these on future work. I am also keen to present audio-related work from my PhD at the conference. It looks to be an excellent arena to do so and get helpful, constructive feedback from experts in the field.
