Abstract: A technique is introduced for generating novel signal processing systems grounded in analog electronic circuits, called model bending. By applying the ideas behind circuit bending to models of nonlinear analog circuits it is possible to create novel nonlinear signal processors which mimic the behavior of analog electronics, but which are not possible to implement in the analog realm. The history of both circuit bending and circuit modeling is discussed, as well as a theoretical basis for how these approaches can complement each other. Potential pitfalls to the practical application of model bending are highlighted and suggested solutions to those problems are provided, with examples.

Abstract: Recent research in deep learning has shown that neural networks can learn differential equations governing dynamical systems. In this paper, we adapt this concept to Virtual Analog (VA) modeling to learn the ordinary differential equations (ODEs) governing the first-order and the second-order diode clipper. The proposed models achieve performance comparable to state-of-the-art recurrent neural networks (RNNs) albeit using fewer parameters. We show that this approach does not require oversampling and allows to increase the sampling rate after the training has completed, which results in increased accuracy. Using a sophisticated numerical solver allows to increase the accuracy at the cost of slower processing. ODEs learned this way do not require closed forms but are still physically interpretable.

Abstract: In this paper we present a virtual analog model of the voltagecontrolled filter used in the EDP Wasp synthesizer. This circuit is an interesting case study for virtual analog modeling due to its characteristic nonlinear and highly dynamic behavior which can be attributed to its unusual design. The Wasp filter consists of a state variable filter topology implemented using operational transconductance amplifiers (OTAs) as the cutoff-control elements and CMOS inverters in lieu of operational amplifiers, all powered by a unipolar power supply. In order to accurately model the behavior of the circuit we propose extended models for its nonlinear components, focusing particularly on the OTAs. The proposed component models are used inside a white-box circuit modeling framework to create a digital simulation of the filter which retains the interesting characteristics of the original device.

Abstract: We have analyzed the Sound Effects Filter from the one-of-a-kind RCA Mark II sound synthesizer and modeled it as a Wave Digital Filter using the Faust language, to make this once exclusive device widely available. By studying the original schematics and measurements of the device, we discovered several circuit modifications. Building on these, we proposed a number of extensions to the circuit which increase its usefulness in music production.

Abstract: In addition to tone, pitch and rhythm, dynamics is one of the expressive dimensions of the performance of a music piece that has received limited attention. While the usage of dynamics may vary from artist to artist, and also from performance to performance, a systematic methodology to automatically identify the dynamics of a performance in terms of musically meaningful terms like forte, piano may offer valuable feedback in the context of music education and in particular in singing. To this end, we have manually annotated the dynamic markings of commercial recordings of popular rock and pop songs from the Smule Vocal Balanced (SVB) dataset which will be used as reference data. Then as a first step for our research goal, we propose a method to derive and compare singing voice loudness curves in polyphonic mixtures. Towards measuring the similarity and variation of dynamics, we compare the dynamics curves of the SVB renditions with the one derived from the original songs. We perform the same comparison using professionally produced renditions from a karaoke website. We relate high values of Spearman correlation coefficient found in some select student renditions and the professional renditions with accurate dynamics.

Abstract: Employing nonlinear functions in audio DSP algorithms requires attention as they generally introduce aliasing. Among others, antiderivative antialiasing proved to be an effective method for static nonlinearities and gave rise to a number of variants, including our AA-IIR method. In this paper we introduce an improvement to AA-IIR that makes it suitable for use in stateful systems. Indeed, employing standard antiderivative antialiasing techniques in such systems alters their frequency response and may cause stability issues. Our method consists in cascading a digital filter after the AA-IIR block in order to fully compensate for unwanted delay and frequency-dependent effects. We study the conditions for such a digital filter to be stable itself and evaluate the method by applying it to the diode clipper circuit.

Abstract: An approach to designing dynamical systems with a three-dimensional state space is described that can be used to build a variety of non-periodic oscillators. The state space is taken to be a 3sphere, which is identified with the manifold of unit quaternions. Any such system can be described as a quaternion-valued ordinary differential equation, which is digitally realized using an approximation as a finite difference equation. Two examples are shown. Compared to previous applications of dynamical systems used to generate audio samples, the approach described here offers a wide choice of specific flows which can neither diverge nor approach a stable limit point.

Abstract: Iterative phase formulations allow for the generalization of many oscillatory sound synthesis methods from circles to general parametrizable loops, with or without explicit geometric contexts. This paper describes this approach, leading to the ability to perform modulation, feedback and chaotic oscillations over deformed circles that can include ill-behaved geometries, while allowing modulations or feedback to be deformed as well.

Abstract: Continuous State Markovian Spectral Modeling is a novel approach for parametric synthesis of spectral modeling parameters, based on the sines plus noise paradigm. The method aims specifically at capturing shimmer and jitter - micro-fluctuations of the partials’ frequency and amplitude trajectories, which are essential for the timbre of musical instruments. It allows for parametric control over the timbral qualities, while removing the need for the more computationally expensive and restrictive process of the discrete state space modeling method. A qualitative comparison between an original violin sound and a re-synthesis shows the ability of the algorithm to reproduce the micro-fluctuations, considering their stochastic and spectral properties.

Abstract: In this paper we investigate into perceptual properties of StyleWaveGAN, a drum synthesis method proposed in a previous publication. For both, the sound quality as well as the control precision StyleWaveGAN has been shown to deliver state of the art performance for quantitative metrics (FAD and MSE of the control parameters). The present paper aims to provide insight into the perceptual relevance of these results. Accordingly, we performed a subjective evaluation of the sound quality as well as a subjective evaluation of the precision of the control using timbre descriptors from the AudioCommons toolbox. We evaluate the sound quality with mean opinion score and make measurements of psychophysical response to the variations of the control. By means of the perceptual tests, we demonstrate that StyleWaveGAN produces better sound quality than state-of-the-art model DrumGAN and that the mean control error is lower than the absolute threshold of perception at every point of measurement used in the experiment.

Abstract: In this paper, we approach the problem of atomic decomposition of audio at the symbolic level of atom parameters through the lens of hyperdimensional computing (HDC) – a non-traditional computing paradigm. Existing atomic decomposition algorithms often operate using waveforms from a redundant dictionary of atoms causing them to become increasingly memory/computationally intensive as the signal length grows and/or the atoms become more complicated. We systematically build an atom encoding using vector function architecture (VFA), a field of HDC. We train a neural network encoder on synthetic audio signals to generate these encodings and observe that the network can generalize to real recordings. This system, we call Hyperdimensional Atomic Decomposition (HD-AD), avoids time-domain correlations all together. Because HD-AD scales with the sparsity of the signal, rather than its length in time, atomic decompositions are often produced much faster than real-time.

Abstract: Nonlinear systems play an important role in musical signal processing, but their digital implementation suffers from the occurrence of aliasing distortion. Consequently, various aliasing reduction methods have been proposed in the literature. In this work, a novel approach is examined that uses samples of a signal’s derivative in addition to the signal’s samples themselves. This allows some aliasing reduction, but is usually insufficient on its own. However, it can elegantly and fruitfully be combined with antiderivative antialiasing to obtain an effective method. Unfortunately, it still compares unfavorably to oversampled antiderivative antialiasing. It may therefore be regarded as a negative result, but it may hopefully still form a basis for further developments.

Abstract: This paper proposes a low-latency quasi-linear-phase octave graphic equalizer. The structure is derived from a recent linearphase graphic equalizer based on interpolated finite impulse response (IFIR) filters. The proposed system reduces the total latency of the previous equalizer by implementing a hybrid structure. An infinite impulse response (IIR) shelving filter is used in the structure to implement the first band of the equalizer, whereas the rest of the band filters are realized with the linear-phase FIR structure. The introduction of the IIR filter causes a nonlinear phase response in the low frequencies, but the total latency is reduced by 50% in comparison to the linear-phase equalizer. The proposed graphic equalizer is useful in real-time audio processing, where only little latency is tolerated.

Abstract: We show a new sufficient criterion for time-varying digital filter stability: that the matrix norm of the product of state matrices over a certain finite number of time steps is bounded by 1. This extends Laroche’s Criterion 1, which only considered one time step, while hinting at extensions to two time steps. Further extending these results, we also show that there is no intrinsic requirement that fil-ter coefficients be frozen over any time scale, and extend to any dimension a helpful theorem that allows us to avoid explicitly per-forming eigen- or singular value decompositions in studying the matrix norm. We give a number of case studies on filters known to be time-varying stable, that cannot be proven time-varying stable with the original criterion, where the new criterion succeeds.

Abstract: Hard sync is a feature appearing in many analog synthesizers: it consists in retriggering a slave oscillator, regardless of its phase, every time a master oscillator completes its cycle. If this process is naïvely implemented digitally, it is subject to aliasing. While for sawtooth, square, and triangle waves several effective antialiasing methods have been developed, the literature is sparser concerning sine hard sync, arguably because discontinuities of infinite order are introduced which are more difficult to handle. In this paper, we introduce a new antialiasing algorithm for sine hard sync which is obtained by filtering the hard-synced sine with a FIR lowpass kernel, as opposed to existing methods based on the windowed sinc function. We show that our method yields lower computational cost and better aliasing reduction.

Abstract: This paper introduces the possibilities of optimizing neural network convolutional layers for modeling nonlinear audio systems and effects. Enhanced methods for real-time dilated convolutions are presented to achieve faster signal processing times than in previous work. Due to the improved implementation of convolutional layers, a significant decrease in computational requirements was observed and validated on different configurations of single layers with dilated convolutions and WaveNet-style feedforward neural network models. In most cases, equivalent signal processing times were achieved to those using recurrent neural networks with Long Short-Term Memory units and Gated Recurrent Units, which are considered state-of-the-art in the field of black-box virtual analog modeling.

Abstract: The motion of a bowed string is a typical nonlinear phenomenon resulting from a friction force via interaction with the bow. The system can be described using suitable differential equations. Implicit numerical discretisation methods are known to yield energy consistent algorithms, essential to ensure stability of the timestepping schemes. However, reliance on iterative nonlinear root finders carries significant implementation issues. This paper explores a method recently developed which allows nonlinear systems of ordinary differential equations to be solved non-iteratively. Case studies of a mass-spring system and an ideal string coupled with a bow are investigated. Finally, a stiff string with loss is also considered. Combining semi-discretisation and a modal approach results in an algorithm yielding faster than real-time simulation of typical musical strings.

Abstract: Digital musical instruments based on physical modelling have gained increased popularity over the past years. This is partly due to recent advances in computational power, which allow for their real-time implementation. One of the great potentials for digital musical instruments based on physical models, is that one can go beyond what is physically possible and change properties of the instruments which are static in real life. This paper presents a real-time implementation of the dynamic stiff string using finitedifference time-domain (FDTD) methods. The defining parameters of the string can be varied in real time and change the underlying grid that these methods rely on based on the recently developed dynamic grid method. For most settings, parameter changes are nearly instantaneous and do not cause noticeable artefacts due to changes in the grid. A reliable way to prevent artefacts for all settings is under development.

Abstract: Discrete-time modeling of acoustic, mechanical and electrical systems is a prominent topic in the musical signal processing literature. Such models are mostly derived by discretizing a mathematical model, given in terms of ordinary or partial differential equations, using established techniques. Recent work has applied the techniques of machine-learning to construct such models automatically from data for the case of systems which have lumped states described by scalar values, such as electrical circuits. In this work, we examine how similar techniques are able to construct models of systems which have spatially distributed rather than lumped states. We describe several novel recurrent neural network structures, and show how they can be thought of as an extension of modal techniques. As a proof of concept, we generate synthetic data for three physical systems and show that the proposed network structures can be trained with this data to reproduce the behavior of these systems.

Abstract: Sound synthesis of a pipe coupled with a reed requires to finely tune the physical parameters of the underlying model. Although the pipe geometry is often well known, the 1 degree of freedom reed model’s parameters are effective coefficients (mass, section, etc) and are difficult to assess. Studies of this coupled system have essentially focused on models without the reed induced flow, and have exhibited two dimensionless parameters γ and ζ, which respectively describe the ratio between feeding pressure and closing reed pressure, and a dimensionless opening of the reed at rest. Including the reed flow in the model, then performing a scaling of the equations, leads to a new third dimensionless quantity that we will call κ. Varying the reed frequency with constant (γ, ζ, κ) on different pipe dimensions shows a certain stability of the model once put under this form. Using a real-time sound synthesis tool, the parameter space (γ, ζ, κ) is explored while the damping of the reed is also varied.

Abstract: Herein, we demonstrate the use of neural nets towards simulating multiport nonlinearities inside a wave digital filter. We introduce a resolved wave definition which allows us to extract features from a Kirchhoff domain dataset and train our neural networks directly in the wave domain. A hyperparameter search is performed to minimize error and runtime complexity. To illustrate the method, we model a tube amplifier circuit inspired by the preamplifier stage of the Fender Pro-Junior guitar amplifier. We analyze the performance of our neural nets models by comparing their distortion characteristics and transconductances. Our results suggest that activation function selection has a significant effect on the distortion characteristic created by the neural net.

Abstract: Sustained contact interactions like scraping and rolling produce a wide variety of sounds. Previous studies have explored ways to synthesize these sounds efficiently and intuitively but could not fully mimic the rich structure of real instances of these sounds. We present a novel source-filter model for realistic synthesis of scraping and rolling sounds with physically and perceptually relevant controllable parameters constrained by principles of mechanics. Key features of our model include non-linearities to constrain the contact force, naturalistic normal force variation for different motions, and a method for morphing impulse responses within a material to achieve location-dependence. Perceptual experiments show that the presented model is able to synthesize realistic scraping and rolling sounds while conveying physical information similar to that in recorded sounds.

Abstract: Automatic detection of musical patterns is an important task in the field of Music Information Retrieval due to its usage in multiple applications such as automatic music transcription, genre or instrument identification, music classification, and music recommendation. A significant sub-task in pattern detection is the realtime pattern detection in music due to its relevance in application domains such as the Internet of Musical Things. In this study, we present a method to identify the occurrence of known patterns in symbolic monophonic music streams in real-time. We introduce a matrix-based representation to denote musical notes using its pitch, pitch-bend, amplitude, and duration. We propose an algorithm based on an independent similarity index for each note attribute. We also introduce the Match Measure, which is a numerical value signifying the degree of the match between a pattern and a sequence of notes. We have tested the proposed algorithm against three datasets: a human recorded dataset, a synthetically designed dataset, and the JKUPDD dataset. Overall, a detection rate of 95% was achieved. The low computational load and minimal running time demonstrate the suitability of the method for real-world, real-time implementations on embedded systems.

Abstract: This paper presents two datasets of room impulse responses (RIRs) for navigable virtual acoustics. The first is a set of 240 mono and Ambisonic RIRs recorded at the Maison Symphonique, a symphonic concert hall in Montreal renowned for its great acoustic characteristics. The second is a set of 67 third-order Ambisonic RIRs which was recorded in the former planetarium of Montreal (currently known as the Centech), a space where the room acoustic includes an acoustic focal point where extreme reverberation times occur. The article first describes the two datasets and the methods that were used to capture them. A use case for these RIRs is then presented: an audio rendering of scene navigation using interpolation among RIRs.

Abstract: Real-time applications of Music Information Retrieval (MIR) have been gaining interest as of recently. However, as deep learning becomes more and more ubiquitous for music analysis tasks, several challenges and limitations need to be overcome to deliver accurate and quick real-time MIR systems. In addition, modern embedded computers offer great potential for compact systems that use MIR algorithms, such as digital musical instruments. However, embedded computing hardware is generally resource constrained, posing additional limitations. In this paper, we identify and discuss the challenges and limitations of embedded real-time MIR. Furthermore, we discuss potential solutions to these challenges, and demonstrate their validity by presenting an embedded real-time classifier of expressive acoustic guitar techniques. The classifier achieved 99.2% accuracy in distinguishing pitched and percussive techniques and a 99.1% average accuracy in distinguishing four distinct percussive techniques with a fifth class for pitched sounds. The full classification task is a considerably more complex learning problem, with our preliminary results reaching only 56.5% accuracy. The results were produced with an average latency of 30.7 ms.

Abstract: Flutter echo is a well-known acoustic phenomenon that occurs when sound waves bounce between two parallel reflective surfaces, creating a repetitive sound. In this work, we introduce a method to recreate flutter echo as an audio effect. The proposed algorithm is based on a feedback structure utilizing velvet noise that aims to synthesize the fluttery components of a reference room impulse response presenting flutter echo. Among these, the repetition time defines the length of the delay line in a feedback filter. The specific spectral properties of the flutter are obtained with a bandpass attenuation filter and a ripple filter, which enhances the harmonic behavior of the sound. Additional temporal shaping of a velvet-noise filter, which processes the output of the feedback loop, is performed based on the properties of the reference flutter. The comparison between synthetic and measured flutter echo impulse responses shows good agreement in terms of both the repetition time and reverberation time values.

Abstract: This paper proposes dark velvet noise (DVN) as an extension of the original velvet noise with a lowpass spectrum. The lowpass spectrum is achieved by allowing each pulse in the sparse sequence to have a randomized pulse width. The cutoff frequency is controlled by the density of the sequence. The modulated pulse-width can be implemented efficiently utilizing a discrete set of recursive running-sum filters, one for each unique pulse width. DVN may be used in reverberation algorithms. Typical room reverberation has a frequency-dependent decay, where the high frequencies decay faster than the low ones. A similar effect is achieved by lowering the density and increasing the pulse-width of DVN in time, thereby making the DVN suitable for artificial reverberation.

Abstract: Computer simulations of room acoustics suffer from an efficiency vs accuracy trade-off, with highly accurate wave-based models being highly computationally expensive, and delay-network-based models lacking in physical accuracy. The Scattering Delay Network (SDN) is a highly efficient recursive structure that renders first order reflections exactly while approximating higher order ones. With the purpose of improving the accuracy of SDNs, in this paper, several variations on SDNs are investigated, including appropriate node placement for exact modeling of higher order reflections, redesigned scattering matrices for physically-motivated scattering, and pruned network connections for reduced computational complexity. The results of these variations are compared to state-of-the-art geometric acoustic models for different shoebox room simulations. Objective measures (Normalized Echo Densities (NEDs) and Energy Decay Curves (EDCs)) showed a close match between the proposed methods and the references. A formal listening test was carried out to evaluate differences in perceived naturalness of the synthesized Room Impulse Responses. Results show that increasing SDNs’ order and adding directional scattering in a fully-connected network improves perceived naturalness, and higher-order pruned networks give similar performance at a much lower computational cost.

Abstract: The cross-correlation of multichannel reverberation generated using interleaved velvet noise is studied. The interleaved velvetnoise reverberator was proposed recently for synthesizing the late reverb of an acoustic space. In addition to providing a computationally efficient structure and a perceptually smooth response, the interleaving method allows combining its independent branch outputs in different permutations, which are all equally smooth and flutter-free. For instance, a four-branch output can be combined in 4! or 24 ways. Additionally, each branch output set is mixed orthogonally, which increases the number of permutations from M! to M 2!, since sign inversions are taken along. Using specific matrices for this operation, which change the sign of velvet-noise sequences, decreases the correlation of some of the combinations. This paper shows that many selections of permutations offer a set of well decorrelated output channels, which produce a diffuse and colorless sound field, which is validated with spatial variation. The results of this work can be applied in the design of computationally efficient multichannel reverberators.

Abstract: In the development of algorithms for sound source detection, identification and localization, having the possibility to generate datasets in a flexible and fast way is of utmost importance. However, most of the available acoustic simulators used for this purpose target indoor applications, and their usefulness is limited when it comes to outdoor environments such as that of a road, involving fast moving sources and long distances travelled by the sound waves. In this paper we present an acoustic propagation simulator specifically designed for road scenarios. In particular, the proposed Python software package enables to simulate the observed sound resulting from a source moving on an arbitrary trajectory relative to the observer, exploiting variable length delay lines to implement sound propagation and Doppler effect. An acoustic model of the road reflection and air absorption properties has been designed and implemented using digital FIR filters. The architecture of the proposed software is flexible and open to extensions, allowing the package to kick-start the implementation of further outdoor acoustic simulation scenarios.

Abstract: The process of creating drum sounds has seen significant evolution in the past decades. The development of analogue drum synthesizers, such as the TR-808, and modern sound design tools in Digital Audio Workstations led to a variety of drum timbres that defined entire musical genres. Recently, drum synthesis research has been revived with a new focus on training generative neural networks to create drum sounds. Different interfaces have previously been proposed to control the generative process, from low-level latent space navigation to high-level semantic feature parameterisation, but no comprehensive analysis has been presented to evaluate how each approach relates to the creative process. We aim to evaluate how different interfaces support creative control over drum generation by conducting a user study based on the Creative Support Index. We experiment with both a supervised method that decodes semantic latent space directions and an unsupervised Closed-Form Factorization approach from computer vision literature to parameterise the generation process and demonstrate that the latter is the preferred means to control a drum synthesizer based on the StyleGAN2 network architecture.

Abstract: Recent neural-based synthesis models have achieved impressive results for musical instrument sound generation. In particular, the Differentiable Digital Signal Processing (DDSP) framework enables the usage of spectral modeling analysis and synthesis techniques in fully differentiable architectures. Yet currently, it has only been used for modeling monophonic instruments. Leveraging the interpretability and modularity of this framework, the present work introduces a polyphonic differentiable model for piano sound synthesis, conditioned on Musical Instrument Digital Interface (MIDI) inputs. The model architecture is motivated by high-level acoustic modeling knowledge of the instrument which, in tandem with the sound structure priors inherent to the DDSP components, makes for a lightweight, interpretable and realistic sounding piano model. The proposed model has been evaluated in a listening test, demonstrating improved sound quality compared to a benchmark neural-based piano model, with significantly less parameters and even with reduced training data. The same listening test indicates that physical-modeling-based models still achieve better quality, but the differentiability of our lightened approach encourages its usage in other musical tasks dealing with polyphonic audio and symbolic data.

Abstract: This paper introduces a novel data-driven strategy for synthesizing gramophone noise audio textures. A diffusion probabilistic model is applied to generate highly realistic quasiperiodic noises. The proposed model is designed to generate samples of length equal to one disk revolution, but a method to generate plausible periodic variations between revolutions is also proposed. A guided approach is also applied as a conditioning method, where an audio signal generated with manually-tuned signal processing is refined via reverse diffusion to improve realism. The method has been evaluated in a subjective listening test, in which the participants were often unable to recognize the synthesized signals from the real ones. The synthetic noises produced with the best proposed unconditional method are statistically indistinguishable from real noise recordings. This work shows the potential of diffusion models for highly realistic audio synthesis tasks.

Abstract: In this paper, we propose a new processor capable of directly changing the microdynamics of an audio signal primarily via a single dedicated user-facing parameter. The novelty of our processor is that it has built into it a measure of relative level, a short-term signal strength measurement which is robust to changes in signal macrodynamics. Consequent dynamic range processing is signal level-independent in its nature, and attempts to directly alter its observed relative level measurements. The inclusion of such a meter within our proposed processor also gives rise to a natural solution to the dynamics matching problem, where we attempt to transfer the microdynamic characteristics of one audio recording to another by means of estimating appropriate settings for the processor. We suggest a means of providing a reasonable initial guess for processor settings, followed by an efficient iterative algorithm to refine upon our estimates. Additionally, we implement the processor as a differentiable recurrent layer and show its effectiveness when wrapped around a gradient descent optimizer within a deep learning framework. Moreover, we illustrate that the proposed processor has more favorable gradient characteristics relative to a conventional dynamic range compressor. Throughout, we consider extensions of the processor, matching algorithm, and differentiable implementation for the multiband case.

Abstract: Recent advancements in deep learning have shown great potential for audio applications, improving the accuracy of previous solutions for tasks such as music transcription, beat detection, and real-time audio processing. In addition, the availability of increasingly powerful embedded computers has led many deep learning framework developers to devise software optimized to run pretrained models in resource-constrained contexts. As a result, the use of deep learning on embedded devices and audio plugins has become more widespread. However, confusion has been rising around deep learning inference engines, regarding which of these can run in real-time and which are less resource-hungry. In this paper, we present a comparison of four available deep learning inference engines for real-time audio classification on the CPU of an embedded single-board computer: TensorFlow Lite, TorchScript, ONNX Runtime, and RTNeural. Results show that all inference engines can execute neural network models in real-time with appropriate code practices, but execution time varies between engines and models. Most importantly, we found that most of the less-specialized engines offer great flexibility and can be used effectively for real-time audio classification, with slightly better results than a real-time-specific approach. In contrast, more specialized solutions can offer a lightweight and minimalist alternative where less flexibility is needed.

Abstract: Many electronic music (EM) genres are composed through the activation of short audio recordings of instruments designed for seamless repetition—or loops. In this work, loops of key structural groups such as bass, percussive or melodic elements are labelled by the role they occupy in a piece of music through the task of automatic instrumentation role classification (AIRC). Such labels assist EM producers in the identification of compatible loops in large unstructured audio databases. While human annotation is often laborious, automatic classification allows for fast and scalable generation of these labels. We experiment with several deeplearning architectures and propose a data augmentation method for improving multi-label representation to balance classes within the Freesound Loop Dataset. To improve the classification accuracy of the architectures, we also evaluate different pooling operations. Results indicate that in combination with the data augmentation and pooling strategies, the proposed system achieves state-of-theart performance for AIRC. Additionally, we demonstrate how our proposed AIRC method is useful for analysing the structure of EM compositions through loop activation transcription.

Abstract: The integration of additional side information to improve music source separation has been investigated numerous times, e.g., by adding features to the input or by adding learning targets in a multi-task learning scenario. These approaches, however, require additional annotations such as musical scores, instrument labels, etc. in training and possibly during inference. The available datasets for source separation do not usually provide these additional annotations. In this work, we explore transfer learning strategies to incorporate VGGish features with a state-of-the-art source separation model; VGGish features are known to be a very condensed representation of audio content and have been successfully used in many music information retrieval tasks. We introduce three approaches to incorporate the features, including two latent space regularization methods and one naive concatenation method. Our preliminary results show that our proposed approaches could improve some evaluation metrics for music source separation. In this work, we also include a discussion of our proposed approaches, such as the pros and cons of each approach, and the potential extension/improvement.

Abstract: Joint time–frequency scattering (JTFS) is a convolutional operator in the time–frequency domain which extracts spectrotemporal modulations at various rates and scales. It offers an idealized model of spectrotemporal receptive fields (STRF) in the primary auditory cortex, and thus may serve as a biological plausible surrogate for human perceptual judgments at the scale of isolated audio events. Yet, prior implementations of JTFS and STRF have remained outside of the standard toolkit of perceptual similarity measures and evaluation methods for audio generation. We trace this issue down to three limitations: differentiability, speed, and flexibility. In this paper, we present an implementation of time–frequency scattering in Python. Unlike prior implementations, ours accommodates NumPy, PyTorch, and TensorFlow as backends and is thus portable on both CPU and GPU. We demonstrate the usefulness of JTFS via three applications: unsupervised manifold learning of spectrotemporal modulations, supervised classification of musical instruments, and texture resynthesis of bioacoustic sounds.

Abstract: Deep learning models applied to raw audio are rapidly gaining relevance in modeling audio analog devices. This paper investigates the use of different deep architectures for modeling audio optical compression. The models use as input and produce as output raw audio samples at audio rate, and it works with noor small-input buffers allowing a theoretical real-time and lowlatency implementation. In this study, two compressor parameters, the ratio, and threshold have been included in the modeling process aiming to condition the inference of the trained network. Deep learning architectures are compared to model an all-tube optical mono compressor including feed-forward, recurrent, and encoder-decoder models. The results of this study show that feedforward and long short-term memory architectures present limitations in modeling the triggering phase of the compressor, performing well only on the sustained phase. On the other hand, encoderdecoder models outperform other architectures in replicating the overall compression process, but they overpredict the energy of high-frequency components.

Abstract: Neural music instrument cloning is an application of deep neural networks for imitating the timbre of a particular music instrument recording with a trained neural network. One can create such clones using an approach such as DDSP , which has been shown to achieve good synthesis quality for several instrument types . However, this approach needs about ten minutes of audio data from the instrument of interest (target recording audio). In this work, we modify the DDSP architecture and apply transfer learning techniques used in speech voice cloning to significantly reduce the amount of target recording audio required. We compare various cloning approaches and architectures across durations of target recording audio, ranging from four to 256 seconds. We demonstrate editing of loudness and pitch as well as timbre transfer from only 16 seconds of target recording audio. Our code is available online1 as well as many audio examples.

Abstract: This paper explores the digital emulation of analog dynamic range compressors, proposing a grey-box model that uses a combination of traditional signal processing techniques and machine learning. The main idea is to use the structure of a traditional digital compressor in a machine learning framework, so it can be trained end-to-end to create a virtual analog model of a compressor from data. The complexity of the model can be adjusted, allowing a trade-off between the model accuracy and computational cost. The proposed model has interpretable components, so its behaviour can be controlled more readily after training in comparison to a black-box model. The result is a model that achieves similar accuracy to a black-box baseline, whilst requiring less than 10% of the number of operations per sample at runtime.

Abstract: Disc jockeys (DJs) use audio effects to make a smooth transition from one song to another. There have been attempts to computationally analyze the creative process of seamless mixing. However, only a few studies estimated fader or equalizer (EQ) gains controlled by DJs. In this study, we propose a method that jointly estimates time-varying fader and EQ gains so as to reproduce the mix from individual source tracks. The method approximates the equalizer filters with a linear combination of a fixed equalizer filter and a constant gain to convert the joint estimation into a convex optimization problem. For the experiment, we collected a new DJ mix dataset that consists of 5,040 real-world DJ mixes with 50,742 transitions, and evaluated the proposed method with a mix reconstruction error. The result shows that the proposed method estimates the time-varying fader and equalizer gains more accurately than existing methods and simple baselines.

Abstract: Deep learning models are mostly used in an offline inference fashion. However, this strongly limits the use of these models inside audio generation setups, as most creative workflows are based on real-time digital signal processing. Although approaches based on recurrent networks can be naturally adapted to this buffer-based computation, the use of convolutions still poses some serious challenges. To tackle this issue, the use of causal streaming convolutions have been proposed. However, this requires specific complexified training and can impact the resulting audio quality. In this paper, we introduce a new method allowing to produce non-causal streaming models. This allows to make any convolutional model compatible with real-time buffer-based processing. As our method is based on a post-training reconfiguration of the model, we show that it is able to transform models trained without causal constraints into streaming models. We apply our method on the recent RAVE model as a case study. This model provides high-quality real-time audio synthesis on a wide range of signals and thus is an ideal candidate to evaluate our method. It should be noted that our method is not restricted to RAVE, and can be straightforwardly applied to any convolutional network. We test our approach on multiple music and speech datasets and show that it is faster than overlap-add methods, while having no impact on the generation quality. Finally, we introduce two open-source implementation of our work as Max/MSP and PureData externals, and as a VST audio plugin. This allows to endow traditional digital audio workstations with real-time neural audio synthesis.

Abstract: The aim for the tutorial is to provide an introduction into the general notion of computational topology as it has appeared in sound synthesis and signal processing as well as discuss computational methods as have emerged over the last two decades. The workshop is structured such that the first block can serve as a non-technical introduction to computational topology with strong ties to DAFx type techniques, and the second block is focused in introducing two prominent methods, persistent homology on time series, and topological filters using sheaves. The tutorial is geared to the working scholar in DAFx with a typical background in signal processing and linear algebra.

Abstract: Sound morphing requires the use of several audio processing algorithms involving the analysis, transformation, and resynthesis of sounds. The aim of this tutorial is to review the techniques implemented in the sound morphing toolbox (SMT). The SMT is open-source and freely available on GitHub. In this tutorial, we will go over the analysis, transformation, and resynthesis steps used in standard morphing techniques applied to isolated notes from musical instrument sounds, such as sinusoidal modeling, spectral envelope estimation, time-scale modification, and resynthesis models. The tutorial will include sound examples to illustrate each step and a hands-on session to show the participants how to use the SMT. The companion webpage has further information.

Abstract: The aim for the tutorial is to provide an introduction into the general notion of computational topology as it has appeared in sound synthesis and signal processing as well as discuss computational methods as have emerged over the last two decades. The workshop is structured such that the first block can serve as a non-technical introduction to computational topology with strong ties to DAFx type techniques, and the second block is focused in introducing two prominent methods, persistent homology on time series, and topological filters using sheaves. The tutorial is geared to the working scholar in DAFx with a typical background in signal processing and linear algebra.

Abstract: The computation of a nonlinear musical circuit model is often just thought of as the last step of its digital realization. Accordingly, an empirical search of the most convenient numerical solver is often conducted aiming at heuristically proving robustness, accuracy, and economy during the processing of a reference system of nonlinear equations. Rarely a preliminary analysis of the computational problem is addressed on the musical circuit model. Starting from existing criteria for the prediction of robustness as well as computational speed of popular solvers such as fixed-point and Newton-Raphson, this tutorial illustrates some recent research results showing that such criteria establish a link between the model and the solver. In particular, these results speak in favor of Wave Digital Filter networks as a modeling domain where fixed-point and Newton-Raphson solvers can be conveniently designed in order to maximize convergence speed and robustness according to physically meaningful procedures.

Abstract: Machine Listening, or AI for Sound, is defined as the general field of Artificial Intelligence (AI) applied to audio analysis, understanding and synthesis by a machine. It can also be viewed as the intertwined domain of Machine learning and audio signal processing and is applicable in a wide range of fields. For AI in general, the access to ever-increasing super-computing facilities, combined with the availability of huge data repositories (although largely unannotated), has permitted the emergence of a significant trend with pure data-driven deep learning approaches. However, these methods only loosely take into account the nature and structure of the processed data. We believe that it is important to rather build hybrid deep learning methods by integrating our prior knowledge about the nature of the processed data, their generation process or if possible their perception by humans. We will illustrate the potential of such model-based deep learning approaches in terms of complexity and controllability on different audio applications, including audio source separation.

Abstract: What are the fundamental elements of sound? What is the best framework for analyzing existing sonic realities and for expressing new sound concepts? These are long standing questions in sound physics, perception, and creation. In the nineteen-forties, Dennis Gabor embraced the mathematics of quantum theory to shed light on subjective acoustics, thus laying the basis for sound analysis and synthesis based on acoustical quanta. In everyday life, it is our body that helps establishing bridges between distal (source-related) and proximal (sensory-related) representations of sound. In particular, it is our vocal apparatus that offers body-based representations of sound, so that vocal imitations can be used as probes into the world of sound at large. In analogy with quantum theory, we can define "observables", "unitary evolution", and "measurement apparati" to represent sound articulations in the phon space, whose three axes can be bound to the articulatory qualities of phonation, turbulence, and clicks. The inherent parallelism and ambiguity of evolving sonic sketches can be expressed by the phon as a pure state in superposition, and the epistemic uncertainty of the listener can be expressed as a mixture of phons. Implications for sound design may be drawn.

Abstract: Much exciting work exists at the intersection of music information research and cardiovascular science, a connection made possible by the commonalities between music and heartbeats and music’s profound impact on the heart. I shall describe how background research on mathematical and computational approaches to representing music expressivity leads to our citizen science approach to marking the structures musicians create in performance, offering new paths to discovering the reasoning behind performance decisions. Next, I will demonstrate how music transcription offers new views on both music expressivity and abnormal heart rhythms, with impact on music generation and potentially disease stratification. Finally, I shall show how these computational approaches to characterising expressivity and performance decisions allow us to link how music is communicated to how it impacts listeners, thus presenting new tools for studying autonomic modulation through music, with promise for formalizing knowledge about the therapeutic effects of music^†.

^† The results presented are part of the projects COSMOS and HEART.FM that have received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement Nos. 788960 and 957532).