-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy pathlatent_source.html
More file actions
168 lines (144 loc) · 8.79 KB
/
Copy pathlatent_source.html
File metadata and controls
168 lines (144 loc) · 8.79 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="description" content="Demo of AMSS-Net-Audio Manipulation on User-Specified Sources with Textual Queries">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>AMSS-Net</title>
<link rel="stylesheet" href="css/normalize.css">
<link rel="stylesheet" href="css/solo.css">
<link rel="stylesheet" href="css/syntax.css">
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css">
<link rel="stylesheet" href="css/rrssb.css">
<script src="../scripts/jquery.js"></script>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
tex2jax: {
inlineMath: [['$','$'], ['\\(','\\)']],
processEscapes: true
}
});
</script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script>
<link rel="canonical" href="" />
</head>
<body>
<article>
<div class="container">
<h1><a href="./index.html">AMSS-Net</a></h1>
<h3 id="how-amss-net-works-latent-source-channels">How AMSS-Net works: Latent Source Channels</h3>
<p>We generate an audio track from a single latent source to check how AMSS-Net works in this demo.</p>
<h4 id="1-latent-source">1. Latent Source</h4>
<ul>
<li>In AMSS-Net, a <strong><em>latent source channel</em></strong> deals with a more detailed acoustic feature than a symbolic-level source</li>
<li>We are used to categorizing symbolic-level classes of sources such as ‘vocals,’ ‘drums’ or ‘bass.’</li>
<li>However, in fact, there are much more ‘instruments’ or sources we have to consider for AMSS</li>
<li>For example,
<ul>
<li>lower frequency band of female-classic-soprano, male-jazz-baritone … $\in$ ‘vocals’</li>
<li>kick, snare, rimshot, hat(closed), tom-tom … $\in$ ‘drums’</li>
<li>contrabass, electronic, walking bass piano (boogie-woogie) … $\in$ ‘bass.’</li>
</ul>
</li>
<li>Such latent source-level analysis enables AMSS-Net to perform delicate manipulation for the given AMSS task</li>
<li>We designed AMSS-Net to perform latent source-level analysis for the given audio.</li>
<li>Similar to [1], we assume that a weighted sum of latent sources can represent a source
<ul>
<li>while [2] assumed that latent sources are independent.</li>
</ul>
</li>
</ul>
<h4 id="2-latent-source-channel-extraction">2. Latent Source Channel Extraction</h4>
<ul>
<li>Each decoding block of AMSS-Net
<ul>
<li>first extracts a feature map $V^{ch}$, in which each channel corresponds to a specific <em>latent source</em> ($\leftarrow$ by Latent Source Channel Extractor),</li>
<li>and selectively manipulates those channels for AMSS</li>
</ul>
</li>
<li>After training, the feature map $V^{ch}$ is expected that
<ul>
<li>each channel of $V^{ch}$ deals with a specific latent source</li>
</ul>
</li>
<li>The conceptual view of the Latent Source Channel is shown in the figure below
<ul>
<li>For example, the second channel deals with the acoustic features observed in the singing voice.</li>
</ul>
</li>
</ul>
<p><img src="ablation/latentsource.png" alt="" /></p>
<h4 id="3-visualization-of-latent-source-channels">3. Visualization of Latent Source Channels</h4>
<ul>
<li>To verify that each decoding block in AMSS-Net extracts a feature map, in which each channel corresponds to a specific latent source,
<ul>
<li>we generate an audio track from a single latent source channel,
<ul>
<li>During the last decoding block, we mask all channels in the manipulated feature map $V’^{ch}$ except for a single latent source channel</li>
<li>and fed it to the remaining sub-networks to generate the audio track.</li>
</ul>
</li>
</ul>
</li>
<li>
<p>Below is the python API that extracts audio from a latent source channel=4 at head=1</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">manipulate_lach</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">head</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">lach</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">amss</span><span class="p">)</span>
</code></pre></div> </div>
</li>
</ul>
<h4 id="4-results">4. Results</h4>
<p>Below are generated audios of a single latent source channel.
For example, if we generate output after masking all channels except for the fifth channel in the second head group, then the result sounds similar to the low-frequency band of drums (i.e., kick drum).</p>
<p>AMSS-Net can keep this channel and drop other drum-related channels to process “apply lowpass to drums.”</p>
<p>However, we found that a latent source channel does not always contain a single class of instruments.
For example, the latent channel of the fourth row in the table deals with several instruments.
Some latent sources were not interpretable to the authors.</p>
<table>
<thead>
<tr>
<th>Latent Source Channel</th>
<th>similar symbol</th>
<th style="text-align: center">Audio</th>
<th style="text-align: center">Spectrogram</th>
</tr>
</thead>
<tbody>
<tr>
<td>N/A</td>
<td>origin</td>
<td style="text-align: center"><audio controls="" class="audio-player" preload="metadata"><source src="https://ws-choi.github.io/personal/presentations/slide/assets/mm_presentation/footprint__original.mp3" type="audio/mp3" /></audio></td>
<td style="text-align: center"><img src="https://ws-choi.github.io/personal/presentations/slide/assets/mm_presentation/latent_original_spec.png" /></td>
</tr>
<tr>
<td>head=5, lach=0</td>
<td>drums (left channel)</td>
<td style="text-align: center"><audio controls="" class="audio-player" preload="metadata"><source src="https://ws-choi.github.io/personal/presentations/slide/assets/mm_presentation/latent_drumsvocals.mp3" type="audio/mp3" /></audio></td>
<td style="text-align: center"><img src="https://ws-choi.github.io/personal/presentations/slide/assets/mm_presentation/latent_drumsvocals_spec.png" /></td>
</tr>
<tr>
<td>head=2, lach=6</td>
<td>vocals</td>
<td style="text-align: center"><audio controls="" class="audio-player" preload="metadata"><source src="https://ws-choi.github.io/personal/presentations/slide/assets/mm_presentation/latent_vocals.mp3" type="audio/mp3" /></audio></td>
<td style="text-align: center"><img src="https://ws-choi.github.io/personal/presentations/slide/assets/mm_presentation/latent_vocals_spec.png" /></td>
</tr>
<tr>
<td>head=3, lach=5</td>
<td>drums (clap-snare) and piano</td>
<td style="text-align: center"><audio controls="" class="audio-player" preload="metadata"><source src="https://ws-choi.github.io/personal/presentations/slide/assets/mm_presentation/latent_drums_pianos.mp3" type="audio/mp3" /></audio></td>
<td style="text-align: center"><img src="https://ws-choi.github.io/personal/presentations/slide/assets/mm_presentation/latent_vocals_spec.png" /></td>
</tr>
</tbody>
</table>
<h5 id="track-info">Track info:</h5>
<iframe width="60%" height="100" scrolling="no" frameborder="no" allow="autoplay" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/108154870&color=%23ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&show_teaser=true&visual=true"></iframe>
<div style="font-size: 10px; color: #cccccc;line-break: anywhere;word-break: normal;overflow: hidden;white-space: nowrap;text-overflow: ellipsis; font-family: Interstate,Lucida Grande,Lucida Sans Unicode,Lucida Sans,Garuda,Verdana,Tahoma,sans-serif;font-weight: 100;"><a href="https://soundcloud.com/choi-hn" title="Woosung Choi" target="_blank" style="color: #cccccc; text-decoration: none;">Woosung Choi</a> · <a href="https://soundcloud.com/choi-hn/2-2u" title="footprint" target="_blank" style="color: #cccccc; text-decoration: none;">footprint</a></div>
<hr />
<h3 id="reference">Reference</h3>
<p>[1] Woosung Choi, Minseok Kim, Jaehwa Chung, and Soonyoung Jung. 2020. LaSAFT:Latent Source Attentive Frequency Transformation for Conditioned Source Sepa-ration.arXiv preprint arXiv:2010.11631(2020).</p>
<p>[2] Scott Wisdom, Efthymios Tzinis, Hakan Erdogan, Ron J. Weiss, Kevin Wilson, andJohn R. Hershey. 2020. Unsupervised Sound Separation Using Mixture InvariantTraining. In NeurIPS. https://arxiv.org/pdf/2006.12</p>
</div>
</article>
<script src="../scripts/rrssb.min.js"></script>
</body>
</html>