Re: Spectral Editing is Not User Friendly

I still suggest you, folks, to see youtube videos with SpectraLayers
and Celemony Melodyne in action. After all, they did a lot of R&D on
spectrogram editing. Melodyne is especially interesting, because it
does a spectrogram editing behind the scenes, showing user only major
features, like vocal, so user basically loads a track into Melodyne
and then apples effect to say only vocal component.

On 10/9/17, Federico Miyara <[email protected]> wrote:
>
> Steve,
>
> On 08/10/2017 22:11, Steve the Fiddle wrote:
>>
>>
>> On 9 October 2017 at 00:34, Federico Miyara <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>>
>>     Steve and friends,
>>
>>>     I like the idea of being able to export the spectrogram, edit in
>>>     an image editor, and then import back into Audacity. I presume
>>>     that what you have in mind is similar to the software
>>>     "Photosounder" (http://photosounder.com/). That would be a
>>>     totally new feature, with only superficial similarity to the
>>>     current "spectral edit" feature.
>>
>>     I don't think it is theoretically possible to edit the spectrogram
>>     in an image processor (even if it were a customized one for
>>     Audacity) and get a valid spectrogram of the processed signal from
>>     which the processed signal could be recovered.
>>
>>
>>
>> See this video:
>> https://www.youtube.com/watch?time_continue=37&v=W8MCAXhEsy4
>>
>> Editing any of those images in an image processor prior to
>> 'converting' them to sound does change the sound that is produced.
>> Editing a spectrogram in an image processor may not give intuitively
>> predictable effects, but it certainly would give an effect. For
>> 'creative' use, the technical limitations may be an asset.
>
> This is quite interesting. I guess they convert first to gray shades
> (conveniently not shown), sample in time domain and frequency domain and
> do not worry about phase. Then it is possible to apply inverse FFT to
> each frame thus obtained and finally to overlapp-add. With some
> refinement it would also be possible to add a frame-dependent (and
> frequency-dependent) phase to improve continuity.
>
> Do you have this program?
>
> It would be interesting to perform some objective tests. For instance,
> take any of those fractals (or compose one with Apophysis 7x),
> synthesize the wav file, close the program, change the file name (just
> in case...! There may be social engineering embedded everywhere!) open
> the file again and see the true spectrogram. I'm sure the beautiful
> fractal smoothness will have been swept out.
>
> This reminds me of Xenakis' UPIC (1977!):
>
> https://en.wikipedia.org/wiki/UPIC
>
> See also this theory:
>
> https://monoskop.org/images/7/74/Xenakis_Iannis_Formalized_Music_Thought_and_Mathematics_in_Composition.pdf
>
> and
>
> http://www.eecs.qmul.ac.uk/~nickbk/papers/ThiebautHealeyKinns-ICMC2008.pdf
>
> Federico
>
>
>>
>> Steve
>>
>>
>>     First of all, the spectrogram has no phase information, so colors
>>     alone cannot represent the correct sound, even the original one.
>>     In other words, the spectrogram drawing is not a reversible
>>     operation, even if no transformation or effect is applied.
>>
>>     Second, the color map is a single-dimension one, defined by the
>>     linear interpolation of five colors which are just points on a
>>     bi-dimensional color map. Unless we could ensure the map color of
>>     the image processor is restricted to the colors contained in the
>>     piecewise linear path representing amplitudes, we would get
>>     meaningless spectrograms.
>>
>>     Third, there is a limitation in resolution given by the size of
>>     the window so spectrograms do not present a continuous spatial
>>     transition of colors. Instead there are small but noticeable
>>     rectangles whose areas depend on the time-frequency uncertainty
>>     principle. If the FFT size is large, they are very short (high
>>     frequency resolution) and wide (low time resolution); if FFT size
>>     is small, they are very tall and narrow instead.
>>
>>     A possibility would be to take advantage of the bi-dimensional
>>     nature of the full visible color map and represent the FFT complex
>>     numbers (retaining the phase information), which are also
>>     bi-dimensional, as colors. I guess several people have already
>>     tried some experiments on this (I mean the graphic representation;
>>     the concept of complex spectrogram is frequently used in signal
>>     processing). I also guess the spectrogram image would be stunning
>>     but far from being intuitive. Moreover, it would be very difficult
>>     to figure out the result of a given graphic effect, as well as
>>     what to do with the image to get the desired signal effect.
>>
>>     The only practical approach is to simulate the spectrogram editing
>>     for very simple operations, such as zero or one gain (or real gain
>>     or attenuation).
>>
>>     This is the approach we had presented (with Luciano). Our plugin
>>     allowed to draw closed regions of arbitrary shape to be either
>>     filtered out or retained . The plugin was based on overlapping
>>     time-varying FFT filter. Whatever the case, the retained part has
>>     gain 1 and the rejected part has gain 0. The idea could be
>>     extended to support drawing with an "attenuating pencil" which
>>     would apply a given gain (picked from a gray shade map).
>>
>>     You mentioned Photosouder. I skimmed through the user guide and
>>     there is no hint as to what is actually doing when importing an
>>     image. What I'm pretty sure is that when editing an already loaded
>>     or computed spectrogram it does not re-synthesize the sound from
>>     the image, but processes the signal and recalculates the spectrogram.
>>
>>     In other words, for an arbitrary image (say, a photo), it tries to
>>     represent it within the available color map and spectral and time
>>     resolution without concern for phase, but they have the advantage
>>     that we don't know a priori how the image "sounds"... And for a
>>     true spectrogram, it just simulates the image editing.
>>
>>     Regards,
>>
>>     Federico
>>
>>
>>
>>     On 06/10/2017 11:19, Steve the Fiddle wrote:
>>>     Hi Nikita,
>>>
>>>     Spectral Editing in Audacity is a fairly new, and not yet fully
>>>     developed, feature. Despite its shortcomings, it was considered
>>>     to still be useful enough to include in the Audacity release.
>>>
>>>     I like the idea of being able to export the spectrogram, edit in
>>>     an image editor, and then import back into Audacity. I presume
>>>     that what you have in mind is similar to the software
>>>     "Photosounder" (http://photosounder.com/). That would be a
>>>     totally new feature, with only superficial similarity to the
>>>     current "spectral edit" feature.
>>>
>>>     The current feature is really a 'simulation' of editing the
>>>     spectrogram, rather than actually editing it. How it works is
>>>     that the selected spectral region sets two frequency parameters
>>>     (high and low). Specially crafted effects (the "Spectral"
>>>     effects) can then use these parameters to control their
>>>     behaviour. The effects then process the audio data as would any
>>>     other audio effect. So the effect is not actually acting on the
>>>     spectrogram - it acts on the audio, and the spectrogram is
>>>     updated from the processed audio.
>>>
>>>     Steve
>>>
>>>     On 6 October 2017 at 13:38, Nikita Sadkov <[email protected]
>>>     <mailto:[email protected]>> wrote:
>>>
>>>         Hi! It was an amazing discovery, that Audacity now supports
>>>         spectral
>>>         editing - arguably one of the most fundamental features,
>>>         especially if
>>>         you want to synth custom sfx for your project or edit some badly
>>>         recorded material from https://freesound.org
>>>
>>>         My complain is that it is very obscure and tedious to use in its
>>>         current form. You have to separately enable spectral
>>>         selection and use
>>>         "Spectral edit ..." tools under "Effect" menu. I.e. you can't
>>>         just
>>>         press Del to clear the frequencies or use Draw Tool (F3) to
>>>         paint your
>>>         sfx with mouse.
>>>
>>>         So I think it would make Audacity order of magnitude more
>>>         useful, if
>>>         it copies UI from Sony SpectraLayers, together with layers,
>>> which
>>>         works just like a raster editor, but unfortunately lacks
>>> waveform
>>>         editing capabilities. Also, it would be nice feature to export
>>>         spectrum plot into png files, so people could edit it in GIMP
>>>         and then
>>>         import back into Audacity.
>>>
>>>
>>> ------------------------------------------------------------------------------
>>>         Check out the vibrant tech community on one of the world's most
>>>         engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>         *********** ASKING FOR HELP *************
>>>
>>>         When asking for help on this list, please include the
>>>         following information, so we can
>>>         help you properly:
>>>
>>>         * What operating system you are using (for example, Windows
>>>         XP or Mac OS X 10.5.1)
>>>
>>>         * Exactly what three digit version number of Audacity you are
>>>         using (Help > About
>>>            Audacity, or Audacity >  About Audacity on a Mac computer)
>>>
>>>         * If this is a recording problem, what equipment you are
>>>         recording with, and how is it
>>>            connected to the computer?
>>>
>>>         Mailing list: [email protected]
>>>         <mailto:[email protected]>
>>>         To UNSUBSCRIBE, use the form at the bottom of this web page:
>>>         https://lists.sourceforge.net/lists/listinfo/audacity-users
>>>         <https://lists.sourceforge.net/lists/listinfo/audacity-users>
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>>     Check out the vibrant tech community on one of the world's most
>>>     engaging tech sites, Slashdot.org!http://sdm.link/slashdot
>>>
>>>     *********** ASKING FOR HELP *************
>>>
>>>     When asking for help on this list, please include the following
>>> information, so we can
>>>     help you properly:
>>>
>>>     * What operating system you are using (for example, Windows XP or Mac
>>> OS X 10.5.1)
>>>
>>>     * Exactly what three digit version number of Audacity you are using
>>> (Help > About
>>>         Audacity, or Audacity >  About Audacity on a Mac computer)
>>>
>>>     * If this is a recording problem, what equipment you are recording
>>> with, and how is it
>>>         connected to the computer?
>>>
>>>     Mailing list:[email protected]
>>>     <mailto:[email protected]>
>>>     To UNSUBSCRIBE, use the form at the bottom of this web page:
>>>     https://lists.sourceforge.net/lists/listinfo/audacity-users
>>>     <https://lists.sourceforge.net/lists/listinfo/audacity-users>
>>
>> ------------------------------------------------------------------------------
>>     Check out the vibrant tech community on one of the world's most
>>     engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>     *********** ASKING FOR HELP ************* When asking for help on
>>     this list, please include the following information, so we can
>>     help you properly: * What operating system you are using (for
>>     example, Windows XP or Mac OS X 10.5.1) * Exactly what three digit
>>     version number of Audacity you are using (Help > About
>>      Audacity, or Audacity >  About Audacity on a Mac computer) * If
>>     this is a recording problem, what equipment you are recording
>>     with, and how is it    connected to the computer? Mailing list:
>>     [email protected]
>>     <mailto:[email protected]> To UNSUBSCRIBE, use
>>     the form at the bottom of this web page:
>>     https://lists.sourceforge.net/lists/listinfo/audacity-users
>>     <https://lists.sourceforge.net/lists/listinfo/audacity-users>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>
>> *********** ASKING FOR HELP *************
>>
>> When asking for help on this list, please include the following
>> information, so we can
>> help you properly:
>>
>> * What operating system you are using (for example, Windows XP or Mac OS X
>> 10.5.1)
>>
>> * Exactly what three digit version number of Audacity you are using (Help
>> > About
>>     Audacity, or Audacity >  About Audacity on a Mac computer)
>>
>> * If this is a recording problem, what equipment you are recording with,
>> and how is it
>>     connected to the computer?
>>
>> Mailing list: [email protected]
>> To UNSUBSCRIBE, use the form at the bottom of this web page:
>> https://lists.sourceforge.net/lists/listinfo/audacity-users
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
*********** ASKING FOR HELP *************

When asking for help on this list, please include the following information, so we can
help you properly:

* What operating system you are using (for example, Windows XP or Mac OS X 10.5.1)

* Exactly what three digit version number of Audacity you are using (Help > About
   Audacity, or Audacity >  About Audacity on a Mac computer)

* If this is a recording problem, what equipment you are recording with, and how is it
   connected to the computer?

Mailing list: [email protected]
To UNSUBSCRIBE, use the form at the bottom of this web page:
https://lists.sourceforge.net/lists/listinfo/audacity-users