Re: Spectral Editing is Not User Friendly

Steve,

On 08/10/2017 22:11, Steve the Fiddle wrote:
>
>
> On 9 October 2017 at 00:34, Federico Miyara <[email protected] 
> <mailto:[email protected]>> wrote:
>
>
>     Steve and friends,
>
>>     I like the idea of being able to export the spectrogram, edit in
>>     an image editor, and then import back into Audacity. I presume
>>     that what you have in mind is similar to the software
>>     "Photosounder" (http://photosounder.com/). That would be a
>>     totally new feature, with only superficial similarity to the
>>     current "spectral edit" feature.
>
>     I don't think it is theoretically possible to edit the spectrogram
>     in an image processor (even if it were a customized one for
>     Audacity) and get a valid spectrogram of the processed signal from
>     which the processed signal could be recovered.
>
>
>
> See this video: 
> https://www.youtube.com/watch?time_continue=37&v=W8MCAXhEsy4
>
> Editing any of those images in an image processor prior to 
> 'converting' them to sound does change the sound that is produced. 
> Editing a spectrogram in an image processor may not give intuitively 
> predictable effects, but it certainly would give an effect. For 
> 'creative' use, the technical limitations may be an asset.

This is quite interesting. I guess they convert first to gray shades 
(conveniently not shown), sample in time domain and frequency domain and 
do not worry about phase. Then it is possible to apply inverse FFT to 
each frame thus obtained and finally to overlapp-add. With some 
refinement it would also be possible to add a frame-dependent (and 
frequency-dependent) phase to improve continuity.

Do you have this program?

It would be interesting to perform some objective tests. For instance, 
take any of those fractals (or compose one with Apophysis 7x), 
synthesize the wav file, close the program, change the file name (just 
in case...! There may be social engineering embedded everywhere!) open 
the file again and see the true spectrogram. I'm sure the beautiful 
fractal smoothness will have been swept out.

This reminds me of Xenakis' UPIC (1977!):

https://en.wikipedia.org/wiki/UPIC

See also this theory:

https://monoskop.org/images/7/74/Xenakis_Iannis_Formalized_Music_Thought_and_Mathematics_in_Composition.pdf

and

http://www.eecs.qmul.ac.uk/~nickbk/papers/ThiebautHealeyKinns-ICMC2008.pdf

Federico


>
> Steve
>
>
>     First of all, the spectrogram has no phase information, so colors
>     alone cannot represent the correct sound, even the original one.
>     In other words, the spectrogram drawing is not a reversible
>     operation, even if no transformation or effect is applied.
>
>     Second, the color map is a single-dimension one, defined by the
>     linear interpolation of five colors which are just points on a
>     bi-dimensional color map. Unless we could ensure the map color of
>     the image processor is restricted to the colors contained in the
>     piecewise linear path representing amplitudes, we would get
>     meaningless spectrograms.
>
>     Third, there is a limitation in resolution given by the size of
>     the window so spectrograms do not present a continuous spatial
>     transition of colors. Instead there are small but noticeable
>     rectangles whose areas depend on the time-frequency uncertainty
>     principle. If the FFT size is large, they are very short (high
>     frequency resolution) and wide (low time resolution); if FFT size
>     is small, they are very tall and narrow instead.
>
>     A possibility would be to take advantage of the bi-dimensional
>     nature of the full visible color map and represent the FFT complex
>     numbers (retaining the phase information), which are also
>     bi-dimensional, as colors. I guess several people have already
>     tried some experiments on this (I mean the graphic representation;
>     the concept of complex spectrogram is frequently used in signal
>     processing). I also guess the spectrogram image would be stunning
>     but far from being intuitive. Moreover, it would be very difficult
>     to figure out the result of a given graphic effect, as well as
>     what to do with the image to get the desired signal effect.
>
>     The only practical approach is to simulate the spectrogram editing
>     for very simple operations, such as zero or one gain (or real gain
>     or attenuation).
>
>     This is the approach we had presented (with Luciano). Our plugin
>     allowed to draw closed regions of arbitrary shape to be either
>     filtered out or retained . The plugin was based on overlapping
>     time-varying FFT filter. Whatever the case, the retained part has
>     gain 1 and the rejected part has gain 0. The idea could be
>     extended to support drawing with an "attenuating pencil" which
>     would apply a given gain (picked from a gray shade map).
>
>     You mentioned Photosouder. I skimmed through the user guide and
>     there is no hint as to what is actually doing when importing an
>     image. What I'm pretty sure is that when editing an already loaded
>     or computed spectrogram it does not re-synthesize the sound from
>     the image, but processes the signal and recalculates the spectrogram.
>
>     In other words, for an arbitrary image (say, a photo), it tries to
>     represent it within the available color map and spectral and time
>     resolution without concern for phase, but they have the advantage
>     that we don't know a priori how the image "sounds"... And for a
>     true spectrogram, it just simulates the image editing.
>
>     Regards,
>
>     Federico
>
>
>
>     On 06/10/2017 11:19, Steve the Fiddle wrote:
>>     Hi Nikita,
>>
>>     Spectral Editing in Audacity is a fairly new, and not yet fully
>>     developed, feature. Despite its shortcomings, it was considered
>>     to still be useful enough to include in the Audacity release.
>>
>>     I like the idea of being able to export the spectrogram, edit in
>>     an image editor, and then import back into Audacity. I presume
>>     that what you have in mind is similar to the software
>>     "Photosounder" (http://photosounder.com/). That would be a
>>     totally new feature, with only superficial similarity to the
>>     current "spectral edit" feature.
>>
>>     The current feature is really a 'simulation' of editing the
>>     spectrogram, rather than actually editing it. How it works is
>>     that the selected spectral region sets two frequency parameters
>>     (high and low). Specially crafted effects (the "Spectral"
>>     effects) can then use these parameters to control their
>>     behaviour. The effects then process the audio data as would any
>>     other audio effect. So the effect is not actually acting on the
>>     spectrogram - it acts on the audio, and the spectrogram is
>>     updated from the processed audio.
>>
>>     Steve
>>
>>     On 6 October 2017 at 13:38, Nikita Sadkov <[email protected]
>>     <mailto:[email protected]>> wrote:
>>
>>         Hi! It was an amazing discovery, that Audacity now supports
>>         spectral
>>         editing - arguably one of the most fundamental features,
>>         especially if
>>         you want to synth custom sfx for your project or edit some badly
>>         recorded material from https://freesound.org
>>
>>         My complain is that it is very obscure and tedious to use in its
>>         current form. You have to separately enable spectral
>>         selection and use
>>         "Spectral edit ..." tools under "Effect" menu. I.e. you can't
>>         just
>>         press Del to clear the frequencies or use Draw Tool (F3) to
>>         paint your
>>         sfx with mouse.
>>
>>         So I think it would make Audacity order of magnitude more
>>         useful, if
>>         it copies UI from Sony SpectraLayers, together with layers, which
>>         works just like a raster editor, but unfortunately lacks waveform
>>         editing capabilities. Also, it would be nice feature to export
>>         spectrum plot into png files, so people could edit it in GIMP
>>         and then
>>         import back into Audacity.
>>
>>         ------------------------------------------------------------------------------
>>         Check out the vibrant tech community on one of the world's most
>>         engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>         *********** ASKING FOR HELP *************
>>
>>         When asking for help on this list, please include the
>>         following information, so we can
>>         help you properly:
>>
>>         * What operating system you are using (for example, Windows
>>         XP or Mac OS X 10.5.1)
>>
>>         * Exactly what three digit version number of Audacity you are
>>         using (Help > About
>>            Audacity, or Audacity >  About Audacity on a Mac computer)
>>
>>         * If this is a recording problem, what equipment you are
>>         recording with, and how is it
>>            connected to the computer?
>>
>>         Mailing list: [email protected]
>>         <mailto:[email protected]>
>>         To UNSUBSCRIBE, use the form at the bottom of this web page:
>>         https://lists.sourceforge.net/lists/listinfo/audacity-users
>>         <https://lists.sourceforge.net/lists/listinfo/audacity-users>
>>
>>
>>
>>
>>     ------------------------------------------------------------------------------
>>     Check out the vibrant tech community on one of the world's most
>>     engaging tech sites, Slashdot.org!http://sdm.link/slashdot
>>
>>     *********** ASKING FOR HELP *************
>>
>>     When asking for help on this list, please include the following information, so we can
>>     help you properly:
>>
>>     * What operating system you are using (for example, Windows XP or Mac OS X 10.5.1)
>>
>>     * Exactly what three digit version number of Audacity you are using (Help > About
>>         Audacity, or Audacity >  About Audacity on a Mac computer)
>>
>>     * If this is a recording problem, what equipment you are recording with, and how is it
>>         connected to the computer?
>>
>>     Mailing list:[email protected]
>>     <mailto:[email protected]>
>>     To UNSUBSCRIBE, use the form at the bottom of this web page:
>>     https://lists.sourceforge.net/lists/listinfo/audacity-users
>>     <https://lists.sourceforge.net/lists/listinfo/audacity-users>
>     ------------------------------------------------------------------------------
>     Check out the vibrant tech community on one of the world's most
>     engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>     *********** ASKING FOR HELP ************* When asking for help on
>     this list, please include the following information, so we can
>     help you properly: * What operating system you are using (for
>     example, Windows XP or Mac OS X 10.5.1) * Exactly what three digit
>     version number of Audacity you are using (Help > About  
>      Audacity, or Audacity >  About Audacity on a Mac computer) * If
>     this is a recording problem, what equipment you are recording
>     with, and how is it    connected to the computer? Mailing list:
>     [email protected]
>     <mailto:[email protected]> To UNSUBSCRIBE, use
>     the form at the bottom of this web page:
>     https://lists.sourceforge.net/lists/listinfo/audacity-users
>     <https://lists.sourceforge.net/lists/listinfo/audacity-users> 
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
> *********** ASKING FOR HELP *************
>
> When asking for help on this list, please include the following information, so we can
> help you properly:
>
> * What operating system you are using (for example, Windows XP or Mac OS X 10.5.1)
>
> * Exactly what three digit version number of Audacity you are using (Help > About
>     Audacity, or Audacity >  About Audacity on a Mac computer)
>
> * If this is a recording problem, what equipment you are recording with, and how is it
>     connected to the computer?
>
> Mailing list: [email protected]
> To UNSUBSCRIBE, use the form at the bottom of this web page:
> https://lists.sourceforge.net/lists/listinfo/audacity-users

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

*********** ASKING FOR HELP *************

When asking for help on this list, please include the following information, so we can
help you properly:

* What operating system you are using (for example, Windows XP or Mac OS X 10.5.1)

* Exactly what three digit version number of Audacity you are using (Help > About
   Audacity, or Audacity >  About Audacity on a Mac computer)

* If this is a recording problem, what equipment you are recording with, and how is it
   connected to the computer?

Mailing list: [email protected]
To UNSUBSCRIBE, use the form at the bottom of this web page:
https://lists.sourceforge.net/lists/listinfo/audacity-users