Re: Spectral Editing is Not User Friendly

> The orchestra man that has simultaneously superior command of software planning, advancerd signal processing, graphic interface design and programming is already too busy working for Melodyne, SpetraLayers, etc., or founding their own enterprise whose product we will probably be commenting here in the future.
Well, Melodyne costs $850 (SpectraLayers would be $400), so you will
surely make a lot of money by producing a more affordable alternative.
And indie game developers like me would applaud you success too :-D

On 10/9/17, Federico Miyara <[email protected]> wrote:
>
> Nikita:
>
> I've seen Melodyne. It is very interesting but notice it is not a true
> spectrogram. It is a sort of spectrally based harmonic
> analizer-synthesizer. The graphic output combines fundamental frequency
> with envelope in a single graph.
>
> It analyzes the sounds and detects probable harmonic complexes, and then
> it represents each complex with a single line at the pitch of the
> fundamental, over which a waveform is plotted.
>
> The spectral editing feature doesn't seem to be a true spectral
> processing but a series of parameter control applied to objects
> (thedetectednotes) and resynthesis.
>
> I think that in order to get such kind of advanced features into
> Audacity the approach to development should change from a do-cracy to a
> team-based work, where the responsibility for getting a new advanced
> feature does not rely on an orchestra-man-nightly-work effort but on the
> combination of co-ordinated specialized work. The orchestra man that has
> simultaneously superior command of software planning, advancerd signal
> processing, graphic interface design and programming is already too busy
> working for Melodyne, SpetraLayers, etc., or founding their own
> enterprise whose product we will probably be commenting here in the future.
>
> Regards,
>
> Federico
>
>
> On 09/10/2017 7:00, Nikita Sadkov wrote:
>> I still suggest you, folks, to see youtube videos with SpectraLayers
>> and Celemony Melodyne in action. After all, they did a lot of R&D on
>> spectrogram editing. Melodyne is especially interesting, because it
>> does a spectrogram editing behind the scenes, showing user only major
>> features, like vocal, so user basically loads a track into Melodyne
>> and then apples effect to say only vocal component.
>>
>> On 10/9/17, Federico Miyara <[email protected]> wrote:
>>> Steve,
>>>
>>> On 08/10/2017 22:11, Steve the Fiddle wrote:
>>>>
>>>> On 9 October 2017 at 00:34, Federico Miyara <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>>
>>>>      Steve and friends,
>>>>
>>>>>      I like the idea of being able to export the spectrogram, edit in
>>>>>      an image editor, and then import back into Audacity. I presume
>>>>>      that what you have in mind is similar to the software
>>>>>      "Photosounder" (http://photosounder.com/). That would be a
>>>>>      totally new feature, with only superficial similarity to the
>>>>>      current "spectral edit" feature.
>>>>      I don't think it is theoretically possible to edit the spectrogram
>>>>      in an image processor (even if it were a customized one for
>>>>      Audacity) and get a valid spectrogram of the processed signal from
>>>>      which the processed signal could be recovered.
>>>>
>>>>
>>>>
>>>> See this video:
>>>> https://www.youtube.com/watch?time_continue=37&v=W8MCAXhEsy4
>>>>
>>>> Editing any of those images in an image processor prior to
>>>> 'converting' them to sound does change the sound that is produced.
>>>> Editing a spectrogram in an image processor may not give intuitively
>>>> predictable effects, but it certainly would give an effect. For
>>>> 'creative' use, the technical limitations may be an asset.
>>> This is quite interesting. I guess they convert first to gray shades
>>> (conveniently not shown), sample in time domain and frequency domain and
>>> do not worry about phase. Then it is possible to apply inverse FFT to
>>> each frame thus obtained and finally to overlapp-add. With some
>>> refinement it would also be possible to add a frame-dependent (and
>>> frequency-dependent) phase to improve continuity.
>>>
>>> Do you have this program?
>>>
>>> It would be interesting to perform some objective tests. For instance,
>>> take any of those fractals (or compose one with Apophysis 7x),
>>> synthesize the wav file, close the program, change the file name (just
>>> in case...! There may be social engineering embedded everywhere!) open
>>> the file again and see the true spectrogram. I'm sure the beautiful
>>> fractal smoothness will have been swept out.
>>>
>>> This reminds me of Xenakis' UPIC (1977!):
>>>
>>> https://en.wikipedia.org/wiki/UPIC
>>>
>>> See also this theory:
>>>
>>> https://monoskop.org/images/7/74/Xenakis_Iannis_Formalized_Music_Thought_and_Mathematics_in_Composition.pdf
>>>
>>> and
>>>
>>> http://www.eecs.qmul.ac.uk/~nickbk/papers/ThiebautHealeyKinns-ICMC2008.pdf
>>>
>>> Federico
>>>
>>>
>>>> Steve
>>>>
>>>>
>>>>      First of all, the spectrogram has no phase information, so colors
>>>>      alone cannot represent the correct sound, even the original one.
>>>>      In other words, the spectrogram drawing is not a reversible
>>>>      operation, even if no transformation or effect is applied.
>>>>
>>>>      Second, the color map is a single-dimension one, defined by the
>>>>      linear interpolation of five colors which are just points on a
>>>>      bi-dimensional color map. Unless we could ensure the map color of
>>>>      the image processor is restricted to the colors contained in the
>>>>      piecewise linear path representing amplitudes, we would get
>>>>      meaningless spectrograms.
>>>>
>>>>      Third, there is a limitation in resolution given by the size of
>>>>      the window so spectrograms do not present a continuous spatial
>>>>      transition of colors. Instead there are small but noticeable
>>>>      rectangles whose areas depend on the time-frequency uncertainty
>>>>      principle. If the FFT size is large, they are very short (high
>>>>      frequency resolution) and wide (low time resolution); if FFT size
>>>>      is small, they are very tall and narrow instead.
>>>>
>>>>      A possibility would be to take advantage of the bi-dimensional
>>>>      nature of the full visible color map and represent the FFT complex
>>>>      numbers (retaining the phase information), which are also
>>>>      bi-dimensional, as colors. I guess several people have already
>>>>      tried some experiments on this (I mean the graphic representation;
>>>>      the concept of complex spectrogram is frequently used in signal
>>>>      processing). I also guess the spectrogram image would be stunning
>>>>      but far from being intuitive. Moreover, it would be very difficult
>>>>      to figure out the result of a given graphic effect, as well as
>>>>      what to do with the image to get the desired signal effect.
>>>>
>>>>      The only practical approach is to simulate the spectrogram editing
>>>>      for very simple operations, such as zero or one gain (or real gain
>>>>      or attenuation).
>>>>
>>>>      This is the approach we had presented (with Luciano). Our plugin
>>>>      allowed to draw closed regions of arbitrary shape to be either
>>>>      filtered out or retained . The plugin was based on overlapping
>>>>      time-varying FFT filter. Whatever the case, the retained part has
>>>>      gain 1 and the rejected part has gain 0. The idea could be
>>>>      extended to support drawing with an "attenuating pencil" which
>>>>      would apply a given gain (picked from a gray shade map).
>>>>
>>>>      You mentioned Photosouder. I skimmed through the user guide and
>>>>      there is no hint as to what is actually doing when importing an
>>>>      image. What I'm pretty sure is that when editing an already loaded
>>>>      or computed spectrogram it does not re-synthesize the sound from
>>>>      the image, but processes the signal and recalculates the
>>>> spectrogram.
>>>>
>>>>      In other words, for an arbitrary image (say, a photo), it tries to
>>>>      represent it within the available color map and spectral and time
>>>>      resolution without concern for phase, but they have the advantage
>>>>      that we don't know a priori how the image "sounds"... And for a
>>>>      true spectrogram, it just simulates the image editing.
>>>>
>>>>      Regards,
>>>>
>>>>      Federico
>>>>
>>>>
>>>>
>>>>      On 06/10/2017 11:19, Steve the Fiddle wrote:
>>>>>      Hi Nikita,
>>>>>
>>>>>      Spectral Editing in Audacity is a fairly new, and not yet fully
>>>>>      developed, feature. Despite its shortcomings, it was considered
>>>>>      to still be useful enough to include in the Audacity release.
>>>>>
>>>>>      I like the idea of being able to export the spectrogram, edit in
>>>>>      an image editor, and then import back into Audacity. I presume
>>>>>      that what you have in mind is similar to the software
>>>>>      "Photosounder" (http://photosounder.com/). That would be a
>>>>>      totally new feature, with only superficial similarity to the
>>>>>      current "spectral edit" feature.
>>>>>
>>>>>      The current feature is really a 'simulation' of editing the
>>>>>      spectrogram, rather than actually editing it. How it works is
>>>>>      that the selected spectral region sets two frequency parameters
>>>>>      (high and low). Specially crafted effects (the "Spectral"
>>>>>      effects) can then use these parameters to control their
>>>>>      behaviour. The effects then process the audio data as would any
>>>>>      other audio effect. So the effect is not actually acting on the
>>>>>      spectrogram - it acts on the audio, and the spectrogram is
>>>>>      updated from the processed audio.
>>>>>
>>>>>      Steve
>>>>>
>>>>>      On 6 October 2017 at 13:38, Nikita Sadkov <[email protected]
>>>>>      <mailto:[email protected]>> wrote:
>>>>>
>>>>>          Hi! It was an amazing discovery, that Audacity now supports
>>>>>          spectral
>>>>>          editing - arguably one of the most fundamental features,
>>>>>          especially if
>>>>>          you want to synth custom sfx for your project or edit some
>>>>> badly
>>>>>          recorded material from https://freesound.org
>>>>>
>>>>>          My complain is that it is very obscure and tedious to use in
>>>>> its
>>>>>          current form. You have to separately enable spectral
>>>>>          selection and use
>>>>>          "Spectral edit ..." tools under "Effect" menu. I.e. you can't
>>>>>          just
>>>>>          press Del to clear the frequencies or use Draw Tool (F3) to
>>>>>          paint your
>>>>>          sfx with mouse.
>>>>>
>>>>>          So I think it would make Audacity order of magnitude more
>>>>>          useful, if
>>>>>          it copies UI from Sony SpectraLayers, together with layers,
>>>>> which
>>>>>          works just like a raster editor, but unfortunately lacks
>>>>> waveform
>>>>>          editing capabilities. Also, it would be nice feature to
>>>>> export
>>>>>          spectrum plot into png files, so people could edit it in GIMP
>>>>>          and then
>>>>>          import back into Audacity.
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>>          Check out the vibrant tech community on one of the world's
>>>>> most
>>>>>          engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>>          *********** ASKING FOR HELP *************
>>>>>
>>>>>          When asking for help on this list, please include the
>>>>>          following information, so we can
>>>>>          help you properly:
>>>>>
>>>>>          * What operating system you are using (for example, Windows
>>>>>          XP or Mac OS X 10.5.1)
>>>>>
>>>>>          * Exactly what three digit version number of Audacity you are
>>>>>          using (Help > About
>>>>>             Audacity, or Audacity >  About Audacity on a Mac computer)
>>>>>
>>>>>          * If this is a recording problem, what equipment you are
>>>>>          recording with, and how is it
>>>>>             connected to the computer?
>>>>>
>>>>>          Mailing list: [email protected]
>>>>>          <mailto:[email protected]>
>>>>>          To UNSUBSCRIBE, use the form at the bottom of this web page:
>>>>>          https://lists.sourceforge.net/lists/listinfo/audacity-users
>>>>>          <https://lists.sourceforge.net/lists/listinfo/audacity-users>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>>      Check out the vibrant tech community on one of the world's most
>>>>>      engaging tech sites, Slashdot.org!http://sdm.link/slashdot
>>>>>
>>>>>      *********** ASKING FOR HELP *************
>>>>>
>>>>>      When asking for help on this list, please include the following
>>>>> information, so we can
>>>>>      help you properly:
>>>>>
>>>>>      * What operating system you are using (for example, Windows XP or
>>>>> Mac
>>>>> OS X 10.5.1)
>>>>>
>>>>>      * Exactly what three digit version number of Audacity you are
>>>>> using
>>>>> (Help > About
>>>>>          Audacity, or Audacity >  About Audacity on a Mac computer)
>>>>>
>>>>>      * If this is a recording problem, what equipment you are
>>>>> recording
>>>>> with, and how is it
>>>>>          connected to the computer?
>>>>>
>>>>>      Mailing list:[email protected]
>>>>>      <mailto:[email protected]>
>>>>>      To UNSUBSCRIBE, use the form at the bottom of this web page:
>>>>>      https://lists.sourceforge.net/lists/listinfo/audacity-users
>>>>>      <https://lists.sourceforge.net/lists/listinfo/audacity-users>
>>>> ------------------------------------------------------------------------------
>>>>      Check out the vibrant tech community on one of the world's most
>>>>      engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>      *********** ASKING FOR HELP ************* When asking for help on
>>>>      this list, please include the following information, so we can
>>>>      help you properly: * What operating system you are using (for
>>>>      example, Windows XP or Mac OS X 10.5.1) * Exactly what three digit
>>>>      version number of Audacity you are using (Help > About
>>>>       Audacity, or Audacity >  About Audacity on a Mac computer) * If
>>>>      this is a recording problem, what equipment you are recording
>>>>      with, and how is it    connected to the computer? Mailing list:
>>>>      [email protected]
>>>>      <mailto:[email protected]> To UNSUBSCRIBE, use
>>>>      the form at the bottom of this web page:
>>>>      https://lists.sourceforge.net/lists/listinfo/audacity-users
>>>>      <https://lists.sourceforge.net/lists/listinfo/audacity-users>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>
>>>> *********** ASKING FOR HELP *************
>>>>
>>>> When asking for help on this list, please include the following
>>>> information, so we can
>>>> help you properly:
>>>>
>>>> * What operating system you are using (for example, Windows XP or Mac OS
>>>> X
>>>> 10.5.1)
>>>>
>>>> * Exactly what three digit version number of Audacity you are using
>>>> (Help
>>>>> About
>>>>      Audacity, or Audacity >  About Audacity on a Mac computer)
>>>>
>>>> * If this is a recording problem, what equipment you are recording
>>>> with,
>>>> and how is it
>>>>      connected to the computer?
>>>>
>>>> Mailing list: [email protected]
>>>> To UNSUBSCRIBE, use the form at the bottom of this web page:
>>>> https://lists.sourceforge.net/lists/listinfo/audacity-users
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> *********** ASKING FOR HELP *************
>>
>> When asking for help on this list, please include the following
>> information, so we can
>> help you properly:
>>
>> * What operating system you are using (for example, Windows XP or Mac OS X
>> 10.5.1)
>>
>> * Exactly what three digit version number of Audacity you are using (Help
>> > About
>>     Audacity, or Audacity >  About Audacity on a Mac computer)
>>
>> * If this is a recording problem, what equipment you are recording with,
>> and how is it
>>     connected to the computer?
>>
>> Mailing list: [email protected]
>> To UNSUBSCRIBE, use the form at the bottom of this web page:
>> https://lists.sourceforge.net/lists/listinfo/audacity-users
>>
>
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
*********** ASKING FOR HELP *************

When asking for help on this list, please include the following information, so we can
help you properly:

* What operating system you are using (for example, Windows XP or Mac OS X 10.5.1)

* Exactly what three digit version number of Audacity you are using (Help > About
   Audacity, or Audacity >  About Audacity on a Mac computer)

* If this is a recording problem, what equipment you are recording with, and how is it
   connected to the computer?

Mailing list: [email protected]
To UNSUBSCRIBE, use the form at the bottom of this web page:
https://lists.sourceforge.net/lists/listinfo/audacity-users