Re: [Csnd-dev] [EXTERNAL] [Csnd-dev] Unity

-------- Original Message --------
From: Giovanni Bedetti <[email protected]>
Sent: Fri Jun 21 11:50:33 GMT+01:00 2024
To: [email protected]
Subject: Re: [Csnd-dev] [EXTERNAL] [Csnd-dev] Unity

Thanks for the tutorials, they are of great help!
The fix to make it run on mobile was very easy and it was exactly what you
suggested.
I needed one more click to wake up the audio context, apparently it takes
some more time on Unity-mobile, so when Csound was being started the audio
context was not unlocked yet.
The build should now work on all mobile devices. If someone can test it on
an iPhone it would be great!
I also fixed the MIME warning on the server adding an .htaccess file with
this content:


# This configuration file should be uploaded to the server as "<Application
Folder>/Build/.htaccess" -> THIS IS WRONG, I HAD TO PUT THE FILE IN THE
APPLICATION FOLDER NOT IN THE BUILD FOLDER
# NOTE: "mod_mime" Apache module must be enabled for this configuration to
work.
<IfModule mod_mime.c>

# The following lines are required for builds without decompression
fallback, compressed with gzip
RemoveType .gz
AddEncoding gzip .gz
AddType application/gzip .data.gz # The correct MIME type here would be
application/octet-stream, but due to Safari bug
https://bugs.webkit.org/show_bug.cgi?id=247421, it's preferable to use MIME
Type application/gzip instead.
AddType application/wasm .wasm.gz
AddType application/javascript .js.gz
AddType application/octet-stream .symbols.json.gz

# The following lines are required for builds without decompression
fallback, compressed with Brotli
RemoveType .br
RemoveLanguage .br
AddEncoding br .br
AddType application/octet-stream .data.br
AddType application/wasm .wasm.br
AddType application/javascript .js.br
AddType application/octet-stream .symbols.json.br

# The following line improves loading performance for uncompressed builds
AddType application/wasm .wasm

# Uncomment the following line to improve loading performance for
gzip-compressed builds with decompression fallback
# AddEncoding gzip .unityweb

# Uncomment the following line to improve loading performance for
brotli-compressed builds with decompression fallback
# AddEncoding br .unityweb

</IfModule>


More info on servers at this link:
https://docs.unity3d.com/Manual/webgl-server-configuration-code-samples.html

Il giorno gio 20 giu 2024 alle ore 18:15 Victor Lazzarini <
[email protected]> ha scritto:

> yes, I got these in the past but I can't remember what I did to fix it. I
> know mobile phones needed some sort of interaction to wake up Webaudio, but
> I think you have that with the screen clicking.
>
> My simple tutorials (https://vlazzarini.github.io/vanilla/) make sound,
> so I am sure you can get your code to run as well.
>
> Prof. Victor Lazzarini
> Maynooth University
> Ireland
>
> On 20 Jun 2024, at 14:41, Giovanni Bedetti <[email protected]>
> wrote:
>
> 
> Yes unfortunately it doesn't work on mobile, the start promise is timed
> out.
>
> This is the log:
>
> TestBuild_0.framework.js:913 starting to await for Csound with flag: 0
> TestBuild_0.framework.js:913 Csound version: Csound: Audio Worklet,
> Single-threaded
> TestBuild_0.framework.js:383 --Csound version 6.18 (double samples) Feb 17
> 2023
> TestBuild_0.framework.js:383 [commit: HEAD]
> TestBuild_0.framework.js:383 libsndfile-1.1.0
> TestBuild_0.framework.js:383 graphics suppressed, ascii substituted
> TestBuild_0.framework.js:383 sr = 48000.0, kr = 3000.000, ksmps = 16
> TestBuild_0.framework.js:383 0dBFS level = 1.0, A4 tuning = 440.0
> TestBuild_0.framework.js:383 orch now loaded
> TestBuild_0.framework.js:383 audio buffered in 256 sample-frame blocks
> TestBuild_0.framework.js:383 SECTION 1:
> TestBuild_0.framework.js:374 start promise timed out
>
> This is the stack of the promise timeout error:
>
> (anonymous) @ TestBuild_0.framework.js:374
> setTimeout (async)
> (anonymous) @ TestBuild_0.framework.js:373
> createStartPromise @ TestBuild_0.framework.js:373
> a @ TestBuild_0.framework.js:898
> _csoundInitialize @ TestBuild_0.framework.js:913
> await in _csoundInitialize (async)
> $func33262 @ 0670e506:0x97e895
> $func34985 @ 0670e506:0x9abd39
> $func36433 @ 0670e506:0x9c759b
> $dynCall_iiii @ 0670e506:0x170a42a
> invoke_iiii @ TestBuild_0.framework.js:913
> $func36417 @ 0670e506:0x9c6632
> $func1111 @ 0670e506:0xa6bf1
> $func60024 @ 0670e506:0x16c471c
> $func57642 @ 0670e506:0x1674cc4
> $func38105 @ 0670e506:0xaabc30
> $func38106 @ 0670e506:0xaabc45
> $func57359 @ 0670e506:0x1664efe
> $func57067 @ 0670e506:0x163f9af
> $func38930 @ 0670e506:0xaf67e8
> $func38930 @ 0670e506:0xaf685c
> $func60744 @ 0670e506:0x16f7e08
> $dynCall_v @ 0670e506:0x170a466
> (anonymous) @ TestBuild_0.framework.js:913
> _JS_CallAsLongAsNoExceptionsSeen @ TestBuild_0.framework.js:913
> $func60497 @ 0670e506:0x16e3bc4
> $dynCall_iiii @ 0670e506:0x170a42a
> (anonymous) @ TestBuild_0.framework.js:913
> touchEventHandlerFunc @ TestBuild_0.framework.js:913
> jsEventHandler @ TestBuild_0.framework.js:913
>
> What could be the cause?
>
> About “HTTPS response Header”, I will investigate the cause, it should
> have no bad effect though.
>
> Il giorno gio 20 giu 2024 alle ore 14:36 Victor Lazzarini <
> [email protected]> ha scritto:
>
>> Tested it here on the Mac and the interaction is working well. On the
>> mobile, I guess the touch screen interaction does not replace
>> the mouse. I get the csoundInitialize dialogue, but that’s about it.
>>
>> On both I get this “HTTPS response Header” message, but that goes away
>> after a while.
>>
>> ========================
>> Prof. Victor Lazzarini
>> Maynooth University
>> Ireland
>>
>> > On 20 Jun 2024, at 11:45, Giovanni Bedetti <[email protected]>
>> wrote:
>> >
>> > *Warning*
>> > This email originated from outside of Maynooth University's Mail
>> System. Do not reply, click links or open attachments unless you recognise
>> the sender and know the content is safe.
>> > I have updated the repo and the build, same links as above.
>> > I have been able to create multiple instances of Csound from Unity,
>> reference a specific instance and set a channel there, so I am making good
>> progress!
>> > I had to rely on callbacks so yes the method signature (and workflow)
>> will probably be very different from the original CsoundUnity
>> implementation.
>> > At the moment there is no sound at all on the editor, so you'll have to
>> build every time after a change.
>> >
>> >
>> >
>> >
>> > Il giorno mer 19 giu 2024 alle ore 03:13 Dr. Richard Boulanger <
>> [email protected]> ha scritto:
>> > I too heard the sine tone in my browser - nice.
>> > - now - a button and a fader in the browser will be something!
>> >
>> > - Dr.B
>> >
>> > Dr. Richard Boulanger
>> > Professor
>> > Electronic Production and Design
>> > Berklee College of Music
>> > Professional Writing & Technology Division
>> >
>> >
>> > On Tue, Jun 18, 2024 at 9:12 PM Dr. Richard Boulanger <
>> [email protected]> wrote:
>> > wow - you are getting there.  this will be awesome too - an important
>> next step.
>> >
>> > - Dr.B
>> >
>> > Dr. Richard Boulanger
>> > Professor
>> > Electronic Production and Design
>> > Berklee College of Music
>> > Professional Writing & Technology Division
>> >
>> >
>> > On Tue, Jun 18, 2024 at 1:36 PM Giovanni Bedetti <
>> [email protected]> wrote:
>> > I made some progress, we have sound!
>> > Check it out in the repo.
>> > At this link you can test the working build! (it's safe I swear)
>> >
>> >
>> >
>> > Il giorno mar 18 giu 2024 alle ore 13:56 Giovanni Bedetti <
>> [email protected]> ha scritto:
>> > I have no idea, this is what the docs say about this, no mention about
>> future plans anywhere:
>> > Because Unity uses FMODto manage audio for platforms, the Web platform
>> supports limited audio functionality, which only includes the basic
>> features. FMOD relies on threads, which the WebGL API doesn’t support. For
>> this reason, Unity uses an implementation based on the internal Web Audio
>> API, which enables the browser to handle audio playback and mixing.
>> >
>> > Il Mar 18 Giu 2024, 13:41 Rory Walsh <[email protected]> ha scritto:
>> > Do they plan to add support for the OnAudioFilterRead, is it on any
>> roadmaps? That's a fairly major part of things.
>> >
>> > On Tue 18 Jun 2024, 12:25 Giovanni Bedetti, <[email protected]>
>> wrote:
>> > Thanks for your replies!
>> > Dr.B let's chat soon! I'll drop you an email ;)
>> >
>> > I'm getting closer! I was able to load Csound! Woho!
>> >
>> > I have created a repository with a test project and a build folder if
>> anybody wants to take a look.
>> >
>> > I grabbed the 6.18.7 dist/csound.js file as Steven suggested, renamed
>> it to .jspre and placed it inside Assets/Plugins.
>> > If I build without any changes it builds ok, but as soon as the game
>> starts on the browser I get this message:
>> > Unable to parse Build/TestBuild_0.framework.js! The file is corrupt, or
>> compression was misconfigured? (check Content-Encoding HTTP Response Header
>> on web server)
>> >
>> > I tried using the other extension, .jslib, and I got this error instead:
>> > Library/Bee/artifacts/WebGL/build/debug_WebGL_wasm/build.js: failure to
>> execute js library "/.../TestCsoundUnityWebGL/Assets/Plugins/csound.jslib":
>> SyntaxError: Unexpected token 'export',
>> >
>> > So I renamed the file back to csound.jspre, and commented the export
>> lines at the end.
>> >
>> > const Csound = Csound$$module$src$index;
>> > Csound.toString = () => 'async (options) => CsoundObj;';
>> > //export { Csound }
>> > //export default Csound
>> > //# sourceMappingURL=csound.js.map
>> >
>> > It builds and runs ok, of course no Csound yet as nothing is calling it.
>> >
>> > So I added a .jslib file and wrote an initialize method (grabbed from
>> the test.js in the 6.18.7 dist above)
>> >
>> >
>> > mergeInto(LibraryManager.library, {
>> >
>> > csoundInitialize: async function(flags) {
>> > window.alert("csoundInitialize");
>> >
>> > const csoundVariations = [
>> > { useWorker: false, useSPN: false, name: "SINGLE THREAD, AW" },
>> > { useWorker: false, useSPN: true, name: "SINGLE THREAD, SPN" },
>> > { useWorker: true, useSAB: true, name: "WORKER, AW, SAB" },
>> > { useWorker: true, useSAB: false, name: "WORKER, AW, Messageport" },
>> > { useWorker: true, useSAB: false, useSPN: true, name: "WORKER, SPN,
>> MessagePort" },
>> > ];
>> >
>> > //Csound.csoundInitialize(flags); // this doesn't work
>> > console.log("starting to await for Csound with flag: + flags")
>> > const cs = await Csound(csoundVariations[flags]);
>> > console.log(`Csound version: ${cs.name}`);
>> > const startReturn = await cs.start();
>> > console.log(startReturn);
>> > // await cs.stop(); // cs.terminateInstance && (await
>> cs.terminateInstance()); },
>> > });
>> >
>> > Then call csoundInitialize from a C# script
>> > void Start()
>> > {
>> > Debug.Log("Calling csoundInitialize");
>> > csoundInitialize(0);
>> > }
>> >
>> > and bang we got it!!
>> >
>> > TestBuild_0.framework.js:913 Calling csoundInitialize
>> > TestBuild_0.framework.js:913 starting to await for Csound with flag: 0
>> > TestBuild_0.framework.js:913 Csound version: Csound: Audio Worklet,
>> Single-threaded
>> > TestBuild_0.framework.js:383 --Csound version 6.18 (double samples) Feb
>> 17 2023
>> > TestBuild_0.framework.js:383 [commit: HEAD]
>> > TestBuild_0.framework.js:383 libsndfile-1.1.0
>> > TestBuild_0.framework.js:383 graphics suppressed, ascii substituted
>> > TestBuild_0.framework.js:383 sr = 48000.0, kr = 3000.000, ksmps = 16
>> > TestBuild_0.framework.js:383 0dBFS level = 32768.0, A4 tuning = 440.0
>> > TestBuild_0.framework.js:383 orch now loaded
>> > TestBuild_0.framework.js:383 audio buffered in 256 sample-frame blocks
>> > TestBuild_0.framework.js:383 SECTION 1:
>> > TestBuild_0.framework.js:913 0
>> >
>> > Only the SINGLE_THREAD csoundVariations print the Csound version, for
>> the other 3 it says undefined, but the rest of the logs are still printed
>> (sr and kr could differ).
>> > Unity apparently doesn't support multi-threading on WebGL yet (this is
>> an interesting post on the subject), so workers don't work? :D
>> > I'll dig deeper inside this as soon as I have sound output.
>> >
>> > So let's get a bit into the details. It was "nice" that I was able to
>> call an async method (csoundInitialize) from a non-async context (the
>> Start() function).
>> > But of course the Start function is not waiting for it to complete
>> before progressing.
>> >
>> > All of C# methods in the CsoundUnity package are synchronous, and
>> sometimes called in a sequence.
>> > Have a look at the initialization sequence of the CsoundUnityBridge
>> > So I guess that to integrate WebGL into the existing CsoundUnity
>> package we will have to use a lot of directives to handle that and it could
>> become hard to maintain.
>> > I mean, the way we use the API in C# is very different from what we
>> should do with the CsoundObj API.
>> > Furthermore, promises don't exist in the .NET environment out of the
>> box, so we should rely on an external package to have them in C#.
>> > We could add callbacks on completion for the methods that need to be
>> awaited but it doesn't look very stylish.
>> > Should the CsoundUnityWebGL be a completely different package?
>> >
>> > The other very important thing to keep in mind is that the
>> OnAudioFilterRead function (what we use to do business in the Audio Thread,
>> grabbing audio data from Csound and writing it into the AudioSource buffer
>> - and the opposite) is not triggered at all on WebGL, and it means that it
>> is not triggered on the editor too (I was very surprised to discover this).
>> > So you won't have the normal CsoundUnity behaviour on the Editor if the
>> selected platform is WebGL. This means that the normal CsoundUnity workflow
>> on the Editor on WebGL is not possible. I mean, the inspector with the
>> control channels grabbed from the csd will still work, but pressing play on
>> the Editor won't produce any sound.
>> > I'm thinking of workarounds but have no ideas at the moment.
>> >
>> > But for now let's focus on the fact that there was some progress! It's
>> a win! :D
>> >
>> > I'll keep you posted as I continue to experiment with this,
>> > thanks again for now!
>> >
>> >
>> > Il giorno dom 16 giu 2024 alle ore 17:59 Steven Yi <[email protected]>
>> ha scritto:
>> > Webassembly Csound already compiles to a single JS file that embeds
>> > the WASM binary into it. I assume this would work with Unity but am
>> > unsure. I think you would only need the dist/csound.js file. You could
>> > also try using the one from 6.18.7:
>> >
>> > https://unpkg.com/browse/@csound/[email protected]/
>> >
>> > It might be possible to rename it .jspre and create a test .jslib that
>> > uses the Csound API.
>> >
>> > On Sat, Jun 15, 2024 at 2:02 PM Giovanni Bedetti
>> > <[email protected]> wrote:
>> > >
>> > > Quick update on this: no luck yet!
>> > >
>> > > Sorry for the long email that follows. Any help would be much
>> appreciated!
>> > >
>> > >
>> > > Unity uses Emscripten for the WebGL build.
>> > >
>> > > There are 2 options using Emscripten:
>> > >
>> > > Compile Csound as a static library to be used as Unity plug-in
>> > > First thing I tried the old emscripten build for Web Csound from the
>> master branch (6.18.1), but I couldn't build it on macOs with this error
>> message:
>> > >
>> > > Error: Could not resolve './libcsound' from
>> src/CsoundScriptProcessorNode.js
>> > >
>> > >
>> > > I also tried older versions of emscripten, but I cannot install the
>> ones older than version 2.0, so no luck there.
>> > > This should be the preferred way of creating the required files for
>> Unity.
>> > >
>> > > The supported native plug-in file formats are
>> > >
>> > > LLVM Bitcode file format (.bc)
>> > > GNU archive file format (.a)
>> > >
>> > > I noticed that the Csound wasm build was exporting a file with that
>> extension so I went this route.
>> > > I was able to build the fix-csound-wasm branch (off CS7) and I can
>> confirm that it works on macOS (commit bac483d).
>> > > I ran yarn build in the wasm folder.
>> > > In the wasm/lib folder I could find a "libcsound-wasm.a" file, that
>> will be accepted by Unity as a native WebGL plug-in.
>> > >
>> > > I was hoping that the .a file would be enough, but it appears that I
>> cannot reach its methods from C#, I guess I need some js glue code with the
>> explicit exports.
>> > > In fact, If I try to build the Unity project with only the above .a
>> file it builds ok, but as soon as I import the CsoundUnity package (with a
>> little change to try and load the library for WebGL) I get loads of build
>> errors (one for each native method we have, like "error: undefined symbol:
>> csoundAddSpinSample (referenced by top-level compiled C/C++ code").
>> > >
>> > > So I started to wonder if I needed the wasm/browser part to be able
>> to use the above library.
>> > > I tried to build the browser package. First of all I ran yarn link
>> "@csound/wasm-bin" to link it with the previously built lib, then yarn
>> build.
>> > > When built, it creates a "dist" folder, with several .js and .map
>> files with the name starting with "__compiled", like
>> "__compiled.vanilla.worker.js", "__compiled.vanilla.worker.inline.js",
>> "__compiled.vanilla.worker.js.map", together with
>> "__csound_wasm.inline.js", "csound.js" and "csound.js.map".
>> > > I'm not sure how to import those files in Unity, I have very little
>> experience on the WebGL platform.
>> > > Unity supports two JavaScript plug-in file types that let you add
>> JavaScript code to your Unity project
>> > >
>> > > .jslib A JavaScript library to use in addition to those in
>> Emscripten’s core libraries:
>> emscripten.org/docs/tools_reference/emcc.html#emcc-js-library
>> > > .jspre A a file whose contents are added before the emitted code:
>> emscripten.org/docs/tools_reference/emcc.html#emcc-pre-js
>> > >
>> > > Furthermore, Unity expects this syntax to retrieve the native methods
>> that can be used in C#:
>> > >
>> > > const hello = {
>> > >     Hello: function() {
>> > >         window.alert("Hello, world!");
>> > >     }
>> > > };
>> > > mergeInto(LibraryManager.library, hello);
>> > >
>> > > to be called in C# like this:
>> > >
>> > > [DllImport("__Internal")]
>> > >     private static extern void Hello();
>> > >
>> > > Bundle the C/C++ code in the Unity project
>> > > This option is very undocumented and there's no mention of any build
>> script or makefile to edit, so at the moment I have no idea how Csound will
>> be built by Unity.
>> > > Out of curiosity, I tried to import some folders from the Csound repo:
>> > > - Engine
>> > > - H
>> > > - include
>> > > - InOut
>> > > - interfaces
>> > > - OOps
>> > > - Opcodes
>> > > - po
>> > > - util
>> > > - util1
>> > > - util2
>> > >
>> > > When I build I get the following error, "Plugins colliding with each
>> other.":
>> > >
>> > > Plugin 'mp3dec_internal.h' is used from several locations:
>> > >  Assets/Plugins/InOut/libmpadec/mp3dec_internal.h would be copied to
>> <PluginPath>/mp3dec_internal.h
>> > >  Assets/Plugins/H/mp3dec_internal.h would be copied to
>> <PluginPath>/mp3dec_internal.h
>> > > Plugin 'mpadec_internal.h' is used from several locations:
>> > >  Assets/Plugins/H/mpadec_internal.h would be copied to
>> <PluginPath>/mpadec_internal.h
>> > >  Assets/Plugins/InOut/libmpadec/mpadec_internal.h would be copied to
>> <PluginPath>/mpadec_internal.h
>> > > Plugin 'stdopcod.h' is used from several locations:
>> > >  Assets/Plugins/Opcodes/stdopcod.h would be copied to
>> <PluginPath>/stdopcod.h
>> > >  Assets/Plugins/Opcodes/gab/stdopcod.h would be copied to
>> <PluginPath>/stdopcod.h
>> > >
>> > > I had a look at the two mp3dec_internal.h and they are identical
>> except for one line, the Assets/Plugins/InOut/libmpadec/mp3dec_internal.h
>> has an additional #include "mpadec_config.h".
>> > > I'm not sure what to try next here, I guess removing one of the two
>> folders is too simple to be an option :D but maybe I'll try it later!
>> > >
>> > > There is also the option to use a pure Javascript plugin, but how to
>> create a javascript version of Csound?
>> > >
>> > > Last option, Unity native audio plugins, but they can't be used for
>> WebGL, see this comment on the official Native plugins repo
>> > >
>> > > Hope someone has some tips here, I'm lost in the Web(GL)!
>> > >
>> > >
>> > > Il giorno ven 31 mag 2024 alle ore 10:07 Giovanni Bedetti <
>> [email protected]> ha scritto:
>> > >>
>> > >> Not yet, there are limitations in the Unity FMOD implementation for
>> the WebGL platform. Specifically, it doesn't support the OnAudioFilterRead
>> callback, which is the function we use to fill the Unity AudioSources
>> content with the Csound output.
>> > >> Technically though it shouldn't be hard to import a Csound library
>> for the Web, and use the API. The difficulty is sending the Csound content
>> to the AudioSources, and so being able to move those sources in the 3D
>> space and having them spatialized (Unity spatializer is first order
>> ambisonic).
>> > >>
>> > >> In the next days I will try something and report here my findings.
>> > >>
>> > >>
>> > >> Il Ven 31 Mag 2024, 09:29 Michael Gogins <[email protected]>
>> ha scritto:
>> > >>>
>> > >>> Is there WebAssembly support for Csound in Unity yet?
>>
>>