Re: [Csnd-dev] Unity

Do they plan to add support for the OnAudioFilterRead, is it on any
roadmaps? That's a fairly major part of things.

On Tue 18 Jun 2024, 12:25 Giovanni Bedetti, <[email protected]>
wrote:

> Thanks for your replies!
> Dr.B let's chat soon! I'll drop you an email ;)
>
> I'm getting closer! I was able to load Csound! Woho!
>
> I have created a repository with a test project and a build folder
> <https://github.com/giovannibedetti/TestCsoundUnityWebGL.git> if
> anybody wants to take a look.
>
> I grabbed the 6.18.7 dist/csound.js
> <https://unpkg.com/@csound/[email protected]/dist/csound.js> file as Steven
> suggested, renamed it to .jspre and placed it inside Assets/Plugins.
> If I build without any changes it builds ok, but as soon as the game
> starts on the browser I get this message:
> Unable to parse Build/TestBuild_0.framework.js! The file is corrupt, or
> compression was misconfigured? (check Content-Encoding HTTP Response Header
> on web server)
>
> I tried using the other extension, .jslib, and I got this error instead:
> Library/Bee/artifacts/WebGL/build/debug_WebGL_wasm/build.js: failure to
> execute js library "/.../TestCsoundUnityWebGL/Assets/Plugins/csound.jslib":
> SyntaxError: Unexpected token 'export',
>
> So I renamed the file back to csound.jspre, and commented the export lines
> at the end.
>
> const Csound = Csound$$module$src$index;
> Csound.toString = () => 'async (options) => CsoundObj;';
> //export { Csound }
> //export default Csound
> //# sourceMappingURL=csound.js.map
>
> It builds and runs ok, of course no Csound yet as nothing is calling it.
>
> So I added a .jslib file and wrote an initialize method (grabbed from the test.js
> in the 6.18.7 dist above
> <https://unpkg.com/browse/@csound/[email protected]/tests/tests.js>)
>
>
> mergeInto(LibraryManager.library, {
>
> csoundInitialize: async function(flags) {
> window.alert("csoundInitialize");
>
> const csoundVariations = [
> { useWorker: false, useSPN: false, name: "SINGLE THREAD, AW" },
> { useWorker: false, useSPN: true, name: "SINGLE THREAD, SPN" },
> { useWorker: true, useSAB: true, name: "WORKER, AW, SAB" },
> { useWorker: true, useSAB: false, name: "WORKER, AW, Messageport" },
> { useWorker: true, useSAB: false, useSPN: true, name: "WORKER, SPN,
> MessagePort" },
> ];
>
> //Csound.csoundInitialize(flags); // this doesn't work
>
> console.log("starting to await for Csound with flag: + flags")
> const cs = await Csound(csoundVariations[flags]);
> console.log(`Csound version: ${cs.name}`);
> const startReturn = await cs.start();
> console.log(startReturn);
> // await cs.stop();
> // cs.terminateInstance && (await cs.terminateInstance());
> },
> });
>
> Then call csoundInitialize from a C# script
> void Start()
> {
> Debug.Log("Calling csoundInitialize");
> csoundInitialize(0);
> }
>
> *and bang we got it!!*
>
> TestBuild_0.framework.js:913 Calling csoundInitialize
> TestBuild_0.framework.js:913 starting to await for Csound with flag: 0
> TestBuild_0.framework.js:913 Csound version: Csound: Audio Worklet,
> Single-threaded
> TestBuild_0.framework.js:383 --Csound version 6.18 (double samples) Feb 17
> 2023
> TestBuild_0.framework.js:383 [commit: HEAD]
> TestBuild_0.framework.js:383 libsndfile-1.1.0
> TestBuild_0.framework.js:383 graphics suppressed, ascii substituted
> TestBuild_0.framework.js:383 sr = 48000.0, kr = 3000.000, ksmps = 16
> TestBuild_0.framework.js:383 0dBFS level = 32768.0, A4 tuning = 440.0
> TestBuild_0.framework.js:383 orch now loaded
> TestBuild_0.framework.js:383 audio buffered in 256 sample-frame blocks
> TestBuild_0.framework.js:383 SECTION 1:
> TestBuild_0.framework.js:913 0
>
> Only the SINGLE_THREAD csoundVariations print the Csound version, for the
> other 3 it says undefined, but the rest of the logs are still printed (sr
> and kr could differ).
> Unity apparently doesn't support multi-threading on WebGL yet (this is an
> interesting post on the subject
> <https://forum.unity.com/threads/webgl-threading-roadmap.1403416/>), so
> workers don't work? :D
> I'll dig deeper inside this as soon as I have sound output.
>
> So let's get a bit into the details. It was "nice" that I was able to call
> an async method (csoundInitialize) from a non-async context (the Start()
> function).
> But of course the Start function is not waiting for it to complete before
> progressing.
>
> All of C# methods in the CsoundUnity package are synchronous, and
> sometimes called in a sequence.
> Have a look at the initialization sequence of the CsoundUnityBridge
> <https://github.com/rorywalsh/CsoundUnity/blob/e192bdd051eeb9434ec1f0dcdf6d7e03e89b2991/Runtime/CsoundUnityBridge.cs#L121C9-L170C51>
> So I guess that to integrate WebGL into the existing CsoundUnity package
> we will have to use a lot of directives to handle that and it could become
> hard to maintain.
> I mean, the way we use the API in C# is very different from what we should
> do with the CsoundObj API
> <https://unpkg.com/browse/@csound/[email protected]/README.md>.
> Furthermore, promises don't exist in the .NET environment out of the box,
> so we should rely on an external package to have them in C#.
> We could add callbacks on completion for the methods that need to be
> awaited but it doesn't look very stylish.
> Should the CsoundUnityWebGL be a completely different package?
>
> The other very important thing to keep in mind is that the
> OnAudioFilterRead function (what we use to do business in the Audio Thread,
> grabbing audio data from Csound and writing it into the AudioSource buffer
> - and the opposite) is not triggered at all on WebGL, and it means that it
> is not triggered on the editor too (I was very surprised to discover this).
> So you won't have the normal CsoundUnity behaviour on the Editor if the
> selected platform is WebGL. This means that the normal CsoundUnity workflow
> on the Editor on WebGL is not possible. I mean, the inspector with the
> control channels grabbed from the csd will still work, but pressing play on
> the Editor won't produce any sound.
> I'm thinking of workarounds but have no ideas at the moment.
>
> But for now let's focus on the fact that there was some progress! It's a
> win! :D
>
> I'll keep you posted as I continue to experiment with this,
> thanks again for now!
>
>
> Il giorno dom 16 giu 2024 alle ore 17:59 Steven Yi <[email protected]>
> ha scritto:
>
>> Webassembly Csound already compiles to a single JS file that embeds
>> the WASM binary into it. I assume this would work with Unity but am
>> unsure. I think you would only need the dist/csound.js file. You could
>> also try using the one from 6.18.7:
>>
>> https://unpkg.com/browse/@csound/[email protected]/
>>
>> It might be possible to rename it .jspre and create a test .jslib that
>> uses the Csound API.
>>
>> On Sat, Jun 15, 2024 at 2:02 PM Giovanni Bedetti
>> <[email protected]> wrote:
>> >
>> > Quick update on this: no luck yet!
>> >
>> > Sorry for the long email that follows. Any help would be much
>> appreciated!
>> >
>> >
>> > Unity uses Emscripten for the WebGL build.
>> >
>> > There are 2 options using Emscripten:
>> >
>> > Compile Csound as a static library to be used as Unity plug-in
>> > First thing I tried the old emscripten build for Web Csound from the
>> master branch (6.18.1), but I couldn't build it on macOs with this error
>> message:
>> >
>> > Error: Could not resolve './libcsound' from
>> src/CsoundScriptProcessorNode.js
>> >
>> >
>> > I also tried older versions of emscripten, but I cannot install the
>> ones older than version 2.0, so no luck there.
>> > This should be the preferred way of creating the required files for
>> Unity.
>> >
>> > The supported native plug-in file formats are
>> >
>> > LLVM Bitcode file format (.bc)
>> > GNU archive file format (.a)
>> >
>> > I noticed that the Csound wasm build was exporting a file with that
>> extension so I went this route.
>> > I was able to build the fix-csound-wasm branch (off CS7) and I can
>> confirm that it works on macOS (commit bac483d).
>> > I ran yarn build in the wasm folder.
>> > In the wasm/lib folder I could find a "libcsound-wasm.a" file, that
>> will be accepted by Unity as a native WebGL plug-in.
>> >
>> > I was hoping that the .a file would be enough, but it appears that I
>> cannot reach its methods from C#, I guess I need some js glue code with the
>> explicit exports.
>> > In fact, If I try to build the Unity project with only the above .a
>> file it builds ok, but as soon as I import the CsoundUnity package (with a
>> little change to try and load the library for WebGL) I get loads of build
>> errors (one for each native method we have, like "error: undefined symbol:
>> csoundAddSpinSample (referenced by top-level compiled C/C++ code").
>> >
>> > So I started to wonder if I needed the wasm/browser part to be able to
>> use the above library.
>> > I tried to build the browser package. First of all I ran yarn link
>> "@csound/wasm-bin" to link it with the previously built lib, then yarn
>> build.
>> > When built, it creates a "dist" folder, with several .js and .map files
>> with the name starting with "__compiled", like
>> "__compiled.vanilla.worker.js", "__compiled.vanilla.worker.inline.js",
>> "__compiled.vanilla.worker.js.map", together with
>> "__csound_wasm.inline.js", "csound.js" and "csound.js.map".
>> > I'm not sure how to import those files in Unity, I have very little
>> experience on the WebGL platform.
>> > Unity supports two JavaScript plug-in file types that let you add
>> JavaScript code to your Unity project
>> >
>> > .jslib A JavaScript library to use in addition to those in Emscripten’s
>> core libraries:
>> emscripten.org/docs/tools_reference/emcc.html#emcc-js-library
>> > .jspre A a file whose contents are added before the emitted code:
>> emscripten.org/docs/tools_reference/emcc.html#emcc-pre-js
>> >
>> > Furthermore, Unity expects this syntax to retrieve the native methods
>> that can be used in C#:
>> >
>> > const hello = {
>> >     Hello: function() {
>> >         window.alert("Hello, world!");
>> >     }
>> > };
>> > mergeInto(LibraryManager.library, hello);
>> >
>> > to be called in C# like this:
>> >
>> > [DllImport("__Internal")]
>> >     private static extern void Hello();
>> >
>> > Bundle the C/C++ code in the Unity project
>> > This option is very undocumented and there's no mention of any build
>> script or makefile to edit, so at the moment I have no idea how Csound will
>> be built by Unity.
>> > Out of curiosity, I tried to import some folders from the Csound repo:
>> > - Engine
>> > - H
>> > - include
>> > - InOut
>> > - interfaces
>> > - OOps
>> > - Opcodes
>> > - po
>> > - util
>> > - util1
>> > - util2
>> >
>> > When I build I get the following error, "Plugins colliding with each
>> other.":
>> >
>> > Plugin 'mp3dec_internal.h' is used from several locations:
>> >  Assets/Plugins/InOut/libmpadec/mp3dec_internal.h would be copied to
>> <PluginPath>/mp3dec_internal.h
>> >  Assets/Plugins/H/mp3dec_internal.h would be copied to
>> <PluginPath>/mp3dec_internal.h
>> > Plugin 'mpadec_internal.h' is used from several locations:
>> >  Assets/Plugins/H/mpadec_internal.h would be copied to
>> <PluginPath>/mpadec_internal.h
>> >  Assets/Plugins/InOut/libmpadec/mpadec_internal.h would be copied to
>> <PluginPath>/mpadec_internal.h
>> > Plugin 'stdopcod.h' is used from several locations:
>> >  Assets/Plugins/Opcodes/stdopcod.h would be copied to
>> <PluginPath>/stdopcod.h
>> >  Assets/Plugins/Opcodes/gab/stdopcod.h would be copied to
>> <PluginPath>/stdopcod.h
>> >
>> > I had a look at the two mp3dec_internal.h and they are identical except
>> for one line, the Assets/Plugins/InOut/libmpadec/mp3dec_internal.h has an
>> additional #include "mpadec_config.h".
>> > I'm not sure what to try next here, I guess removing one of the two
>> folders is too simple to be an option :D but maybe I'll try it later!
>> >
>> > There is also the option to use a pure Javascript plugin, but how to
>> create a javascript version of Csound?
>> >
>> > Last option, Unity native audio plugins, but they can't be used for
>> WebGL, see this comment on the official Native plugins repo
>> >
>> > Hope someone has some tips here, I'm lost in the Web(GL)!
>> >
>> >
>> > Il giorno ven 31 mag 2024 alle ore 10:07 Giovanni Bedetti <
>> [email protected]> ha scritto:
>> >>
>> >> Not yet, there are limitations in the Unity FMOD implementation for
>> the WebGL platform. Specifically, it doesn't support the OnAudioFilterRead
>> callback, which is the function we use to fill the Unity AudioSources
>> content with the Csound output.
>> >> Technically though it shouldn't be hard to import a Csound library for
>> the Web, and use the API. The difficulty is sending the Csound content to
>> the AudioSources, and so being able to move those sources in the 3D space
>> and having them spatialized (Unity spatializer is first order ambisonic).
>> >>
>> >> In the next days I will try something and report here my findings.
>> >>
>> >>
>> >> Il Ven 31 Mag 2024, 09:29 Michael Gogins <[email protected]>
>> ha scritto:
>> >>>
>> >>> Is there WebAssembly support for Csound in Unity yet?
>>
>