Re: [Csnd-dev] Unity

I made some progress, we have sound!
Check it out in the repo
<https://github.com/giovannibedetti/TestCsoundUnityWebGL.git>.
At this link you can test the working build!
<https://www.giovannibedetti.com/csound/csound_unity_webgl/TestBuild_0/index.html>
(it's
safe I swear)



Il giorno mar 18 giu 2024 alle ore 13:56 Giovanni Bedetti <
[email protected]> ha scritto:

> I have no idea, this is what the docs say about this, no mention about
> future plans anywhere:
>
> Because Unity uses FMOD <https://www.fmod.com/>to manage audio for
> platforms, the Web platform supports limited audio functionality, which
> only includes the basic features. FMOD relies on threads, which the WebGL API
> doesn’t support. For this reason, Unity uses an implementation based on the
> internal Web Audio API, which enables the browser to handle audio playback
> and mixing.
>
> Il Mar 18 Giu 2024, 13:41 Rory Walsh <[email protected]> ha scritto:
>
>> Do they plan to add support for the OnAudioFilterRead, is it on any
>> roadmaps? That's a fairly major part of things.
>>
>> On Tue 18 Jun 2024, 12:25 Giovanni Bedetti, <[email protected]>
>> wrote:
>>
>>> Thanks for your replies!
>>> Dr.B let's chat soon! I'll drop you an email ;)
>>>
>>> I'm getting closer! I was able to load Csound! Woho!
>>>
>>> I have created a repository with a test project and a build folder
>>> <https://github.com/giovannibedetti/TestCsoundUnityWebGL.git> if
>>> anybody wants to take a look.
>>>
>>> I grabbed the 6.18.7 dist/csound.js
>>> <https://unpkg.com/@csound/[email protected]/dist/csound.js> file as
>>> Steven suggested, renamed it to .jspre and placed it inside Assets/Plugins.
>>> If I build without any changes it builds ok, but as soon as the game
>>> starts on the browser I get this message:
>>> Unable to parse Build/TestBuild_0.framework.js! The file is corrupt, or
>>> compression was misconfigured? (check Content-Encoding HTTP Response Header
>>> on web server)
>>>
>>> I tried using the other extension, .jslib, and I got this error instead:
>>> Library/Bee/artifacts/WebGL/build/debug_WebGL_wasm/build.js: failure to
>>> execute js library "/.../TestCsoundUnityWebGL/Assets/Plugins/csound.jslib":
>>> SyntaxError: Unexpected token 'export',
>>>
>>> So I renamed the file back to csound.jspre, and commented the export
>>> lines at the end.
>>>
>>> const Csound = Csound$$module$src$index;
>>> Csound.toString = () => 'async (options) => CsoundObj;';
>>> //export { Csound }
>>> //export default Csound
>>> //# sourceMappingURL=csound.js.map
>>>
>>> It builds and runs ok, of course no Csound yet as nothing is calling it.
>>>
>>> So I added a .jslib file and wrote an initialize method (grabbed from
>>> the test.js in the 6.18.7 dist above
>>> <https://unpkg.com/browse/@csound/[email protected]/tests/tests.js>)
>>>
>>>
>>> mergeInto(LibraryManager.library, {
>>>
>>> csoundInitialize: async function(flags) {
>>> window.alert("csoundInitialize");
>>>
>>> const csoundVariations = [
>>> { useWorker: false, useSPN: false, name: "SINGLE THREAD, AW" },
>>> { useWorker: false, useSPN: true, name: "SINGLE THREAD, SPN" },
>>> { useWorker: true, useSAB: true, name: "WORKER, AW, SAB" },
>>> { useWorker: true, useSAB: false, name: "WORKER, AW, Messageport" },
>>> { useWorker: true, useSAB: false, useSPN: true, name: "WORKER, SPN,
>>> MessagePort" },
>>> ];
>>>
>>> //Csound.csoundInitialize(flags); // this doesn't work
>>>
>>> console.log("starting to await for Csound with flag: + flags")
>>> const cs = await Csound(csoundVariations[flags]);
>>> console.log(`Csound version: ${cs.name}`);
>>> const startReturn = await cs.start();
>>> console.log(startReturn);
>>> // await cs.stop();
>>> // cs.terminateInstance && (await cs.terminateInstance());
>>> },
>>> });
>>>
>>> Then call csoundInitialize from a C# script
>>> void Start()
>>> {
>>> Debug.Log("Calling csoundInitialize");
>>> csoundInitialize(0);
>>> }
>>>
>>> *and bang we got it!!*
>>>
>>> TestBuild_0.framework.js:913 Calling csoundInitialize
>>> TestBuild_0.framework.js:913 starting to await for Csound with flag: 0
>>> TestBuild_0.framework.js:913 Csound version: Csound: Audio Worklet,
>>> Single-threaded
>>> TestBuild_0.framework.js:383 --Csound version 6.18 (double samples) Feb
>>> 17 2023
>>> TestBuild_0.framework.js:383 [commit: HEAD]
>>> TestBuild_0.framework.js:383 libsndfile-1.1.0
>>> TestBuild_0.framework.js:383 graphics suppressed, ascii substituted
>>> TestBuild_0.framework.js:383 sr = 48000.0, kr = 3000.000, ksmps = 16
>>> TestBuild_0.framework.js:383 0dBFS level = 32768.0, A4 tuning = 440.0
>>> TestBuild_0.framework.js:383 orch now loaded
>>> TestBuild_0.framework.js:383 audio buffered in 256 sample-frame blocks
>>> TestBuild_0.framework.js:383 SECTION 1:
>>> TestBuild_0.framework.js:913 0
>>>
>>> Only the SINGLE_THREAD csoundVariations print the Csound version, for
>>> the other 3 it says undefined, but the rest of the logs are still printed
>>> (sr and kr could differ).
>>> Unity apparently doesn't support multi-threading on WebGL yet (this is
>>> an interesting post on the subject
>>> <https://forum.unity.com/threads/webgl-threading-roadmap.1403416/>), so
>>> workers don't work? :D
>>> I'll dig deeper inside this as soon as I have sound output.
>>>
>>> So let's get a bit into the details. It was "nice" that I was able to
>>> call an async method (csoundInitialize) from a non-async context (the
>>> Start() function).
>>> But of course the Start function is not waiting for it to complete
>>> before progressing.
>>>
>>> All of C# methods in the CsoundUnity package are synchronous, and
>>> sometimes called in a sequence.
>>> Have a look at the initialization sequence of the CsoundUnityBridge
>>> <https://github.com/rorywalsh/CsoundUnity/blob/e192bdd051eeb9434ec1f0dcdf6d7e03e89b2991/Runtime/CsoundUnityBridge.cs#L121C9-L170C51>
>>> So I guess that to integrate WebGL into the existing CsoundUnity package
>>> we will have to use a lot of directives to handle that and it could become
>>> hard to maintain.
>>> I mean, the way we use the API in C# is very different from what we
>>> should do with the CsoundObj API
>>> <https://unpkg.com/browse/@csound/[email protected]/README.md>.
>>> Furthermore, promises don't exist in the .NET environment out of the
>>> box, so we should rely on an external package to have them in C#.
>>> We could add callbacks on completion for the methods that need to be
>>> awaited but it doesn't look very stylish.
>>> Should the CsoundUnityWebGL be a completely different package?
>>>
>>> The other very important thing to keep in mind is that the
>>> OnAudioFilterRead function (what we use to do business in the Audio Thread,
>>> grabbing audio data from Csound and writing it into the AudioSource buffer
>>> - and the opposite) is not triggered at all on WebGL, and it means that it
>>> is not triggered on the editor too (I was very surprised to discover this).
>>> So you won't have the normal CsoundUnity behaviour on the Editor if the
>>> selected platform is WebGL. This means that the normal CsoundUnity workflow
>>> on the Editor on WebGL is not possible. I mean, the inspector with the
>>> control channels grabbed from the csd will still work, but pressing play on
>>> the Editor won't produce any sound.
>>> I'm thinking of workarounds but have no ideas at the moment.
>>>
>>> But for now let's focus on the fact that there was some progress! It's a
>>> win! :D
>>>
>>> I'll keep you posted as I continue to experiment with this,
>>> thanks again for now!
>>>
>>>
>>> Il giorno dom 16 giu 2024 alle ore 17:59 Steven Yi <[email protected]>
>>> ha scritto:
>>>
>>>> Webassembly Csound already compiles to a single JS file that embeds
>>>> the WASM binary into it. I assume this would work with Unity but am
>>>> unsure. I think you would only need the dist/csound.js file. You could
>>>> also try using the one from 6.18.7:
>>>>
>>>> https://unpkg.com/browse/@csound/[email protected]/
>>>>
>>>> It might be possible to rename it .jspre and create a test .jslib that
>>>> uses the Csound API.
>>>>
>>>> On Sat, Jun 15, 2024 at 2:02 PM Giovanni Bedetti
>>>> <[email protected]> wrote:
>>>> >
>>>> > Quick update on this: no luck yet!
>>>> >
>>>> > Sorry for the long email that follows. Any help would be much
>>>> appreciated!
>>>> >
>>>> >
>>>> > Unity uses Emscripten for the WebGL build.
>>>> >
>>>> > There are 2 options using Emscripten:
>>>> >
>>>> > Compile Csound as a static library to be used as Unity plug-in
>>>> > First thing I tried the old emscripten build for Web Csound from the
>>>> master branch (6.18.1), but I couldn't build it on macOs with this error
>>>> message:
>>>> >
>>>> > Error: Could not resolve './libcsound' from
>>>> src/CsoundScriptProcessorNode.js
>>>> >
>>>> >
>>>> > I also tried older versions of emscripten, but I cannot install the
>>>> ones older than version 2.0, so no luck there.
>>>> > This should be the preferred way of creating the required files for
>>>> Unity.
>>>> >
>>>> > The supported native plug-in file formats are
>>>> >
>>>> > LLVM Bitcode file format (.bc)
>>>> > GNU archive file format (.a)
>>>> >
>>>> > I noticed that the Csound wasm build was exporting a file with that
>>>> extension so I went this route.
>>>> > I was able to build the fix-csound-wasm branch (off CS7) and I can
>>>> confirm that it works on macOS (commit bac483d).
>>>> > I ran yarn build in the wasm folder.
>>>> > In the wasm/lib folder I could find a "libcsound-wasm.a" file, that
>>>> will be accepted by Unity as a native WebGL plug-in.
>>>> >
>>>> > I was hoping that the .a file would be enough, but it appears that I
>>>> cannot reach its methods from C#, I guess I need some js glue code with the
>>>> explicit exports.
>>>> > In fact, If I try to build the Unity project with only the above .a
>>>> file it builds ok, but as soon as I import the CsoundUnity package (with a
>>>> little change to try and load the library for WebGL) I get loads of build
>>>> errors (one for each native method we have, like "error: undefined symbol:
>>>> csoundAddSpinSample (referenced by top-level compiled C/C++ code").
>>>> >
>>>> > So I started to wonder if I needed the wasm/browser part to be able
>>>> to use the above library.
>>>> > I tried to build the browser package. First of all I ran yarn link
>>>> "@csound/wasm-bin" to link it with the previously built lib, then yarn
>>>> build.
>>>> > When built, it creates a "dist" folder, with several .js and .map
>>>> files with the name starting with "__compiled", like
>>>> "__compiled.vanilla.worker.js", "__compiled.vanilla.worker.inline.js",
>>>> "__compiled.vanilla.worker.js.map", together with
>>>> "__csound_wasm.inline.js", "csound.js" and "csound.js.map".
>>>> > I'm not sure how to import those files in Unity, I have very little
>>>> experience on the WebGL platform.
>>>> > Unity supports two JavaScript plug-in file types that let you add
>>>> JavaScript code to your Unity project
>>>> >
>>>> > .jslib A JavaScript library to use in addition to those in
>>>> Emscripten’s core libraries:
>>>> emscripten.org/docs/tools_reference/emcc.html#emcc-js-library
>>>> > .jspre A a file whose contents are added before the emitted code:
>>>> emscripten.org/docs/tools_reference/emcc.html#emcc-pre-js
>>>> >
>>>> > Furthermore, Unity expects this syntax to retrieve the native methods
>>>> that can be used in C#:
>>>> >
>>>> > const hello = {
>>>> >     Hello: function() {
>>>> >         window.alert("Hello, world!");
>>>> >     }
>>>> > };
>>>> > mergeInto(LibraryManager.library, hello);
>>>> >
>>>> > to be called in C# like this:
>>>> >
>>>> > [DllImport("__Internal")]
>>>> >     private static extern void Hello();
>>>> >
>>>> > Bundle the C/C++ code in the Unity project
>>>> > This option is very undocumented and there's no mention of any build
>>>> script or makefile to edit, so at the moment I have no idea how Csound will
>>>> be built by Unity.
>>>> > Out of curiosity, I tried to import some folders from the Csound repo:
>>>> > - Engine
>>>> > - H
>>>> > - include
>>>> > - InOut
>>>> > - interfaces
>>>> > - OOps
>>>> > - Opcodes
>>>> > - po
>>>> > - util
>>>> > - util1
>>>> > - util2
>>>> >
>>>> > When I build I get the following error, "Plugins colliding with each
>>>> other.":
>>>> >
>>>> > Plugin 'mp3dec_internal.h' is used from several locations:
>>>> >  Assets/Plugins/InOut/libmpadec/mp3dec_internal.h would be copied to
>>>> <PluginPath>/mp3dec_internal.h
>>>> >  Assets/Plugins/H/mp3dec_internal.h would be copied to
>>>> <PluginPath>/mp3dec_internal.h
>>>> > Plugin 'mpadec_internal.h' is used from several locations:
>>>> >  Assets/Plugins/H/mpadec_internal.h would be copied to
>>>> <PluginPath>/mpadec_internal.h
>>>> >  Assets/Plugins/InOut/libmpadec/mpadec_internal.h would be copied to
>>>> <PluginPath>/mpadec_internal.h
>>>> > Plugin 'stdopcod.h' is used from several locations:
>>>> >  Assets/Plugins/Opcodes/stdopcod.h would be copied to
>>>> <PluginPath>/stdopcod.h
>>>> >  Assets/Plugins/Opcodes/gab/stdopcod.h would be copied to
>>>> <PluginPath>/stdopcod.h
>>>> >
>>>> > I had a look at the two mp3dec_internal.h and they are identical
>>>> except for one line, the Assets/Plugins/InOut/libmpadec/mp3dec_internal.h
>>>> has an additional #include "mpadec_config.h".
>>>> > I'm not sure what to try next here, I guess removing one of the two
>>>> folders is too simple to be an option :D but maybe I'll try it later!
>>>> >
>>>> > There is also the option to use a pure Javascript plugin, but how to
>>>> create a javascript version of Csound?
>>>> >
>>>> > Last option, Unity native audio plugins, but they can't be used for
>>>> WebGL, see this comment on the official Native plugins repo
>>>> >
>>>> > Hope someone has some tips here, I'm lost in the Web(GL)!
>>>> >
>>>> >
>>>> > Il giorno ven 31 mag 2024 alle ore 10:07 Giovanni Bedetti <
>>>> [email protected]> ha scritto:
>>>> >>
>>>> >> Not yet, there are limitations in the Unity FMOD implementation for
>>>> the WebGL platform. Specifically, it doesn't support the OnAudioFilterRead
>>>> callback, which is the function we use to fill the Unity AudioSources
>>>> content with the Csound output.
>>>> >> Technically though it shouldn't be hard to import a Csound library
>>>> for the Web, and use the API. The difficulty is sending the Csound content
>>>> to the AudioSources, and so being able to move those sources in the 3D
>>>> space and having them spatialized (Unity spatializer is first order
>>>> ambisonic).
>>>> >>
>>>> >> In the next days I will try something and report here my findings.
>>>> >>
>>>> >>
>>>> >> Il Ven 31 Mag 2024, 09:29 Michael Gogins <[email protected]>
>>>> ha scritto:
>>>> >>>
>>>> >>> Is there WebAssembly support for Csound in Unity yet?
>>>>
>>>