Re: status of nio version

Thanks Robert,

Just got home.

Interesting and more to think about.  1.5 didn't fix anything concerning
ByteBuffers?  Anybody familiar with the channels?  Without know how they
work I do have a case where reading in a 290 MB file it really works:-)
But there is memory files which I was thinking could be employed for
huge data situations.  Determining when is the problem.

Robert Hastings wrote:

>When I think of [out] parameters I think of two cases.
>
>1) The [out] parameter is a primitive type. COM requires that the type of
>the [out] parameter is a pointer to the primitive type so that it can use
>the pointer to change the value.
>
>2) The [out] parameter is a pointer to a pointer type. The passed pointer
>points to a memory area that will be over-written with a pointer value that
>points to the [out] value. Stay that ten times.
>
>Thinking in terms of byte buffers the first case it pretty workable for
>direct byte buffers, but those will each use a page of memory (4K). With
>direct byte buffers you could get pointer to the memory used on the JNI side
>and pass that as the [out] param. When the primitive value is written into
>the bytes they will show up on the java side and you could use on of the
>many get methods.
>
>For the pointer case, you'd end up with a byte buffer that contains a
>pointer to native memory. I suppose you could write a JNI call that will
>wrap those bytes with a byte buffer and then you'd have access. You have to
>free those bytes with the correct native call. That is if you have a BSTR
>you have to use SysFreeString, if you have a SAFEARRAY, you have to use
>SafeArrayDestroy, if you have a VARIANT you need VariantClear, and all other
>cases should map down to CoTaskMemFree (unless I forgot a case :-).
>
>To address the direct byte buffer 4K problem you might setup some JNI code
>that will use malloc/free to create smaller byte buffers. JNI allows you to
>create a byte buffer to wrap a smaller memory area. I don't know why this
>feature is only available on the JNI side.
>
>Robert
>
>-----Original Message-----
>From: Discussion of Java/Win32/COM integration with Jawin
>[mailto:[email protected]] On Behalf Of Roger I Martin PhD
>Sent: Thursday, June 02, 2005 3:31 PM
>To: [email protected]
>Subject: Re: [JAWIN] status of nio version
>
>Thanks Morten,
>
>A more indept discussion will follow.
>
>Project organization I leave the howto in your hands:-)  Let me know
>what works for new development being cvs'ed.  Branch?
>
>I started an OleControl.cpp so I could work without damaging
>GenericStub.cpp but have no plans for pushing it into cvs that way.  As
>I went I kept using less and less of the original linear stream
>transforms and testing.  For the valid issues you raise about complex
>types, am looking at filling the ByteBuffer with the Variant structure
>at the beginning and the rest of it with the data in direct and native
>byte order.  Then make a pre invoke call to the native to set the
>pointers in place.  This may be problematic if memory changes but is
>there a way to protect from this?
>
>You mention [OUT] parameters in the return and I must display my true
>ignorance of the linear byte cpp marshaling:-)  I never understood  it
>and never applied it using all of the return capability:-)  Did JNI code
>generation instead:-) Originally I wanted to get the stub generator
>working in Java with native type info support and xml technology.  Then
>Josh Passenger built the TypeLib Browser and [whisper] we both wanted
>some of the original marshallers to return with enthusiasm.  Currently
>what I'm trying to do is return everything back thru their place in the
>arg list or ByteBuffer array.  Maybe I'm changing it too radically.
>
>I can tell you that for the prior marshalling I can only tinker with
>fixes but don't understand the native marshaling code.  I never saw
>anything constructed by the COM side appear on the Java side.  Or if a
>string changed dimension I never got it. Maybe we didn't finish this
>part of new stubbing for the returns?
>
>Quickly,  the member id FUNCDEC.memid is used by IDispatch and the
>FUNCDESC.oVft is the vtable offset which will figure in coclasses (I did
>not have the tlb retrieval supply memid before).  In a small test
>FUNCDESC.oVft is zero if it is not a virtual function and vice versa for
>the FUNCDEC.memid.  We can make the GetIDofName
>orIDispatch.getIDsofNames functional for those who do not have hard info
>on the memid and choose not to use the stub generation?  By this it can
>be retrieved during run time by the function name as usual.
>int id[1];
>String names[1];
>names[0]=functionName;
>getIDsofNames(...names...id
>invoke(id[0],....);
>
>every time or they can get all the ones they want during initialization
>and use as needed and without repeated calls to getIDsofNames.
>
>More later,
>
>Roger
>
>
>Morten Andersen wrote:
>
>
>
>>Hi Roger,
>>
>>Impressive thoughts you seem to have put into this. I have added a
>>couple of comments, here and there below (I have far from comments or
>>ideas on everything, as there are lots I don't know much about)
>>
>>Roger I Martin PhD wrote:
>>
>>
>>
>>>There are a number of issues which I would like to know if the changes
>>>adversely affect anyone.
>>>
>>>0) All current marshalling functionality remains available while new
>>>functionality is added.
>>>
>>>
>>>
>>That is of course the safe option. But I fear the risk of bit-root,
>>where we suddenly have two different ways to do the same thing, and then
>>there will be a risk that one of them doesn't get appropriate attention
>>and will degrade over time. I assume I am more to a "hard-cut" switch,
>>although I am aware that this may burn some bridges to existing
>>developers.
>>
>>But that is solely up to you.
>>
>>
>>
>>>1)  Is it a problem for anyone if the IdentityManager uses the IUnknown
>>>interface to work with the Jawin classes that implement it instead of
>>>the COMPtr directly?
>>>
>>>
>>The details of the IdentityManager is not entirely fresh in my memory.
>>But if it is possible to work on the interface only, that is of course
>>preferable. But will it be possible without exposing the internal Jawin
>>reference counting for GIT-cookie and/or vtable pointer information.
>>
>>But again this is far away in my memory, so I assume if you have a fresh
>>overview of the internal working of IdentityManager, you have considered
>>pro and cons.
>>
>>[SNIP]
>>
>>
>>
>>
>>>3)  Everything I am doing is aimed at speed and seamless bridging.
>>>
>>>
>>With respect to the speed, a couple of the unittests are actually some
>>indicate perfomance tests. That is the classes in test/org/jawin/perf,
>>which can be run with "ant test", after this you can inspect the run
>>time for these in JUnit-report. I assume you should probably dublicate
>>these tests for the new NIO-interface and se what speed gain we can
>>expect (we could set up a betting pool in advance :-)).
>>
>>
>>
>>>The
>>>member id from the function description  will be used instead of passing
>>>a String with the function name to the new nio native calls and then
>>>converting to a BSTR and calling the  method to get the member id to
>>>pass to the IDispatch invoke.  The Jawin Type Browser is updated to
>>>provide the xml output with the member id.  The vtableoffset will be
>>>used for virtual function events and callbacks.  Is there any reason
>>>relying on the member id from the tlb's would be problematic?  I am
>>>imitating what I see when a OleControl is dropped into a Visual C++
>>>project.
>>>
>>>
>>That depends a little bit on what direction we want the project to take.
>>If I understand you correctly this will disable the option to do
>>"type-un-safe" VB-script coding style
>>(http://jawinproject.sourceforge.net/jawin.html#callingScript ) and
>>force developers to generate and use stubs?
>>
>>I am not that strong on COM-terms, but is it correctly understood that
>>what you suggest is that all stubs should be VTable based, and that we
>>won't really use and/or expose the functionality offered by the
>>IDispatch interface (or is this just me mixing up terms?).
>>
>>When discussing if this is direction we want to take, we should probably
>>also discuss what support we want for calling old style (non-COM) DLL
>>entry points. Is this something we want to keep supporting, or should we
>>give up on that part, and refer people to more mature projects for such
>>functionality (most of the under-documented marshalling instructions are
>>actually for this). Or this something we should direct attention to?
>>
>>[SNIP]
>>
>>
>>
>>>5) Am providing an invoke method for every type of return( similar to
>>>the way JNI is organized).  The stylesheets will pick the right one when
>>>generating code.
>>>invokeVoidMethod
>>>invokeIntMethod
>>>...
>>>invokeObjectMethod
>>>
>>>These eliminate the need for
>>>       return MarshalAllocator::allocateByteArray(env,
>>>javaOut.getMem(), javaOut.getPos());
>>>and replaces this with the direct application of the return Variant
>>>such as
>>>       return retVar.lVal;
>>>for integer returning methods.
>>>
>>>
>>>
>>But the "javaOut" byte-array contains both [RETVAL] and [OUT]
>>parameters. Will a "random" mix of [RETVAL] and [OUT] parameters still
>>be supported with this change?
>>
>>
>>
>>>6) The argument list for every Jawin native invocation call is being
>>>reduced to jint dwDispID, jobjectArray argArray, jint peer, jint
>>>unknown.
>>>dwDispID = member ID
>>>argArray = array of java.nio.ByteBuffer's. One for each argument and
>>>each contains a direct memory access to Variant.
>>>
>>>example:
>>>
>>>JNIEXPORT jint JNICALL
>>>Java_org_jawin_marshal_OleControlStub_invokeIntMethod
>>> (JNIEnv* env, jobject obj,  jint dwDispID, jobjectArray argArray, jint
>>>peer, jint unknown)
>>>
>>>Notice the instString is gone which eliminates
>>>
>>>       JNIComUtil::jstostr(env, instString, str);
>>>       IStreamOnMemory instructions(str.c_str(), str.length() + 1);
>>>...
>>>       Transform tin(env, &br, javaIn, instructions, comIn, NULL);
>>>...
>>>       //write retval into stream
>>>       Transform tret(env, &br, result, instructions, javaOut, NULL);
>>>       //write out params into stream
>>>       Transform tout(env, &br, comOut, instructions, javaOut, NULL);
>>>The direct access to the ByteBuffer variants eliminates
>>>       IStreamOnMemory javaIn(request, requestSize, env);
>>>       jint stackSizeWithRet = stackSize + 1;
>>>
>>>       // Allocate memory for variants
>>>       size_t vntsSize = sizeof(VARIANT) * stackSizeWithRet;
>>>       VARIANT* vnts = (VARIANT*)MarshalAllocator::SafeMalloc(vntsSize,
>>>NULL); // should we pass the BatchReleaser?
>>>       for (int i = 0; i < stackSizeWithRet; ++i) {
>>>           ::VariantInit(vnts + i);
>>>       }
>>>       OStreamOnMemory comIn(vnts, vntsSize, true, true);
>>>...
>>>       IStreamOnMemory result((byte*)comIn.getMem() +
>>>stackSize*sizeof(VARIANT), sizeof(VARIANT));
>>>       IStreamOnMemory comOut((byte*)comIn.getMem(),
>>>stackSize*sizeof(VARIANT));
>>>with
>>>       int arrayLength=env->GetArrayLength(argArray);
>>>       VARIANTARG* pVar=new VARIANTARG[arrayLength];
>>>       for(int i=0;i<arrayLength;i++)
>>>       {
>>>
>>>
>>>
>>>
>pVar[i]=((VARIANT*)env->GetDirectBufferAddress(env->GetObjectArrayElement(ar
>gArray,
>
>
>>>(jint)i)))[0];
>>>       }
>>>       VARIANT retVar;
>>>The marshalling code is simplified to
>>>JNIEXPORT jint JNICALL
>>>Java_org_jawin_marshal_OleControlStub_invokeIntMethod
>>> (JNIEnv* env, jobject obj,  jint dwDispID, jobjectArray argArray, jint
>>>peer, jint unknown)
>>>{
>>>   long result;
>>>   se_translator translator;
>>>   try {
>>>       int arrayLength=env->GetArrayLength(argArray);
>>>       VARIANTARG* pVar=new VARIANTARG[arrayLength];
>>>       for(int i=0;i<arrayLength;i++)
>>>       {
>>>
>>>
>>>
>>>
>pVar[i]=((VARIANT*)env->GetDirectBufferAddress(env->GetObjectArrayElement(ar
>gArray,
>
>
>>>(jint)i)))[0];
>>>       }
>>>       VARIANT retVar;
>>>       CComPtr<IDispatch> cpUnk;
>>>       getUnknown(IID_IDispatch, peer, unknown, (IUnknown**) &cpUnk);
>>>       CComDispatchDriver disp(cpUnk);
>>>       IOleControl* olectl=NULL;
>>>       cpUnk->QueryInterface(IID_IOleControl, (void **)&olectl);
>>>       HRESULT hr;
>>>               DISPID dispidPut = DISPID_VALUE;//DISPATCH_METHOD;
>>>               DISPPARAMS dispparams;// = { pVar, NULL,
>>>(int)stackSize+1, 0};
>>>               memset(&dispparams, 0, sizeof(dispparams));
>>>               dispparams.rgvarg=pVar;
>>>               //dispparams.rgdispidNamedArgs=&dispidPut;
>>>               dispparams.cArgs=arrayLength;
>>>               //dispparams.cNamedArgs=1;
>>>               EXCEPINFO excepInfo;
>>>               unsigned int argErr;
>>>               //IDispatch* idisp=(IDispatch*)disp;
>>>               hr = ((IDispatch*)olectl)->Invoke(dwDispID, IID_NULL,
>>>LOCALE_USER_DEFAULT, DISPATCH_METHOD, &dispparams, &retVar, &excepInfo,
>>>&argErr);
>>>               //hr = disp.InvokeN((int)dwDispID, pVar, arrayLength,
>>>&retVar);
>>>       olectl->Release();
>>>            JNI_HR(hr);
>>>       return retVar.lVal;
>>>   }
>>>   HANDLE_WIN32_EXCEPTIONS()
>>>   HANDLE_JNI_EXCEPTIONS()
>>>   return 0;
>>>}
>>>
>>>
>>This looks very exciting. So I am understanding it correctly if what you
>>want to change the marshalling to is a scheme like:
>>  "the-java-side-marshall-directly-into-the-native-format"
>>which can be pushed directly into the "Invoke" method.
>>
>>I am not that deep into native Variant types, but is it also possible to
>>allocate the Variant-memory for the types with special
>>allocation/deallocation methods (I assume this is SafeArray and BSTR and
>>perhaps other?), or are this just helper methods MS added to make it
>>easier to remember memory-allocation/deallocation for these more complex
>>types.
>>
>>If you want some "weird-type" methods to test on, I think we have some
>>for e.g. SafeArrays in SafeArrays in the test class
>>test/org/jawin/DispatchTestBase.
>>
>>Best Regards
>>Morten
>>
>>
>>
>
>
>