Wednesday, 16 March 2011

Aaargh, yet another set-back in our attempt to make Domino a bit more accessible...

It was planned to provide a (web service based) file download and upload in the next release of soapgate Q! Unfortunately we found that for certain files, data is lost when they are stored in an XSD_BASE64BINARY object defined in lsxsd.lss. The latter being the very library one would use to implement advanced Domino web services.

The issue has been published by IBM recently and is effecting Domino 8.5.1 and 8.5.2...AND soapgate Q! (unfortunately)...

The comment "The problem will be fixed in a future release if there is one." unfortunately doesn't sound too good. So it looks like we are forced to go back to the "drawing boards" to come up with a different solution.


  1. I can explain more on this issue (as I have worked on it). The limitation is in the LotusScript part of the API.

    The XSD_BASE64BINARY stores data as an array of strings. You can see this in the LSXSD.LSS file on your client/server.

    So you are locked down to the limitations that are in LotusScript.

    Java web services on the other hand does not have the same limitations that the LS has. So it should work.

    However using XSD_BASE64BINARY to send/receive data is inefficient. A better method is to send the URL to where the data can be pulled from.

    So for example instead of creating multiple large SOAP responses, the consumers can pull from the same file on the server.

  2. Just to add to the "If there is one", that is generic generated text for "Fixed in next release". It just means that if another release is planned for product X then this is flagged to be addressed in the future.

    It is a bit misleading, like "No plans to fix" and "No plans to fix ever" don't actually something will never be fixed.

  3. Simon, thanks for the info. Much appreciated. Was already looking into lsxsd.lss, but your pointer will speed up things :-)

    Yes, I'm limited to LS as I'm an LS and Action Script developer. I know a bit of Java, but don't feel too comfortable with it.

    With regards to downloads, you are correct, one could simply use a URL request, but that wouldn't solve some of the issues we try to address with a web service based up- and download.

    For instance a DMZ to inner network scenario. You can't simply point with the URL to an attachment this way. And there are other concerns.

    In any case, we are implementing a web service based data access API, so the idea is to keep it web service based even though it is not the best performing method.

    I will see if I can fix the issue myself before RNext.

  4. You're right about the "if there is one" comment, as right on top of the IBM posting it actually says status closed as fixed in Next.

    The problem is we have to wait at least 18 month for that.

  5. Simon, I just looked through the lsxsd.lss and can't find any use arrays whatsoever with regards to XSD_BASE64BINARY.

    XSD_BASE64BINARY seems to boil down to a String Proxy Class returning a base64 encoded string as a stream.

    So this leads either to a problem in the base64 encoder method in the XSD_DATATYPE_CONVERTER class or in the NotesStream class. Whilst for the latter I obviously will not be able to fix anything, if the problem is the converter, I will find the problem and fix it (hopefully).

  6. Problem found...ns.ReadEncoded(ENC_BASE64, 76) ' 76 being an IBM internal code is loosing part of the stream.

    The alternative "fallback" encoder method in lsxsd.lss is extremely slow and keeps the CPU quite busy over a long period. I'm testing with a 3MB jpg image file.

    The reason for this is that the Function notesStreamToBase64 method is of type String and is internally concatenating the return value.

    Concatenating 3MB of Byte to data to a string is quite obviously a never ending task as the time to concatenate strings grows exponentially with its length.

    I started the test run during my writing of this post and after 5 minutes I got a time out. So this is a no go.

    I will try to convert the base64 encoder to use streaming. Let's see if this will do the trick.

  7. You will hit the same limitation with streaming. The limitation is in the LotuScript part.

    You also can't change the class in LSXSD.LSS as this is used by special internal class in Domino which will most likely break if you do.

  8. OK, I imported the lsxsd.lss into a new script library and modified the code to use streaming instead of string operations.

    The base64 encoding method provided in the lsxsd.lss using streaming is working fine and I managed to download my test images without loss of data. However, the performance of the encoding algorithm isn't yet satisfying.

    The IBM internal (not documented) ReadEncode() method of the NotesStream class is still not functioning. This means this method has a bug. This is not a limitation issue.

    I will soon post the modified code.