Monday, 28 March 2011

Bookstore Demo for Playbook soon available on Blackberry Appworld

Today we have posted the well known Domino Bookstore Sample application (available on Openntf.org) as an Adobe Air application for the Playbook on the Blackberry Appworld. Of course we still await the confirmation of acceptance.


This Bookstore-Demo - originally developed to demonstrate Domino web services functionality - shows the functionality of SoapgateQ! in conjunction with the Adobe Flex Libraries for SoapgateQ!, both enabling Adobe Air applications to talk to a Lotus Domino back-end.  The application is based on Adobe AIR 2.5 and is build with Flash Builder 4 Burito.

The source to this application as well as the Flex Libraries for SoapgateQ! are open and licensed under OSI - Apache 2.0. SoapgateQ! is web service based data access API for Domino.  The (soon available) free personal edition of SoapgateQ! allows access for up to 3 users.  

We will make all source code available in next few days and will also provide a download for SoapgateQ! 2.0 Beta.

Visit www.qkom.net for more information and to see further demos.

Friday, 25 March 2011

Flexdomino.net download pages now open for unregistered access

Due to technical difficulties with the registration procedure of the currently used CMS of flexdomino.net, we decided to open the download pages for unregistered access.

Simply go to http://www.flexdomino.net/home/downloads to download all sample projects and code.

Thursday, 24 March 2011

Fixed LSXSD.LSS - Some more info...

With all my excitement having fixed LSXSD.LSS, I have posted the code without any further comments as to the changes I have implemented. I actually received an email asking me for the reason of having renamed the PortTypeBase class to PortTypeBase_NOT_A_CONSUMER, and this reminded me that there a few things that require an explanation:


PortTypeBase_NOT_A_CONSUMER

LSXSD.LSS is normally included in the code using the %include directive. Apparently when including ans LSX, the compiler understands a compiler directive like this...


%If WEB_SERVICE_CONSUMER_SCRIPTLIB
Public Class PortTypeBase
...
End Class
%End If

...which means the PortTypeClass is only included if the web service is a consumer. In my case the web service is a provider and hence I do not need that class.

Unfortunately the above compiler directive doesn't work when importing the LSXSD.LSS into a script library and then using the USE statement to include the script library. Hence I renamed the class to render it ineffective.

So to make it clear. If you replace the existing LSXSD.LSS on your Domino server or you copy this version of code with a new name to still use the %include directive, you need to rename the above class to its original name (removing the _NOT_A_CONSUMER part) and enclose the class with the %If WEB_SERVICE_CONSUMER_SCRIPTLIB...%end if directive.

If you want to use the code in  a script library, but you write a web service consumer rather then a provider, you also need to rename the class to its original name (of course without using the %if..%end if directive, as it would cause a compiler error).

There have been a few other changes as well...

CLASS XSD_DATATYPE_CONVERTER

notesStreamToBase64()

The method notesStreamToBase64() previously defined as of type String is now returning a NotesStream. The reason being is that the original method used string concatenation to build the encoded return string, which causes the method to slow down exponentially to the length of the encoded stream; to a point where one needs to kill the server task.

The method now converts the input stream into a encoded output stream. One can then use the NotesStream.Readtext() method to receive the previously return encoded string.

notesStreamToBase64Ext()

Consequently the notesStreamToBase64Ext() method originally returning a string now returns a NotesStream.

I also removed the call of the IBM internal NotesStream.ReadEncoded() call as this method has a bug, which lets the method skip chunks of encoded data in the returned string. This happens randomly according to IBM, but my experience is, it happens to larger file sizes.

Another issue a found looking at the original method is that it uses On Error statements without the corresponding Resume statements.


The notesStreamToBase64Ext() method now simply calls notesStreamToBase64().

base64ToNotesStream()


Though the code of this method is completely re-written, from a parameter and return value point of view nothing has changed.

base64ToNotesStreamExt()

Though the by IBM reported issue seems to be related to NotesStream.ReadEncoded() only, and NotesStream.WriteDecoded() seems to be uneffected, I have again removed the call of the IBM internal method and call only base64ToNotesStream().

NOTE: NotesStream.WriteDecoded() is obviously much much faster than even my improved decoding algorithm written in LotusScript. So you might want to switch back to it and only use the base64ToNotesStream() call if you experience problems with IBM's internal decoding method.

notesStreamToHexBinary()

Similar to notesStreamToBase64() this method now returns a NotesStream. I implemented the same improvements.

hexBinaryToNotesStream()

No improvements here yet.

NOTE: If you have utilised the upload/download code provided in the Bookstore database available on Openntf.org, you need to amend the code to reflect the new return values where applicable. 

Wednesday, 23 March 2011

Fixed LSXSD.LSS

For those of you who require to use the Base64 upload/download features of the LSXSD.LSS and having trouble because of the by IBM reported issue...

http://flexdomino.blogspot.com/2011/03/aaargh-yet-another-set-back-in-our.html
https://www-304.ibm.com/support/docview.wss?dc=DB550&rs=463&uid=swg1LO57765&context=SSKTWP&cs=utf-8&lang=en&loc=en_US

...and cannot wait for the fix...

I have re-written the Base64 encoding and decoding methods and use it by importing the code into a script library. Instead of using the %Include statement, I use the Use statement. Hence instead of rolling out a custom LSX, I place the custom code in my databases.

Whilst the original Base64 coding algorithms in LSXSD.LSS are performing dead slow (anything beyond a few KB file size can actually keep the server busy for very prolonged periods (hours!) - see ...

http://flexdomino.blogspot.com/2011/03/ouch-that-hurtshello-ibm-quality.html

... - the code provided in the link below manages MB sized files in seconds:

3.2MB encoding in approx. 20 seconds
3.2MB decoding in approx. 10 seconds

Whilst this performance is still considerably slower then the in Domino 8.5 introduced internal encoding/decoding methods of the NotesStream class (for the same file size about 1 second), the modified code works reliable!

The code can be found here...

http://www.flexdomino.net/lsxsslss

Sunday, 20 March 2011

Ouch that hurts...hello IBM quality control are you out there?

Further analysing the code in lsxsd.lss I'm utterly disgusted about the code I find. I'm sorry to use such strong words, but it reflects how bad it is. I have looked at the NotesStream to Base64 encoding (XSD_DATATYPE_CONVERTER.notesStreamToBase64) in particular. For my file download tests a used a 3.2MB image file and the method got the server to its knees. Here are the reasons for it:

  1. The  method  returns a string and hence uses string concatenation to build the return value. The problem with that is that as longer a string gets, the longer a string concatenation takes. Now this is not a linear increase,  but an exponential one. To a point that any string concatenation with a 2MB string takes pretty much forever in server code execution terms.
  2. The method reads in the Notes stream byte by byte...OUCH!!! This is probably the best way to render the NotesStream class useless.
  3. Though only handling Strings, the concatenation is done not using the + (String) operator, but using the much slower & (Variant) operation.  
Here is the original code (I have highlighted the most significant 2 lines of code):

Function notesStreamToBase64 (ns As NotesStream) As String
Const b64Chars$ = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
Dim nsLength As Long
nsLength = ns.Bytes
ns.Position = 0
Dim numPads As Integer
numPads = (3 - (nsLength Mod 3)) Mod 3
While nsLength > 0
' Output lines are limited to 76 chars, and because every
' three input bytes produce 4 output chars we process
' up to 57 input bytes at a time. (57 = 76 / 4 * 3)
Dim inLength As Long
inLength = nsLength
If inLength > 57 Then inLength = 57
nsLength = nsLength - inLength
Dim outString As String
outString = ""
Dim idx As Integer
idx = 0
While idx < inLength
' Collect up to 24 bits (3 bytes) of input data
Dim outBits As Integer
outBits = 0
Dim bits24 As Long
bits24 = 0
Dim i As Integer
For i = 0 To 2
bits24 = bits24 * 256
If idx + i < inLength Then
Dim buf As Variant
buf = ns.Read(1&)
bits24 = bits24 + buf(0)
outBits = outBits + 8
End If
Next
idx = idx + 3
Dim numChars As Integer
numChars = 4
If outBits <> 24 Then
numChars = 4 - numPads
End If
For i = 1 To numChars
Dim bits6 As Integer
bits6 = (bits24 And &HFC0000&) / 262144
outString = outString & Mid$(b64Chars, bits6 + 1, 1)
bits24 = (bits24 And &H3FFFF&) * 64
Next
If numChars <> 4 Then
For i = 1 To numPads
outString = outString & "="
Next
End If
Wend
' Add another line of base64 output to the return string
notesStreamToBase64 = notesStreamToBase64 & outString & Chr$(13) & Chr$(10)
Wend
End Function

And this is how it could (should) look like:

Function notesStreamToBase64 (ns As NotesStream) As NotesStream
On Error GoTo errhandle
Print Now
Const b64Chars$ = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
Dim batchsize As Long
Dim inLength  As Long
Dim outString As String
Dim idx       As Long
Dim outBits   As Long
Dim bits24    As Long
Dim i         As Long
Dim nsLength  As Long
Dim numPads   As Long
Dim buf       As Variant
Dim numChars  As Long
Dim bits6     As Long
Dim bufcnt    As Long
Set notesStreamToBase64 = session.Createstream()
batchsize   = 57
bufcnt      = 32766
nsLength    = ns.Bytes
inLength    = batchsize
ns.Position = 0
numPads = (3 - (nsLength Mod 3)) Mod 3
While nsLength > 0
' Output lines are limited to 76 chars, and because every
' three input bytes produce 4 output chars we process
' up to 57 input bytes at a time. (57 = 76 / 4 * 3)

If nsLength < batchsize Then 
inLength = nsLength
End if

nsLength  = nsLength - inLength
outString = ""
idx       = 0
While idx < inLength
' Collect up to 24 bits (3 bytes) of input data
outBits = 0
bits24  = 0

For i = 0 To 2
bits24 = bits24 * 256

If idx + i < inLength Then
If bufcnt = 32766 Then
bufcnt = -1
buf    = ns.Read(32767)
End If

bufcnt  = bufcnt  + 1
bits24  = bits24  + buf(bufcnt)
outBits = outBits + 8
End If
Next
idx      = idx + 3
numChars = 4
If outBits <> 24 Then
numChars = 4 - numPads
End If
For i = 1 To numChars
bits6     = (bits24 And &HFC0000&) / 262144
outString = outString + Mid$(b64Chars, bits6 + 1, 1)
bits24    = (bits24 And &H3FFFF&) * 64
Next

If numChars <> 4 Then
For i = 1 To numPads
outString = outString + "="
Next
End If
Wend
' Add another line of base64 output to the return string
retbyt = notesStreamToBase64.Writetext(outString, EOL_CRLF)
Wend

errexit:
Print Now

Exit Function
errhandle:
Print Erl & ", " & Error
Sleep 1
Resume next

End Function


This is maybe not the best way of implementing a Base64 encoding algorithm. I'm not an expert in Base64 encoding in the first place. I simply made the original code work (more efficiently). There is still room for improvement, but at current stage I managed to get the processing time from "forever" (I restarted the server after 10 minutes of processing the 3MB file) down to 24 seconds. Still a bit slow, but reasonable.

Coming back to the title of this posting, I'm not amused that such code leaves IBM quality control unchecked. If this is an example of code that can be found in the Lotus Notes Domino code stream, then it is no wonder why the Notes Clients became the slow and heavy tankers they are today.

I hope someone at IBM looks at this postings.


Wednesday, 16 March 2011

Aaargh, yet another set-back in our attempt to make Domino a bit more accessible...

It was planned to provide a (web service based) file download and upload in the next release of soapgate Q! Unfortunately we found that for certain files, data is lost when they are stored in an XSD_BASE64BINARY object defined in lsxsd.lss. The latter being the very library one would use to implement advanced Domino web services.

The issue has been published by IBM recently and is effecting Domino 8.5.1 and 8.5.2...AND soapgate Q! (unfortunately)...

https://www-304.ibm.com/support/docview.wss?dc=DB550&rs=463&uid=swg1LO57765&context=SSKTWP&cs=utf-8&lang=en&loc=en_US

The comment "The problem will be fixed in a future release if there is one." unfortunately doesn't sound too good. So it looks like we are forced to go back to the "drawing boards" to come up with a different solution.