Richard Pajerski  Software development and consulting

This is the sign by which you are to know him; you will find a child still in swaddling-clothes, lying in a manger.
(Luke 2:12; Knox Version).

Image (public domain): L'Adoration des Bergers by Georges de La Tour

Among the technical reasons Domino has enjoyed longevity in the application server market are its stability and reliability.  But of course technology changes rapidly and being able to successfully adapt to new trends has also kept Domino relevant over the years.  Striking a balance between trendy and relevant is not always obvious but the stewards of Domino have generally taken a conservative approach to introducing and supporting new technology which continues to benefit the product in the immediate and long term.

Very briefly then, beyond the tried-and-true world of Notes client server application development, let's look at three modern technology options developers can choose from in 2021:

1.  Domino AppDev Pack (https://doc.cwpcollaboration.com/appdevpack/docs/en/homepage.html)

From the site:  "The AppDev Pack primarily adds Node.js support to HCL Domino Server."

So developers already familiar with or wanting to explore the JavaScript-centric Node.js have a custom-designed set of tools for programmatically and securely accessing Domino applications and data.

2.  Domino REST API (Project Keep) (https://opensource.hcltechsw.com/domino-keep-docs/)

From the site: "Domino REST API is designed to re-establish Domino as a world class, modern, standards-compliant, cloud native and enterprise-level collaboration platform. It adds contemporary REST APIs to Notes and Domino, enabling a modern programming experience with the tools of your choice."

This is still in beta but will hopefully be out in Domino 12.0.1 later this fall.  Domino has for many years had the ability to offer data via a REST API (Domino Access Services) but Keep modernizes that effort by implementing (among other things) OpenAPI and JWT Authorization.  In addition, developers will be able to introduce their own server-side modules called "verticles" as Keep runs on top of Vert.x.

3.  Tasklets with DOTS - Domino OSGi Tasklet Service (https://help.hcltechsw.com/domino/12.0.0/admin/wn_dotsredux.html)

From the documentation: "DOTS is a generic Domino add-in task that lets users create Domino server tasks by creating a tasklet container using Java OSGi plugins."

Ok, this is not exactly modern -- DOTS was an OpenNTF project that's been available since Domino 8.5.3.  Although officially dropped in the Domino 10/11 era it was recently updated and re-introduced in Domino 12.

Strictly-speaking, DOTS is the Domino server task that manages the tasklets which are Java server-side plugins.  Tasklets use a standard mechanism for starting/stopping (OSGi-defined) which makes them similar to Java agents but have the additional benefits of (a) not being tied to a specific .nsf file, (b) not requiring a full JVM reload on every execution and (c) better access to the Domino C API.

How are you developing Domino server-side code these days?

Having recently installed a Notes/Designer/Admin 12 client on Windows 8.1 Pro on an aging Intel i7 Quad-Core (with eight logical processors), I was surprised to see that after startup, Notes was consistently grabbing one logical processor and pegging it at around 15%.  Windows Task Manager showed that nlnotes.exe was the process and it wouldn't let go of that processor until Notes was closed:

Notes, Designer and Admin clients all worked fine and interaction with local- and server-based applications was normal.  Without giving it too much thought, I tried a few things like removing cache.ndk, stopping a couple Notes-related services and making sure the preference "Enable scheduled local agents" was disabled.

But none of that worked and strangely, Disk and Network utilization were both at 0%:

The client was set up in the normal way by connecting to a Domino server with an existing mail file on the server.  However, this workstation needed to use a Location that sends and receives mail from a POP3 server instead of Domino.  For that purpose I used an existing names.nsf that already had Account and Locations documents in place.  All of that connectivity worked and the mail flowed normally.

The next step was internet search but nothing obvious jumped out and most references to similar problems ended with Notes client crashes which I was not experiencing.

The Location document for this POP3 configuration kicks off replication and runs it every 10 minutes.  I noticed that after starting the client, there was a gap of a few seconds where the nlnotes.exe process was at 0% and didn't go up to 15% until replication started (and then stayed there).  Thinking the culprit was the Replicator, I disabled replication and restarted Notes... but nlnotes.exe was back to 15%!

But this time I could clearly see that process spiked when "Notes configuration settings have been refreshed" scrolled across the Status bar:


Based on that message and the fact that a POP3 configuration like this is not commonly used, I kept pursuing the Location document as the source of the problem.  And the problem was indeed there.

When you configure the Notes client for POP3 mail retrieval, only the "Mail" tab of the Location document needs to be filled out:

The "Servers" tab can remain empty -- and that was the problem!  At a minimum, the "Home/mail server" field MUST have some value in it to calm down the processor:

It doesn't matter if the server is down or the value entered isn't even a Domino server -- nlnotes.exe will report that the server is not responding but it leaves the processor alone after that:

Ok, problem solved, back to work. 

Early in the morning on the first day of the week, while it was still dark, Mary Magdalen went to the tomb, and found the stone moved away from the tomb door.  (St. John 20:1; Knox Version).

Image (public domain): The Resurrection by Carl Heinrich Bloch

Ray Ozzie is among the 2021 Computer History Awards honorees recognized "For a lifetime of work in collaborative software and software entrepreneurship".

https://computerhistory.org/press-releases/chm-honors-tech-legends-for-lifetime-of-contributions-and-impact-on-humanity/

A virtual event took place to honor Ray on March 18, 2021 and will be generally available in the coming days.  Thank you, Mr. Ozzie for your contribution to computer history!

In an effort to cover some specific cases, a minor feature was just added to CertMatica (3.6.0) to automate copying LE certificates to additional locations on the server's file system.

The idea is to allow Domino to share the LE certificate with one or more services running on the same machine (such as a reverse proxy like Nginx or an alternate SMTP service) without further manual intervention.

CertMatica 3.6.0 Trial

First of all, a big "Thank you!" to all CertMatica customers!

Just out this week, CertMatica 3.5.1 is a maintenance release with simplified switching between Let's Encrypt test and production modes and minor improvements to logging and documentation.  However, for those running on earlier versions of Domino 9.x and 10.x servers, this release also includes an important update to the CertMatica Cacerts Utility which can be used to address potential connectivity errors caused by missing or expired intermediate certificates in the Domino JVM truststore (cacerts).  For information on Let's Encrypt infrastructure changes related to this update see https://letsencrypt.org/2019/04/15/transitioning-to-isrg-root.html.

As always, feedback and commentary is always greatly appreciated!

Start Your Engines! HCL Domino v12 Beta is Here! Are You Ready?

FlexNet download:  https://hclsoftware.flexnetoperations.com/flexnet/operationsportal/logon.do

Image (public domain): Nativity of Jesus by El Greco.

In addition to Domino, CertMatica now includes the ability to install and renew Let's Encrypt certificates for Sametime 11 Proxy servers.  Using the simplicity (and power) of a single .nsf solution, CertMatica can help to centralize administration for LE certificates across both server types and seamlessly manage service restarts.  No extra dependencies needed.

For more information, please visit the CertMatica 3.5.0 product page.

As always, comments and suggestions are welcome!

[Edit August 27, 2020: HCL has updated the documentation to include: "MongoDB needs to remain active during upgrade"]

Quick note on Sametime 11 FP2 which is now available on FlexNet.  [The fixlist is located here and the accompanying upgrade instructions are here.]

The instructions tell you to:

-- Close all applications on the server, including the Domino server administrator and the web browser.
-- Stop all Domino and Sametime services.

You might think this includes stopping the MongoDb service as well but you should actually keep that running, at least before you run the proxy upgrade.  The reason is that the proxy upgrade script tries to connect to the MongoDb service and if it isn't running the upgrade will not succeed and you'll be asked to:

"Please verify/update dbconfig.properties and stproxyconfig.xml manually."

It's not a major issue... you can simply restart the MongoDb service and rerun the upgrade.  However, since the upgrade process makes a backup of the previous Tomcat install, you'll now have a duplicate set of backup files.

Since many development and administrative tasks in Notes/Domino can conveniently be carried out with great front-end tools like Domino Designer or Administrator, it can sometimes be inconvenient when we're required to use the command line or terminal to get things done.  Working with Domino keyrings is a case in point and one of the reasons why I developed Aperture.

Aperture is a lightweight desktop application that allows you to work with those .kyr files without having to resort to the command line.  It works with both the KYRTool and OpenSSL to allow you to visually create keyrings, view their contents, create Certificate Signing Requests and several other tasks you'd normally being doing on the command line.

Please visit the Aperture product page for more details:  https://www.rhpconsult.com/aperture.html.

As always, comments and suggestions are appreciated!

The latest update to CertMatica (CertMatica 3.1.0 - ACME certificate renewals for Domino)
is now available.  This latest release fixes a few bugs on Linux installations but also includes a new feature to auto-restart the Traveler task.  Enjoy!

And when the sabbath was over, Mary Magdalen, and Mary the mother of James, and Salome had bought spices, to come and anoint Jesus.  So they came to the tomb very early on the day after the sabbath, at sunrise.  Then they looked up, and saw that the stone, great as it was, had been rolled away already.
(St. Mark 16:1,2,4; Knox Version).

HCL Sametime 11 has been out for a few months now and brought important technical changes that, when used in conjunction with the Sametime 11 Proxy Server, make it a more compelling offering than previous versions released by IBM.

The most fundamental change is the streamlined installation that removes the DB2 and Websphere dependencies needed for the proxy server. Those components have been replaced by MongoDB and a Tomcat-based proxy server respectively, both of which are indeed simpler to install and configure.  It's been rumored that an .nsf storage option will be offered in the next release and that should further smooth out the installation process.

That being said, getting everything up and running is more difficult than it should be.  One notable problem is the documentation.  The language is at times too informal (even ambiguous), the formatting could use some tidying up (unclear headings/inconsistent fonts for samples, etc.) and a URL for details on setting up SSL/TLS sends you to the wrong version (10) -- apparently, no version 11-specific documentation is available.  More importantly, if this is a fresh Sametime installation (which is the only supported option), what's *left out* of the documentation might lead to broken communication between the proxy and the Sametime server.

After successfully completing the Windows installation, I attempted to log in to the proxy from a browser and was greeted with "Sametime is temporarily unavailable":

The problem wasn't immediately obvious because in the proxy's logs I found entries like "Sametime Proxy server is successfully connected to the Sametime community, ...".  But digging further into the logs, I found "generateTokenFailed reason: 80000000".  So network communication was there but
SSO wasn't working.

PRO TIP: Get more verbose logging on the proxy by uncommenting these two lines in Tomcat's logging.properties file (sametimeproxy\conf directory):
com.ibm.level=FINE
com.ibm.handlers = 2localhost.org.apache.juli.AsyncFileHandler

During the Sametime server installation, the installer creates a Web/SSO document in the Domino Directory called "LtpaToken", sets the Session authentication field to "Multiple Servers (SSO)" (in the Domino server document) and correctly references the Web/SSO document.  However, on this fresh installation, the DNS Domain name field of the Web/SSO document was blank:

That's going to leave you with "HTTP Server: Error loading Web SSO Cookie Name Configuration 'LtpaToken' for Web Site ..." on the Domino server and will prevent Sametime from properly creating an SSO token to send to the proxy server. 

Adding the DNS domain name (in my case, .testlab.com) to the Web/SSO document should fix the login problem for most installations.  But in my Windows installation, there was a further complication that kept producing "Sametime is temporarily unavailable".  It turns out the fully-qualified hostname for the Sametime server was not being passed to the proxy.  From the proxy's logs:

serverFQDN: S1
cluster: CN=S1/O=TestLab
serverURL: 192.168.0.102

The fix for this was to enter the fully-qualified domain name for the Sametime server in the Net Address field of the Domino Server document (Ports > Notes Network Ports tab).  After that, the login worked and the proxy reported:

serverFQDN: s1.testlab.com
cluster: CN=S1/O=TestLab
serverURL: 192.168.0.102

Conclusion
So what's *left out* of the documentation is any reference to the Web/SSO and LtpaToken configuration on the Sametime server.  Whether or not the blank DNS Domain name field is an "out-of-the-box" configuration error, some mention of the Web/SSO details back on the Sametime server would be a helpful addition to the documentation.

How are your Sametime 11 installations coming along?