The Multi-Multi-tenant Upgrade Jigsaw

One of the most commonly cited advantages of an app sitting on multi-tenant architecture is the fact it is frequently upgraded so each customer is always on the 'latest version'. Historically the upgrade path for on-premise enterprise software had been a complete pain. Taking the 'one-click' upgrade path was about as effective as throwing a bag of spanners at the thing, as it had been butchered beyond all recognition. Not 'out of the box' anymore, the box had been burned and the ashes whacked repeatedly with a baseball bat. So the standard upgrade routine would basically have a look and then run away laughing.

With a multi-tenant app, the abstraction layer prevents customers from butchering their app to such an extent. This results in a fairly smooth upgrade process, but then a new issue rears its head. The upgrade itself and the timing are forced. It is not possible on a true multi-tenant platform to choose when the upgrade occurs. All customers are upgraded around the same time. So, let's take a couple of multi-tenant examples from my experience:

Salesforce.com : Upgrade occurs to the Production instance every 4 months.

The fun: One month prior to the production instance upgrade, certain sandboxes are upgraded by the Salesforce upgrade fairy. Now, in an enterprise environment you will likely have at least 2 sandboxes - Dev and Test. You may also have a separate UAT sandbox, maybe a Staging sandbox also. You might have a parallel development stream with another Dev and Test. Anyway, the point is that to avoid a sandbox being upgraded early by that naughty but ultra-keen fairy, the only way is to refresh it in a certain window of time, differing depending on what server instance it sits on. This kills the current sandbox and creates a new instance copied from Production. So this needs to be planned well in advanced into the dev schedule to avoid for instance your dev sandbox being on a different version than your test one, and to make sure the release path sandboxes are on the same version as production. And of course if you are midway through your dev schedule when the Production upgrade occurs, and your sandboxes are upgraded at the same time, you need to regression test the in-development functionality. It is also a good idea to have one sandbox upgraded early to regression test current Production functionality against the forthcoming release. To get the refresh schedule right I try to get my head around it all by drawing a couple of timelines. Firstly for the Summer 12 Salesforce release I drew myself this:

And generically as a reminder for the future I summarised the above paragraph into this:

Now lets look at Zuora:

Zuora: Upgrade occurs to the Production instance every month.

The fun: One week prior to the Production upgrade, all Zuora sandboxes are upgraded by the Zuora upgrade pixie. This necessitates a regression test of the current dev cycle, and a regression test against the current production functionality to pre-empt any Production issues. However, outside of this schedule is an absolute avalanche of patch releases by the mini-pixies, which with the best intentions, it is impossible to keep up with. Usually there will be about 1 days notice before the sandboxes are patched, then a few days until production. If you are lucky you get some release notes, but usually not until after the sandboxes are patched.


NOW integrate Salesforce with Zuora, and the solution as a whole has 2 multi-tenant platforms, with sandbox and Production environments upgraded on different schedules. Bloody fairies and pixies running around all over the place. The joy of forced upgrades. Then maybe add on Eloqua, or BigMachines - the more the merrier!

But we choose to enter the multi-tenant world so can't complain. Three points to make:

-When planning your dev/release schedule, review the upgrade schedules and ensure the impact of forced upgrades on resources and release dates is taken into account
-Automate your regression testing to the max to gain confidence in the upgrades quickly, because you will have to! Review release notes and prioritise areas that have changed.
-When cloud platforms prove themselves to be more robust in the future, the need for regression testing may reduce. Maybe.

Upgrades are more likely to have an impact on the more complex implementations which are found in enterprise implementations involving lots of integration, customisation etc, and spanning more functional breadth than smaller implementations. You probably don't need to worry too much about your little standard Sales Cloud thing you stood up for the old lady next door. But who wants to work on a small implementation?

- Posted using BlogPress from my iPad