(If you are reading this article to fix a specific issue my workaround is at the bottom of this post).
But even with the best possible patching techniques in place there are some risks we must mitigate against. The trouble is, by definition patches are changing small pieces of functionality within a much bigger context. And, to make matters worse, they are probably changing small but critical pieces of functionality to work in a more "secure" or "reliable" way than before. If they fixed an entire system or made no functional changes they wouldn't be "patches". By changing single components we are relying on the developer to have fixed a specific issue without causing knock on issues in other related components.
To use an analogy, it's a bit like taking your car to a garage for a service. They change the oil filter and we rely on them to have made no changes to any other part of the car. We also rely on the manufacturers of the oil filter to have constructed a perfect oil filter that won't break. And we rely on everyone to ensure that the oil we have in our car is compatible with the new oil filter components.
If we look at a simple example of a patch we can see this in action.
In this case we have an application installed on our Windows machine. It's a 64 bit machine and our Windows installation is 64 bit too. On top of that the .NET framework is providing lots of extra functionality. It is 64 bit as well. And we have installed our application in 64 bit mode meaning everything is nice and compatible.
Now we install our patch, in this example to the .NET framework.
Microsoft developed Windows. Microsoft developed .NET. We can be reasonably sure our patch for .NET is compatible with Windows (or if there are going to be issues we can be reasonably sure Microsoft have plenty of documentation on how to resolve them).
This means all we really need to worry about is whether our application is compatible with our patched version of the .NET components. And with millions of .NET based apps out there it's pretty reasonable to assume that any compatibility issues are going to be blogged about within seconds of the patch hitting.
So, as we know, it's a good idea to test our patches. It's a good idea to check the documentation before applying the patches. And it's a good idea to have a way back if a patch causes something unexpected. But otherwise life is simple.
BizTalk is a bit different to this. First lets look at our diagram after we add our out-of-the box BizTalk components.
Immediately we have much more complexity. We are invoking a stack of extra Windows components including MSDTC. Then, instead of simple application layer with some compatible executables and DLLs we now have a stack of components designed to run under service accounts, called from Windows services in a mixture of 64 bit and 32 bit architecture. Our same patch must now simultaneously support the behaviour of all those layers in both 32 bit and 64 bit modes, fixing security, fixing loopholes, fixing issues while ensuring every component of BizTalk is entirely unaffected. It's a big ask.
But that's the easy bit. Now we need to add our BizTalk apps themselves.
Think about how this works. When you build a BizTalk app you use Visual Studio to build a bunch of DLLs which are going to be deployed to your BizTalk server. These can be 32 bit or 64 bit and may take the form of BizTalk templates such as maps and orchestrations or could be chunks of .NET code. And now we are going to expect BizTalk to invoke these DLLs dynamically based on some settings in a SQL database. And, to add to the fun, we want it to flip between 32 and 64 bit on demand while asynchronously invoking these components to handle any message flow we throw at it. In fact it's so dynamic and so unpredictable it almost sounds like the behaviour malware would get up to. Want to bet your house on this stack of goodies working after a quick Windows Update + Restart combo? Hmm, I think not.
If this doesn't make you stop and think about having a robust patching strategy including thorough testing let me give you a specific real world, "happened to me" example.
Microsoft patches KB3110329, KB3121918 and KB3175024 all deal with the way that the Windows Kernel invokes DLLs and, more specifically, all tighten security in relation to how DLLs can be called by other DLLs and executables to reduce the risk that one of the DLLs is some kind of malware. We have already installed these patches on multiple servers without any problems.
Now we put the same patches on our pre-production BizTalk server. This server has a number of WCF-Custom send ports connecting to SQL Servers. The ports are invoked by a number of orchestrations running on a 64 bit host instances called "OrchestrationHost64". However the ports are configured to run on two different host instances called "SendPortHost32" and "SendPortHost64".
So, installation of the patches goes well. The server is restarted. BizTalk restarts successfully. No errors are reported by BizTalk. Messages start flowing. All looks good.
And then a message is processed which sends to two WCF-Custom/SQL ports, one on our "SendPortHost32" and one on our "SendPortHost64".
“System.IO.FileLoadException:
Loading this assembly would produce a different grant set from other instances.
(Exception from HRESULT: 0x80131401)”
This is the error in our Windows Application Event Log as one of the two sent messages drops off into a black hole. Helpful isn't it? Try Googling for it now and see how many articles explain what this message means in relation to BizTalk. Any good ones? Any from Microsoft?
We were fortunate. We caught the error before it hit our production environment. We found it because the tests we had in place allowed us to catch this error. We had seen similar errors before outside of the world of BizTalk. And, most importantly, we were lucky enough to have a large enough spread of send ports for a repeatable pattern to emerge quickly. We restored our server to a backup from before the patching, retested using identical tests, proved it was fixed, applied the patches and proved it was broken. Within a day we were back up and running.
Ask yourself the question, would you be as fortunate as us? Take patching seriously everyone!
Anyway, as promised, here is my workaround which fixed our problem (and probably highlights the need to be very consistent in the way you configure BizTalk).
1) Confirm that the error you see is...
“System.IO.FileLoadException: Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401)”
2) Confirm that you have ports of the same Port Type running on a mixture of 32 and 64 bit hosts
3) Confirm that you have the capacity to run all ports of that Port Type on a single host and, if not, spin up further host instances as required
4) Amend the port bindings to ensure all ports of the same Port Type are running on host instances of a single processor architecture
5) Restart the host instances (including the host instance that you have moved away from!)
What we do is hard.
No comments:
Post a Comment