Our walled gardens, and other interop hurdles

Well. It's finally happened. The other day I finally had to sit with the "MS Guy" and work with him on providing access to calls on my system via SOAP.

And now, SOAP that once held hope for me, holds only sadness and despair, Let me explain why.

Some time ago I chose not to base myself on Microsoft products, not because I truly hate Microsoft, I DO hate Microsoft's marketing and selling tactics and politics, but being, or trying to be a pragmatic person and developer, I try to use the technology that fits the best for the requirements, but as many technical experts would know, and as most Perl developers hold to heart "there's more than one way to do something".

So when several solutions would apply equally, or near equally well to the problem, it's up to the developer's taste (in cases where he makes the call, as is always the case for me, I like making the calls so I look for jobs where I can make them) to decide which solution to use, which of course can be degraded in some cases, I think we all know about people forcing the frameworks or systems they know into a project however bad a fit it is for the project.

This hasn't been much of a problem to me since my biggest ability has always been learning new stuff rather than becoming a deep expert into one topic, it's a double edged sword but it's how I work and function and I'm not going to change it now.

Anyhow before this evolves fully into a rant. What I intend to say here is that we all have a style of working, a platform and a set of tools we feel most confortable in, and a personal vision on how stuff should be. And I'd call anyone who pretends to be entirelly agnostic or pragmatic a liar for I don't think we humans are able to attain pure objetivity.

So what the hell does this has to do with SOAP? Well there's a guy working for the company as a consultant developer trying to install a new, local "enterprisey" costly and powerfull invoicing system called Profit (Personally I've always found enterprise and powerfull solutions to be... well unmanageable do-it-all wannabes that require too many years of training to atain a pitifull performance, but that's MY vision as a agile programmer).

So, this system is coded in Visual FoxPro, and we've been trying to figure out a manner on communicating, I wouldn't allow him to plunge into my PostgreSQL Database (He almost sneered at me not using MS-SQL) for I fear for the integrity of my data and my DB has always been a sacred thing for me. I'll rarely allow anyone to drink my DB without some kind of protecting API wrapped around. So finally we decided on SOAP.

Here I hit the first hurdle. SOAP in itself is overkill, I agreed so before doing this (even without having done it before) and I agree with it now. As a pure solution I preffer json-rpc since I like the syntax of JSON a lot, and it's unhindered by all the layers of crust. But try selling a .NET guy on REST or on JSON-RPC, it's near impossible. So with the intention of being able to work together, we went the merry SOAP road, after all it's a well defined standard by now, and .NET has in theory (I don't use the product) very good tools for dealing with SOAP... Well the first hurdle I found was the disagreement from my ruby on rails peers.

For starters, DHH (it seems, this hasn't been verified by me) has decided SOAP is in many respects too inferior to REST, so the soap serving package, ActionWebServices has been dropped from the main rails code. It's relegated now to the disterred repository and not afaik too actively maintained by the core programmers... it IS maintained, but let's just say that I personally find it lacking in some areas, and installing it alongside a normal rails install was a fight. It was such a fight that I haven't been able to make my staging environment work with AWS yet, only my development box, and that's incredibly frustrating since debugging my staging is not so easy, more so considering no ammount of logging has helped here and it seems I'll have to remotely debug the code step by step.

Then, when my SOAP services where coded and returning nice XML responses that seemed to my inexperienced eye to be perfect, I tried hooking with this guy... and it took us all a night of work to get it to function.

First of all, in his version of Visual FoxPro, the "Professional tools found in .NET for SOAP services" turned up to be a soap script library glued into VF by this guy, and whose correctness I still suspect but being not an expert in either SOAP or VF I can't assure. To make a long story short, the guy is expecting a form of structured XML as a response, it is in a way an XML representation of a data table, with an embeded schema in XML and all. I could pass very basic types to him via AWS and soap, stuff like strings, dates, and integers. But when I tried to send a structured type, all he got was "garbage" he couldn't understand. And sending back an array of an atomic type like :int caused AWS to construct a custom complex type, and from this type the VF library was able to correctly obtain only the first element in the array (And wathever the guy said to me I still insist I'm sure there's a way to iterate the array on his side).

So for the two calls that we use what did we do?

Well for the (single) structured type I'm composing and sending him back a :string with semicolon separated variables... ugh... and for the array type I'm using REXML to construct a very basic copy of an example XML table structure taken from the VF help, and then embeding the resulting XML as a :string onto the SOAP response, which he parses into one of his XMLData objects to parse it.

Now it's entirely probable that %100 of our troubles have to do with the fact that both the guy and me are newbies at SOAP, hell the guy couldn't even understand he needed to hook into my WSDL definition URL rather than trying to hook onto the API access URI with his library until I pointed it to him from seeying the debug he was performing.

But there is also something very distinct here, while I perused SOAP documents, prior, during and after doing this first test run with him (we'll continue workign together until release and somewhat beyound), I saw a distinctive pattern. All SOAP materials where highly language and toolset centric.

That is, every single SOAP text I found explained SOAP not even generically but in the proper terms of the language and toolset it was geared for, and testing the results was even performed in those tools and languages as well!! This, in my view, for a protocol designed specifically to interconnect heterogeneous systems is more than a terrible practice, it's practically poison.

So, that's my first point against SOAP. Everyone seems to be doing a "universal" conduit in their own manner, or even not doing it at all... that does not bode well for a universal conduit's future.

The second point is complexity.

SOAP is horrible. I glanced over the W3c specs and I almost developed skin sores, the thing is large, horribly large and convolute. What's the problem here? well I think SOAP was infected by the EAV people. And don't think I have always been against that, I once was an EAV kid myself. I once thought "My programs need to be able to deal with anything, anytime, without my intervention, forever" so I designed for example a CRM with enough expandability in it to be able to also be used to calculate the distribution of sugar cubes in shipments to mars if it was necessary someday... of course these systems grew horribly complicated and unmaintanable, and very terribly innefficient at what they originally HAD to do.

With time I grew over it. And I know that the more lean my programs are, the easier it'll be for me to maintain them. And the more custom fit they and their data sets are to the application, the more solidly it'll run... and it's not like deploying a new version has to be a transparent even always, and even for those cases where it has to be I have planned a certain type of program that will help me a lot to perform them and of which I'll tell you more latter one day.

So still it seems the SOAP group was contaminated by people wanting it to be ready for anything and do anything. Say I for one would've been completely happy to have a parameter type akin to the capabilities of JSON which I think will cover all my basic needs, without complicating things too much, instead of that ComplexType abomination SOAP relies on....

But it gets worse, regarding databases and XML I was looking around, and it seems so far that what Ruby understands for a complex type is not the same C# understands of it and it's not the same VisualFox understands it to be... so, let me recap, the SOAP group produced a too varied, too complex definition with a minimum common denominator that I consider to be anything but minimum, and still SOAP is easy to use only between homogeneous systems? so heterogeneous system engineers still have to sweat blood and fight for the communication to work....

And then what are the alternatives? XML-RPC? JSON-RPC? or better yet in my view REST? well the problem with that on my mind is when you work with loosely fit teams on both sides or with no communication at all... the complexity in SOAP comes from the process of creating an universal standard with every side adding details they deem necessary or vital, and which all eventually end up agreeing on. So if SOAP being a standard in all it's complexity, sometimes fails misserably to interconnect heterogeneous systems, what would happen if you rely on a standard or half standard that is not backed by a giant like the W3c and pushed by behemots like Microsoft? it'd be worse, much much worse.

At least with SOAP, if I want to, say for example use Google's API over SOAP and there are some incompatibilities between what they produce and what I expect, I can juggle them and handle them on my side, as long as they conform to the SOAP standards, without any help from Google, so I don't need to interact with them.

But if I where to try so on a more laxly designed or regulated standard, or semistandard, it would eventually end up with me being unable to use the published API, or being forced to interact with the publisher or team to be able to make it work.

So the short, short conclusion is. Everyone is doing everything from their viewpoint, and in their manner... do you guys think it will be ever ever ever possible for systems and companies to exchange data and APIs in a universal form, without it becomming an overcomplex behemot that makes things more difficult, and without each person/company's policies and views poisoning the communicating pipe?