Sunday, April 20, 2008

Tool Poll

Hi friends, readers and sufferers.

I post this one today to ask on your advice and views on a particular topic.

I'm trying to find and decide on a nice set of software and hardware to use as an organizing and data moving system for my work and life.

So far, the ideas of Getting Things Done, to which foucist introduced me, have worked pretty well for me. I'm not an organizational expert and I'm still kind of a walking mess, but these ideas have permeated my brain somehow and taken hold, and I'm trying to organize.

As far as organizing myself so far I rely on Google Calendar for appointments, which I enter on my phone via Opera Mini, or in my Palm on the calendar and then sync with GooSync. Then the calendar warns me of upcoming meetings via SMS (thank the gods SOMEONE can use SMS to my country, unlike twitter). For actions I need to perform, code and such, I post them on via either Opera Mini on the phone, Email from the phone, or Firefox on the desktop. And then when I'm sitting and have nothing on my brain to do I check it either on Firefox or the Phone via the web.

I do all these centralizations on the internet because I have two jobs. I work with around 3 computers a day, and you never know when I need to be out of the office.

The only drawback is in the remote case I lose all connectivity, which hasn't happened yet, and which I think would anyhow send me into a panic attack (I live more than half my life online so far) so it wouldn't matter.

What's my problem so far? well firstly, typing is slow on the phone and worse on the palm. T9 helps a lot but it's not perfect, and it specially hinders bilingual people like me since we constantly switch between languages by context and it's a slow process with T9. And on the palm even with tealscript which helps, writting or virtual-keyboarding stuff is agonizingly slow, specially when I have to correct missinterpreted letters from the graffiti...

So far I was opting for carrying a small paper agenda onto meetings to annotate in pen (blue and red to denote importante), basically capturing stuff there that I latter need to enter into the computer sites and catalogue, since writting on pen and paper is much easier and faster than the phone's T9 or the palm for me. The troubles are varied. First I've noticed it's much easier for me to capture and organize actions at the same time. If I capture them on paper and then I have to sit and organize them and type them again it's not only a waste of time, but I offer subconscious resistence to it, so I actually "forget" a lot to enter stuff to the web pages from my paper agenda.

The other is that every time more I see that in meetings I need not only enter data but seek data too, when I am asked certain stuff I need to check my numbers, my online stats for some services and Knowledge Base (I have created an In-Office mediawiki KB), so the paper Agenda serves me to enter stuff but not to search existent, real time data to share during meetings.

So I think I need a new digital solution. And I'm trying to look for something.


Hardware wise I'm very constrained by money right now. As some might know or not, to avoid economical bleeding my coutnry has established an exchange embargo, in wich we're able to obtain a certain ammount of foreign currency every year, and nothing over that for anything that is not importing goods for sell and that would require lots of paperwork.

Considering this, my dream machine right now would be a CloudBook machine. You see it's not much much larger than the paper agenda I already carry around, and for the reasons I mentioned before I want a real, usseable keyboard. No on screen keyboards, not even the iphones which I've tried and is good but not confortable enough yet. And stuff like T9 and thumbboards are completely out of the question since so far I've tried a few and typing on them is hell.

Now I do have two devices that might help now, a 12" G4 iBook, and a Palm TX, the good thing about the iBook is the power, but even it being 12" it's too heavy compared with the agenda, and it's beautyful white is a little too enticing though a skanky folder/bag could disguise the machine. I could also get a bluetooth keyboard for the Palm, after all I have LOTS of bought software for the palm, and using it to it's extreme would be helpfull, the trouble there is that the palm is kinda slow and at the top of it's limitations in some areas, specially network wise, I'm very limited with my browsers in the palm, even the great Opera Mini runs slowly and barely on the Palm.

So I'm thinking of trying to save and somehow getting dollars, and buying either a cloudbook or an EEE, both are small enough, very powerfull, and most importantly linux based. I'm thinking the cloudbook since it's battery life is better afaik, and the HDD capacity makes it great for carrying even live apache based services on the road.

The rules basically are. cheap, if possible using what I have. It must have a keyboard a real keyboard, and it must be small. It must run only Linux, OS X, anything but windows/windows ce. and if possible symbian since all versions I've tried are terribly slow.


This is a particular need for software. For organizing so far I'm very happy with google calendars, and vitalist, both work well and both I can kind of sync around. My trouble is capturing stuff that will not become an action or a date. I need an efficient way to capture data for reference. Now I've learned the lesson, a lessom mom once thaught me. As a professional translator one of the worse days in her work life was when she was forced to move from Wordstar in DOS, to Word in windows. You see when she was typing fast (and she typed REALLY fast) it broke her stride to have to go to the mouse to select an option or command.

Likewise I've realized that while typing captured data it would be a real break to go for the mouse, doubly so if I am using a laptop pad to try and graph a diagram. So what do I think I need? I need something that can translate simple text to formated stuff, very much like a wiki works, using asterisks for unordered lists, and other wiki type.

For diagrams, something like built in graphviz, with a few more options, I've tought long ago that for a programmer like me it'd be easier to graph stuff on the fly programatically uwing the keyboard than trying to draw pretty diagrams by hand.

My trouble and where I want you to help me, is that I want all this in a standalone program, something I can drop on my usb pen drive, and run from Linux, OS X or even windows. And do this in a manner easy to save and search...

I'd preffer a native program rather than a Java based one, for speed reasons. I might even end up having to code my own in Qt4 for the 3 platforms, but I'd preffer not to. still. Well.

All in all those are my questions, and your feedback is very welcome and expected.

Our walled gardens, and other interop hurdles

Well. It's finally happened. The other day I finally had to sit with the "MS Guy" and work with him on providing access to calls on my system via SOAP.

And now, SOAP that once held hope for me, holds only sadness and despair, Let me explain why.

Some time ago I chose not to base myself on Microsoft products, not because I truly hate Microsoft, I DO hate Microsoft's marketing and selling tactics and politics, but being, or trying to be a pragmatic person and developer, I try to use the technology that fits the best for the requirements, but as many technical experts would know, and as most Perl developers hold to heart "there's more than one way to do something".

So when several solutions would apply equally, or near equally well to the problem, it's up to the developer's taste (in cases where he makes the call, as is always the case for me, I like making the calls so I look for jobs where I can make them) to decide which solution to use, which of course can be degraded in some cases, I think we all know about people forcing the frameworks or systems they know into a project however bad a fit it is for the project.

This hasn't been much of a problem to me since my biggest ability has always been learning new stuff rather than becoming a deep expert into one topic, it's a double edged sword but it's how I work and function and I'm not going to change it now.

Anyhow before this evolves fully into a rant. What I intend to say here is that we all have a style of working, a platform and a set of tools we feel most confortable in, and a personal vision on how stuff should be. And I'd call anyone who pretends to be entirelly agnostic or pragmatic a liar for I don't think we humans are able to attain pure objetivity.

So what the hell does this has to do with SOAP? Well there's a guy working for the company as a consultant developer trying to install a new, local "enterprisey" costly and powerfull invoicing system called Profit (Personally I've always found enterprise and powerfull solutions to be... well unmanageable do-it-all wannabes that require too many years of training to atain a pitifull performance, but that's MY vision as a agile programmer).

So, this system is coded in Visual FoxPro, and we've been trying to figure out a manner on communicating, I wouldn't allow him to plunge into my PostgreSQL Database (He almost sneered at me not using MS-SQL) for I fear for the integrity of my data and my DB has always been a sacred thing for me. I'll rarely allow anyone to drink my DB without some kind of protecting API wrapped around. So finally we decided on SOAP.

Here I hit the first hurdle. SOAP in itself is overkill, I agreed so before doing this (even without having done it before) and I agree with it now. As a pure solution I preffer json-rpc since I like the syntax of JSON a lot, and it's unhindered by all the layers of crust. But try selling a .NET guy on REST or on JSON-RPC, it's near impossible. So with the intention of being able to work together, we went the merry SOAP road, after all it's a well defined standard by now, and .NET has in theory (I don't use the product) very good tools for dealing with SOAP... Well the first hurdle I found was the disagreement from my ruby on rails peers.

For starters, DHH (it seems, this hasn't been verified by me) has decided SOAP is in many respects too inferior to REST, so the soap serving package, ActionWebServices has been dropped from the main rails code. It's relegated now to the disterred repository and not afaik too actively maintained by the core programmers... it IS maintained, but let's just say that I personally find it lacking in some areas, and installing it alongside a normal rails install was a fight. It was such a fight that I haven't been able to make my staging environment work with AWS yet, only my development box, and that's incredibly frustrating since debugging my staging is not so easy, more so considering no ammount of logging has helped here and it seems I'll have to remotely debug the code step by step.

Then, when my SOAP services where coded and returning nice XML responses that seemed to my inexperienced eye to be perfect, I tried hooking with this guy... and it took us all a night of work to get it to function.

First of all, in his version of Visual FoxPro, the "Professional tools found in .NET for SOAP services" turned up to be a soap script library glued into VF by this guy, and whose correctness I still suspect but being not an expert in either SOAP or VF I can't assure. To make a long story short, the guy is expecting a form of structured XML as a response, it is in a way an XML representation of a data table, with an embeded schema in XML and all. I could pass very basic types to him via AWS and soap, stuff like strings, dates, and integers. But when I tried to send a structured type, all he got was "garbage" he couldn't understand. And sending back an array of an atomic type like :int caused AWS to construct a custom complex type, and from this type the VF library was able to correctly obtain only the first element in the array (And wathever the guy said to me I still insist I'm sure there's a way to iterate the array on his side).

So for the two calls that we use what did we do?

Well for the (single) structured type I'm composing and sending him back a :string with semicolon separated variables... ugh... and for the array type I'm using REXML to construct a very basic copy of an example XML table structure taken from the VF help, and then embeding the resulting XML as a :string onto the SOAP response, which he parses into one of his XMLData objects to parse it.

Now it's entirely probable that %100 of our troubles have to do with the fact that both the guy and me are newbies at SOAP, hell the guy couldn't even understand he needed to hook into my WSDL definition URL rather than trying to hook onto the API access URI with his library until I pointed it to him from seeying the debug he was performing.

But there is also something very distinct here, while I perused SOAP documents, prior, during and after doing this first test run with him (we'll continue workign together until release and somewhat beyound), I saw a distinctive pattern. All SOAP materials where highly language and toolset centric.

That is, every single SOAP text I found explained SOAP not even generically but in the proper terms of the language and toolset it was geared for, and testing the results was even performed in those tools and languages as well!! This, in my view, for a protocol designed specifically to interconnect heterogeneous systems is more than a terrible practice, it's practically poison.

So, that's my first point against SOAP. Everyone seems to be doing a "universal" conduit in their own manner, or even not doing it at all... that does not bode well for a universal conduit's future.

The second point is complexity.

SOAP is horrible. I glanced over the W3c specs and I almost developed skin sores, the thing is large, horribly large and convolute. What's the problem here? well I think SOAP was infected by the EAV people. And don't think I have always been against that, I once was an EAV kid myself. I once thought "My programs need to be able to deal with anything, anytime, without my intervention, forever" so I designed for example a CRM with enough expandability in it to be able to also be used to calculate the distribution of sugar cubes in shipments to mars if it was necessary someday... of course these systems grew horribly complicated and unmaintanable, and very terribly innefficient at what they originally HAD to do.

With time I grew over it. And I know that the more lean my programs are, the easier it'll be for me to maintain them. And the more custom fit they and their data sets are to the application, the more solidly it'll run... and it's not like deploying a new version has to be a transparent even always, and even for those cases where it has to be I have planned a certain type of program that will help me a lot to perform them and of which I'll tell you more latter one day.

So still it seems the SOAP group was contaminated by people wanting it to be ready for anything and do anything. Say I for one would've been completely happy to have a parameter type akin to the capabilities of JSON which I think will cover all my basic needs, without complicating things too much, instead of that ComplexType abomination SOAP relies on....

But it gets worse, regarding databases and XML I was looking around, and it seems so far that what Ruby understands for a complex type is not the same C# understands of it and it's not the same VisualFox understands it to be... so, let me recap, the SOAP group produced a too varied, too complex definition with a minimum common denominator that I consider to be anything but minimum, and still SOAP is easy to use only between homogeneous systems? so heterogeneous system engineers still have to sweat blood and fight for the communication to work....

And then what are the alternatives? XML-RPC? JSON-RPC? or better yet in my view REST? well the problem with that on my mind is when you work with loosely fit teams on both sides or with no communication at all... the complexity in SOAP comes from the process of creating an universal standard with every side adding details they deem necessary or vital, and which all eventually end up agreeing on. So if SOAP being a standard in all it's complexity, sometimes fails misserably to interconnect heterogeneous systems, what would happen if you rely on a standard or half standard that is not backed by a giant like the W3c and pushed by behemots like Microsoft? it'd be worse, much much worse.

At least with SOAP, if I want to, say for example use Google's API over SOAP and there are some incompatibilities between what they produce and what I expect, I can juggle them and handle them on my side, as long as they conform to the SOAP standards, without any help from Google, so I don't need to interact with them.

But if I where to try so on a more laxly designed or regulated standard, or semistandard, it would eventually end up with me being unable to use the published API, or being forced to interact with the publisher or team to be able to make it work.

So the short, short conclusion is. Everyone is doing everything from their viewpoint, and in their manner... do you guys think it will be ever ever ever possible for systems and companies to exchange data and APIs in a universal form, without it becomming an overcomplex behemot that makes things more difficult, and without each person/company's policies and views poisoning the communicating pipe?