(generalisation warning)

Working for the ministry of education, science, and culture has been has given me a whole new experience on working with other people. I have usually been working with students, hackers, and software developers (yes I’m talking about free software hackers – not crackers or criminals – and yes, I make a distinction between hackers and software developers) but they all differ from the typical governmental worker (at least the typicial Icelandic governmental worker I have been working with). I have realised that these two groups of people I’ve worked with are essentially driven by two different things.

Students, hackers and software developers focus more on solutions, while governmental workers focus more on processes. There are of course individuals in both groups which relate more to the focus of the other group, but I’m speaking generally here.

This is not necessarily a bad thing and this mindset is quite understandable. Students, hackers, and software developers are used to being under the pressure of delivering stuff. Students have to hand in assignments and deliver answers on exams. I have never been a part of student project where any member even suggests that we should establish a formal hierarchy, decide on work flow, set up communication lines, and manage all work products in a predetermined and coherent way. The student project is of course managed somehow, but mostly by “first to suggest – first to decide” basis. Students usually ask: “What does the teacher want?” and then they decide when to meet to work on things and communications are just on the go, or through email or instant messaging (probably Facebook by now). The process is a by-product of delivering assignments.

Software developers are under pressure to deliver working solutions on time and within budget. Agile software development values individuals, interactions, working software, customer collaboration, and responding to change over processes and tools, comprehensive documentation, contract negotiation, and following a plan. In essence, agile software development favours deliverables over getting bogged down by bureaucracy. Again, there are of course some processes used to make decisions or keep people informed but these tend to be minimal since when you’re under a time pressure you can’t afford to let your developers hang in long coffee drinking status meetings. Keep the meetings short so that people can get to work.

Hackers are also focused on deliverables. There are some predetermined means to participate but they aren’t really processes. If somebody would post a message to a free and open source software mailing list and ask what the workflow is for contributing to the software the answer is likely to be something similar to: “Here’s the bug list/feature request tracker pick something or add something you think is missing. When you have something send your patch to the dev mailing list for review, if it’s good it will get added.” Once again, there are some processes used, for example release cycles with feature freeze etc. However, these processes are more of a paste onto the inherent community processes. You can continue to develop new features and work as normal but that won’t go into this release, it’ll just go into the next one. The hacker processes don’t disturb the community by forcing them to abide to specific rules, these processes just make sense of the community and show when one can expect deliverables in a neverending development cycle.

Government workers on the other hand are more concerned with how to do things rather than what they’re doing. This is based on the requirement that government institutions have to be able to show every step leading up to, and the reason for, a decision. This has the effect on government workers that they focus more on how they do things and constantly check if they are following the predetermined “right way” instead of focusing on what they’re trying to achive and trying to do it in the most efficient way.

I noticed this today when sitting in on a meeting about a new defibrillator. In the meeting we were shown how to use the defibrillator and during this presentation one coworker raised a hand and said: “We have a special security committee, would it not be better if they assigned one or two individuals on every floor the position of being a defibrillator user, and we could then call on these people in the time of need.” I can’t even begin to imagine what people would do if they should follow a predetermined process if a coworker would have a heart attack. “Wait, I have to find the defibrillation process document in our document management system to know who the security committee has assigned as defibrillator managers so I can run around and try to find them in process status meetings (which hopefully are being held in-house) to ask them to fetch the defibrillator and help bring Johnny back!” Is that really more efficient than just running down, getting the defibrillator and using it by themselves as quickly as possible? Human lives are more important than following the correct procedure!

This is a dangerous scenario but there are other problems which are not as dangerous, just really inefficient and irritating, like for example the fact that I have 5 bosses in a work place of 70 people (that’s about 7% of my coworkers and I still have to make all the decisions about software issues by myself).

I’m not saying that processes are bad or that solution driven individuals are better than process driven ones. But there’s a time and place for everything. I don’t think solution driven people mix very well with process driven people. Somebody will end up kicking and screaming. Either it will be: “You can’t just DO this (even it is logical), we have to decide whether we want to do this by having a meeting with the stakeholders” or it will be: “How long should I have to wait until you’ve called a meeting to decide on something we could get answers to through email or the telephone? I have nothing to do until then but waste tax payers money by drinking coffee!”

For me… I’m a doer, not a coffee drinker (meaning: I drink coffee while I do stuff, I don’t sit down to drink coffee since according to the clock we are now in the coffee break phase of our daily process) but it’s good to know why I’m different than most of my coworkers. It helps me cope.

Newton probably never thought his face would ever be mathematically defined, but while creating the video I posted in the last post I played around with creating an svg (Scalable Vector Graphic) which is just mathematical vectors (with some additions) used to draw a picture. I tried to imitate Godfrey Kneller’s portrait of Newton as hard as I could but I wasn’t happy with the result (I think I could have done the eyes better).

I still spent a lot of time doing this so I thought why let this work disappear with the next hard disk crash and decided to post it here.

Here is the SVG rendering of Sir Isaac Newton and this is the resulting image:

Both for pleasure and business I created a short 30 second video about why I think Open Access is important. In the video I probably butcher a hero of many scientists, Sir Isaac Newton (as hard as I try I won’t be able to change my voice or get a perfect 17th/18th century British English).

I hope you can at least see my intention and since it’s under Creative Commons Attribution ShareAlike you can go ahead a remix it. I would just love to see the idea of the video rendered perfectly. So without further ado, here’s the video:

I finally gave a presentation at the Nordic Perl Workshop 2010 today. It was a difficult birth, not the presentation, but the conference. It was planned to hold the workshop on May 1 but due to a volcanic eruption in Eyjafjallajökull. The workshop on May 1 was cancelled (instead Salve J. Nilsen got a bunch of Perl hackers to hack on Kaizendo.

When people started asking whether Nordic Perl Workshop 2010 would ever be held in 2010 Jonas Brømsø Nielsen took on the job of organizing it. I instead organized the Reykjavik Digital Freedom Workshop which included a workshop on open genealogy databases, Creative Commons, OpenStreetMap, the Nordic Perl Workshop, and others.

My talk was titled Quis custodiet ipsos custodies?, summarized as follows:

Testing software is a difficult but rewarding task. Software testers have to foresee all that can go wrong and viciously try to make software fail before it hits production in order to prevent failurs in a production environment. Various test procedures and approaches exist to aid the software tester and software developers in general. Unit tests, system integration tests, regression tests, black box testing, box testing, and white box testing.

Software quality matters and because all humans make mistakes we need to test our software. Tests are important but, like the code itself, tests are created by humans so how can we be sure that they can catch all defects in our code? This is an old philosophical question, initially raised by Plato in The Republic, but rephrased by Juvenal in his Satires: “Quis custodiet ipsos custodes?” Who guards the guardians? Who tests the tests?.

One approach to testing the tests is called mutation testing, an approach initially proposed by Richard Lipton. Mutation testing is about making slight modifications to the original software to see whether the tests will catch the modified versions. A popular tool for Java based mutation testing is Jeff Offut’s µJava (muJava). This talk introduces the reimplementation of Offut’s µJava in Perl. The answer to who tests the Java tests, can therefore be: Perl.

A PDF version of the slides I used for my talk.

With great power, there must also come — great responsibility!
– Stan Lee (Narrative in Amazing Fantasy #15)

All around us, in enterprises, schools, hobbies, and at home, we have computers. They are the modern day work tool. We live and work through these digital boxes. I remember, from my university days, students walking around like mindless drones whenever the university’s workstations stopped working. “What should we do now?”

It doesn’t matter if you’re a student or the president. We are all dependent on digital computer systems. All computer systems are dependent on the system administrator. The system administrators have more power combined than all international councils combined (after all some system administrators must be keeping the council’s operations running via their computer systems. There is a reason this shirt from Thinkgeek is funny:

It’s funny because it’s true.

Their job bears a heavy responsibility. It probably contributes to their much rumored ill-temperament that users and crackers continually try to use their systems in a way the systems aren’t designed to work (I would however say this problem is a result of the design and implementation of the software behind the system). Because of constant attacks and exploit trials it shouldn’t come as a surprise that the recommended way to run a system on a network is with an “open-up-only-if-necessary” approach. By closing ports, system administrators free themselves from attacks that exploit those ports. By opening up only necessary ports the system becomes safer.

But as the system becomes safer from malicious users, normal users must deal with the added burden of not being able to try out or develop new solutions. Users will have to restrict themselves to ports the system administrator is familiar with and trusts. The system administrator might open up the port for MSN instant messaging or Skype but close the port for Jabber. Lotus Notes port might be open because that’s what Lotus Notes uses but the IMAP port might be closed (there is also the possibility that the Lotus Notes system administrator doesn’t enable IMAP).

Any system administrator would become irritated by a user’s request to open a port just because they want to try out this or that software. What irritates me is that while they choose to close ports (often ports used by for free software because they don’t know that software very well) they still choose closed, proprietary software which they know nothing about the internal runnings of and open up the ports to that application (well I am also irritated over my system administrator who opened up the SSH port on a DHCP connection which meant I continually lost my port access everytime I got a new IP address — but that’s another story). This security measurement is still understandable. The whole system is their responsibility. You just don’t open up everything and invite people in. That would just be a bad judgment call. Even though it restricts users freedom to use what software they want to use.

But it just doesn’t seem right that they close ports but open up access to software they don’t know anything about. And they all open up port 80 for HTTP access. When evil-doers know what’s always open why shouldn’t they just focus all the effort to exploit that opportunity. Instead of making the exploitations of malicious users harder by deploying a more diverse system, system administrators just shown them the door.

But wait, there is a way to restrict HTTP access. Block access to certain sites, i.e. don’t let your users visit malicious sites. This improves security (which is why browsers have begun maintaining a list of sites, users shouldn’t go to). However, some system administrators decide to use some crappy web content-control software, like WebSense, to further restrict users. Content-control software has been criticized before (e.g. for choosing sides or for false alarms). An awful decision that restricts productivity while trying to increase it.

This crappy content-control might increase security (probably not that much) but is today more used to increase productivity and decrease bandwidth use. Users can’t stream media, users can’t access sites which are categorized as “a personal blog” or Facebook. This all sounds good. Why would anyone want to visit Facebook when they should be working. That’s just bad for productivity (some employees do waste a lot of precious time and thus productivity by hanging out on Facebook and blog sites but those sites might also enable them to do things the system administrators can’t foresee).

I believe an old way of thinking is getting in the way of a new way of working. We increasingly reach out to peers for information and peers are increasingly putting information online. Content-control software restricts this access. I have been searching online for solutions to a problem and had to go out of my way to get access to information Google pointed to (which had the answer I was looking for). Would it have been better if I had contacted a private company who’d have charged my employer an arm and a leg for information that’s available online for free, after I had gone through all of the hassle of contacting the other company and waited while they dug up the information (which would probably be the same information or something worse)?

Although I’m not on Facebook, I see the potential in the huge network you can just ask for help. We shouldn’t block these sites, we should help employees use these sites more efficiently. We’re restricting development of information retrieval and sharing. That’s just stupid and irritating and a bloody waste of potential productivity.

But we face an even more threat from system administrators. I have sat on meetings with heads of IT departments who brag about finally being able to control what software users can use. They brag about stopping users from installing anything except the software they make available. That’s just sadistic and it’s nothing to brag about. It’s the wrong kind of development.

I understand that this is supposed to decrease complexity in the system and make the system as a whole more maintainable. But they are effectively deciding how people should work and what tools to use. They are, again, making it easier for exploiters and attackers to focus on specific products. But far worse is the effect this will have on employees. They are effectively creating worker drones. They are defining how we should work and by doing so destroying employee creativity and problem solving.

This would be kinda (but still not at all) understandable if system administrators were professionals who knew everything about our work and tasks. But they are only system administrators (albeit powerful ones). We hire professionals to do their work the best way possible, but then we also hire system administrators who go ahead and decide how the professionals should work. That’s just plain wrong. System administrators have too much control. This is not a bright future. We’re going faster towards the digital dark ages than I anticipated. Those system administrators that take away the freedom to choose the best tools are restricting development and productivity.

System administrators have a lot on their plate. They have a big responsibility and do all they can to be responsible but it’s just outrageous that they have the power to constrain and restrict work. Information technology has the potential to increase productivity, creativity, and development. So why are we trying to take it away? Systems administrators have the power to constrain development by constraining how we do our work.

I for one can’t wait to get my most recent Amazon purchase Hacking Work: Breaking Stupid Rules for Smart Results. I hope the book will show me that development can’t be constrained and that the future is bright.

The human mind is a wonderful mystery. The hacker mind is a fun mystery. The creativity of hackers continue to amaze me. As I was playing around on GitHub I noticed a project called Inception. I had recently seen Christopher Nolan’s movie Inception which blew my mind. I have always liked Christopher Nolan’s work (especially his take on Batman).

Inception, the movie, is a fascinating movie about the human mind, our dreams and then dreams within dreams, etc. The movie can be quite complicated to follow for those with a short attention span (I had been warned but didn’t find it hard to follow, only hard to discuss and speculate afterwards).

So the software Inception, which I was merely interested in because of its title, was surprisingly (well for me) a piece of software related to the movie. Apparently, someone (A. R. Karthick) wrote a small piece of software in C, which gives its user (through command-line) a programmatic representation of the movie, something similar to (although the programmatic representation is the code itself):

…
[Arthur] Fighting Fischers projections in level [2]
[Fischer] interacting with Mr. Browning in hijacked state at level [1]
[Eames] doing recovery on [Fischer] who is shot at level [3]
[Yusuf] while falling into the river triggers Arthurs fall in level [1]
[Arthur] experiencing a FALL in his dream at level [1]
…

This piece of software, which I would guess is, for this California hacker, as well as a tribute to Christopher Nolan, just a fun, little hack where Karthick is playing around with C and assembly, but still by sharing it with the world I was able to take a look at the software, and play around with it.

I was very pleased. A small hack like this would most often just remain on someone’s computer, perhaps only shown to siblings or friends. But Karthick shared it with the world on GitHub (under GPL) and in the process gave me immense joy as I ran the software, looked at the output, looked at the code and contemplated about how fun hackers are. This little hack (which doesn’t look useful at all, has even been forked by Chris Dickinson who has ported it to Mac OS X).

Free software is great and a large part of its greatness comes from the community of free software hackers. These playful hacks are just wonderful and make our community more fun to be a part of.

In celebration of EFF’s 20 years as the cyberspace’s national guard Nina Paley made a really nice animated video covering the threat’s of cyberspace. I really like the judge there making fun of the Hadopi (three strikes and your out) laws. To congratulate EFF for 20 years, I decided to include the video here and at the same time try out HTML5 (stream EFF’s ogg).

So happy birthday EFF! May our lives online continue to prosper thanks to you!

I work part-time (soon full-time) as a government official for the ministry of education, culture, and science. I am responsible for all bespoke (custom) software development projects being worked on for the department of education. I am also hired to be a specialist on free and open source software since the ministry is actively pushing for the use of free and open source software in schools. This push is based on the Icelandic government’s policy on free and open source software which states that “[s]tudents in Icelandic educational institutions shall be given the opportunity of learning about and using free and open-source software on a par with proprietary software.”

When I started work I was handed a computer with Microsoft Windows XP. I did one thing on that computer before I got permission from people within the ministry to install Ubuntu GNU/Linux on it. The only thing I did (except for shutting it down) was to check that this 600 Mhz with 1.5G RAM computer could handle Microsoft Windows XP (and it barely did).

I have now been working happily on Ubuntu for the two months I’ve been working here but always been irritated by the mail system. The ministry (like many others) is using Lotus Notes for mail. I of course don’t want to use it but since I want to be able to send and receive mail I was kind of forced to. I therefore chose to access my mail via the webmail version of Lotus Notes. Everyday I found new bugs and irritating, non-configurable “features”. I have no idea why anyone says Lotus Notes is good software and why anyone would choose to use it, let alone so many Icelandic institutions. If this was the time for much used proprietary software to show that it is actually better than free software, it failed tremendously.

I therefore contacted the computer department (for all ministries) and asked whether they could open up for IMAP + TLS/SSL (also because the webmail only ran through simple, plain text HTTP) so I would be able to use a software I like. The computer department replied to this request with a visit to my office. The visitors were my contact whom I had asked and his boss, the department chief. The department chief did all the talking and explained to me that I couldn’t use Ubuntu since they hadn’t approved of the software and out of security reasons they were going to install Microsoft Windows XP on the computer. I have one day to store my files somewhere (they recommended that I use a personal USB pen) and then they’ll install Microsoft Windows XP and I will have a much more secure computer than with Ubuntu GNU/Linux on it.

Microsoft Windows XP with a standard set of software for general use is going to be much more secure and less vulnerable than Ubuntu GNU/Linux (although I asked when I installed the system whether I should use an encrypted file system and I was told that much security wasn’t necessary). I was told that we have to use Microsoft Windows XP because I have access to the minister’s files. So a couple of guys in a computer department with their crappy, brute-forcible, unmodifiable passwords, their standard software packages, their plain text webmail, their non-existent encryption and digital signature means, and their automounted (or what it’s called in Windows) disk drives have made a professional decision that they know Microsoft Windows XP is better than GNU/Linux. I wonder what Google missed?

I know what Google didn’t take into account. My computer will be much more secure when they’ve installed Microsoft Windows XP on it than when I was managing my own Ubuntu GNU/Linux system. That’s just because when they install Microsoft Windows XP on the computer, they’re going to turn a working computer into an expensive desk decoration since I’m not going to use it. It’s back to bureaucracy basics for me. Plain paper (perhaps using some form of cipher) and taxi cabs to send documents for me.

It’s just like Tenacious D said:

“The government totally sucks, you motherfucker”

For the last weeks I’ve been trying to get transactions to work with Mule. I’ve been integrating two databases which have to use information from another database and send a message to a Java Message Service (JMS). Things are not really going as planned and I’ve become irritated at transactions.

What I have to do is fetch information from an Oracle database, this information is an XML document split into parts saved in varchars (yes I know, this is not smart and this is not how you do things but I am only the integrator and I’m just supposed to work with things as they are). This XML document I need to parse and get different information which I perform some computations on (with xpath) as well as fetch new information from another Oracle database, and then insert the final information into three different tables in a Microsoft SQL Server.

It didn’t take me long to design the whole Mule model which works with all these data sources and performs all the computations, even though I had ill-defined requirements and needed to talk to other developers quite often. After that I wanted to test things out and talked to another developer who created the tests for me (he also did a informal code review which resulted in: “Yup, looks all right!”).

Running the tests was the beginning of headaches and frustration. Everything worked, except that if one of the three inserts into the MS SQL databases failed they didn’t rollback. Of course not, I wasn’t using any transactions.They hadn’t been using them either and were using a home-made, bottleneck message manager and monitor. The reason they hadn’t been using transactions before was basically: “Well, we weren’t smart back then”. That’s forgivable. Let’s just get transactions working on this piece of Mule model and then we can reapply them to former Mule models.

Getting these transactions working was and still is a nightmare. First, let’s define the transactions. Nothing works. Things that shouldn’t be running after a failed transaction still run. Why? After days upon days we finally figure out that you cannot split transactions between services. They all have to be a part of the same service. Redesign of the code. Everything important is no in one big, ugly service but at least we can use transactions, right? No.

Now we face the problem that the transactions aren’t running on the same resources. They’re running on a MS SQL Server, they’re running on an Oracle server (which updates the status in the Message Manager to finished), and they’re running on an JMS server. That shouldn’t be a problem. We’ll just use XA transactions. Mule supports them. They’re pretty straight-forward, right? … right? No!

After changing all the sources to XA data sources (and XA connectors) I run a few tests and everything works great. Error rollback and clean messages run through with a smile on their faces. Everything is too good to be true which turns out to be true when I run a stress test. After only a couple of hundred messages the whole system grinds to a halt. The databases scream in agony over all of the connections that are open. Wait what? After some investigation, with another developer, we figure out that each message which comes in gets a separate thread, and each thread creates three connections to the databases and then after everything is done, Mule doesn’t close the connections. So every message results in three open connections which are never closed. It didn’t use to do that before switching to XA. What’s so special about XA transactions?

After many effortless refactoring trial and errors we give up. XA transactions are only for grown-ups. We decide to just go with regular transactions and declare no transaction on the JMS and Oracle end points. After all it’s only the Microsoft SQL databases which really need to rollback if something happens.

Stress testing works great. Mule closes the connections and the databases are happy… but I’m not happy. There are other problems with normal database transactions. In some cases when a bogus message is sent, it should be forwarded to the exception handler, which it does but the exception handler complains it is not a part of the transaction. I’ve declared that it shouldn’t be a part of the transaction (I don’t want to rollback an error status). I try everything. I put a <jdbc:transaction action="NONE"/> on every endpoint in the model just to try things out and I still get a message saying there is a transaction available but the endpoint (which doesn’t tell it’s name but from what I can see it is a part of the exception handler) says that it hasn’t been told to participate in the transaction.

First of all that’s just stupid. Some cocky endpoint is saying: “Hey, these guys are special, I want to be special too”. Why should it, I haven’t told it to be a transaction (well actually, from what I can understand from the error messages, I haven’t told it to be a part of the transaction or not to be part of the transaction). I don’t understand this. Transactions are a good idea but why must they be so difficult to work with. I’ve actually started thinking about just throwing them out the window and write my own transaction handler in my exception handler. That only takes me a few seconds instead of a few days:

delete from my_table where id = id_of_newly_inserted_row;


…but this is not as good a solution as working transactions which are simple to define and use.

Last year I was contacted by someone who wanted to form a company with focus on free and open source software solutions. We haven’t formed the company at this moment even though we’ve started working together, because we wanted to see if there is an open slot in the marketplace (which became obvious after just a week of collaboration since customers just piled in) and then see if we could work together. So we are basically three individuals, investigating the opportunity of forming a business. I am the only one who has no experience in the business, the other two have 10 and 20 years of work experience. I was only working on my Ph.D. dissertation when I was contacted and had no idea about how to run a business. Now I sit here happy we’re just looking at the opportunity. Because they, plain and simply, suck!

I’m currently sitting at a work meeting with my maybe-soon-to-be business partners discussing free and open source solutions for a particular customer. A rather large customer. It has taken three weeks of meetings with a representative from the customer to understand what solutions they are looking for. I haven’t been a part of this project since I’ve had a lot of other projects that I’ve been working on but they called me in to go over things before the final report.

Sitting at a working meeting and blogging shows my interest in what they’re doing. I am not interested! What I am seeing is that my maybe-soon-to-be partnership has in a few minutes become holy-crap-I-must-get-out-of-this partnership. For the last three weeks they haven’t done anything at all.

What are the needs of the customer? They don’t know. They are sitting and talking about how awesome Alfresco is, how awesome Zimbra is, how awesome this and that is. Then they throw out the statement: “vTiger looks cool, lets recommend vTiger” and then they ask: “What can vTiger do for them?”. I just sit baffled, at a loss of words. I can’t even manage to say: “Does the customer need a CRM solution?” My soon-to-be-ex-possible business partners have no idea what the customer needs.  They don’t even know how many the users are. They’re just shooting into the dark. Guessing what the customer needs. Now they’re looking at videos on the Alfresco website. So it appears that their idea is to recommend solutions, they know nothing about, to a customer, they know nothing about!

This is not how I would like to do things. This is not what I think of as a good work procedure. The problem is that I can’t do everything. I can’t hold their hand and tell them how they work with customers to specify the requirements. I can’t show them how they measure software against requirements or how they can estimate the maturity of free and open source solutions. Well I can, but in order to do it I’d have to bend the space time continuum just so I can have the time to help them out in the small amount of time they have before the deadline (which is in three days).

The only thing I can do is sit here and think how the heck I’m going to get out of this possible partnership. I need to find some sane business partners. This is insanity!