Personal projects, ideas, and the occasional comments on life

I have been in the job market for a few months now looking for work as a web developer. A certain percentage of companies have decided not to work with me but I have decided not to work with a larger percentage of them. Why turn down a job in this market? To put it bluntly, because most of you people are showing up for a pay check and that is the only reason you show up. If that statement shocks you then perhaps I am directing this post at you. Yes, you.

Take a seat, you need to see a guy about a thing.

Pop quiz: How do you kill a man?
The obvious aside, one sure fire way is to stick him in a situation, virtually every day, virtually all day, where he is unhappy. Some place that creeps into his head and, after waking up one day, he decides he hates. It will probably sneak up on him. Not long after this realize will feed into itself and turn into less and less productivity for your company. This is the state of the “code factories” in most IT departments. All the benefits and job stability in the world is not worth it one byte. I, personally, will take a job way below my previous pay grade at an unstable start up in order to care for my family before I come home and take all that out on them. Although I have nothing against working for larger companies I do not favor them because they tend to be less interested in the people and making good product.

I have discovered a few warning signs that a place is too far gone to be worth any amount of compensation:

• They have a sprawling HR department. Your interview goes well. Everyone loves you and wants you on their team. Now they need to go through HR. All of a sudden the best person for the job is reduced to numbers. “Sorry but we have decided to move in another direction.” What is a programmer with storage experience worth? What is the minimum a second person is willing to accept just to manage the storage when there is not enough storage-related work to go around? I honestly have little idea but I can not imagine penny wise, pound foolish is the way to go with your critical infrastructure.
• They think they just need a “frontend” or “backend” person. Maybe you learn about a subject, not because you love it, but because that knowledge will improve the thing you do love. Maybe it is a necessary evil and you can not do without the related technology. No matter the reason you do not write “real” web-based applications in a vacuum; There are other parts you need to consider for a solid product. Sure, you are no DBA but I think you may agree that knowing that MySQL almost never runs your query as you entered it has a big impact on your final decision of how to do something. Did you know that PHP arrays (pretty much the only data type they give you) are not arrays? They are hash tables with syntaxical sugar. That tight loop you are porting from your legacy CGI C executables are going to need to be rewritten from scratch if you are going to reasonably support the same user base you did before the port.
• Company is hopping on the latest thing for no good reason. I love new technology and techniques. I love to play with them. I love to build real-world products with them that you can put your hands on and get that “wow” response. One problem with the web, however: Anything new is useless for years until it is further adopted. Either that or it is not new at all and you have been taken in by clever salesmen. Obviously if you need to wait there is nothing you can do until most of your audience abandons Windows XP and Internet Explorer 6 (come on, Mom, give it up already). But what about the “new” things? Take MVC, for example. I am a massive fan of MVC, even back when it was called OOP. There is nothing new there. Although I am all for OOP– and thus MVC– having a non-technical manager making a decision based on something they do not understand as well as they think they do is a problem. You hired a programmers to program, let them write the code.
• Culture can only be forced so far. When I talk about “culture” I am really referring to people getting along with each other, complementing each others skill sets, being willing to admit there are things they do not know, and working towards an end goal as a unified front. People who want to be there are there because… well, because they want to be there. People who joined the company softball team because they are afraid it will reflect badly on them were already unhappy. Happy people work harder. Money is one of those things that is very important when you do not have any of it. We all need to eat, stay out of the cold, and not have our hair turn white when we think of the basic things we can not do for our children. Getting a nice salary bump is a fantastic pick-me-up but it is no long-term solution to your employees ability to be more productive for you. I have no links to back me up on this but, in my experience, the turn around time for companies that think money is the answer to everything is much higher than people who truly enjoy their jobs. Hell, people– such as myself– will accept less money for other perks that cost a company virtually nothing in comparison. I have my own personal preferences but compensation time or surprise days off for a job well done can go a really long way for some ones mental state.

I realize some of (all of?) this comes off as a war cry against The Man™ and makes me sound like a rebel. I suppose in some respects I even am. I am not anti-big company nor am I trying to convince anyone to change anything they are already doing that works; A company is there the make a profit and my job, as a programmer, is to do with what I have and churn out code as fast as I reasonably can for said company. There is a balance there that, in a perfect world I would rally against, but in reality I see it as a necessary evil. These are just my thoughts on the subject as I have seen them over and over recently. Most of the old ways of doing things have lasted this long for good reasons, but so has evolution. Keep in touch with your people, communicate often, and remember the long-term far out weights the short-term nine times out of ten.

I have been working with a whole mess of data as of late in a series of MyISAM tables. The short of it is I have a 256GB XML file that I want in MySQL. A few tools already exist designed for this specific data. Problem is they import everything and I only want about 75% of the data and they were not as fast as I know they could be. So, being how I have so much free time, I decided to explore this a bit further and wrote my own PHP script to handle the job.

I came the conclusion that the indexes were what was killing me. The database ends up being ~3.5 billion records– yes, it is nearly that much– and indexes are kind of a very important thing. First I tried with the indexes enabled for the whole import. I do not know how long it would have taken because the larger the database grew the slower it got, kind of like how you can never quite reach the speed of light. After that I tried creating the tables without any indexing and using ALTER TABLE after. It looked good at first but when you have such huge tables it gets slower and slower with each added column (IE the first column indexed plenty fast but the second was half, the third half that, ect). The problem was, using ALTER TABLE, you have to add them one-by-one.

After further researching it seems to me that the best way to do this is to add the indexes while creating the table and then just disabling them. Before any of the data is inserted you do a ALTER TABLE table DISABLE KEYS and SET FOREIGN_KEY_CHECKS=0. After you have done your massive, massive import you just do SET FOREIGN_KEY_CHECKS=1 and ALTER TABLE table ENABLE KEYS to re-enable them. This way MySQL does all the work at once and virtually any time you let MySQL handle the work itself– versus multiple calls via external script– you end up with a big benefit.

A few notes:

• The primary keys are still updated during insertion. I suppose you could use ALTER TABLE after to add those but I am pretty happy with the speed of this setup.
• I also tried INSERT DELAYED (both alone and in combination with the above) and it resulted in a ~10% increase in time.

Edit 2011.10.30
I just found this article. It seems I was wrong about having to index one column at a time but there are a few caveats. The most notable one is that the order in which you specify the columns has a direct result on if it can be used or not. Any way, this is a huge topic and that is why there are DBAs. Good luck to us all…

I was out with friends last night and one of them brought his Kindle with him. He showed it to an inquisitive second friend. This second friend expressed an interest in owning one of his own. I, of course, had to step in.

Some animals have remained virtually unmodified for a long, long time. Why? Because there was no reason to change. They were already doing something that worked and, evidenced by the fact they are still here, had no immediate need for something new. This approach tends to give us everything we need in the long run without over complicating things. For example, paper-based books work great. They are light, easy to use, universally understood without any learning curve, and have remained virtually unchanged in most of human memory.

So why do I fear books moving into the future? Take Amazon for example. They have removed books from users Kindles without notice before. Unless someone enters my apartment or steals from me on the subway once I have the information it is mine; I own it and no one can take it away from me. This Big Brother thing Amazon has going on is not only wrong but only gives the illusion of control. Also while on the topic, what happens in a few years when Amazon has moved on and they no longer support their 10+-year-old tablet? If the service is gone so are "my" books.

The new e-ink displays are very neat. They only require power to display an image, which they only need to do once, and that power usage is impressively low. Still, though, why did I need to stick a battery in my perfectly functioning book in the first place? Sure, an electronic device may be able to stay in stand by for a month or more but my paper-based books can sit on a shelf forever and still work as intended.

Kindle-Man also felt it very important to tell me that he can carry some insane amount of e-books at once. The obvious response to this is that I am only in the habit of reading one book at a time. Perhaps if I am planning a long plane flight I may pack two books. Is this really so much of a benefit that I would drop so much money and risk the above “features?”

I love technology. I make my living on technology and do it in my free time as well. I, however, see the need to keep technology out of certain areas for at least these reasons. Also let us not forget the Battlestar Galactica, people!