Browsed by
Author: Nicholas Ingrassellino

Recursively Remove .cvs/.svn/.git Directories

Recursively Remove .cvs/.svn/.git Directories

I tend to keep backups using several methods depending on the situation. Some times I run a script that invokes rsync with rolling, date-based backup. Lately I have been experimenting with compressed/dedup archives/filesystems.

One nearly constant annoyance, though, are those pesky .svn, .cvs, and .git directories. They serve a purpose, but not within my backup that already versions its data.

In order to be rid of them I just run rm -rf `find ./ -type d -name [directory to remove]`. If you wanted to, you could stick this into a script within your path:


if [ $# == 1 ]; then
rm -rf `find ./ -type d -name $1`
echo "Script requires one argument."

This would allow you to pass the directory you want to recursively be rid of in whatever directory you call it from. Note this script will not handle spaces in the argument but for this we do not need it.

Minimal Ubuntu 12.04 Install with Only MATE

Minimal Ubuntu 12.04 Install with Only MATE

In a previous article I talked about replacing Unity with MATE on Ubuntu 12.04. It is working out very well for me but I still feel like there is more that can be done. Too many resources are being spent for things that got left over from Unity and it is bugging me.

Minimal Ubuntu Install

First things first, we need to do a minimal install of Ubuntu. You have two options here:

  • You can use the netboot disc. When prompted later in the installation do not install any of the pre-configured setups as we will install the packages we need later.
  • You can use the alternative disc. When the disc first boots just press F4 Mode and select “Install a command-line system“.

I like the netboot option best because it downloads all the latest packages during install so there is less mess. I have also had random problems with it not being able to find some packages since Ubutnu 12.04 was released. If that happens the alternative disc works just as well and will be faster since it has all its packages on the disc. It really does not matter which you choose for our purposes.

Both support encryption and both support RAID so feel free to use those if you like. I will not be covering them or the rest of the installation in this article.

Install MATE

Since everything here requires root just do an sudo -i first. Remember to log out of root (Ctrl-D) before starting MATE. Would be a good idea to do an apt-get update && apt-get upgrade before, too.

First we need to add the MATE repositories. Using nano add the following to /etc/apt/sources.list:

deb precise main
deb-src precise main

Next we need to install the packages (you can answer the hddtemp question however you like when it comes up):

apt-get update
apt-get install -y --force-yes mate-archive-keyring mate-desktop-environment xinit
apt-get update

Now all you need to do is login, type startx, and you are done! … sort of… if you are happy with the way things are you may now go away.


You now have an fast and low-resource traditional desktop at your finger tips with the stability of Linux and all the packages Ubuntu has to offer. I am so proud of you.

On my dinky little Atom-based netbook, with MATE running, this setup eats virtually 0% CPU and ~150MB of memory. Disk usage sits at ~1.7GB which is a little higher than I would like but storage is cheap so I am not too worried about it (probably all the xinit dependencies). Boot time is in the mid-single digits and that is on a 5,400 RPM disk. I imagine a decent USB stick will be very similar but I have yet to test that.

I am very happy with all this. Of course you, dear reader, are not so you will continue reading… blood from a stone, Internet Person…

MATE Extras

As of right now your menus are looking pretty bare and your audio might not work. By virtue of what I set out to accomplish there is very little installed. Here are some packages that may prove very helpful:

apt-get install zram-config preload synaptic gparted brasero mate-media-pulse mate-settings-daemon-pulse mate-bluetooth bluez-cups cups cups-pdf system-config-printer-gnome mate-conf-editor wine libreoffice libreoffice-pdfimport firefox pidgin thunderbird xul-ext-lightning vlc gimp gimp-data-extras jockey-gtk usb-creator-gtk network-manager-gnome

I am sure you will want to install a hell of a lot more but I will leave that up to you. You may want to install less, in which case do an aptitude show [package] to see what a package does before installing.

For some reason there is no supplied way to do “nothing” when the laptop lid is closed. If you want this functionality back start up mateconf-editor, set /apps/mate-power-manager/buttons/lid_ac and /apps/mate-power-manager/buttons/lid_battery to nothing.

Auto Boot into MATE

Maybe you want to automatically boot into MATE. Maybe you are just that lazy. Maybe a white-on-black terminal killed your father and now, as a result, you are too afraid to face one… fear not, citizen! We did not install a display manager since MATE does not currently come with one but we can easily fix all that with a single command:

apt-get install slim

slim is very light weight which is why I chose it. The trade off is it does not support much besides logging you in. After installed just open /etc/slim.conf and set default_user to your username and auto_login to yes (make sure you uncomment both, of course). The MATE Wiki also recommends that you stick exec ck-launch-session mate-session in .xinitrc but everything works fine without it for me so experiment with it.

If you really, really do not want to install a display manager (like me) you could just stick startx at the bottom of .bash_profile (if it does not exist create it). This file sits in your home directory and is executed every time you log in.

Known Issues/Notes

There are a few things that are not show-stoppers for me but I want to spend some time fixing later.

  • Want to get Compiz and all those snazzy effects working. I have been experimenting with varying success. Have not gotten it to work just right yet.
  • I miss my Open With Archive Mounter from nautilus. Just either need to figure out the right package or Caja configuration (not sure which yet).
  • VLC does not inhibit screen blanking when running in full screen. This is really, really annoying to me as I use my netbook for entertainment while on long trips.
  • For some odd reason nm-applet (part of network-manager-gnome) will not start up until you comment out all references to your primary network adapter in /etc/network/interfaces and reboot. I only noticed it after I tried to connect to my wireless network so if you only have a wired connection you may not care about this.
  • Putting startx in .bash_profile may cause an issue if you log in from anywhere other than the terminal. For example, an SSH connection. Also if you need a terminal after breaking X, MATE, or something this might make it harder to fix. I am sure there is a better place to put startx but I have to look up where. For now it is probably better to just go with the display manager method above if you do not want to be bothered.

I am sure I will come across some things as I play with this some more. When I do I will update this article. Please feel free to leave any comments with any issues you find and we will see what we can do about them.

Of course, if you are not insane like me, you could have just installed Linux Mint with MATE in the first place.

Update 2012.09.06
I am not playing with this any longer since I got Linux Mint 13 encrypted up and running. It is not likely I will update this article further. The two are not exactly the same but I have little free time so something has to give.

Replacing Ubuntu 12.04 Unity with MATE

Replacing Ubuntu 12.04 Unity with MATE

I am a stubborn man. But part of that stubbornness was the result of a personal kind of evolution, not an unwillingness to change.

For example, my desktop. Microsoft, with Windows 95, gave us something great: the desktop we know and love. Trying my best to ignore Metro— for the sake of this post, at least– some of my habits have changed to fit this model but, more importantly, I have figured out how to change it best to fit my needs. There is room for improvement but I can only imagine those improvements adding new functionality, not removing old, show-stopping hurdles.

Ubuntu, in their awesome, massive growth and other-wise fantastic direction from their leadership, have decided to make a major change. They have decided to abandon our old friend for something pretty radical with Unity. Unity is, for the most part in its latest incarnation, a pretty great thing. It leaves some old hardware behind (always takes balls but some times is needed) in order to innovate and move us all forward and I have grown to like it for some tasks. For example, my parents and grandparents have flourished using it. They all love different things about it but it works for them and I am happy to use the right tool for the job.

However, although my family finds it easier with is bigger buttons and everything-merged-into-one-place design, I find it much harder to use for what I do. I would call myself a power user and, as such, I have certain requirements. Instead of going into those requirements here I will quote Linus Torvalds because I feel he summed up the meat of it pretty well:

I used to be upset when gnome developers decided it was “too complicated” for the user to remap some mouse buttons. In gnome3, the developers have apparently decided that it’s “too complicated” to actually do real work on your desktop, and have decided to make it really annoying to do.

Here’s an example of “the crazy”: you want a new terminal window. So you go to “activities” and press the “terminal” thing that you’ve made part of your normal desktop thing (but why can’t I just have it on the desktop, instead of in that insane “activities” mode?). What happens? Nothing. It brings your existing terminal to the forefront.

That’s just crazy crap. Now I need to use Shift-Control-N in an old terminal to bring up a new one. Yeah, that’s a real user experience improvement. Sure.

I’m sure there are other ways, but that’s just an example of the kind of “head up the arse” behavior of gnome3. Seriously. I have been asking other developers about gnome3, they all think it’s crazy.

I’m using Xfce. I think it’s a step down from gnome2, but it’s a huge step up from gnome3. Really.

As such I had, for a while, switched to Linux Mint like so much of my cohort. I still love Mint but there is a smaller community there and less Mint-oriented information around the web. Sure, majority of the stuff I have come across for Ubuntu also works for Mint, but I only left Ubuntu because of Unity. If I can remove that annoyance I have no reason not to move back as there is safety in numbers. After all, is that not the whole idea behind the open-source philosophy?

Let us rip that mother-loving (see, I made a funny) interface out and replace it with what was not broken in the first place, shall we?

Remember to backup anything you touch before doing this because who knows if I am secretly working for Microsoft and spreading false information to make people think Linux sucks? It could happen, people!

Open up a terminal and run these commands as root:

add-apt-repository "deb $(lsb_release -cs) main"
apt-get update
apt-get install -y --force-yes mate-archive-keyring
apt-get update
apt-get install mate-core
apt-get install mate-desktop-environment

These will install the necessary packages. Note that you must install mate-core and mate-desktop-environment in separates command for– what I assume is– a race-condition situation. If you do not it may work, or it may break your UI. I never really looked into why as the solution is plenty easy to implement and does not harm anything.

Next you have to change a few configuration files so Ubuntu knows where to find some things. Edit /etc/xdg/autostart/gnome-settings-daemon.desktop as root and replace the contents with

[Desktop Entry]
Name=Mate Settings Daemon

Next edit /etc/xdg/autostart/mate-settings-daemon.desktop as root and find the line that starts with Exec=. Replace it with

Exec=/usr/bin/mate-settings-daemon --no-daemon &

Next edit /etc/X11/Xsession.d/80overlayscrollbars (you may have to create it) as root and add the line


If this is a well-traveled installation of Ubuntu you may also want to change the default text editor to pluma (MATE’s version of gedit) and the default file manager to caja (MATE’s version of nautilus). pluma is not such a big deal– I just hate the latest replaces for the find and find and replace functionally in gedit— but caja added some wwwaaayyy-overdo functionality like undo/redo.

I am not sure if Unity will work after this and I never tried despite it only being a few clicks away. Honestly I do not care one tiny bit. I did notice that a significant amount of memory is no longer being used so my netbook is much happier when I am pushing it a little too hard. Once I get Ubuntu running on my Slate 2 I am also betting it will appreciate the spare CPU cycles, as well.

That crack about sabotaging Linux hurt to write. I owe me an apology. :'(

Dear Women: Us No Think… You Purdy… Too Good

Dear Women: Us No Think… You Purdy… Too Good

A male buddy of mine just send me a link to Cosmo’s 44 Most Ridiculous Sex Tips. I read the first few and I was overcome with two thoughts: “this is hilarious” followed by “holy crap, what if people are taking this seriously?

To help out the second category of you– which I pray I am imagining– here is my open letter to the females who have had to put up with us:

Dear women,
Having trouble understanding a man? Are you so confused that you have turned to a magazine for something other than its entertainment value? Let me help you, in my limited capacity.

Try the following: Consider the dumbest woman you know. Now imagine taking a pencil, sharpened to the finest point you can imagine. Cover it in acid, red ants, cosmic radiation, and violently shove it into said womans brain. Wiggle it a bit to make sure you get all those IQs!

Now, given the severe brain injury, consider all the simple things this woman could want or desire compared to before. No longer will she want “iced water, with a twist of lemon, in a tall glass” but just “water.” The hose will be more than fine. No longer is she looking at a map for the best way to get from point A to point B. Now she just starts walking in the general direction of point B. How does her hair look? She has no idea because it never occured to her before leaving the house and now has to go out of her way to find a mirror.

Now replace “she/her” with “he/his.”


(Try To) Love,

If you are still confused my advice to you is simple, female person: Learn to love boobs as much as I do or to happy while also very frustrated. If you are trying to think of a third possibility you are over thinking this, too.

Yes, yes, I know. These are Cosmo’s “Most Ridiculous Sex Tips“. Lighten up, tight ass.

What Interview Questions to Ask and How to Land a Job You Love: My Experience

What Interview Questions to Ask and How to Land a Job You Love: My Experience

As I have previously mentioned, I had been in the job market for months. Had a number of pretty solid offers but I did not feel the cultural fit was there for me to be happy. In the interest of sharing my experiences, here are the questions I wrote down to ask every time I went on a first or second interview. This is not a complete list but a mix of important questions plus the (odd) ones most important to me. I am also adding notes for my reasoning behind them for the purposes of this entry. I would suggest, after looking at the list, you simply disregard the things that do not apply to you– or more importantly– the things you do not care about.

  • At will? Contract? 1099? W2? Corp-to-corp?
    Consider the tax implications of a 1099 or corp-to-corp. Seriously, big difference. Remember that if an at-will employee can leave any time– two weeks notice is a good idea because everyone can always use more references– it also means they can let you go for absolutely no reason at all.
  • What benefits are offered? Are they fully paid for by the company? How much do I have to contribute if not 100%? How long will it take to kick in?
    Will a lapse of insurance be a problem for me due to an existing condition?
  • Insurance offered by the company? Can it cover other people? Worried about in-network? HMO, EPO, ect?
    If you are getting married it may be a great idea to be able to offer such a thing to your spouse. That way they do not have to worry about it and do can whatever (maybe even take care of the rest of the familiy if needed instead of the cost of a day care).
  • Do they offer a retirement plan (401k, ect)? Do they do any matching? If so how much?
    If you start this at 18 you are sitting pretty. Starting later is not nearly as good but still well worth the tiny amount it costs you now. Trust me on this, you will not be sorry. If a potential employer does not offer it you can always hire a company to do it for you and it is still very, very much worth it.
  • What is the dress code?
    I just hate folding and ironing things with a pasaion. Feel free to ignore my insanity.
  • Flexible hours? If I put in 16+ hour days is there any compensation time? Can I work from home at least one day a week assuming I am not required in the office?
    This was very important to me. Even more so if there was no overtime. Your time is the most valuable thing you have because it allows you to do everything else. Do not kill yourself so someone else can get rich and you get shit. There are lots of things worth more than money and, if you are willing, a company may be willing to make non-standard accommodations if you just ask so everyone is happy.
  • How do meetings with the other departments go? How often (on average, not with the last big, once-in-a-life-time project)?
    I, personally, used this as a metric to see how big a company was and how much red tape I would have to deal with. Not really a great measure of anything but an indication.
  • Tandem projects? How many things will I be working on at once?
    This may not pertain to you or maybe you like it. I just wanted to make sure no one expects the impossible out of me.
  • Amount of time spent working vs meetings? Would they say my department makes the decisions it has the skills to make or other departments?
    I was worried that some big-wig boss man would be making decisions based on nothing but cost. Penny wise, pound foolish kind of thing and I wanted no part of that.
  • Can they walk me through what most of my days would look like?
    Just because they say they are filling one role does not mean I think they know what they want. Make sure you are clear on what you are committing yourself to. Use your own judgment but do not be afraid to say “no” if the job is too huge and there is little relative up side for you. Again, do not kill yourself so someone else can get rich and you get fired because they sold the company that you have no real stake in.
  • Is there a Do Not Compete clause? If so does it apply to post-employment and for how long?
    Wwhen you leave you want to make sure you can still do what it is you do. A DNC that is too broad will fuck you. Do not be afraid to request changes if you feel the spirit of the law is there but the wording is not. I have done this with every DNC I have ever signed and never once has anyone become upset because of it.
  • Are any personal projects you generate outside of company time and company equipment property of the company?
    Are things you do on your own time on your own equipment property of the company? For someone who loves what they do this would be a really, really bad thing. I was very, very afraid of this.
  • What is the salary they can offer?
    Is it a job you really want? Is it worth it to accept a little less for a better chance at the gig? Perhaps it is cheaper for them to put that difference into benefits when it would have cost you a lot more if you were to do it on your own. Make suggestions if you want something.
  • Provide equipment?
    Are they expecting you to spend your own money to do your job? Will you be reimbursted? Is there a limit?
  • Bring own equipment?
    Some companies do not like the idea of you using your own equipment (especially things with long-term storage, like laptops, ect). Make sure, if you need something and they will not let you use yours, that they are willing to spend the money required to get the job done right. If they think you suck because you produce crap it may just be because they were too cheap and not because of you. Still, it reflects badly on you and that is not something they will hear very well.
  • What time do you start your day? Flexibility?
    I hate mornings. Like, a lot. This may not matter to you. I also want to be able to stay 16 hours one day and four the next so that was important to ask (but be careful about how it comes off).

Check out the recruiters! There are people out there that have just as much stock in you getting a good job and you do not have to pay them. Generally they only get paid if you get the job, get paid relative to however much you are offered, and none of that comes out of your pocket. It is win-win for you. I am more familiar with the IT-related resume-posting sites such as Dice but there is also Monster so use those to get in front of a lot of eyes (recruiters included). I, personally, did not do any of the heavy lifting myself (craigslist, for example) and let the recruiters find the jobs but there is no rule you can not be doing both. Hell, there is no rule you can not be doing five interviews all at the same time (except for the drain on your energy, that is). This is not a marriage and you do not owe anyone anything until you sign those papers: Be a whore.

This whole process can be intimidating so I offer this advice: Do not act like it– seriously, most people hate me for this very reason– but think of yourself as the best; Fuck everyone else, they suck. The truth of the matter is 95% of the population are just doing what they need to do to get by and keeping their heads down. Always put your best foot forward but remember that doing 50% of your best will be worlds better than most of your competition. Also show off what you did in the past because you wanted to and why you loved doing it. Try something new. Tell them about your failures, why you failed, but how it was still worth doing. Do not be afraid of what you will do, not do, or say in an interview. This is as much about you getting what you need and want as much as it is for your interviewer.

Hope this helps anyone who might be looking for a job. It is the accumulation of me adding to and modifying this list over the course of a few months of non-stop, ~five-interviews-a-day job hunting. If you were happy being suck in a cubical you could easily do that but there is also no chance of you finding that awesome place you do not want to leave at the end of the day. Chief Technology Officer (CTO) was a great title I had at my last company, had a nice office over looking Rockefeller Plaza, and the pay was great… but I was dying inside. The more time that past doing something I did not want to be doing the more I could feel myself growing resentful of a place that did not deserve that. Remember that if you are so unhappy your hair is changing color the job is not worth any amount of money. You can find a cheaper place to live but destroying your relationships because you brought home that stress is no longer only effecting you.

Technology Is As Technology Is

Technology Is As Technology Is

I have been in the job market for a few months now looking for work as a web developer. A certain percentage of companies have decided not to work with me but I have decided not to work with a larger percentage of them. Why turn down a job in this market? To put it bluntly, because most of you people are showing up for a pay check and that is the only reason you show up. If that statement shocks you then perhaps I am directing this post at you. Yes, you.

Take a seat, you need to see a guy about a thing.

Pop quiz: How do you kill a man?
The obvious aside, one sure fire way is to stick him in a situation, virtually every day, virtually all day, where he is unhappy. Some place that creeps into his head and, after waking up one day, he decides he hates. It will probably sneak up on him. Not long after this realize will feed into itself and turn into less and less productivity for your company. This is the state of the “code factories” in most IT departments. All the benefits and job stability in the world is not worth it one byte. I, personally, will take a job way below my previous pay grade at an unstable start up in order to care for my family before I come home and take all that out on them. Although I have nothing against working for larger companies I do not favor them because they tend to be less interested in the people and making good product.

I have discovered a few warning signs that a place is too far gone to be worth any amount of compensation:

  • They have a sprawling HR department. Your interview goes well. Everyone loves you and wants you on their team. Now they need to go through HR. All of a sudden the best person for the job is reduced to numbers. “Sorry but we have decided to move in another direction.” What is a programmer with storage experience worth? What is the minimum a second person is willing to accept just to manage the storage when there is not enough storage-related work to go around? I honestly have little idea but I can not imagine penny wise, pound foolish is the way to go with your critical infrastructure.
  • They think they just need a “frontend” or “backend” person. Maybe you learn about a subject, not because you love it, but because that knowledge will improve the thing you do love. Maybe it is a necessary evil and you can not do without the related technology. No matter the reason you do not write “real” web-based applications in a vacuum; There are other parts you need to consider for a solid product. Sure, you are no DBA but I think you may agree that knowing that MySQL almost never runs your query as you entered it has a big impact on your final decision of how to do something. Did you know that PHP arrays (pretty much the only data type they give you) are not arrays? They are hash tables with syntaxical sugar. That tight loop you are porting from your legacy CGI C executables are going to need to be rewritten from scratch if you are going to reasonably support the same user base you did before the port.
  • Company is hopping on the latest thing for no good reason. I love new technology and techniques. I love to play with them. I love to build real-world products with them that you can put your hands on and get that “wow” response. One problem with the web, however: Anything new is useless for years until it is further adopted. Either that or it is not new at all and you have been taken in by clever salesmen. Obviously if you need to wait there is nothing you can do until most of your audience abandons Windows XP and Internet Explorer 6 (come on, Mom, give it up already). But what about the “new” things? Take MVC, for example. I am a massive fan of MVC, even back when it was called OOP. There is nothing new there. Although I am all for OOP– and thus MVC– having a non-technical manager making a decision based on something they do not understand as well as they think they do is a problem. You hired a programmers to program, let them write the code.
  • Culture can only be forced so far. When I talk about “culture” I am really referring to people getting along with each other, complementing each others skill sets, being willing to admit there are things they do not know, and working towards an end goal as a unified front. People who want to be there are there because… well, because they want to be there. People who joined the company softball team because they are afraid it will reflect badly on them were already unhappy. Happy people work harder. Money is one of those things that is very important when you do not have any of it. We all need to eat, stay out of the cold, and not have our hair turn white when we think of the basic things we can not do for our children. Getting a nice salary bump is a fantastic pick-me-up but it is no long-term solution to your employees ability to be more productive for you. I have no links to back me up on this but, in my experience, the turn around time for companies that think money is the answer to everything is much higher than people who truly enjoy their jobs. Hell, people– such as myself– will accept less money for other perks that cost a company virtually nothing in comparison. I have my own personal preferences but compensation time or surprise days off for a job well done can go a really long way for some ones mental state.

I realize some of (all of?) this comes off as a war cry against The Man™ and makes me sound like a rebel. I suppose in some respects I even am. I am not anti-big company nor am I trying to convince anyone to change anything they are already doing that works; A company is there the make a profit and my job, as a programmer, is to do with what I have and churn out code as fast as I reasonably can for said company. There is a balance there that, in a perfect world I would rally against, but in reality I see it as a necessary evil. These are just my thoughts on the subject as I have seen them over and over recently. Most of the old ways of doing things have lasted this long for good reasons, but so has evolution. Keep in touch with your people, communicate often, and remember the long-term far out weights the short-term nine times out of ten.

Cross Compile Allegro 5 Programs in Linux for Windows

Cross Compile Allegro 5 Programs in Linux for Windows

The Allegro game programming library has released v5 of their popular library and with it comes a whole mess of great changes. Thing is, since most of the applications you make with it are going to be games, your main audience lives in Windows. Since I am really upset with Microsofts offerings in this area I needed a way to capture this audience without having to drive myself insane.

What follows is how I am making Windows executables in Linux using Allegro. Please note that I live in Ubuntu (currently 10.10, Maverick Meerkat). You may have to make some slight changes to fit your distro but that should not be a big deal. These instructions assume a clean installation where no other copy of Allegro has been installed (not sure if that would be a problem or not as I have not tested).

  1. Install the required programs:
    sudo apt-get install cmake mingw32
  2. Retrieve and uncompress DirectX:
    Download and copy the DirectX headers and libraries to /usr/i586-mingw32msvc/. Note the file structure within the archive should compare to the mentioned directory. When prompted to overwrite any files do so but make sure you have a backup first in case something explodes.
  3. Retrieve and uncompress Allegro 5:
    Download allegro-5.x.x.tar.gz from their site. Uncompress some place easy to get to. I used my desktop as we can delete this when done.
  4. Compile from source:
    In a terminal type
    cd [path to uncompressed archive] && mkdir build && cd build && cmake -DCMAKE_TOOLCHAIN_FILE=../cmake/Toolchain-mingw.cmake .. && make && sudo make install
    This may take a little while depending on your hardware.

You should now have a functioning cross compiler setup for Allegro 5. Just replace gcc with i586-mingw32msvc-gcc or g++ with i586-mingw32msvc-g++ (for example, I compiled my first test with i586-mingw32msvc-g++ alleg.cpp -lallegro.dll -lallegro_image.dll -lallegro_font.dll -o alleg.exe). The DLLs you will need are in /usr/i586-mingw32msvc/bin/. You may now delete all of our working files on your desktop (or wherever you put them).

There are still one or two things I need to figure out. For one, dynamically linked programs are peachy on Linux; I am comfortable in my assumption that most people using Linux either already expect this or are willing to learn. Windows, not so much. I want to statically link for that platform but I have yet to experiment with that. Another thing is the fact that my current method has Windows opening up a console window in addition to the “main” window. I am sure this is also very simple but have not yet played with it.

Improving MySQL Index Performance on Bulk Insert

Improving MySQL Index Performance on Bulk Insert

I have been working with a whole mess of data as of late in a series of MyISAM tables. The short of it is I have a 256GB XML file that I want in MySQL. A few tools already exist designed for this specific data. Problem is they import everything and I only want about 75% of the data and they were not as fast as I know they could be. So, being how I have so much free time, I decided to explore this a bit further and wrote my own PHP script to handle the job.

I came the conclusion that the indexes were what was killing me. The database ends up being ~3.5 billion records– yes, it is nearly that much– and indexes are kind of a very important thing. First I tried with the indexes enabled for the whole import. I do not know how long it would have taken because the larger the database grew the slower it got, kind of like how you can never quite reach the speed of light. After that I tried creating the tables without any indexing and using ALTER TABLE after. It looked good at first but when you have such huge tables it gets slower and slower with each added column (IE the first column indexed plenty fast but the second was half, the third half that, ect). The problem was, using ALTER TABLE, you have to add them one-by-one.

After further researching it seems to me that the best way to do this is to add the indexes while creating the table and then just disabling them. Before any of the data is inserted you do a ALTER TABLE table DISABLE KEYS and SET FOREIGN_KEY_CHECKS=0. After you have done your massive, massive import you just do SET FOREIGN_KEY_CHECKS=1 and ALTER TABLE table ENABLE KEYS to re-enable them. This way MySQL does all the work at once and virtually any time you let MySQL handle the work itself– versus multiple calls via external script– you end up with a big benefit.

A few notes:

  • The primary keys are still updated during insertion. I suppose you could use ALTER TABLE after to add those but I am pretty happy with the speed of this setup.
  • I also tried INSERT DELAYED (both alone and in combination with the above) and it resulted in a ~10% increase in time.

Edit 2011.10.30
I just found this article. It seems I was wrong about having to index one column at a time but there are a few caveats. The most notable one is that the order in which you specify the columns has a direct result on if it can be used or not. Any way, this is a huge topic and that is why there are DBAs. Good luck to us all…

Easily Install An Updated Ubuntu Kernel

Easily Install An Updated Ubuntu Kernel

I recently bought a new laptop. Since this laptop is so new and Ubuntu normally ships with kernel versions a few revisions back not all of my laptops features were supported. Bluetooth, for example, is detected but just refuses to turn on. The solution? A newer kernel with newer everything. Sure, I could just update the drivers but this is more fun.

Canonical offers pre-compiled kernels for Ubuntu. All you have to do is go over to and download which ever version you want. Just make sure you grab the headers (all), headers (architecture), and the image (architecture) packages. So if you wanted to upgrade to 2.6.39 you would download linux-headers-2.6.39-02063901_2.6.39-02063901.201106030905_all.deb, linux-headers-2.6.39-02063901-generic_2.6.39-02063901.201106030905_amd64.deb, and linux-image-2.6.39-02063901-generic_2.6.39-02063901.201106030905_amd64.deb. Just make sure you install them in that order and reboot.


Some Tools Remain Unmodified for a Reason

Some Tools Remain Unmodified for a Reason

I was out with friends last night and one of them brought his Kindle with him. He showed it to an inquisitive second friend. This second friend expressed an interest in owning one of his own. I, of course, had to step in.

Some animals have remained virtually unmodified for a long, long time. Why? Because there was no reason to change. They were already doing something that worked and, evidenced by the fact they are still here, had no immediate need for something new. This approach tends to give us everything we need in the long run without over complicating things. For example, paper-based books work great. They are light, easy to use, universally understood without any learning curve, and have remained virtually unchanged in most of human memory.

So why do I fear books moving into the future? Take Amazon for example. They have removed books from users Kindles without notice before. Unless someone enters my apartment or steals from me on the subway once I have the information it is mine; I own it and no one can take it away from me. This Big Brother thing Amazon has going on is not only wrong but only gives the illusion of control. Also while on the topic, what happens in a few years when Amazon has moved on and they no longer support their 10+-year-old tablet? If the service is gone so are "my" books.

The new e-ink displays are very neat. They only require power to display an image, which they only need to do once, and that power usage is impressively low. Still, though, why did I need to stick a battery in my perfectly functioning book in the first place? Sure, an electronic device may be able to stay in stand by for a month or more but my paper-based books can sit on a shelf forever and still work as intended.

Kindle-Man also felt it very important to tell me that he can carry some insane amount of e-books at once. The obvious response to this is that I am only in the habit of reading one book at a time. Perhaps if I am planning a long plane flight I may pack two books. Is this really so much of a benefit that I would drop so much money and risk the above “features?”

I love technology. I make my living on technology and do it in my free time as well. I, however, see the need to keep technology out of certain areas for at least these reasons. Also let us not forget the Battlestar Galactica, people!