I notice a lot of people these days using Heroku to host applications they’ve written with Django or Rails. This may save them some time and effort, but they are spending way more money on this specialized hosting than they would pay for a generic Linux host. They are also giving up control of a large portion of their application. I consider the entire stack to be part of my application, and complete control over all the pieces is mandatory.

Because I encourage people to administer their own servers, I have written this tutorial to remove just one more of their excuses not to do so. This tutorial covers everything you need to do to configure a blank Ubuntu Linux 12.04 LTS 64-bit server to run a Django application properly. I will not be covering anything about writing the application itself. I will also only provide minimal discussion of some topics, like database configuration. If you can finish this tutorial, you can Google the necessary documentation for other things. The only thing I completely skip is configuration of search using Haystack and Solr since it is not needed by most applications.

If you want to use a different Linux distribution, most of this tutorial will still be correct. You will have to translate all of the apt-get statements and package names to your distribution of choice. Configuration file paths and layouts may also differ. I chose Ubuntu 12.04 because it is very popular, available on almost every host, and its long term support will keep this tutorial relevant for a longer period of time.

Formatting

In this tutorial all commands and lines of code that you should actually enter are displayed in preformatted text. Within the preformatted text there will be lines that begin with an octothorpe (#). Those lines contain my personal comments on the surrounding lines, and should not be typed. Lines beginning with the dollar symbol ($) are commands that should be typed on the shell. In cases where there are variables that you should replace with appropriate values, I will surround the variable with <> symbols and name it with all capital letters.

# This text is describing the command below it. Don't type it.
$ this is a shell command, type it

# Type the following command, but replace the <USER> with your username
$ sudo adduser <USER>

In cases where edits must be made to a file I will begin that section with a shell command to open the appropriate file in vim, which is my text editor of choice. You may use whichever text editor you wish. I will also describe the edits to be made in the comment text.

$ vim test.txt

# add this line to the end of the file
TEST = True

Tutorial

Beginning Steps

Begin by installing Ubuntu on a machine you own, or getting a new machine from a hosting provider. I recommend Linode or Amazon EC2. If you choose EC2 the Ubuntu AMI finder should be your starting point.

Once the machine is installed, you will need to login, mostly likely using SSH. I can not more strongly recommend you use SSH Agent Forwarding. I will not cover how to set it up in this tutorial, as it varies based on your client machine. If you haven’t done this already, do it now.

Once you are logged into the machine, you will probably notice it is telling you that a lot of packages need updating. You should update packages whenever they need updating, especially if there are security patches. Just be aware this may disrupt your site for a few minutes. If your site were important enough to where a minute of downtime was a problem, you would not need this tutorial. Here is how you update all packages on an Ubuntu or Debian system.

# update the local package index
$ sudo apt-get update

# actually upgrade all packages that can be upgraded
$ sudo apt-get dist-upgrade

# remove any packages that are no longer needed
$ sudo apt-get autoremove

# reboot the machine, which is only necessary for some updates
$ sudo reboot

You may notice frequent use of sudo. Sudo is a program that allows us to execute other commands with root privileges. If you are bothered having to type your password when using sudo, then you can edit your sudoers file to make it easier in the future.

distribute and pip

Many of the Python packages we need are available through Ubuntu and apt-get. However, we do not want to use them. The packages provided by Ubuntu are very likely to be old and out of date. Also, it is only possible to install them globally, whereas we would like to install them in an isolated fashion. If we ran two sites on this server that used different versions of Django, this method would not work.

Many of the Python packages are wrappers around C libraries that will need to be available for linking. Those C libraries must be installed using apt-get before we install the associated Python packages. There are two such packages that are needed for just about everything, and should be installed now. They are build-essential and python-dev. Build-essential is the Ubuntu package which provides all the standard C compilation tools. Python-dev provides the necessary files for compiling Python/C modules.

$ sudo apt-get install build-essential python-dev

Now, there are various tools to install Python packages, and there is much drama and debate in the community about them. I will ignore all that, and say to use Distribute and pip. I have used them for years now, and have had no issues at all.

# download distribute
$ curl -O http://python-distribute.org/distribute_setup.py

# install distribute
$ sudo python distribute_setup.py

# remove installation files
$ rm distribute*

# use distribute to install pip
$ sudo easy_install pip

Virtualenv(wrapper)

Now that we have used distribute to install pip, we will use pip to install every other Python package. This solves the problem of getting packages, but it does not solve the problem of isolating them. For that we must use virtualenv. Because virtualenv itself is a little clumsy, we will also use virtualenvwrapper to make things a little easier on ourselves.

# install virtualenv and virtualenvwrapper
$ sudo pip install virtualenv virtualenvwrapper

# edit the .bashrc file
$ vim .bashrc

# to enable virtualenvwrapper add this line to the end of the file
source /usr/local/bin/virtualenvwrapper.sh
#save and quit your editor

# exit and log back in to restart your shell
$ exit

You will notice after exiting and logging back into the shell that virtualenvwrapper will do a bunch of magic. That is a one-time setup that you will never see again. If you don’t see a bunch of stuff happen, then there is a problem you should fix.

Now, how do you use virtualenv? You create as many virtualenvs as you like, each with a unique name. Only one of these virtualenvs will be active at a given time. Any time you use pip the packages will only be installed into the active virtualenv. In order for a Python program to make use of those Python packages, it must be executed with that virtualenv activated. Let’s create a virtualenv for our project and activate it. You should just keep it active for the rest of this tutorial for ease. You can tell when a virtualenv is active because it will appear in your shell prompt.

# create a virtualenv, I usually give it the same name as my app
$ mkvirtualenv <VIRTUALENV_NAME>

# The virtualenv will be activated automatically.
# You can deactivate it like this
$ deactivate

# to activate a virtualenv, or change which one is active, do this
$ workon <VIRTUALENV_NAME>

One very helpful command you can use now is pip freeze. Pip freeze lists all available Python packages and their versions. It will let you easily see what has been installed. Try these commands out. You will see just how many packages have already been installed system-wide by Ubuntu. You will also see that your virtualenv is isolated from the rest of the system, and doesn’t have much installed in it yet.

$ deactivate
$ pip freeze
$ workon <VIRTUALENV_NAME>
$ pip freeze

Django

Installing Django is very simple. Make sure the virtualenv is activated. This is the last time I will remind you to activate.

# install the newest version of Django
$ pip install django

# also install docutils, which is used by the django admin
$ pip install docutils

# If you want to test django, start a new project and run it
$ django-admin.py startproject <APP_NAME>
$ cd <APP_NAME>
# make manage.py executable
$ chmod +x manage.py
# start the dev server
$ ./manage.py runserver 0.0.0.0:8000

PIL / Pillow

PIL is the Python imaging library. It is needed if your application is going to do anything involving at all involving images. Even if you simply allow user to upload images, you will need it. There is a drop in replacement available called Pillow. I’ve never had problems with either one, but some people have problems with PIL. If you don’t know, just choose Pillow. Whichever one you choose, you will need the same set of prerequisite C libraries. If you don’t install these, then PIL might fail if a user tries to upload a jpg, or you try to perform certain image operations.

# install libraries
$ sudo apt-get install libjpeg8-dev libfreetype6-dev zlib1g-dev

# choose ONE of these
$ pip install pillow
$ pip install pil

Databases MySQL / PostgreSQL

It’s up to you to choose your database. I prefer MySQL because it is easier. The Django devs prefer PostgreSQL because it is more robust. You can learn to configure the databases using their respective documentation. It’s also up to you to edit the Django settings.py file to point at the appropriate database server. If you choose MySQL, you should configure it so that utf8 is always the default character set, utf8_general_ci is the default collation, and that InnoDB is the default storage engine.

# do this to go with Postgres
$ sudo apt-get install postgresql libpq-dev
$ pip install psycopg2

# do this to go with MySQL
$ sudo apt-get install mysql-server libmysqlclient-dev
$ pip install MySQL-Python

Whichever database you choose, you will have to create a database and a user for your application to use. Then you will have to make sure that the user has full permissions on that database.

South

South is a mandatory part of every Django project as far as I am concerned. It manages schema changes to your database about as well as can be expected, and makes life much easier in general.

$ pip install south

# add south to the INSTALLED_APPS in your settings.py
$ vim <YOUR_APP>/settings.py
INSTALLED_APPS = 
    ...
    'south',
    ...

memcached

Just about every web site can benefit from having memcached. It creates a key:value store directly in memory that is very fast and easy. This can be used to store information to reduce database queries and increase performance. There have been many different libraries and modules for memcached, but PyLibMC is presently the preferred choice.

# install memcached server and library
$ sudo apt-get install memcached libmemcached-dev

# install pylibmc
$ pip install pylibmc

# edit the CACHES setting in your project's settings.py
$ vim <YOUR_APP>/settings.py
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
        'LOCATION': '127.0.0.1:11211',
    }
}

Asynchronous Task Execution

Almost every site can benefit from asynchronous task execution. Even if your site isn’t big enough to make it a necessity for scaling and performance, why not do it? It’s very easy to set it up, and requires only free software.

What is asynchronous task execution? Let’s say you have a simple contact form. If you call send_email from the contact form view, then the user who submitted the form will not receive an HTTP response until the email has been sent. Their browser will sit there spinning while the server works. It will also tie up one of your web server threads. To make matters worse, if there is an email error, it will return that error to the user.

What you really want to do is send the user to a thank you page immediately after they submit. Then you can send the email some time later without tying up a web server. If there is an error, you can keep retrying it without bothering the user. To do this we use a Python package called celery.

Celery requires a few pieces to work. First, it needs celery workers. These are the programs that actually do the work, such as sending the emails. Next it needs a message queuing server. This is where the celery workers look to see if there is any work they should be doing. For this you should use RabbitMQ.

There is also celerycam, which will monitor celery by taking snapshots every few seconds. This will allow you to see the state of what celery is doing from the Django admin interface. Lastly, we need a place to store results if any celery tasks produce them. For that we will just use the existing MySQL or PostgreSQL database.

RabbitMQ

When you install RabbitMQ you also have to create a user and grant them permissions on a virtual host. You can pick whatever username, password, and vhost name you like, but I always use the name of my app for all three.

$ sudo apt-get install rabbitmq-server
$ sudo rabbitmqctl add_user <RABBIT_USER> <RABBIT_PASSWORD>
$ sudo rabbitmqctl add_vhost <RABBIT_VHOST>
$ sudo rabbitmqctl set_permissions -p <RABBIT_VHOST> <RABBIT_USER> ".*" ".*" ".*"

Celery

Installing celery itself is a one liner, but it requires a lot of modifications to the Django settings file to get it working properly.

$ pip install django-celery

$ vim <YOUR_APP>/settings.py

# add djcelery to INSTALLED_APPS
INSTALLED_APPS = 
    ...
    'djcelery',
    ...

# add these settings
BROKER_URL = "amqp://<RABBIT_USER>:<RABBIT_PASSWORD>@localhost:5672/<RABBIT_VHOST>"
CELERY_RESULT_BACKEND = "database"

# choose the setting that matches your database of choice
CELERY_RESULT_DBURI = "mysql://<DB_USER>:<DB_PASSWORD>@localhost/<DB_NAME>"
CELERY_RESULT_DBURI = "postgresql://<DB_USER>:<DB_PASSWORD>@localhost/<DB_NAME>"

# put these two lines at the very bottom of the settings file
import djcelery
djcelery.setup_loader()

To test celery you can start up the celery daemon using a management command. If your application is going to have recurring tasks, you should enable events and celerybeat. Otherwise, you can ignore those options.

# start celery with Beat and Event
$ ./manage.py celeryd -B -E

# start celery normally
$ ./manage.py celeryd

# press Ctrl+C to quit celery once you see it is working

gunicorn

When you want to serve a modern Python web application, you use a thing called the web server gateway interface (WSGI). That is how your application will talk to the web server. There are a lot of choices when it comes to using WSGI. Apache with mod_wsgi is the old trustworthy option. uWSGI is also very popular. I personally prefer Gunicorn because it works extremely well, and runs great out of the box without any configuration.

$ pip install gunicorn

# add gunicorn to INSTALLED_APPS
$ vim <YOUR_APP>/settings.py
INSTALLED_APPS =
    ...
    'gunicorn',
    ...

Like celery, you can test Gunicorn by running it from a management command. The Gunicorn management command has many command line parameters. You can read about the options in the Gunicorn documentation. Personally, the only options I use are to set Gunicorn to use four workers and to enable gevent. You will see more about this in the next section.

$ ./manage.py run_gunicorn -w 4 -k gevent

# pres Ctrl+C to exit Gunicorn once you see it is working

Supervisor

You may have noticed by now that Ubuntu is very clever in the way it handles servers. When you install MySQL with apt-get it immediately starts it with a default configuration. The same goes for RabbitMQ, memcached, SSH, and almost every other such server. If the machine reboots, these services will start themselves automatically. We can trust the distribution to take care of this for us.

But we have a problem now. We need to start Gunicorn and celery as services, but Ubuntu won’t help us with things we didn’t install with apt-get. One way to solve this problem is to write upstart or init.d scripts for these services. That will work, but it’s a pain in the ass. Those kinds of scripts are not simple to write, and are worse to maintain. Luckily supervisor exists.

Supervisor is installed with apt-get, so it will start automatically. We give supervisor very simple configuration files for any further services we would like it to manage, and it will start those up for us. It has the added benefit that it can restart any service it is managing in the case of failure. If Gunicorn crashes, it will come right back without us having to do anything. Because of this feature, some people choose to manage all their services under supervisor. I do not do this because it is extra work, and it is very rare that it will help you. If something like MySQL crashes, odds are it will not be successfully or safely restarted by supervisor.

Installing supervisor is as simple as apt-get, but configuring it is another matter. We need to create a configuration file for each service that supervisor manages. These configuration files must be located in /etc/supervisor/conf.d/ and must have a name with the conf extension. I have included here the full contents of my three supervisor configuration files, but have only documented the first one, as they are so similar.

$ sudo apt-get install supervisor

# contents of /etc/supervisor/conf.d/celeryd.conf

# the name of this service as far as supervisor is concerned
[program:celeryd]

# the command to start celery
command = /home/<USERNAME>/.virtualenvs/<VIRTUALENV_NAME>/bin/python /home/<USERNAME>/<APP_NAME>/manage.py celeryd -B -E

# the directory to be in while running this
directory = /home/<USERNAME>/<APP_NAME>

# the user to run this service as
user = <USERNAME>

# start this at boot, and restart it if it fails
autostart = true
autorestart = true

# take stdout and stderr of celery and write to these log files
stdout_logfile = /var/log/supervisor/celeryd.log
stderr_logfile = /var/log/supervisor/celeryd_err.log

# contents of /etc/supervisor/conf.d/celerycam.conf
[program:celerycam]
command = /home/<USERNAME>/.virtualenvs/<VIRTUALENV_NAME>/bin/python /home/<USERNAME>/<APP_NAME>/manage.py celerycam
directory = /home/<USERNAME>/<APP_NAME>
user = <USERNAME>
autostart = true
autorestart = true
stdout_logfile = /var/log/supervisor/celerycam.log
stderr_logfile = /var/log/supervisor/celerycam_err.log

# contents of /etc/supervisor/conf.d/gunicorn.conf
[program:gunicorn]
command = /home/<USERNAME>/.virtualenvs/<VIRTUALENV_NAME>/bin/python /home/<USERNAME>/<APP_NAME>/manage.py run_gunicorn -w 4 -k gevent
directory = /home/<USERNAME>/<APP_NAME>
user = <USERNAME>
autostart = true
autorestart = true
stdout_logfile = /var/log/supervisor/gunicorn.log
stderr_logfile = /var/log/supervisor/gunicorn_err.log

You will notice the command for each of the supervisor configurations is strange. Instead of just running manage.py directly, we are specifically calling the python interpreter located inside of our virtualenv. There is no way for supervisor to easily activate itself the way we activate in our shell with the workon command. By specifically using the python interpreter from the virtualenv instead of the system version, celery and gunicorn will have the correct path settings, and will use the packages within the the virtualenv. Also, note the celeryd and gunicorn command line parameters we had discussed earlier to enable beat, event, gevent, etc.

To make supervisor recognize the new configuration files, we must restart it. It’s also pretty obvious from the example blow how to manually control the services that supervisor manages.

# restart supervisor itself
$ sudo service supervisor restart

# restart/stop/start all services managed by supervisor
$ sudo supervisorctl restart all
$ sudo supervisorctl stop all
$ sudo supervisorctl start all

# restart just celeryd
$ sudo supervisorctl restart celeryd

# start just gunicorn
$ sudo supervisorctl start gunicorn

Static and Media

There is a great deal of confusion about static and media in Django. It is partially because the separation of the two was not very clear in earlier releases. This has since been fixed, and the static files system has been completely reworked. While the problem is solved in  the current versions, the big changes may have added to the confusion. Allow me to help.

Static files are files that you would put into your repository that are not code. These are almost always css, jss, and image files. Your favicon would also be a static file. This is a file that you make, must be served to web browsers, but does not change unless you update your application.

Media files are files that are also served statically, but they were not made by you. They were uploaded by users. So on a site like YouTube, the YouTube logo in the top left would be a static file, but the actual videos themselves would be media. On Flickr, the css and the login buttons would be static, but the photos uploaded by users are media.

For media files we must simply take files that users upload, put them in a folder, and then serve the files in that folder. For static files we have a slightly trickier issue because they are not so easily located. The django admin keeps its static files in one place. You put your static files in another place. Maybe you pip install another third party module that includes some static files. In this case, you run a management command to collect the static files to a single folder from which they will be served.

# Create directories to hold static and media
sudo mkdir /var/www
sudo mkdir /var/www/static
sudo mkdir /var/www/media

# set permissions on these directories
sudo chown -R <USERNAME>:www-data /var/www

$ vim <YOUR_APP>/settings.py

# these settings are explained below
MEDIA_ROOT = '/var/www/media/'
MEDIA_URL = '/media/'
STATIC_ROOT = '/var/www/static/'
STATIC_URL = '/static/'

# run this whenever static files are changed
$ ./manage.py collectstatic

In the django settings file there are four settings. MEDIA_ROOT, MEDIA_URL, STATIC_ROOT, and STATIC_URL. The ROOT settings are the local directories where the files are located. The URL settings are the URLs where these files will be served. Assume we have a file named css/main.css Using the above example, the file will be located in /var/www/static/css/main.css after it is collected. It will be served from the URL http://yourdomain.com/static/css/main.css.

One more benefit of the static system is that you can use pip to add static files to your project. For example, you can pip install django-staticfiles-jquery and django-staticfiles-bootstrap to automatically put jquery and boostrap into your static files when you run collectstatic.

Nginx

This brings us to the final piece of the puzzle. How exactly are these static files served? Gunicorn handles serving all the dynamically generated pages of our application, but it doesn’t know anything about these static files. The Django runserver will do some magic to handle these files to make development easier, but once you enter production and set DEBUG=False, you no longer have this luxury. That is why we put Nginx as our web server out in front of Gunicorn. It will serve all the static files, and also ask Gunicorn to handle any requests it doesn’t know what to do with.

# install nginx
$ sudo apt-get install nginx

Much like Apache, Nginx uses two folders for configuration. There is /etc/nginx/sites-available/ and /etc/nginx/sites-enabled/. You put all your nginx configuration files into the sites-available directory. Then if you want a site to be active, you create a symlink to that file from the sites-enabled directory. You have to restart nginx after any configuration changes. Because Ubuntu starts nginx with a default configuration when you install it, you must first remove the symlink to that default configuration.

# remove the default symbolic link
$ sudo rm /etc/nginx/sites-enabled/default

# create a new blank config, and make a symlink to it
$ sudo touch /etc/nginx/sites-available/<YOUR_APP>
$ cd /etc/nginx/sites-enabled
$ sudo ln -s ../sites-available/<YOUR_APP>

# edit the nginx configuration file
$ vim /etc/nginx/sites-available/<YOUR_APP>

Here is what the nginx configuration file should look like

# define an upstream server named gunicorn on localhost port 8000
upstream gunicorn {
    server localhost:8000;
}

# make an nginx server
server {
    # listen on port 80
    listen 80;

    # for requests to these domains
    server_name <YOUR_DOMAIN>.com www.<YOUR_DOMAIN>.com;

    # look in this directory for files to serve
    root /var/www/;

    # keep logs in these files
    access_log /var/log/nginx/<YOUR_APP>.access.log;
    error_log /var/log/nginx/<YOUR_APP>.error.log;

    # You need this to allow users to upload large files
    # See http://wiki.nginx.org/HttpCoreModule#client_max_body_size
    # I'm not sure where it goes, so I put it in twice. It works.
    client_max_body_size 0;

    # THIS IS THE IMPORTANT LINE
    # this tries to serve a static file at the requested url
    # if no static file is found, it passes the url to gunicorn
    try_files $uri @gunicorn;

    # define rules for gunicorn
    location @gunicorn {
        # repeated just in case
        client_max_body_size 0;

        # proxy to the gunicorn upstream defined above
        proxy_pass http://gunicorn;

        # makes sure the URLs don't actually say http://gunicorn 
        proxy_redirect off;

        # If gunicorn takes > 5 minutes to respond, give up
        # Feel free to change the time on this
        proxy_read_timeout 5m;

        # make sure these HTTP headers are set properly
        proxy_set_header Host            $host;
        proxy_set_header X-Real-IP       $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Restart nginx and your server is now running and serving your django application perfectly!

# restart nginx
$ sudo service nginx restart

Conclusion

I’m sure there are going to be errors in this no matter how much I proofread. I also fully admit that there is always more to learn. I welcome any suggested fixes or improvements to any part of this. Just leave a comment below.

I also hope this saves someone some money by letting them run their site on a cheap Linux host instead of a managed host. Most of all I hope it gets some people to view their application as not just the one piece they wrote, but as a whole made up of many separate working parts. And that those other parts are not to be outsourced, but to be embraced. When you control them, you can solve problems with them.

At this day camp we played a lot of four square. The thing was, we played it inside a cabin. The cabin was square. There was no furniture. There were built-in benches all around the inner edge. It had a roof to block the rain. The square was already painted very clearly on the wooden floor by campers of years past. It was an arena that beckoned for the red Voit.

However, because there were walls in this cabin all sorts of crazy situations developed. Is it in or out if it bounces off the benches or the ceiling? What if it goes out of the cabin? Are double taps and other cheap maneuvers permitted? What if someone sitting on a bench interferes?

In all the time of this summer camp, nobody had bothered to actually solve these problems. Nobody took games are seriously as I did, and still do. The solution in place was simply that whoever was in the ace position had the serve and dictated the rules. The game changed rules every time the serve changed hands. The basic idea was that when you earned the serve you would choose rules favorable to yourself, so that you would keep the serve for as long as possible. It was the simple children’s logic of winner’s ball.

The other kids didn’t want a fair game of skill. They didn’t want to have to earn victory. They wanted victory handed to them on a silver platter. To them it had been a game of using trickery, cheap maneuvers, and hax. I, on the other hand, had always played honestly. I felt righteous, even in defeat. I wouldn’t use a bullshit double-tap or spike, even when it was permitted. I told myself I only lost because they were all cheap and/or cheating. I was always quickly eliminated.

One rainy day most activities were canceled, and we played four square for many hours. Perhaps for the first time, I became the ace. I had begrudgingly played by their rules until now, but things were about to change. In about five minutes I rewrote the rules of the game to be not completely broken and unfair. I laid down the law on all sorts of exceptional situations that had occurred, and could theoretically occur. It was a four square revolution.

Now you might think my new regime would have pissed everyone off. They would have moaned and groaned because of the change. They would have called me names and not obeyed my rulings. I even expected it at the time. I was happy to just play a fair game, even if it were to last only a single round.

After I laid down the law I actually won a few rounds as the other kids adjusted out of habits that were now illegal. Of course I was eventually eliminated from the square, and the king moved to the ace position. It was time for a new ruler to lay down the law.

Two words did he then utter.

“Scott’s Rules”

You may or may not have to play the game to win the game. But once you are in charge, you can dismantle the game. Obviously the dictatorship of four square is relatively simple, but it is often frightening how often schoolyard politics resemble world politics. Only those with the power in the system can fix the system. If those in power truly hated the state of the game, they could change it. The fact that politicians not only play the crooked game, but keep the game crooked, means that they prefer the crooked game. That is an undeniable fact. They may say they hate the system, but if they do not fix it, then they support it.

Remember, if they are in power, that means they have won the game of politics at least once, and probably many times. Why would they change the rules of this unfair and corrupt game when they themselves are already so good at it? Fixing the system likely means removing themselves from power. Anyone like myself who would change the game can not possibly get into a position to do so. Our refusal to cross ethical boundaries makes it next to impossible to attain a position of power.

Some politicians are more corrupt than others. Some are clearly more preferable than others. Yet, any one that plays the game of politics, and does not at least attempt to fix it or even speak out against it, is, in my eyes, a corrupt piece of shit. The level of stink may vary, but shit is shit.

Now, my rules didn’t last forever. They were not written down, and I only taught them once. Some rules were forgotten or misinterpreted. Often kids would reach the ace and say “Scott’s Rules plus spike on serve,” or some other nonsense. They knew my rules made the game more fair and more fun, but they could not rid themselves of corruption when in a position of power. They wanted to win, and had no qualms about modifying the rules to their advantage. It wasn’t enough to fix the system once. It had to be constantly maintained lest it fall back into disrepair.

Look, ye mighty.

In the days before the Internet, national borders were solid walls. If a movie was released in country A, and you lived in country B, it was almost impossible to see it. Maybe you could see it during a vacation in country A. Your only realistic hope was for someone to physically import the film. Even with VHS that would only help you if you spoke the language of the other country, or someone with equipment and time generously subtitled it. Without that, you simply had to wait and hope some local company would license and redistribute the movie in your home country. Any work that managed to achieve international distribution was nothing short of a miracle.

It wasn’t all bad. Miracles do happen. The most popular of things did get released internationally. Beatles albums, for example, were available in many countries.  Disappointingly, they couldn’t leave well enough alone. The UK release of Rubber Soul had different tracks from the North American release. The same was true for almost every international release across all media in those days. Completely unchanged releases across regions were the true miracles.

Why would they do this? The producers had some ideas, that may or may not be true, about what would appeal to different cultures in different countries. If you have enough money to produce different versions of your art for different audiences, and your ideas about cultural preferences are correct, why wouldn’t you segregate to sell more copies? When playing a live concert, doesn’t a musician play different songs in different cities? Why wouldn’t your album have different tracks in different countries? As long as every country gets every song, I really don’t see a problem with having them reconfigured onto different discs in different orders.

And there wasn’t much of a problem. The vast majority of fans in North America had no idea that the tracks were different in the UK, and vice versa. They got away with it so well for so long, that they still continue this behavior today. It is now a humongous problem. How can you possibly have the exact same business strategy as fifty years ago, and not even realize there’s a problem?

Today you absolutely can not get away with these kinds of edits. Everyone who cares will find out immediately. When video games and comic books are censored for different markets, articles on the web instantly crop up listing in excruciating detail every single difference between the versions. When video games are released with different box art, or books have different cover art, everyone ends up seeing every version. Even board games are not immune to this. It is impossible to keep the secret, but they still try.

Not only is it impossible to keep the secret, but heaven help you if one country gets more than another. The DVD from one country has bonus features not released in the other? It will take maybe two seconds for people to flood the post office and start exporting the superior version.

Even worse is when companies try to set different prices in different countries. A movie on DVD in Japan might cost $40+ while the same DVD in the US is $15 at most. Even with the cost of shipping, importing DVDs from the US is definitely a money saving move. When international shipping is a cost savings, something is extremely wrong.

I’ve also seen cases where simply knowing that their country is being overcharged causes people to stop buying. In Australia video game prices are famously much higher than in other countries. Some people import games, but many just don’t buy them at all. Even if they could afford to buy them, they don’t. They know the international prices. They know they are being overcharged, so they buy fewer games.

There have been a few attempts to technologically prevent importing and exporting of media. All of them are futile. DVD region codes have been useless for a very long time. Even now with the Nintendo 3DS being region locked a hack will definitely appear, if it hasn’t already. If it doesn’t, people will import the console itself. Creating this region locking technology must cost companies time and money. They must be better off not fighting such a futile battle. Heck, you regularly see entire arcade machines that are technically not supposed to cross international borders. If such big machines can make the trip, there’s no stopping digital copies of anything.

The most hilarious is when you go on Spotify and it lists the top tracks. Any songs not available in your country are listed, but grayed out. What do they think happens? Everyone immediately will go to The Pirate Bay and type in the names of these artists. Why do they list them in gray when they could just remove them from the list entirely? My guess is that it is an intentional design decision by Spotify. They probably don’t agree with the region locking and are forced into it by the record companies. They know that you are going to pirate those songs, so they keep them in the list to help those artists create an international fanbase.

Here is one particularly egregious example concerning My Little Pony: Friendship is Magic. Clarification From Hasbro About Regional DVD Sales and Episode Releases.  You have a product that has become internationally popular because it was illegally, and inevitably, distributed worldwide on the Internet. Now you completely fail to have it legally available in any form. Your customers who want to give you money are left with almost no choice but to pirate it.

Could they wait for the legal release? Yes, but there’s no guarantee if or when it will happen. There’s also no guarantee it won’t be modified and ruined by poor translation or censorship. If someone is forced to wait, there are so many other entertainment options they are guaranteed to look elsewhere. Most likely they will forget about you, and now you’ve really lost a sale. At least if they pirate they might become a huge fan and remember you when you finally get around to a legal release, or buy your other merchandise.

Now famous is the story of how Valve software reduced game piracy in Russia. Russia is a country where piracy rates are through the roof. Lots of people don’t even bother releasing their products there because piracy is so widespread they will make almost no sales. Then how come Valve made sales? Did they use magic? No, they simply released the exact same product in Russia at the exact same time they released in every other country on Earth, and let them pay in rubles. A true global release. And of course, it works.

Russia is a classic chicken and egg situation. Not enough things being available lead to pirating. Pirating lead to less things being available. That lead to more pirating. The only way to break the cycle is to make things legally available. Valve broke the cycle. Can the bone-headed old world companies ever do the same? They haven’t yet, and I doubt they ever will.

In this day and age, even time zones matter. If you have a midnight release, guess what? It’s midnight on the East Coast of the USA three hours before midnight on the West Coast. Your product can be uploaded and transferred to the entire world before the Central Time Zone release, let alone the Pacific. If it’s a midnight release in the East, then it’s a 10PM release in the West. You have no choice in this matter. The people of California aren’t going to wait three hours. You had better open your doors before they close their wallets.

What I find most hilarious is when web sites try to restrict themselves to one geographic region. Welcome to new social network X, now available in Canada. Uh, dude. It’s the world wide web. When you put something on the web, it’s available world wide. You can’t keep anyone out. Look at how hard Hulu, Netflix, and BBC fail to keep people in different countries from getting access, and it never works. Lots of online games in South Korea try unsuccessfully to keep out players from other countries. Why do they even try?

There is always an issue of translation cost. If there is any language in your product, you must localize it for each market. Each localization costs money. And what if a Swahili translation is a big economic risk considering how few sales you are likely to get? What can you do about that?

Don’t do anything. Don’t do the Swahili translation. But you should still release it in Swahili speaking nations, and include every localization that you have available. If someone buys it and does a fan translation, now you have discovered, or even created, some fans. Maybe by the time the sequel is done the risk of translating to Swahili will be much lower. Maybe you will know for sure that it’s not worth the cost to translate into Swahili, but hey, you sold three extra copies in English and two in French. Five is more than zero, and it cost you nothing extra to make it available to them.

Another thing I’ve noticed is companies that feel an absolute necessity to market something. They can’t release a product unless they market the hell out of it, and they really care about the timing of the marketing coinciding with the release.

First of all, digital marketing is also international. If you advertise on the web, you’ve already advertised to the world. A big part of the problem is when companies unintentionally create demand in countries they aren’t serving. If I make you want something very badly, and then don’t offer it to you legally, of course you are going to pirate it if that is your only available option.

Also, there is nothing wrong with making something available without marketing it. Ok, you only made two sales in Argentina because you didn’t market there. Who cares? You made two sales! It cost you nothing to make it available to them. That’s only positive.

Lastly, the timing means nothing. Even Apple seems to get this one wrong. You released in Argentina a year ago. Now you decide you want to start marketing there. The people will not care that your product was released so long ago. It’s not food. It doesn’t spoil. Your advertising will be no less effective. In fact, some of the biggest advertisers are food companies like McDonald’s, Pepsi, and Coca-Cola. Their advertising still works on products that were released decades ago. Advertising your one year old product will be just fine.

Here’s what it all comes down to. In terms of selling any product that can take a digital form, every release is a worldwide simultaneous release, whether you like it or not. If you fail to treat it as such, you will simply be hurting yourself and nobody else. Whatever it is you are selling, you need to make the exact some product available in every single country on Earth at the exact same second at the exact same price, no exceptions, period. If the product is digital, it costs nothing extra whatsoever to simply make it available. Do it.

Trying to remove the world from the world wide web is impossible. It’s simultaneously baffling and hilarious to see so many people try so hard to achieve this impossible goal. There has to be more than one company on this planet who is willing to end this charade.

Definitions

Before we begin there are some definitions I have to set down.

Facebook Account

The actual Facebook account you use to talk to friends and family.

Facebook Page

A separate page you have setup for your brand or online community.

Facebook Application

An application you create which is basically just an set of IDs you need to connect to the Facebook API.

Setup

Before we begin you have to create your facebook account, page and application. I’ll assume you already have an account, or know how to create one. That’s clearly outside the scope of this tutorial. Make sure you remain logged into your Facebook account throughout this entire process.

If you don’t already have one, you need to create a Facebook page. Go to https://www.facebook.com/pages/create.php, and follow the instructions. When you are all done you will end up looking at your new page. It will have a URL that will have the following format.

https://www.facebook.com/pages/PAGE_NAME/PAGE_ID_NUMBER

Save the PAGE_NAME and PAGE_ID_NUMBER. If you already have a page, visit it and look at the URL for the name and/or id number. Make sure that your Facebook account has permission to post status updates on that page. I’ve only tested this on pages where my account is the page administrator. Your mileage may vary otherwise, but I see no reason it wouldn’t work as long as you have the necessary permissions.

The next step is to create a Facebook application. To do that go to https://www.facebook.com/developers/createapp.php and follow the instructions. You will go through a configuration process. You can leave all settings as the defaults. Just save the Application ID and Application Secret, and you will be good to go.

Get the Access Tokens Through OAuth

Now it’s time for the tricky parts. You have to give the application permission to access your account. Take the URL below and replace both instances of APP_ID with the ID of the application you just made.

https://www.facebook.com/dialog/oauth?client_id=APP_ID&
redirect_uri=http://www.facebook.com/appcenter/APP_ID&scope=manage_pages,offline_access,publish_stream

Visit this URL in your browser and you will get a dialog box that will ask you whether to allow or deny granting permissions on your account to the application. The page needs to redirect somewhere after you click allow, and it will only allow you to redirect to a page your application owns. That’s why we put the URL of your application’s profile page as the redirect_uri.

The three permissions we are granting are manage_pages, offline_access, and publish_stream. Manage_pages grants the application access to pages which your account has access to. Offline_access means that the application will get an access token that never expires so we never have to do this complicated process ever again. Publish_stream allows the application to post new updates to your account, or any other place your account is able to post updates to, including the page we just created.

Click the allow button. Immediately go to the location bar of your browser and copy and paste the URL to a text editor. Do not lose it. You will see that the URL looks something like this.

https://www.facebook.com/apps/application.php?id=APP_ID
&code=CRAZY_LONG_CODE

That crazy long code is the magic we need. Take that code and use it to create yet another URL in the format that follows. Replace APP_ID with the Application ID as usual. Replace APP_SECRET with the Application Secret. Lastly, put that CRAZY_LONG_CODE where it belongs. Paste the resulting gigantic URL into the location bar of your browser and visit it.

https://graph.facebook.com/oauth/access_token?client_id=APP_ID
&redirect_uri=https://www.facebook.com/apps/application.php?
id=APP_ID&client_secret=APP_SECRET&code=CRAZY_LONG_CODE

If you did everything right you should now be looking at a big beautiful access_token. We will call this the account access token. It gives your facebook application permission to access your account. If you just want to post status updates to your personal account, you can take this access token to the next section. Otherwise, proceed to get the page access token.

In order to get the page access token, we need to visit yet another URL, it looks like this.

https://graph.facebook.com/USER_ID/accounts/?
access_token=ACCOUNT_ACCESS_TOKEN

The USER_ID is the user id of your Facebook account. Mine is apreche because my Facebook profile can be found at https://www.facebook.com/apreche. You might not have a profile URL, in which case you can use a numerical ID instead. If your personal profile is at a URL such as https://www.facebook.com/profile.php?id=NUMBERS then use the NUMBERS as your USER_ID. Take the account access token we got from the previous step and put it in place of ACCOUNT_ACCESS_TOKEN.

Now visit this URL in your browser. You are going to see some slightly complicated text that is in a format known as JSON. It should look something like this.

{
   "data": [
      {
         "name": "PAGE_NAME",
         "access_token": "PAGE_ACCESS_TOKEN",
         "category": "Website",
         "id": "PAGE_ID"
      },
      {
         "name": "APPLICATION_NAME",
         "access_token": "APPLICATION_ACCESS_TOKEN",
         "category": "Application",
         "id": "APPLICATION_ID"
      }
   ]
}

Depending on how many different pages and applications you have on your Facebook account, you may see more than this. Regardless, you are only interested in one thing. Find the section with the PAGE_NAME of the page you want to post to and extract the PAGE_ACCESS_TOKEN from that section. This is the token we have been looking for. Save it and keep it safe.

Making the Post

Now that we have the page access token, we can finally do the actual status update. I will use curl in my examples since that is language agnostic. All you really need to do is make an HTTP POST. In Python you could use urllib. You could also use an appropriate Facebook library for your language, such as the Facebook PHP SDK. It’s up to you to learn how to get this done in your programming language of choice.

Here is what a complete POST looks like using curl in the shell. Remember to put the PAGE_NAME and PAGE_ACCESS_TOKEN where they belong.

$ curl -F access_token="PAGE_ACCESS_TOKEN" \
-F message="testing" \
-F picture="http://i.imgur.com/2uLkT.jpg" \
-F link="http://frontrowcrew.com" \
-F name="Front Row Crew.com" \
-F caption="FRC In the house!" \
-F description="Homepage of GeekNights and other fine free entertainment." \
-F source="http://www.youtube.com/e/NKWpGJ4Xhw8?autoplay=1" \
https://graph.facebook.com/PAGE_NAME/feed

You don’t actually have to include all of that information with every post. Just include the information that is relevant. I have simply included all of the possible information to demonstrate a complete example. Here is what the post above will look like on your Facebook page.

The avatar and the username on the posting are going to be the avatar and username of the page itself. You can’t change those from post to post.

The word “testing” that you see there is the message of the post. Most of the time I imagine you are going to be posting just messages without any other information.

After that is a separate section with the link, picture, video, and everything else. Depending on which combination of information you post, that section will work differently. You only get one of these sections per post, so depending on what you want to do, you might need to stretch it out over multiple postings. If your post only contains a message, this section won’t exist at all.

The link field controls the URL where the user will be sent if they click on the blue text at the top of the section. The name field controls the blue text itself. If you don’t specify a name then Facebook will visit the link and figure out a name to use. In this example the link is http://frontrowcrew.com and the name is “Front Row Crew.com”. If you know HTML you can think of it like a basic “A” tag, since that is what it is.

<a href="LINK_GOES_HERE">NAME_GOES_HERE</a>

The caption controls that light gray text directly beneath the link. The description controls the block of text that is one space beneath the caption. If you leave either of these blank then Facebook will visit the link in question and try to figure out a caption and description on its own.

The picture is where things start to get a little bit complicated. First of all, if you specify a link without a picture, then Facebook will actually pick an appropriate picture based on the link, just like it does with the name, caption, and description. Since it will probably pick a bad picture, I suggest you set the picture manually whenever specifying a link. The picture is just a link to any image on the web. Clicking on the picture thumbnail will take the user to the URL of the link.

You can actually specify a picture all on its own without any link at all. In that case clicking on the picture will take the user to see the picture in its full size. The caption, and description will be extremely barebones unless you manually specify them.

Source may as well be called video instead of source. It needs to be a URL directly linking to a video file, usually a flash video. I’m pretty confident other HTML5 video formats will work, but I haven’t tested them myself. Don’t make the mistake of setting the source to a video’s page like this.

http://www.youtube.com/watch?v=NKWpGJ4Xhw8

That won’t work. You have to set the source to be the actual video file as in the example.

http://www.youtube.com/e/NKWpGJ4Xhw8?autoplay=1

You are on your own on how to structure these links on other video sites besides YouTube. I think I can safely assume the vast majority of people are using YouTube.

If a source is set without a picture, you may get an error if Facebook can’t figure out a video thumbnail on its own. In this case, you must specify a picture to successfully make the post. The picture will be used as the thumbnail. Clicking on the picture will cause the video to play. For YouTube the player is embedded within Facebook itself. As always, your mileage may vary with other video formats and sites.

Now, Facebook is actually pretty smart. Let’s say you specify just a link, and that link goes directly to a YouTube page. Facebook will automatically figure out the source and picture on its own, assuming you do not manually override them.

Lastly, what if you specify absolutely everything as in the example above? Obviously you will have complete manual control over all of the text appearing in the post as well as the picture. Clicking on the picture will play the source video. Clicking the blue text link will always go to the specified link. In this way you can actually link to two different things. Clicking on the picture will play a video, but clicking the link can take people to your blogpost about the video or other related place on the web.

Also, it should be noted that there is nothing forcing the picture and video to match. You can post a thumbnail for a video that is not related to the video at all. Users might be very confused if you abuse it, but there are legitimate reasons for doing so. You might just a brand logo as a thumbnail on a video which does not include that logo.

One last thing. The post will always return some information formatted in JSON. If it fails there will be an error for you to read. If it success there will be an ID that is the ID of the status update you just posted. You can save and use that ID for later if you want to edit or delete the post.

Conclusion

Figuring this out cost me many hours. If this saves even one person from that struggle, then it was worth it.

I hope this can also serve as an example to other people who write technical tutorials or howtos. You can’t just skip over details. You need to include every step and explain the what, why, and how of all those steps. Otherwise readers have to go to ten different sites to piece together the complete picture like I did. Even worse, users might blindly copy and paste without understanding what they are doing.

And of course, I hope this shows just how much of a pain OAuth can be when it is used unnecessarily. With my suggestion of every account also being an application, many of the OAuth steps would be eliminated without sacrificing any security or privacy.

But now we run into problems of security. What if someone else calls the lab impersonating your doctor and gets your test results? What if you change doctors and the lab releases the test results to your old doctor against your wishes? What if the lab is full of jerks and they send all sorts of crap to your doctor in your name. We have this exact same problem on the Internet, and that is why OAuth was created.

For decades the most popular means of authentication on computers has been usernames and password combinations. You want to prove to some system that you are a certain person, so you enter in a secret piece of information that only you know to prove you are you. Now that you have identified yourself the system will allow you to engage in activities permitted to your account.

Now let’s say you have a newsletter system, and you want it to send out emails on your behalf. Even to this day a very popular way of doing this is to give the newsletter system the username and password to your email account. The newsletter system will authenticate on your behalf, so your email server will think that the newsletter system is you. That’s what authentication is all about. As far as your email server is concerned the newsletter system is you, and has permission to do everything you do, including changing your password or deleting your account.

It’s obvious that this is a problem. Even though that goes against everything you’ve ever learned about security and passwords, people still do things this way. You should never tell your password to any other person ever, not even your closest and most trusted people like spouses. You definitely shouldn’t tell it to other computer systems. You should only ever send a password to the system for which it is intended. Your email password is for you, your email server, and that’s it.

Well, I made this fun game on the web, and my users want to tweet out their high scores. I could just suggest they do it with copy/paste, but that’s like driving back and forth across town to get the test results. Instead, I want my game to do the tweeting on behalf of the users for the greatest efficiency. Now I could, and many systems did, just ask for the users to give me their twitter usernames and passwords. It is possible I could be honest and not abuse that, but do you trust me? Even if I was an entirely client-side application, and your password is being sent directly to Twitter, this game is not officially endorsed by Twitter. How do you know I’m not secretly sending your password to my own servers or doing something else shady?

Thankfully we have solved this problem, and the solution is OAuth. OAuth is slightly complicated, but it gets the job done. Let me explain it as simply as I can using our high score game tweet example. We’ll call the game shootyguy.

You have a Twitter account, and I want you to let shootyguy send tweets on that account. First shootyguy gets its own special twitter application account. That account has an ID and a secret, which is basically the same as a username and password. Now shootyguy can authenticate with Twitter and prove that it is indeed shootyguy.

When you get a high score for the first time it will give you a link saying “click here to let me send tweets for you!” That link is a link to Twitter. It contains shootyguy’s ID number and a list of the things shootyguy wants you to let it do. In this case the game is only asking permission to send tweets, but it could also ask permission to edit your avatar, change your profile, etc.

You click on this link and you are sent to Twitter itself. It’s safe to give your Twitter password to Twitter, obviously. You do so, and Twitter asks you, “Hey, do you want to let shootyguy do the following things with your account?” If you say no, nothing will happen. If you say yes, then Twitter will remember that shootyguy has permission to send tweets for you. It will also send shootyguy a special secret password that it can use to act on your behalf. If shootyguy is a popular game, it’s going to have to remember those secret passwords for every user who has tweeting enabled. As long as the game holds onto these secrets, and you don’t take permission away from it, it won’t have to ask permission again. This painful user interface problem of jumping back and forth between web sites only has to happen once ever.

This is what is called authorization. You see, both you and shootyguy authenticate with Twitter to prove you are who you say you are. You are authorized to do whatever you want with your own account. Shootyguy is authorized to send tweets, but nothing else, on a whole bunch of accounts that have granted it permission. Up until relatively recently we didn’t have authorization, just authentication. Now that we have it, it’s easy to see why it is so incredibly important.

Perhaps the most important feature of authorization is the ability to deauthorize. Imagine if shootyguy got hacked like the PSN and it had a database full of Twitter passwords. That would be extremely bad. You would have to change your password before hackers started using it. If they had used OAuth, you could just remove authorization for the compromised application, and your account would be perfectly safe. Worst case the hacker starts doing the few things you authorized them to do before you turn them off.

I think I have made it very clear that in cases where you want to give a third party access to one of your accounts, OAuth, or a similar authorization system, is an absolute necessity. Anything less is grossly negligent.

Given that, what is up with the title of this blog post? Shouldn’t it be “Always OAuth”? Well, that seems to be the opinion of many sites out there such as Twitter and Facebook who are forcing OAuth as the only way to interact with their APIs. I can see why they do this. OAuth is more complex to implement, so developers would keep taking the easy way out by collecting passwords if they were not forced to use OAuth. It’s also next to impossible to educate users on not giving out their passwords.

Yet, there are still those rare occasions when OAuth becomes a huge pain in the ass for no benefit whatsoever. Let’s turn the tables around a little bit. Let’s say that we make an official shootyguy twitter account. We want to automatically tweet on that account whenever a player beats the game at shootyguy.com. How does that work?

In this case, we don’t need to ask the player for authorization. We aren’t using the player’s Twitter account. Shootyguy is using its own account. There’s no benefit to using OAuth in this case. It would just be easier for shootyguy to authenticate with its own username and password, and have full authorization to tweet with its own account.

The problem is that shootyguy is a program, not a person. It can’t just visit twitter.com and send a tweet. It has to use the API to tweet programmaticaly. But the API forces you to use OAuth no matter what. Since it is a program, and not a person, how can it click the “Allow” button in the browser? It can’t. What you have to do is manually do the OAuth in a browser and collect and save all the secret keys along the way. Then you put all the secret keys on the shootyguy.com server so it can send tweets.

Is it really that hard? No, there are many things that are more difficult, but it’s still a pain. In a way it makes sense. The shootyguy account is giving permission to the shootyguy.com application to tweet on its behalf. But Twitter is actually pretty simple as far as OAuth goes. If you want to see frustration, try it with Facebook.

Pretend that instead of posting to a shootyguy twitter account, we want to post onto the wall of the official shootyguy Facebook page. Well, a Facebook page isn’t a Facebook account, or is it? And a Facebook account isn’t a Facebook app. I’m going to write a tutorial on every step involved in doing this, but here’s the TL;DR version. You have to create a facebook account, a page, and an application. You have to authorize the account to post onto the page. Then you have to authorize the application to have a bunch of permissions on the account. Then the application has to use its secret keys on the account to get yet another secret key it can use to access the page.

Again, this is actually a great system when you are using it for what OAuth was designed for. If you want to give the game permission to post on your personal Facebook wall, this is fantastic. For the game to post on its own Facebook wall this is awful.

Of course, my complaining isn’t for naught. I actually have a solution that doesn’t require changing OAuth itself in any way. The solution is actually quite simple, and it actually opens the door to more features for users that have been unexplored.

I won’t hold back, the answer is to make every account also an application that has full authorization of itself. Why are applications and accounts separate entities? Why do I have to make my own Facebook application and connect it to my account? If my account were also an application, it would only need authorization to do things with other accounts, but not as itself. I shouldn’t have to pretend to be an authorized third party application to post on my own personal Facebook wall through the API.

One side benefit of this is that every user account can be authorized to perform actions on other user accounts. That can be dangerous if people give too much authorization away, but I think it will be very clear to people what is going on. “Do you want to give your friend joeyjoejoe1337 the ability to edit your profile? Yes or No?” It’s not a question too many people are going to get wrong.

While you may not immediately realize it, there are many cases where people will want to say yes. A parent can give their children their own personal private accounts, but can then authorize their own accounts to have a certain amount of access. You could authorize your trusted friends to handle your accounts in case of emergency without giving anyone your passwords. You could have a company account and give many people access to it without sharing a password around the office. You could then easily add and remove authorization as employees come and go. There are many duct tape third party systems that add this functionality for businesses that could be done away with completely if my idea were implemented.

I hope from reading this that you have a good conceptual understanding of the difference between authentication and authorization. I also hope you understand what OAuth is, why we have it, and why it is so great. And though I know it won’t change anything, I just wanted to complain about the unnecessary complexity that can make OAuth sucky under certain circumstances. If you are building an application, make sure you use it appropriately, and please try out my idea of every account being an application. And remember, never ever give your password for anything to absolutely anyone at any time under any circumstance. It is never necessary. No, not even in that circumstance you just suggested.

Firstly, these attacks are not really all that effective. Defacing web sites? Releasing customer data? That doesn’t make any difference in the grand scheme. Do you remember viruses like the ogre? It erased all drives in a computer. When was the last time someone wrote a virus that actually did serious damage like that? I’m not aware of any in recent memory.

Why don’t we see such harmful attacks anymore? Is it because that sort of attack doesn’t serve the motivations of the hackers? Is it because our modern systems are properly hardened against these kinds of attacks? Are these attacks happening and not being publicized?

Whatever the reason is, it is both disappointing and encouraging. You see, I can respect this sort of attack in the way that I can respect someone like Mike Tyson. His boxing is a sight to behold, but I have no desire to behold a man having his ear bitten off. You can impress me by writing a virus that turns off CPU fans and forces people to buy new PCs, but I don’t actually want to see that to happen. That makes the world a worse place by wasting a bunch of valuable resources for no benefit.

Secondly, these attacks lack technological sophistication. SQL injection and DDoS against soft targets? Please. These techniques aren’t demonstrative of deep technological knowledge. Regardless of my feelings on the motivations or target of an attack, as a technology professional I have a hard time respecting such easy hacks.

Imagine someone who burglarizes an old grandma who left the door unlocked. Even if it was a mean old grandma who needed to be taught a lesson about security, that thief isn’t going to get love from anyone. Because the burglary was so easy, the perpetrator is easily branded as a coward and lowlife scum.

Meanwhile, there are two other kinds of thieves who do get love and respect that makes up for their evil ways. The first is the professional thief or con-man. Their talents are so great that we can romanticize their stories and root for them. Despite acting immorally, their biographies become hit movies. The other kind is the Robin Hood who commits a crime for a good purpose. They do something that is illegal, but is not viewed as wrong given the circumstances.

What I want to see happen are some attacks that are both technologically difficult and have effects that benefit the world in some great way. If you are willing to so blatantly ignore the law, unlike myself, then cut it out with the lame hacks. Do something big and serious that in and of itself will have a huge positive effect for all people on the planet. Don’t be that guy who robs a convenience store at gunpoint. Instead, strive to be like Lupin III or Frank Abagnale.

Since things like this haven’t really been done before, I think I have to give some examples. Here are my top ten ideas of possible hacks that would have a huge positive benefit on society. The perpetrators of these attacks would definitely earn a great deal of respect to go with their time in prison. Post your own ideas in the comments!

1. Hack stores like iTunes or Steam to give everyone free games, movies, music, etc. Remove region locking on services like Hulu and Spotify. Allow art and culture to be spread around the world for free.
2. Release the source code to important programs like Adobe Creative Suite, Microsoft Windows, iOS, the Google Search algorithm, etc.
3. Take control of ISPs to remove filtering, bandwidth limiting, wire tapping, and other nefarious systems.
4. Hack into the systems that host and sell academic journal articles, and get the full text of all the articles onto the net somewhere for free.
5. Hack into the world’s financial systems and bring down all the evil investment banks like JP Morgan, Goldman Sachs, etc.
6. Hack into government and corporate back office systems to expose corruption around the world. I’m sure there are plenty of Enrons waiting to be discovered.
7. Completely disable industrial facilities that are hazardous to the environment. Even if they come back online, a few days without something like coal burning makes a big difference even at the expense of a power outage.
8. Completely bring down the great firewall of China or other Internet censorship systems.
9. Attack the television satellites and turn off almost every TV in the world for a significant time period. At the very least replace Fox News with Al-Jazeera English.
10. Disable the nuclear arsenals of every country that has them. Let the power plants stay, just disable all the missiles.

I think you get the idea now. LulzSec and others have been thinking too far inside the box. The lack of imagination and lack of technological skill is why nothing like the attacks I’ve describe has happened, or will happen anytime soon. If anything like the items on my list even comes close to happening, I will be more impressed than I have been in my entire life. As of right now, I’m still yawning.

I fully expect to wake up tomorrow to see that my computers will not boot, and they just display laughing skulls all over the screen. I’d be mad that I would have to fix all my computers, but a little bit happy that that can still happen.

Just about every day I read about some article about turning New York City into Silicon Valley 2. What bothers me is they take it for granted that this is a thing that is a good idea, and move right along to discussing how it can be done. It’s obvious why the startup people want to do this. There are a lot of people and, more importantly, a lot of dollars in NYC. It’s the most important city in the world. Yet, there are few technology-centric businesses based here. To these people the city is a huge untapped mine filled with gold.

The thing is, the location of technology companies doesn’t really matter that much does it? If I were starting a tech company I would probably pick Kansas City once Google brings the fiber there. The Netherlands is also a good choice, and has the fiber right now. The only real reason they come to NY is because they need more developers. The valley is tapped.

If they have marketed these entrepreneurial ideas so well, why are they having a hard time getting these developers? Why can’t they replicate the valley in New York? What is giving them all this trouble? I’ll make a general trollish statement and say that New Yorkers aren’t morons like those Californians are. Less trollish statement, New Yorkers aren’t as naive as Californians.

You see anyone who has a silicon valley attitude already probably moved there, or wants to move there. If someone stayed in New York it’s because they have a New York attitude. We aren’t the kind to work all night eating only instant noodles in exchange for a miniscule chance of getting rich and making some venture capitalist much richer. We have something here called rent, and it’s really expensive. We also like to eat good food, which is also very expensive. If you want us to work for you you have to pay us in cash, and lots of it. Venture capitalists don’t like that. They want kids right out of college who will work for peanuts because they don’t know any better.

Also, while New York lacks technology companies, it doesn’t lack technologists. Every desk has a computer on it. There are thousands of technologists in this city, many of them among the best in the world. The thing is, we all work for companies that do something else besides technology. Many work for financials that make money by having money to begin with. Some work for insurance companies, media companies, advertising firms, or *gasp* companies that actually have a product to sell. In other words, we work for businesses that actually make a real profit with a real sustainable business model that hasn’t been made obsolete like Hollywood has.

All these venture funded startups, how many make real profits? LinkedIn just had a huge IPO, how much profit do they actually make? Apparently  the company is valued at 521 times its profits. I don’t know how math works in CA, but in NY we know how to use a calculator. That doesn’t add up. If we’re going to work somewhere and get paid, it better be someplace that actually makes money. If you promise me the salary I demand, you better damn well not bounce that check.

These people are treating investment dollars like they’re revenues. In NY people are still living in the real world. We’ll get excited about real things, like sales figures going way up. If some morons decide to invest a huge sum in a company that isn’t turning a profit, that’s not something to cheer about. It’s something to be very worried about. If I see the champagne opening, my first instinct is to run.

I also have a suspicion that the weather makes a big difference. New York has nice weather, but the winter is cold and snowy. That has a huge psychological effect on people. Because we are often grumpy, we want to escape. If you make a New Yorker rich, you know what they’ll do? They’ll buy a luxury apartment, a house in the Hamptons, and a tropical beach house. Then they’ll retire immediately never to be seen again. Make a valley entrepreneur rich, and they’ll show up to work the next day like nothing happened. The weather is so nice, they are happy with their late nights and ramen dinners. They don’t want to escape, they are happy to work forever.

If you’re one of those people trying to push your silicon valley mentalities in New York, let me be the first to tell you to literally get out of town. You’re not wanted here. If anything, I think you guys should take a little New York back home with you. When you have real money and are ready to pay my rent, then give me a call. Until then, save your cash for plane tickets.

Despite the greatness in the indie video game scene, there is also a great deal of fail. Because indie developers lack the resources of giant corporations like Nintendo or Electronic Arts, there are certain flaws in their games that we have to accept. The controls won’t be as polished and smooth. There might be graphical glitches on certain video cards. The game might crash under weird circumstances. Without the money or time for thorough testing, this is just a reality of life we must accept. I forgive the indie games for these kinds of flaws.

That being said, there is one area in which failure is absolutely not acceptable. That area is online multiplayer. It seems that just about every week a cool new multiplayer game comes out on Steam, but the networking is a complete disaster. This is absolutely unacceptable. If it’s primarily a single player game with a small online component, then it’s no big deal if that part doesn’t work. However, if it’s primarily an online game then the game may as well not exist if the networking is busted.

As a programmer I know full well how difficult it is to write netcode that works. There are so many factors to balance including latency, synchronization, cheating, firewalls, etc. It’s no doubt a huge pain to write good net code. To those that have succeeded in this task, I salute you.

Even worse, writing netcode is a distraction from making your game. A game developer wants to make their game as good as possible. In a competitive online multiplayer game that means working on things like balance, controls, user interface, level design, and adding more modes of play. Every minute spent working on netcode is a minute a developer is distracted from the game itself. It’s no wonder that developers don’t spend much time on it.

That being said, if your game is primarily an online multiplayer game, it may as well not even exist if the netcode doesn’t work. I acknowledge that it is difficult code to get right, and that it distracts from the task of actually making the game. I just don’t accept those as valid excuses. Networking is hard, not impossible.

In 1996 Id software released QuakeWorld. The original Quake had netcode that was only suited for LAN play. QuakeWorld fixed it so that Quake could be played beautifully over the Internet. Tribes 2 was released in 2001. The netcode was so amazing that the game was playable with a 56k modem. We are living in the year 2011. No matter how hard the netcode problem is, it’s been a solved problem for fifteen years. I’m sorry. There is absolutely no excuse for getting it wrong.

One reason people fail is because they try to do something other than the classic dedicated server model. In my opinion, this is the way all non-MMO games should do online play. It’s a design that works beautifully for over a hundred thousand Counter-Strike players every second of every day. I’m all for trying new things. Rotary engines are pretty cool because they actually work. If you want to try to make a new kind of network engine, that’s great, but it had better work.

Not following this advice is how we get games like Frozen Synapse trying to use centralized servers that they can’t keep available. It’s a turn-based game, and they can’t even keep the server up. We also get games like Ace of Spades which wants to launch the game from in-browser links instead of typing IP addresses into the game itself. Most times when you try to play you just get a connection error. If these developers had done things the old fashioned way, they might be in better shape.

Another mistake developers make is they don’t build the networking in from the ground up. Having online play affects the overall design of your game in many ways. If you don’t code your game with networking in mind from the very beginning, it will be enormously difficult to properly add it later on.

Minecraft has this problem, and that’s why so many people keep making all sorts of server mods for it. Even with the mods I consider the Minecraft multiplayer too awful to be even worth playing. Notch has the money now. If he wants multiplayer to not suck, he should build a new Minecraft that is designed for networking from the start. They can call it Multicraft.

Civilization V is a massive failure in this regard. Not only are their lag and connection issues, but the actual game mechanics actually don’t hold up in multiplayer. The simultaneous turns give a huge advantage to the first mover in combat. When Civ IV came out, people stopped playing Civ III. The fact that many people are still playing Civ IV after Civ V has been out for months shows you they really messed something up.

Magicka is a game with a ton of network problems. Even after tons of patches, the game still gets out of sync. Diablo II was perfect in 2001, so why is Magicka flawed? The Torchlight guys have the right idea. They have no networking whatsoever in Torchlight. If you can’t do it right, don’t do it at all. They are making Torchlight II with networking in mind from the start. I have high hopes for them doing a good job.

Natural Selection is my second favorite FPS of all time after Tribes 2. It’s a Half-Life mod, so the networking just works perfectly. I was very excited for Natural Selection 2, and I pre-ordered it immediately. The developers started out on the Source engine, but abandoned it to write their own. While their engine is very impressive, they have failed at networking. Their experience comes from making mods, so they had no experience with networking. Perfect netcode was provided for them by the underlying engine. Natural Selection 2 is still in beta, but the networking is awful. There is insane lag. It gets better with every patch, but it is still nowhere near satisfactory. At least we’ll always have NS1.

You might argue that some of these games are commercially successful despite their sub-par networking. That is true, but they are very lucky. Introversion software released a great RTS called Multiwinia. When it was first released many people could not connect to any servers due to network errors. I don’t know if they eventually fixed it because I stopped playing. The broken online play prevented a community from forming around the game, and the game died. Compare that to Defcon, another Introversion game that has working online play, which still has a community to this day.

I’m sure everyone is also familiar with the Nintendo way of failing at online multiplayer. Friend codes and other weird systems create a huge barrier to online play, even if the netcode works. Borderlands has fine netcode, but reliance on GameSpy accounts makes it annoying as hell to get it running. Steam and XBox Live are both perfect. Every single game should integrate as much as possible with one or both of those systems. Why would you intentionally create a barrier to entry for your game?

Networking is hard, so it is obvious why so many game developers fail at it. It’s just incredibly frustrating for us as gamers to spend money on a multiplayer game that has broken online play, where there is really no excuse. Almost every game should use the standard model of dedicated servers integrated with XBox Live and/or Steam as fully as possible. It’s bad enough that so many games with big potential die out from lack of interest, it’s an even greater shame if it’s due to technological failure.

Right now I’m at a point where I buy all kinds of cool indie games only to be disappointed when the networking doesn’t function. I’m at the point where I’m not even going to buy games now until I have confirmation that they work. With more technological failures more people might take on this attitude, and it could be a serious blow to indie game developers, and thus a serious blow to video game innovation. Nobody wants to see that, so just make the networking work. There’s no valid excuse.

Take for example the Nintendo DS vs. something like the Pandora. The Pandora is really no comparison. Just looking at it makes me shudder it’s so hideous. There’s little incentive to actually develop anything for it since nobody owns one. Hooray, you made a game for the Pandora that nobody will ever play.

Meanwhile, the Nintendo DS is a closed system. Nintendo will only give you a dev kit if they approve you, and you pay them money. Then you can only release your game if they approve of that. This hasn’t stopped people from reverse engineering it and homebrewing the heck out of it. There is more homebrew action going on the DS than on even the most popular open systems out there. The incentives of developing for such a widely used well designed system make with worth the hassle of hacking and reverse engineering it.

The same is true for almost all consumer electronics. We have a bunch of open devices that suck and don’t really work, and we have awesome closed devices that we have a hard time pushing beyond their limitations. What would happen if one of the companies making closed devices would just cave and open it up?

Actually, this has happened. Do you remember the Linksys WRT54G router? I don’t have sales figures, but it’s obviously one of, if not the, best selling routers of all time. Because of the GPL, Cisco was forced to open the source code of the firmware for early versions of the router. The result is that a huge community sprung up of people buying this router and flashing it with firmware that added a ton more features. This obviously helped their sales by a ton, so why did they change the software in later versions and keep their future routers closed? If they had kept their routers open, Netgear, D-Link, and everyone else would have no chance of competing unless they opened up as well.

Now, depending on the device, going completely open can pose problems. Sometimes it can allow software piracy, like with the iPhone or DS. Sometimes it can allow media piracy, such as with an open source Blu-Ray player. But even in those cases, an official dev kit that anyone can buy will not help piracy that much.

Even with completely closed Blu-Ray players, the whole system is already cracked wide open. Pirates are pirating, and will continue to pirate. Little to nothing can be done to change the rate of piracy. Yet, imagine if Sony made one particular model of Blu-Ray player that was open. Without marketing it heavily, they just had a page on their site with documentation and development tools for the player. Maybe they even discreetly sell a dev kit for a token fee. That will almost immediately become the #1 Blu-Ray player, and no other will be able compete with it.

Imagine if Nintendo discreetly sold official DS flash cartridges with a development kit. They would make a ton of money since all those dollars spent on R4 cards would be in Nintendo’s wallet. Piracy of DS games is rampant and unstoppable anyway, this would at least give more money to Nintendo instead of the people who make the R4.

Of course, this will never happen. These big old companies are set in their ways, and there is pretty much zero chance they will be smart enough to do something like this. They are too stubborn and old fashioned.

That doesn’t mean it can’t happen. I’m calling out all the people who make open hardware to change their ways. Instead of making something like Pandora, make a product with a consumer focus. If you design for developers, you’ll only get developers buying it. Be like a real company and focus completely on consumer sales. Just while you’re at it, make the device open and make development tools available or for sale. Don’t heavily promote the openness, promote the device itself. By keeping the openness quiet you won’t scare anyone away, and you’ll grow a sleeper hit from the underground. Think of it as Linksys router on purpose.

Imagine yourself right now trying to buy something like, oh, a television. There are so many to choose from in different sizes with different specs. But one feature that no television has is openness. Imagine if there was just one model of television that had a dev kit available on some page of the manufacturer’s web site. They didn’t promote it or draw attention to it, but they had it there. Would that not immediately become the most popular television? Of course it would. It would be the obvious choice of almost every nerd. After the nerds got done with it, it would become the obvious choice of non-nerds as well.

If you are a consumer electronics manufacturer, please make your stuff open. It’s in the best interest of your company and your customers. Throw away your old ways of thinking. Just do it. Just try it on one model. If it doesn’t work, cancel it. At least try. What’s the worst that can happen? It sells the same as it would have sold if it were closed? That’s not likely.

What fascinated me even more was that he had trouble actually getting the game to work. Even if this day and age when people can run Doom on everything from iPods, pocket watches, to microwave ovens, it’s very strange that someone would have trouble running it on desktop x86 PC. Because I think it is so important that every person, and especially every gamer, should at least have experience with this game, I present to you a tutorial on how to play Doom.

The first step is acquiring the game. This isn’t really as obvious as you might think. You see, the Doom engine has been open source for a very long time. The only part that is not free and open source is the game data itself. The maps, the art, the textures, all those things are still commercially owned by Id software, and you must pay money to acquire them legally.

The Doom engine stores all of those things in WAD files. An entire game for the Doom engine is stored in a single WAD file. You can, of course, acquire them illegally. If you don’t pay money, but want to stay legal, you can only acquire the Doom demo version WAD files. Personally I suggest you go on Steam and purchase the id super pack. It’s a great deal that will also give you Quake, Quake 2, Commander Keen, and many other games which are required playing. Make sure after you purchase the games that you install them in Steam to get the files downloaded to your machine. It should take a matter of seconds on a fast connection.

Now, if you are running Windows, you could attempt to play these games directly in Steam. This will launch the original Doom engine using DOSBox. This might work for you (it does for me), or it might not (it didn’t for Totilo). Even if it works, it is not a great experience. This engine was designed for DOS, and it shows its age on modern machines. It will run at a really crappy resolution, and it has limited options to make things any easier on you. Stick with it if you are a purist, but I’m betting you aren’t.

I do suggest that if it works for you, you should play classic controls mode for a few minutes. This way you can learn what Doom was like in the olden days. You had to hold a button down to enable strafing. You couldn’t aim up or down. You couldn’t jump. You had to hold a button to run. You couldn’t really mouse look. It’s important to know what it was like so you can fully appreciate the great luxuries we have today.

Now that you know what it was like all those years ago, you don’t need to suffer like that anymore. Whether you are running Windows, Mac, or Linux, grab yourself a free copy of the Doomsday Engine. Because the Doom engine is open source, many people have rewritten the entire thing to work well on modern computers. Doomsday engine is just one of these, but it has worked very well for me on all three platforms. I recommend it.

Installing the Doomsday engine is pretty straight forward. The only tricky part is that you have to tell it which Doom engine games you own, and tell it the locations of the WAD files. Check off the boxes for every game you have purchased on Steam. If you purchase the id complete pack, that’s every game except for the demo versions. The WAD files should be located in c:\Program Files (x86)\Steam\steamapps\common\GAMENAME\base\FILENAME.WAD

Just replace GAMENAME with whatever game you are looking for, such as ultimate doom, and replace FILENAME with the filename of the wad file that Doomsday is looking for. The only thing that might be tricky is that the Hexen: Deathkings of the Dark Citadel game actually requires two WAD files. Don’t worry, they’re both in the correct folder. If you are running a 32 bit Windows, you can leave off the (x86).

Once you’ve told Doomsday which games you have, and where the WAD files are, you’re in modern PC gaming territory. Doomsday’s configurations are almost exactly like those on every other PC game you are used to. You can select modern resolutions, completely configure your controls, go nuts. You can even do things like add in a mini map that wasn’t in the original game. The world is your oyster.

Just remember when you’re playing this old game with the new engine that it’s not the same as the original. You couldn’t jump, look up or down, strafe without an extra button, or change weapons with the scroll wheel. It’s an impure experience, but it’s good enough for you to learn about the Doom family of games. Most importantly it works, and it won’t have a person used to modern games quitting in frustration after a few minutes.

One more suggestion. In Totilo’s article he mentions not finding a shotgun for quite some time. The reason is that he was playing on a sissy difficulty level. He’s probably too young to die. I suggest you turn up the difficulty, otherwise you really aren’t going to learn anything. Nightmare is supposed to be nearly unbeatable, but you probably want at least hurt me plenty. If you do that, you’ll probably be seeing a shotgun toting bad guy right away.

One more hint. If you aren’t taking out one imp, or two or three regular guys, with one blast from the shotgun, then you’re doing it wrong.

Prepare to meet your Doom.

My first experience with Linux was in the very late ’90s. The first time I saw it, some kids at the nerd summer camp I went to were running it. I didn’t use it, and I didn’t learn it. Yet, as with all things related to computers, I was curious. In high school a friend lent me a Corel Linux CD-ROM. I tried with no avail to make it work on the family computer (486 100mhz). I didn’t realize I had to boot from the CD-ROM, and I don’t think that computer even had that capability.

It wasn’t until 1999 that I downloaded Red Hat ISOs over my 56k dial-up connection. It was the first computer I ever built and personally owned, a Pentium /// 450mhz. I was able to dual boot Red Hat 6.0 and Windows 98SE. The thing is, I soon deleted it. It didn’t work with my USB dial-up modem. Linux was apparently just a different desktop with some different looking applications, and bad hardware support.

It wasn’t until I got to college that I actually learned something. All the computer science labs ran Solaris, and I started to learn UNIX for real. I soon discovered Mandrake Linux. It actually worked with all my hardware, even my weird PCI IDE controller. Most importantly, I learned to SSH into the Solaris systems to do my school work, and even use X11-forwarding.

At this point I was hooked. I was running Linux as the primary OS in a dual boot system. I only loaded up Windows (2000) for PC gaming. The thing is, hardware support was still a problem. My hardware didn’t all work perfectly, and Mandrake didn’t update quickly enough to give me the updates that fixed those problems. That’s when I had the distro rodeo and picked Gentoo.

I tried every major Linux distro at the time, and BSD as well. None of them really impressed me, and Gentoo didn’t even seem to work. Yet, I kept going back to it. The splash screen was so good looking, the community forums were so helpful, and the documentation was so great, that I kept at it. Eventually I came to realize that yes, I actually had installed it properly multiple times over, but unlike other Linux distros X was not a default part of the system. Booting into a command line wasn’t a failure, it was success.

From then on I was Gentoo crazy. I would constantly be rebuilding packages updating packages, even reinstalling the whole system. I would constantly be tweaking and twisting system settings to see what they did. Mostly I messed around with the user interface. I even fell into using fvwm, the most customizable X window manager. By default it sucked, but if you configured it, you could go beyond anything else in existence. Boy, did I ever configure it.

Late in my college years, and after graduation, I was soon employed. My free time evaporated. I was spending a lot of time commuting to work. My iPod became a lot more important than my desktop. I needed Windows to run iTunes, and to play games. I had little desire to code outside of work. Yet, I still kept a dual booted Linux because it was nicer to SSH into my blog/podcast server from that, than from PuTTY.

By then, I was rocking Ubuntu. It just worked. It installed in 15 minutes, not three days. I didn’t need to jump through any hoops to get my NVidia card working. I no longer needed to edit the X configuration to get dual monitor support and proper resolutions. I still had a natural inclination to customize the user interface, but it was kept at a minimum simply due to the fact I had no free time for it.

Eventually I was running Ubuntu and Windows. Then my desktop was dual booted, but my laptop was just Ubuntu. And then when I discovered VirtualBox, and it became good enough, I ran Ubuntu on the desktop in a virtual machine, and only had Windows installed on the bare metal. And just recently I replaced that laptop with a new one. This new laptop is so powerful that it too can run Linux in a virtual machine. And thus, my days of installing Linux are over. I wish I could run Linux in the bare metal and Windows in a virtual machine. The thing is that I use Windows for gaming, and VMs are not so great for that. I use Linux for web development, and running it in a VM makes no difference for that at all.

You see, in college I had a ton of free time. I spent that free time working on my computer. I do not regret spending my time in that manner. Most of the Linux skills I use every single day I learned from rocking Gentoo. Even so, I can now look at my old self and laugh. I was not unlike a mechanic who always worked in the garage and never drove anywhere. I spent all that time trying to customize the computer to my preferences, that I hardly ever computed. That’s why I don’t have my own software business. While I was compiling and configuring other people’s code, the other guys were writing that code.

Thankfully I learned my lesson, and I hope you can learn the same. It is often quite difficult to change computers. You can spend hours just customizing keyboard shortcuts, let alone other settings. Then, as soon as you use a different computer, all of your customizations are gone. Even if your customization can increase your efficiency, is it really worth it if it takes you an hour to set it up?

Take it from someone who has been there. Instead of changing the computer, change yourself. It’s easy to change yourself, you have 100% full control. You don’t even need to spend time looking up how to do it. Get used to the default keyboard shortcuts instead of making your own. Learn to live with the default settings. Once you do, you can be just as efficient as you would have been with your customizations. Better still, you will be able to sit at any computer and get going immediately without being frustrated at a different setup.

It used to be that my computer felt like home, and every other computer felt like a foreign country. Thanks to cloud computing, any computer with an Internet connection and a web browser can become your home as quickly as you can login. You no longer have to spend a day installing and configuring software after you have a fresh OS install. You just have to install your favorite browser, and maybe one or two other things like Steam or iTunes.

Learning default settings only makes this even easier. I truly feel at home on absolutely any computer I sit in front of. I am familiar with all three major OSes and all the major browsers. I still do not prefer OSX, which I am forced to use at work, but I know it. I’ve also made vim my text editor of choice, and I use it with a very default configuration. Vim or vi is already installed on almost every system, I can easily use any server I come across. People who are perhaps used to graphical editors, like Eclipse, may have some slight difficulty if they have to SSH into a server and fix something.

If you get frustrated behind the wheel of any car other than your own, then do you really know how to drive? A real driver can sit in any driver’s seat and be off to the races after a quick seat and mirror adjustment. That’s not to say you should never spend time under the hood. It’s very important to know how things work, and to be able to fix them when they go wrong. Just don’t spend so much time under there that you never leave your garage.

You won’t get very far in this world by working on your computer. If you want to make it somewhere, you have to use the computer to compute. Start now.

It dawned on me that because of HTML5, it should actually be possible to create an HTML presentation with video that is seamless. Today I went ahead and prepared a video-centric panel for Connecticon, and I did it all in HTML5. I realized afterwards that I should build this into a tool, so that others can use it as well. Hence, Presentoh was born.

http://github.com/Apreche/Presentoh

The idea is really simple. There are only four kinds of slides I think you should be using: title slides, bulleted lists, videos, and images. I built a simple HTML template which could handle all four types of slides. I made a brain-dead simple CSS style for it which entails white text on black background. The black background also looks more professional because photos and videos really stand out. Also, it looks better when using a projector in a dark room because you can’t see the borders of the projection.

The only catch was that I wanted to use my Logitech Cordless Presenter. I went out and found jquery.hotkeys, which allowed me to bind any key on the keyboard to move between slides by redirecting in JavaScript. Problem solved.

All you have to do to use Presentoh is create a file containing JSON which defines all the slides in your presentation. Then you just run the presentoh.py script, tell it which json file you want to use, and it spits out all the HTML files for your presentation. Find the first slide in your presentation, open it in your browser, go full screen, and rock and roll.

I think it’s pretty awesome that while this tool is really hacky, and doesn’t have any error checking, it still beats out Google Docs, Powerpoint, Open Office, and all the rest in terms of how it handles video. There are some slight catches, but nothing that can’t be worked around. For example, you have to make sure all of your videos are the right codec for the browser you are using. My H.264 videos obviously didn’t work in Firefox.

Also another issue is that there is no way to automatically make an HTML5 video go full screen. If you’re wondering why, the answer is on stackoverflow. Therefore you have to manually specify the height and/or width of each video depending on its aspect ratio, and the resolution you will be presenting in. This is to prevent videos from going off the edges of the screen if they are too wide or too tall, and to prevent aspect ratios from being broken.

All the rest of the information can be found in the README file. I hope Presentoh helps somebody out there with their presentation needs. Enjoy.

I have recently started a new project, Project D.O.R.F., and many of the other developers are new to Git. Thus, they have been making some mistakes and not utilizing Git to its fullest. Thus, I have created this guide using Project D.O.R.F. as an example. As you become more experienced with Git, you can figure things out on their own. Until then, this flow will get you a very long way.

Assuming you have Git installed properly, the first thing you need to do is configure Git. It is done like so.

$ git config --global user.name "Your Username"
$ git config --global user.email "your@email.com"

These two commands will configure your username and e-mail address for every Git repository you work on with that user account on that machine. Every commit in a Git repository has an author, so this is necessary to see which people are responsible for what work. There is also a git blame command that can show who is responsible for each individual line of code in a repository. If these settings are not configured, those features will not work. Alternatively, you can change these settings on a per-repository basis, but I’ve never had a need for it.

The next step is to go to GitHub and create a fork. First you go to http://github.com and register for it, as you would any other web site. There is no need to pay any money unless you would like to have private repositories that nobody else can access. Next go to the GitHub page for the project you would like to work on, in this case it is http://github.com/Apreche/Project-DORF/. Then, click the fork button. This will create a copy of the project in your GitHub account. You can see that lackofcheese has a fork of the project at http://github.com/lackofcheese/Project-DORF/.

Now that you have your own personal fork of the project, you need to actually get the code onto your local machine to begin working on it. This is done with the git clone command. Remember, you want to clone your forked repository, not the original one. You also want to clone using the SSH method. Follow the directions on GitHub for configuring your SSH key.

$ git clone git@github.com:yourusername/Project-DORF.git

This command will create a folder named Project-DORF which contains a clone of the repository. If you would like to put the repository into a different folder, feel free to move or rename it. Alternatively, you can do something like this which will put the repository in a directory named foobar

$ git clone git@github.com:yourusername/Project-DORF.git foobar

So now it’s time to get to work. Go into the folder and take a look at the files, and learn the project. You’ve got to read code before you can write it.

$ cd Project-DORF

You’ve read the code, and you’re ready to make some changes, but wait! What branch are you on? Let’s check.

$ git branch
* master

The master branch, you don’t want to be working directly on the master branch. Do all of your work on separate branches. And if you are working on more than one task, put each task in a separate branch. For example, let’s say you are planning to improve performance of the renderer, but also fixing a bug in a path-finding algorithm. Don’t do those things on the same branch. Do them separately. That is the power and magic that Git has to offer you, so there’s no point if you don’t use it. We’re going to fix a bug on a separate branch, let’s go.

$ git checkout -b bugfix
$ git branch
* bugfix
master

Perfect, we have a new branch. Now, start coding! Are you done coding? Time to commit. Let’s pretend we edited game.py, and we made a new file foo.py.

$ git status
# On branch bugfix
# Changed but not updated:
# (use "git add ..." to update what will be committed)
# (use "git checkout -- ..." to discard changes in working directory)
#
# modified: game.py
#
# Untracked files:
# (use "git add ..." to include in what will be committed)
#
# foo.py
no changes added to commit (use "git add" and/or "git commit -a")

It seems like git recognizes that we have a new file, but it’s untracked. It also recognizes we have modified game.py, but it’s not updated yet. You can see that there are instructions right there on the screen as to how to commit our changes. We can checkout a file to reset it to its pre-edited state. This command will undo our changes to game.py.

$ git checkout -- game.py

But we want to commit our changes, which means adding them first. We can explicitly add the files with the git add command.

$ git add game.py
$ git add foo.py

Then we can commit the changes with the git commit command.

$ git commit

Let’s say you only want to commit some of the changes, but not others? That’s perfectly fine. Just add the files you want to commit, and don’t add the files you don’t want to commit. Protip: Use the special commands like:

$ git add -i
$ git add -p

to interactively select line-by-line which changes you want to add. If you add two bits of code to a file, and only want to commit one of them, then only commit one of them. Nothing is stopping you.

The thing is, most of the time you are working, you aren’t making new files. You are just editing existing ones, and you want to commit all of your changes. In these cases use this shortcut command.

$ git commit -a

This will automatically add and commit all changes to files that already exist. You will still need to explicitly use the git add command to add in any newly created files. You will also need to use the git rm command to remove files. If you move or rename a file, just be sure to git add the new one and git rm the old one in the same commit, and git will figure it out, like magic. git rm is just like git add, so you have to commit afterwards. You should also use commands such as git add -A and git add -u to easily add many files at once. See the official Git documentation for more on that.

You’ve got your code in a separate branch from master, and you’ve committed your changes. Now it’s time to share them with the world. But first, you’ve got some work to do. While you were working, other people were also on the job. There is probably new code out there, and you have to make sure that your code plays nicely with it. The first thing you have to do is to add the canonical repository for the project as an upstream source. You only have to do this command once per clone.

$ git remote add upstream git://github.com/Apreche/Project-DORF.git

You don’t have to call it upstream, you can call it whatever you like, but upstream is an appropriate name. Regardless, this command creates the hook you need to be able to get updates from the “lead” repository. How do you get those updates? Let me tell you.

There are three commands you have to be concerned with here: fetch, merge, and pull. Fetch will get the new patches from the remote repository. Merge will merge those patches into your current branch. Pull does both at the same time. Let’s do it.

Go back to the master branch.

$ git checkout master

Fetch and merge changes from the upstream master into your local master.

$ git pull upstream master

Ok, so now your master branch has all the latest goodies that everyone else has been working on. But are your changes compatible? Let’s find out.

Switch back to your bugfix branch.

$ git checkout bugfix

Rebase!

$ git rebase master

Rebase can be a scary thing, but it’s not scary if you are careful. In this case, what we are using it for is pretty simple. It takes your changes that you have made on your bugfix branch, and sets them aside. Then it takes all the new goodies from the master branch, and puts them on your bugfix branch. Then it takes your changes and puts them back on top of that. If there is a conflict, it will provide you with instructions on how to fix it. Git has three different merging algorithms, though, so that rarely happens. Now that you are all rebased, test your branch and make sure it works. If changes are necessary, go back a few steps. Add and commit the changes. Make sure upstream doesn’t have any new changes. If it does, rebase again, and so on. Eventually, you will have a bugfix branch worthy of sharing with the world. Let’s do it.

$ git checkout master
$ git merge bugfix
$ git push

You’ve now pushed your changes out to your GitHub page. Congratulations. Now the rest of the world can go to your page and see those changes. Other people working on the project might even take your changes and start fiddling with them themselves. Don’t worry about that, though. You are interested in getting your changes upstream into the canonical repository. So go back to the GitHub web site for your repository, and click the pull request button. Send a pull request out to your collaborators, in this case Apreche, and they will be notified that your changes are ready for merging. Wait patiently, and if your patches are good, they will be part of the upstream. More information on pull requests is available at http://github.com/guides/pull-requests

Now let’s get advanced for a second. Git really doesn’t believe in the idea of a canonical repository. It’s just a concept we use to make things easier for ourselves. Someone coming to GitHub for the first time is going to see 10 repositories with similar code, how do they know which one is the right one? This is why picking one to be the one is a good idea. However, since there really is no such thing, you can actually do some fancy stuff. You might want to use the git remote add command to add links to some of the other contributors on the project. For example, lackofcheese might do this to be able to fetch amelim’s changes that are not yet available at upstream.

$ git remote add amelim git://github.com/amelim/Project-DORF.git
$ git fetch amelim
$ git merge amelim/master


It also might be a good idea to merge different contributor’s patches into separate local branches which you then merge together. Never be shy about making a ton of local branches. You can delete the ones you are no longer using to clean up cruft, and the cost of making them is very minimal. Don’t hold back, branch like a mad man. Here’s an example of how I would merge changes from two other contributors with my code and then push it out on the master branch. When I was done with this, I would send out a pull request.

$ echo "These remote add commands only need to be done once ever per clone"
$ git remote add ameleim git://github.com/amelim/Project-DORF.git
$ git remote add lackofcheese git://github.com/lackofcheese/Project-DORF.git
$ git checkout -b mycode
$ git add somefiles
$ git commit
$ git checkout -b cheese
$ git fetch lackofcheese
$ git merge lackofcheese/master
$ git checkout -b melim
$ git fetch amelim
$ git merge amelim/master
$ git rebase cheese
$ echo "The melim local branch now has amelims changes merged with lackofcheese's"
$ git checkout mycode
$ git rebase melim
$ git checkout master
$ git merge mycode
$ git push
$ echo "mycode on top of amelim's on top of lackofcheese's pushed out to the world"
$ echo "let's delete the branches we are done with
$ git branch -D mycode
$ git branch -D cheese
$ git branch -D melim

When you execute the rebase commands, you may be prompted to fix conflicts. If so, just follow the directions on the screen carefully. They usually involve editing the files in question with your editor to fix the appropriate sections by hand, re-adding, and re-committing the changes. It doesn’t happen often, and it’s not nearly as hard as it is to fix with non-distributed version control systems. Just make sure to test your code after resolving he conflicts.

That pretty much covers my basic work flow. Just to reiterate. Make separate local branches. Do work on them. Pull (fetch and merge) the upstream changes to your master branch. Rebase your changes on top of master then merge them into master before pushing them out to the world. If necessary, pull the changes of other collaborators into separate branches, and rebase/merge their changes together. Then take the sum of those changes and treat them as you would your own. Rebase them on top of the upstream master, merge them in, and push them out.

I have followed this work flow for quite some time, and it has served me well. Feel free to ask any question. Happy coding.

It all comes down to one thing. Why must we be forced to choose between a good user experience and openness? How come we can’t have both? Every open platform has a god-awful user interface, relatively speaking. All Apple products, especially the portable devices, have a great user experience. I don’t personally like OSX, but I can’t deny that it gets a lot of things right that everyone else gets wrong. Font rendering is a good example.

People who don’t care about openness, or aren’t aware of the problem, are thrilled. They get everything they want. Hooray for them. People who do care about openness are enraged. Why are they mad about something they aren’t going to buy or use? This is easy to explain.

They are mad because they see an unfulfilled potential. Think of a screwdriver. A typical screwdriver can screw any screw that is of appropriate size. Imagine a copper screw and a steel screw that are otherwise identical. For some reason the screwdriver works on the steel screw but not the copper one for no apparent reason. I think most people would consider such a screwdriver to be broken.

A device like the iPad has all the hardware necessary to do a billion amazing things. However, it only does a few thousand things. Why? Because Steve Jobs says so. Who is he to tell me what I can and can’t do with this thing that I have bought? If you compare it to the screwdriver example, you can say that the iPad/iPhone/iPod are all broken. Broken technology enrages the geek, especially when they can’t fix it.

The nerd goes out to get a different screwdriver that is not broken. All these other screwdrivers are great. They work on all screws, as expected. However, they all have really uncomfortable handles. Every single screwdriver out there which is not broken is really uncomfortable. There is not a single exception.

Why can’t someone just take the good screwdriver and stick it on the comfortable handle? It’s so simple. There is no law of physics preventing it from happening. It’s 100% possible, but for whatever reason it just doesn’t exist. Every single screwdriver on the shelf is either broken or uncomfortable. The geeks can not be satisfied. Their rage multiplies exponentially.

This is the source of the rage against Apple. Apple seems to be the only company capable of making screwdrivers with comfortable handles, but their screwdrivers can’t do half of the things that other cheaper screwdrivers can do. Meanwhile, all those other cheaper screwdrivers give you blisters when you use them. I’m getting angry just thinking about this imaginary screwdriver scenario.

There are only two solutions I see. One is that Apple opens up. Apple is clearly capable of open sourcing things. They could just open their portable hardware and software, and that would be that. Obviously it will never happen, but I don’t understand why. If Apple did it, they would have immediate complete market dominance. If they had a comfortable and fully featured screwdriver, all competition would crumble instantly. There would be no reason to buy any competitor’s product.

The other solution is for the open source people to just shamelessly start ripping Apple off. If we can’t invent something better, which we clearly can not, then let’s reverse engineer and straight-up steal all of Apple’s stuff. Take the screwdriver handle down to the metal shop, copy it exactly, and attach it to a real screwdriver. Patent laws my ass. Why should we settle for anything less than achieving our full technological potential? Patents are supposed to help, not hinder, progress. If they are hindering, let’s just ignore them.

Google, I’m looking at you. You must know that Android’s UI sucks compared to iPhone. You have a jillion dollars. Just hire the best designers in the world, and get it done. What are you waiting for? You’re supposed to be a company of geniuses, why didn’t you just do this in the first place for Android 1.0? What can you possibly be thinking? What are all the non-Apple companies thinking?

The talk itself is insightful, and the research Jane is doing at the Institute For The Future is very interesting. I especially enjoyed the part of the talk where she discussed Herodotus’ story of the Lydians using dice games to ease famine possibly being true. It at least rings true considering the use of video games to distract ill people from their pain and suffering.

Regardless of that Jane’s talk seems to gloss over many obvious factors. Mainly she completely fails to deeply analyze the mechanics of the games themselves. The sad fact is that games can not change the world, at least not in the way she presents.

In the talk, Jane says that in games people present the best version of themselves. Players will never quit until their goals are achieved. Not only that, but they give their strongest effort, and are ready and willing to collaborate. If you ignore the griefers and shitcockers out there, this is largely true. What she doesn’t discuss very deeply are the reasons behind these behaviors.

The simple and obvious reason is that games have no true failure. If someone tries to be a hero in real life, and they fail, there are permanent negative consequences. In the real world, you risk losing your money, possessions, freedom, health, and the health of those you care for. It’s a dangerous world, and many mistakes can not be undone. This is the cause of the anxiety people feel in the real world that they do not have in the game world.

In the gaming world there is a reset button. Even if you fail, you can keep trying with no repercussions other than lost time. In Mario, you don’t feel much anxiety trying to jump over the gigantic pit because you have extra lives. If you’re down to your last life, then you actually do feel some anxiety because failure means having to start over from the beginning again. When games do actually have negative consequences for failure, players get upset and play something else.

I think the reason she misses this is because she focuses on World of Warcraft, a game with no consequences. Death isn’t even a real consequence in WoW, or other modern MMORPGs. It’s just a momentary obstacle that can be completely reversed. Take a look at some other games out there that actually have harsh consequences for failure. How about some old NES games that have no continues, passwords, or extra lives? How about Steel Battalion which would erase your save data if you failed to eject on time? These games are obviously ludicrously unpopular.

As for the topic of virtuosos and 10,000 hours of practice, I personally don’t buy it. Measures of talent are strictly relative. How many people out there have played over 10,000 hours of golf or tennis? Now how many Tiger Woods and Roger Federers are out there? How many people in South Korea have played that much Starcraft? What percentage of them are professionals? I know I personally have played many thousands of hours of video games, but you don’t see me sponsoring PC hardware like Fatal1ty. The first person to ever play the violin was a virtuoso in their time, though they would be terrible by today’s standards. Being a virtuoso simple means you are better than everyone else. Not everyone can be great. That’s what makes greatness great.

This fact directly interferes with the factor of epic purpose. Every person who plays Zelda to the end will defeat Ganon and save the world of Hyrule. In the real world, there is only one champion, and it’s probably not you. Even if you collaborate and give your full effort, odds are you will still fail to be exceptional in the real world. Tons of people around the world dedicate their lives to sports, music, and other things from a young age. A ludicrously small percentage make it big. Strong effort greatly facilitates great achievement, but does not guarantee it.

For the sake of discussion, let’s imagine that everyone actually can be a virtuoso. We’ll pretend that we can create an army of Steve Vai’s by practicing enough guitar. If that is true, then being a virtuoso at World of Warcraft is like being a virtuoso at the player piano. If an activity has no possibility of failure, then either every human being is a virtuoso at it, or none are, regardless of how much practice. It’s meaningless and unremarkable to be a virtuoso at an activity at which you are guaranteed to succeed. If every scientist got Nobel Prize, it would lose all meaning.

Let’s take this same exact TED Talk, and replace all of the World of Warcraft examples with Counter-Strike. People are not ready and willing to collaborate in Counter-Strike, often prioritizing personal kill ratios above the objectives of the team. They will not work hard towards their goal, often switching servers when faced with a superior opponent they can not beat. People are quick to cheat their way to victory because true victory is too much work, if it’s even possible. Most players quit and never try again when they can not achieve success with minimal effort. These are the same people that quit in real life. They are the boomerang generation running home to mommy because college is too rough. Sorry, I don’t believe that these people can save the world.

If you want to really change the world with games, you need games that mimic the real world. These are known as simulations. The military uses them to train people for various things, most notably flight. If you play a racing simulator for 10,000 hours as a child, you do actually have a good chance of becoming a professional race car driver. The thing is, an accurate simulation is just as difficult as driving a real race car, and will not attract many players putting in many hours. Tons of people play Mario Kart. Relatively few have even heard of Forza or Gran Turismo. If you could get people to play simulation games as much as they play Bejeweled, then you could change the world. Sadly, that will never happen because simulations are simply not fun to most people. This is why rhythm games have ludicrously simplified instruments. They make rhythm games with real instruments, and almost nobody plays them.

I do think that making the real world more like a game can save it. Make it impossible to fail at life. Give straight A+ grades to every student just for showing up to class. Remove all negative irreversible consequences from everything. Guarantee success for anyone who puts in enough effort, regardless of skill or talent. Do all that, and an army of gamers will revolutionize the real world. Then again, does such a utopian world really need changing?

Until then, I think this talk is actually kind of harmful. A lot of people who are otherwise wasting their lives away will use this to validate that time wasting. Some may falsely believe that they are saving the real world while saving Azeroth. Rather than changing the world with games, it may just be giving people another excuse to play games when they could be saving the world.

The fundamental problem is that people do not understand the concept of switching inputs. They simply do not get it that video and audio are separate. They don’t get that the audio and video come out of one box into another box. They do not comprehend that on the receiving box you must manually select which input you wish to currently be active. Now, people should learn this, but with a change to the HDMI spec, I think we can eliminate the problem in the long run.

Every cable box I have seen has a power outlet on it. Why is that? The reason is that you are supposed to plug your television into the cable box, and then plug the cable box into the wall. This will allow you to have the TV turn on and off when you turn the cable box on and off. Ideally this will eliminate the need for the TV remote control. Sadly, this only makes trouble in a more complicated system. For one thing, it means I have to turn the cable box on in order to play a video game. That’s just a waste of electricity.

The solution is to add one single additional wire to every HDMI cable. That wire will allow the devices to tell each other if and when they are on or off. The devices will also be able to command each other to turn on or off, or to switch inputs. It’s really so simple, I’m amazed it doesn’t already exist.

Here is an example scenario. You have a TV with an XBox and a DVD player. They are all off. You turn on the XBox. The XBox signals over the wire to the TV that it has turned on. The TV notices that it is off, so it turns on. Then it notices the XBox is the device which told it to turn on, and is the only device sending a signal. Therefore it automatically switches to that input.

Think about the difference that makes for the end user. They push the green button on the XBox controller, and they’re good to go. No fiddling with other remote controls. No juggling inputs. No nothing. Logitech probably wouldn’t like it, though, because it would ruin their Harmony product line.

I’m sure that you intelligent readers can figure out how the system will work in all sorts of other typical situations, but let me make a couple more examples. You turn off the TV, and it tells everything connected to it to turn off as well, saving lots of electricity. You turn off the XBox, and nothing else connected to the TV is still on, so the TV turns itself off. You have the XBox already on, but you turn on the Blu-Ray player. Well, it would be worse to switch away from a game in progress, so it should stay on the XBox. However, if the XBox turns off, it can switch to the Blu-Ray automatically.

Of course, the user will still need the ability to take manual control over which inputs are active, because the automatic setting will not be able to handle every single use case. Even so, such a system will drastically decrease the amount of times the user will have to manually switch inputs or turn devices on or off. It will also drastically reduce electricity usage because devices will almost never be on when they are in use.

Now, there are some extremely complicated situations that require figuring out. The best example I can think of is picture in picture. The TV and/or receiver has to be aware that picture in picture is a possibility. Maybe if you have picture in picture, and you turn on a second device, it opens in the small picture. Then if you turn off the second device, the small picture disappears. Even this complicated situation isn’t that difficult, you just have to spend some time figuring out the best possible default behavior for every possible scenario.

They’re adding all sorts of other great features to the HDMI specification like Ethernet, higher video resolution, audio return, etc. Those are far more complex than this simple one-wire feature. It will cost next to nothing to implement, and it will make a huge difference. There’s no rush, we can just do the usual thing we do with car safety equipment. All new products as of X date must support this feature. It will save a ton of electricity and reduce a ton of headaches. Too bad I’m just some guy and not an electronics manufacturer. I don’t even know the second step towards trying to make this a reality. The first step was this blog post.

I think the primary thing that confuses people about Git is that it assumes that if you are developing software on a team, that you actually talk to people. Whether it’s by IM, IRC, a web site, Twitter, Git doesn’t care. It just assumes that you actually have some form of regular communication with the other developers.

You see, the non-distributed version control systems do not make this assumption. If you are using subversion, your work flow is dictated by the limitations of the software. You do some code. When you commit, you also share. Everyone commits to the same place.

With this sort of system, you don’t need to talk to the other developers to know what to do. There is only one canonical current version of the code in one repository. If you do an svn update, you know everything you need to know. If you change the code, you just commit, and you don’t need to tell anybody. They will see your changes when they eventually do an svn update.

This system inevitably leads to the many problems that Git solves. The one I always like to point to is the situation where you need to make a quick bug fix while you are in the middle of working on a big new feature. You need to make a version of your code that has the fix, but not the new feature. If you committed the feature, it’s going to be a nightmare to get a version of the code without it. If you didn’t commit the feature, it is very hard to collaborate with others. Branching solves it, but branching in subversion is slow and difficult, and it’s shared. You might ruin everyone else in the process.

Git lets you do absolutely anything that is possible to do with your code. You can branch, tag, commit, share, patch, and unpatch your code any which way you can imagine, and some you can’t yet imagine. As long as you are aware of all the commands and features, you can never dig yourself into a hole. You will always be able to deliver the code with the combination of patches that you need.

The problem with this is that since you have so many capabilities, there is no dictated work flow. Should you have a central repository? Should you push back and forth between individual developers? Should we have any shared branches? Should we use tags? Should you use soft tags or hard tags? Should you merge? Should you rebase? All of these questions, and more, are the subject of much debate.

I think that all of these debates are silly. The correct answer is that there is no correct answer. All of the features of Git are just tools to help you get your code the way you need it to be. If rebasing saves the day, then rebasing is awesome. If sending patches by email works for you, then that’s also awesome. Whatever process you come up with for your team to work together on the same code is fantastic, if it works for you.

Even if you decided to follow a subversion style of development, you can absolutely just do it with Git instead. Git can work exactly the way SVN works, if you wish it to. The best part is that you can use it exactly like SVN, but because it is Git, you can escape the usual SVN traps when they come up. Therefore, I see absolutely no reason for a new project to use SVN when you can simply use Git and have developers agree to work in an SVN flow.

And that is the beauty of Git, is that it lets you come up with your own process. How you work is no longer dictated by the limitations of the software you are using. It is decided by the developers themselves. And even if developers like to work differently, as many do, they can each work in a way that is comfortable for them without interfering with anyone else. As long as the team communicates with each other, and agrees on how to share code, all is well.

Just to drive the point home, Git was made by Linus Torvalds to use for developing the Linux Kernel. How do you submit a patch to the kernel using Git? You put your patch up in your Git repository, and you email someone requesting them to pull the patch from you. Email!? With SVN you just commit, no need to talk to anybody. I don’t know about you, but do you think that forcing code changes on other developers without having to talk about it a good idea?

If you actually bother to talk to people then 500+ forks is not a problem. You ask someone and they tell you what fork to clone. Just ignore all those graphs, they mean nothing.

However, I must admit that the 500+ forks is indeed a symptom not of Git itself, but github’s design, and users not understanding Git. Github makes the fork button very prominent, but forking a project on Github is not really something that people should do that often. Github is not Git. The design of Github, intentionally or unintentionally, leads users into doing something they don’t actually want to do. It doesn’t really help people understand Git, and it doesn’t help people communicate.

What most people should be doing is just cloning repository to their local machines, and making changes there. If they want to submit the changes, then they should use non-Git communication tools to talk to the person they cloned from. They only need to fork on github if they want to publicly share changes that have not yet been, or will not be, merged into the actual project. Otherwise, they should not fork, just clone.

If you do have changes that are not accepted, then sharing them publicly on github in a fork is a very useful thing. Maybe there is someone out there who needs a copy of djata with a slight tweak, but someone else made the tweak already. The tweak may never be accepted to the canonical codebase, but it can still be made available in a form. That might save someone the time of making that same tweak themselves on their own, which is what they would likely be doing on a project managed by subversion.

Right now I see people with SVN mindsets who won’t switch to Git because they don’t truly understand DVCS. Likewise there are people with SVN mindsets who don’t understand DVCS who switch to Git anyway for whatever reason. The missteps of the latter group provide many of the stories that will keep the former group scared away.

All in all, I’m not too worried. When I was in college, less than a decade ago, the CS department taught us RCS and the software engineering department taught us CVS. In my senior years some people starting using SVN, which was bleeding edge at the time. Git was released in 2005. The world of version control moves surprisingly rapidly. I think Git’s dominance is pretty much guaranteed at this point. It will just take time for programmers to escape the SVN mindset and understand the, admittedly more complicated, DVCS world.

We Git users are here to help, so ask before you run away or jump to conclusions.

HTML5 is on the move. Pretty much every browser other than Internet Explorer is supporting it pretty well. However, the codec issue on the video tag is going to be a major headache for Firefox. YouTube, which is really the only video site that matters, has decided to go with H.264. Firefox can’t include H.264 because of patents, and can only support Ogg Theora. Chrome and Safari do not have this problem. IE will also not have this problem, if and when they get their act together.

This is due to the balance of open source that people like RMS do not understand. If you demand everything be purely open source, you will be missing out on something. The same goes if you demand everything be purely closed source. Some applications only exist closed, and some only exist open. You will have to use both to get an optimal user experience, especially on the desktop.

The overwhelming majority of users do not care about whether something is open or not. They only care that it is free as in beer, and that it has all the features they desire. Regardless of speed benefits, Chrome will beat Firefox because of its willingness to not be 100% pure open source, and include things like the patented H.264 video codec.

For those of you who do not know, Ubuntu has a very strict release cycle. Every six months they release a new version of Ubuntu. The most recent release is 9.10, released in October 2009. It includes version 2.6.31 of the Linux kernel. If there is a bug or security patch to that kernel, Ubuntu will push out an update to the users, such as the current version 2.6.31-17. The thing is, the current version of the Linux kernel is actually 2.6.32.5. Ubuntu 9.10 users will never get version 2.6.32. In order to get a newer version of the kernel, they will have to build it by hand, or wait for Ubuntu 10.4 in April.

What’s the big deal? The kernel version barely makes a difference in the user experience. Most users will probably not notice or care, even if they are stuck with a very old kernel version. Well, what if you apply the same update policy to something like Firefox? Firefox 3.6 was just released, but Ubuntu users will not get that update. Windows and Mac users will, because Firefox updates itself on those platforms. Ubuntu users will have to wait for Ubuntu 10.4, or jump through hoops to update Firefox by hand. They have to wait 3 months for a very significant user experience update.

The problem with these two open source projects, and others, is not that they are open source. Open source is awesome, and it’s the only reason these projects can even compete in the first place. The problem is that unlike say, the Apache web server, these target every day users. However, their philosophies and policies do not make the experience of the user their number one priority.

Yes, there are good reasons behind these policies. I support stable software releases and being as open as possible. It’s simply that the user experience is even more important than either of those. You need to make exceptions to policy when the user is suffering. If a store didn’t make a reasonable exception to its return policy, you would give it a bad review and maybe even stop shopping there. Why can’t open source projects also make reasonable exceptions? If they don’t, they’ll just slowly lose ground to their competitors.

I remember when I was a kid, and landline phones were still huge. Sprint made a big deal about their ten cents a minute.

The number of cents you paid per minute was a huge deal. I remember my grandmother would dial a five digit number before making a long distance call in order to get the price down to five cents per minute. With truphone you can now make many international calls for under three cents per minute. Taking inflation into account, voice calls have gotten ultra cheap over the years.

Now, I’m someone who doesn’t talk on the phone much at all. I still need a phone number because it’s sort of necessary for engaging with society. I also do call family and relatives, and occasionally friends when necessary. But most of my calls are very very short.

So what am I paying for this voice service I hardly use? I have AT&T’s “Nation 450 Rollover & 5000 Night/Weekend & Unlimited Mobile-To-Mobile Minutes” plan. It costs me $39.99 a month. There is no smaller plan available. If you do the math, this comes out to under ten cents per minute. If I’m paying for just the 450 minutes alone, each minute is under 9 cents. Good deal right?

It would be a good deal if I talked on the phone. As it stands, 4135 rollover minutes available. Oh they’re so generous letting me rollover the minutes, right? Wrong. The rollover is just an excuse. It makes you feel less bad about paying for all these minutes you never use. The thing is, if you never use them, you’re paying lots of money for absolutely nothing.

On my last bill I used 20 of the 450 minutes, 4 unlimited M2M minutes, and 64 night & weekend minutes. Even if I were paying a ludicrously high price of 25 cents per minute, for all the minutes, my bill would have been $22. Instead, I’m paying 45 cents per minute.

The way they structure the plans is highly deceptive. You don’t think about your voice costs in terms of cents per minute anymore, because you are paying a fixed rate. But when you really think about it, you’re either paying for something you don’t use, or you’re paying way too much for the little that you do use.

Now, for someone who talks on the phone a lot, this probably isn’t a problem. If I actually used all those minutes, I would actually be getting a pretty good deal. It wouldn’t be an amazing deal, but certainly not a rip-off.

The other carriers are no better. T-Mobile and Verizon have similar minimum plans of 450-500 minutes for $40. Sprint has a voice plan of 200 minutes for $30. I think somewhere in the fine print there’s the option to not pre-pay for any minutes, and pay only for the minutes you actually use. Of course, they charge 45 cents for those minutes, making it moot.

There is an argument to be made that wireless minutes should be more expensive than land line minutes. That’s acceptable, but it’s not true. Remember, if I used all my minutes, I’d be paying a reasonable price for them. The problem is that the minimum tier is much too high.

I’m willing to bet if you looked at the books for any of these wireless companies you would see millions upon millions of dollars paid for unused minutes. I’m paying as much money for unused minutes as I am for unlimited data, which I use a lot. It’s about 30% of my phone bill, and I wouldn’t be surprised if it was 20-30% of their revenues.

Yes, text messages are a rip-off. Yes, data plans are secretly and evilly limited. But most of the money people like me are losing is paying for minutes we never use. Over the course of my two year iPhone contract, I’ve probably paid over $700 for unused voice minutes.

It’s nice that we’re actually seeing a little price war for the unlimited plans. These guys are actually starting to cut some of their ludicrously high margins on the high-end in order to push high-end smart phones to more customers. That’s well and good, but let’s also see some competition for the lower end plans. I want to see unlimited data and text for $30-40 with 100 voice minutes for $10-20. Even those prices are a little high, but they are far more reasonable than what I’m forced to pay now.

Maybe soon I’ll be able to ditch voice service altogether and use a VOIP solution. Perhaps the reason they don’t improve their data networks and expand coverage more quickly is to prevent me from doing that very thing.

Mark Maunder suggests we crowdsource the terrorism fight. He’s on the right track, fundamentally speaking. Our government is clearly incapable of intelligently combating a non-military threat. It must be the citizens themselves who help to stop terrorism. However, Mark’s specific suggestions are not all that great. None of them would stop a terrorist with a working explosive device. The people were only saved by the stupidity of the terrorist. There was nothing they could have done to save themselves from an intelligent attacker.

And that is the one thing that gives me hope. So many of the terrorists are just complete morons. Terrorism isn’t really rocket science. Just about any reasonably good engineer could cause massive amounts of destruction. I can personally come up with any number of plans to cause a very large amount of destruction. The reason we are save is because these smart people are smart enough to have jobs, and money, and such. When you’re smart enough to be a good terrorist, you’re smart enough not to be a terrorist in the first place.

Despite this, we can not rest easily. Even morons who fail most of the time are dangerous. They did succeed in 2001, in the London underground, in Oklahoma City, and many other times throughout history. They will fail often due to their lack of brains, but they will succeed sometimes, and it is those sometimes we need to prevent.

Start thinking about ways to prevent terrorist attacks. What do you think of? You think of physical defenses. Locking doors, searching people and bags, detecting explosives and weapons, these are all physical defenses. They all try to stop an attack right before, or as, it is about to happen. They also don’t work. If someone has a bomb at an airport, they can just set it off in the security line. Heck, trains have almost no security whatsoever. As long as you have a situation where there are many people together, it is next to impossible to physically prevent a terrorist attack. Even if everyone is on the lookout, and the crowd reports suspicious activities, that isn’t going to stop anything.

What you need to do is prevent terrorism way in advance. Intelligence is absolutely the most effective weapon against terrorism. If you know that a terrorist is going to go on a shooting spree tomorrow, you can have police arrest him today. If you know that someone bought a whole bunch of fertilizer, in the winter, and they aren’t a farmer, and it’s for a crop that doesn’t grow in the area, that they are probably up to something.

You might think this information is difficult and expensive to collect, parse, and act upon. You’re right, it is. But this is exactly where the crowdsourcing comes in. We need to crowdsource the intelligence to pre-empt terrorism. Let ordinary citizens keep a watchful eye about them. Then we can pool, share, and filter to turn the information into intelligence. Then the authorities can use that to pre-empt villainy.

This kind of crowdsourcing can work, but you can already see it’s a minefield. Untrained citizens are not that great at collecting intelligence. He who has a hammer sees nails everywhere. If you’re on the lookout for terrorism all the time, suddenly everything looks like terrorism. Is that a homeless guy, or is it a disguise? Is he hiding a bomb in his shopping cart? Is that guy really a utilities worker, or is he putting a bomb in the subway? What’s that weird device he has? That computer guy’s screen is weird, is he hacking?

Is there another kind of crowdsourcing we can do that is actually useful, and doesn’t require as much training? I think there is. We can find terrorists while sitting on our asses, on the Internet. For real.

I assume people reading this are fairly technologically competent. If you wanted to get illegal things online, like pirated media, guns, drugs, child porn, etc. do you think you could do it? It’s quite trivial. Yet, law enforcement agencies can’t seem to figure it out. If my only job was to find illegal activity on the Internet, I could have people arrested faster than they could be processed.

Do terrorists use the Internet just like pirates and pedophiles? You bet they do! Our recent terrorist Mutallab posted all over the Internet. He wasn’t particularly secretive either. He openly displayed his unhappiness and fundamentalist beliefs. It was a textbook case of a terrorist to be, right out in the open where anyone could see it. The same sort of thing goes not just for terrorists, but school shootings, and other criminals. A great many of these people posted their craziness all over the web before they acted.

These terrorists are not ultra smart super villains like Doctor Doom. They’re mostly just really sad individuals who reach the breaking point. They post their plans openly. They’re not hatching an evil scheme in secret, they’re crying for help. By searching Google, newsgroups, IRC, and Facebook we can find the next Mutallab or Seung-Hui Cho before they strike. The authorities can’t even figure out BitTorrent, but we can. We can find these sad people on the ‘net and then do something to make sure they never fall so far as to actually try something.

You’re probably sitting at your computer right now, and you’ve had enough free time to read this blog post. You probably also have enough free time to do some searching. Go out there, and see if you can find people on the web who are actually likely to be real terrorists in the making. Maybe you can save lives without getting out of bed.