Django+nginx+uwsgi+supervisor

Because I've heard so many people talking about switching from Apache to this combination, I thought I would give it a try. After scouring the Internet and reading some blog posts, I came up with a setup, which I'm documenting here, because there were some gotchas for me.

I use Ubuntu 11.04, so this is the list of required packages:
• python-pip
• python-virtualenv
• postgresql
• postgresql-dev
• python2.7-dev (uwsgi requirement)
• libxml2-dev (uwsgi requirement)
• libpcre3 (nginx requirement)
• libpcre3-dev (nginx requirement)
• nginx

I install nginx with apt-get to have the necessary directory created in /etc/, but I'll replace the binary with the one I get after building the source of the latest version:

wget http://nginx.org/download/nginx-1.0.11.tar.gz
tar -xzf nginx-1.0.11.tar.gz
cd nginx-1.0.11/
./configure --conf-path=/etc/nginx/nginx.conf\
--sbin-path=/usr/sbin\
--without-http-cache
make
make install

The configuration options are so few because this is what I need in development, but for production more should be added.

Next, I just use django-startproject, to create a Django project and start working. My fork of this application does some extra things. First, it assumes very similar development and production environments, so the same stack is also used for development (nginx, uwsgi, supervisor, postgresql). Second, it automates a number of tasks done at the start of each project or on a new deployment. A typical work-flow is like this:
1. Install django-startproject:
    pip install -e git+https://github.com/LucianU/django-startproject#egg=django-startproject

2. Go to the location where you want to create your new project and run:
    django-startproject.py projectname
    At this point, you are prompted for some values and then the setup is done. During the setup, all the placeholders are replaced in the project template with the values corresponding to the current machine and the ones provided at the prompt. Afterwards, a virtual env is created and the packages used for development are installed: Django, psycopg2, supervisor, uwsgi, django-debug-toolbar. I prefer installing uwsgi and supervisor in a virtual env and not globally.

3. Create a database and put your credentials in the project's settings/local.py file.

4. Activate your environment and run django-admin.py syncdb

5. Symlink project/confs/development/nginx.conf into the 'sites-enabled' directory of your nginx installation and restart the server

If there weren't any problems, you should be done and running supervisord should start uwsgi, and you should be able to access the Django greeting page.

django-startproject also offers a script that can be used for deployment. For example, if someone wants to start working on a project created with django-startproject, they just clone the repository, run bin/bootstrap.py dev, and then follow the quickstart from step 3 onwards.

Last, I want to mention some things about nginx's configuration system because this was unclear to me at the start. nginx has a global configuration file (located at /etc/nginx/nginx.conf in case of Ubuntu) and two directories, sites-available and sites-enabled. Generally, in nginx.conf are found settings applying to the server as a whole, like used modules, compression, and so on. On the other hand, the configuration files placed in sites-enabled are specific to the served app, containing the domain name and the location of application and its logs.

Setting the terminal default editing mode

I had the unexpected surprise after I installed 64-bit Ubuntu, to discover that the terminal wasn't set to Emacs editing mode by default. It's not that I'm a big lover of Emacs (I use Vim), but I've gotten used to the Emacs key bindings in the terminal. After some searching in ~/.bashrc and /etc/bash.bashrc, I found that the "problem" came from ~/.inputrc. The first line of that file was 'set editing-mode vi', so I commented it and all was well again.

How to setup wireless for Inspiron N3010 in Ubuntu 11.04

It took me a day and the help of allandee from #ubuntu to be able to setup wireless on my laptop, so I decided to write down the steps and put them here in case I or someone else needs them.

First, the laptop comes with Broadcom Corporation BCM4313 802.11b/g/n Wireless LAN Controller (rev 01), so the instructions are for it.

Here are the steps:

 1. sudo apt-get install git (in case it's not installed)
 2. git clone git://git.bu3sch.de/b43-tools.git
 3. cd b43-tools
 4. make #build the b43-fwcutter binary

 5. cd
 6. mkdir tmp #create a temporary dir for the driver
 7. cd tmp
 8. wget http://mirror2.openwrt.org/sources/broadcom-wl-5.10.56.27.3_mipsel.tar.bz2
 9. tar -xjf broadcom-wl-5.10.56.27.3_mipsel.tar.bz2 #extract the archive
 10. sudo ~/b43-tools/b43-fwcutter -w /libs/firmware/ 
~/tmp/broadcom-wl-5.10.56.27.3_mipsel/driver/wl_apsta/wl_prebuilt.o #extract the firmware
 #the line above is wrapped to properly show up on screen 
 11. modprob b43
 a restart might be needed afterwards.
 

This is it.

Geospatial Applications

I've started learning about the topic, so I'm getting familiar with PostGIS, OpenLayers, Mapnik, and GIS in general. I want to make some blog posts detailing the steps I've gone through when rendering my own tiles for example.
Anyway, I find the topic fascinating because it allows you to visually link statistics with locations on the globe, which I think is very cool. To be continued...

How to upgrade the SQLite that comes with Python

If you're using Python with SQLite and you want to be able to read and write to the database concurrently, you need to enable WAL. The problem is that only SQLite versions 3.7.x support this new functionality, and Python comes with older versions (at least Python 2.6.x does).

So, you need to upgrade SQLite, but fortunately this is surprisingly easy. If you're on Windows, like I am, you should go to the SQLite download page and download the sqlite.dll. Then, you simply take that dll, and put it in Your_Python_Installation\DLLs, where Your_Python_Installation is the folder where Python is installed, in my case Python26.

After you've done that, you can simply do

PRAGMA journal_mode=wal

and that should activate WAL. To be sure, check the return of value of curs.fetchone() which should be (u'wal',).

Installing oursql on Windows

I had some difficulties installing oursql on Windows, so I thought I'd write down the steps, in case I need to do it again in the future or anyone else has the problems.

First, I tried installing with pip, but that gave me some problems (something about a missing file) even after I installed Cython.

On second try, I noticed that there are already some binaries available, so I tried with those. The archive which can be downloaded from here http://www.habnabit.org/software/oursql/ contains an installer which in turn contains an egg and a .pyd file. Those go in the site-packages directory. I'm saying this in case anyone wants to install oursql in a virtualenv, which is what I wanted to do and I couldn't because the installer only detected my main site-packages and wouldn't let me choose.

I put those in their place, but when trying to import oursql, I got a "ImportError: DLL load failed...". Apparently, I needed the MySQL C Connector. I installed that, but I still got the error, which I solved by copying libmySQL.dll from my WAMP package into site-packages(supposedly, this isn't necessary when MySQL has been installed directly).

Ok, to recap. The steps were:
1) Download an archive with the binaries from here http://www.habnabit.org/software/oursql/
2) Extract the archive and also the installer.
3) Put the egg and the .pyd in the site-packages.
4) Install MySQL C Connector
5) Copy libmySQL.dll into site-packages.
6) Done.

Update

I thought I'd come back and write something here because I haven't done so in a month, I think. One of the things I did in the mean time was starting to learn assembly. My main goal is to cover my bases, so to speak, and learn the fundamentals of computers and computer science, and I thought that learning assembly would provide that. I haven't started writing code because the book that I'm learning from (Assembly Language Step-by-Step: Programming with Linux) is big, but I'm close to that point and I'm looking forward to it.

So far I like the book. It starts really slow, which is a bit annoying but to be expected considering it's for people with assembly as their first programming language (I wonder how many people start with assembly), but then goes into a lot of detail and careful explanation in order to make you understand what's going on. It has analogies and a lot of drawings which are so useful for visual learners. That's all I can say for now. I'll add to this once I finish the book.

Oh, and here's something inspiring. A guy who wrote his own compiler, editor, OS and a few others.

How to properly install Node.js and npm

I'm documenting here the proper way to install node.js in case I might need to do it again. I'm using the word "proper" because the first time I installed node.js, I did it system wide and that
made it very difficult for me to install npm, the node.js package manager. So, without further ado:

Installing Node.js

git clone git://github.com/ry/node.git
cd node
./configure --prefix=$HOME/my_node
make install

Open $HOME/.profile and add:

export PATH=$PATH:$HOME/my_node/bin

Then run:

source $HOME/.profile

Installing npm
Simply do:

curl http://npmjs.org/install.sh | sh

Scraping

I have been doing some scraping jobs lately, mostly for fun because none has brought me money yet. Now, since Scrapy 0.10 came out, I'm planing on integrating Django with Scrapy. Up until now it wasn't that easy, but 0.10 introduces a Scrapy daemon, persistent queues and other stuff that makes it easier to schedule scraping jobs remotely.

This project will also give me the opportunity to learn jQuery because I want to use Ajax for the user input and the display of the scraping results. I imagine the site will look somewhat like a search engine, the difference being that the crawling will be done in real time.

When I finish it, I'll put it on Bitbucket for anyone interested.

Namespace vs scope

I'm on a row today because I've become aware of some new things. Besides lambdas, I've also been trying to understand scope and how it works in Python. As far as I can tell, the rule is as follows: names in the global scope are available in a local scope as long as they aren't "shadowed" by a local name, but local names aren't available from the global scope.

There is an exception to this. A non-local name can be accessed from a local scope by using the 'global' or the 'nonlocal' (available only in Python 3.x) statements. The 'global' statement allows access to names in the global scope, while the 'nonlocal' statement allows access to names in an enclosing non-global scope. However, in both access, these names can only be accessed for the purpose of binding the name to a new object, and this must be done before 'shadowing' a global name.

Here is some code exemplifying all this gibberish:

x = 5

# you can access a global name from a local scope
def foo():
    def bar():
        print x
    return bar

f = foo()
f() # prints '5'

# binding a name to an object makes it local to the current scope without affecting an identical name in the global scope
def foo():
    x = 10
    print x

foo() # prints '10'
print x # prints '5'

# a name bound in a local scope can't be accessed from an enclosing scope
def foo():
    def bar():
        y = 1
    print y

foo() # raises NameError: the global name 'y' is not defined

# using 'global' we can access a name from the global scope in a local scope and rebind it to a different object

def foo():
   global x 
   x = 10
   print x

foo() # prints '10'
print x # prints '10'; it was '5' before

# using 'nonlocal' we can access a name from an enclosing non-global scope in the local scope and rebind it to a different object (only in Python 3.x)

def foo():
    y = 0
    def bar():
        nonlocal y
        y = 1
    bar()
    print y # prints '1'

Now, what about namespace vs scope? The difference between the two is that a namespace is a scope which can be accessed from a unique identifier. In Python we have module, class, and function namespaces, but names from a function's namespace can't be accessed the same way they can be accessed in the case of a module or a class, namely 'namespace.name'.

Lambdas

I've just realized that up until now I had made the wrong assumption that a lambda expression is evaluated and also executed at its definition. That's practically impossible if it takes arguments, but that didn't occur to me until now.

I think that the reason for this misconception is that I have mostly seen lambda used in situations where a value was needed. But that doesn't sound right either. I think I was confused, really confused.

I'm glad that changed.

Decorators Revisited

I was saying the other day that I've finally understood decorators after reading that response from Stack Overflow, but there was still something I was unsure of. I was wondering if you can make class decorators, not just function decorators, and today I found out you can. I did a little bit of experimenting and I came up with this:

>>> class Foo(object):
        def __new__(cls, kls):
            class Bar(object):
                def __init__(self, length, width, height):
                    try:
                        self.length = kls.length
                    except AttributeError:
                       self.length = length
                    try:
                       self.width = kls.width
                    except AttributeError:
                       self.width = width
                    self.heigth = height
            return Bar


>>> @Foo
class Buzz(object):
    def __init__(self, length, width):
        self.length = length
        self.width = width


>>> Buzz(1, 2, 3)
<__main__.Bar object at 0x00BD5390>
>>> b = Buzz(1, 2, 3)
>>> b.length
1
>>> b.width
2
>>> b.height
3

As you can see, 'b' is an instance of Bar, but that can be fixed by decorating Bar with the wraps decorator from the functools module.

Hmm, I just realized I'm turning class attributes into instance attributes with this code which is not something someone usually wants. I also realized that functools.wraps needs to be passed some extra arguments to work on classes, because the __doc__ attribute is read only and dictproxy objects don't have an update method.

This causes something which might become a problem: the wrapped object's dict contains the wrapper's class attributes (the ones used to preserve the default values).

Anyway, here's a different version that keeps instance attributes the way they are and also preserves their default values:

>>> class Foo(object):
        def __new__(self, kls):
        @functools.wraps(kls, assigned=('__module__', '__name__'), updated=())
        class Bar(object):
            try:
                length = kls().length
            except AttributeError:
                length = 0
            try:
                width = kls().width
            except AttributeError:
                width = 0
            def __init__(self, height=0, length=length, width=width):
                self.length = length
                self.width = width
                self.height = height
        return Bar

>>> @Foo
class Derp(object):
    def __init__(self, length=8, width=9):
        self.length = length
        self.width = width


>>> Derp.__name__
'Derp'

It's not very pretty, and I can't think of a use case right now, but it's interesting as an experiment.

Python Decorators

Thanks to this answer on Stack Overflow, I've finally understood decorators. Up until now I understood the syntax, I understood what decorators did, but it was only after I hand coded all the examples in that post that I really "got" what and how decorators work.

I understand now that a decorator simply takes a function and wraps it or its return value in another function which it returns afterwards. I guess this is useful if you have a bunch of different functions that you want to have some common functionality. Instead of writing the same code in each function, you can write a decorator and decorate each of the functions.

Next, I'll just have to use a decorator in a real-life situation, so that I can remember it better.

Hello, Internet!

I've decided to start a blog in which I will document my experience in learning programming. When you saw the title, you probably thought of Lambda the Ultimate. That's what I thought as well, but since I'm pretty much a beginner, I thought I'd call my blog Lambda the First, although I know realize that that might imply I consider myself the first. Well, not yet.

So far, I'm familiar with Python and JavaScript, so I will mostly discuss about these. I have also used Scrapy (a web crawling framework written in Python) for the last couple of months, so I'll be mentioning it too.

Next, I'm planning to learn to use Twisted, so you can imagine that it will drip in here. I actually intend to create a mind map to help me learn better, but I'll see how that goes because, from what I've seen, Twisted is huge. Maybe several mind maps, each focusing on a specific module, will be more appropriate.

Creating mind maps is one of my hobbies, so I'll be doing it a lot. I already have a few for Python, Scrapy, JavaScript, Django, which I will publish shortly.

Ok, that's about it for my first post. I'm curious to see how this will evolve.