How to upgrade the SQLite that comes with Python

If you're using Python with SQLite and you want to be able to read and write to the database concurrently, you need to enable WAL. The problem is that only SQLite versions 3.7.x support this new functionality, and Python comes with older versions (at least Python 2.6.x does).

So, you need to upgrade SQLite, but fortunately this is surprisingly easy. If you're on Windows, like I am, you should go to the SQLite download page and download the sqlite.dll. Then, you simply take that dll, and put it in Your_Python_Installation\DLLs, where Your_Python_Installation is the folder where Python is installed, in my case Python26.

After you've done that, you can simply do

PRAGMA journal_mode=wal

and that should activate WAL. To be sure, check the return of value of curs.fetchone() which should be (u'wal',).

Installing oursql on Windows

I had some difficulties installing oursql on Windows, so I thought I'd write down the steps, in case I need to do it again in the future or anyone else has the problems.

First, I tried installing with pip, but that gave me some problems (something about a missing file) even after I installed Cython.

On second try, I noticed that there are already some binaries available, so I tried with those. The archive which can be downloaded from here http://www.habnabit.org/software/oursql/ contains an installer which in turn contains an egg and a .pyd file. Those go in the site-packages directory. I'm saying this in case anyone wants to install oursql in a virtualenv, which is what I wanted to do and I couldn't because the installer only detected my main site-packages and wouldn't let me choose.

I put those in their place, but when trying to import oursql, I got a "ImportError: DLL load failed...". Apparently, I needed the MySQL C Connector. I installed that, but I still got the error, which I solved by copying libmySQL.dll from my WAMP package into site-packages(supposedly, this isn't necessary when MySQL has been installed directly).

Ok, to recap. The steps were:
1) Download an archive with the binaries from here http://www.habnabit.org/software/oursql/
2) Extract the archive and also the installer.
3) Put the egg and the .pyd in the site-packages.
4) Install MySQL C Connector
5) Copy libmySQL.dll into site-packages.
6) Done.

Update

I thought I'd come back and write something here because I haven't done so in a month, I think. One of the things I did in the mean time was starting to learn assembly. My main goal is to cover my bases, so to speak, and learn the fundamentals of computers and computer science, and I thought that learning assembly would provide that. I haven't started writing code because the book that I'm learning from (Assembly Language Step-by-Step: Programming with Linux) is big, but I'm close to that point and I'm looking forward to it.

So far I like the book. It starts really slow, which is a bit annoying but to be expected considering it's for people with assembly as their first programming language (I wonder how many people start with assembly), but then goes into a lot of detail and careful explanation in order to make you understand what's going on. It has analogies and a lot of drawings which are so useful for visual learners. That's all I can say for now. I'll add to this once I finish the book.

Oh, and here's something inspiring. A guy who wrote his own compiler, editor, OS and a few others.

How to properly install Node.js and npm

I'm documenting here the proper way to install node.js in case I might need to do it again. I'm using the word "proper" because the first time I installed node.js, I did it system wide and that
made it very difficult for me to install npm, the node.js package manager. So, without further ado:

Installing Node.js

git clone git://github.com/ry/node.git
cd node
./configure --prefix=$HOME/my_node
make install

Open $HOME/.profile and add:

export PATH=$PATH:$HOME/my_node/bin

Then run:

source $HOME/.profile

Installing npm
Simply do:

curl http://npmjs.org/install.sh | sh

Scraping

I have been doing some scraping jobs lately, mostly for fun because none has brought me money yet. Now, since Scrapy 0.10 came out, I'm planing on integrating Django with Scrapy. Up until now it wasn't that easy, but 0.10 introduces a Scrapy daemon, persistent queues and other stuff that makes it easier to schedule scraping jobs remotely.

This project will also give me the opportunity to learn jQuery because I want to use Ajax for the user input and the display of the scraping results. I imagine the site will look somewhat like a search engine, the difference being that the crawling will be done in real time.

When I finish it, I'll put it on Bitbucket for anyone interested.

Namespace vs scope

I'm on a row today because I've become aware of some new things. Besides lambdas, I've also been trying to understand scope and how it works in Python. As far as I can tell, the rule is as follows: names in the global scope are available in a local scope as long as they aren't "shadowed" by a local name, but local names aren't available from the global scope.

There is an exception to this. A non-local name can be accessed from a local scope by using the 'global' or the 'nonlocal' (available only in Python 3.x) statements. The 'global' statement allows access to names in the global scope, while the 'nonlocal' statement allows access to names in an enclosing non-global scope. However, in both access, these names can only be accessed for the purpose of binding the name to a new object, and this must be done before 'shadowing' a global name.

Here is some code exemplifying all this gibberish:

x = 5

# you can access a global name from a local scope
def foo():
    def bar():
        print x
    return bar

f = foo()
f() # prints '5'

# binding a name to an object makes it local to the current scope without affecting an identical name in the global scope
def foo():
    x = 10
    print x

foo() # prints '10'
print x # prints '5'

# a name bound in a local scope can't be accessed from an enclosing scope
def foo():
    def bar():
        y = 1
    print y

foo() # raises NameError: the global name 'y' is not defined

# using 'global' we can access a name from the global scope in a local scope and rebind it to a different object

def foo():
   global x 
   x = 10
   print x

foo() # prints '10'
print x # prints '10'; it was '5' before

# using 'nonlocal' we can access a name from an enclosing non-global scope in the local scope and rebind it to a different object (only in Python 3.x)

def foo():
    y = 0
    def bar():
        nonlocal y
        y = 1
    bar()
    print y # prints '1'

Now, what about namespace vs scope? The difference between the two is that a namespace is a scope which can be accessed from a unique identifier. In Python we have module, class, and function namespaces, but names from a function's namespace can't be accessed the same way they can be accessed in the case of a module or a class, namely 'namespace.name'.

Lambdas

I've just realized that up until now I had made the wrong assumption that a lambda expression is evaluated and also executed at its definition. That's practically impossible if it takes arguments, but that didn't occur to me until now.

I think that the reason for this misconception is that I have mostly seen lambda used in situations where a value was needed. But that doesn't sound right either. I think I was confused, really confused.

I'm glad that changed.