Blog | Caffeine Fueled Dreams

PyCon Ireland Rewrite Talk 22:00 Sunday the 22nd of October 2017 0 comments
Slide deck (including speaker notes) for my talk on Software Rewrites at PyCon Ireland today can be found here.
Adding Jobs to MRQ from code 20:47 Friday the 1st of September 2017 0 comments
The MRQ documention is vague on how to add jobs to a queue from code. Particularly if you want to do it from a remote system (one the worker is not running on).

The following snippet does the job
```
from mrq.job import queue_job, context

context.setup_context(
 extra={
 'redis': 'redis://my_redis_url',
 'mongodb_jobs': 'mongodb://my_mongo_url',
 })

queue_job("tasks.MyTask", {'arg1': "myarg"}, queue="myqueue")
```

A little Django troubleshooting 20:10 Thursday the 24th of August 2017 0 comments

We hit an annoying apache problem this month. When attempting to serve a Django view on the root of the site, Django was attempting to match '/index.html' rather than /.

Looking at the uwsgi logs sent us down the wrong path, as the log files looked like:

[pid: 10093|app: 0|req: 1/1] 172.16.0.1 () {42 vars in 797 bytes} [Wed Jul 19 17:25:12 2017] GET / => generated 86199 bytes in 258 msecs (HTTP/1.1 200) 1 headers in 63 bytes (1 switches on core 0)

Leading us to believe that Django was recieving a request for /, and so the problem must be on the Django layer.

Raising an exception in the view with debug mode turned on set us back on the right track. Looking at the Meta information for the request showed us that REQUEST_URI was set to "/", but PATH_INFO was set to "/index.html".

At the time, this left us confused, as we where pretty sure that apache and uwsgi where not responsible. We decided to verify using tshark.

tshark -V -i lo -f "dst port 3031"

-V has tshark print packet details
-i lo says to watch the local loopback device (might be different for you)
-f "dst port 301" says only way traffic going to port 3031 (which we where running uwsgi on).

tshark does a good job of showing you the content of the request. This part:

0000  00 1d 03 00 0e 00 52 45 51 55 45 53 54 5f 4d 45   ......REQUEST_ME
0010  54 48 4f 44 03 00 47 45 54 0c 00 51 55 45 52 59   THOD..GET..QUERY
0020  5f 53 54 52 49 4e 47 00 00 0b 00 53 45 52 56 45   _STRING....SERVE
0030  52 5f 4e 41 4d 45 1b 00 70 6f 72 74 61 6c 2e 76   R_NAME..portal.v
0040  61 67 72 61 6e 74 2d 72 72 69 2d 6c 61 62 73 2e   agrant-rri-labs.
0050  63 6f 6d 0b 00 53 45 52 56 45 52 5f 50 4f 52 54   com..SERVER_PORT
0060  02 00 38 30 0f 00 53 45 52 56 45 52 5f 50 52 4f   ..80..SERVER_PRO
0070  54 4f 43 4f 4c 08 00 48 54 54 50 2f 31 2e 31 0b   TOCOL..HTTP/1.1.
0080  00 52 45 51 55 45 53 54 5f 55 52 49 01 00 2f 0b   .REQUEST_URI../.
0090  00 52 45 4d 4f 54 45 5f 41 44 44 52 0a 00 31 37   .REMOTE_ADDR..17
00a0  32 2e 31 36 2e 30 2e 31 0b 00 52 45 4d 4f 54 45   2.16.0.1..REMOTE
00b0  5f 55 53 45 52 00 00 0d 00 44 4f 43 55 4d 45 4e   _USER....DOCUMEN
00c0  54 5f 52 4f 4f 54 0d 00 2f 76 61 72 2f 77 77 77   T_ROOT../var/www
00d0  2f 68 74 6d 6c 0b 00 53 43 52 49 50 54 5f 4e 41   /html..SCRIPT_NA
00e0  4d 45 00 00 09 00 50 41 54 48 5f 49 4e 46 4f 0b   ME....PATH_INFO.
00f0  00 2f 69 6e 64 65 78 2e 68 74 6d 6c 09 00 48 54   ./index.html..HT
0100  54 50 5f 48 4f 53 54 1b 00 70 6f 72 74 61 6c 2e   TP_HOST..portal.
0110  76 61 67 72 61 6e 74 2d 72 72 69 2d 6c 61 62 73   vagrant-rri-labs
0120  2e 63 6f 6d 0f 00 48 54 54 50 5f 43 4f 4e 4e 45   .com..HTTP_CONNE
0130  43 54 49 4f 4e 0a 00 6b 65 65 70 2d 61 6c 69 76   CTION..keep-aliv
0140  65 12 00 48 54 54 50 5f 43 41 43 48 45 5f 43 4f   e..HTTP_CACHE_CO
0150  4e 54 52 4f 4c 09 00 6d 61 78 2d 61 67 65 3d 30   NTROL..max-age=0
0160  1e 00 48 54 54 50 5f 55 50 47 52 41 44 45 5f 49   ..HTTP_UPGRADE_I
0170  4e 53 45 43 55 52 45 5f 52 45 51 55 45 53 54 53   NSECURE_REQUESTS
0180  01 00 31 0f 00 48 54 54 50 5f 55 53 45 52 5f 41   ..1..HTTP_USER_A
0190  47 45 4e 54 69 00 4d 6f 7a 69 6c 6c 61 2f 35 2e   GENTi.Mozilla/5.
01a0  30 20 28 58 31 31 3b 20 4c 69 6e 75 78 20 78 38   0 (X11; Linux x8
01b0  36 5f 36 34 29 20 41 70 70 6c 65 57 65 62 4b 69   6_64) AppleWebKi
01c0  74 2f 35 33 37 2e 33 36 20 28 4b 48 54 4d 4c 2c   t/537.36 (KHTML,
01d0  20 6c 69 6b 65 20 47 65 63 6b 6f 29 20 43 68 72    like Gecko) Chr
01e0  6f 6d 65 2f 35 39 2e 30 2e 33 30 37 31 2e 31 31   ome/59.0.3071.11
01f0  35 20 53 61 66 61 72 69 2f 35 33 37 2e 33 36 0b   5 Safari/537.36.
0200  00 48 54 54 50 5f 41 43 43 45 50 54 55 00 74 65   .HTTP_ACCEPTU.te
0210  78 74 2f 68 74 6d 6c 2c 61 70 70 6c 69 63 61 74   xt/html,applicat
0220  69 6f 6e 2f 78 68 74 6d 6c 2b 78 6d 6c 2c 61 70   ion/xhtml+xml,ap
0230  70 6c 69 63 61 74 69 6f 6e 2f 78 6d 6c 3b 71 3d   plication/xml;q=
0240  30 2e 39 2c 69 6d 61 67 65 2f 77 65 62 70 2c 69   0.9,image/webp,i
0250  6d 61 67 65 2f 61 70 6e 67 2c 2a 2f 2a 3b 71 3d   mage/apng,*/*;q=
0260  30 2e 38 08 00 48 54 54 50 5f 44 4e 54 01 00 31   0.8..HTTP_DNT..1
0270  14 00 48 54 54 50 5f 41 43 43 45 50 54 5f 45 4e   ..HTTP_ACCEPT_EN
0280  43 4f 44 49 4e 47 0d 00 67 7a 69 70 2c 20 64 65   CODING..gzip, de
0290  66 6c 61 74 65 14 00 48 54 54 50 5f 41 43 43 45   flate..HTTP_ACCE
02a0  50 54 5f 4c 41 4e 47 55 41 47 45 0e 00 65 6e 2d   PT_LANGUAGE..en-
02b0  55 53 2c 65 6e 2b 71 3d 30 2e 38 0b 00 48 54 54   US,en;q=0.8..HTT
02c0  50 5f 43 4f 4f 4b 49 45 57 00 52 52 53 50 53 45   P_COOKIEW.RRSPSE
02d0  53 53 49 44 2d 76 33 69 65 62 62 35 70 6b 6c 71   SSID=v3iecc5pklq
02e0  43 75 6a 32 66 35 64 39 36 77 38 71 59 6a 76 68   suj3f5c97w9p9jvi
02f0  33 33 31 35 6e 3b 20 63 73 72 66 74 6f 6b 65 6e   3305n; csrftoken
0300  3d 75 41 62 78 75 35 74 55 4d 52 6a 38 32 56 6d   =uAbxu5tTMRj82Vm
0310  47 76 65 71 35 6e 6f 50 30 52 35 4a 54 61 4c 42   Gveq5noP0R5JTaLB

contained the important detail:

00e0  4d 45 00 00 09 00 50 41 54 48 5f 49 4e 46 4f 0b   ME....PATH_INFO.
00f0  00 2f 69 6e 64 65 78 2e 68 74 6d 6c 09 00 48 54   ./index.html..HT

Showing that Apache was setting PATH_INFO to ./index.html

The culprit turned out to be Apache's dir module.

We dont need it, so we disabled it, you could also do the following in your apache config (for 2.4+)

<Location />
 DirectoryIndex disabled
 SetHandler uwsgi-handler
 uWSGISocket 127.0.0.1:3031
</Location>

Micropython on the WeMos D1 Mini 01:26 Tuesday the 1st of November 2016 0 comments
I've been playing with MicroPython lately, and more specifically on a WeMos D1 Mini . It's a really nice, cheap ($4 at the time I write this) ESP8266 based board, that can run MicroPython.

I have an original pyboard from the MicroPython Kickstarter, but never got around to doing much with it. Lately I've been fiddling around trying to make a Wifi to Infrared bridge, so I can control my TV from my computers/phones. Unfortunately the pyboard does not come with Wifi, and addons seem to be at least $25 and often far more. A D1 mini on the other hand is $4 and competely replaces the pyboard for an application like this.

Compared to the pyboard the D1 mini is slower (default clock speed is about half a pyboard, but it can be overclocked to come close), and if Wifi is running, you only have about 36Kb of memory available for your own application. It supports 2.4 GHz Wi-Fi (802.11 b/g/n, supporting WPA/WPA2) and has a built in antenna. It also has 4 Mbytes of Flash storage built in, compared to the pyboards 1 Mbyte.
If you are shopping for a D1 mini you will also come accross the D1 mini Pro. It comes with 16 MBytes of flash, is smaller and lighter than a D1 mini and has an external antenna connector as well as a built in antenna. The only catch right now is that Micropython only detects 1MB of storage. If you can live with that until Micropython supports it fully, its probably a better buy and only costs $1 more. (Check this issue to see if the full 16 MBytes has support yet).

If you want something with slightly better support and documentation, take a lok at the Adafruit Feather HUZZAH. At $15 its much more expensive than a D1 mini. But still a lot cheaper than a pyboard and an adaptor. Adafruit has extensive documentation and tutorials on their site. These are worth a read even if you do go with a D1 mini.

The D1 mini usually comes with Arduino or NodeMCU preinstalled. So to use it you have to either find prebuilt firmware, or build it yourself. I found several tutorials, but all either assumed a slightly different operating system, or skipped steps that caused me a lot of frustration. What follows are build and installation instructions assuming you are running Ubuntu 16.04 (Xenial).

Before we start

In order to connect to your D1 mini, you will need your user to be a member of the dialout group. Run the following command:
```
sudo addgroup $USER dialout
```
and then log out and log in again

Getting prebuilt firmware

If you are happy to use prebuilt firmware, you can find it here. You are looking for the ESP8266 section. Then skip to the Installation Instructions below.

Building the firmware on Ubuntu 16.04

First we need to make sure we have all the packages and libraries we will need installed:
```
sudo apt-get install gperf bison flex help2man libncurses5-dev make autoconf texinfo libtool libtool-bin g++ python unzip python-serial git screen make
```
Now clone the ESP SDK (All the instructions from here on in assume you start in your home directory, if you want to do it elsewhere, modify the commands to match)
```
cd ~
git clone --recursive https://github.com/pfalcon/esp-open-sdk.git
```
Now lets build the SDK, this took about 20 minutes on my laptop
```
cd esp-open-sdk/
make STANDALONE=y
```
Now the SDK is ready to use, lets put in on our path.
```
export PATH=$HOME/esp-open-sdk/xtensa-lx106-elf/bin:$PATH
```
You will need to run that command any time you open a new terminal and want to use the SDK.

Now we are ready to build Micropython, lets check it out from github.
```
cd ~
git clone https://github.com/micropython/micropython.git
cd micropython
git submodule update --init
```
And finally, build our firmware
```
make -C mpy-cross
cd esp8266/
make axtls
make
```
And now we should our firmware in a file called firmware-combined.bin in the build directory.

Installing Micropython on the D1 Mini

Find the port your board shows up on, if you dont know, the easiest way is to run
```
ls /dev/tty*
```
then plug in your MicroPython board and run
```
ls /dev/tty*
```
again. Compare the two lists and find the entry that appears after the board is plugged in.

Mine shows up as /dev/ttyUSB0

Before we install MicroPython, its best to erase the current contents of the flash drive.
```
esptool.py --port /dev/ttyUSB0 erase_flash
```
If this refuses to write, check your user is a member of the dialout group and if not add them as shown above.

Now we write our firmware file to the board.
```
cd $HOME/micropython/esp8266

esptool.py -p /dev/ttyUSB0 write_flash 0x0000000 ./build/firmware-combined.bin
```
Press the little reset button on the side of the board. The board will reboot, the blue LED on the side will blink briefly and MicroPython should be running.

Connecting to our micropython board.

We can use screen to connect to the board
```
screen /dev/ttyUSB0 115200
```
You should now see a python REPL!

Lets try something simple
```
>>> 1 + 1
2
>>>
```
Hurray, we have Python running on this tiny computer!

The D1 Mini has a blue LED built in, lets blink it!
```
>>> from machine import Pin
>>> p2 = Pin(2, Pin.OUT)
>>> p2.high()
>>> p2.low()
>>> p2.high()
>>> p2.low()
```
As you switch from high to low, you should see the LED turn on and off. But the real point of this board is the Wifi support, lets connect to a Wifi network.
```
>>> import network
>>> wifi = network.WLAN(network.STA_IF)
>>> wifi.active(True)
>>> wifi.connect('your-ssid', 'your-password')
```
Nothing too exciting here, but lets make a HTTP request
```
>>> import usocket as socket
>>> s = socket.socket()
>>> address = socket.getaddrinfo("google.com", 80)
>>> print("Address:", address)
>>> connect_address = address[0][-1]
>>> print("Connect address:", addr)
>>> s.connect(connect_address)
>>> s.send(b"GET / HTTP/1.0\n\n")
>>> while True:
>>>     data = s.recv(4096)
>>>     if data:
>>>         print(str(data, 'utf8'), end='')
>>>     else:
>>>         break
```
You should see the HTML for the google homepage come back. Congratulations! Your tiny computer is connected to the internet.

I'll try and follow this up with more details on the Wifi -> Infrared work I'm doing.

From Del.icio.us to Pinboard.in with Python 09:21 Sunday the 19th of December 2010 3 comments

The sad news that Yahoo plans to shut down del.icio.us reached me this week (although theres still hope). I use del.icio.us pretty much every day and was a little traumatized upon hearing this. Once I had finished wailing and gnashing my teeth I set out looking for somewhere to go.

There are many bookmarking sites/services out there, but I fear change, and pinboard.in seemed like the closest thing to a plain replacement. It even supports the same API as del.icio.us. Theres a small charge for signing up, but no recurring fee, so I broke out the credit card and joined up.

The next step was to figure out how to migrate my bookmarks. del.icio.us provides a export to HTML feature in its settings area, but a quick look at the export revealed some data was missing (mostly extended descriptions). Rabid googling revealed a lesser known XML export mechanism. To use it visit https://api.del.icio.us/v1/posts/all, enter your username and password and save the resulting XML file.

Now to get my bookmarks into pinboard.in. I broke out my trusty text editor and battered together the script below which works just fine, a few hours later all my bookmarks are in pinboard.in, their bookmarklets are installed in my browser, and I'm loving their read later features. Sean is a happy geek again.

You can download my migration script. To use it :

python delmigrate.py backup.xml username password

Heres the source for the curious.

from xml.dom import minidom
import sys

import urllib
import urllib2
import time

user = sys.argv[2]

password = sys.argv[3]

endpoint = "https://api.pinboard.in"

url = "/v1/posts/add?"

#open the xml file to import from and parse it
f = open(sys.argv[1], "r")

doc = minidom.parse(f).documentElement

#keep count of how many urls have been imported
urlcount = 0

count = 0
ellength = len(doc.childNodes)

failcount = 0
while count < ellength:
    e = doc.childNodes[count]

    if e.nodeType == e.ELEMENT_NODE:
        print "import url %s" % urlcount

        #get the attributes from the xml
        href = e.getAttribute("href")
        description = e.getAttribute("description")
        extended = e.getAttribute("extended")
        tags = e.getAttribute("tag")

        dt = e.getAttribute("time")
        rargs = dict(url=href, description=description, extended=extended,
                        tags=tags, dt=dt)
        shared = e.getAttribute("shared")

        if shared.strip() == 'no':
            rargs['shared'] = 'no'

        #convert them to unicode
        rargs = dict([k, v.encode('utf-8')] for k, v in rargs.items())

        print rargs
        #build the request to send
        #set up http auth for pinboard.in
        #doing this for every request may seem wasteful, but urllib2
        #seems to forget the auth details after a half dozen requests
        # if you dont
        password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
        password_manager.add_password(None, endpoint, user, password)

        auth_handler = urllib2.HTTPBasicAuthHandler(password_manager)
        opener = urllib2.build_opener(auth_handler)

        urllib2.install_opener(opener)


        request = urllib2.Request(endpoint + url + urllib.urlencode(rargs))

        #set the user agent
        request.add_header('User-Agent','SeansDeliciousMigrater')
        try:

            r = opener.open(request)
            #send the request and read the response
            response = minidom.parse(r).documentElement.getAttribute("code")

        except Exception, e:
            response = str(e)

        #if we get an invalid response, abort, proabbly throttled
        if response !="done":
            failcount += 1

            print "Failure: Invalid response: %s" % response
            if failcount > 4:

                print "Aborting: Invalid response %s"
                break
            else:
                print "waiting for 30 seconds and retrying"

                time.sleep(30)
        else:
            failcount = 0

            count += 1
            #put in a delay between requests to reduce odds of throttling
            time.sleep(1)

            urlcount += 1
    else:
        count += 1

print "%s urls imported" % urlcount

Download:delmigrate.py

Page 1 of 19
←
1
2
3
4
5
6
7
8
9
10
→

Before we start

Getting prebuilt firmware

Building the firmware on Ubuntu 16.04

Installing Micropython on the D1 Mini

Connecting to our micropython board.