Code Fudging

Stumbled on a neat little feature of Google Chrome today. You can
paste in a URL or search term to the address bar and use Cmd
(or Alt) + Return to open it in a new tab. When the new tab
opens, the search bar in your current tab switches back to the current
address.

Very handy, saves a Cmd + T here and there..

Just posting this because I always forget how to rename a window in a
screen session:

Ctrl-A, Shift-A

Comes in handy with >5 windows in a screen session

There seems to be a dearth of resources for posting a new blog entry
to Wordpress via Ruby, and some confusion around which method needs to
be invoked. That method is metaWeblog.newPost:

 
require "xmlrpc/client" 
 
my_new_blog_post = { 
 :description => 'foo', 
 :title => 'Title of the Post', 
 :excerpt => 'ex', 
 :categories => ['Category1', 'Category2'] 
 :post_status => 'publish' 
 } 
 
server = XMLRPC::Client.new( blog[:url], blog[:path]) 
 
# Returns the ID of the new post, or fail 
resp = server.call("metaWeblog.newPost", 0, "username", "password", my_new_blog_post) 

Your categories array should contain the category names of the
categories on your blog, you can't just populate them with new ones.
You may also want to change the post_status to "draft" if you want to
review it before posting.

Lesson #1 for scraping results from Google's search results - don't over do it.

After mining out some information, my Ruby script seems to be permanently banned. Not that I particularly blame them, even with a generous 10 second pause.

Must send user agent info next time.

Making use of the tzinfo gem (gem install tzinfo) in Ruby makes it easy to list, iterate over and display the countries of the world.

require 'tzinfo'


# Get one country 
us = TZInfo::Country.get('US')


# Get all countries 
all = TZInfo::Country.all


# Get all codes 
codes = TZInfo::Country.all_codes


# One liner to dump out all codes and their associated country name 
TZInfo::Country.all_codes.map { |code| puts "#{code} : #{TZInfo::Country.get(code).name}"

Paul Dix maintains a small github profile with consistently useful Ruby code; small libraries designed to do exactly what they were designed to do, with no cruft.

A little snippet for calling the MongoDB $inc (which increments a
field by N) function from pymongo:

 
import pymongo 
from pymongo import Connection 
 
# Set up connection 
con = Connection() 
db = con.my_database 
 
# Set up collection 
db_blog_posts = db.posts 
 
db_blog_posts .update({'_id': "my_id"}, {"$inc" : { "post_views": 1 }}) 

Links:

Updating in MongoDB: http://www.mongodb.org/display/DOCS/Updating
Updating in pymongo:
http://api.mongodb.org/python/1.3%2B/api/pymongo/collection.html#pymongo.collection.Collection.update

Having spent some time recently on multi-threading a process to download a few thousand files from the Internet, I started to notice some interesting differences between cURL and the indomitable wget.

Some observations:

1. The out of the box config for cURL leads to far more refused connections, empty content or HTTP 302 (document moved) codes than wget. Whether this is because of the default auto-follow behaviour of wget which cURL doesn't do, or the way they identify, I don't know.

2. cURL seemed a little faster, but not in a significant way.

3. When asking both tools to download the file only if it is newer, cURL truncates the old file and puts header information into the file instead. Wget behaves properly. I assume there is an override for this behaviour, but the hackish work around is to keep two copies of your files - an archive copy to reference modified infomation, and the new copy if cURL sees there is a newer version available.

4. I find that cURL is probably more powerful (because of complexity), but wget is easier to get working Right Now.

And finally, if you are attempting anything similar, threading is a MUST - a single thread doing this kind of operation will block for timeouts, redirects and the like.

Introducing more threads will get the job done in almost 1/N time. Your connection is big enough these days, but dealing with thousands of remote servers is too uncertain to rely on and wait on one thread.

The pymongo library - a Python interface to MongoDB - makes it pretty
easy to insert a new document which is related to another (usually in
another collection).

This is particularly useful if you need to maintain a relationship
between two items, but don't want to be restricted by embedded
documents in one collection.

 
import pymongo 
from pymongo import objectid 
from pymongo import Connection 
 
con = Connection() 
db = con.my_blog # Choose relevant DB 
 
collection_comments = db.comments # Pre load the collection 
 
post_id = 1 
 
# Find the reference (i.e. _id) to the post we want to relate to 
post_dbref = pymongo.dbref.DBRef("posts", str(post_id)) 
 
# Build a new comment, and give it a "post_id" or the post in question 
new_comment = { 
 'body': 'lorem ipsum', 
 'post_id': post_dbref. id 
} 
 
# Insert 
collection_comments.insert(new_comment) 

(Note there is an extra space on "post_dbref. id" - Posterous is trying to linkify it)

Et voila. Please post any optimizations if you can think of any.

Having not used Python much, or for long (as is probably evident), it
took a while to work out why the sqlite library wasn't handling UPDATE
statements properly.

Short answer: The connection needs to commit changes.

An example:

 
import sqlite3 
 
db = sqlite3.connect('mydb.db') 
c = db.cursor() 
 
c.execute("UPDATE posts SET title = 'new_title'") 
 
# Update hasn't happened yet... 
 
db.commit() 
 
# There we go.. 

Probably elementary Python, but slightly different to what I'm used to.