For the last week or so I've been wrapping my brain around Solr and Solango. The whole time that I've been doing this I've had the feeling that they can do awesome, powerful things but they're documentation is so poor that I couldn't figure anything out beyond the basic examples. Ultimately I had to dig through a bunch of code and do some experimentation. Now that I've finally figured out how to do what I've been trying to do and have wrapped my brain around some of the trickier bits I'm going to share some of the gotchas and solutions I've found.
sPaste Update
After a bit of a break, there is now a bit of new functionality in sPaste. Now when creating a snippet users can select a date and a number of views, after which the snippet will self-destruct. This update adds a great extra layer of security by only allowing information to exist in sPaste just as long as it needs to, and no longer. All while keeping sPaste very simple and easy to use.
Check it out and get a copy of sPaste to run on your own server at https://spaste.com!
Check it out and get a copy of sPaste to run on your own server at https://spaste.com!
sPaste|source - a secure, open source, pastebin
A few months ago, I found myself with a frequent itch at work. Often I needed to send sensitive information such as access credentials to co-workers and clients. Obviously email isn't an acceptable way to send this information, but anything much more complex than email overly frustrates who ever I am sending information to.
My solution to this problem was to create a new site called sPaste. sPaste is a secure pastebin where small snippets of text can be easily submitted, secured, and sent to other people.
At a personal level, sPaste was great success. sPaste took me less than 40 hours to write (thanks to the magic of Django and jQuery) and it perfectly meet my needs. Unfortunately in its existing form, it could never really be useful to most other people. Since sPaste is running on my server, anybody using sPaste would have to trust that I'm not doing anything evil with their data. Accordingly, for the past couple of weeks I've been working on cleaning things up and putting together an open source release.
Today I finally finished that work and have released sPaste|source. sPaste|source is a ready to go site, based on Django, which gives all of the features an functionality of sPaste.com on your own server. Simply grab the source, set it up like any other full Django project, and you will be ready to go.
Check it out! Feedback and comments are always welcome.
My solution to this problem was to create a new site called sPaste. sPaste is a secure pastebin where small snippets of text can be easily submitted, secured, and sent to other people.
At a personal level, sPaste was great success. sPaste took me less than 40 hours to write (thanks to the magic of Django and jQuery) and it perfectly meet my needs. Unfortunately in its existing form, it could never really be useful to most other people. Since sPaste is running on my server, anybody using sPaste would have to trust that I'm not doing anything evil with their data. Accordingly, for the past couple of weeks I've been working on cleaning things up and putting together an open source release.
Today I finally finished that work and have released sPaste|source. sPaste|source is a ready to go site, based on Django, which gives all of the features an functionality of sPaste.com on your own server. Simply grab the source, set it up like any other full Django project, and you will be ready to go.
Check it out! Feedback and comments are always welcome.
Quick Django Trick
Recently while playing around with a Django model in the always awesome iPython shell I discoved a neat feature of the Django ORM. It's basically a way to get the id of a related object without actually triggering a query to get all of the related object's data.
Frequently when working with a model which has a foreign key, I simply want to access the id of the related item and I don't care about any of the related item's other information. Situations where this comes up include generating links and building queryset filters. Unfortunately if I follow the normal Django style and do something like "item.related.id", the Django ORM will fire off an extra query to get all of the information of the related object (which I don't care about). While this is far from tragic it is still unnecessary work since all I care about is the object's id and that is already contained in the "item" object.
Fortunately there is an alternative! Instead of getting the id via "item.related.id" one can say "item.related_id". Using this method, no extra query is performed and I get just the id value I was looking for.
There are two things to be aware of with this trick. First I have not found this feature documented anywhere with a cursory search of the Django docs. This means that I am not sure how much this feature is actually supported and how permament it may or may not be. Second while I have not tested it, I suspect that if your foreign key has a custom database field name, the field on the model will match that custom field name.
As I come across any other quick Django tricks I may start making them a regular feature here on the blog. If you found this intersting or have your own quick django trick, let me know and leave a comment!
Annoucning sPaste.com
Tonight I am officially launching my new side project, sPaste.com! sPaste is a simple tool to help you quickly and securely send small snippets of data around the web. As usual, the site is written entirely in Django on the server side and jQuery on the client side. At the moment it is little more than a pastebin which forces the use of SSL but soon it will offer additional features and tools.
Check it out at https://sPaste.com and let me know what you think!
Check it out at https://sPaste.com and let me know what you think!
Update to django-ae-utils
I've been hanging out at the NYC App Engine Hackathon today and I've had some time to work on django-ae-utils. Accordingly I've just posted an update to django-ae-utils. This update includes a User model which uses the App Engine datastore as a backend, as well as a few generic views for working with the model.
Check it out at Google Code!
Check it out at Google Code!
Introducing django-ae-utils
Recently I've started to play around with using Django on Google's new App Engine. As mentioned by many others at this point, Django technically runs on App Engine but there are significant parts which do not. Accordingly I've started to write some code to replace or supplement the holes left in Django by App Engine. Since I figure this code may be of use to others I've released it as an open-source project called django-ae-utils (github mirror coming soon).
Google Launches App Engine
Google just launched a new platform called App Engine. Basically it allows you to run your web application on Google's servers using much of their main infrastructure such as BigTable and GFS. The best part is, you write App Engine apps in Python and they include Django out of the box! Mike Arrington has a video of Guido Van Rossum talking at the launch event as well as a pretty good summary at TechCrunch.
It should be very interesting to see what this platform can do and how people will put it to use.
It should be very interesting to see what this platform can do and how people will put it to use.
Working around MySQL's horrible "ORDER BY rand()" in Django
Recently I've been working on a web 2.0ish community site written in Django. As is frequently the case with such sites I often need to create lists or collections of psuedo-randomly selected items. For example on a user's profile page there may be a box showing a few of the user's friends, another box with some of the user's
pictures, or a box with random comments the user has made.
Ideally in this type of situation I would simply preform a query with what ever rules I need and a random "order by" statement such as the following:
Which generates SQL similar to:
In an ideal world this works great, I only get back as much data as I need and the site stays interesting because what the user sees is constantly changing/rotating. Unfortunately unless you are using PostgreSQL or have less than a few hundred rows in the pictures table you are not in an ideal world. Under both MySQL 4 and 5 random ordering is horrifyingly inefficient. Adding that "ORDER BY rand()" can make a query that took a few milliseconds to run instead take tens of seconds. The problem gets even worse when you are working with a table containing millions of rows and thousands of rows which meet the criteria of your query. What is one to do?
pictures, or a box with random comments the user has made.
Ideally in this type of situation I would simply preform a query with what ever rules I need and a random "order by" statement such as the following:
pictures = User.picture_set.filter(private=False).order_by('?')[:6]Which generates SQL similar to:
SELECT * FROM `pictures`
WHERE private = 0 AND user_id = 5
ORDER BY rand()
LIMIT 6
In an ideal world this works great, I only get back as much data as I need and the site stays interesting because what the user sees is constantly changing/rotating. Unfortunately unless you are using PostgreSQL or have less than a few hundred rows in the pictures table you are not in an ideal world. Under both MySQL 4 and 5 random ordering is horrifyingly inefficient. Adding that "ORDER BY rand()" can make a query that took a few milliseconds to run instead take tens of seconds. The problem gets even worse when you are working with a table containing millions of rows and thousands of rows which meet the criteria of your query. What is one to do?
Installing Django on OS X 10.5 (Leopard)
This is a quick update to my previous installing django posts.
Today I got my hands on Leopard, so far I've been happy but went I went about reinstalling Django I hit a snag. MacPort would encounter an error whenever it tried to build PostgreSQL. If you can wait a little while the port maintainer for PostgreSQL should fix the problem.
On the other hand if you need Django running today you can follow the steps at Lee Packham's Corner to get PostgreSQL running. Unfortunately this will get you PostgreSQL 8.2 (not inherently a bad thing) which means you will need to manually build and install psycopg2 since the MacPorts psycopg2 package depends on PostgreSQL 8.1. Here's what you will need to do to manually build and install pyscopg2:
- Download the source from http://www.initd.org/tracker/psycopg/wiki/PsycopgTwo.
- Extract the files using the command tar -xzf psycopg2-latest.tar.gz.
- Open the file setup.cfg file and look for the line that starts with pg_config=
- Change the line to read pg_config=/opt/local/var/macports/software/postgresql82/8.2.5_0+darwin_9/opt/local/lib/postgresql82/bin/pg_config
- Run the command python setup.py build
- Run the command python setup.py install
Now you should be all set!
On a related note if your interested in learning about Djago and you'll be around Poughkeepsie, NY this Wednesday stop by at the MHVLUG meeting, I'll be giving a Intro to Django talk.
Installing and Running Django on a Mac - Part 2 - Install
This is a continuation of Part 1 - Getting Ready.
Let's just get into it. Before you can run Django you will need to install several dependencies. To do this I'm using MacPorts but it can be done just as easily using PortAuthority. In PortAuthority just search for the package name, select the package in the results and click on the install button (the brown box thing). With MacPorts open a terminal and run the command sudo port package_name. Be aware that each of these will take a while to run while MacPorts downloads and compiles each package and its dependencies.
Let's just get into it. Before you can run Django you will need to install several dependencies. To do this I'm using MacPorts but it can be done just as easily using PortAuthority. In PortAuthority just search for the package name, select the package in the results and click on the install button (the brown box thing). With MacPorts open a terminal and run the command sudo port package_name. Be aware that each of these will take a while to run while MacPorts downloads and compiles each package and its dependencies.
- python25 - Python 2.5 (Django will run on any version of Python 2.3 or newer, I personally like 2.5)
- openssl - Handles encryption between python and the database server.
- py25-setuptools - Setup tools for Python 2.5
- postgresql81 postgresql81-server - My database of choice. Obviously install the database that makes sense for you.
- py-psycopg2 - Python interface for postgres. Additionally py-psycopg2 will install two dependencies, PortAuthority/MacPorts will take care of
- subversion - Django doesn't need SVN to run but SVN is the easiest way to get the latest version of Django.
Installing and Running Django on a Mac - Part 1 - Getting Ready
I'm a recent Mac convert. For the most part I've loved all the perks of being a Mac user but I've run into a road block recently. Apparently getting my favorite web application framework, Django and its various dependencies running on OS X to do local development is not as easy or clear as it is on other platforms like windows or linux. This may be thanks to my limited knowledge of BSD's querks or simply that not many people run Django on the Mac. To make things even more complicated I want to run the NewFormsAdmin branch instead of the current 0.96 release. Since I'm sure I'm not the only person running into troubles and confusion with this here's a step-by-step guide on how to get Django running on your Mac!
At this point you have all the resources you will need to install Django and it's dependencies. Check back tomorrow to see how to do the actual install.
- Install Xcode - You will need the tools included in Xcode to build and install many of Django's Pre-requisites. To get Xcode you can either install it from your OS X install DVD or you can download it from Apple Developer Connection (ADC). If you choose to download Xcode be aware that you will need to register (for free) and that the download is a little under 1 GB in size. Otherwise if you install from DVD remember to run Software Update (System Prefrences > System > Software Update), there have been several updates to Xcode since its release.
- Download and install MacPorts - What's MacPorts you ask? MacPorts is a package mananagement tool that handles obtaining, building, and installing dozens of open source applications and tools onto a Mac. Once you download MacPorts mount the DMG image and run the installer package. Generally accepting the defaults for the installer is fine. Once installation you will have a new command available in your terminal called port.
- (Optional) Download and install PortAuthority - PortAuthority is a GUI frontend to MacPorts. I love the command-line but sometimes for things like package managers a GUI can be nice to more easily search and browse around. NOTE - PortAuthority costs $20 but offers a 30 day trial.
At this point you have all the resources you will need to install Django and it's dependencies. Check back tomorrow to see how to do the actual install.