Chronosbox

There are dozen of articles and snippets about how work with JSON
and Django, however I saw that most are a bit vague and solution involves use of simplejson to serialize and HttpResponse to send response from the view. There are some details that normally aren’t covered and now I show a easy way and a functional example of how work with Django, JSON and forms.
(continue reading…)

This weekend I reworked on my .vimrc and put some extra time in make things work
better for me in python and django coding . I did a nice job and now I can
code with much more goodies.

Results are:

• A very commented and customized .vimrc.
• Support for python omnicompletion.
• Adding django and python paths automatically to vim.
• Setting DJANGO_SETTINGS_MODULE automatically to vim, so you can complete your own code.
• SnipMate support for django and django templates.
• Django templates with support for highlighting in tags and use of ‘%’ with matchit.
• Nice python folding and jumping.
• A custom gui dark/grey theme called wombat.
• A session manager to easily save session and load all when you back.

Here is files to download:

• vim-confs-latest.tbz2: The bziped tar with all files, unpack in your home.
• vim-confs-latest.tbz2.md5: The md5 checksum for file.

If you prefer, here is url of github: http://github.com/chronossc/my-vim-confs

This pack can’t be created with some plugins and docs, here is reference:

• UsingVimWithDjango from Django Wiki
• SnipMate vim plugin by Michael Sanders
• SnipMate snippets for django by Rob Hudson (Demo)
• Matchit vim plugin by Benji Fisher
• Updated Syntax highlighting for Django templates by Dave Hodder
• py_jump vim plugin by Sergiy Matusevych
• python fold vim plugin by Jorrit Wiersma
• Session Manager for vim by Yuri Klubakov

Additionally (thx Handrus), I suggest for who write much HTML the SparkUp vim and textmate plugin . I not tested, yet, but sounds nice.

Thanks for you that read it, thank you for all that wrote all nice plugins,
thanks very much to freenode #vim channel and awesome vim doc! Now let’s code
with vim

** Updated
2010-03-07: I repacked files because I removed some debug messages, so for who already downloaded, download again
2010-03-08: I repacked again and added to github with some fixes.

Sometime ago I have to change the charset and line breaks of a web application from charset iso-8859-1 (US-Latin) to UTF-8 and linux break lines to windows break lines. At that time chronos helped to create this small script that I’m sharing now:

1
2
3
4
5
6
7
8
9
10

#!/bin/sh
#change extensions filter as you need ;)
find -type f | egrep "\.php|\.css|\.js|\.html|\.htm" | egrep -v "\.svn" > ./charset_list.txt;
while read -r line;do
	#charset change
	iconv -f LATIN1 -t UTF-8 "$line" > "${line}2";
	mv "${line}2" "$line";
	#we adopted dos format for line breaks
	unix2dos "${line}";
done < ./charset_list.txt > ./charset_after.log 2>&1 #receive list of params and send output to log

Explanationss: The script generate a list with files in actual and subfolders (according to the filter described in 2nd line).
After that reads the file, create a UTF-8 version of it in another file and move it over the old one, after this it changes the final file line breaks. Very simple, I LOVE BASH!

Nowadays it is a standard that all web sites and servers run at least with UTF-8 encoding as default, however you can find cases where it is LATIN1 or even ASCII.

The case we are dealing with today is a Debian server, version 5.0.3 (etch), which used encode LATIN1 as default encoding when installed and configured the Postgresql, and not let me create a database with UTF-8 encode, showing following error:

debian:~# su postgres -c 'createdb teste3  -E UTF-8'
createdb: database creation failed: ERROR:  encoding UTF8 does not match server's locale en_US
DETAIL:  The server's LC_CTYPE setting requires encoding LATIN1.

(continue reading…)

Multiple python environments

Work with various python versions with multiple versions of python environments has become something common in my life, and, I believe that an easy way of setting this environments are something very useful, and not simple for most of people.

I created a simple shell script that set this environment, that is a function that you add in your .bash_profile. Run this function in your shell with python version as argument and all is set. Let’s take a look on my code:

export ORIGINALPATH=$PATH
loadpyenv(){
    # $PV is our control variable, get from $1, first argument on shell
    # anything that you need to put here and depend for python version ${PV} is your var ;)
    export PV=$1
    echo -e "\n:: Setting environ to use python $PV"
 
    # Put here various aliases that you can use
    # put a python alias
    alias python="python${PV}"
    # m and runserver a kick alias for Django common actions
    alias m="python${PV} manage.py"
    alias runserver="python${PV} manage.py runserver :8000"
    # easy and fast install of python packages, in right path with localpyinstall
    alias localpyinstall="python${PV} setup.py install --prefix=~/.py/${PV}/"
    echo -e "\n:: New aliases"
    # print aliases :)
    alias python m runserver localpyinstall
 
    echo -e "\n:: New PATH"
    # setup applications path, be careful about not use $ORIGINALPATH.
    #
    # I use a path structure like that:
    # ~/.py/
    # |-- 2.4
    # |-- 2.5
    # `-- 2.6
    export PATH="${HOME}/.py/${PV}/bin/:${ORIGINALPATH}"
    echo $PATH
    echo -e "\n:: New PYTHON PATH"
    # setup new python path, don't need to add system wide path, just our custom paths
    export PYTHONPATH="${HOME}/.py/${PV}/lib/python${PV}/site-packages/"
    # print more some info
    python$PV -c "import sys;print sys.path"
    echo -e "\n:: Django version & file"
    python$PV -c "import django;print 'version %s ... file %s' % (django.get_version(),django.__file__);"
}

Now you create a directory structure like my own with command:

$ mkdir -pv ~/.py/{2.4,2.5,2.6}

And any time that you need to use a diferent version of python, load with:

$ loadpyenv VERSION

Where VERSION is python version that you choose.

Multiple python versions in Gentoo

The Gentoo Linux, my preferred distro, give me a bonus about multiple python versions, configure in your /etc/make.conf the following line:

USE_PYTHON="2.4 2.5 2.6"

Now install all python versions and run following command to re-compile your installed packages:

emerge $(eix -I dev-python/* | grep "\[I\]" | cut -d' ' -f2)

The eix is a simple app that index portage, very common in Gentoo machines.
Each python package are recompiled for all tree versions (in this case), but, I have some problems with some packages and python 2.4

Well, that’s it, see ya!

As a user of the old friend gentoo I always missed the lack of possibilities to compile softwares to have a better performance.Mainly in server-side softwares like Java, PHP, Apache, Mysql and Postgres.In my tests I always felt the diference between pre-compiled versions and the softwares compiled in loco.
In Debian/Ubuntu the discover of apt-build helped me to solve the problem qith the same class and almost the same efficiency that portage.

Lets go to the step-by-step:

To install apt-build:
sudo apt-get install apt-build auto-apt

Configure your machine:
dpkg-reconfigure apt-build
Choose your architecture and the level of optimization. The medium is indicated.
The options will be saved at /etc/apt/apt-build.conf ; Lets just adjust the number of cores, in the file search for “make_options” and change as follow:
make_options = " -j4"
The number is equal to # of “cores” + 1. So a quadcore should be “-j5″.

To prevent your compiled packages to be overwriten by the official ones edit this file /etc/apt/preferences adding:
Package: *
Pin: release o=apt-build
Pin-Priority: 990

Don’t forget to add the source repositories (deb-src) to sources list and you’re done!
Now use apt-build install instead of apt-get install

Sample: sudo apt-build install xchat emesene

Quick tutorial on apt-build

apt-build install –reinstall to reinstall something
apt-build upgrade upgrade all installed softwares
apt-build world Recompile all system
apt-build remove programa Removes software
apt-build clean-build Remove compiled packages
apt-build clean-sources Remove downloaded sources.

Hello people. Recently I used Django Auth module and his permission system. Is very interesting how we can define custom permissions per model and use in view, templates, etc.

Today, our code come from necessity of easily work with forbidden urls, where Django give us only HttpResponseForbidden class, that isn’t really interesting to handle, once that ideal is something like Http404 exception.

Based on this necessity, I developed a Middleware that intercept Http403 exceptions, and return, in following order, one of options:

1. One custom view, set on handler403 variable at project urls.py
2. Render 403.html template
3. Render one hard-coded template

To use Middleware, follow instructions:

1. Download the project tgz at end of this article. We use this project as example and I called it of http403project.
2. Unpack and copy file http.py to some place of your project, in http403project project I put in project root.
3. Add the middleware to settings.py:

   MIDDLEWARE_CLASSES = (
         # ...
         'http403project.http.Http403Middleware',
   )

4. Raise a Http403 exception in view, ex:

   from http import Http403
   def Http403View(request):
     raise Http403(u"This is custom message for permission denied.")

5. Configure your custom view or template if want. Your template should be named 403.html. Your custom view should receive request and exception arguments. Exception argument for view transport custom message that user raised, you can view a simple example in http403project urls.py

Project tgz: http403project.tgz

Ok, that’s it. Use is very simple, be happy

Today I saw a post at one of the blogs I often watch, where a guy said he got 1000% of speed boost in a aplication by changing: uma aplicação trocando um:

SELECT * FROM TABLE WHERE fieldname ILIKE 'abc%';

For

SELECT * FROM TABLE WHERE lower(fieldname) LIKE 'abc%';

(changed ilike to lower … like)
Kinda ‘newbie’ (hehehe), obviously this absurd improve happened in a giant data mass and probably in a field without appropriated index.
BTW, my tip is other:
Ilike isn’t supported for mysql and sqlite. So use it its extremely inappropriate, especially if you are working with Django, Rails or any application layer that provides a multi-db.
Sure the way to make it happens with the above frameworks is “forcing” ilike.Their ORM are pretty much able to handle this comparisons.

Conclusion
Avoid “ilike” on sql statements, use “lower()… like” or “upper… like”; And remember to change variables to lower/upper case!

Here I show a sinple script to block IPs that try brute force at your host, preventing you from use a alternative ssh port, as example.

The script runs over /var/log/messages and detect via regex brute force attemps, and add IPs to /etc/hosts.deny

Code:

#!/bin/bash</code>
 
cp /etc/hosts.deny /var/tmp/hosts.deny
 
# Jun  8 00:39:43 mintaka sshd[23332]: User root from 222.77.183.51 not allowed because not listed in AllowUsers
egrep "$(date +%b) ($(date +%d)|$(date +%e)).*User .* from [a-z0-9\.]* not allowed because not listed in AllowUsers" /var/log/messages | sed -e 's/.* from \([a-z0-9\.]*\) .*/\1/' | sort | uniq | sed -e 's/^/ALL:/' &gt; /etc/hosts.deny ;
#Jun  8 11:54:33 mintaka sshd[31221]: Invalid user bob from 62.204.145.224
egrep "$(date +%b) ($(date +%d)|$(date +%e)).*Invalid user .* from [a-z0-9\.]*$" /var/log/messages | sed -e 's/.*Invalid user .* from//' | sort | uniq | sed -e 's/^/ALL:/' &gt;&gt; /etc/hosts.deny ;
 
diff -u /etc/hosts.deny /var/tmp/hosts.deny &amp;&gt;/dev/null || (echo "** hosts.deny updated **" ; cat /etc/hosts.deny)

Script working:

Sep 2 16:55:02 mintaka -- MARK --
Sep 2 13:58:10 mintaka sshd[20790]: Did not receive identification string from 189.2.118.72
Sep 2 13:58:34 mintaka sshd[20795]: Invalid user suporte from 189.2.118.72
# Brute force attemp
Sep 2 13:58:41 mintaka sshd[20798]: Invalid user suporte from 189.2.118.72
Sep 2 13:58:43 mintaka sshd[20800]: Invalid user suporte from 189.2.118.72
Sep 2 13:58:46 mintaka sshd[20802]: Invalid user suporte from 189.2.118.72
Sep 2 13:58:49 mintaka sshd[20804]: Invalid user suporte from 189.2.118.72
Sep 2 13:58:51 mintaka sshd[20806]: Invalid user suporte from 189.2.118.72
Sep 2 13:59:24 mintaka sshd[20817]: Did not receive identification string from 189.2.118.72
# Script running
** hosts.deny updated **
ALL:74.221.239.100
ALL: 189.2.118.72
ALL: 200.27.79.101
ALL: 74.221.239.100
# and now IPs are blocked
Sep 2 14:00:09 mintaka sshd[20826]: Invalid user teste2008 from 189.2.118.72
Sep 2 14:00:10 mintaka sshd[20864]: refused connect from 189.2.118.72 (189.2.118.72)


I run in my machine every two minutes