Entries Tagged 'Programming' ↓

Backup your iphone SMS’s as a conversation transcript.

At the point in writing there aren’t many ways of backing up your SMS’s from your iPhone, but you do a system backup when you sync with itunes but what if you want your SMS conversions backed up as a simple non proprietary format? Well the answer is here!

Shea recently upgraded to an iPhone, and was having trouble with bluetoothing the data across from her old phone. She told me she had saved the most important SMS’s but was a shame to loose the record of our entire SMS communication history.
And she’s right, in today’s world where everything is digital a lot of important relationship related stuff was discussed and it would be a shame to loose it all. So I started googling, at first I thought it would be a feature of itunes.. i was wrong.. which is a shame. But it turns out people have done it before, and some applications where written to do just that, unfortunately though all report that they only work for the iphone OS version 2.0. Sure it would be alot easier if the phone was jail broken, but there must be an easier way.. and there is!

Step 1 – Extract the SMS database from one of your iPhone backups:

I came across this OSX app, I’m not sure if there is a windows equivalent but seeing as I sync my iPhone under OSX I don’t really care.
Anyway this app allows you to access one of your iPhone backups and extract parts of it. For this post we are only interested in the SMS’s so once you have chosen a backup from the list scroll to the bottom and extract “System Files” or “Other Files” (can’t remember the name will check when I get home).

You’ll be prompted for a location to extract to, I suggest you extract the contents to an empty folder.
Once the files have been extracted you should find a sms.db file under:

<extracted folder>/System Files/Library/SMS/sms.db

This sms.db turns out to be a sqlite file.. and for those in the know, know that this is good news! With a few lines of python we can access and extract what we need from the file, but first we need to find the structure, which leads us to step 2.

Step 2 – Determine the sms.db table internal table structure.

There are many sqlite applications, but I’ll point you to 2 of them. A OSX app and a Linux app.
For OSX there isΒ  sqlitebrowser and for Linux I simply used sqliteman which to install is as simple as:
For Debian/Ubuntu:

apt-get install sqliteman

For Fedora:

yum install sqliteman

Now inside the sms.db file there turns out to be 5 tables:


All actual SMS text are stored in the ‘message’ table, and as the conversion I needed to backup was a simple 1 on 1 conversation all I needed was to query this one table.
While we are here what’s the structure of the ‘message’ table, well there are 17 columns but the only ones that I required where address, date, text and flags.

  • address – Is the number of the person you were having the SMS communication with.
  • date – Date of the text in epoch format.
  • text – The text itself.
  • flags – numerical flags attached to the message, but just looking at the table I realised that if the flag field contained a 2 then the text was from the recipient, a 3 indicated it was send from you.

With all that information I was ready to write my simple script, which leads to step 3.

Step 3 – The basic script

This python script does need some work, I only wrote it as a once off, so adding more exception handling and passing in the main parameters into the script rather then using variables would be useful, but outside the scope.

It is also worth a mention that I am using python 2.6 and it does also require the sqlite module, under fedora it is as simple as:

yum install python-sqlite2

Note: Yes its the 2nd version of the python sqlite module, but is actually supports sqlite version 3, so inside python you ‘import sqlite3’ so it actually is the sqlite3 module.

Now for the script, don’t forget to change the <data place holders> with the data you require:

#!/usr/bin/env python                                                                                                                                        

import sqlite3
import time
import sys
import os
import codecs 

DEBUG = True
names = {'2' : "<Recipient>", '3': "<your self>"}
key = "<number>"               

SQL = "select flags, address, date, text from message where address = '%s'"

output = """%s - %s


def getDate(epoch):
        return time.strftime("%a, %d %b %Y %H:%M:%S",time.localtime(epoch))

def main(dbfile, outputfile):
        outFile = codecs.open(outputfile, encoding='utf-8', mode='w')

        conn = sqlite3.connect(dbfile)
        c = conn.cursor()
        c.execute(SQL % (key))

        count = 0
        firstDate = ""
        lastDate = ""

        for row in c:
                flags = str(row[0])
                for name in names.keys():
                        if name in flags:
                                user = names[name]
                date = getDate(row[2])
                text = unicode(row[3])

                outStr = output % (user, date, text)

                if DEBUG:
                        print outStr


                # Store the first Date
                if count == 0:
                        firstDate = date
                lastDate = date
                count += 1
        outStr = "Date Range: %s - %s" % (firstDate, lastDate)
        if DEBUG:
                print outStr


if __name__ == "__main__":
        if len(sys.argv) < 3:
                print "%s  " % (sys.argv[0])

        dbfile = sys.argv[1]
        outFile = sys.argv[2]

        if not os.path.exists(dbfile):
                print "%s doesn't exist" % (dbfile)

        main(dbfile, outFile)

This creates a transcript like:

Matt - Mon, 04 Feb 2010 08:01:38
This is a text message

Other Person - Mon, 04 Feb 2010 08:02:38
This is the response.

Anyway happy backing up!
Needless to say I believe Shea was happy πŸ™‚

MySQL Datetime precision… I think not!

We use Hibernate at work, and I’ve been working on getting all our JUnit tests to work on MySQL… Yes hibernate takes care of most of that, and it does, BUT some of our unit tests were failing when I pointed it at MySQL rather then PostgreSQL.

After some debugging I noticed that when we pulled one of our objects out of the database via hibernate, the Date object inside wasn’t the same as the Date inserted:
originalDate.getTime(); // = 1261613807262
retrievedDate.getTime(); // = 1261613807000

The two epoch dates are almost the same, except the last three digits are zero’d out. The keen observers might have already figured out these are microseconds.
It seems MySQL isn’t storing or retrieving the microseconds from the datetime datatype.Β  Even though MySQL does have the MICROSECOND() function. A quick search on Google supports my findings there is a “Feature Request” opened on the MySQL bug tracker, annoyingly though it was opened on 15 Feb 2005, yet nothing is yet implemented.

For those who want to get around this issue, and want to Assert the dates, and want them to actually work, you simply need to zero out the microseconds component of the date, here is the simple function I used:
private Date removeMicroseconds(Date date) {
return new Date(date.getTime() - (date.getTime() % 1000));

Installing Sun Java on Fedora 12

By default Fedora 12 doesn’t install Sun’s Java, and it isn’t in the repository. This isn’t a mistake, in fact I think this is a good decision! Fedora is only dealing with free open source software. You can add other repositories to give you the extra non OSS software you want, for those who cannot live without certain software.

Fedora uses the OpenJDK, which I think is awesome.. but unfortunately as I am a Java developer at the moment, and it seems some of the software I work requires the Sun version of Java, at least to compile.

So I needed to install Sun JDK on my 64bit machine.. this is how I did it:

  1. Download the Sun Java JDK 64bit Linux bin installer.
  2. Run it to install.
  3. Even though I ran it as root it installed in the current folder. So move the folder to we it should be installed:
    sudo cp -a jdk1.6.0_16 /usr/lib/jvm/
  4. Use the alternatives command to tell Fedora to use the new Java binary, to do so we need to “install” the new binary as an option in alternatives:
    sudo /usr/sbin/alternatives --install /usr/bin/java java /usr/lib/jvm/jdk1.6.0_16/bin/java 20000
    Then use alternatives and make sure our new binary is selected:
    sudo /usr/sbin/alternatives --config java
  5. Use the following command to verify that Fedora is pointing to the right binary:
    java -version

That’s it, Sun’s Java should now be installed!

Eclipse @author

Most Java developers know eclipse is an awesome IDE, it’s customisation and auto-completeness is great.. It even auto completes annotations in the code far ya.

When added a @author annotation to some code, it’ll automatically try and add your name.. which is great, but it uses your logon name, not my real name. It’s not that big a deal, cause retyping your name is easy!

But curiosity got the better of me, I wanted to know if there was a place you can set the author name correctly.. and there is! When starting eclipse you can pass in an option:
eclipse -Duser.name=Matthew Oliver

But most people want to run eclipse from the menu, sure you can update the menu item, but there is another way. In the eclipse installation directory there is an eclipse.ini file, just append -Duser.name=<name> and that’s it.

For those of you following along at home, or those of you who are sysadmins, may realise that is a major problem.. can you figure it out?

By setting the user.name in the ini file, you forcing the user.name for everyone who uses eclipse on that machine, if there is only one person then there is no problem, but if there’s more however…

Maybe editing the menu item is better after all πŸ˜‰ or maybe there is an eclipse.ini file under ~/.eclipse somewhere!! Who knows.

Bash autocompletion + Java

I think the title says it all, I was just checking something at work and needed to run one of our Java apps using the command line, I did the standard:
java -cp jarfile1.jar:jarfile2.jar au.gov.

and hit tab key twice out of habit and to my surprise bash had checked the jar files inside the command I was writing and presented me with all the class options!!

Bash auto completion just continues to impress me! Thank you bash, you have just made my life SO much easier yet again.

For those of you who have no idea what bash is, its a terminal shell used by default in Linux.. so for those of you out there not yet using Linux, here is yet another reason to do so.

OSX also uses bash as the default shell for the terminal, so I’ll now have to test it on my macbook.. after a quick test the answer is no, OSX is either using an older version of bash which doesn’t support this feature, or hasn’t turned on/set up all the autocompletion features by default.

Ant vs Proxy

I’ve been updating our ant build scripts to compile all our third party library’s from source for two reasons:

  1. So we don’t keep binary blobs under CVS, and
  2. So it is easy to distribute the source of the libraries we use.

Ant is a very powerful build system, and everything was moving along swimmingly until I was working on calling the JavaHelp build script, which is nice and smart, but it’ll seem too smart for it’s own good! It automatically downloads its dependencies for you, the only problem is at work we are stuck behind a very annoying web proxy.

But never fear ant came to the rescue! Ant allows you to put a <setproxy .. /> tag into your build script to set a proxy, yay!!


Can anyone see a problem here? When we are working on build scripts for some open source apps we write here at work, then adding the proxy to the script is good and fine here, but it WILL break for all users out in the big wide world who use our software… if only there was an easier way.

I talked to Chris and he came solution… one that I should have thought of myself, environment variables!

Ant checks an evironment variable called ANT_OPTS for, strangely enough, ant options. To set the proxy settings:

export ANT_OPTS="-Dhttp.proxyHost=localhost -Dhttp.proxyPort=5865 -Dhttps.proxyHost=localhost -Dhttps.proxyPort=5865"

NOTE: Change localhost and the port to the port of your proxy server.

JVMs have a GPL’d implementation of the JavaMail API thanks to GNU!

This has been around for a while, but I’ve only just stumbled across this as I have been looking at into the licensing of some 3rd party jars we use at work.

GNU have a free implementation of the Sun JavaMail API specification, which means we now have some more GPL’d jars we can use.

So lets build em!
First we need to download all the source:

wget http://ftp.gnu.org/gnu/classpathx/activation-1.1.1.tar.gz
wget http://ftp.gnu.org/gnu/classpath/inetlib-1.1.1.tar.gz
wget http://ftp.gnu.org/gnu/classpathx/mail-1.1.2.tar.gz
NOTE: The above links are on the javamail link above, so you can just download it from there!

Extract and start building.. as an example I’ll just use the ‘/tmp’ directory, remember to change this to your desired location.

tar -xvf activation-1.1.1.tar.gz
tar -xvf inetlib-1.1.1.tar.gz
tar -xvf mail-1.1.2.tar.gz

We need to build activation.jar and inetlib.jar first as they are dependancies of the GNU mail api.

First build activation.jar:

cd activation-1.1.1/
cd ..

Then build inetlib.jar:

cd inetlib-1.1.1/
cd ..

Now with those jars built we need to build the gnumail jars:

cd mail-1.1.2/
./configure --with-activation-jar=/tmp/activation-1.1.1/ --with-inetlib-jar=/tmp/inetlib-1.1.1/
cd ..

Lets move all the Jar files into /tmp so I can show you them all:

cp activation-1.1.1/*.jar .
cp inetlib-1.1.1/*.jar .
cp mail-1.1.2/*.jar .

ws -l *.jar
-rw-r–r– 1 matt matt 44375 2009-06-12 15:53 activation.jar
-rw-r–r– 1 matt matt 163942 2009-06-12 15:54 gnumail.jar
-rw-r–r– 1 matt matt 95250 2009-06-12 15:54 gnumail-providers.jar
-rw-r–r– 1 matt matt 182002 2009-06-12 15:53 inetlib.jar

Now just put them in your Java path and away you go!

Happy GNU Java’n