Arrdino

A few of us here at work are using Fedora 12, two of the guys upgraded Fedora to find X not starting, seemingly a broken nvidia driver. Because they had issues I was careful about updates. But the inevitable happened.. I updated killing my Xorg.

I decided I’d do a bit of research into what was actually happening.. the fix the others used at work was to install the free nvidia driver or use a driver without proper 3D support, I didn’t like this so I started by checking my logs and then some google-fu.

The Xorg log file showed (/var/log/Xorg.0.log):

(II) May 11 11:18:49 NVIDIA: Using 768.00 MB of virtual memory for indirect framebuffer
(II) May 11 11:18:49 NVIDIA: access.
(II) May 11 11:18:49 NVIDIA(0): Initialized GPU GART.
(II) May 11 11:18:52 NVIDIA(0): Setting mode
(II) May 11 11:18:52 NVIDIA(0): "DFP-0:nvidia-auto-select+0+0,DFP-1:nvidia-auto-select+1680+0"
(EE) May 11 11:18:56 NVIDIA(0): WAIT: (E, 0, 0x827d, 0)

Notice it seemed to hang on a wait.

So I checked the kernel log (/var/log/messages) and found a lot of repeating entries:

May 11 11:19:55 localhost kernel: DMAR:[DMA Read] Request device [01:00.0] fault addr 22d226000
May 11 11:19:55 localhost kernel: DMAR:[fault reason 01] Present bit in root entry is clear
May 11 11:19:55 localhost kernel: DRHD: handling fault status reg 2

These repeated hundreds of times.
OK so time to start googling, it turns out these errors seem to point to the PCI bridge or something related to that, the device [01:00.0] turns out to be the address of my video card.

Some google hits talked about a bios issue and recommend adding “intel_iommut=off” to the kernel line, I also found mention in a kernel mailing list post about “intel_iommu=off”, as my PCI bus was made by Intel I thought I’d give it a go.
NOTE: To check your PCI Bus you can use the “lspci” command.

For the keen observers you’ll notice intel_iommut and intel_iommu, I didn’t know if one had a typo so I added them both to my kernel line and booted.. and SUCCESS, Xorg started!

Updated Note: Thanks to Chris the only option you need is “intel_iommu=off”.

For those who don’t know how to add the extra stuff to your kernel command line permanently, then edit /boot/grub/menu.lst.

In this file there should be an entry for each kernel and operating system you have installed, each new entry starts with a “title” line, under which is a root, kernel and initrd line, just append the new option(s) to the end of the kernel line for the entry/kernel you want. e.g:

title Fedora (2.6.32.11-99.fc12.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32.11-99.fc12.x86_64 ro root=/dev/mapper/system-fedora noiswmd LANG=en_AU.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet nouveau.modeset=0 vga=0×318 intel_iommu=off
initrd /initramfs-2.6.32.11-99.fc12.x86_64.img

NOTE: The kernel line above is actually wrapping to a new line in, but should actually be all on the same line.
So there you go.

At the point in writing there aren’t many ways of backing up your SMS’s from your iPhone, but you do a system backup when you sync with itunes but what if you want your SMS conversions backed up as a simple non proprietary format? Well the answer is here!

Shea recently upgraded to an iPhone, and was having trouble with bluetoothing the data across from her old phone. She told me she had saved the most important SMS’s but was a shame to loose the record of our entire SMS communication history.
And she’s right, in today’s world where everything is digital a lot of important relationship related stuff was discussed and it would be a shame to loose it all. So I started googling, at first I thought it would be a feature of itunes.. i was wrong.. which is a shame. But it turns out people have done it before, and some applications where written to do just that, unfortunately though all report that they only work for the iphone OS version 2.0. Sure it would be alot easier if the phone was jail broken, but there must be an easier way.. and there is!

Step 1 – Extract the SMS database from one of your iPhone backups:

I came across this OSX app, I’m not sure if there is a windows equivalent but seeing as I sync my iPhone under OSX I don’t really care.
Anyway this app allows you to access one of your iPhone backups and extract parts of it. For this post we are only interested in the SMS’s so once you have chosen a backup from the list scroll to the bottom and extract “System Files” or “Other Files” (can’t remember the name will check when I get home).

You’ll be prompted for a location to extract to, I suggest you extract the contents to an empty folder.
Once the files have been extracted you should find a sms.db file under:

<extracted folder>/System Files/Library/SMS/sms.db


This sms.db turns out to be a sqlite file.. and for those in the know, know that this is good news! With a few lines of python we can access and extract what we need from the file, but first we need to find the structure, which leads us to step 2.

Step 2 – Determine the sms.db table internal table structure.

There are many sqlite applications, but I’ll point you to 2 of them. A OSX app and a Linux app.
For OSX there is  sqlitebrowser and for Linux I simply used sqliteman which to install is as simple as:
For Debian/Ubuntu:

apt-get install sqliteman

For Fedora:

yum install sqliteman

Now inside the sms.db file there turns out to be 5 tables:

_sqlitedatabaseproperties
group_member
message
msg_group
msg_peices

All actual SMS text are stored in the ‘message’ table, and as the conversion I needed to backup was a simple 1 on 1 conversation all I needed was to query this one table.
While we are here what’s the structure of the ‘message’ table, well there are 17 columns but the only ones that I required where address, date, text and flags.

• address – Is the number of the person you were having the SMS communication with.
• date – Date of the text in epoch format.
• text – The text itself.
• flags – numerical flags attached to the message, but just looking at the table I realised that if the flag field contained a 2 then the text was from the recipient, a 3 indicated it was send from you.

With all that information I was ready to write my simple script, which leads to step 3.

Step 3 – The basic script

This python script does need some work, I only wrote it as a once off, so adding more exception handling and passing in the main parameters into the script rather then using variables would be useful, but outside the scope.

It is also worth a mention that I am using python 2.6 and it does also require the sqlite module, under fedora it is as simple as:

yum install python-sqlite2

Note: Yes its the 2nd version of the python sqlite module, but is actually supports sqlite version 3, so inside python you ‘import sqlite3′ so it actually is the sqlite3 module.

Now for the script, don’t forget to change the <data place holders> with the data you require:

#!/usr/bin/env python                                                                                                                                        

import sqlite3
import time
import sys
import os
import codecs 

DEBUG = True
names = {'2' : "<Recipient>", '3': "<your self>"}
key = "<number>"               

SQL = "select flags, address, date, text from message where address = '%s'"

output = """%s - %s
        %s         

"""

def getDate(epoch):
        return time.strftime("%a, %d %b %Y %H:%M:%S",time.localtime(epoch))

def main(dbfile, outputfile):
        outFile = codecs.open(outputfile, encoding='utf-8', mode='w')

        conn = sqlite3.connect(dbfile)
        c = conn.cursor()
        c.execute(SQL % (key))

        count = 0
        firstDate = ""
        lastDate = ""

        for row in c:
                flags = str(row[0])
                for name in names.keys():
                        if name in flags:
                                user = names[name]
                date = getDate(row[2])
                text = unicode(row[3])

                outStr = output % (user, date, text)

                if DEBUG:
                        print outStr

                outFile.write(outStr)

                # Store the first Date
                if count == 0:
                        firstDate = date
                lastDate = date
                count += 1
        outStr = "Date Range: %s - %s" % (firstDate, lastDate)
        if DEBUG:
                print outStr

        outFile.write(outStr)
        outFile.close()

if __name__ == "__main__":
        if len(sys.argv) < 3:
                print "%s  " % (sys.argv[0])
                sys.exit(1)

        dbfile = sys.argv[1]
        outFile = sys.argv[2]

        if not os.path.exists(dbfile):
                print "%s doesn't exist" % (dbfile)
                sys.exit(1)

        main(dbfile, outFile)

This creates a transcript like:

Matt - Mon, 04 Feb 2010 08:01:38
This is a text message

Other Person - Mon, 04 Feb 2010 08:02:38
This is the response.

Anyway happy backing up!
Needless to say I believe Shea was happy

FUSE (Filesystem in userspace) gives not only non privileged users a little more power, but also allows file systems that for legal reasons cannot be linked to the kernel life on a Linux system.

Thanks to this fuse layer, you can access gmail as a harddisk, ssh mount devices, and now you can finally access your iphone or ipod as a harddisk under linux thanks to the iFuse filesystem!
Here is a link to an ubuntu geek howto on installing and using iFuse!

Isn’t it great when you find a command line gem in Linux!

I’ve been using the ‘time’ command for years, and it was only up recently that I was told that the time command I’ve been using was the time command built into bash. There is a GNU time command, which has always been installed but I’d never used!

Chris was the one who stumbled across this while attempting to use some time options specified in the time man page which wasn’t working. That’s when he realised the man page was for GNU time not bash’s implementation.

GNU time is awesome!! You can grab a whole heap of useful stats about the process your timing, not just the time. To quote the GNU time website:

The `time’ command runs another program, then displays information about the resources used by that program, collected by the system while the program was running.

I had a play with GNU Time and set up some format strings I find kinda cool, I’m not using all information from the file, check the man page if you wanna add extra goodies but here is what I’ve got:
A formatted time display:

/usr/bin/time -f "Exit Status: %x\nCPU %%: %P\nMemory:\n Unshared: %D\t\t\tAvg Total Mem: %K\n Major Page Faults: %F\t\tMinor Page Faults: %R\n No. Swaps Out of Mem: %W\tNo. Invol Context Swiches: %c\tNo. Vol Context Switches: %w\nTime:\n Realtime: %E\t\tSystem Time: %S\t\tUser Time: %U"


In which the time output looks like:

Exit Status: 0
CPU %: 98%
Memory:
Unshared: 0 Avg Total Mem: 0
Major Page Faults: 0 Minor Page Faults: 3241
No. Swaps Out of Mem: 0 No. Invol Context Swiches: 35 No. Vol Context Switches: 2
Time:
Realtime: 0:00.29 System Time: 0.01 User Time: 0.27


And a simple tab seperated version easy for parsing:

/usr/bin/time -f "%x\t%P\t%D\t%K\t%F\t%R\t%W\t%c\t%w\t%E\t%S\t%U"

NOTE: The parse time command doesn’t have any labels, the columns are the same as the detailed format string.

Which looks like:

0 100% 0 0 0 3240 0 153 2 0:00.30 0.03 0.26


Unfortunately, the memory usage stats don’t seem to work… I’ll have to look in to why, is it a bug? am I doing something wrong? … not sure but if I figure it out I’ll let y’all know!.

UPDATE

Well as it turns out GNU Date is a little outdated…I had a quick look at the source code to see what was happening as to why the memory stats would only ever be 0. As it turns out to calculate memory usage they are using an algorithm that requires the number of kernel ticks. And for those of you who are unaware the Linux kernel has been tick-less since version ~2.6.21.

So no ticks equals no memory calculations.. hopefully GNU will rectify this problem cause as I stated earlier, GNU Date is a gem.