Category Archives: Blog

Backup your Google Apps (or Gmail) e-mail over IMAP using imapsync

Posted on August 8, 2010 by Tony Duckles

I have confidence in Google’s ability to keep their e-mail service up-and-running and keep proper backups of data, but my e-mail history is my data and I like to have my own copy of it. Since Google provides access to your Google Apps (and Gmail) e-mail over IMAP, you can do all kinds of things using standard tools, e.g. synchronize your Gmail e-mail to a local mailbox using IMAP.

I run an Ubuntu box at home and it was easy to install the “dovecot-imapd” package to get an IMAP server installed. Since my box is behind my router/firewall, I’m wasn’t that concerned with tweaking Dovecot’s default configuration, but I’m sure you could fiddle with the config to ensure that Dovecot only binds to 127.0.0.1.

From there, it’s just a matter of using “imapsync“, just like I ended-up using previously to initially transfer all my e-mail to my Google Apps account.

Here’s the script:

# Sync e-mail from username@yourdomain.com to localhost, using IMAP
imapsync --host1 imap.gmail.com --user1 username@yourdomain.com
         --passfile1 /path/to/gmail_mirror.passfile1 \
         --host2 localhost --user2 username
         --passfile2 /path/to/gmail_mirror.passfile2 \
         --ssl1 \
         --useheader 'Message-Id' --skipsize --allowsizemismatch \
         --syncinternaldates --noauthmd5 -nofoldersizes\
         --split1 100 --split2 100 \
         --regextrans2 's/\[Gmail\]/username\@somedomain/' \
         --include "All Mail|Sent Mail" --delete2 --expunge2

The “--regextrans2” option rewrites IMAP folder-names on-the-fly, so that my local IMAP folder structure can be different than the structure on Gmail’s server. For example, the top Gmail IMAP folder is “[Gmail]” which wasn’t all that useful for me, so instead I rewrote that top-level folder to be “username@somedomain” so that the local folder name (e.g. in ~/mail/) would match the source e-mail address.

You can also use the “--include” option to decide which IMAP folders to copy. I opted to just copy “All Mail” and “Sent Mail”, which gives me a copy of all my mail but doesn’t preserve any information about the labels I might have had assigned to those messages in Gmail.

The initial copy will definitely take a few hours (or more), depending on how much e-mail you have in your Gmail account. But this works great for me and stores the mail in “mbox” format locally so I can even access the mail locally via mutt/alpine/etc.

Posted in Blog | Tagged , , , , | Leave a comment

Moving E-Mail from Gmail to Google Apps over IMAP using imapsync

Posted on August 7, 2010 by Tony Duckles

Two years ago, I transitioned from using regular Gmail (i.e. “…@gmail.com”) to setting up Google Apps for “nynim.org” (i.e. “…@nynim.org”). I had found the following article back then which gave me helpful hints on how to use imapsync to push data to my new Google Apps e-mail address:
http://gemal.dk/blog/2008/04/08/completed_the_gmail_migration/

Here is the final script I ended-up with:

#!/bin/sh
# Copy all e-mail from source@gmail.com to target@yourdomain.com (Google Apps)
imapsync \
  --host1 imap.gmail.com --port1 993 --user1 source@gmail.com \
  --passfile1 /path/to/gmail_mirror.passfile1 --ssl1 \
  --host2 imap.gmail.com --port2 993 --user2 target@yourdomain.com \
  --passfile2 /path/to/gmail_mirror.passfile2 --ssl2 \
  --useheader 'Message-Id' --skipsize \
  --syncinternaldates --noauthmd5 -nofoldersizes \
  --split1 100 --split2 100 \
  --maxage 50 \
  --regexmess 's/Delivered-To: source\@gmail.com/Delivered-To: target\@yourdomain.com/g'

The “--syncinternaldates“, “--useheader 'Message-Id'“, and “--skipsize” options are all recommended by the imapsync FAQ (search for “Gmail”):
http://www.linux-france.org/prj/imapsync/FAQ

I opted to use the “--passfile1/passfile2” options rather than passing in a plain-text password in via a command-line param for two reasons: first because anyone with access to your system can use “ps” to view active processes and hence would see your password plain-as-day; second because it just better abstracts the script-logic from the password-text, and we can control the file-permissions of those password files.

All your Gmail labels should sync-over automagically. Since this is going Google-to-Google, because Google presents labels as separate IMAP folders in their Gmail IMAP implementation, the process above should sync and preserve all your Gmail labels for free.

It’s been a two years since I’ve used this script, but I seem to remember it working pretty painlessly. I wanted to share it here for anyone else who might be looking to do this same thing.

Posted in Blog | Tagged , , , | Leave a comment

Nynim.org: New & Improved

Posted on December 20, 2008 by Tony Duckles

I’ve resurrected my nearly abandoned website and breathed new life into it! I’m now running SweetCron, an open-source project for rolling you own personalized lifestream.

I find that I don’t always have the inspiration or motivation to write blog posts that often, but I do have a Web 2.0 foot-print that other people might want to watch, whether it be sharing interesting news tidbits on Google Reader to bookmarking useful webpages on Del.icio.us, to what songs I love on Last.fm.

Hence, the birth of this lifestream: all my various online activities aggregated and presenting in a unified “lifestream”. This gives others an easy way to check-out what I’m up to. If you have any comments or feedback, drop me an e-mail!

Posted in Blog | Tagged , , | Leave a comment

Maintaining Online Identities

Posted on September 11, 2007 by Tony Duckles

I stumbled across this neat site called Clipperz a few weeks back. On the surface, it’s “a free and anonymous online password manager“. This was immediately interesting to me because I’d really like to find a (secure!) web-based password manager. More and more lately, I’ve been trying to find web apps that can replace the desktop applications I use on a daily basis. It’s just so incredibly convenient to have the portability to be able to do everything you need from inside a web-browser, and more importantly to have the persistent-state data stored on the server-side rather than on each individual machine I use throughout the day. I’m already a huge fan of webapps like del.icio.us (for tracking all my bookmarks) and Google Reader (for all my RSS aggregation and reading needs), and finding a web-based password manager would be ideal. It really becomes a chore remembering what username or e-mail address I registered with at various websites: Amazon, eBay, Newegg, Google, Yahoo, credit cards, travel sites, online bill pay, banks, etc. (Oy!) It’s just a lot of identities to juggle. I’ve thought about using KeePass or something, but that’s a desktop app and I’d have to keep my database in-sync between work and home.

What really interested me about Clipperz is their claim at being a “zero-knowledge” web-application: they only store encrypted data on their server-side, and use Javascript to decrypt the data client-side based on the username and password which you supply. Your password is never sent to the server at all; it’s merely used to decrypt the data locally. That’s a pretty slick idea, IMO. But, I’m also a little nervous about trusting some third-party website will all my personal sensitive online account information. So, I’m tempted to sit down and write my own webapp to recreate that wheel, except that it’s all code I can vouch for and know that it’s not secretly sending my data off to some third-party site. You can already find open-source Javascript-based implementations of the various crypto algorithms that I’d need to use. And it seems like a simple AJAX-based webapp to save/load data from the server.

Also really intriguing is Clipperz’ one-click-sign-in feature. They have a Javascript bookmark which pulls HTML FORM information off a given login page and then allows you to link the various FORM INPUT fields with the appropriate data-fields on the “card” for that website (e.g. username, password). The one-click sign-in then just has to do a HTTP POST to the desired web-address with the correct FORM data to act just like the real login page. It’s simple enough, but that seems like the real time-saver here. Not only could you have a webapp which stores all your personal data, but it also provides a quick launchpad to login any site you need. That just takes it a step further and makes the webapp almost a blackbox: you don’t really need to know what your username and password is anymore because the webapp database knows it and can feed it to the target website’s login form for you.

This just seems like a really awesome idea, and I’m really tempted to just use Clipperz natively so that I don’t have to re-invent the wheel, but I’m still just nervous about using someone else’s website as a database to store all my personal sensitive data. It’s basically just a trust issue, and I don’t think I have any good reason to trust that their Javascript code will never do anything malicious. I’d much rather control the data myself on my private webserver. If only Clipperz was a SourceForge project… ;)

Posted in Blog | Tagged , , | Leave a comment

Simple Off-Site Backups…?

Posted on September 11, 2007 by Tony Duckles

Lately I’ve been consumed with the idea of getting more serious about backing-up all the data I have. I tend to be a digital pack-rat: my music collection, digital photos I’ve taken, all the songs I’ve ever recorded, various tarballs filled with code-stuffs for college projects I did, etc. There’s a wealth of nostalgia there, and I realized that I really wouldn’t want to lose a lot of that stuff.

I already use a quasi-RAID system: I have two identical hdd’s in my file-server box and I rsync the “master” drive to the “slave” drive every so often. Not only does this provide me some amount of rollback-ness (i.e. because I only rsync so often), but the decision to not RAID-mirror the drives was intentional: if the filesystem or partition table on the RAID somehow became corrupted, all my data could be lost.

But…what if my house burns down? (*gasp* Oh noes!!) Yes, that would be sad indeed! So, rather than just providing a single layer of redundancy locally, if I really want to invest in the survival of my important data, I really need to spread that data around; I need to diversify.

I’ve looked around at various web-based back-up solutions like Amazon’s S3 service, but those don’t seem very optimal for me because of the amount of data I want to backup (~150GB) and because it seems like it would be a PITA to do a full restore over my home cable internet pipe. Not to mention the monthly fees, paying someone else to store my data safe and sound. But, dare I trust my important (and partially sensitive) data to a stranger?

Currently, I’m tempted by a seemingly simple solution: just get an external USB hdd, mirror my data once locally, and then throw it at a friend’s house and use rsync to keep it up-to-date. The main cost involved is the cost of the new hdd; there’s no monthly fee because I already need to pay for my internet-access. And I can even return the favor by hosting drives on my end too. And this could even be expanded to a multiple people, if you wanted to back-up your data in multiple off-sites. This seems almost too easy to me, but it seems perfectly effective. Anything, it’s making use of the hidden geek-factor: you’re a geek and you have geek friends, so why not make the most of it and use them for geeky endeavors like helping each other backup each other’s data? ;)

The main problem I have with this plan is that my data wouldn’t be encrypted at all. I’m not sure how paranoid I really need to be about my friends snooping around my data. Though, I think this is basically the general idea as: what would happen if someone stole your computer? So, that’s really more an argument that I should be locally encrypting my data so that even if prying eyes were to get at my local/master copy, they still wouldn’t be able to do much with it.

Has anyone else given thought to getting more serious about backing up their data?

Posted in Blog | Tagged | Leave a comment

del.icio.us director

Posted on May 19, 2007 by Tony Duckles

Yes, somewhere along the line…I became a geek. I started using multiple computers, not just between home/work/school/etc., but even multiple computers at home. (Though, I mainly use my MacBook for all my casual web-surfing needs these days at home…) I was always annoyed by having different bookmarks on the different computers I used, which meant frustration when I had to remember which computer I bookmarked that useful such-and-such on. I wanted a way to have a centralized webpage for my bookmarks and hence could access from wherever I was. So, inspired by an entry on 24ways.org, I created one. It was my little pet project, and it was good…

Then I found out that someone else had already done the same thing and they had done it better. Enter del.icio.us. (Ooh, the pretty pretty tag-clouds! Yes, my tag-cloud is quite delicious indeed…) Tags are such a simple concept, but oh-so effective. It just makes so much sense to have a webapp where you save bookmarks and can assign arbitrary tags to each entry. del.icio.us then makes it very easy to navigate around your list of links by tags, even providing nifty “easy” URLs like http://del.icio.us/tduckles/madison, which shows me everything I’ve tagged with “madison”.

Even better, people out there have written even slicker tools to access and navigate around your del.icio.us bookmarks. John Vey wrote this amazingly cool tool called del.icio.us direc.tor. In it’s native form, it’s meant to be a bookmarklet which you use at api.del.icio.us after you’ve done all your HTTP-Auth stuff, so that the JavaScript code doesn’t need to worry about auth when trying to access your bookmarks.

Not only is this a uber-slick UI for navigating around your tag-tree and for quickly finding a link based on (real-time!) searching, but the page itself also goes into the design and architecture of direc.tor. This was a really fascinating read for me, from a web-dev perspective. What really blew my mind about this is the whole concept of using AJAX methods to grab the del.icio.us XML DOM, and then use XSLT and XPath features which are built-in to most modern browsers to do all the searching in real-time on the client-side. So, really, the whole bottom pane of direc.tor is the output of an XSTL on the del.icio.us XML DOM of your bookmarks. I had never realized that you could do XSLT and XPath operations via Javascript before. This was a watershed moment for me, because a whole new perspective on web-development opened up for me: you could use XMLHttpRequest to grab an XML document, and then use Javascript to do an XSLT to transform that XML data into HTML or whatever presentation-layer you wanted. In the past, I had always only thought of AJAX calls returning some chunk of HTML which you could then insert somewhere into the document DOM.

Posted in Blog | Tagged , | Leave a comment

Google Reader

Posted on May 19, 2007 by Tony Duckles

I really dig the whole web-based RSS aggregator/reader thing. For the past year or so, I’ve been using reBlog on my Linux box at home. It had it’s quirks, but it was great to use.

Enter Google Reader. Being the show-off’s they are, Google has created a fantastic RSS reader web-app. I’ve ditched my home-based reBlog setup for using Google Reader for all my RSS feed reading needs. Here are the things which really won me over:

  • Super-slick UI, with fantastic keyboard navigation support.
  • Nifty AJAX tricks, like only loading enough items into the ListView to fit on screen initially and then fetching more as you scroll down. (Of course, they do this for performance reasons, but it’s still implemented very cleanly and just “feels” nice.)
  • You can “star”/flag entries that are useful, and then go back and look at all the articles you’ve flagged. It remembers the timestamp of when you flagged the entry, which I find handy. (Though most useful links I consume in a del.icio.us-fashion these days…)
  • You can easily sort either by Oldest or Newest, and it saves that preference on each “view” (i.e. viewing entries from just one feed vs all entries) you have.
  • Has an “infinite” scrollback feature, to allow you to scroll backwards through all the posts which Google has ever fetched for that feed.
  • It’s free. ;)

I would definitely recommend Google Reader to anyone that uses a fat-client RSS reader/aggregator, especially if you want something that does the aggregating 24-7 for you and is accessible from any computer.

Posted in Blog | Tagged , , , | Leave a comment

Firefly & Fan Documentary DVD

Posted on July 19, 2006 by Tony Duckles

I finally succumbed to recommendations of friends and picked-up the Firefly TV series on DVD…and it definitely didn’t fail to impress. It’s the best sci-fi show I’ve seen in a long time. It reminds me of ST:TNG in some ways, in so much as it’s very character-driven, with engaging characters who are developed very nicely. I would certainly recommend it to any sci-fi fans who haven’t had a chance to see it yet.

The fan community for Firefly seems to be alive and strong, and it looks like a fan-compiled Firefly documentary is on the verge of being released. I’m still debating picking it up… I know I can get fanatical about some things, but I’m not sure if I’m too that point with Firefly yet. I certainly wish that the TV series hadn’t been cancel mid-1st-season.

http://www.donetheimpossible.com/

Posted in Blog | Tagged , , | Leave a comment

reBlog – Web-Based, Server-Side RSS Aggregator

Posted on March 21, 2006 by Tony Duckles

Like many people these days, I use RSS feeds to keep up on news from various feeds. But, with some of the higher traffic sites, since the RSS XML file only keeps the last 40 headlines or so, if I don’t update the feed every ~12 hours, then I’ll miss some posts. Since my iBook has been on the fritz lately, I’ve just been leaving it on 24-7, so I was able to have it auto-refresh every few hours.

This weekend, I was suddenly struck with the idea that it would be cool to have an RSS reader with a web-interface, and the RSS fetcher could run as a crontab job. This has the benefits of being able to be accessed anywhere, and to fetch new posts automagically, without needing some desktop computer somewhere to handle that.

At first, I thought it might be a cool home project to write up. Using some AJAX techniques, I could come up with a pretty slick interface. But, surely someone else must have had this same idea too, so I did some searching for existing open-source projects like this. And, sure enough, enter reBlog. This article gives a good overview of the feature-set, and also has a Google Video giving a short workflow run-through.

It uses a lot of AJAX tricks, and it even has full keyboard navigation shortcuts. It uses “,” and “.” as the default row up/down, but I was able to fiddle with the source code to map “j” and “k” to these same functions, so I could use it vi-style. ;) So, it’s really easy to scroll through posts, mark them as read or flag them for follow-up. All the posts you flag (or “Publish” in reBlog terminology) go to a different bucket, which can be rendered to a RSS feed itself, I guess so you can point some other RSS reader to the list.

I’m just very impressed with the project. It’s really just what I was looking for. The 2.0 version is still in beta, and I’m hoping by the final version they offer more flexibility/customizing. So, if you’ve ever wanted a web-based RSS reading solution, check out reBlog.

Posted in Blog | Tagged , , | Leave a comment