WorkMachine

I am working on a project called WorkMachine which allows the creation of arbitrary workflows which might or might not involve human workers. It is generic enough that any workflow can be created.

What is it's purpose? There is a lot of duplication happening in effort when it comes to creating workflows. Companies need workflows of some kind for internal work whether it be to verify a document has correct data to augmenting with machine learning output with human input. Most of the time this is done manually or in ways which can be made more efficient and optimal. That is the goal of WorkMachine to simplify and make more efficient the waste in that process.

I hope that this will make it easier to facilitate creation of those workflows.

Instapaper Kindle Two way Sync

Well today might not be the best day for Marco of Instapaper with the introduction of Reading List feature in iOS and Mac OS X by Apple. But I love Instapaper nonetheless. But one of the annoyances I have is that there is no two way sync with the Kindle. Well there is http://goephemera.com/ but it is unmaintained and no longer works for multiple articles.

So in a binge of insomniac coding I decided to write such a sync. Now, I realize that I am not going to be putting a lot of work into maintaing this so take it as is and improve it as you like.

Somethings you will need:

Leave a comment if you need help. But this does assume you have knowledge of the command line.

require 'rubygems'
require 'oauth'
require 'json'
require "sqlite3"

# Get your key here. http://www.instapaper.com/main/request_oauth_consumer_token
INSTAPAPER_CONSUMER_KEY = 'getyourownkey'
INSTAPAPER_CONSUMER_SECRET = 'getyourownkey'
INSTAPAPER_USERNAME = 'user@mail.com'
INSTAPAPER_PASSWORD = 'password'

KINDLEGEN = "~/Dropbox/KindleGen/kindlegen"

class Instapaper
  Url = "http://www.instapaper.com"

  def initialize(consumer_key, consumer_secret)
    @consumer_key    = consumer_key
    @consumer_secret = consumer_secret
  end

  def authorize(username, password)
    @consumer = OAuth::Consumer.new(@consumer_key, @consumer_secret, {
        :site              => "https://www.instapaper.com",
        :access_token_path => "/api/1/oauth/access_token",
        :http_method => :post
      })

    access_token = @consumer.get_access_token(nil, {}, {
        :x_auth_username => username,
        :x_auth_password => password,
        :x_auth_mode     => "client_auth",
      })

    @access_token = OAuth::AccessToken.new(@consumer, access_token.token, access_token.secret)
  end

  def request(path, params={})
    @access_token.request(:post, "#{Url}#{path}", params)
  end
end


$instapaper = Instapaper.new(INSTAPAPER_CONSUMER_KEY, INSTAPAPER_CONSUMER_SECRET)
$instapaper.authorize(INSTAPAPER_USERNAME, INSTAPAPER_PASSWORD)


$db = SQLite3::Database.new('i2ksync.db')
$db.results_as_hash = true

# status 1. download, 2. still_there 3. archived
$db.execute(%{
   create table if not exists bookmarks (
                id integer PRIMARY KEY,
                title text,
                status text,
                UNIQUE(id))
})

# Create the path if it doesn't exist
path = File.join("/Volumes/Kindle/documents", "_instapaper")
Dir.mkdir(path) unless File.exists?(path)

num_not_there = 0
still_there_ids = []

still_there = $db.execute("select * from bookmarks where status = 2")
still_there.each do |bookmark|
  unless File.exists?(File.join(path, "#{bookmark['title']}.mobi"))
    $instapaper.request("/api/1/bookmarks/archive?bookmark_id=#{bookmark['id']}")
    $db.execute("update bookmarks set status = 3 where id = #{bookmark['id']}")

    num_not_there += 1
  else
    still_there_ids << bookmark['id']
  end
end

articles = JSON.parse($instapaper.request("/api/1/bookmarks/list?limit=#{25 - num_not_there}&have=#{still_there_ids.join(',')}").body)

articles.each do |article|
  next unless article.has_key?('title')

  filename = article['title']
  filename = "Article #{article['bookmark_id']}" unless article['title']

  file_path = "#{filename}.html"

  File.open(File.join(path, file_path), 'w') do |f|
    f << $instapaper.request("/api/1/bookmarks/get_text?bookmark_id=#{article['bookmark_id']}").body
  end

  $db.execute("insert or ignore into bookmarks (id, title, status) values (?,?,?)", article['bookmark_id'], filename, 2)
end

`cd #{path} && find . -name "*html" -exec #{KINDLEGEN} {} \\;`
`cd #{path} && find . -name "*html" -exec rm {} \\;`

Gnus configuration

Here is the configuration for Gnus.

(load-library "smtpmail")
(load-library "nnimap")
(load-library "starttls")

(setq gnus-select-method '(nntp "news.gmane.org"))
(setq gnus-secondary-select-methods
      '((nnml "")
        (nnimap "imap.gmail.com"
                (nnimap-address "imap.gmail.com")
                (nnimap-server-port 993)
                (nnimap-authinfo-file "~/.imap-authinfo")
                (nnimap-expunge-on-close 'never)
                (nnimap-stream ssl))))

(setq smtpmail-starttls-credentials '(("smtp.gmail.com" 587 nil nil))
      smtpmail-smtp-server "smtp.gmail.com"
      smtpmail-default-smtp-server "smtp.gmail.com"
      send-mail-function 'smtpmail-send-it
      message-send-mail-function 'smtpmail-send-it
      smtpmail-smtp-service 587
      smtpmail-auth-credentials
      '(("smtp.gmail.com" 587 "ykabhinav@gmail.com" nil)))

(add-hook 'gnus-topic-mode-hook 'gnus-topic-mode)

;(setq gnus-summary-rescan-group 'all)
;(setq gnus-permanently-visible-groups ".*INBOX")


(setq gnus-thread-sort-functions
      '(gnus-thread-sort-by-most-recent-date
        (not gnus-thread-sort-by-total-score)))

(gnus-demon-add-handler 'gnus-group-get-new-news 5 nil)



(cond (window-system
       (setq custom-background-mode 'light)
       (defface my-group-face-1
         '((t (:foreground "Red" :bold t))) "First group face")
       (defface my-group-face-2
         '((t (:foreground "DarkSeaGreen4" :bold t)))
         "Second group face")
       (defface my-group-face-3
         '((t (:foreground "Green4" :bold t))) "Third group face")
       (defface my-group-face-4
         '((t (:foreground "SteelBlue" :bold t))) "Fourth group face")
       (defface my-group-face-5
         '((t (:foreground "Blue" :bold t))) "Fifth group face")))

(setq gnus-group-highlight
      '(((> unread 200) . my-group-face-1)
        ((and (< level 3) (zerop unread)) . my-group-face-2)
        ((< level 3) . my-group-face-3)
        ((zerop unread) . my-group-face-4)
        (t . my-group-face-5)))

(setq-default
 gnus-summary-line-format "%U%R%z %(%&user-date;  %-15,15f %* %B%s%)\n"
 gnus-user-date-format-alist '((t . "%Y-%m-%d %H:%M"))
 gnus-summary-thread-gathering-function 'gnus-gather-threads-by-references
 gnus-thread-sort-functions '(gnus-thread-sort-by-date)
 gnus-sum-thread-tree-false-root ""
 gnus-sum-thread-tree-indent " "
 gnus-sum-thread-tree-leaf-with-other "|-> "
 gnus-sum-thread-tree-root ""
 gnus-sum-thread-tree-single-leaf "\-> "
 gnus-sum-thread-tree-vertical "│")

(setq gnus-use-cache t)

Thoughts on Using Emacs

I converted to Emacs all of seven months ago and I wish I started earlier. The editor is way powerful, it is a Lisp Machine for the rest of us. Considering its origins it makes sense. It’s an editor you build up to the way you think instead of stooping down to think the way the editor works.

I won’t recommend jumping full into the Emacs way using ERC, GNUS, VM, and all, but start off small. I jumped in by using org-mode.

Alos, learn something new everyday, learn Elisp and add small custom functionality. You will be spending most of your time in your editor if you are a coder so might as well learn its language. Always be sharpening your tools!

mutt with Emacs and BBDB

I wanted to really use Emacs and BBDB with my mail client and that’s partially why I spent so much time trying out different Emacs mail clients. But I can do the same with mutt and Emacs + BBDB and don’t have to bang my head at the slowness.

First step is to add the following to your .muttrc:

set edit_headers
set editor = "emacsclient -a emacs"

This allows you to edit the headers when you are editing the message and also use emacsclient. This assumes that you are running Emacs in daemon mode and will start a new instance if it isn’t. I also set the mail-mode for the mutt compose buffers. So add the following in your .emacs file.

(start-server)

(add-to-list 'auto-mode-alist
             '("\\.*mutt-*\\|.article\\|\\.followup"
                . mail-mode))

Now install BBDB I am using BBDB 3.0 which is still in development but quite stable. I have the following in my .emacs file.

(require 'bbdb)
(require 'bbdb-message)
(bbdb-insinuate-message)

(if bbdb-complete-mail
    (define-key mail-mode-map "\C-t" 'bbdb-complete-mail))

Now you will be able to use bbdb for completing names and have Emacs’s mail mode at your disposal when editing emails. Sort of like this post.

Emacs VM config for Gmail IMAP

Make sure that you have stunnel 4.33 or lower for the trunk version of VM. https://launchpad.net/vm

(add-to-list 'load-path "~/Dropbox/vm/lisp")
(require 'vm-autoloads)

(setq mail-user-agent 'vm-user-agent)

(setq vm-folder-directory "~/Mail")
(setq vm-primary-inbox "~/Mail/vminbox")
(setq vm-crash-box "~/Mail/inbox.crash.mbox")

(setq vm-imap-expunge-after-retrieving nil)

(setq vm-mutable-frames nil)
(setq vm-imap-account-alist
      '(("imap-ssl:imap.gmail.com:993:login:ykabhinav@gmail.com:*" "gmail")))

(setq vm-primary-inbox "imap-ssl:imap.gmail.com:993:inbox:login:ykabhinav@gmail.com:*")

(setq vm-debug t)
(setq vm-keep-imap-buffers t)
(setq vm-imap-log-sessions t)

(setq vm-use-menus 1)
(setq vm-stunnel-program "stunnel")

(setq send-mail-function 'smtpmail-send-it
      message-send-mail-function 'smtpmail-send-it
      smtpmail-starttls-credentials
      '(("smtp.gmail.com" 587 nil nil))
      smtpmail-auth-credentials
      (expand-file-name "~/.authinfo")
      smtpmail-default-smtp-server "smtp.gmail.com"
      smtpmail-smtp-server "smtp.gmail.com"
      smtpmail-smtp-service 587
      smtpmail-debug-info t)
(require 'smtpmail)

(setq mail-default-headers "From: Abhi Yerra <ykabhinav@gmail.com>\n")


(setq  vm-mime-type-converter-alist
      '(("text/html" "text/plain" "elinks -dump /dev/stdin")
        ("message/delivery-status"  "text/plain")
        ("application/zip"  "text/plain" "listzip")
        ("application/x-zip-compressed"  "text/plain" "zipinfo /dev/stdin")
        ("application/x-www-form-urlencoded"  "text/plain")
        ("message/disposition-notification"  "text/plain")
        ("application/mac-binhex40" "application/octet-stream" "hexbin -s")))

(define-key vm-mode-map "#" 'vm-imap-synchronize)

Return to mutt

I’ve sort of given up on Emacs mail clients. I really tried, I did. But all of them had their own problems and at the end of the day I need to get work done. I can’t worry whether my mail client is working. But I wanted a terminal interface to my mail client because I don’t really like using the gmail interface nowadays, it’s a bit slow and takes a while to load, etc.

I’ve decided to head back to mutt. One of the initial things I disliked about mutt was wrongfully thinking I couldn’t see my imap folders with the unread count. This was a wrong assumption. If you press ‘c’ in the pager or index views and then press you can see the folders/unread count. You can also press ‘y’ to get the same view.

set folder_format="%2C %t %N %8s %d %f"

Now the only problem I have is what to do about contact completion. That will be the next battle I face.

However, I do think mutt should incorporate a few more features within its fold. I mean I could totally see it having a Google Voice and Twitter integration.