[Top] [Contents] [Index] [ ? ]


Consume and digest your history.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

1. Overview

Keine is a lightweight time tracking system. "Time tracking" meaning she is something like a stopwatch. "Lightweight" meaning she is not that much more complicated than a stopwatch. Keine is a very simple solution to a not so simple problem.

This document is meant to be a manual to help others use this tool, but the manual part is rather sparse since Keine is such a simple tool (See section Usage). Most of this manual is more like a diary in behavioral engineering.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

1.1 Motivation

Short story: I want to find where all my time went.

Long story: I know where all my time went, I just want to know how bad it is.

I don't think I am too bad as an engineer, but even I miss milestones once in a while. It isn't because I have been playing games at work (which is not true) or because I work slower than I claim (which I don't believe). My theory is that my time are lost to unplanned things outside my control – debugging unexpected problems, being interrupted to answer questions, and so on and so forth. I want something to tell me where all the time went for each of the events.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

1.2 Solution

The first step to solving most problems would be to gather data, which calls for a tool to track time. It need not be too fancy, something that automatically timestamps each event and sum the deltas between them would be sufficient.

Some combination of tcsh's command history and a small Perl script might have solved it, but they still a bit short of what I wanted. Thus I set out to write my own tool, with the following requirements:

Non-goals (See section Patterns):

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

1.3 Results

I have implemented Keine to the requirements specified above, gathered enough data, presented the results to my manager, and planned my next quarter's schedule accordingly. More things are happening on schedule now, much to my joy and I believe to my manager as well.

What really happened? I can't really give details on what I do at work, but it's roughly:

The solution: plan for the unplanned, and move some of the middle 1/3 time to the scheduled 1/3 time to derive a more realistic schedule. There will always be some time lost that are just beyond your control, you just have to plan for it.

On a side note, I found the most productive day last quarter was actually when I went on vacation to do work. No time lost to context switches! Moral of the story – no amount of time management is going to better than just having more free time in the first place. You will need to be workaholic enough to have excessive vacation accrued, then even more workaholic to use vacation for work. Results not typical.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

2. Usage

There are only a few things you can do, and few ways to do them. I don't plan to add any more commands than these, and if there is anything I can remove, I would have removed them already.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

2.1 Running

Keine operates in one of two modes, either interactively:

./keine log.txt

Or in batch mode:

./keine < log.txt

In interactive mode, Keine runs in a loop that accepts event entries and commands, maintaining persistent state in the log file specified in command line arguments. In batch mode, Keine loads the log file, prints out a summary report (See section Report), and exit immediately. Most of this chapter describes the interactive mode.

After Keine starts, she prints out a message that looks something like this:

Loaded 28 events from sample_log.txt
Enter ? for help

From there on, you can perform actions described in the rest of this chapter. Entering ‘?’ will give a short summary of all possible commands.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

2.2 Events

Entering most text messages at the ‘>’ prompt causes events to be recorded, for example:

> deploy hoge
2007-10-01 20:12:34 deploy hoge

This means "I am going to start doing this task now". Keine echoes both the timestamp and event to confirm what you have just entered.

Besides this pattern, you would often want to record "I have been doing this other task since this time." In that case, you can prefix the line with a timestamp:

> 2007-10-31 14:00 halloween party
2007-10-31 14:00:00 halloween party
> 15:30 debug alerts
2007-10-31 15:30:00 debug alerts

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

2.3 Shortcuts

Since we often perform only a small set of tasks throughout the week, Keine supports a few shortcuts for recording recurring tasks:

These shortcuts can also be prefixed with a timestamp, follows the same semantics described in the previous section.

> i = interrupted by hage
2007-11-01 10:00:00 interrupted by hage
> 10:30 i
2007-11-01 10:30:00 interrupted by hage

These concepts came from ‘!?’, ‘!’ and ‘alias’ commands found in most shells. There are no plans make Keine any more shell-like than these functions.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

2.4 Report

Entering the empty string (pressing enter at the prompt) will cause Keine to print out a report summarizing all event times. The report looks like this:

Total time:    39779 (11h 2m)
    9637 (2h 40m): implement hoge  [24.2%]
    6728 (1h 52m): update documentation for  [16.9%]
       3983 (1h 6m): hogehoge  [10.0%]
       2745 (0h 45m): hoge  [6.9%]

    5764 (1h 36m): meeting with  [14.5%]
       3723 (1h 2m): piyo  [9.4%]
       2041 (0h 34m): piyopiyo  [5.1%]

    4282 (1h 11m): interrupted by hage  [10.8%]
    3676 (1h 1m): lunch  [9.2%]
    3209 (0h 53m): design fuga  [8.1%]
    2422 (0h 40m): dinner  [6.1%]
    1645 (0h 27m): review  [4.1%]
       1524 (0h 25m): hogera  [3.8%]
        121 (0h 2m): hogepiyo

    1643 (0h 27m): breakfast  [4.1%]
     773 (0h 12m): mail  [1.9%]

The events are grouped by common prefixes, then sorted by event duration. For example, the report above says that I have spent 1 hour and 52 minutes updating documentation, 45 minutes of which are for hoge, which took up 6.9% of all time tracked by Keine.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

2.5 Reload

Entering ! will cause Keine to reload the log file. This is useful for synchronization after after making edits in an external text editor. See section Editing text.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

2.6 Exit

To exit Keine and stop logging, send EOF (<Ctrl+D> on Linux, <Ctrl+Z> followed by enter on Windows).

<Ctrl+C> or SIGINT might also work, but you may or may not need to hit enter after the <Ctrl+C> depend on which version of Keine you are using.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

3. Patterns

Using Keine could change your life. Really.

More like, to make use of Keine, you may need to change some of your working habits, hopefully for the better.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

3.1 Recording events

First hurdle to using Keine would be just remembering to run the program in the first place. Keine is not obtrusive by design, and for the first few days you might even forget that she is sitting there waiting to for events to be entered. This might be a difficult habit to acquire depending on your current working style – in particular, how many different things you do every day.

If you find that using Keine is too much overhead, you are probably switching between tasks too often, or perhaps trying to multitask too much. In that case, recognize that you are doing mostly one thing at any one time, and record only the main event, and that should reduce the overhead significantly.

Another pattern that tends to happen often is not recording an event because they are perceived to have low costs ("it will take me just a few seconds to answer this mail! Just a few minutes! A few...") No task is really zero cost in time, if it's going to take you more than a few seconds to think about whether it's going to take a long time or not, you should just record the event anyways, and make it a habit.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

3.2 Entering time

Using Keine requires the ability to predict the future. Quite a difficult task for most people, even if it's only a few minutes into the future. Alright, so it's not quite that bad, but you really need to get into the habit of thinking about "this is what I am going to do next" as opposed to "this is what I just did."

For a while, I considered the alternative design of recording the end times of each event, as opposed to start times. It's either one or the other, and start time is more intuitive in many ways (including making manual edits to logs), so there was a bias toward using start times since the first version. But recording end times does have one advantage – most people do much better at remembering what they just did, compared to planning what they are going to do. Recording end times would therefore make the event log reflect reality more accurately.

Philosophically, though, I am against this backward looking business. Just by forcing yourself to think about what you are going to do next tend to makes it more likely that you will actually do them. Personally, in working with this peculiarity of Keine, I found myself more focused on fewer tasks, and having fewer context switches than before I started using Keine. This might be yet another habit that takes a bit of time to be acquired, but it's one that's worth the effort.

As a workaround, Keine does accept an optional timestamp in event entry, specifically for the purpose of tracking things you just did. But if you find yourself using that feature a lot, it may be time to try a change in working style.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

3.3 Entering descriptions

Keine's report generation algorithm is hierarchical, meaning you a bit of additional clustering for free if the events were entered a certain way. For example, if your daily activities are mostly code or documentation related, and you want to summarize how much total time you have spent on coding versus documentation, those words should be placed at the beginning of each event description.

> implementing hoge
> documenting hoge
> implementing piyo
> documenting piyo

This would produce reports that are summarized by activities:

   ####: implementing
      ####: hoge
      ####: piyo
   ####: documenting
      ####: hoge
      ####: piyo

If you want to summarize tasks by milestones, but don't care too much on what's done for each milestone, then the activities should be placed near the end:

> hoge code
> hoge docs
> piyo code
> piyo docs

This produce reports that are grouped by milestones:

   ####: hoge
      ####: code
      ####: docs
   ####: piyo
      ####: code
      ####: docs

Depending on which type of report is more useful to you, you should try to phrase the descriptions accordingly. Either way, try to develop a convention and stick to it. If you had entered totally random events, the summary report would be much harder and less useful to read.

Whenever possible, try to use one of the shortcuts to enter repeated events (See section Shortcuts). Besides saving a few keystrokes, they would make the event descriptions more consistent.

Maintaining a consistent convention over a long period of time is going to take some effort. Often it's useful to have a separate Perl script to unify the different log event variations, and have Keine generate report from the cleaned up version rather than the raw version (this is why Keine supports reading log files from pipes). Often I need to write this filter script anyways to anonymize the events before publishing the summary.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

3.4 Editing text

Keine has a reload function because there are no functions to edit the log events in memory (if you haven't noticed yet). This is intentional, as soon as I add one editing function or support one editing model, there would be no end to it. Keine is not a text editor.

As a workaround (actually, "intended behavior"), if you need to correct some previously entered entry, edit the log file in a real text editor, then use the reload function to make the edits visible to edit.

Of course, it would be much more straightforward if everyone simply enter all events correctly on the first try. It's not that hard, but may take some practice.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

4. Implementation

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

4.1 Log format

Keine keeps all state in a plain text file, which consist of two types of lines: event definitions and abbreviation definitions (lines that do not match either format are silently ignored).

2007-02-01 12:34:56 some event
# key = value

Log file format has been designed to make manual edits easy. Actually that's the only design consideration. No efforts has been made to make the log files parser friendly, reduce disk footprint, or protect data integrity. For all these other bits, consider rotating log files once every few months, and do your own backups and cleanups on old logs. Rotating logs once in a while also tend to make them more manageable for manual edits.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

4.1.1 Events

Events are plain text strings. They are mostly kept as is without much additional processing, other than stripping some redundant whitespaces.

Lack of additional processing also means lack of special support for internationalization. Assuming your terminal supports something other than ASCII, Keine will accept those bytes as is, but what happens after that isn't always well defined. Shift-JIS has been tested to work, UTF-8 will probably work. Regardless of the encoding, if the event tokens are not separated by half-width whitespaces, hierarchical summary reports won't work as expected (See section Report).

Keine treats most events equally, except ‘{{{’ and ‘}}}’, which marks the beginning and end of a session, respectively. ‘{{{’ is mostly a no-op, but useful for finding the corresponding ‘}}}’ on most editors. ‘}}}’ terminates the previous event without starting a new event. These were selected because they are language neutral, but also because they are the default fold markers supported by VIM, so manual edits with VIM may be slightly easier than some other editors.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

4.1.2 Timestamps

Timestamps are human readable time strings, in ‘yyyy-mm-dd HH:MM:SS’ format. One thing worth noting is that they are always in local time to simplify manual edits. This means if you are in a place affected by daylight saving time, and use Keine during those two or three special hours of the year, Keine might not handle those times correctly, so don't do that.

Timestamps must be valid, in the sense that they should be monotonically increasing. If they are not, Keine would print out line numbers of the offending lines when they are loaded, so that they can be corrected or removed. If they can't even be parsed (that is, they do not match ‘yyyy-mm-dd HH:MM:SS’ format), those lines are silently ignored.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

4.1.3 Abbreviations

Abbreviation definitions consist of (key, value) pairs, on lines prefixed by ‘#’. Keys must be one word (no spaces), but values can be anything.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

4.2 Internals

Keine is basically a list and a hashtable, the list contains all the events and the hashtable contains all the abbreviations. The basic input loop looks like this:

Because log files are written immediately (without buffering), a crash would not cause any loss in event data. This also means the writes are not quite efficient if user wants to enter many long events at once, but the expected usage pattern is one event every few minutes, so this shouldn't be a problem.

When summary is requested, Keine does the following:

This means the time needed to generate the report will be roughly proportional to the number of words in event log. It's fairly easy to craft an event log that will take a long time to summarize, but those logs are not expected to occur often in regular use. I do recommend (again) to rotate logs once in a while, to reduce report processing time for these operations.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

4.3 History

Keine is implemented first in Perl, then again in Perl, and finally in OCaml.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

4.3.1 Initial implementation

A "lightweight stopwatch" really isn't that complicated, basically two parts:

For the event entry part, I realized that there are a lot of overlaps with how shells work: "!" shortcuts to match prefixes, "!?" shortcuts to match substrings, aliases, etc. But I don't really want to reimplement any shell. And actually I don't really like the "!" and "?" too much, since they require pressing the shift key (personal quest of mine – avoid shift keys as much as I can to reduce stress on my fingers, although not to the point where I would stop typing uppercase characters).

Keeping the minimal and easy to implement features from shells, I had prefix and substring search, and also aliases all implemented in the first version of Keine. It was enough to enter most events efficiently without having to cut and paste a lot. Most other things are not there, including something as basic as "undo last entry". Keine is not a shell, nor is Keine a text editor.

For the time summary part, it was simply summing all the deltas, then presenting them in sorted order. For the initial version, there was also the concept of "interrupts" – time lost due to annoyances. This was to confirm my early theory of where times went by summing the annoyances separately. In practice, I realized that the same set of events always get marked as interrupts (and conversely, the other set of events never get marked as interrupts). This lead me to design the next version, which had a more clever summary routine, but otherwise treated all events uniformly.

In terms of actual code, this was all done in not too many lines of Perl (much smaller than this manual, at least). Part of the complexity came from treating the interrupts differently, all more reasons to drop it in the next version.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

4.3.2 Current implementation

The interrupt concept was a bit hackish in the first version, so I rewrote the summary to be more generalized and hierarchical. This means by entering events using a certain convention, times lost to interrupts and other annoyances will be tracked automatically (See section Entering descriptions).

Another major change was to accept optional timestamps before each event, so that I can back populate history more easily (See section Entering time). This is a less than satisfactory solution to a larger problem, but it's better than what I used to do before (editing the log in an external text editor and then reloading the log file).

The underlying code is actually quite different. Cleaner, I would say. Some of the abbreviation semantics broken in the previous version have been fixed here, and more comprehensive tests were added. I also made Keine use Term::ReadLine for input, so I get a bit of console text editing for free.

Then I realized... it's still all Perl, I can't get an executable for Windows people.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

4.3.3 Final re-implementation

For our Perl-less friends, I rewrote Keine in OCaml so that I can get an executable. It didn't have to be OCaml, I did it mostly for kicks. It's tested using the exact same regression test as the Perl version, so the behavior should be mostly identical. Known differences:

This being OCaml, I can claim the code works entirely as intended since it compiles, ha!

Really, all implementations of Keine have been tested for quite a while, and I haven't had to change the code ever since I started writing this manual. There are still some potential optimizations and cleanups left to do, but as far as I care, the code is good enough to be frozen.

That is to say, I don't really want to spend more time on this time tracking software. Keine was developed entirely outside of Company time, since I thought using Company time to track Company time has a bad ring to it. That said, it's amazing that what used to be a short script written over one weekend ends up to be this executable two months later (but does more or less the same thing), so I am obviously not very efficient at using my time outside of work. Perhaps I need to gather data on how my weekends are spent as well...

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

4.3.4 The fine manual

Like most other projects these days, the last thing I do is to complete the documentation. It took especially long for Keine, mostly to cover the minimalistic design and implementation. There are good reasons (and excuses) for minimalistic programs, but I don't think documentation should ever be minimalist.

Of course, it shouldn't be one large blob of unorganized text either, it should be just enough so that when I tell people RTFM, they can find what they want quickly.

[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

4.4 Keine

Keine was named after Kamishirasawa Keine from Touhou Eiyashou, for her ability to consume and create history.

Extra was easier than Lunatic, heh.

[Top] [Contents] [Index] [ ? ]

Table of Contents

[Top] [Contents] [Index] [ ? ]

About This Document

This document was generated by omoikane on November, 5 2007 using texi2html 1.78.

The buttons in the navigation panels have the following meaning:

Button Name Go to From 1.2.3 go to
[ < ] Back Previous section in reading order 1.2.2
[ > ] Forward Next section in reading order 1.2.4
[ << ] FastBack Beginning of this chapter or previous chapter 1
[ Up ] Up Up section 1.2
[ >> ] FastForward Next chapter 2
[Top] Top Cover (top) of document  
[Contents] Contents Table of contents  
[Index] Index Index  
[ ? ] About About (help)  

where the Example assumes that the current position is at Subsubsection One-Two-Three of a document of the following structure:

This document was generated by omoikane on November, 5 2007 using texi2html 1.78.