Local storage of arbitrary 'did' stats

[calmrat]

did focuses on pulling data from external sources about things that a user 'did' during a time period.

Many times we 'do' things that aren't stored in any external source though, and thus can't be queried later and included into our reports. eg 'drafted really important slides in libraoffice for project X'

Users should be able to save (aka, cache) this type of arbitrary 'did' stat locally.

[calmrat]

I think pulling in sqlalchemy as a dependency using local sqlite (~/.did/$USER.db) as the default store could be a good place to start. We know this way too we can later extend out to enable syncing/caching of data to other DB's as well (eg, some shared repo where people can sync and track each other's activities)

[calmrat]

some extra syntax that could later enable 'social' features

#tag 
@mention

[calmrat]

maybe the command could be 'idid'

idid [date] #projectx #othertag drafted really important slides in libraoffice

then

.did/config

[idid]
source = '~/.did/cward.db'

[Project X]
    type=idid
    filter="tag in ['projectx']"

and report

>> did --projectx
Project X
    '#projectx #othertag drafted really important slides in libraoffice'

[psss]

Thanks for creating the issue, Chris. I was thinking about the custom log module a little bit diferrently. I'll dump my thoughts here and we can discuss whether/how it's compatible with yours:

My idea was to store the custom logs in plain text file to allow easy editing and review of the content, something like this:

2015-11-22 11:22:33 Learned GitHub & TravisCI & ReadTheDocs

The format should support tags, for example

2015-11-22 11:22:33 Learned this [training]
2015-11-22 11:22:33 Created documentation [docs]
2015-11-22 11:22:33 Did something on the tooling  [tools]

Not sure about the tag format. I don't like that #tag does not support spaces in tags. But it's quite common. Don't know... As far as command line is concerned I'd suggest to keep a single command did and implement sub-commands similarly as git does:

did log now 'I did this'
did log yesterday 'I did that'

The config could perhaps look like this:

[work]
type = log
name = "Various stuff"
source = /home/psss/.did/work.log

List of the accomplished stuff could be done either as a single section:

* Various stuff
    * Learned this [training]
    * Created documentation [docs]
    * Did something on the tooling [tools] [did]

or grouped by the first tag:

* Various stuff (training)
    * Learned this
    * Learned that
    * Learned something else

* Various stuff (tools)
    * Did something on the tooling [did]
    * And a bit of community work [upstream]

That's all from my brainstorm. What do you think about it?

[liuhangbin]

I'd prefer the plain text style. Also I saw we could add 'Plans, thoughts, ideas' in the footer(even though not accomplished yet). like a todo list. We should be able to keep the todo list whenever we run did. and after we finished one item, then we can delete this item.

[TomasHofman]

To complicate thinks further, I will add some suggestions of my own :).

I can imagine three features that could be enabled by local storage:

1. some form of custom log messages - like you suggested

2. marking issues I was working on so they appear in the report even if their state wasn't modified, could be something like:

   did workon bz-12345
   

   and then bz-12345 would appear in the report under particular bugzilla section.

3. adding comments to issues, which would then appear in the report under particular issues. Could be

   did comment bz-12345 failed verification, need more effort
   

The tricky part would be linking the issue codes with real issues, and it would probably be more invasive than just adding another plugin...

[psss]

Quite interesting thoughts, Tomáš! Probably deserve a separate issue for detailed discussion. I'll just add one quick thought to the brainstorm: Adding Bugzilla comment with the first line starting with the did: prefix and summarizing what was done could result in displaying those one-line updates under the --bz-commented section if --verbose is enabled.

[psss]

Note to myself: Here's syntax I would like to use when I discover one of the joys of the week:

did log joy 'http://inside.github.io/vim-presentation/#/37 [vim]'

Seems it would make sense to use a separate file joy.log for keeping similar stuff together.

[psss]

Here's summary of my draft implementation proposal: Data saved as plaintext with possibility to include tags:

2015-11-22 11:22:33 Learned this [training]
2015-11-22 11:22:33 Created docs for that [docs]
2015-11-22 11:22:33 Did something on the tooling [tools]
2015-11-22 11:22:33 Drafted new automation tool [tools]
2015-11-22 11:22:33 Reviewed a bunch of patches [tools]
2015-11-22 11:22:33 Pushed did to pip, copr & fedora [joy]

Command line allows to specify log where the record should be saved:

did log joy 'Announced did, the feedback was very positive'
did log work 'Reviewed a bunch of patches [tools]'

Optionally, a date could be provided as well:

did log work yesterday 'Reviewed a bunch of patches [tools]'
did log work 2015-11-22 'Reviewed a bunch of patches [tools]'

The first configured log instance would be used by default:

did log 'I like short command lines'

Config file would state the header name and the log location:

[work]
type = log
log = /home/psss/.did/work.log
name = Work on various stuff

Now the resulting output would look like this:

* Work on various stuff
    * Learned this [training]
    * Created docs for that [docs]
    * Did something on the tooling [tools]
    * Drafted new automation tool [tools]
    * Reviewed a bunch of patches [tools]
    * Pushed did to pip, copr & fedora [joy]

The config would also allow to highlight specific tags by creating a separate section in the report output. In this way it's possible to have everything in a single file if preferred:

[work]
type = log
log = /home/psss/.did/work.log
joy = Joy of the week
tools = Work on the tooling
name = Various stuff

And here is the corresponding output:

* Joy of the week
    * Pushed did to pip, copr & fedora

* Work on the tooling
    * Did something on the tooling
    * Drafted new automation tool
    * Reviewed a bunch of patches

* Various stuff
    * Learned this [training]
    * Created docs for that [docs]

Ideas? Concerns? Chris, what do you think? Does this draft match your use case? Are you willing to implement this functionality? Or shall I assign the issue to myself?

[calmrat]

I like it.

Something tells me integrating Logging loggers into the mix could be an interesting solution rather than rewriting the print -> output code; LogHandlers could be extended easily to send and store log data to different backends.

Only reservation is still the overloading of did to both pull and now possibly push data somewhere.

tldr; consider breaking the push and pull components into separate executables /bin/did and /bin/idid

Doesn't it break the api when the first argument could be either 'default' or subcommand?

# did last: show all the pulled data about my historical activities
did last [week]

vs

# did log: submit a new 'did log' entry
did log [work yesterday 'Reviewed a bunch of patches [tools]']

The first argument's expected value seems too fuzzy. Maybe that's a feature?

idid work yesterday 'Reviewed a bunch of patches [tools]'

of some different command like that, same package, seems like a feasible alternative. Only overhead is the /bin/idid executable.

[calmrat]

OK, working off your proposal @psss mostly I have come up with one approach.

https://github.com/kejbaly2/did/commits/feat.log.git

It's functional already. It includes/depends on the fix.date branch I pull requested yesterday. Here's an example .did/config entry:

[logg]
type = logg
joy = Joy of the Week
tools = Working on Tooling
work = What I do

Then, an example usage:

did logg joy '@X brought me coffee from Italy! #ilovemonday'
did this week
Status report for this week (2015-10-12 to 2015-10-18 16:57:49.980797+00:00).
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Chris Ward <[email protected]>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* Logg stats: 1
* 2015-10-13 @X brought me coffee from Italy! #truelove [joy]

The logg command resulted in a text file with the loggs stored as psss suggested

cat ~/.did/loggs/did-cward.txt 
2015-10-13 X brought me coffee from Italy! #truelove

If you specify 'engine' in the config to point to a git repo you can use a git repo with branches as topics as the 'backend' for saving the logs.

[logg]
type = logg
engine = git:///home/cward/.did/loggs/did-cward.git
joy = Joy of the Week
tools = Working on Tooling
work = What I do

The same commands above will result in the following git repo

# config points to git backend
did logg joy '@X brought me coffee from Italy! #ilovemonday'
cd ~/.did/loggs/did-cward.git/
git log
commit 19e4013edd3ccfd52bfe47e9565e6cc03b19f30c
Author: Chris Ward <[email protected]>
Date:   Tue Oct 13 19:05:18 2015 +0200
    @X brought me coffee from Italy! #ilovemonday [joy]

commit 60e82b479e9e8792394c7adafca3c44616444496
Author: Chris Ward <[email protected]>
Date:   Tue Oct 13 19:05:17 2015 +0200
    New did repo added

The 'topics' must be specified in this version otherwise logg command returns a ConfigError. This ensures a description is always provided.

I haven't implemented user specific searching.

I have an issue still with the ambiguous nature of the CLI API though. These words, currently found in the README file which made, me laugh when I thought about how they're no longer true... "Usage is straightforward: ..."

Minimally Synopsis will need to now show both psuedo-sub-commands::

Synopsis

Usage for loading stats from configured sources is straightforward::

did [last] [week|month|quarter|year] [opts]

Usage for saving loggs to configured sources is too::

did logg [today|yesterday|$DATE] TOPIC 'LOGG RECORD' [opts] 

More over, --help is just for the standard usage. 'logg' usage isn't available via --help at this time, which is a significant problem. Any thoughts on how to solve this aspect of the forcing the logg functionality into the did CLI, but not doing it with actual sub-parsers would be appreciated.

[calmrat]

Here's my go at this feature

https://github.com/psss/did/pull/75

There's still a lot of potential for improvement, obviously. I think this is a good place to start though.

The basic use-case (appending logs to a text file) is covered. So is the experimental use-case for using git repo as the storage with topics as branches (with all commits synced to 'master') and commits as idid "logg" stats

Comments, questions, bugs, enhancements!?

[tcerna]

Hi, there are many great ideas which will improve this tool well. This is what I really would like to use.