May 10, 2013

GSOC Proposal: Code Recommenders for Javadoc!

Hi,

my name is Patrick Gottschaemmer and I'm a computer science student at the Darmstadt University of Technology. Besides that, I work for the Code Recommenders project as a Eclipse contributor since October 2012. I'd like to participate in the Google Summer of Code 2013 and this is my GSOC Proposal about an integration of Code Recommenders statistics into Javadoc. Any feedback would be welcome!

Abstract



The Eclipse Code Recommenders project [1] provides a smart Code Completion for the Eclipse IDE in Java, based on the context where code completion is triggered and gained knowledge by analyzing source code of open source projects. These machine learned statistics of API and framework usages (e.g. JFace or the JDK itself) can be summarized and made abstract from the current code context. This way, typical usage patterns of API calls or code snippets (lines of code which are usually appear together) can be identified.
These gained informations can be a great advantage when learning how to use a new API or a framework.
Furthermore, a directly integration into the standard documentation platform of the programming language would be useful, because that is normally the starting point for developers which want to learn something about core classes, package usages and interfaces in frameworks or APIs. In this GSoC Proposal, I’d like to implement the following additional features for the Javadoc generation tool [2]:
  • Integration of Code Recommenders statistics for packages, classes and methods on Javadoc

  • A little helper utility which fetches source code from a repository, runs javadoc on it, integrates the Code Recommenders statstics and uploads the updated documentation back to the repository

  • Enable developers to directly edit, comment or rate on API documentation and new code snippets, which are provided by the Snipmatch subproject of Code Recommenders or posted by developers themselves

Introduction



The standard Code Completion of the Eclipse IDE (or Code Completion in general) is a great feature to speed-up your workflow in programming. Not only saving you a dozens of keystrokes per day by complete your not yet finished lines of code, it can also be used to show you syntactically correct proposals for your method calls or your overrides.
While this is a great way to learn how to use new APIs and frameworks, it can quickly get confusing if you trigger code completion for example on a bloated ui class and get flooded by hundreds of method call proposals:


(all permissions granted by codetrails.com)



But actually, only a few methods are interesting, depending on your current code context:



(all permissions granted by codetrails.com)

This is exactly what Code Recommenders does. The knowledge, which methods are relevant for a programmer at a certain point in his code context are gained from machine learning and analyzing of source code from open source projects at the Eclipse Foundation.


Unfortunately, if you’re completely new to an API or a framework, your first step wouldn’t be to trigger code completion on some unfamiliar classes. Usually, you start paying attention on some documentation, whether it be a book, an online web page or a demo talk. For a specific problem, you’d lookup a forum or something like stackoverflow.com
But unfortunately, the machine learned source code statstics of Code Recommenders are only available in your Eclipse IDE. At least until now!

Javadoc & Code Recommenders - What will it look like?



There are several questions and statistics from Code Recommenders which you could insert in a Javadoc HTML generated website, e.g.


  • When selecting a package:
    'What are the most frequently overwritten (used) classes in package org.eclipse.xyz*?'
  • When selecting a method:
    'Which methods are frequently called together with m()?'
  • When selecting a type:
    'Which methods of type X frequently used by clients?'
  • and many more...


While these are just some basic ideas, I’d like to prioritize them (and others) for the implementation with a Google Docs poll for the Eclipse GSoC mentors.

An integration of the Code Recommenders statistic to the generated HTML output of Javadoc could look like this:



I’ve created some more Javadoc examples in HTML with the Javadoc framework, check them out at
The following elements are annotated with some statistics from Code Recommenders:
  • The package org.eclipse.jface.viewers (at the bottom)
  • The class org.eclipse.jface.viewers.TableViewer(right beneath the class description)
  • The method getTable() in class TableViewer


Javadoc & Code Recommenders - How are they working together?



For this generation part, I plan to extend the Javadoc Tool framework by one or more 'Doclets' (see [2]) that allow different extdoc providers (based on the extdoc providers of Code Recommenders) to contribute directly to the generated HTML output via the Javadoc api.
In contrast to the normal way how the javadoc command is used, I'd like to develop a little helper utility that fetches and unzips the sources directly from a maven repository, runs the normal javadoc tool and adds the recommenders documentation to the standard javadoc output, and finally uploads the updated javadocs to the repository or deploys them on a website.


Going further - Let’s get social!



In addition (when there is still some timet), I'd like to enable users to directly edit, comment or rate on API documentation and new code snippets, which are provided by the Snipmatch subproject of Code Recommenders. Alternatively, these code snippets could also be posted directly by developers through the webinterface or even via a REST API (so you could implement an additional IDE plugin for this afterwards). The code snippets shall be grouped by java classes and/or packages, which then can be tagged, commented and rated by other developers. Top rated code snippets ('favorites') shall appear at the top of the list, indeed similar to stackoverflow.com.


Here are two mock-ups for a first impression:


API Commentation feature for developers



Social Code Snippets Integation


Server Backend



For these social features, an additional backend (a RESTful api) will be needed, which stores and organizes the code snippets, including tags and comments by users.
The snippets shall be inserted dynamically in the (so far static generated) Javadoc website. For the server-side part I'm thinking of using the OData [3] protocol backed up by a SQL database like MySQL.


Milestones and Timeline



Milestone #
Description
Duration
1 -
Javadoc Generation Utility
Provide a little helper utility which
  • fetches and unzips sources directly from a maven repository
  • runs the javadoc tool (with recommenders documenation)
  • upload updated javadoc
~3 Weeks
2 -
Extend Javadoc with CoRe statistics
Extend the Javadoc Tool framework by one or more 'Doclets' which allow different extdoc providers (based on the extdoc providers of Code Recommenders) to contribute directly to the generated HTML output via the Javadoc API
~7 Weeks
3 -
Server Backend für Social Javadoc
RESTful API Server backend, which stores and organizes the code snippets, tags and documentation comments by users.
Implementation with the OData protocol backed up by a SQL database like MySQL.
~4-5 Weeks
4 -
Webpage Frontend for API Documenation
Enable users to directly edit, comment or rate on API documentation through a webpage or the REST API (maybe an Eclipse IDE plugin afterwards)
Remaining Time
5 -
Webpage Frontend for Code Snippets
Integration of code snippets, which are provided by the Snipmatch subproject of Code Recommenders. Alternatively, the code snippets could also be posted directly by developers through the webinterface or via the REST API (maybe an Eclipse IDE plugin afterwards)
Remaining Time


18 Weeks GSoC


About my work and contact information



Until now, I’ve done some various bugfixes and corrections in the frontend of the Code Recommenders Eclipse Plug-in, improved the drag’n’drop provider ranking in the Extdoc View and worked out a documentation for the different code completion scenarios (method calls, overrides, annotations etc.) in Java and how we want to treat them in the future at Code Recommenders.


Please feel free to contact me for further informations on my work or my GSoC proposal:


email
mail@gottschaemmer.net
mailing lists
recommenders-dev@eclipse.org,
soc-dev@eclipse.org
gtalk/jabber
patrick@gottschaemmer.net
icq
216653447
skype
paddyg89
astra
paddyg89


Links
[3] http://www.odata.org/