Aug 26, 2011

Eclipse Code Recommenders 0.3!

We've been a bit quiet for a while, right? Yes, that's true, but today we announced quite a lot new stuff. At least too much stuff for one blog post, so we split things up ;-)

So what has changed?

Model store for recommendation models:
For 0.3 we completely rewrote the way how we manage and deliver our models for code completion. In a nutshell, the models are no longer shipped on installation but are downloaded automatically from the web as needed.  This reduces download size to a minimum and enables services such as automated model updates, manually generated models and many things more as you shall see below.

Support for version ranges:
Assuming that the way we have to use the same API will not change on every build but only on major version changes (3.6-->3.7), we now generate models that match multiple versions of a framework. For example, we generate a single model for org.eclipse.jface matching versions for 3.6.x and another one for all 3.7.x releases.

Automated model updates:
The current drop will contain all you need to automatically download models matching your used framework libraries. Code Recommenders just computes the fingerprint of your library, and extracts the library-name and -version for OSGI, Android, and Maven-build files and sends this to the server to find the appropriate model. 

Community-sourcing and custom model support:
One of the long-term goals of Code Recommenders is to enable every developer to build his own models for his libraries. Unfortunately, this feature is not ready for 0.3 yet. However, there is a student project that aims to add this feature to Code Recommenders and it may be ready for 0.4. If you are keen to build your own models from your own Eclipse Workspace, please check out this page for a (pretty rough) description how to get started. There is also a link to 90 minutes video (in German) available that shows how we added support for Android live to a local Code Recommenders installation. If you'd like to build your own models, follow the instructions there. If you got stuck, just send a comment to the forum. We'll improve on the scrips and documentation as soon as you request it. Of course: Your assistance in generating models and improving Code Recommenders community-sourcing approach is greatly appreciated!

What else has changed?

Let's step over to the other things that changed:
Codesearch gets two new friends and is not alone in the sandbox category anymore. One of the two new experimental projects is code completion based on subwords matching and the other is ExtDoc, an integrated documentation platform implemented by a student as google summer of code project. Both are described in a separate blog post: Enjoy reading!

Get the new stuff :
All plug-ins are available form the Eclipse Marketplace and our DEV update-site  - and don't hesitate to tell us what you think about it. We'd like to get your feedback!

Extdocs: Extended Documentation Platform for crowd-sourced, mined, and other documentation!

Writing documentation is one of those tasks a programmer is typically not very keen about. Good and comprehensive documentation,  however, is crucial for the success of open source software. But creating such documentation takes time and energy, is boring and has almost no immediate rewards. Consequently, documentation is (too) often incomplete - or even worse: outdated (meaning plain wrong) as this example shows:

I can't say what's more annoying: Having outdated and wrong documentation or having no documentation at all... 

Some guys might postulate that well written (aka clean) code doesn't need any documentation - and they may be right. Others may argue that before documentation gets outdated it should be generated from code - since "the code is the design" this sounds reasonable too. 

Based on this observation (and maybe driven by some project manager) a resourceful programmer came up with a tool that generates high-quality API documentation directly from code:

Impressive, right? I especially like its ability to take a statement  such as "throws IllegalArgumentException" and create the JavaDoc comment "@throws the illegal argument exception"!  Awesome. (OMG...)

No, seriously. Generating documentation using tools like JAutodoc at most pleases our team lead since they improve the documentation ratio - from a programmer point of view this documentation is pretty useless.

OK, but what can Code Recommenders do about it? Can we improve it?

Yes. To some extent. In an earlier post we described the idea of how example code that uses a framework's API could be leveraged to create valuable usage-driven documentation from, such as subclassing directives, which self-calls to execute, displaying code snippets showing common usages etc. If you haven't read this post before, you should reserve a few extra minutes to get the idea of mining documentation from example code.

However, mined documentation extracted from example applications is not the only information you want to leverage when using unfamiliar APIs. Often you refer to other resources on the web such as tutorial sites, codesearch engines, or code snippet repositories. So what you rather would like to have is this: A documentation platform, that tightly integrates various information into your IDE:

Something like this:

I'm extremely glad to announce that a first draft of this extended documentation platform is ready for download from the Eclipse Markeplace. This platform has been developed by Stefan Henss during this year's Google Summer of Code. An outstanding work which I would like to present with this post.

The text below is copy of Stefan's final Project summary which I cross-post here:

Stefan Henss: Summing Up Eclipse ExtDoc (GSOC 2011)

This year I’ve participated in Google Summer of Code by contributing to Eclipse Code Recommenders. The goal of Code Recommenders is to develop various tools which analyze code frameworks and their usage to mine information useful for code documentation and to increase productivity. For example, it compares the user’s code with code from various sources and makes statistical assumptions on which methods should be called next. ExtDoc was created to compile all available information into a handy documentation format integrated in Eclipse.

The Idea
ExtDoc’s approach is to adopt the way Eclipse displays Javadoc, i.e. in a view or when hovering code elements, and to create a framework which allows to contribute any information to similar displays. For instance, Code Recommenders holds statistical information of many kinds for many popular frameworks, but currently lacks a way of displaying all in one place. In order to have information displayed with ExtDoc, each information contributor simply has to implement a “provider” through which he is asked to submit his data for the current user selection. ExtDoc then puts together all providers and allows the user to navigate through the information in multiple spots.

The ExtDoc View
Just like the console or the Javadoc view, the ExtDoc view can be located somewhere around the main editor and reacts to any selection in the editor. In the following, all available providers are depicted as they appear in the ExtDoc view. Additional to the providers’ contents, which are displayed as one scrollable “document” on the right, there is a table on the left indicating all available providers and their status. For example, providers are grayed out if they have nothing available for the current selection or are disabled by the user.

The table also offers the possibility to change the provider order by drag and drop and to disable/re-enable providers perminately. Finally, on top of the table the selection’s location type is displayed (e.g. “field declaration”), since some providers distinct between up to 9 location types. For example, a user might find different information useful when selecting a type in an “extends” statement than in a method body. Therefore the user’s selection of providers is also location-sensitive.

Currently available Providers
The method calls provider offers information on which methods should typically be called for the selected Java element. For example, when selecting SWT’s Text type, it displays the most frequently called methods. When selecting a local variable of type Text, it even takes into consideration which methods have already been called on the variable, which usually influences the probabilities of further method calls.

Sometimes even whole blocks frequently occur in different source codes, e.g. the configuration of reoccurring elements. As a result, not only single method calls but also whole code examples are proposed by ExtDoc. Note: this provider is currently in development and only displays information for a few selected elements.

The usage of a framework typically involves extending provided interfaces, abstract classes or even concrete classes. Not seldom, one is expected to override already implemented methods in order to achieve certain tasks and therefore has to understand the superclass(es) code or rely on a comprehensive documentation. Code Recommenders computes which methods of a class typically are overriden, so-called subclassing directives, which are also displayed in ExtDoc. When selecting an overriden method, ExtDoc even displays which of the superclass’ methods are typically called from the method implementation, e.g. for configuration purposes.

Often framework objects are such comprehensive that the set of overriden methods differs between tasks, i.e. one method might be required to be specified in one scenario, but of no interest at all in the other. Therefore also subclassing patterns are mined, as depicted in the following screenshot.

Finally, the social bookmarks provider allows the users to share web resources. For example, when one experiences difficulties with an API element, he might submit a solution he found on the web as a reference for all other users that might come along this element and have similar issues. In order to filter the most relevant resources, the community is able to rate each link.

Community Feedback

As depicted in all screenshots, all current providers allow users to rate their content through a 5-star system, either the displayed element information as a whole, or single items such as a specific patterns. Furthermore, there is a standard ExtDoc widget which allows commenting the provider content. The following screenshot depicts the expanded comments section underneath the subclassing patterns provider.

The ExtDoc community widgets allow both, communication between the users, e.g. additional information to the displayed data, and feedback to the provider author.
Further ExtDoc Locations

All above screenshots displayed the providers as they appear in the ExtDoc view. However, similar display is also available in two additional locations in which users typically look for documentation. First, the Javadoc pop-up, which is displayed when hovering an element in the editor, is replaced with an ExtDoc pop-up. It contains the same providers “document” as the view does and an additional toolbar underneath, which allows jumping/auto-scrolling to the specific provider. Now, there is no more Javadoc available!? Don’t worry, ExtDoc also contains a provider displaying the traditional Javadoc, as depicted in the very first screenshot :-)

The third location for ExtDoc is the documentation of elements selected in the code assistant pop-up. It offers the same features as the hover pop-up does, as shown below:

Obtaining ExtDoc

Since ExtDoc, among some other Code Recommenders features, is in the testing phase for an upcoming drop, you have to access it either through the head update site or the ExtDoc update site. Both links are given below:

[It's here: Install it directly from the Eclipse Marketplace!]

Feedback on the current state as well as suggestions for further development are much welcome :-)

Disclaimer ;-)
Code Recommenders' Extended Documentation Platform is experimental - and so is the mined documentation shown for various Eclipse APIs. For 0.4 we are planning to add a community sourcing approach that enables us to support arbitrary frameworks.

Whatever comments you have (ranging from like to dislike), let us know about it. Here, in the forum or via twitter.

Marcel, Stefan and Johannes.

Eclipse Clean Code Method Sorter: Try #1

A few months ago, a colleague and I had a hot discussion on good styles of sorting methods in source code, and (of course) we also discussed the advantages of the newspaper-style as proposed by Robert C. Martin:
Have you ever chased your tail through a class, hopping from one function to the next, scrolling up and down the source file, trying to divine how the functions relate and operate, only to get lost in a rat’s nest of confusion? Have you ever hunted up the chain of inheritance for the definition of a variable or function? This is frustrating because you are trying to understand what the system does, but you are spending your time and mental energy on trying to locate and remember where the pieces are.
Concepts that are closely related should be kept vertically close to each other. We want to avoid forcing our readers to hop around through our source files and classes.

Robert C. Martin "Clean Code" pp. 80

Unfortunately, Eclipse doesn't support sorting members in "clean-code style". Thus, we decided to give away this idea to a pair of students as an Eclipse hand's-on training. This post is a follow-up of my post in May.

To get you started, here is a quick refresh of the problem:

Typically, there are two types of code ordering. Alphabetically (+ modifiers) or completely hand-crafted (which in many cases means "without any particular order"). But is it that complicated to let the IDE sort a class' members for you? Reducing the need to hop from top to the bottom of the file and thus minimizing scrolling and confusion?

The slide below gives you an example of how a well-sorted piece of code might look like. As you see the number of hops required to read the whole code is minimal since there is a clear ordering between the code:

How complicated can it be?
It's amazingly complex :) We had to learn that an optimal solution would be APX-hard to find (did you no that complexity class before? I didn't.) Anyways, we had quite tough discussions with our students and sometimes even bar room fights what might be the best solution. To make that part of the story short,  Mateusz and Fabian came up with two solutions. One solution is already available for testing.

If you like the newspaper-style as described by Robert C. Martin, you should give it a try! Please check this wiki page for further information about the plug-in and installation instructions (it's not an official part of Code Recommenders yet).

To track the progress of this feature, add yourself to this bugzilla report.


Stay in touch with Code Recommenders via Twitter.

Eclipse Subwords Completion #1 - Here You Are:

Did you ever type something like text.lay<^Space> and wished Eclipse would complete it to text.setLayoutData()? If you do that with Eclipse as it is today you will get all proposals that start with lay* as shown below:

No surprise yet. Eclipse has a strict prefix matching that ensures that every completion proposal matches the prefix entered by the user. This limitation is somewhat relaxed by allowing CamelCase matches. For instance if you trigger code completion on container.sLD<^Space> you get all completions that match the upper-case letter as shown here:

CamelCase is excellent. However, most of the time when using CamelCase matchers we have to carefully think about what CamelCase shortcut would match the proposal we want to see. But given the benefit that I only have to type 3 letters instead of 11 to make the match unique (or by typing setl+↓CamelCase is a real relief.

Yet, quite some people aren't happy enough with the way CamelCase abbreviations work. They would like to have a subword matching rather than a strict prefix or CamelCase matching (see [1], [2], [3] for  recent discussions on this).

Subwords is coming to rescue those. A few weeks ago we had been asked to provide such a completion engine. It didn't seem very complex (at first). Thus I spent a few hours on implementing a first prototype and announced it via twitter and Code Recommenders' newsgroup.  I got a lot of feedback and helpful discussions on what expectations users have how such a completion engine should work. Thanks to Maxime Jeanmart, Lars Vogel, Deepak Azad, Sascha Vogt, Sebastian Proksch, Gary Fritz, Michael Kutschke and quite a few others for their comments on this prototype.

Today, we are happy to announce the first "official" "drop" of this subwords completion engine!

Below you find a few screenshots showing the subwords completion engine in action.

Completion proposals if triggered at container.lay<^Space>
Proposal when triggering completion at text.<^Space> and then typing lay with open completion window
Subwords completion matching on variable names

You will experience a few more features as you go.

I would like to thank Paul-Emmanuel Faidherbe who implemented large parts of this system and contributed it to Code Recommenders!  

What's coming next?
Subwords is experimental and needs your feedback. If you like the completion let us (Code Recommenders and the JDT team) know by adding yourself to CC list at and by adding comments and new feature requests to this bug report.

And of course: If you experience strange behavior, let us know by sending a post to the code recommenders forum or by creating a bug report at Eclipse Bugzilla.

If you'd like to support subwords: all ideas and issues are tracked by bug 350000 and wait for your help.  If you would like to add a feature (such as completion on type names for instance), just let us know!