How much can the IDE predict what you will write in the next seconds?

How should code search work?

2011-12-02T17:16:00.001+01:00

Hey all. As you probably all know, this is the Code Recommenders project blog we typically use to announce new features, releases or other noteworthy topics related to Code Recommenders. This blog post is a bit different than the others: It's our first "guest blog post" written by Tobias Boehm, a master student doing his master thesis in the scope of the Code Recommenders project.
This post is basically a brain dump how he thinks code search engines like Google Code Search, Krugle, or Koders *should* work and how we should be able to use them. He introduces an early prototype of a code search query language (which he will implement using Xtext) and a client tightly integrate into the Eclipse IDE. His work will be based on the previous code search engine and Eclipse client we already blogged about a few months ago here: "Why is Google Codesearch not google for code search?"

This blog post is a "heads-up! Your feedback is wanted!" post. So, please do not hesitate to ask tough questions or provide any other kind of feedback. All kind of feedback is appreciated and will help Tobias to catch idea bugs early! If you are interested in joining the work on code search engines, get in contact via the Code Recommenders forum. We are looking forward to your feedback.

Thanks,
Marcel

How should code search work?
We all know that learning an API is hard. But we do it day by day by day... When learning a new framework we often ask things like "Which classes should I extend?", "Which methods should I invoke on this object?", "How can I create an instance of this particular type?", "How does the code of others look like that is similar to mine?". Hopefully, some documentation is available that answers these kinds of questions. But how many frameworks do you know that have such excellent documentation? If you know a few: How many do you know which have not?

Let's assume that there is some documentation somewhere. Who wants to work through heaps of documents just to know how obtain an instance of the famous IStatusLineManager? In my opinion, the best possible documentation already exists. It's existing and tested code which is available in masses in code repositories where these API are actually used, the classes are extended, the methods called and the objects actually instantiated. The code is there, it just has to be found!

But searching for source code is still a tedious task. Although many search engines exists and even Google has a product targeted at code search - they are not too useful in certain situations. First it seems most of them treat source code the way web search engines treat websites they index - as plain text. While that might make sense for some code search use cases - it is not enough for most others. Moreover they are too generic to be useful. Most code search engines seem to identify the programming language the code is written in, yet they are not using the language-specific semantic that lies underneath.

And then there is "availability". For a developer to be able to search for source code this source code has to be indexed by the search engine. So the code available is always restricted to open source code publicly available in the web with all personal and company repositories being unused.

Lastly, there is no IDE integration. What that means is that every time we want to issue a query we are shifting focus away from our IDE to a website.

In this blog post, I describe may plans on how to implement a Code Search engine with Apache Lucene. I'll go through a set of sample queries and explain what get's indexed and how developers can query the index to solve common day-to-day tasks.

If you are interested in code search and are maybe seeking for good alternative to (almost closed) Google Code Search or want to build your own code search engine for your own company - please continue reading and don't hesitate to ask questions about it here or in the Code Recommenders forum.

Query
As said above, search query capabilities of todays code search engines are somewhat limited. Code Recommenders might come to rescue here. The heart of this prototype currently in development is a novel query language. This query language must be very simple to use. We want to create queries very easily. Yet it must be so powerful that we can express all the requests we might have to a code base. What might these requests be? Before we dig into the search criteria let's take a step back and think about what it is we would like to find. Are we interested in source files? Probably not. Java developers don't think in source files. At the bottom level we think in classes, methods and maybe even smaller blocks of code. Then these are probably the units of code we want to find. Now what questions might a developer have? She might for example want to find methods that have a certain name. The query will look like the following.

METHODS WHERE Name IS "set.*"

This query will return those methods with a name starting with "set". Let's say we are interested in methods that add something to a java.util.Set.

METHODS WHERE CalledMethods CONTAINS {java.util.Set.add}

By combining multiple search criteria and using negation the developer is able to refine the query to get exactly what she needs.

METHODS WHERE ReturnType IS {org.eclipse.jface.action.IStatusLineManager}
AND +IsPublic AND !IsStatic

A Query that will search for public, non-static methods that should return an instance of IStatusLineManager. When using the prefix "+" and "!" the developer can explicitly mark criteria as mandatory or non-occurring. If we omit the prefix the condition is optional and the results we get might not meet the criteria. In this particular example we would want the query to look like this.

METHODS WHERE ReturnType IS {+org.eclipse.jface.action.IStatusLineManager}
AND +IsPublic AND !IsStatic

This way we can be sure that the methods we find will return IStatusLineManager. What if we - for whatever reason - are interested in public methods that use an IStatusLineManager, are annotated with SuppressWarnings and that should be constructors? Here's the query for that.

METHODS WHERE IsConstructor
AND UsedTypes CONTAINS {+org.eclipse.jface.action.IStatusLineManager}
AND +IsPublic
AND ANNOTATED WITH {+java.lang.SuppressWarnings}

This is really just supposed to be an example of how detailed we can get. There are many more criteria available and they are not bound to just methods. Many of the criteria applicable to methods can be applied to classes too. We might as well search for classes with a certain name.

CLASSES WHERE Name IS "set.*"

With a more complex query for example we search for abstract classes that implement the interface ASTVisitor preferably using the type java.util.Set.

CLASSES WHERE +IsAbstract
AND ImplementedInterfaces CONTAINS {+org.eclipse.jdt.core.dom.ASTVisitor}
AND UsedTypes CONTAINS {java.util.Set}

Or how about classes that contain deprecated methods?

CLASSES WHERE
CONTAINS METHOD WERE ANNOTATED WITH {java.lang.Deprecated}

The list goes on. And we don't stop at classes and methods. Sometimes we would like to bring in more context information. For instance a typical question a developer sometimes asks is how other developers handled a certain exception. Did they close the stream afterwards or did others just log the exception?

CATCHBLOCKS WHERE CaughtType IS {+java.io.IOException}

What we can express here is a question many of us ask themselves over and over again. What have other developers done in my situation?

Repositories
The quality of the results is dependent mainly on the quality and the volume of the search index. The public index will consist of many open source types (http://eclipse.org/recommenders/documentation/completion/) that we can offer code examples from. A dilemma arises when we think about precious code that lies dormant in hundreds of types and thousands of methods in the developer's company repository. These sources are most likely not open source and hence can't be put into a public index. The solution is a private search index that is build and stored inside the company infrastructure. The query can then be performed on the public as well as the local search index.

IDE integration
Queries of this kind can then already be used easily from inside the Eclipse IDE. The editor assists with things such as the query grammar and resolving of types. The goal is to make it as efficient as possible to express a query that reflects what the developer searches for. While a query of this kind can easily be created by hand its full potential is still not exploited. In many use cases where the developer would like to find code examples, the query will consist of information that reflect the user's current code context. That might be the interfaces the current class implements, the overridden method we are in and the uninitialized type that we desperately need an instance for. Most of this information the query consists of are usually ones that the IDE could provide. So why shouldn't it? By abstracting further and bundling the search into a few easy to understand search types the query complexity is now hidden underneath plain proposals in the Eclipse content assist popup window. Let's consider the following situation.

IStatusLineManager slm = null;
|<^Space>

We just declared an IStatusLineManager and set it to null. One of the proposal would probably be a search for code that initializes this type in some way and might find code that for example creates an actual instance of IStatusLineManager or just a method that returns it. Do we now copy the whole method or even the whole class? Or do we just reuse an already existing method that we now knows exists? It's up to us the developer how to use the code examples that are found.

How does that sound. Like something that could make your life easier? Let us know.

Do you still read source code to learn how to use an API?

2011-11-17T14:06:00.001+01:00

We all agree that learning new APIs is a challenging and time consuming task. However, we can't get around this, right? It is the way how software development works nowadays. But it's also part of the fun we have when creating new software and make it work. However, sometimes we wish we would be little more efficient - especially when digging into the source code written by others to learn how to use an API, or when using code search to find code examples.

Apparently, reading source code is valuable. But have you ever though about how the knowledge how others used an API could be leveraged to improve IDE services such as Code Completion, Code Search, or even API documentation?

In early 2011, the Eclipse Code Recommenders project was created to tread new paths on how the next generation of IDEs could enable developers to share knowledge with each other over their IDEs and to improve tools like code completion, code-search, and even to enrich existing documentation by leveraging the knowledge of the masses.

Since it's inception, the project made an amazing progress. Many tools have been developed since such as:

Intelligent (context-aware) Code Completion,
Dynamic (mined) Code Templates,
Code snippet search engines,
Extended (mined) Documentation platform,

and a lot more. 9 Months later, we published v0.4 of Code Recommenders and presented it at EclipseCon Europe. If you haven't heard about Code Recommenders yet, it's time to spent a few minutes now to learn how IDEs will change your way of developing software in the next year:

The video is made available by FOSSLC. Thanks to FOSSLC and FOSSLCde to make this public available.

Talk Abstract:

This talk introduces Code Recommenders’ tool suite, demonstrates the potential to support you and your team on your daily work, and shows in a live session how to setup your own knowledge base for your own products by adding support for Google’s Android to Code Recommenders. The talk provides you a substantial understanding of how Code Recommenders tools work, which knowledge gets extracted from code and how recommendations for code search, code completion and other tools are made.

This talk targets on developers, team leads, and CTOs. It aims to familiarize you with the ideas of Code Recommenders, to interactively demonstrate the current state-of-the-art, and provides perspectives how to leverage the knowledge of the masses for your products in your own company.

If you want to learn more about Eclipse Code Recommenders, visit http://eclipse.org/recommenders.
If you have any comments about Code Recommenders and its vision of IDE 2.0, let me know via twitter, google plus, or the project's forum.

Best,
Marcel

It's even not your father's IDE...

2011-10-31T09:02:00.003+01:00

As Sven outlined in his last post, much has changed since Java's inception somewhere in the mid 90s and it's worth looking on how things are going to change in the future. I'm looking forward to this session - even more after Sven's announcement (Java Puzzlers). I'm now pretty curios how this session will look like. I guess, it will be fun!

Regarding the general topic: programming languages have been subject to change in the past years, each claiming to improve developer productivity. But as advances in programming language improve developer productivity, so do advances in todays IDEs. And there are massive changes coming to your favorite IDE!

In my session at EclipseCon on Friday morning, I'll demonstrate how leveraging your knowledge about how to use an API helps others to quickly learn how to use these APIs correctly - or the other way round: how other's knowledge help you to get productive in minutes.

You should attend this session if:

you have to learn about new/unfamiliar APIs often.
your team frequently gets new team members that have to learn how to use your (internal) APIs.
you have customers that have to use your APIs and you wanna support them in a novel, pioneering way.
you wanna know how leveraging source code works to improve code completion, API documentation, code search, and many more tools.
you wanna see Code Recommenders in Action!

In 50 minutes, I'll take you through Code Recommenders tool suite by demoing its intelligent code completion, chain completion, subwords completion, crowd-sourced template completion, extended javadoc platform, and code search, and explain which information is leveraged to make these tools work. At the end of the session, I'll demonstrate how you can use all these tools for your own development teams or customers to improve their productivity by adding support for Google Android to Code Recommenders - live.

This session will be packed with latest features and tools which you haven't seen in any other IDE before - nor your father :) Looking forward to see you on Friday, 10:30 in the Theater: "Code Recommenders: Code Completion on Steroids!"

Eclipse Code Recommenders 0.3!

2011-08-26T16:20:00.001+02:00

We've been a bit quiet for a while, right? Yes, that's true, but today we announced quite a lot new stuff. At least too much stuff for one blog post, so we split things up ;-)

So what has changed?

Model store for recommendation models:

For 0.3 we completely rewrote the way how we manage and deliver our models for code completion. In a nutshell, the models are no longer shipped on installation but are downloaded automatically from the web as needed. This reduces download size to a minimum and enables services such as automated model updates, manually generated models and many things more as you shall see below.

Support for version ranges:

Assuming that the way we have to use the same API will not change on every build but only on major version changes (3.6-->3.7), we now generate models that match multiple versions of a framework. For example, we generate a single model for org.eclipse.jface matching versions for 3.6.x and another one for all 3.7.x releases.

Automated model updates:

The current drop will contain all you need to automatically download models matching your used framework libraries. Code Recommenders just computes the fingerprint of your library, and extracts the library-name and -version for OSGI, Android, and Maven-build files and sends this to the server to find the appropriate model.

Community-sourcing and custom model support:

One of the long-term goals of Code Recommenders is to enable every developer to build his own models for his libraries. Unfortunately, this feature is not ready for 0.3 yet. However, there is a student project that aims to add this feature to Code Recommenders and it may be ready for 0.4. If you are keen to build your own models from your own Eclipse Workspace, please check out this page for a (pretty rough) description how to get started. There is also a link to 90 minutes video (in German) available that shows how we added support for Android live to a local Code Recommenders installation. If you'd like to build your own models, follow the instructions there. If you got stuck, just send a comment to the forum. We'll improve on the scrips and documentation as soon as you request it. Of course: Your assistance in generating models and improving Code Recommenders community-sourcing approach is greatly appreciated!

What else has changed?

Let's step over to the other things that changed:

Codesearch gets two new friends and is not alone in the sandbox category anymore. One of the two new experimental projects is code completion based on subwords matching and the other is ExtDoc, an integrated documentation platform implemented by a student as google summer of code project. Both are described in a separate blog post: Enjoy reading!

Get the new stuff :

All plug-ins are available form the Eclipse Marketplace and our DEV update-site - and don't hesitate to tell us what you think about it. We'd like to get your feedback!

Extdocs: Extended Documentation Platform for crowd-sourced, mined, and other documentation!

2011-08-26T15:38:00.000+02:00

Writing documentation is one of those tasks a programmer is typically not very keen about. Good and comprehensive documentation, however, is crucial for the success of open source software. But creating such documentation takes time and energy, is boring and has almost no immediate rewards. Consequently, documentation is (too) often incomplete - or even worse: outdated (meaning plain wrong) as this example shows:

I can't say what's more annoying: Having outdated and wrong documentation or having no documentation at all...

Some guys might postulate that well written (aka clean) code doesn't need any documentation - and they may be right. Others may argue that before documentation gets outdated it should be generated from code - since "the code is the design" this sounds reasonable too.

Based on this observation (and maybe driven by some project manager) a resourceful programmer came up with a tool that generates ~~high-quality~~ API documentation directly from code:

Impressive, right? I especially like its ability to take a statement such as "throws IllegalArgumentException" and create the JavaDoc comment "@throws the illegal argument exception"! Awesome. (OMG...)

No, seriously. Generating documentation using tools like JAutodoc at most pleases our team lead since they improve the documentation ratio - from a programmer point of view this documentation is pretty useless.

OK, but what can Code Recommenders do about it? Can we improve it?

Yes. To some extent. In an earlier post we described the idea of how example code that uses a framework's API could be leveraged to create valuable usage-driven documentation from, such as subclassing directives, which self-calls to execute, displaying code snippets showing common usages etc. If you haven't read this post before, you should reserve a few extra minutes to get the idea of mining documentation from example code.

However, mined documentation extracted from example applications is not the only information you want to leverage when using unfamiliar APIs. Often you refer to other resources on the web such as tutorial sites, codesearch engines, or code snippet repositories. So what you rather would like to have is this: A documentation platform, that tightly integrates various information into your IDE:

Something like this:

I'm extremely glad to announce that a first draft of this extended documentation platform is ready for download from the Eclipse Markeplace. This platform has been developed by Stefan Henss during this year's Google Summer of Code. An outstanding work which I would like to present with this post.

The text below is copy of Stefan's final Project summary which I cross-post here:

Stefan Henss: Summing Up Eclipse ExtDoc (GSOC 2011)

This year I’ve participated in Google Summer of Code by contributing to Eclipse Code Recommenders. The goal of Code Recommenders is to develop various tools which analyze code frameworks and their usage to mine information useful for code documentation and to increase productivity. For example, it compares the user’s code with code from various sources and makes statistical assumptions on which methods should be called next. ExtDoc was created to compile all available information into a handy documentation format integrated in Eclipse.

The Idea
ExtDoc’s approach is to adopt the way Eclipse displays Javadoc, i.e. in a view or when hovering code elements, and to create a framework which allows to contribute any information to similar displays. For instance, Code Recommenders holds statistical information of many kinds for many popular frameworks, but currently lacks a way of displaying all in one place. In order to have information displayed with ExtDoc, each information contributor simply has to implement a “provider” through which he is asked to submit his data for the current user selection. ExtDoc then puts together all providers and allows the user to navigate through the information in multiple spots.

The ExtDoc View
Just like the console or the Javadoc view, the ExtDoc view can be located somewhere around the main editor and reacts to any selection in the editor. In the following, all available providers are depicted as they appear in the ExtDoc view. Additional to the providers’ contents, which are displayed as one scrollable “document” on the right, there is a table on the left indicating all available providers and their status. For example, providers are grayed out if they have nothing available for the current selection or are disabled by the user.

The table also offers the possibility to change the provider order by drag and drop and to disable/re-enable providers perminately. Finally, on top of the table the selection’s location type is displayed (e.g. “field declaration”), since some providers distinct between up to 9 location types. For example, a user might find different information useful when selecting a type in an “extends” statement than in a method body. Therefore the user’s selection of providers is also location-sensitive.

Currently available Providers
The method calls provider offers information on which methods should typically be called for the selected Java element. For example, when selecting SWT’s Text type, it displays the most frequently called methods. When selecting a local variable of type Text, it even takes into consideration which methods have already been called on the variable, which usually influences the probabilities of further method calls.

Sometimes even whole blocks frequently occur in different source codes, e.g. the configuration of reoccurring elements. As a result, not only single method calls but also whole code examples are proposed by ExtDoc. Note: this provider is currently in development and only displays information for a few selected elements.

The usage of a framework typically involves extending provided interfaces, abstract classes or even concrete classes. Not seldom, one is expected to override already implemented methods in order to achieve certain tasks and therefore has to understand the superclass(es) code or rely on a comprehensive documentation. Code Recommenders computes which methods of a class typically are overriden, so-called subclassing directives, which are also displayed in ExtDoc. When selecting an overriden method, ExtDoc even displays which of the superclass’ methods are typically called from the method implementation, e.g. for configuration purposes.

Often framework objects are such comprehensive that the set of overriden methods differs between tasks, i.e. one method might be required to be specified in one scenario, but of no interest at all in the other. Therefore also subclassing patterns are mined, as depicted in the following screenshot.

Finally, the social bookmarks provider allows the users to share web resources. For example, when one experiences difficulties with an API element, he might submit a solution he found on the web as a reference for all other users that might come along this element and have similar issues. In order to filter the most relevant resources, the community is able to rate each link.

Community Feedback

As depicted in all screenshots, all current providers allow users to rate their content through a 5-star system, either the displayed element information as a whole, or single items such as a specific patterns. Furthermore, there is a standard ExtDoc widget which allows commenting the provider content. The following screenshot depicts the expanded comments section underneath the subclassing patterns provider.

The ExtDoc community widgets allow both, communication between the users, e.g. additional information to the displayed data, and feedback to the provider author.
Further ExtDoc Locations

All above screenshots displayed the providers as they appear in the ExtDoc view. However, similar display is also available in two additional locations in which users typically look for documentation. First, the Javadoc pop-up, which is displayed when hovering an element in the editor, is replaced with an ExtDoc pop-up. It contains the same providers “document” as the view does and an additional toolbar underneath, which allows jumping/auto-scrolling to the specific provider. Now, there is no more Javadoc available!? Don’t worry, ExtDoc also contains a provider displaying the traditional Javadoc, as depicted in the very first screenshot :-)

The third location for ExtDoc is the documentation of elements selected in the code assistant pop-up. It offers the same features as the hover pop-up does, as shown below:

Obtaining ExtDoc

Since ExtDoc, among some other Code Recommenders features, is in the testing phase for an upcoming drop, you have to access it either through the head update site or the ExtDoc update site. Both links are given below:

http://www.eclipse.org/recommenders/download/~~http://vandyk.st.informatik.tu-darmstadt.de/updates/extdoc/~~

[It's here: Install it directly from the Eclipse Marketplace!]

Feedback on the current state as well as suggestions for further development are much welcome :-)

Disclaimer ;-)

Code Recommenders' Extended Documentation Platform is experimental - and so is the mined documentation shown for various Eclipse APIs. For 0.4 we are planning to add a community sourcing approach that enables us to support arbitrary frameworks.

Whatever comments you have (ranging from like to dislike), let us know about it. Here, in the forum or via twitter.

Best,

Marcel, Stefan and Johannes.

Eclipse Clean Code Method Sorter: Try #1

2011-08-26T15:35:00.000+02:00

A few months ago, a colleague and I had a hot discussion on good styles of sorting methods in source code, and (of course) we also discussed the advantages of the newspaper-style as proposed by Robert C. Martin:

Have you ever chased your tail through a class, hopping from one function to the next, scrolling up and down the source file, trying to divine how the functions relate and operate, only to get lost in a rat’s nest of confusion? Have you ever hunted up the chain of inheritance for the definition of a variable or function? This is frustrating because you are trying to understand what the system does, but you are spending your time and mental energy on trying to locate and remember where the pieces are.
Concepts that are closely related should be kept vertically close to each other. We want to avoid forcing our readers to hop around through our source files and classes.

Robert C. Martin "Clean Code" pp. 80

Unfortunately, Eclipse doesn't support sorting members in "clean-code style". Thus, we decided to give away this idea to a pair of students as an Eclipse hand's-on training. This post is a follow-up of my post in May.

To get you started, here is a quick refresh of the problem:

Typically, there are two types of code ordering. Alphabetically (+ modifiers) or completely hand-crafted (which in many cases means "without any particular order"). But is it that complicated to let the IDE sort a class' members for you? Reducing the need to hop from top to the bottom of the file and thus minimizing scrolling and confusion?

The slide below gives you an example of how a well-sorted piece of code might look like. As you see the number of hops required to read the whole code is minimal since there is a clear ordering between the code:

How complicated can it be?
It's amazingly complex :) We had to learn that an optimal solution would be APX-hard to find (did you no that complexity class before? I didn't.) Anyways, we had quite tough discussions with our students and sometimes even bar room fights what might be the best solution. To make that part of the story short, Mateusz and Fabian came up with two solutions. One solution is already available for testing.

If you like the newspaper-style as described by Robert C. Martin, you should give it a try! Please check this wiki page for further information about the plug-in and installation instructions (it's not an official part of Code Recommenders yet).

To track the progress of this feature, add yourself to this bugzilla report.

Best,
Marcel

Stay in touch with Code Recommenders via Twitter.

Eclipse Subwords Completion #1 - Here You Are:

2011-08-26T14:32:00.000+02:00

Did you ever type something like text.lay<^Space> and wished Eclipse would complete it to text.setLayoutData()? If you do that with Eclipse as it is today you will get all proposals that start with lay* as shown below:

No surprise yet. Eclipse has a strict prefix matching that ensures that every completion proposal matches the prefix entered by the user. This limitation is somewhat relaxed by allowing CamelCase matches. For instance if you trigger code completion on container.sLD<^Space> you get all completions that match the upper-case letter as shown here:

CamelCase is excellent. However, most of the time when using CamelCase matchers we have to carefully think about what CamelCase shortcut would match the proposal we want to see. But given the benefit that I only have to type 3 letters instead of 11 to make the match unique (or by typing setl+↓) CamelCase is a real relief.

Yet, quite some people aren't happy enough with the way CamelCase abbreviations work. They would like to have a subword matching rather than a strict prefix or CamelCase matching (see [1], [2], [3] for recent discussions on this).

Subwords is coming to rescue those. A few weeks ago we had been asked to provide such a completion engine. It didn't seem very complex (at first). Thus I spent a few hours on implementing a first prototype and announced it via twitter and Code Recommenders' newsgroup. I got a lot of feedback and helpful discussions on what expectations users have how such a completion engine should work. Thanks to Maxime Jeanmart, Lars Vogel, Deepak Azad, Sascha Vogt, Sebastian Proksch, Gary Fritz, Michael Kutschke and quite a few others for their comments on this prototype.

Today, we are happy to announce the first "official" "drop" of this subwords completion engine!

Below you find a few screenshots showing the subwords completion engine in action.

Completion proposals if triggered at container.lay<^Space>

Proposal when triggering completion at text.<^Space> and then typing lay with open completion window

Subwords completion matching on variable names

You will experience a few more features as you go.

I would like to thank Paul-Emmanuel Faidherbe who implemented large parts of this system and contributed it to Code Recommenders!

What's coming next?
Subwords is experimental and needs your feedback. If you like the completion let us (Code Recommenders and the JDT team) know by adding yourself to CC list at https://bugs.eclipse.org/bugs/show_bug.cgi?id=350000 and by adding comments and new feature requests to this bug report.

And of course: If you experience strange behavior, let us know by sending a post to the code recommenders forum or by creating a bug report at Eclipse Bugzilla.

If you'd like to support subwords: all ideas and issues are tracked by bug 350000 and wait for your help. If you would like to add a feature (such as completion on type names for instance), just let us know!

Best,
Marcel

Eclipse DemoCamp in Darmstadt 21.06. is Approaching!

2011-06-10T11:16:00.002+02:00

I'm happy to announce this year's Eclipse DemoCamp in Darmstadt. The DemoCamp is organized by Deutsche Telekom and Darmstadt University of Technology - and it's going to be great again :)

The last time we had ~120 attendees and quite packed program: Lars Vogel spoke about e4, Benjamin Muskalla and Holger Staudacher talked about the regular day of a committer at Eclipse.org, Manuel Bork presented UMLLab, Karsten Thoms introduced Maven Tycho, Kai Kreuzer demonstrated how to control your home with OSGI, Berthold Daum showed his photo managing software Zora, Stefan Lay gave first insights into EGit, Janos Koppany spoke about DVCS, and Lothar Wieske highlighted Bespin. Who had known at this time that Orion was going to come, Tycho would be that popular, e4 looks that neat, and life as a committer has so much to do with latency :)

This program was hard to beat. But we spent quite some time to get comitters for the latest "en vogue" topics. This time we have 8 talks. Each one will be 15 minutes + one extended talk with 30 minutes. This is the program of this year's Eclipse DemoCamp Darmstadt:

Andreas Hoegger and Matthias Zimmermann (BSI Business Systems Integration AG) introduce Eclipse Scout. Thanks for presenting Scout the first time ever on a DemoCamp!
Christian Schneider (Yatta Solutions) speaks about creating Taks-focused diagrams with Mylyn and UML Lab.
Simon Kaufmann and Jochen Hiller (SAP AG / Deutsche Telekom) speak about Eclipse Skalli - the latest approach to project management portal which brings more transparency to your teams.
Sascha Scholz (SAP AG) talks about integrating Git/Gerrit with Jenkins and Mylyn (AKA Mylyn Builds ++)
Stefan Lay and Matthias Sohn (SAP AG) let us know whether or not "Egit is ready for prime time". I bet it is!
Holger Staudacher (EclipseSource) let's you know what's going on in Eclipse RT Universe.
Falko Menge (Camunda) gives you the spirit of modeling your business with Activiti and BPMN2.
Marcus Ficner (Itemis) gives an extended talk about cross-platform mobile development with APPLause. Friends of Android and iOS keep your mobiles ready for this talk!

And of course, there will be free food and drinks. This event is sponsored by BSI Business Systems Integration AG, msg systems, Deutsche Telekom, and Darmstadt University of Technology. Thanks to our sponsors for their support and helping us to make this event happen again the third time now!

The event starts on 21.06., 17:00 at Deutsche Telekom AG, T-Online-Allee 1 in Darmstadt. For location details and latest news please check http://wiki.eclipse.org/Eclipse_DemoCamps_Indigo_2011/Darmstadt.

Oh Stacktrace, My Stacktrace...

2011-05-24T21:51:00.005+02:00

Do you remember the first time you wrote a Wizard for Eclipse? Do you still know what went wrong the first time (and maybe a few times more)? Let's assume you wrote a wizard page like the one below which just contains a simple text widget inside the page's composite:

It's not necessarily a fancy UI nor complete but it doesn't look obviously wrong, right?

However, when you run the wizard it just fails without any error message.

A look into the error log reveals that there is a problem with JFace - but not your code:

Now, you might guess that this exception is caused by you, but frankly: When I wrote my first wizard page I was quite sure I didn't make any mistakes... God, yes. How wrong I've been...

After googleing for an explanation for a while, I found out that I was missing just one simple method call. Namely, calling setControl(..) on my wizard page. I figured this out thirty minutes later - what a great waste of time...

But as I've learned from my talk at Eclipse Summit, I'm actually not the only jerk doing this wrong. I asked the audience to raise their hands if they know that error - and after a while they started grinning and one-by-one raised the hand to show that they experienced the same problem just some time ago. How about you? Do you know it too?

Let's see what Google returned me when I pasted the first line of my exception into its search box: ("null argument: has no page control" is part of the query but was cut off in the screenshot):

Google obviously does not a good job here. It still doesn't know the answer - even after years this error (better: wrong usage) exists.

So what can we do to get rid of this? Bug trackers like Jira and Bugzilla (with Mylyn extension) have shown the way. These tools have duplicate stacktrace detectors that, whenever they observe a stacktrace in a new report, query the bug database for bugs that contain similar stacktraces.

And guess what: They work - and not too bad! To evaluate that, we replayed the bug reports from Eclipse Bugzilla from 2001 to 2008 one by one and for each new bug report containing a stacktrace we tried to find the corresponding bug id this new report was marked as duplicate of). The amazing result: In ~70% of all cases we found the right bug id under the top 3 positions! Goodbye duplicates - if you submit a stacktrace...

Just another example. Maven had a neat feature that whenever an exception occurs during a build, it printed out a message that directs you to some wiki page which may contain further help about this error. Unfortunately, these pages have been mostly empty (at least for the cases I looked at them). However, the idea is great!

What if there would be a place where you can send your stacktrace to and get a list of resources in the web that discuss exactly your problem? With such a help it should be quite easy to nail down and fix your problem in just a few minutes, right? You may think of it as being similar to stackoverflow.com - but just for stacktraces. Something like stacktraces.org - or stacktraces.eclipse.org - a repository which contains stacktraces of the users, error descriptions and solutions including pointers to resources in the web that deal with these problems.

What we have in mind is something like this: An Eclipse client that listens for exceptions printed on the console or written into the log:

which are parsed automatically by the tool Whenever it observes an exception it may show a pop-up (3) which starts a search dialog that shows your most recent stacktraces:

If you know click on "search" it will search stacktraces.org for similar stacktraces and returns pointers to resources in the web that deal with similar stacktraces - it may even include a discussion on this stacktrace with some explanation of what caused this exception and how to fix it. This summary could be displayed in your browser or directly in Eclipse as shown below:

Additional tabs may provide additional information like similar stacktraces, related web-resources etc.:

That's just a thought about how we could get rid of it. Best of it: in its minimum version there is no additional effort of a user required. We may just give pointers to existing resources or we may integrate the results into a forum etc.

What do you think?

Would you use such a stacktrace search engine that leverages the community knowledge about common pitfalls and usage errors of your favorite open source project? How much time you could have saved if someone had pointed you directly to the forum post telling you that you simply forgot to call setControl.

Just click on the like/dislike button below - and if you have a few minutes, feel invited to drop a comment below.

Thanks,

Marcel

Update #1:

If you like the idea you may also consider commenting on it at/for stackoverflow.com.

Stay in touch: Follow me on Twitter.

A Subword Matching Completion Engine for Eclipse?

2011-05-15T15:59:00.019+02:00

A few hours ago stumbled across a post at stackoverflow.com titled "Eclipse: Fulltext autocompletion for Java?" where a programmer is essentially asking whether Eclipse' code completion supports partial proposal matching.

I copied the question here to make this post a bit more self-contained:

================================================

Hi, can I have fulltext autocompletion for Java @ Eclipse? Let's demonstrate:

Final piece of code:

getVariants().add(new Variant(MediaType.TEXT_XML));

How do I code now:

getv[ctrl+space].a[Enter]new V[ctrl+space, down arrow, Enter]M[Ctrl+Space, Enter].text_x

Basically, Eclipse completes word "TEXT_XML" when I provide letters "TEXT_X".

How would I like to code:

getv[ctrl+space].a[Enter]new V[ctrl+space, down arrow, Enter]M[Ctrl+Space, Enter].xml

and Eclipse should realise I meant "TEXT_XML" (fulltext autocompletion).

================================================

Well, thanks to JDT's quite comprehensive APIs this is not too complex to implement, and I wrote down a simple version of this completion engine (10 locs).

This is how it looks like in action now:

But now I wonder:

How often do you just remember a few letters but don't know the complete name of a method or member?
Would you like to see such a completion engine in Eclipse?
What else would be missing?

I would be glad if you could send your thoughts to the code recommenders forum.

Best,
Marcel

Stay in touch. Follow me on Twitter.

Update #1: Neat research on this topic.

I just found a paper (almost) exactly on this topic. Could you check out these screencasts?

Maybe it's even more than you/we would like to see :-) But I think that's pretty cool stuff and having a similar completion engine for Eclipse could be very, very cool. Internally it's based on Hidden Markov Models which learn how people abbreviated certain identifiers (like stx --> 'setText') in the past. The nice thing about it is that we don't have to come up with our own matching algorithm but just use what the programmers used before... I like the Han's work :)

Update #2: Prototype available.

After a quick and dirty monday afternoon hack a first prototype is ready. In a nutshell, it takes your prefix entered and creates a regular expression from it. For example, when entering button.stx|<^Space> the engine only presents those proposals that match this regular expression .*s.*t.*x.* which finally may match "setText(text)". This is roughly what Maxime proposed in his comment below.

In addition it now respects upper case letters as suggested by Deepak. For example button.sT<^Space> now evaluates to '.*s.*T.*' whereas button.st would evaluate to .*s.*[tT].*.

Currently, it does not reduce the proposal score as recommended by another comment nor does is give precedence to those words that match the prefix - but we may do that if you find this more useful. Just drop a comment in the forum mentioned below.

Please find the installation instructions here:
http://www.eclipse.org/forums/index.php/mv/msg/209269/671088/#msg_671088

Please send your comments to the forum here:
http://www.eclipse.org/forums/index.php/f/211/

Stay in touch. Follow me on Twitter.

Google Summer of Code 2011 - Extended Documentation Platform Get's Going

2011-05-13T21:12:00.000+02:00

IDEs do a great job when it comes to searching and presenting documentation on how to use an API. However, one recurring problem with open source software is that it is frequently outdated, misleading or even missing. In these cases, example code that used an API successfully before become an interesting source of information.

However, extracting patterns from reading source code is a tedious and time-consuming process and the question arises whether we couldn't support developers better by, for example, automatically extracting frequent patterns from example code and provide them as additional documentation. There is quite a large body of research on how useful information can be extracted from code and put into documentation but means to present such mined documentation inside Eclipse are missing, however.

I'm happy to announce that Stefan Henß will fix this during this year's Google Summer of Code. He will implement an "extended documentation platform" for Eclipse - an extensible documentation platform that enriches existing documentation with various information contributed from various documentation providers (not necessarily just mined information).

Stefan already started to outline his thoughts on his blog. If you like the idea just visit his post, drop a comment, and keep in touch to get the latest news about this GSOC project either by following us on Twitter (Stefan, me) or by subscribing to Stefan's news feed (a request to add Stefan's blog to planet-eclipse is currently on the way).

Good Luck Stefan! I'm looking forward to your first prototype screenshots!

"Clean Code" Method Sorter for Eclipse?

2011-04-29T19:12:00.003+02:00

Have you ever chased your tail through a class, hopping from one function to the next, scrolling up and down the source file, trying to divine how the functions relate and operate, only to get lost in a rat’s nest of confusion? Have you ever hunted up the chain of inheritance for the definition of a variable or function? This is frustrating because you are trying to understand what the system does, but you are spending your time and mental energy on trying to locate and remember where the pieces are.

Concepts that are closely related should be kept vertically close to each other. We want to avoid forcing our readers to hop around through our source files and classes.

Robert C. Martin "Clean Code" pp. 80

If you have to read other people's code quite often you probably know this, right?

Typically, there are two types of code ordering. Alphabetically (+ modifiers) or completely hand-crafted (which in many cases means "without any particular order"). But is it that complicated to let the IDE sort a class' members for you? Reducing the need to hop from top to the bottom of the file and thus minimizing scrolling and confusion?

The slide below gives you an example of how a well-sorted piece of code might look like. As you see the number of hops required to read the whole code is minimal since there is a clear ordering between the code:

How complicated can it be?

In the end, there are just a few things to do:

Extracting the caller-callee relationships from code,
Sorting these relations using some topological sorting algorithm,
Dealing with some pat/remi situations where several elements may have the same ordering, and finally
A "physically"reordering these methods in your source file.

That's all. At least in theory. It's well known that "in theory, there is no difference between theory and practice", and thus, it should be straight-forward to implement :-P

I'm glad that we found two students who will implement this nice Clean Code Method Sorter during their Eclipse hands-on this semester. I'm really looking forward to their clean code!

Good Luck, Mateusz and Fabian!

Track progress of this feature via Bugzilla.

Stay in touch with Code Recommenders via Twitter.

Say "Hello Android!"

2011-04-29T17:24:00.004+02:00

Android is all around you, isn't it? It's in your smartphone, in your tablet - and chances are not too bad that you already wrote your own app for your device.

Me myself is still using an old-school Nokia 6230 that is used more often as watch than as phone or even internet device.

This is nagging me. I'm looking for a new phone. A smarter phone. A smartphone.

I'd like to have this one.

A phone which I can use to explore the internet, write emails, and even to write apps for myself (and sometimes to call people).

Unfortunately, I've no knowledge about how to write apps with Android. How about you? Do you know how to write apps in Android? How smooth was your start with Android?

For beginners (and experts alike) Lars' blog vogella.de is an excellent resource to learn about Android and related technologies.

However, not all details are yet covered in tutorials and two weeks ago Lars asked me wether we couldn't run Code Recommenders on Android?

It took me a few hours to figure out but yes - Code Recommenders is capable to learn models for Android too!

The only disadvantage with Android is, that there are not too many open source projects available. That means, that is quite tricky to get a reasonable amount of data to train Code Recommenders. Thus, we have to come up with a purely community driven approach that allows developers to share their knowledge about how to use Android directly from their IDE and - in turn - generating new models from this usage data on a daily basis. With such a system in place we think it should be possible to create a continuously growing knowledge base for Android.

Since my spare time is currently very low, I'm glad that Andreas Frankenberger signed up to implement this upload-train-deliver life-cycle for Android in his bachelor thesis. I'm really looking forward to his results. Until now he has collected a set of 119 open source Android applications (taken from http://en.wikipedia.org/wiki/List_of_open_source_Android_applications) which he will use to train the initial Android recommendation models used as proof of concept before implementing the upload and train cycle.

If you like the idea let us know. We will soon create an early set of recommendation models for Android and we would be glad to get your feedback about it. In addition, if you know any other open source Android projects not listed above, please feel free to give us a pointer to them (preferably as Eclipse ADT projects). Andreas will be glad to add them to the initial training repository.

If you have any other comments or ideas on this idea just discuss them here on the blog or in our forum.

See you!
Marcel

Stay in touch with Code Recommenders via Twitter.

At Lunch with JDT Team Zurich...

2011-04-01T11:30:00.007+02:00

Just a few weeks ago I had the pleasure to present Code Recommenders to Frank Tip from IBM Research - a nice guy who, as I learned during the demo, has implemented some of the refactorings we use day in, day out in Eclipse! Today he's not working on Eclipse anymore but manages the Program Analysis and Transformation Group at IBM Research which is developing (among many other things) the outstanding WALA analysis toolkit we use inside Code Recommenders - definitely a must-have when doing static analysis in Java/JS.

During a 30 minutes demo I presented him the whole collection of tools we worked on in the group - a quite compact session with lots of ideas and concepts in it. After I finished, Frank asked whether we already presented the project to the JDT team and (as I negated) just said "well I will tell Erich about it. I think he should see that." Wow. Is it actually that easy to get in contact?

Just a few days later I was introduced to Daniel Megert and Markus Keller from IBM Research Zurich, two guys who are working on Eclipse JDT from its first day (and even before :)). We arranged a meeting in Zurich last Monday where I got the opportunity to present Code Recommenders during a so called "lunch talk"... I had no idea what a lunch talk actually looks like - but hey! No worries :)

On Monday morning I arrived at IBM Rationale in Zurich downtown. I had a nice chat with Daniel and learned quite a few things what the JDT team keeps busy behind the scenes. Then at 12 o'clock everyone in the office moved out to grab some food from the surrounding restaurants and went back to the group's common room where they unpacked their lunch and started eating.

Except me. I started talking.

Okay, so that's a lunch talk: 10 people having lunch while one guy is sitting in front of the desk entertaining the audience... Well, this looks odd for moment but frankly: it was a great atmosphere! I only wonder how previous speakers managed to eat their food. I guess they didn't... :)

However, after one hour we finished the lunch talk and started a detailed discussion on how Code Recommenders uses JDT and where JDT's defensive APIs affect code recommenders performance. We identified several potential changes / extensions to JDT which I would like to summarize in this post.

More Chatty JavaCompletionContentAssistContext

So far, the java content assist context provides plug-in developers with just a small number of interesting facts:

The position in text where code completion has been triggered on,
Some completion prefix like 'setT' as in 'button.setT<^Space>',
The expected type the proposal should return if any,
The enclosing Java element, i.e., the method or class in which code completion was triggered.

But what to do if you need the information of which kind the receiver of a code completion event is, or which name the receiver has? In a completion event like 'b.setT' you are almost on your own to figure out information like type and name of the receiver, the exact type of completion event (CompletionOnMemberAccess, CompletionOnMessageSend...) by either parsing the source file or rerunning the completion parser as we do in Code Recommenders. Both ways are time-consuming and take between 10 to 50 ms in our cases. Fast in general but unnecessary given that JDT completion already computed (almost) all relevant information before.

During discussion we identified a set of potential changes how content assist context could be improved to help other plug-ins to leverage the analysis results of JDT and how to provide access to more structural information like variable names and types etc.

If you are interested in discussing advances to JDT JavaCompletionContentAssistContext, just add your comments to Bug 340945 - Extension Request of JavaContentAssistInvocationContext.

Code Recommenders' Next Steps: Interviews at jaxenter.com and jaxenter.de

2011-03-29T16:34:00.001+02:00

Two interviews about Code Recommenders' current state and next steps have been published at jaxenter.de (German) and jaxenter.com (English). Comments appreciated - just send them to the forum here.

Interviews are published here:

jaxenter.de: Das Eclipse der Zukunft: Intelligent, sozial, kollaborativ
jaxenter.com: Changing the Way Developers Learn About New APIs

Best,
Marcel

Eclipse Code Recommenders 0.2!

2011-03-01T15:53:00.005+01:00

We are happy to announce the second drop of Eclipse Code Recommenders today! Two weeks after the first drop two very exciting new features are ready to be released - and we are curios to hear what you think about them!

Feature #1: Java chain completion for Eclipse

How many times did you ask yourself "How do I get an instance of, say, IStatusLineManager" in scenarios like the one below?

And how much time did you then spent with shimming from one method call to another to find a call chain returning the type you were looking for - OR- reading Javadoc's Uses documentation to learn how to obtain such an instance?

If you have the feeling that this was quite too long, then this feature is for you:

Code Recommenders chain completion (described in more detail here) supports you in situations like the one above by automatically searching the API jungle for you, i.e., it takes every local variable, every field and every method it can invoke and follows its members until it finds a field or method returning an instance of the requested type.

For example when searching for an IStatusLineManager inside a ViewPart it returns:

Code Recommender chain completion also works on static classes:

And directly on method returns:

And even on method returns with type casts:

Many thanks to Andreas Kaluza and Gary Fritz who implemented this feature for Eclipse Code Recommenders!

Feature #2: Dynamic, example-driven templates engine

The second feature is something I'm getting really excited about.

Have you ever wondered how to deal with a framework's object you are not used to? While working with SWT you may stumbled across SWT Templates. SWT Templates offer common best practices of how to use SWT directly inside your IDE as code completion proposals. For instance, when triggered on the literal "Text" in your current editor, they propose a complete object creation pattern as shown below.

These templates are excellent for occasional users of SWT. However, these templates suffer from one major drawback: They are hand-crafted by experts and consequently only a small number of such templates exist - and only for SWT.

But what if you want to use JFace? Or core parts of Eclipse UI? Or whatever framework you use in your projects?

Let's stick with JFace to illustrate Code Recommenders' template completion engine. Assume that we want to use a JFace TableViewer inside our own view. When triggering SWT Templates on the literal "TableViewer" Eclipse tells you that there are no matching SWT templates (well, not a big surprise, eh?) :

But the question is: Couldn't we generate something similar to SWT Templates for every other framework too? Automatically? Maybe by just looking on how others used the APIs before?

Right, that's Code Recommenders' template completion engine! The idea behind our template engine is to identify common usage patterns from example code and to create something similar to SWT templates for every type we came across when analyzing code examples.

The result of this is shown below: The screenshots show the same situation as above, but just with our example-driven template completion engine and its proposals for JFace TableViewer:

Neat, isn't it? Well, as you might see, these templates are not always as perfect as hand-crafted examples but good enough to be valuable, right? What do you think? Do you like it? Let us know.

And as you may expect from call completion: Of course these templates adopt themselves based on what you have typed before! See this screencast for a detailed example of this:

Many thanks to Stefan Henß who did an amazing job to integrate Code Recommenders intelligent call completion engine into this pretty neat templates completion engine!

Where do I get it?
If you have Code Recommenders 0.1 installed already you can simply choose Help > Check for Updates in Eclipse. Otherwise you can get it from the Eclipse Marketplace or from our update site:
http://download.eclipse.org/recommenders/updates/dev/. You may also check the project homepage for more screencasts one code recommenders.

It's a drop! (not a release)
To avoid any confusions we had with our first drop, we want to explicitly state that we are currently publishing drops only but no releases as defined by the Eclipse Incubator guidelines. Before we can officially release new versions of Code Recommenders we have to pass an official release review by the Eclipse EMO (read more about that here). So far we didn't request a release review because some dependencies are under examination by the IP team - but should do so for later versions to avoid further confusion :-)

What's next?
For 0.3 the goal is to stabilize the current tool suite. Thus do no hesitate to file bugs against the project. The more the better!

Also some exciting news:
We will start collaboration with UC Irvine's Sourcer Research Group - a research group which has an excellent body of knowledge in how to create next generation of code search engines. This will be an exciting project - we will blog about it separately when more details are out. But just to let them know that you like the idea: Click this link

Thanks,
Johannes & Marcel

Eclipse Code Recommenders 0.1!

2011-02-15T18:52:00.006+01:00

We are happy to announce the first ~~release~~ drop of Code Recommenders as Eclipse project! It's just 4 weeks ago that the project was provisioned - but quite a lot has changed since.

During last weeks Code Recommenders stepped out of its fairly ad hoc build and test process and moved towards a somewhat more mature and defined process. We'll see whether the love we put into the build system pays out in future :-)

Apropos love: The project website is up with a new project logo and new screencasts. Please check out eclipse.org/recommenders/ and read eclipse.org/recommenders/about/logo for an explanation of the logo. We are curios to hear how you like website and logo!

But despite all this, let's get to the nitty-gritty stuff. What has changed in the code?

Okay, okay... let's get to the important stuff!

The guiding principle for the first ~~releases~~ drops of Code Recommenders is "It shouldn't hurt!" This means we spent most of the time on improving performance.

For instance, a lot effort has been spent on making Code Recommenders' code completion fast. There is still some extra overhead when a recommendation model has to be loaded the first time, but after that, computing recommendations for these types takes 10-50 milliseconds only! Compared to previous versions this is at least one order of magnitude faster!

Furthermore, given that most of the time is spent (re-) doing work that JDT code completion has done before (computing proposals, figuring out which completion kind was requested etc.), there is quite a lot potential push performance down to just a few milliseconds! The pure recommendation process is that fast!

What else changed? Let's look at the screenshots below:

Code Completion on Variables:
Well, this stayed the same compared to previous versions. Thus, just for completeness:

Of course, this works with prefix tokens as well:

Completion in Anonymous Inner Classes:

This is pretty new - and fast! Whenever you trigger code completion on variables even declared in an enclosing class, Code Recommenders can propose you a set of likely methods for this context too.

Completion on This:
Maybe obvious but somewhat tricky to implement is this:

Completion on Anonymous Method Returns:
Earlier versions of Code Recommenders required you assign anonymous types to local variables before anything was proposed. This is history now:

Completion on Class Bodies:

Want to know which methods you may override in the given class? Here you go:

Cool! What else changed?

The configuration. Code Recommenders is now enabled on a per project basis. Thus to enable Code Recommenders you have to either:

Enable Code Recommenders via code completion shown below:

or by selecting Add Recommenders Nature from the project's context menu:

Can I Use It For My Projects? Which Frameworks Are Supported?

For this ~~release~~ drop we worked on performance and IP cleanliness. Over the next ~~releases~~ drops we will add more and more frameworks. Luckily, the guys around Yoxos / EclipseSource support our mining efforts by providing us with tons of example applications. Many thanks to the team around Jochen Krause, Jordi Böhme Lopez - and Holger Staudacher as the man-in-the-middle supporting us with tips and tricks in many ways :-)

To see which frameworks are currently well supported please checkout this temporary listing here. We will provide more detailed statistics as soon as we have the new data collection framework online. Promised.

Where do I get it?
Pick it from the Eclipse marketplace or direct your Eclipse update client to this update site: http://download.eclipse.org/recommenders/updates/dev

What Comes Next?
Exciting features are almost ready:

Andreas Kaluza further improve the call chain completion as described in this post. This feature will probably be part of the next ~~release~~ drop.

Stefan Henß' templates completion engine (somewhat similar to the SWT templates engine but driven by the calls recommender engine) is almost ready for ~~release~~ next drop.

But there is quite a lot in the pipeline... stay tuned.

Special Thanks

First, the Code Recommenders team now consists of two full time committers (and a bunch of students :-)). Johannes Lerch joined me back in December and has made a fantastic job since then. My thanks to him for his great passion on working on code recommenders' website, build/test system and in particular for improving code recommenders.

Second, I wanna thank the Eclipse IP team, namely Sharon Corbett and Gunnar Wagenknecht, who already spent quite some time on checking our dependencies - and will probably spent much more time later on. Thanks!

Third, thanks to my project mentor Chris Aniszczyk for pushing me through the IP process and initial steps, and thanks to Jochen Krause and Ralph Mueller for the meetings and discussions on Code Recommenders.

Please enjoy this ~~release~~ ~~update~~ drop but report all errors (even the smallest one) to the forum or directly to our bugzilla!

Thanks,
Marcel

P.S.: Say it thousand times: "It's a drop, it's a drop, it's a drop..." until there is a official review by the EMO :-)

How do I get an instance of ... IStatuslineManager?

2010-12-29T00:28:00.047+01:00

As of yesterday the Code Recommenders Demo Tour 2010 is over. This year I had the pleasure to talk at JAX, Andrena Developers Days, Eclipse Summit, FoSER, as well as on several (five or so?) Eclipse demo camps in Darmstadt, Bonn, Kassel, Dortmund and Karlsruhe. Further talks have be given by my colleague Martin at ECOOP and MSR. Quite a lot of events but is was fun and my pleasure to talk to all these audiences. Thanks for your interest in code recommenders and thank you to the organizers of these events. 2011 will be a great year: there are already a few talks scheduled for Java User Groups and others in January, February and March and I'm looking forward to these events. See you there in 2011!

However, aside traveling and talking about code recommenders we also worked on several new features - one of which I want to give a preview in this post. It's a feature I was looking for several years now and now it's there!

Whenever coding, how often did you spent hours on finding a path in the API jungle that returns you a highly required instance of a certain type? For Eclipse developers one such example might be

"How do I get an instance of an IStatusLineManager inside my Eclipse View?"

Personally, it took me more than half an hour to figure out how to obtain such an IStatusLineManager when doing it the first time. Most of the time I spent in manually traversing the Eclipse API jungle to find a method call chain that would return me an instance of the requested IStatusLineManager. The tough part in traversing the API call graph is that my base class ViewPart already provides more than 50 methods I might (!) call - not counting the methods I might invoke on all accessible fields and locals!

"Okay, but how hard can it be to scan 50 methods?" you say. Let's do some math. The call chain I was searching was three hops away. Each method gave me in average 20 more methods to invoke. Summing things up for a call chain of length 2 only, I already ended up in 50*20=1000 potential call chains to check! Make a guess how many call chains to check for a call chain of length 3... (granted, it is much more clever to check the Javadoc uses pages of each accessible type to reduce the search scope.)

The call chain I was looking for was this.getViewSite().getActionBars().getStatusLineManager(). Simple for the expert who knows the API. Challenging to figure out for a developer new to the framework.

I guess, you have been in many very similar situations, right? Haven't you ever thought "Damn, couldn't someone create a smart completion engine that does this **** tedious search for me?" Actually there are some guys who did. They created a fully automated call chain completion engine that searches the API jungle for all possible call chains that - starting from something that is available in your active working context - results in an instance of the requested type (like IStatusLineManager in the example above). This idea was developed by Naiyana Sahavechaphan, Kajal Claypool, and others in 2006 and their tool XSnippet was developed for Eclipse. Although an amazing idea and cited many times in academic world, the idea behind XSnippet never made it into Eclipse. But I can hear IntlliJ users yelling "Yeah, we have this features for years already!" ... right ... sad enough. Time to close that gap.

Java call chain completion is now available for Eclipse!

Whenever you tip ctrl+space twice now, you get this nice completion window which contains all call sequences that will result in an instance of the requested type on the left-hand-side of an assignment. Note that the exact behavior can (needs to) be configured in Eclipse Preferences as depicted below.

The call chain completion is currently under active development by Gary Fritz and Andreas Kaluza. Thanks to you both for your superb work! They made available a prototype implementation of their call chain completion engine. If you wanna check out the current prototype simply download it from http://goo.gl/fMHfh, unzip it into your eclipse/dropins/ folder, restart Eclipse and configure code completion as shown above. There is also a short demo video showing code recommenders' call chain completion in action http://goo.gl/aTgTX. Please note that this is a sneak preview and as such we (Gary, Andreas, and me) appreciate all your comments, bug reports and feature requests! The final version will come along with the first incubator release of Eclipse Code Recommenders. To make this tool as helpful as possible, please comment on this idea and tool, tell your friends, and don't forget to send all bugs and feature requests to our issue tracker. Don't be quiet - let us know what you think about this idea. Your feedback motivates!

Is this all?

No, reimplementing XSnippet is just the beginning. Can you imagine what happens when developers start sharing usage statistics for such code snippets among each other - or even share their own code snippets in a central snippet repository? We could create tons of extremely valuable code snippets for our daily work and - in combination with smart completion engines that know what we need and how to leverage these knowledge base - provide you with code snippets you actually need for your task at hand! This is where we go! Stay tuned... we are working hard on this vision!

All the best,
Marcel

Gathering Community... now!

2010-11-10T21:37:00.000+01:00

Finally, Eclipse Code Recommenders has been proposed officially by the Eclipse Foundation today. We are now in the Gathering Community Phase which precedes the Creation Review. But what exactly is the goal of this phase?

Basically it's a reality check that aims to figure out whether there is a community which is interested in the project. More formally, the Eclipse Development Process says "The proposers, in conjunction with the destination PMC and the community, collaborate in public to enhance, refine, and clarify the proposal". And "when the proposers and the EMO are confident that the proposers have sufficient community support for the proposal, the process can progress to the next step: the Creation Review."

So, here we go :-) The proposal is available here. It presents a set of of five initial (groups of) tools we want to bring to Eclipse:

Intelligent Code Completion Systems
Smart Template Engines
Usage-Driven and Crowdsourced API Documentation
Stacktrace Search Engine
API Misuse / Bug Detector

Read more about these tools in the project proposal, and send your comments, questions, support/ "I like" statements etc. to the proposals forum - and don't hesitate to ask tough questions :-)

We'd also appreciate to hear about your ideas and personal visions of how code recommenders may/should improve Eclipse.

And one last favor: Please help spread the word, for instance, via twitter, blogs, facebook or good old email! Thanks!

All the best,
Marcel

Code Recommenders Goes Eclipse!

2010-10-29T00:33:00.005+02:00

Today is probably one of the most exciting days I had in the last year and a half. The EMO gave its ack for officially proposing Code Recommenders as Eclipse Incubator Project. For me this is a welcome point in time to look back what happened in the past 18 months and to give an outline of what will happen in the next 18 months (or so...:-) ).

The Past.

Last year in July I decided to present my research project Code Recommenders the first time on an Eclipse Demo Camp in town (Darmstadt, organized by Jochen Hiller from T-Systems) - just to see whether people would like or dislike the idea of having tools that learn what is relevant for a developer and make this knowledge available in a somewhat "educated version of code completion".

Well, I must admit that my first demo left some remarkable room for improvements... However, after my presentation I had nice and motivating discussions with Bernd Kolb and others which encouraged me to continue my work and to intensify building tools that leverage various kinds of collective intelligence and to integrate them into Eclipse.

At the beginning of September my advisor received an email of Ralph Mueller in which he invited several universities to submit a poster to ESE 2009 - and we submitted. This is the poster we presented at ESE 2009:

Roughly at the same time my colleague Martin Monperrus and I worked on a paper how to extend javadocs by mined real-usage patterns and somehow Martin managed to get in contact with Boris Bokowski who reviewed some of our mined documentation snippets. After some mails we met in Darmstadt and we presented him the project and the current tools we had developed so far. But he wasn't completely hooked. Okay, back to training :-P

A few weeks later I received a mail from S&S Editor Hartmut Schlosser who asked me whether I would like to present code recommenders on a November Eclipse Demo Camp in Frankfurt. This was pretty cool experience: This was my first invited talk. Maybe this was another highlight in the past year :-P

Summit 2009: The conference was pretty cool and the poster session was real fun. Starting at 6PM I rolled up my poster and quit service at 10PM. I got to know quite a lot well-known people from Eclipse. Among them Jochen Krause from EclipseSource whose Yoxos platform was incredibly valuable for my studies because it contained thousands of plug-ins to analyze and to learn from. Over the whole evening I received a lot of input of what people liked and disliked, what they say is missing in current IDEs or had just nice and encouraging chats.

A few weeks later I attended the Eclipse Demo Camp in Frankfurt organized by Lars Martin from Itemis. This time the presentation was a lot better than a few months ago in Darmstadt and I had the pleasure to meet Stephan and Leif from Andrena (Project Usus) , Hartmut and Sebastian Meyen from S&S, Karsten Thoms and Benny Muskalla. We had a nice evening in a Greek tavern, lots of good discussions, and frosty beverages.

A few weeks later Hartmut asked me to write an online-article about code recommenders for JAX which was a huge honor. This article made it into the February edition of the German Eclipse Magazin. Another huge honor - and pleasure - for me.

Next, I received the opportunity to give a short talk at JAX 2010. There I chat up Mik Kersten from Mylyn and he took his time and patience, stepped through the slides of my talk (in an incredible speed :-) ) and checked out the prototype on my laptop. Then he sat back, looked at me and asked: "And what now? What are your plans?" Hmm, plans... "What do you think? Would it be cool for Eclipse?" "Of course!" - and there the idea of becoming an Eclipse project materialized much more than ever before...

To shorten the remainder. In June I had a long talk at Andrena Developer's Day in Karlsruhe where I presented the complete tools suite developed so far. In July I had the pleasure to assist Jochen Hiller in organizing the Eclipse Demo Camp in Darmstadt and present code recommenders again on a demo camp. This November I'm on tour: Eclipse Demo Camps in Bonn, Dortmund and Kassel. BTW: if you are near these locations consider attending these demo camps which demo pretty interesting stuff!

And I'm also very glad to present Eclipse Code Recommenders to the Eclipse Community at the Eclipse Summit Europe 2010:

Ok, that's a quick summary of the last 18 (Eclipse-centric) months. What has changed since the first demo? In the meanwhile we refined our implementations, developed new tools like bug detectors, new kinds of code search engines, stacktrace search engines, variuos code completion engines and many things more. The word 'We' may need some refinement: In the past three semesters more than 50 students supported the code recommenders project by various hands-ons, bachelor or master theses and spent innummerable hours in designing and implementing all these ideas. Many thanks to you doing all this great work!

The Future?
So what happens next? As I mentioned, the project proposal is underway and Code Recommenders will become an Eclipse Incubator. The proposal is available for review here. However, the proposal hasn't been officially published yet because we are seeking your voice to support this project!

Thus, if you like the ideas described here in the blog or in the project propsal tell your friends about this project proposal and add a comment on this post containing your name so that we can put you on the list of interested parties (or put your name on the wiki page directly)!

Many thanks,
Marcel

IDE 2.0: Bringing Collective Intelligence into Software Development

2010-08-24T17:38:00.009+02:00

A few months ago Chris started a discussion about Eclipse and Academia and how Eclipse could support research projects to participate in the Eclipse Ecosystem. Furthermore, the upcoming Eclipse Magazin will also contribute to this discussion. However, the discussion how Eclipse could help and benefit from research projects is dangling. With this post, I would like to pick pick up Chris' blog post and present an idea how eclipse and research community could get together to create something (I think) very fancy...

In my last posts, I presented our preliminary work on improving IDEs leveraging the hidden knowledge available in example code that uses other APIs (visit code-recommenders@eclipselabs and the official project homepage for more details). A few weeks ago we wrote down our vision of how future IDEs should work---which you can find and comment below. This post is basically the preprint version of this paper (which got accepted today at the Working Conference "Future of Software Engieering Research") and we would love to get the your feedback to the vision we present here. As said above, we did a lot of work to get where we are today and the question is now: Should we continue to let the visions below come reality? Clearly, this vision will only work with a very vital community around the project - which I think can be found nowhere else than at Eclipse. But how do you feel about that? Just read the vision and tell us about your opinion (yes, I know it's longer than a standard post. Sorry for that but I hope it's worth reading ;-) ).

IDE 2.0: Collective Intelligence in Software Development

Marcel Bruch, Eric Bodden, Martin Monperrus, and Mira Mezini

Software Technology Group

Department of Computer Science

Technische Universität Darmstadt, Germany

{bruch,bodden,monperrus,mezini}@cs.tu-darmstadt.de

ABSTRACT

Today’s Integrated Development Environments (IDEs) only integrate the tools and knowledge of a single user and workstation. This neglects the fact that the way in which we develop and maintain a piece of software and interact with our IDE provides a rich source of information that can help ourselves and other programmers to avoid mistakes in the future, or improve productivity otherwise. We argue that, in the near future, IDEs will undergo a revolution that will significantly change the way in which we develop and maintain software, through integration of collective intelligence, the knowledge of the masses. We describe the concept of an IDE based on collective intelligence and discuss three example instantiations of such IDEs.

1 Introduction

Under the right circumstances, groups are remarkably intelligent and are often better than the smartest person in them. – James Surowiecki: Wisdom of the Crowds

During the past decades, software systems have grown significantly in size and complexity, making software development and maintenance an extremely challenging endeavor. Integrated Development Environments (IDEs) greatly facilitate this endeavor by providing a convenient means to browse and manipulate a system’s source code and to obtain helpful documentation on Application Programming Interfaces (APIs). Yet, we argue that there is great space for improvement by exploiting collective intelligence, the knowledge of the masses.

The leveraging of user data to build intelligent and user-centric web-based systems, commonly summarized as the Web 2.0, is the source of our inspiration. A Web 2.0 site allows its users to interact with each other as contributors to the website’s content, in contrast to websites where users are limited to the passive viewing of information that is provided to them. Web 2.0 examples include web-based communities, web applications, social-networking sites, video-sharing sites, wikis, blogs, mashups, and folksonomies.

Amazon, for instance, creates recommendations based on purchase behaviors of its customers or finds interesting similar products based on how customers interact with search results. Netflix, a video-on-demand service, features a web application that leverages user ratings on movies to recommend likely interesting movies to other users. These systems have in common that they leverage crowds to continuously improve the quality of their services, either through implicit feedback (e.g., user click-through behaviors), explicit feedback (e.g., ratings for movies) or user-generated content (e.g., product reviews and movie critics).

Today’s IDEs behave more like traditional “Web 1.0” applications in the way that they do not enable their users to contribute and share their knowledge with others, neither explicitly nor implicitly, and thus hinder themselves to effectively exchange knowledge among developers. What would it mean to bring collective intelligence into software development? Figure 1a shows the current state of the practice: software developers use IDEs that are “integrated” only in the sense that they integrate all tools necessary to browse, manipulate and build software on a single machine. If a programmer has a question about a particular piece of code, for instance an API, she has to browse the web for solutions—by hand. After she has found the solution and solved her problem, the newly gained knowledge is usually lost.

Figure 1: Our vision: in the future, IDEs will be linked through global knowledge bases

Figure 1b shows our vision of the near future: IDEs will support developers through integration with a global knowledge base. This knowledge base will receive information from implicit and explicit user feedback. By implicit feedback we mean anonymized usage data that the cross-linked IDEs will send to the knowledge base automatically and spontaneously (in the figure, we represent such spontaneous activity through dashed arrows). The knowledge base will also comprise explicit user feedback in the form of user-written documentation, error reports, manuals, etc. In this work, we will show that such data can help, for example, to improve ranking heuristics, or to focus developer activity.

Crucially, the knowledge base itself is intelligent: it will use novel data-mining techniques to integrate the different sources of information to produce new information that has added value. For instance, if the knowledge base discovers that people who write an equals method in Java often write a hashCode method on the same type at the same time, or do so after a longer debugging session, then the knowledge base may be able to discover the important rule that, in Java, every type that implements equals should also implement hashCode, and that missing this rule likely causes bugs.

The remainder of this paper is organized as follows. In Sec. 2, we materialize IDE 2.0 by discussing example intelligent IDE services that leverage implicit and explicit user feedback to aid programmers in everyday software-development tasks. We show that not only feedback data itself but in particular derived information, obtained through data mining, has the potential of greatly easing the software-development process as a whole. Moreover, as the data is persisted, it will survive over time, unlike today, where much information gets lost and needs to be re-discovered over and over again. In Sec.3, we materialize IDE 2.0 by drawing parallels between the main characteristics of IDE 2.0 and those of Web 2.0. Finally, Sec. 4 summarizes the paper.

2 From IDE 1.0 towards IDE 2.0

In the following we give three examples of how research in collective intelligence can improve existing IDE services. We split the discussion of each example into three sections. IDE 1.0 sections describe the state-of-the-art in today’s IDEs. Under IDE 1.5, we briefly summarize current research to improve IDE 1.0 services. IDE 2.0 sections discuss how collective intelligence could solve some of the issues of these approaches.

Intelligent Code Completion

IDE 1.0: Code completion is a very popular feature of modern IDEs, a life without which many developers find hard to imagine. One major reason for its popularity is that developers are frequently unaware of what methods they can invoke on a given variable. Here, code completion systems (CCSs) serve as an API browser, allowing developers to browse methods and select the appropriate one from the list of proposals. However, current completions are either computed by rather simplistic reasoning systems or are simply hard-coded. For instance, for method completion, CCSs only consider the receiver’s declared type. This often leads to an overwhelming number of proposals. Triggering code completion on a variable of javax.swing.JButton results in 381 method proposals. Clearly, developers only need a fraction of the proposed methods to make their code work. Code templates are an example for hard-coded proposals. Templates (like the Eclipse SWT Code Templates) serve as shortcuts and documentation for developers. Manual proposal definitions are labor intensive and error prone.

IDE 1.5: Researchers have recognized these issues. For instance, approaches exist that analyse client code to learn which methods the clients frequently use in certain contexts, and rearrange method proposals according to this notion of relevance [2]. Tools like XSnippet, Prospector and Parseweb [7 9 10] attempt to solve the issue of hard-coded code templates by also analyzing source code, identifying common patterns in code. Although obviously useful, these systems didn’t made it into current IDEs. We argue that the primary reason for this is the lack of a continuously growing knowledge base. To build reliable models, source-code based approaches require example applications and full knowledge about the execution environment (i.e., classpath, library versions etc.). However, finding a sufficiently large set of example projects is difficult and tedious, and creating models for new frameworks is too time-consuming yet. While such approaches can sufficiently support a few selected APIs, we argue that they do not scale when tens of thousands of APIs should be supported.

IDE 2.0: So, how can we build continuously improving code completion systems then? To solve the scalability problem, code completion systems must allow users to share usage information among each other in an anonymized and automated way—from within the developer’s IDE. This continuous data sharing allows recommender systems to learn models for every API that developers actually use. IDEs are very powerful when it comes to extracting information: they have access to information about the execution environment and about user interactions, even with respect to certain APIs. But the new, massive data sets derived from this information pose a challenge. We will likely require new algorithms to find reliable and valuable patterns in this data. Whatever means future code completion systems will use to build better recommendation models, the systems will be based on shared data. It will be the users who provide this data, and it is important to realize that, as the user base grows, the recommendation systems will be able to continuously improve over time, making intelligent completions that are useful for novice developers and experts alike.

Example Code-Snippet Recommendations

IDE 1.0: Source-code examples appear to be highly useful to developers, whenever the documentation of the API at hand is insufficient [8]. This is evident by the raise of several code search engines (CSEs) over the last few years, like Google Codesearch, Krugle, and Koders, just to name a few. However, current CSEs almost exclusively use standard information-retrieval techniques that were developed for text documents. While source code is text, it also bears important inherent structure. Disregarding this structure causes less effective rankings and misleading code summaries.

IDE 1.5: Researchers have presented a number of approaches [3 5 11] that improve certain aspects of CSEs. All these approaches exploit structure, like inheritance relations, method calls, type usages, control flow and more, however they face two severe problems. First, source code provides much more structure than text. Thus, ranking systems have to take into account many more features when building the final ranking for a search query. Consequently, it is hard to derive optimal weights for these features, so that the resulting scoring function will perform as well as possible. Often, a fixed scoring systems will perform "well enough" but not be optimal. Another issue with current CSEs is that they ignore the personal experience of the user who issued the query. Many current web search engines now support “personalized search”, which leverages the personal background and interests of a user to find documents that are likely to be interesting for this user, but not necessarily for others. Current CSEs lack such functionality.

IDE 2.0: How can one improve ranking and realize personalized search in CSEs? The key to solving both problems is to leverage implicit user feedback. To solve the manual-weight-tweaking problem of search engines, recent work [4] has shown that leveraging observations of how users interact with the search results can significantly improve the precision of existing search engines. The authors used the information whether or not the user inspects a search result to automatically adjust feature weights. This produces an optimized ranking where all inspected results are listed above those that the user did not investigate. To implement personalized code search engines, one can infer the personal background (or experience) of a developer by the code she has already written. Then, CSEs could first display code examples that are similar to examples previously explored or, on demand, code examples that allow the developer to learn new information. We are certain that IDE services in general, not only those that we discussed, can greatly benefit from leveraging implicit user feedback.

Extended Documentation

IDE 1.0: Software engineers widely accept that documenting software is a tedious job. Especially open-source projects frequently lacks sufficient resources to produce comprehensive documentation. Both Sun and the Eclipse Foundation recently started to address this problem by opening their documentation platforms to their users. Eclipse asks its users to provide and update tutorials at the central Eclipse Wiki. Sun’s “Docweb” allows users to edit Javadoc API documentation, and to provide code examples or cross references to other interesting articles in the web. These tools aim to leverage a Wikipedia-style approach tailored to software documentation. Past experience has shown, however, that such systems often suffer from a lack of user participation. We believe that the primary cause for this lack of participation is the fact that people may not be willing to document APIs which they have no control over, because these APIs may change rapidly at any time: they may be completely outdated in just a few months.

IDE 1.5: Recent research therefore addresses the problem from another angle, enriching existing documentation with automatically mined documentation [1 6]. Such approaches identify frequent patterns or interesting relations in code, and generate helpful guidelines from these relations. However, generated documentation may not always be helpful. Like text mining, documentation mining uncovers any relation between code elements, no matter whether or not this relation is useful to consider. The problem is aggravated by the fact that it is sometimes the surprising relations that are the most useful. Another drawback of mining approaches is that they cannot provide rationales for their observations, leaving it up to the developer to make sense of the data.

IDE 2.0: How could collective intelligence address the issues mentioned above? The key to a solution is a mixture of explicit user feedback and user-provided content. In the future, we expect generated documentation to be judged by thousands of users, enabling people to evaluate the quality of their services immediately—tool developers and documentation providers alike. Furthermore, we expect collective intelligence to enable us to migrate documentation from older to newer versions more easily. For example, when a new version of an API becomes available, explicit user feedback will make apparent which parts of the documentation remain valid for the newer version and which parts require updating. Explicit user feedback will also allow users to attach rationale to mined documentation, allowing the documentation to not only state that users must follow a certain principle but why.

These examples are just the tip of the iceberg. We are confident that the software engineering research community will invent many more interesting techniques to generate, judge, and complete documentation.

3 From Web 2.0 to IDE 2.0

We have used the analogy to “Web 2.0” to indicate that this new generation of web applications and our view of future IDEs have something in common. In the following, we discuss the similarities between Web 2.0 and IDE 2.0 to make this analogy more concrete.

In this section, we define a set of principles that we expect successful IDE 2.0 services to follow. Some of the concepts are paraphrased from Tim O’Reilly’s principles for successful Web 2.0, described in his article “What is Web 2.0?”.

1. The Web as Platform. The web as platform is the core concept of Web 2.0. In various ways, clients and servers share data over the web. We expect the same to hold for future collaborative IDE 2.0 services. These services rely on client-side usage data and thus, the web is also fundamental to them. A notable difference between IDE 2.0 and Web2.0 is that IDEs offer a much larger spectrum of data and also allow for client-side pre-processing of data like static analysis code analysis. Such pre-processing may even be crucial to allow for proper privacy. Furthermore, one needs to distribute to clients recommendation models that are built on the server-side. Local databases or caches can increase the scalability of these systems; crucial, when dealing with millions of request per day. Whatever the particular technology may be, the web will be the platform for IDE 2.0.

2. Data is key. Data is key to any IDE 2.0 service. However, here we fundamentally differ from Tim O’Reilly’s understanding of who owns this data. In Web 2.0, data is the key factor for the success of an application over its competitors. In contrast, we strongly believe in Open Data: all collected data is publicly available. This fosters a vital ecosystem around the concepts of IDE 2.0 and enables sustainable research. Successfully IDE 2.0 services will use both raw data and derived knowledge will facilitate innovation instead of locking in data or users.

3. Harnessing Collective Intelligence. Leveraging the wisdom of the crowds is the third fundamental concept of successful Web 2.0 applications—and same holds for IDE 2.0. The examples introduced in the previous section used either user-provided content (like source code, updated documentation or code snippets), implicit feedback (like user click-through data used to improve rankings), or explicit feedback (like ratings for judging the quality of relevance of generated documentation) to build new kind of services. It is important to recognize that, while individuals may be able to build these services, these services cannot unleash their potential without the crowds sharing their knowledge. Only with collective intelligence, IDE services like intelligent code completion, example recommenders or even smart documentation systems become possible.

4. Rich User Experiences. The appearance of AJAX gave web applications a new look and feel, bringing web applications much closer to desktop applications than ever before. In the context of IDE 2.0, intelligent, context-sensitive recommender systems will evolve that recommend relevant APIs or documentation where appropriate and help to reduce the clutter in IDEs at the same time. However, providing a rich user experiences is fundamental for users to accept such services. Similar to Google Search, simple and intuitive interfaces seamlessly integrated into existing IDE concepts like code completion, quick fixes etc. are the major key to success.

5. Lightweight Programming Models. In web 2.0, mashups (applications that combine several other (web) applications to build new services on top of existing ones) evolved, building new services the application developers never considered. Excellent IDE 2.0 services will encourage others to build their services on top of existing ones by providing public and easy-to-use APIs. Clearly, in the early days we expect such services to be data-driven, i.e., they will leverage the same data for enhancing several aspects of current IDEs or to port existing services to other IDEs. Note that Open Data is necessary to enable such services. However, over time, services will use other services to build what we call IDE mashups.

4 Summary

The concepts behind Web 2.0 are a great fit for future IDE services and we expect future services to meet at least one if not almost all of these properties. However, the Software Engineering research community has to play a key role in unleashing the full power of the crowds. First, and most importantly, it has to provide an appropriate environment for building and evaluating IDE 2.0 services. Strong partners like the Eclipse Foundation or Sun/Oracle already support and promote such new IDE concepts today, and their help will be crucial to providing access to large user communities in the future. But there is an incentive for these partners: they will profit from new exciter features, making the IDE itself appear very innovative.

Second, the Software Engineering research community is the connective link between practitioners and researchers in machine learning. Most IDEs only contain instances of rather primitive machine-learning algorithms. It will be our job to identify the problems that developers face in their day-to-day work, to provide appropriate data as input for machine learners, and to evaluate and reintegrate these results into IDEs. Thus, IDE 2.0 research will create new fascinating and challenging applications of machine learning aside the current markets.

To sum up, IDE 2.0 services have much potential to improve developer productivity and provide a fantastic playground for new algorithms. They bring together several research communities at the same time, to solve a new generation of challenges in software engineering. When tackling the problem now and in a farsighted, IDE 2.0 will be one of the major research areas of the near future.

5 References

[1] Marcel Bruch, Mira Mezini, and Martin Monperrus. Improving the quality of framework subclassing directives. InMSR, 2010.

[2] Marcel Bruch, Martin Monperrus, and Mira Mezini. Learning from examples to improve code completion systems.In FSE, 2009.

[3] Reid Holmes and Gail C. Murphy. Using structural context to recommend source code examples. In ICSE, 2005.

[4] Thorsten Joachims. Optimizing search engines using clickthrough data. In KDD, 2002.

[5] Erik Linstead, Sushil Bajracharya, Trung Ngo, Paul Rigor, Cristina Lopes, and Pierre Baldi. Sourcerer: mining andsearching internet-scale software repositories. Data Min. Knowl. Discov., 18(2), 2009.

[6] Fan Long, Xi Wang, and Yang Cai. Api hyperlinking via structural overlap. In FSE, 2009.

[7] David Mandelin, Lin Xu, Rastislav Bodík, and Doug Kimelman. Jungloid mining: helping to navigate the api jungle.In PLDI, 2005.

[8] Martin Robillard. What makes apis hard to learn? answers from developers. IEEE Software, 2009.

[9] Naiyana Sahavechaphan and Kajal Claypool. Xsnippet: Mining for sample code. In OOPSLA, 2006.

[10] Suresh Thummalapenta and Tao Xie. Parseweb: a programmer assistant for reusing open source code on the web.In ASE, 2007.

[11] Hao Zhong, Tao Xie, Lu Zhang, Jian Pei, and Hong Mei. Mapo: Mining and recommending api usage patterns. InECOOP, 2009.

(Please note, this list is by far incomplete but a 4 pages limitation requires you to select just a few publications)

6 Disclaimer ;-)

This is a vision of what we want to achieve with the code recommenders project. So far you have seen preliminary versions of intelligent code completion, extendend javadocs, and example code search. In the pipeline is a API misuse detector we will present in a few weeks. However, we are currently starting to make all these tools "ide 2.0-ready" and would propose this project as Eclipse Incubator project. But this project would need your help in many ways to be successful! Thus:

Let us know whether you like the idea and would support this project when becoming an open source / open data Eclipse Project. And even if you would not support it: Tells us what would prevent you from using it. If it is a technical issue I'm sure we can fix it. In other cases we would love to learn what causes "rumbling in the tummy" ;)

If you want to learn more about the project drop us a mail and/or visit the project homepage

All the best,
Marcel

Why is Google Codesearch not 'google for code search'?

2010-07-05T15:04:00.004+02:00

Whenever we are searching some information in the web, we use Google to find it. Google is actually so popular that we often use ‘google’ as a synonym for searching the web – and sometimes we ask ourselves: “Who did we ask before Google?”

Google works amazingly well when searching the web for articles or documentation etc. However, with the large scale availability of open source code, developers started to search for example code using Google. Although text documents too, applying the same techniques (like word stemming, word splitting etc.) on source code doesn’t work too well. Therefore in October 2006 Google announced their own search engine specially designed to find source code: Google Codesearch. And with Google Codesearch a plenty of other code search engines appeared on the screen like Koders, Krugle, and ByteMyCode - just to name a few.

However, although considered being extremely useful, we had the feeling that code search engines still aren’t used by the majority of software developers. But why? For that reason we started some research on what may hinder the widespread use of such search engines. This post is about some (more or less obvious) findings and a new prototype (re)search engine that aims to solve some of the issues of existing approaches.

Issues with existing code search engines

#1: Usage of plain text-based information retrieval techniques.

The most critical issue with current code search engines (CSEs for short) is that they treat source code still more like text documents than like code. Of course, CSEs adopted their stemming algorithms and phrase detections etc. to better fit source code syntax, but these algorithms still ignore the large amount of structured information that is available in source code. For illustration, consider the code snippet below taken from the Eclipse JavaSnippetEditor. What information can you extract from this piece of source code?

protected void showStatus(String message) {
IEditorSite site=(IEditorSite)getSite();
site.getActionBarContributor().getActionBars()

.getStatusLineManager().setMessage(message);
}

Well, basically only the literals used in the snippet. You can extract the names of the methods and variables used and the simple type names given in the method or variable declaration. But how can you determine what kind of object the method getActionBars() is working on? You can’t – at least not from looking at this snippet or its enclosing Java source file only. This information can only be resolved if you have a detailed knowledge about the classpath this piece of code will be executed with and that’s what the compiler does for us: When compiling the source code to binary it resolves all these implicit dependencies using the classpath and thus knows exactly about the types and methods used (this is more or less correct since it only knows about the static types but not the runtime types – but it’s far more knowledge than we get by just looking on the source code of a single file). Without that information we loose most of the structural information in code. To compensate this, CSEs allow us to use regular expressions to get at least some meaning back into our queries. However, regex aren’t a perfect compensation of what we have lost…

This little example reveals the most fundamental problem of current CSEs: They just look in the source code without resolving the implicit (method and type) bindings. For sure, current CSE have to address many other challenges but we think that knowing about your types, methods and even inheritance hierarchies is important to find the best matching code examples. If you are interested to learn how for instance Krugle implemented its search engine I recommend you to get a copy of Lucene In Action Ed. 2. Here Krugle developers wrote a case study chapter about how they implemented their search engine using Apache Lucene and which challenges and design decisions they had to make…

#2: Ignorance of prior knowledge.

A second issue of current CSEs is their ignorance of the developer’s prior knowledge. Consider for instance a developer who is very familiar with SWT and JFace but not familiar with, say, Eclipse Help. A code search engine that presents to him hundreds of examples that use SWT and JFace are less helpful to him than examples that actually use Eclipse Help – these examples carry some new knowledge for him, and thus are much more interesting than code that does not contain any new information.

Consequently, CSEs should leverage the personal knowledge of a developer to build personalized search results. This is, however, difficult when using a web browser to submit and display query results. But it becomes much easier when we integrate this process into the IDE. We will detail on that later.

#3: Almost no IDE integration.

When using CSEs we typically have to leave the IDE and enter our code search query manually into the webpage’s search field. This is disturbing and - in conjunction with the limited query language support (using simple terms and regular expressions for search) - makes it very hard to for developers to specify their query.

Here, approaches are needed that automatically extract the relevant search terms from the code the developer is currently working , automatically sends the query and presents the results within the IDE. Such a tool would greatly ease the use of CSEs.

#4: Scoring of example code is based on words rather than on semantics.

Current scoring of example code works comparable to the scoring of plain text documents. However, as said above, code contains much more structure than text. For instance all used types, all implemented interfaces, all superclasses, overridden methods, called methods etc. Since all existing approaches do not (at least rarely) leverage this structure their current scoring mechanisms are rather simple. But when the structure in code is preserved during search, new, much more complex scoring functions become available.

In the end of this post we will present an approach how CSEs could learn what is actually relevant for a given user or query by looking at how users interact with the search results – and in response to this user feedback updates its scoring function to produce better results next time it is used.

Four issues. How can we solve them?

Let’s start with a simple example how searching for example code from inside your IDE looks like with our current prototype:

Let’s assume we created a new Eclipse View by extending the class ViewPart class as depicted in the figure below. Let’s assume further that we want to update the view’s status line using an IStatusLineManager whenever we do some background computation. The problem here is how can we obtain a handle on IStatusLineManager (cf. line 18)?

When using a CSE like Google Codesearch or Krugle we have to enter some keywords on the web page’s search field on our own like “IStatusLineManger setMessage” and hope that we get som appropriate code snippets showing how to obtain such a reference. However, when doing so, you get probably something like this:

Which you may or may not find very helpful.

How could a fully integrated code search look like then?

Well, first of all it could create the query fully automatically from the class code itself or just a user selected portion. The figure below highlights various information that could be added automatically to the query when searching for example classes similar to the whole class:

At the end the query contains the information about the overridden methods, the extended classes, the declared fields, used types and methods (IStatusLineManager and IStatusLineManager#setText() for instance).

Next the tool sends this query to the server and displays the results inside the IDE as shown below. The developer now browses the example recommendations and double clicks on those examples he find interesting to look on them in detail. In this case the second example might be very interesting since it contains a method call to IActionBars#getStatusLineManager() which is very likely to contain the information we are actually looking for:

The Code Examples Results view gives a very brief summary of each code snippet. It shows both the types and methods used within the code snippet, and gives some inheritance information like the name of the super-class and implemented interfaces where appropriate. It serves like an abstract of the class.

As said above, example #2 looks quite interesting and the developer may decide to open the source code by double-clicking on the summary. Next, a java editor opens showing the source code of the example but also highlights the relevant statements in code and put some markers on the right bar to indicate other relevant places in code:

From here, he may copy the interesting lines into his code. All within ~30 seconds :-)

That’s the prototype UI implementation of our code search engine. To summarize, it’s most important features are:

Automatic query creation from the current editor’s content (whole classes or selected lines)
Sending the query and displaying its results directly inside the IDE
Uses structure preserving queries that uses fully qualified method and type names (requires a special indexing structure on server-side too)
Condensed code summaries
Opens source code with relevant statement highlighting and markers to indicate interesting locations in code.

The UI is, however, just one side of the coin and probably the most challenging part is the backend that is actually performing the search. Although important, I tend think that details on how the backend is actually implemented is not that interesting here. However, there is one important feature I want to discuss: Learning what has been important to continuously improve the search results.

Best results always on top
How to continuously improving the ranking

As we said before, code contains much more structure than actually leveraged by current CSEs. With the availability of such a rich feature set we have to decide which features we consider to be more important than others so that at the end the best examples are ranked on top. Since we take into account different information in code (like the inheritance information, used methods and types etc.) we currently have a set of ~20 different features we use to score a code example. But determining the best weights for each of these features is rather difficult; tweaking one parameter here may affect another parameter there, and thus finally may result in a configuration worse than before.

To overcome this need a machine learning algorithm that learns the optimal parameters/weights for our scoring function. The intuition behind this algorithm is what I want to explain in a nutshell.

Consider that a developer issued a query that returned six results as shown below. Consider further that the user gave some feedback about the quality of these examples by explicitly saying that example #1 and #4 was actually very helpful, example #3 wasn’t helpful, example #6 was somehow helpful and example #2 and #5 weren’t looked at.

How could such a feedback be used to improve the quality of the search engine? Well, we could use this feedback to learn what the user actually looked at and liked or disliked. Therefore we create a partial ranking of the examples that were deemed helpful by the user, look at the features that caused the examples to be ranked that high in the actual ranking and automatically tweak the weights for these features so that the search engine would have produced an optimal ranking where all useful examples where ranked on top and the less useful ones at the bottom.

Or in really simplified: If a user clicked on the second result and ignored the first one (based on a not-interesting summary) we want the next time example #2 to appear before example #1:

This is one of the core features of the backend we implemented: A ranking engine that learns what users actually find helpful and adopts the weights/parameters of the scoring function according to the user’s implicit (or explicit) feedback. This approach could also be used to for personalized preferences and many other things and together with a (purely local) personal knowledge tracker new personalized code search engines become possible. This will be one of our most interesting future work areas :-)

Wrap up

In this post I presented the first prototype of our code search engine, an engine that learns what the user actually liked or disliked and leverages this knowledge to improve its own ranking capabilities over time. Furthermore, we outlined how structure in code could be leveraged to improve code search and how all these features could be seamlessly integrated into the Eclipse IDE.

Disclaimer :-)

This (re)search engine is ready to use and available for download (http://recommenders1.st.informatik.tu-darmstadt.de/updates/eclipse3.6/). It currently works with some initially trained weights but will improve itself over time. If you like – give it a try. Its examples database consists of all source code file of the Eclipse 3.6 (RC4) classic edition, and thus might be helpful for Eclipse plug-in developers.

As always, we would love to get your opinions on this tool and whether you feel such a code search engine could serve you better than Google code (or not - which is also a valid finding. Then your arguments would be very interesting :) ). Whatever comments you have – let us know! This engine is build to stay and to be continuously improved – for free. If you like, submit your ideas or issues to our issue tracker at eclipselabs.org and point us to the right search engine for software developers.

It might be that we are wrong with our perception of how good code search engines actually are. Tell us about your opinion by taking this 2 minutes survey: http://recommenders1.st.informatik.tu-darmstadt.de/survey/index.php?sid=91172 This would help much in learning what features you find important and which features you don't like...

For more details on the project itself visit our homepage.

All the best,
Marcel

Credits

Most of the work in this project is done by volunteers, and same is true for the code search engine. The search engine, its intelligent scoring function and basic UI has been developed by Peter Schroeder as part of his master thesis. The UI is currently improved by Sheip Dargutev and Nikolay Shindov. Thanks for your work!

Eclipse Code Recommenders at JAX2010

2010-05-04T06:24:00.000+02:00

It’s a great pleasure for me to present the Eclipse Code Recommenders tool tomorrow (Wednesday) evening at Eclipse Tools Day @ JAX2010.

I’m very happy to talk about the ideas and concepts behind code recommenders ranging from intelligent code completion and mined Javadocs up to smart bug detectors that can identify “strange” API uses and warn developers in the case of serious programming mistakes…

I’m looking forward to see you – and I would be happy to take your questions :-)

Cheers,
Marcel

Time and Location: 05.05.2010 | 18:10 - 18:30 | Watfordsaal I

It’s all about intelligent code completion...

2010-05-03T00:02:00.002+02:00

On the advantages of code completion

The code completion feature of modern integrated development environments (IDEs) is extensively used by developers, up to several times per minute. The reasons for their popularity are manifold.

First, usually only a limited number of actions are applicable in a given context. For instance, given a variable of type java.lang.String, the code completion system would only propose members of this class but none of, say, java.util.List. This way, the code completion prevents developers from writing incompilable code by proposing only those actions that are syntactically correct.

Second, developers frequently do not know exactly which method to invoke in their current context. Code completion systems like that of Eclipse use pop-up windows to present a list of all possible completions, allowing a developer to browse the proposals and to select the appropriate one from the list. In this case, code completion serves both as a convenient documentation and as an input method for the developer.

Another beneficial feature is that code completion encourages developers to use longer, more descriptive method names resulting in more readable and understandable code. Typing long names might be difficult, but code completion speeds up the typing by automating the typing after the developer has typed only a fraction of the name.

On the limitations of code completion

However, current mainstream code completion systems are fairly limited. Often, unnecessary and rarely used methods (including those inherited from superclasses high up in the inheritance hierarchy) are recommended. Current code completion systems are especially of little use when suggestions are needed for big (incoherent) classes with a lot of functionality that can be used in many different ways.

For illustration, consider the code snippet depicted in listing above. Let’s assume that the developer triggered code completion on the swtTextWidget variable in line 3. Take a break and think about what code completion should offer to the developer in this context…

Well, let’s see what it actually offers:

164 methods seems to be too much, right? We don’t need methods like getMonitor or removeDisposeListener here. But before continuing the discussion (and to make clear that I’m not bashing SWT :-)), let’s see how code completion looks like for SWING classes like, say, JButton, after we invoked the constructor on such an instance:

One can’t see an improvement, right? But let’s get back to the example.

Because of the overwhelming number of proposals, we looked at the source code of several Eclipse plug-ins to see how developers actually use instances of SWT Text in their code. Thereby we observed that developers rarely used more than five methods of Text (in the context of IDialogPage#createControl()) and typically only setText, setLayoutData, addModifyListener, and setFont. Thus, the remaining 160 methods unnecessarily bloated the code completion window making it hard for developers to see what is actually relevant for their task at hand.

Can we reduce the clutter in code completion?
The example above outlines the basic problem of current code completion systems. Our solution to this problem - a context-sensitive code completion - is shown in below. Instead of presenting all 164 potentially callable methods on the text variable, code completion only presents the most likely ones in this context to the user, i.e., setText, setLayoutData, and addModifyListener:

In a (very small) nutshell, the tool works as follows: (i) we grabbed a few Eclipse plug-ins, (ii) looked on how developers actually used Text widgets in their code, (iii) created a database from these usages, (iv) built an intelligent code completion engine that (every time code completion is triggered) looks up the appropriate methods a developer may be interested in, and (v) presents them to the user.

Of course, we Text is not the only class we examined. We created databases for Eclipse SWT, JFace and some parts of Eclipse UI – and if you are interested to check out the tool, visit our project page and see a demo screencast showing the Eclipse Code Recommenders tool in action.

Disclaimer ;-)

The intelligent code completion feature of the Eclipse Code Recommenders tool is a research prototype. And as such, we appreciate your comments about the idea, implementation issues and other cool ideas.

So, what’s your feeling about having such an intelligent code completion? Do you think having this in Eclipse would be a cool (and helpful) feature? Should we start a large-scale training for more Eclipse Frameworks?

All the best,
Marcel

Introducing Myself to Planet Eclipse

2010-03-30T17:33:00.004+02:00

Unfortunately, my first post is already published. However, I would like to introduce myself, my new blog and the topics I want to present here to planet eclipse.

I’m a PhD student at Darmstadt University of Technology, Germany. My research work deals with code recommender systems, i.e., I’m working on tools that (aim to) support developers on their daily work by improving several features of current IDEs.

My research focus is on mining large-scale code repositories to find valuable knowledge in example code and bringing back this knowledge to the IDE by the means of (i) intelligent code completion, (ii) extended Javadocs, or (iii) smart bug detection tools that warn developers if they misuse the framework’s API. The Eclipse Code Recommender is my tool where I’m trying to put all things in I’ve found valuable and want others to give it a try and to report whether these ideas are actually useful to someone.

(BTW: Thanks to the guys from EclipseSource and Yoxos which provide this incredible huge code repository of Eclipse plug-ins. I "misused" this one already several times on late Sunday evenings to get tons of example code for my studies :-) ) .

So, what is my objective for this new blog? It’s twofold. First, I would like to share my ideas and new tools with the Eclipse community to learn what might help us on our daily work (and maybe get some feedback on these ideas). But additionally, I want to write about other research tools that do fancy things with Eclipse and encourage others to give them a try. And as you have read recently, there is much interesting work going on (like Code Bubbles but also many other interesting tools) that are worth be recognized by the community.

A few words on my own Eclipse experience: I made my first steps with Eclipse with version 2.1 (how long is this...?) and started to write my first plug-ins with version 3.0. Since then I’ve been using Eclipse and (for whatever reasons) never touched another IDE (except a short time where I tried Netbeans but came back rather quickly). As part of my teaching assistant work I’m organizing Eclipse Hands-on trainings where we “create” round about a dozen Eclipse programmers per semester - somehow my personal, little summer of code, I guess :-D

I think, that’s it. I would love to get in contact with you, hear your thoughts about research projects, ideas and tools I’m going to present, but I'm also interested in whatever you think that could be improved to help Eclipse to stay the best IDE on the planet. See you!

Cheers,

Marcel

The Problem of Incomplete Javadocs

2010-03-30T12:12:00.002+02:00

The Problem of Incomplete Javadocs

Good and comprehensive documentation is crucial for the success of open source software. But creating such documentation takes time and energy, is boring and has almost no immediate rewards. Consequently, documentation of open source frameworks is (too) often incomplete or outdated.

However, whenever there are users of a framework there is example code that uses the framework's API. And if there is example code, the question arises whether information about how to use the framework's API can be extracted directly from example code.

We think so, and thus started to study how documentation could be completed by automatically mined documentation. So far we concentrated on mining documentation required fore developers that plan to extend a given baseclass and created what we called "subclassing directives" from code.

In a nutshell, subclassing directives are generalizations of frequently made observations in code like "Subclasses of Wizard always override its method addPages()" or "Reimplementors of Dialog.createContents() may call its super implementation." etc. Our findings are summarized in our paper "Mining Subclassing Directives" published on the 7th Working Conference on Mining Software Repositories 2010 which takes place in May 2010. The Extended Javadoc View presented here is a result of this research work.

This post describes the basic concepts behind the Extended Javadoc View, provides some examples of how mined documentation could be integrated non-intrusively into Eclipse, and how others may extend the view to provide their own documentation providers. Please note that this project is still work in progress. That means that there is much more work ongoing (see Sketchbook Page about the proposal) and appreciates your feedback.

The Extended Javadoc View

The extended Javadoc View is essentially an aggregator of different information sources for a single code element like a class, method, field or parameter. It is designed as a replacement for the existing Eclipse Javadoc. It provides basically the same functionality as the Eclipse Javadoc View. Let's walk through the existing documentation providers.

Javadoc Tab

The screenshot below shows the view displaying the standard Javadoc information of the JFace Dialog class.

But replacing on view with another one is not a big deal. The interesting part comes with the other tabs in the view: Subclassing Directives and Subclassing Patterns. These tabs contain mined information about how developers typically extended the selected code element. Let’s look on the Subclassing tab in more detail now.

Subclassing Directives Tab

As said above, subclassing directives are generalizations of frequently made observations in example code like "Subclasses of Wizard always override its method addPages()" or "Reimplementors of Dialog.createContents() may call its super implementation". The screenshots below give two examples for these mined directives are presented to a user.
The first screenshot gives a quick summary which methods are typically overridden by subclasses of JFace Wizard. The second screenshot shows a detailed look on Wizard's addPages() method and informs a developer which methods are frequently called within the control-flow of addPages(), namely, Wizard.addPage() and Wizard.addPages(). For both methods the percentage is given how frequently these methods actually have been called to allow developers to decide whether these methods are relevant for him and his task at hand or not.

Such subclassing directives are currently mined for almost all Eclipse 3.5 classes were extensions of these classes could be found in our example code base.
However, displaying which methods to override and to call is just one thing you can do with an extended documentation provider. Let's look on the Subclassing Patterns tab in more detail.

Subclassing Patterns Tab

Subclassing patterns try to group observed extensions of a base class into typical extension patterns, i.e., they cluster subclasses by similarity to find patterns in data. For illustration of the results, look on the following screenshots below. The first picture shows the frequent subclassing patterns found for the JFace ViewerComparator class. It states that typically either the method ViewerComparator.compare() is overridden or ViewerComparator.category() but typically not both at the same time (even if possible). It also states that extenders typically stick with the first pattern (~82%) and only in 19% follow pattern two.

Also for the JFace Dialog class some patterns can be found. Here developers typically overwrote the createDialogArea() method and often the methods okPressed and configureShell. However, also other patterns exist that directly respond to buttonPressed events.

For JFace Wizard two patterns can be found: The standard pattern (overriding performFinish and addPages) and a mixture of several other ways of extending Wizard.

Venturing a look at the Future

To my opinion, the current Extended Javadoc View is an interesting approach that shows what can be found in client code. But much more things can be found in code that might be used to enrich existing documentation. But to make this come true much more aspects need to be considered. We are currently working on a draft for a task-oriented, crowd-sourced API documentation which grounds (at least partially) on mined documenation. How do you feel about that? Does this sound interesting for Eclipse? We appreciate your comments!