I started working on a huge multi-module maven project these days. Trust me, working with ~150 maven modules at a time is a pain for a developer. On the other hand there’s a lot of room for me to improve the development environment & make dev cycles efficient.
Some more context: The team is pretty big; over 100 developers on a single codebase working in a ‘continuous integration’ fashion: little branching, keeping the integration gap small, automated testing, CI servers etc. Each developer checks-out entire trunk that contains ~150 maven modules linked via the default ‘x-SNAPSHOT’ dependency. Maven veterans probably flinch now because this way of managing internal dependencies in a large project is probably considered ‘lame’. Maven ‘by-the-book’ approach may put emphasis on keeping the internal dependency links to a specific version. I’m a pragmatic developer and I see pros & cons of both approaches. This blog post, however, does not dwell on the subject of internal dependencies. I want to focus on the most effective dev environment set-up and the choice of the IDE.
“Toggle IDE/local-repo dependency” pattern
Couldn’t find a good name for this pattern. Theoretically, it can give a nice dev performance boost in a huge multi-module project. Here’s the idea: out of the 150 modules we will work on the handful only. Therefore I want to be able to quickly decide which module should be a concrete project/module in the IDE and which should a mere jar dependency resolved via the local maven repo. It is a sort of a toggling dependency IDE/local-repo.
Here’s how I see it working:
1. The developer pulls in a smaller module set into an IDE by selecting a relevant sub-parent in the multi-module tree.
2. At some point the developer needs to perform changes in the module that is not pulled into the IDE. He pulls in the relevant module(s). Dependencies inside IDE are updated from jar-dependencies to: project-dependencies (Eclipse) or module-dependencies (IDEA).
Here’s how my team so far coped with the project and how the above pattern could be used.
At the moment I think more than half of the team uses maven eclipse:eclipse plugin to generate configuration that is consumable by Eclipse. In our environment it is not very efficient because after every svn update developer has to regenerate project configs (in case someone changed internal/external dependencies) and restart Eclipse (‘just’ refreshing Eclipse didn’t work well occasionally).
Team members who use IntelliJ IDEA never use idea:idea maven plugin. I cannot confirm that but I heard feedback that it was not generating the correct configurations for latest & greatest IDEA. However, most devs simply don’t need idea:idea because maven support built-in to IDEA 9+ is simply brilliant. I hardly imagine someone wanting to step back and generate configurations via idea:idea.
Toggling dependencies IDE/local-repo is possible but somewhat inconvenient. It requires specifying project names by hand at command line; then refreshing/restarting IDE.
I don’t recall anyone using it in my team. The plugin seems to be making progress – I heard several good reviews. In our particular environment developers complained on reduced perceived performance of Eclipse and they fell back to generating configs via eclipse:eclipse. I tested m2eclipse myself recently with our project; there were things I liked about it. Yet, I still found Eclipse choking (even If I fine-tuned the plugin).
Toggling dependencies IDE/local-repo is possible but I cannot comment how it behaves as we haven’t used m2eclipse too much.
The built-in maven plugin for IDEA is getting more & more popular in my team. It does not seem to reduce the perceived performance of the IDE. However, IDEA tends to be a bit slower than Eclipse anyway. IDEA 9+ is fast enough for me – I tend to use it more often than Eclipse. My experience with IDEA 8- is limited because I rarely had enough patience to wait until it started. Spanking new IDEA 10 is marketed as faster than 9. I have not tested it yet but I want to extend my thanks to JetBrains for working on it!!!
Back to the maven support in IDEA and the context of our huge multi-module project. Options that we absolutely needed to tweak were:
1. Disable files/folders synchronization on IDEA frame activation. In our case, IDEA choked every time one ran any maven goal outside of the IDEA.
2. Disable automatic maven project reimporting (it’s the default anyway). It is related to the #1.
Also, be sure to read some of the posts on how to improve IDEA performance in general (most notable: choose wisely the plug-in set enabled at IDEA; get SSD drive; etc.)
In our project we found this feature very useful: the separation of 2 different actions: ‘reimporting maven projects’ and ‘updating folders’. The first one makes sure the project internal/external dependencies are linked well. The latter makes sure sources are generated for modules that need them (generate-sources).
Finally, toggling dependencies IDE/local-repo works like a charm in IntelliJ!!! Basically, all I need to do is “Import module from existing model”, point out the relevant pom.xml. IDEA takes care of updating the dependency links of my existing modules to use IDE links rather than local-repo links. How cool is that?!
I still keep myself using both Eclipse & IDEA as I work with various developers. However, it’s going to be harder to get back to Eclipse…
Disclaimer: Please notice the date of this blog post! I don’t know if the problems I mentioned with some of the tools will still be valid in future. Hopefully all those tools will be great, fast & very convenient for working with huge maven projects. Respect to all the people who work on the dev tools that we use today (Eclipse, IntelliJ IDEA, eclipse:eclipse, idea:idea, m2eclipse). Thanks a lot!!!
Hi,
I disagree with you that eclipse:eclipse goal in Maven is bad for big project. I use these tool for most of Apache projects like Karaf, ServiceMix and Camel, which have tens of modules. I really don’t understand why some of you have to restart your Eclipse.
If you build all from command line you don’t have to speed up IDE because shell will be always faster.
Hey,
I wouldn’t say eclipse:eclipse is bad for big project. “Big” is relative and there’s also context that determines everything. eclipse:eclipse is simply not efficient for us. For example, it forces generate-sources phase (at least the version of this plugin as of today). I cannot turn it off and it takes very long in our environment due to many inplaced war dependencies. Hopefully this will be implemented with the next version (I saw it in their tracker).
Restarting Eclipse after eclipse:eclipse is what developers do here. They claim that sometimes the project dependencies/configs are not correctly refreshed without it. I faced this problem few years ago so, maybe it’s fixed by now.
>If you build all from command line you don’t have to speed up IDE
We do build from command line. The reason we need to speed up the IDE is because it’s a big web. This requires decent support for hot deploy, instant view of the change, etc.
I really recommend using m2eclipse for the Eclipse users. It solves the problem of having to regenerate the project settings on each update and has a number of other benefits.
If you are working against a slow m2 repo, I also recommend putting it in offline mode in the settings.
Regarding pulling in more modules / projects, I find it useful to have only a couple of groups of projects per workspace, organized into working sets.
If you discover that you need to work on more projects, import them into a new workspace. Remember to start Eclipse with -showlocation so you know which one you are in.. :)
Also, m2eclipse can download source code automatically when you go into code that you haven’t imported the project for, as long as the source coded is deployed in the Maven repo.
Hope that helps someone..
I agree with Viktor. I favor m2Eclipse ofer eclipse:eclipse. We use m2Eclipse on a project with approx. 50 modules and it works. Switching from workspace dependency to jar is easily done by opening/closing the project in Eclipse. But I try to have as few modules open as possible because it tends to start building in the background, which slows down the IDE. To create the initial config from Maven you have to use eclipse:m2eclipse only once. But the target is only available in Maven Eclipse plugin 2.7, so you might have to set the plugin version. And from time to time it does not get the changed dependencies from changes in poms, it might be necessary to build the module first on the commandline and refresh. (So yes, it’s not without problems … ;-)
I have to thank you for the efforts you have put in penning this site.
I’m hoping to see the same high-grade blog posts from you later on as well. In truth, your creative writing abilities has encouraged me to get my very own site now ;)
Asking questions are really good thing if you are not understanding anything fully, but this piece of writing presents pleasant understanding even.
naturally like your web-site however you need to
take a look at the spelling on quite a few of your posts. Many of
them are rife with spelling problems and I to find it very troublesome to
tell the truth on the other hand I will surely come
back again.
Nice post. I learn something new and challenging on websites I stumbleupon on
a daily basis. It will always be interesting to read through
content from other authors and use something from other web sites.
Ahaa, its good discussion regarding this paragraph here at this web site,
I have read all that, so at this time me also commenting here.