Using JDT’s ASTParser with External Java Files

Eclipse’s Java Development Tools (JDT) are a powerful set of libraries. Of particular interest to me is the Abstract Syntax Tree (AST) API that it has which is extremely robust and full-featured. You can generate an AST representation of existing code (for modification or analysis), you can use the AST API to generate code, and probably a bunch of other things. JDT is intended to be used as part of an Eclipse plug-in, but that doesn’t mean it can only be used that way. I was interested in creating a standalone application that could generate and analyze the AST for a given Java program. Not just for Java programs that are within an open Eclipse project, but any Java program. There are plenty of articles out there for doing this as a plug-in. Here are a couple:

These articles did not, however, provide sufficient information on how to parse an external Java file using JDT. I searched for a while on the subject, but didn’t come up with much of anything, so I came up with my own solution which I will present below.

I realized that an ASTParser is perfectly content consuming a char[] as its source program. This meant that all I had to do was convert the contents of a Java file to an array of characters. The following code is what I came up with for that part of the solution:

public char[] getFileContents(File file) {
	// char array to store the file contents in
	char[] contents = null;
	try {
		BufferedReader br = new BufferedReader(new FileReader(file));
        	StringBuffer sb = new StringBuffer();
		String line = "";
		while((line = br.readLine()) != null) {
			// append the content and the lost new line.
			sb.append(line + "\n");
		}
		contents = new char[sb.length()];
		sb.getChars(0, sb.length()-1, contents, 0);

		assert(contents.length > 0);
	}
	catch(IOException e) {
	        System.out.println(e.getMessage());
	}

	return contents;
}

(see the gist on GitHub)

In the above method, we take the given File and start reading from it line by line with a BufferedReader. Each line that we read, we append to a StringBuffer along with the new line character that gets dropped. After the end of the file is reached, we stick the entire contents of the StringBuffer into a character array which is then returned.

All that remains then is to create the ASTParser that uses the character array created by the above method. That can be done with the following code:

public CompilationUnit createParser(char[] contents) {

	// Create the ASTParser which will be a CompilationUnit
	ASTParser parser = ASTParser.newParser(AST.JLS4);
	parser.setKind(ASTParser.K_COMPILATION_UNIT);
	parser.setSource(contents);
	parser.setResolveBindings(true);
	CompilationUnit parse = (CompilationUnit) parser.createAST(null);

	return parse;
}

(see the gist on GitHub)

In the above code, createParser takes in a char array like the one we got using the other method as input. It then creates an ASTParser for JLS4 of the kind CompilationUnit. The given contents are then passed in as the source of the parse. The parser is then cast to a compilation unit which is then returned. This CompilationUnit is the root node of the AST of the given source program and it can be traversed and manipulated using the rest of the JDT AST API.


Edit:

I found an approach similar to mine here.

Advertisements
Tagged , ,

13 thoughts on “Using JDT’s ASTParser with External Java Files

  1. Vasa says:

    When I tried the same code, I got the following exception. I have no clue how to fix this. Sould you help me with this?

    Exception in thread “main” java.lang.NoSuchFieldError: ignoreMethodBodies
    at org.eclipse.jdt.core.dom.CompilationUnitResolver.parse(CompilationUnitResolver.java:491)
    at org.eclipse.jdt.core.dom.ASTParser.internalCreateAST(ASTParser.java:1194)
    at org.eclipse.jdt.core.dom.ASTParser.createAST(ASTParser.java:801)
    at itjava.industry.testAST.runTest(testAST.java:29)
    at itjava.industry.testAST.main(testAST.java:37)

    • Hi Vasa,

      I don’t have access to your code, so I don’t know what the root of the problem is. However, it looks like a field called ignoreMethodBodies is causing the problem. For an explanation of NoSuchFieldErrors, check out this StackOverflow post (http://stackoverflow.com/questions/6686753/nosuchfielderror-java). This might give you some insight into troubleshooting your error. My guess is that there has been some compilation mismatch and ignoreMethodBodies doesn’t exist anymore.

      • Vasa says:

        Thanks Josh. Sure, I shall check it out. I used exactly the same code given in this article. Made the methods static,had a main and then ran the file

        public static void main (String args[]){
        final File file = new File(LocalMachine.home+”files/db.java”);
        char[] contentsOP = getFileContents(file);
        CompilationUnit parseOP = createParser(contentsOP);
        }

        I have the following jars..
        org.eclipse.core.contenttype_3.4.100.v20100505-1235.jar
        org.eclipse.core.jobs_3.5.1.R36x_v20100824.jar
        org.eclipse.core.resources_3.6.0.v20100526-0737.jar
        org.eclipse.core.runtime_3.6.0.v20100505.jar
        org.eclipse.equinox.common_3.6.0.v20100503.jar
        org.eclipse.equinox.preferences_3.3.0.v20100503.jar
        org.eclipse.jdt.compiler.apt_1.0.300.v20100513-0845.jar
        org.eclipse.jdt.compiler.tool_1.0.100.v_A68_R36x.jar
        org.eclipse.jdt.core_3.6.1.v_A68_R36x.jar
        org.eclipse.jface_3.6.1.M20100825-0800.jar
        org.eclipse.osgi_3.6.1.R36x_v20100806.jar
        org.eclipse.osgi.util_3.2.100.v20100503.jar
        org.eclipse.text_3.5.0.v20100601-1300.jar

        I realize that it might be some incompatibility issue.
        Will this info help you to figure out anything?

      • Where is the field ignoreMethodBodies located? I am guessing this is somewhere in the file that you are trying to parse… is it?

  2. Vasa says:

    Nope.. ignoreMethodBodies is located in org.eclipse.jdt.core_3.6.1.v_A68_R36x.jar

    I don’t know how to see the code within the jar as they are all class files. But when I searched online I was able to see a version that has the method ignoreMethodBodies in it.

    http://grepcode.com/file/repository.grepcode.com/java/eclipse.org/3.6/org.eclipse.jdt/core/3.6.0/org/eclipse/jdt/core/dom/CompilationUnitResolver.java#CompilationUnitResolver.0IGNORE_METHOD_BODIES

    • Well I have a couple of suggestions for you:
      1. you might try reinstalling the JDT because it seems like the issue is there.
      2. you should consider inquiring about your issue on the mailing list or reporting a bug (http://www.eclipse.org/mail/)
      3. you can try posting your question in more detail on StackOverflow (http://stackoverflow.com). There is a large community of developers there and perhaps someone has seen this problem before.

      • Vasa says:

        Thanks a lot josh.. I was able to finally fix it. This was due to an eclipse environment issue. The Apache server lib has ecj.jar and it is not compatible with org.eclipse.jdt.core_3.6.1.v_A68_R36x.jar

        Removing ecj.jar fixes the problem.

      • @Vasa Glad you were able to fix it. Thanks for posting how you troubleshot the issue. Good skill on the rest of your project!

  3. Vasa says:

    Thank you Josh, for your directions and suggestions. It was really helpful.

  4. Radu Mirescu says:

    Hi! I tried to serialize a Java AST in Xml. Everything worked fine except the MethodDeclaration that contains in their body an “assert” keyword. For these MethodDeclaration their body appears to be empty.

    Checked the documentation and I found for assert that I should do something like

    parser = ASTParser.newParser(AST.JLS3);
    HashMap compilerOptions = new HashMap();
    compilerOptions.put(“Source Compatibility Mode”,”1.4″);
    parser.setCompilerOptions(compilerOptions);

    but didn’t help ( with “1.5” and “1.6” neither).

    Do you have any clue to address this issue ?!

  5. Kalyani says:

    Thanks a lot. This is exactly what I wanted.

  6. Adrian says:

    I was trying to do something similar. But I found that:

    > parser.setResolveBindings(true);

    by itself (like in your example) does not work. The bindings are not
    set when reading from an external file. When I try to actually use
    the binding (e.g., in a “visit(VariableDeclarationStatement node)” in
    a class that extends “ASTVisitor”), the binding is NULL.

    In fact, the documentation of ASTParser.setResolveBindings() says that
    you need to use some setProject(IJavaProject) or
    setEnvironment(String[], String[], String[], boolean) and
    setUnitName(String).

    Does anybody have an example that works for binding?

    • Adrian says:

      All right, solved it. After:

      “parser.setResolveBindings(true);”

      you need:

      …………………………………………………………….
      parser.setUnitName(“dummyProjectName/pathToSourceRelativeToDummyProjectName”);
      String[] classPath = {“pathToBinaries”};
      String[] srcPath = {“pathToSource”};
      parser.setEnvironment(classPath, srcPath, null, false);
      …………………………………………………………….

      or variations of this, depending on what you need. I.e., just like
      the documentation for “setResolveBindings” says. But again, without
      this, you won’t get the bindings.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Advertisements
%d bloggers like this: