Eclipse Xtext Importing- Martin Baker

How Do I Call Native Java Code?

I can call native java code from my DSL like this:

Here is some simple code for my DSL.

// my DSL code
package example1
import example1.myJavaCode

class HelloWorld {
	def static void main(String[] args) {
		val myJavaCode m=new myJavaCode()
		m.getString("me")
	}
}

It calls this simple native java code

//java code written by me
package example1;

class myJavaCode {
	String getString(String x) {
		return "hello "+x;
	}
}

Notes:

I was originally going to write println(m.getString("me")) but my DSL does not yet handle static imports so the code does not yet do anything.
We need to explicitly import example1.myJavaCode even though it is in the same package. If we want to automatically import from our own package we need to add scoping fragment as described later.

I put both of these files in my 'src' directory and they work fine.

How does this work? Well, of course, our DSL generates java code which can then call other java code. However this does not mean that each class in our DSL is self contained because it still needs to check that it is valid to call this code. Our DSL code needs to 'see' the model for native java code so it 'knows' the signature of the method call. To demonstrate this, if I replace this line in the DSL code:

m.getString("me")

with this line:

m.getString(1)

Then we get the expected error message:

Incompatible types. Expected java.lang.String but was int

and so it will give error messages if the signature is wrong, which means that it must be able to 'see' the structure of our own java code. How does it do that?

The imports work because the grammar for our DSL uses the magic name 'importedNamespace' where 'import' is defined in the grammar as we shall describe below.

Our DSL Code

So lets attempt to build our own DSL (using xbase) that can run the code above and call native java code. I created a xtext project (see screenshots on this page for a step-by-step guide - but use the following grammar instead).

I put the following code into the grammar:

grammar com.euclideanspace.importTest.MyTest with org.eclipse.xtext.xbase.annotations.XbaseWithAnnotations

generate myTest "https://www.euclideanspace.com/importTest/MyTest"

import "http://www.eclipse.org/xtext/xbase/Xbase" as xbase
import "http://www.eclipse.org/xtext/xbase/Xtype" as xtype
import "http://www.eclipse.org/Xtext/Xbase/XAnnotations" as annotations
import "http://www.eclipse.org/xtext/common/JavaVMTypes" as types

File returns MyTestFile :
   ('package' package=QualifiedName ';'?)?
   (imports+=Import)*
   (MyTestTypes+=Type)*
;

Import returns MyTestImport :
'import' importedNamespace=QualifiedName ';'?
;

Type returns MyTestTypeDeclaration :
  {MyTestTypeDeclaration} annotations+=XAnnotation*
  ({MyTestClass.annotationInfo = current}
    'public'? abstract?='abstract'? 'class' name=ID ('<' typeParameters+=JvmTypeParameter (',' typeParameters+=JvmTypeParameter)* '>')?
    ("extends" extends=JvmParameterizedTypeReference)?
    ('implements' implements+=JvmParameterizedTypeReference (',' implements+=JvmParameterizedTypeReference)*)?'{'
    (members+=Member)*
    '}'
  )
;

Member returns MyTestMember:
  {MyTestMember} annotations+=XAnnotation*
  (
    {MyTestField.annotationInfo = current}
    (extension?='extension' (final?='val' | 'var')? type=JvmTypeReference name=ID?
    | static?='static'? (type=JvmTypeReference | (final?='val' | 'var') type=JvmTypeReference?) name=ID)
    ('=' initialValue=XExpression)? ';'?
    | {MyTestFunction.annotationInfo = current}
    ('def' | override?='override') static?='static'? (dispatch?='dispatch'?)
    ('<' typeParameters+=JvmTypeParameter (',' typeParameters+=JvmTypeParameter)* '>')?
    ( =>(returnType=JvmTypeReference name=ID '(')
    | =>(returnType=JvmTypeReference name=ID '(')
    | name=ID '('
    )
    (parameters+=Parameter (',' parameters+=Parameter)*)? ')'
    ('throws' exceptions+=JvmTypeReference (',' exceptions+=JvmTypeReference)*)?
    (expression=XBlockExpression)?
    | {MyTestConstructor.annotationInfo = current}
    'new'
    ('<' typeParameters+=JvmTypeParameter (',' typeParameters+=JvmTypeParameter)* '>')?
    '(' (parameters+=Parameter (',' parameters+=Parameter)*)? ')'
    ('throws' exceptions+=JvmTypeReference (',' exceptions+=JvmTypeReference)*)?
    expression=XBlockExpression
) ;

Parameter returns MyTestParameter:
  annotations+=XAnnotation*
  parameterType=JvmTypeReference varArg?='...'? name=ID;

I then ran the mwe2 file.

and put the following code into the 'jvmmodel/MyTestJvmModelInferrer.xtend' file:

package com.euclideanspace.importTest.jvmmodel

import com.google.inject.Inject
import org.eclipse.xtext.xbase.jvmmodel.AbstractModelInferrer
import org.eclipse.xtext.xbase.jvmmodel.IJvmDeclaredTypeAcceptor
import org.eclipse.xtext.xbase.jvmmodel.JvmTypesBuilder
import com.euclideanspace.importTest.myTest.MyTestFile
import com.euclideanspace.importTest.myTest.MyTestClass
import com.euclideanspace.importTest.myTest.MyTestConstructor
import com.euclideanspace.importTest.myTest.MyTestField
import com.euclideanspace.importTest.myTest.MyTestFunction
import static org.eclipse.xtext.util.Strings.*
import org.eclipse.xtext.common.types.JvmOperation
import org.eclipse.xtext.common.types.JvmField
import org.eclipse.xtext.common.types.JvmTypeReference
import org.eclipse.xtext.common.types.JvmConstructor
import org.eclipse.xtext.common.types.JvmParameterizedTypeReference
import org.eclipse.emf.common.util.EList

class MyTestJvmModelInferrer extends AbstractModelInferrer {
     
  @Inject extension JvmTypesBuilder
  
  def dispatch void infer(MyTestFile element,
      IJvmDeclaredTypeAcceptor acceptor,
      boolean isPrelinkingPhase) {
      for (classElement : element.myTestTypes) {
      	if (classElement instanceof MyTestClass) {
      	  val MyTestClass ec=classElement as MyTestClass
      	  buildClass(acceptor,ec,element.getPackage())
        }
      }
  }

  def void buildClass(IJvmDeclaredTypeAcceptor acceptor,
  	                 MyTestClass ec,String pck){
  	var String qualifiedName = ec.name
  	if (pck != null){
  	  qualifiedName = pck + "." + ec.name
  	}
    acceptor.accept(ec.toClass(qualifiedName)).initializeLater [
      documentation = ec.documentation
      var JvmParameterizedTypeReference ext = ec.getExtends()
      if (ext!=null && superTypes!=null) {
        superTypes += ext.cloneWithProxies
      }
      var EList imps = ec.getImplements()
      for (imp:imps){
      	superTypes += imp.cloneWithProxies
      }
      for (methodElement : ec.members) {
        if (methodElement instanceof MyTestFunction) {
          val MyTestFunction me=methodElement as MyTestFunction
          members += buildMethod(me)
        }
        if (methodElement instanceof MyTestConstructor) {
          val MyTestConstructor me=methodElement as MyTestConstructor
          members += buildConstructor(me)
        }
        if (methodElement instanceof MyTestField) {
          val MyTestField fe=methodElement as MyTestField
          members += buildField(fe)
        }
       }
     ]
    }

    /**
     * method definition, starts with 'def' like xtend
     */
    def JvmOperation buildMethod(MyTestFunction me){
      var String methodName = me.name;
      var JvmTypeReference methodType = me.returnType
  	  return me.toMethod(methodName,methodType) [
        for (par : me.parameters) {
          if (par.name != null && par.parameterType != null)
            parameters += par.toParameter(par.name,par.parameterType)
        }
        documentation = me.documentation
        body = me.expression	
      ]
  	}

    /**
     * constructor, starts with 'new' like xtend
     */
    def JvmConstructor buildConstructor(MyTestConstructor me){
        return me.toConstructor() [
        for (par : me.parameters) {
          if (par.name != null && par.parameterType != null)
            parameters += par.toParameter(par.name,par.parameterType)
        }
        documentation = me.documentation
        body = me.expression	
      ]
  	}

    /**
     * variable/value definition, starts with 'var' or 'val' like xtend
     */
    def JvmField buildField(MyTestField fe){
      if (fe.type == null) {
      	println("type = null")
      	return null;
      }
  	  return fe.toField(fe.name,fe.type) []
  	}

}

If we now run this in a second instance of Eclipse, in the usual way, and input the code at the start of this page it works as described.

Using Import

Imagine that we are implementing a java-like language that starts with 'package' and 'imports'. Our code needs to know about different namespaces, for instance within expressions we may need to call various methods or use parameters and so on.

So we may need to know about:

Structures within the current file.
Structures using the same language in the same package.
Native Java code in the same package.
Structures using the same language specified by import statement.
Native Java code specified by import statement.
Java classes from library code such as 'String'.

Out of the box support

As discussed on the grammar page, there are 3 magic names that can be used for types: 'name', 'importURI' and 'importedNamespace'

When we use xtext by specifying the grammar and the generator but leaving the remaining files with their default contents then we can only see some of these namespaces. Apart from the grammar file and the generator (myJvmModelInferrer) file everything else is left with the default file contents. (so I am using QualifiedNamesFragment).

Importing native java classes or interfaces does not work properly (the classes are visible but I can't call methods on them).

We can import structures using the same language specified by import statement. We do this by using the magic 'importedNamespace' name in the grammar, like this:

File returns EuclidFile :
  ('package' importedNamespace=QualifiedName ';'?)?
  (imports+=Import)*
  (euclidTypes+=Type)*
;


Import returns EuclidImport :
  'import' (
  importedNamespace=QualifiedName
  | importedNamespace=QualifiedNameWithWildCard) ';'?
;

The ability to import java classes from library code such as 'String' works out of the box in this case.

However this solution does not support item 2 (Structures using the same language in the same package.) or item 5 (Native Java code specified by import statement.)

I can get round item 2 like this:

package myPackage
import myPackage.*

but I would like to avoid the need for the additional import.

I can use methods on the built-in java library such as 'String' but not java code that I have written myself (in the same eclipse project).

Attempt to find Example from 7 Languages

There are some limitations to using the magic 'importedNamespace' name, for instance, limited debugging support and code completion. Since Xtend can do these things it must be possible to do them in my own DSL. However Xtend is a complicated piece of code and its not always easy to find out how it works by looking at its source code. So it might be better to try to find an example in 7 Languages For The JVM that might possibly help us.

Most of them use the magic 'importedNamespace' name like this:	'import' importedNamespace=QualifiedNameWithWildcard;
The only exception is httprouting which uses:	'import' importedType=[ types::JvmType \| QualifiedName];

plus a custom ImportedNamespaceScopeProvider

The code from NumberGuessing.route was:

GET /guess
	do controller.handleGuess(request.getParameter('theGuess'))

This points to an xtend file: GuessTheNumber.xtend which starts like this:

As you can see it has a method 'handleGuess' which takes a string parameter.

package com.acme

import com.google.inject.Inject
import java.io.OutputStreamWriter
import javax.servlet.http.HttpServletResponse

import static extension com.google.common.io.CharStreams.*
import static extension java.lang.Integer.*

class GuessTheNumber {
	
	@Inject extension MagicNumber 
	@Inject HttpServletResponse response
	
	def handleGuess(String theGuess) {
<snip>

If we call this method but give it an 'int' parameter, instead of a 'String' as shown here:

importing xtend

Then we get the expected error message:

Incompatible types. Expected java.lang.String but was int

This shows that the DSL code understands the model of Xtend code. How does it do this?

I think it must be something to do with the 'inject' keyword defined in the grammar here:

Import :
	'import' importedType=[ types::JvmType | QualifiedName];

AbstractDeclaration :
	Dependency | Route;

Dependency :
	'inject' annotations+=XAnnotation? type=JvmTypeReference name=ID;

and the corresponding generator code in RouteJvmModelInferrer:

				// translate the dependencies to fields annotated with @Inejct
				for (field : model.declarations.filter(typeof(Dependency))) {
					members += field.toField(field.name, field.type) [
						annotations += field.toAnnotation(typeof(Inject))
						field.annotations.translateAnnotationsTo(it)
					]
				}

If we replace the reference to GuessTheNumber.xtend to a java file say GuessTheNumber2.java (and provide a suitable java file with the same method signatures) then we get:

adapt to java

This gives the error messages:

Feature handleGuess is not visible

So this method does not seem to work when linking to native java, only when linking to xtend source code.

So this http routing example does not help much?

Handling Concept of a Package

In the example at the start of this page we needed to explicitly import example1.myJavaCode even though it is in the same package. If we want to automatically import from our own package we need to add scoping fragment.

Adding Custom ImportNamespacesScopingFragment

In order to tackle these issues I did the following:

I created a scoping directory. For some reason when I run mwe2 on an xbase project it does not build a scoping directory even though it has the following line: fragment = scoping.ImportNamespacesScopingFragment {}
In that directory I constructed a class called: EditorImportedNamespaceScopeProvider I then added the code below:

/**
 * extends ImportedNamespaceAwareLocalScopeProvider
 * http://git.eclipse.org/c/tmf/org.eclipse.xtext.git/plain/plugins/org.eclipse.xtext/src/org/eclipse/xtext/scoping/impl/ImportedNamespaceAwareLocalScopeProvider.java
 * 
 * This is a local scope provider that understands namespace imports.
 * 
 * It scans model elements for an EAttribute importedNamespace. The value of this attribute is interpreted
 * as qualified name to be imported. Wildcards are supported (see {@link #getWildCard()} for details).
 * 
 * Imports are valid for all elements in the same container and their children.
 * 
 * In the case of xtend then XbaseImportedNamespaceScopeProvider
 * is extended instead.
 * http://git.eclipse.org/c/tmf/org.eclipse.xtext.git/plain/plugins/org.eclipse.xtext.xbase/src/org/eclipse/xtext/xbase/scoping/XbaseImportedNamespaceScopeProvider.java
 */
public class EditorImportedNamespaceScopeProvider extends
		ImportedNamespaceAwareLocalScopeProvider {

	public static final QualifiedName JAVA_LANG = QualifiedName.create("java","lang");
	public static final QualifiedName EUCLID_LIB = QualifiedName.create("com","euclideanspace","euclid","Editor");

	/**
	 * Converts QualifiedNames to strings and back
	 */
	@Inject private IQualifiedNameConverter qualifiedNameConverter;
	
	// automatically import all types from the package we are in
	@Override
	protected List internalGetImportedNamespaceResolvers(
			EObject context, boolean ignoreCase) {
		  if (!(context instanceof EuclidFile)) return Collections.emptyList();
		  EuclidFile file = (EuclidFile) context;
		  List importedNamespaceResolvers = Lists.newArrayList();
		  // add the import statements
		  for (EuclidImport imp : file.getImports()) {
		      String value = imp.getImportedNamespace();
		      ImportNormalizer resolver = createImportedNamespaceResolver(value, ignoreCase);
		      if (resolver != null)
		        importedNamespaceResolvers.add(resolver);
		  }
		  // then add types from own package
		  if (!Strings.isEmpty(file.getImportedNamespace())) {
		    importedNamespaceResolvers.add(
		      // construct ImportNormalizer with wildCard set to true
		      // ImportNormalizer constructor has this form:
		      // ImportNormalizer(QualifiedName importedNamespace, boolean wildCard, boolean ignoreCase)
              // https://github.com/eclipse/xtext/blob/master/plugins/org.eclipse.xtext/src/org/eclipse/xtext/scoping/impl/ImportNormalizer.java
		      new ImportNormalizer(
		        qualifiedNameConverter.toQualifiedName(
                  file.getImportedNamespace()
                ), true, ignoreCase
              )
		    );
		  }
		return importedNamespaceResolvers;
	}

	/**
	 * This allows us to import classes from the java library.
	 * This is supported out-of-the-box when the 'magic' name
	 * 'importedNamespace' is used but has to be put back in
	 * when ImportedNamespaceAwareLocalScopeProvider is
	 * overridden.
	 */
	@Override
	protected List getImplicitImports(boolean ignoreCase) {
	  return newArrayList(new ImportNormalizer(JAVA_LANG, true, false),
	    new ImportNormalizer(EUCLID_LIB, true, false));
	}
}

I edited the already existing file: EditorRuntimeModule which contained an empty class. I copy/pasted this code into it and again changed the name to correspond with my project.

package com.euclideanspace.euclid;
import com.euclideanspace.euclid.scoping.EditorImportedNamespaceScopeProvider;
import com.google.inject.Binder;
import com.google.inject.name.Names;

/**
 * Use this class to register components to be used at runtime / without the Equinox extension registry.
 */
public class EditorRuntimeModule extends com.euclideanspace.euclid.AbstractEditorRuntimeModule {
	@Override
	public void configureIScopeProviderDelegate(Binder binder) {
		binder.bind(org.eclipse.xtext.scoping.IScopeProvider.class).annotatedWith(Names.named(org.eclipse.xtext.scoping.impl.AbstractDeclarativeScopeProvider.NAMED_DELEGATE)).to(EditorImportedNamespaceScopeProvider.class);
	}
}

This still does not support native Java code specified by import statement.

Xtend supports this so I will try to see how it does it:

File returns XtendFile :
  ('package' package=QualifiedName ';'?)?
  (imports+=Import)*
  (xtendTypes+=Type)*
;


Import returns XtendImport :
  'import' (
    (static?='static' extension?='extension'? importedType=[types::JvmType|QualifiedName] '.' '*')
  | importedType=[types::JvmType|QualifiedName]
  | importedNamespace=QualifiedNameWithWildCard) ';'?
;

It uses both the magic 'importedNamespace' name and also 'importedType'

How does xtend distinguish between these? Is it that namespaces in the native language can only be specified with a wildcard but native java classes, interfaces and parameters must always be specified individually without a wildcard?

See this code from 7-languages HTTP Routing Language.

override internalGetImportedNamespaceResolvers(EObject context,
      boolean ignoreCase) {
    val model = context.getContainerOfType(typeof(Model))
    return model.imports.map [
      createImportedNamespaceResolver(importedType.qualifiedName,ignoreCase)
    ].filterNull.toList
}

Conclusion

This page has shown how to call our own native java code from our DSL (based on xbase).

However, there are still a lot of open questions, such as:

How do we do static imports?
How can I do this by using 'import' importedType=[ types::JvmType | QualifiedName]; in grammar.

Also, altough we can get a lot of functionality out-of-the-box by using the magic 'importedNamespace' name, it would be good to understand how this works. Does xbase have code which can build a model instance from java source code?

metadata block

see also:	itemis blog
Correspondence about this page

Eclipse Xtext Importing