Saturday, September 27, 2014

Getting Scalatra to work on Google App Engine - Part 2

In the previous post we made the necessary modifications to Scalatra to make it compatible with Servlets API 2.5 , which allow us to use it on App Engine. One compromise we had to make is to use web.xml to register all the servlets, instead of doing it programmatically.

There is indeed no way to make it possible to register the servlets programmatically in a 2.5 API environment we have no control of, but we can work around it by not really using servlets. This is actually a very old concept in Java web frameworks - register just one servlet in web.xml and have it dispatch the request to application non-servlet handlers. With a little work we can modify Scalatra to work this way, taking advantage of its API, and using an internal mapping mechanism for the top level resources, instead of the Servlet 3.0 registration.

Do note that as opposed to the changes we made in the previous post, the changes demonstrated here result in a different way the requests flow in your application, which may or may not affect your specific cases, especially when you have different frameworks/components in your web tier.

One of the objectives in the solution presented here is to stick to Scalatra's API as much as possible, so that it would be easier to port existing Scalatra code to App Engine, and to drop this solution when App Engine upgrades to Servlet API 3.0+ (if/whenever that happens).

The steps we need to take are:
  • Implement a resource class that will operate similar to ScalatraServlet but will ignore the URI prefix that corresponds to the top-level mapping
  • Implement a mapping structure to back the mounting operation
  • Override Scalatra's enriched ServletContext mount operation to delegate to the above mapping
  • Implement a dispatcher servlet that will use the above mapping to dispatch requests to the appropriate resource
First we implement our "resource" class (think JAX-RS resource or SpringMVC controller). Since it should be almost identical to ScalatraServlet, we inherit from it, and the only change we need to make is to trim the URI prefix from the top-level mapping, because Scalatra's original behavior is to use the part of the URI after the servlet mapping, and we need to use the part after the dispatcher mapping. Therefore we add a member to hold the length of the top-level mapping, and override requestPath() to return the path after that.

package org.scalatra.gae

import org.scalatra.ScalatraServlet
import javax.servlet.http.HttpServletRequest
import org.scalatra.Handler

/**
 * Instances of this class are meant to be mounted via 
 * [[org.scalatra.gae.AppMapper]] , as opposed to 
 * [[javax.servlet.ServletContext]], which cannot be done in API 2.5 .
 */
class ScalatraResource extends ScalatraServlet {

  /**
   * The length of the top-level URI mapping
   */
  protected[gae] var basePathLength: Int = _

  /*
   * Trim the top-level mapping from the URI before using Scalatra's mapping
   */
  override def requestPath(implicit request: HttpServletRequest): String = {
    super.requestPath(request).substring(basePathLength)
  }
}

To store the mappings, we implement an object that lets us add (mount) mapping of URI prefixes to resources, and get a resource that matches a URI prefix. We use a mutable map for storage. When we mount a resource, we set the top-level mapping length so it knows how to trim it.

package org.scalatra.gae

/**
 * This object holds the top-level resource mappings
 */
object AppMapper {

  val mapping = scala.collection.mutable.Map[String, ScalatraResource]()
  
  /*
   * Map a URI prefix to a resource
   */
  def mount(handler: ScalatraResource, path: String) {

    // Consistent handling of "/path", "/path/" and "/path/*"
    val fixedPath = if (path.endsWith("/*"))
      path.stripSuffix("*")
    else if (!path.endsWith("/"))
      path + "/"
    else
      path

    handler.basePathLength = fixedPath.length - 1
    mapping.put(fixedPath, handler)
  }
  
  /*
   * Get the resource that matches the prefix of a URI
   */
  def getHandler(uri: String) = mapping.find(m => uri.startsWith(m._1))
}

We could use AppMapper directly from our code, but to keep it close to the Scalatra way we will extend Scalatra's enriched ServletContext class and update ServletApiImplicits so our ScalatraBootstrap code will be decoupled from this solution.

package org.scalatra.gae

import org.scalatra.servlet.RichServletContext

import javax.servlet.ServletContext

/**
 * Enriched [[javax.servlet.ServletContext]] which delegates the mounting 
 * operation to application-level mapping, instead of 
 * [[javax.servlet.ServletContext#addServlet(String, javax.servlet.Servlet)]]
 * which is not available in API 2.5
 */
class RichAppServletContext(sc: ServletContext) extends RichServletContext(sc) {

  def mount(handler: ScalatraResource, urlPattern: String) = AppMapper.mount(handler, urlPattern)
}


package org.scalatra
package servlet

import javax.servlet.ServletContext
import javax.servlet.http.HttpServletRequest
import javax.servlet.http.HttpServletResponse
import javax.servlet.http.HttpSession
import org.scalatra.gae.RichResponseLocalStatus
import org.scalatra.gae.RichAppServletContext

trait ServletApiImplicits {
  implicit def enrichRequest(request: HttpServletRequest): RichRequest =
    RichRequest(request)

  // Changed to overcome lack of HttpServletResponse.getStatus() in Servlet API 2.5
  implicit def enrichResponse(response: HttpServletResponse): RichResponse =
    new RichResponseLocalStatus(response)

  implicit def enrichSession(session: HttpSession): RichSession =
    RichSession(session)

  // Changed to overcome the lack of ServletContext.mount() in Servlet API 2.5
  implicit def enrichServletContext(servletContext: ServletContext): RichAppServletContext =
    new RichAppServletContext(servletContext)

}

object ServletApiImplicits extends ServletApiImplicits

Finally we implement the dispatcher servlet. It uses the request URI to get the matching resource instance from AppMapper and dispatches the request to it.

package org.scalatra.gae

import scala.collection.JavaConversions._
import javax.servlet.http.HttpServlet
import javax.servlet.http.HttpServletRequest
import javax.servlet.http.HttpServletResponse

/**
 * This is the only servlet that needs to be registered in web.xml .
 * It dispatches the request to the resource that matches the URI prefix
 * as mounted to [[org.scalatra.gae.AppMapper]]
 */
class Dispatcher extends HttpServlet {

  override def service(request: HttpServletRequest, response: HttpServletResponse) {

    val uri = request.getRequestURI.substring(request.getServletPath().length())
    val o = AppMapper.getHandler(uri)
    if (o.isDefined) {
      val (basePath, handler) = o.get
      handler.handle(request, response)
    }

  }

}

Now all we need to do in our application code is:
  • Register and map the Dispatcher servlet in web.xml
  • Inherit from ScalatraResource instead of ScalatraServlet
Other than that, our code would be similar to any Scalatra application on a Servlet API 3.0+ environment, including calls to ServletContext.mount() in ScalatraBootstrap.

A complete code example is available here.

This is not the most generic code, for example - it doesn't handle filters, but it is a working starting point to get Scalatra to work on App Engine with minimum changes. If you have ideas for improvements - I'd love to know.

Wednesday, September 10, 2014

Getting Scalatra to work on Google App Engine - Part 1

Scalatra is a neat lightweight Scala web framework. Scalatra 2.3.0 is based on the Servlet API 3.1 , while App Engine Java runtime only supports Servlet API 2.5 . App Engine imposes additional restrictions as described here, but they are not as significant as lack of support for Servlet 3.0/3.1 .

After digging and experimenting, I succeeded in getting Scalatra to work on App Engine with a few workarounds. Basically we need to refrain from using Servlet 3.0/3.1 specific features in our code, and override Scalatra's parts that do use them with Servlet 2.5 compatible alternatives.

The first obstacle you'll encounter is that you can't mount your servlets and filters programmatically in your ScalatraBootstrap class like this:

import org.scalatra.LifeCycle

import javax.servlet.ServletContext

class ScalatraBootstrap extends LifeCycle {

   override def init(context: ServletContext) {

      // This will fail
      context mount (new MyCustomServlet, "/*")
   }
}

Instead, you leave the init() method empty, and add your servlets and filters to web.xml :

<servlet>
   <servlet-name>my-custom-servlet</servlet-name>
   <servlet-class>MyCustomServlet</servlet-class>
</servlet>
<servlet-mapping>
   <servlet-name>my-custom-servlet</servlet-name>
   <url-pattern>/*</url-pattern>
</servlet-mapping>

Then, at runtime when accessing a path which is mapped by your Scalatra endpoints, you will encounter a NoSuchMethodError because Scalatra attempts to call HttpServletResponse.getStatus(), which was added in Servlet API 3.0 . The problematic code is in RichResponse.status(). RichResponse is an implicit replacement for HttpServletResponse as defined in ServletApiImplicits.enrichResponse(). To overcome the reading of the status from the original response, we'll subclass RichResponse and change the setter and getter of the status to use a local variable:

import org.scalatra.servlet.RichResponse
import javax.servlet.http.HttpServletResponse
import org.scalatra.ResponseStatus

class RichResponseLocalStatus(response:HttpServletResponse) extends RichResponse(response) {

   var statusCode:Int = _

   override def status = ResponseStatus(statusCode)

   override def status_=(statusLine: ResponseStatus) {
      statusCode = statusLine.code
      super.status_=(statusLine)

   }
}

The Scalatra code made it impossible to override the implicit definition for our need, so we have to rewrite ServletApiImplicits to look like this:

package org.scalatra
package servlet

import javax.servlet.ServletContext
import javax.servlet.http.{HttpServletRequest, HttpServletResponse, HttpSession}
import test.RichResponseCustom

trait ServletApiImplicits {

   implicit def enrichRequest(request: HttpServletRequest): RichRequest =
      RichRequest(request)

   implicit def enrichResponse(response: HttpServletResponse): RichResponse =

      // This is changed
      new RichResponseLocalStatus(response)

   implicit def enrichSession(session: HttpSession): RichSession =
   RichSession(session)


   implicit def enrichServletContext(servletContext: ServletContext): RichServletContext =
      RichServletContext(servletContext)
}

object ServletApiImplicits extends ServletApiImplicits

Remember to place the changed ServletApiImplicits with your sources in the "org.scalatra.servlet" package. If you place it in another JAR, remember to make sure it takes precedence over Scalatra's JAR using App Engine's classloader JAR ordering.

That's it. It's not a perfect solution as it won't support Servlet 3.0/3.1 features such as asynchronous processing, and you have to define and map each servlet and filter in web.xml, but these are limitations you have to live with anyway on App Engine and this solution does let you use the nice and clean Scalatra code to write your web tier.

In the next post we will implement an alternative mapping mechanism, which will allow us to reduce the web.xml code and call mount operations in our ScalatraBootstrap code.