Saturday, September 26, 2009

Proxying large Amounts of Data using UrlRewriteFilter

Finally, I found some time to submit a feature request with patch to the UrlRewriteFilter project.

http://code.google.com/p/urlrewritefilter/issues/detail?id=53

In one of my projects we needed to provide functionality to post (upload) and download data via a servlet container to/from another url/port - basically we needed to implement proxying.

A great library out there is UrlRewriteFilter, a Java library that provides Apache's mod_rewrite functionality. Not only can you use it to make complex urls more user-fiendly, or re-map old url to new ones but it also provides proxying capabilities.

UrlRewriteFilter uses Apache HttpClient for doing proxying. Unfortunately, I ran into memory issues when proxying large amounts of data.  The issue is that the current version of UrlRewriteFilter (3.2) is doing buffered requests while proxying. This probably works fine for 90% of all use-cases but for the project I am working on we need to basically support unlimited amounts of data to be proxied (multiple 100s of MB).

Thus I provided a patch, that worked really well in my project without increasing memory consumption.
In Apache HttpClient you can implement a custom class using the RequestEntity interface that allows you to stream the data directly.

No comments: