All posts by Brian Hoffmann

Issues to POST streamed data to a webservice

Recently I had quite some issues with sending data via POST to a webservice from an Android app.

Let me quickly set the stage: I needed to upload data to a webservice via HTTPS. This data were dynamically generated PDF files created in the app. Since the files can become potentially large, I wanted to make sure that data would be streamed byte-wise to the webservice. I already had Volley and were using the HurlStack (which uses HttpUrlConnection).

In order to up-stream data would one needs to explicitly call either setFixedLengthStreamingMode() or setChunkedStreamingMode() on the UrlConnection. Without that the data that will go up would actually be stored in memory first. As I didn't want to take the risk that 3MB (or more) big files could blow my app with an OutOfMemoryException (especially on low-end devices, which are always short on memory!), I needed to take the safe road.

Setting up the workflow in the applications architecture was done quickly. Add some callbacks for preparing the data, counting data length and finally streaming data with an OutputStream. Worked perfectly! The problems kicked in, as I started to actually send data. I tried with fixed-length and chunked streaming mode. For fixed-length streaming mode one needs to the length of the full request body that is about to be sent. Contrary to the chunked streaming, were the connection would buffer up a certain amount of data and send it in (you guessed it!) chunks.

Unfortunatelly, chunked streaming mode wasn't supported by the webservice, so I had no alternatives left but use the fixed-length mode. But this resulted in exceptions complaining that the connection "expected 0 bytes but got 1 byte". Though, I was pretty sure that the content length used wasn't 0. Also going plain HTTP without SSL wouldn't be a real solution, I hoped maybe it's just the SSL connection causing problems. But it wasn't.

After lots of fiddling around with parameters, stepping through code and reading hidden platform classes, I came to the conclusion: something must be wrong with the HTTP(S) stack implementation in Android.

What came first to mind was OkHttp: an HTTP client with up-to-date TLS, SNI and SPDY support. No hesitation: get the library and its dependency Okio and drop everything into the project. To fully integrate OkHttp I needed to tell Volley to not use the platform's implementations. All that's needed is a subclass of HurlStack:

Without changing anything else of my implementations, the fixed-length upload streaming suddenly worked (which I take as a proof that indeed something is wrong with the platform implementations).

Setup Dagger with Eclipse

The dependency injection library Dagger by Square setup for using it with Eclipse is a bit different when not using Maven.

At the time of writing, there seem to be no complete explaination on what is needed to setup everything. I got everything working with combined information from the comments in issue 126 by staxgr and arichiardi.

  • Prepare the Eclipse project and create the directories libs and libs-compile.
  • Download the following libraries:
      • dagger-x.x.x.jar goes into the libs directory
      • dagger-compiler-x.x.x.jar goes in the libs-compile directory
    • the javawriter jar and put it into libs-compile
    • javax.inject goes into libs
  • Enable Annotation Processing and add all four libraries to the factory path of the annotation processing settings
    • Project Properties → Java Compiler → Annotation Processing
      • check "Enable project specific settings" and "Enable annotation processing"
    •  Project Properties → Java Compiler → Annotation Processing → Factory Path
      • "Add JARs..." for each downloaded jar

When starting a build, the generated files should appear in .apt_generated. This directory should be automatically configured as a source folder after annotation processing has Bern enabled.

Becoming Agil: Magic Estimation

Usually we have certain constrains on our project which define what we can achieve and most likely also what we have to do first. Having a backlog packed with user stories makes it very difficult to get an overview of the upcoming workload. To be able to actually get an idea for the upcoming sprints we need to estimate the product backlog. Doing that detailed for each user story would take us quite some time. So instead of estimating each single user story (as exact as possible) we could try to estimate a bunch of similar user stories at once. This is Magic Estimation.

Let's assume we have a backlog packed with 80 user stories. Each of them has a decent description and also has acceptance criteria defined.

As a first step we need to set up complexity buckets. In those buckets we'll put stories with similar complexity. For simplicity we choose shirt sizes: XS, S, M, L and XL. The XS bucket is for stories which can be easily achieved in a couple of hours, whereas the XL bucket is for stories which we don't really have a clue on how to solve. All other stories would be evenly distributed among the other buckets.

The buckets get put on a board in form of cards. The user stories will be put next to the bucket cards.

The product owner starts by briefly presenting each user story. The team then has just a couple of seconds to decide on the complexity of the story. When time is up (the ScrumMaster could help here by watching the time) the story goes into its complexity bucket. This process shouldn't really take much longer than 30 seconds, certainly not longer than a minute. If the team cannot make a reasonable decision, just put the story in one of the higher ranked buckets -- we are not in a sprint planning, we just want to get an idea of the workload. Don't think about it too long, just shoot right away.

After all stories are distributed the team gets time to have a look over all buckets (in the meantime the product owner can go to get a coffee refill). Are there some stories which doesn't quite fit to the others? The team can take maybe ten minutes to reorder all stories. If there are stories which could make up there own bucket, create a new one. There might be real no-brainers which could all go into an XXS bucket. But don't try to create too many buckets, this wouldn't be much helpful.

Now all user stories are clustered into their complexity. In the next step we can put story points on the buckets: is the S bucket three or five points; the XL bucket might be 20 points. When all buckets have story points assigned, we can add it all up.

Board with user stories arranged into complexity categories
Board with user stories arranged into complexity categories

The sum of story points gives us a rough direction of how many sprints it would take to finish all user stories. At the end the product owner can use the estimated story points to prioritize user stories for the upcoming sprints. Furthermore, he can compare the estimation outcome with the product's time table if it is actually possible to do or maybe if some feature need to be cut down.

This approach is just one possible way of doing a magic estimation. There are other ways the team could do this:

Becoming Agile: Principles

Continuously integrate new code

As soon as new code is finished it should be integrated into the code base. By that it is not only ensured that all members of the development team get access to new code, it also minimises the risk of running into incompatibilities and reduces the complexity of merging changes.

In a recurring fashion this could happen at the end of development day. Also partly finished features can already be pushed into the base as long as they meet the project's quality minimums (etc. testable/tested, documented). At the beginning of the next development day changes would get individually pulled into the working copy.

Maximise coverage of automated tests

Testing code is a crucial part in reaching a certain level of quality and stability. To minimise the workload needed the process should be automated where possible. This can be done with unit tests using state of the art testing frameworks (JUnit, Selenium, etc) to further reduce workload.

It should be also possible to combine testing with continuous integration. This gives the advantage of running tests as often as possible and getting the chance to react to issues quickly.

Think ahead and stay flexible

Each team member should keep the future constantly in mind. As the application grows requirements might (or most likely "will") change. To be able to react and implement these changes it is important to always develop for scalability and flexibility. This accounts not only for the development members but also for the management members of team. The management should always evaluate new features with what might come next.

Keep it simple

Always create the simplest solution for a problem that will work and deliver for the minimum requirement. Maintaining a simple, clean and open design ensures that the application can be easily extended while staying focused and meeting dead lines.

StringEntity Doesn't Play by the Rules of Android

So here is the situation: you build up some JSON objects and pop them into a StringEntity so your HTTPClient can send the stuff to server. We know on Android, the default charset is UTF-8 and use that knowledge through out the whole application, not considering that this fact might change in some places. But it does! Using a default StringEngtity with data supposed to be encoded in UTF-8 won't give us the expected results on Android. A look into the code explains why:

// ...
if (charset == null) {
// ...

And guess what, HTTP.DEFAULT_CONTENT_CHARSET does not take advantage of Charset.defaultCharset() but instead is set up with "ISO-8859-1".

To get real UTF-8 data we need to do something like new StringEntity("âáàéèê", HTTP.UTF_8);.