Tuesday, June 30, 2009

Fighting LiveFolders

Cupcake introduced "Live Folders" to the android world and since then I've wondered why the phone does not come with a LiveFolder implementation for bookmarks by default. In an effort to try and get code into core-android I set out to write it, but I've been struggling with all sorts of issues.

  1. ContentProvider.query returns a CursorWrapper, not a real cursor, which means if an existing content provider does not provide a live folders implementation and you want to write your own little content provider to proxy the requests with correctly formatted responses you'll have to implement your own cursor to delegate to the cursor wrapper. I resolved this by creating a MatrixCursor in my content provider into which I copy the columns from the BrowserProvider.
        public Cursor query(Uri uri, String[] projection, String selection, String[] selectionArgs,
                String sortOrder) {
            Cursor c = getContext().getContentResolver().query(Browser.BOOKMARKS_URI,
                    Browser.HISTORY_PROJECTION, "bookmark = 1", null, null);
    
            String[] liveFolderColumnNames = {
                    LiveFolders._ID, LiveFolders.NAME, LiveFolders.ICON_PACKAGE,
                    LiveFolders.ICON_RESOURCE,
            };
            MatrixCursor mc = new MatrixCursor(liveFolderColumnNames, c.getCount());
    
            int columnCount = c.getColumnNames().length;
            while (c.moveToNext()) {
                Object[] row = {
                        c.getString(Browser.HISTORY_PROJECTION_ID_INDEX),
                        c.getString(Browser.HISTORY_PROJECTION_TITLE_INDEX),
                        getContext().getPackageName(), R.drawable.ic_launcher_shortcut_browser_bookmark
                };
                mc.addRow(row);
            }
            return mc;
        }
    
  2. I can't figure out how in hell I'm supposed to use ICON_BITMAP to set an icon for each item in the LiveFolder -- Looking at the source code it looks like its expecting a byte[], which I am in fact providing, but then the provider is having difficult constructing a bitmap out of that data and NPE's trying to (and I don't know why it is doing so) thumbnail the icon!

Monday, June 29, 2009

Dumpcatcher: Uncaught Exceptions and Client IDs

Alright, about halfway through my flight and I now have an Uncaught exception handler that will post messages to dumpcatcher.

It doesn't work with android at the momement... There is a thread I know I have to read on android-developers. I think I starred it because I knew I'd find it uuseful.

The first user of this class was going to be Foursquared, but it turns out the code I wrote doesn't work.

I quickly added the ability to create a client id attached to each crash in order to allow me track each individual client and in the process refactored a bit of the code here and there. I'll have to remeber to update the design doc as I have made some changes to the design after noticing flaws in my original work.

Sunday, June 28, 2009

Dumpcatcher: Java Client

I'm on my way back from Cambodia... More on that later...

After spending about two-three hours working on hmac signing with Java I now have a working Dumpcatcher client for Java, this is similar to the python client

clients/python/logging/handler.py:Dumpcatcher
in that its usage is:
Dumpcatcher dc = new Dumpcatcher(PRODUCT_KEY, SECRET, "http://localhost:8080/add", 2);
HttpResponse response = dc.sendCrash(
    new NameValuePair("short", "Some short dump"),
    new NameValuePair("long", "some long dump")
)
    assertNotNull(response);
    assertEquals(200, response.getStatusLine().getStatusCode());

Have a look: http://code.google.com/p/dumpcatcher/source/browse/clients/java/src/com/googlecode/dumpcatcher/logging/Dumpcatcher.java?r=20090629r0

Now onto making a logging client for this guy.

Monday, June 22, 2009

Dumpcatcher: Python Client

For the python dumpcatcher client I decided to go with the standard python logging infrastructure. While there was definitely something lacking in the python documentation for logging.py I found all the information I needed with a combination of the docs and the source code itself. Creating a python client for the dumpcatcher is now pretty simple:
import handler  # from dumpcatcher.clients.python.logging

logger = logging.getLogger('my_tag')
logger.setLevel(logging.DEBUG)  
logger.addHandler(
      DumpcatcherHandler('agtkdW1wY2F0Y2hlcnINCxIHUHJvZHVjdBgCDA',
                         '9b6c65910428419db0d0b730278b72e3',
                         'http://localhost:8080/add',
                         5))


try:
  'a' + 1  # TypeError!
except Exception, e:
  logger.exception('Oh man! Look at this!')
The output of which looks something like:
2009-06-21 06:36:57 demo 40
handler.py:110:broken Oh man! Look at this!
handler.py:110:broken Oh man! Look at this!
Traceback (most recent call last):
File "/home/jlapenna/code/dumpcatcher/clients/python/logging/handler.py", line 108, in broken
  'a' + 1
TypeError: cannot concatenate 'str' and 'int' objects
File "/home/jlapenna/code/dumpcatcher/clients/python/logging/handler.py", line 110, in broken
  logger.exception('Oh man! Look at this!')
Looking at this now, it seems that all I have left is the java client and data aggregation before I'm feature complete!

Dumpcatcher: Python HTTP and Dates libraries

Sheesh,

Python really shows its age when you have to deal with dates, http and urls. I was able to implement my signing and verification bit of code just now and it was quite a pain. I have a sample client in clients/python/example.py and the endpoint/crash.Add handler should show you the inverse side of the handshake.

The thing that was definitely most challenging was writing the example client. Because the python libraries for http, etc evolved over time there are about seventy-five ways to do an http request instead of python's normal "one."

Same can be said for datetime modules. Long ago when we had to use mx time there was a DateTime object, now its built into python so everything should be peachy? Yeah, date handling in python was improved by this change but timezones are still outrageous! Considering I'm now about 10 hours off UTC, I have had to think about how to submit a request in string form. For now though I'm going to punt and require that times submitted are in the UTC time zone.

Now that I've got that baked in, I can acutally write the code that logs the requests to the data store. After that, I need to write a bit of code to do sorting and aggregating of the crash dump data.

Dumpcatcher: HTML

Wow, its been quite a while since I've written any HTML. Probably about one year. Its also been that long since I last used AppEngine and as a result I'm really, really rusty.

I'm now about 3 hours into the hack-a-thon (on a plane instead of of a coffee shop) and I have about 10 hours of battery life remaining.

Things I've learned so far:

I totally forgot the basic IO operations for the app engine data store and with this kind of knowledge its actually pretty hard to re-learn -- I remember such random bits and pieces I forget which parts are bad memory or actuality.

The design doc was fun to write, I had to think about a few different problems that I figured I would encounter. I even realized that my initial idea for request signing had some security vunerabilities which I had to think about how to resolve.

I have one third of the whole project done, but its the smallest one third. Users are now able to register their account and create an unlimited number of products.

Next, I need to write the server side component of crash logging. Eg, when a user submits a crash the request must be authorizedand then it needs to get into the datastore.

Dumpcatcher: Design Doc

As I do with any project at work, I want to put together a short doc describing the scope and scale at which I will write this app.

Summary

Dumpcatcher is a simple web service that takes authorized requests from remote clients and logs key-value pairs in its datastore for future analysis. These pairs are typically an aribitrary identifier and an exception/stack-trace.

Features

  • Crash Stack Message storage
  • Clients must be able to submit stack traces (well, arbitrary strings) along with various bits of meta-data. Version, app name, etc.
  • Client libraries for Python, Java
  • I am targeting my <a href="http://joelapenna.com/git/foursquared.git">Foursquared Android Client

    Unknown end tag for </a>

    as well as any other pet projects I may use in the future.
  • Data Aggregation
  • I plan on allowing data aggregation by exception type, custom label and line.
  • Authenticated Client Requests
  • All requests by clients must be sent by authorized clients to prevent the service from becoming a black hole for spam. Design:

Design

App Engine has a very simple data store and webapp framework that I intend to utiltize for the basic functionality of the app.

Users

Users represent a single Google Id and a particular developer using the system.

Products

A product is an application that uses the dumpcatcher to log crashes. A user may have multiple products.

Each product registered will have two values associated with it, a productKey which will be passed as a paramter in all HTTP requests to the server and a secret which will be used to HMAC sign a request.

Product secrets will be randomly generated UUIDs.

HTTP Request

All requests to the dumpcatcher service will be secured with an HMAC hash. The hash will be keyed by a unique identifier provided to the client

productKey

Each client -> server request will include a productKey, an identifier used to differentiate between different products using the service.

HMAC

All requests must be submitted with an HMAC-SHA1 hex digest of the request query paramters as well as an increasing "request" identifer. The message consists of a standard http "query", sorted by keyname and quoted, request

For: http://localhost/add?product_keyd=1234&some=pair&other=pair we would construct the digest like so:

TODO(jlapenna): Probably don't want to split on & if the contents of the request might contain one though, they should already be encoded. Something like that...

sorted_query = ''.join(sorted(request.query_string.split('&'))) hash = hmac.new('SOME KEY', sorted_query, hashlib.SHA1)

And, as such, the actual request made to the server will be:

'http://localhost/add?product_key=12345&some=pair&other=pair&hmac=%(hash)s'

On the backend the server will take the reverse steps and using the secret associated with the provided productKey, will verify the authenticity of the request by encoding the query paramters the same way it is done on the client, keying the result by secret associated with the provided productKey.

Datastore

Initially there will be three models, one corresponding to "crashes," another to "users" and the third to "products."

Each user will be associated with a specific Google ID but a single Google ID can have many products.

Security

Security and validity of client-> server requests will be handled via the usage of HTTPS for securing communications and for HMAC to verify authenticity of a client request.

Replay Attack

An attacker with access to the HTTP stream a client -> server request is sent over will be able to execute a replay attack by capturing the HTTP post made by the client and submitting it as its own, at any rate he so desires.

The solution as such is to only allow requests over HTTPS. This gains the added advantage of preventing any private data from leaking via a network observer packet sniffing.

Caveats

It is likely and highly reasonable that an app like this exists in a highly more polished and featureful way. I chose this project because I felt like it would be a good way to explore some new technologies and have a fun time; not because this is in any way "new" or "exciting"

Friday, June 19, 2009

Dumpcatcher

I took the day off in order to spend it writing some code. My goal for today is to finish with a fully-functional exception catcher for my projects so that I don't have to require users to send me tracebacks or exceptions when they occur.

I started by registering http://dumpcatcher.appspot.com. I will be pushing frequently to this site as I add features. I chose App Engine because I like that its hassle-free application deployment. Launch the app, run it, walk away and it should just work!

Second, I registered http://dumpcatcher.googlecode.com. I will be pushing the code here pretty much just as frequently as I push the site. I chose Mercurial as the SCM because I am sick and tired of git.

So... Let me begin...

Addendum: I actually ended up going into work instead of taking the day off and coded this on a flight to Cambodia.

Monday, June 15, 2009

Foursquared!

In early April I started working on an Android client for Foursquare I started out with an idea that I could screenscrape enough of the mobile app to make a worthy native api but I was quick to discover other nefarious methods to get at the API that Foursquare was using on the iPhone client. Armed with wireshark and a couple of python scripts, I was able to auto-generate a Java API based on data samples pulled from the TCP dump.
Over the next several weeks I started designing a UI for the app. First and foremost I developed a venue search + checkin activity. I then implmentented another screen to scratch an itch, the CheckinsActivty. This is neat because it displays in a list and in a map which each of your foursquare friends may be.
I had to take a break from the UI work when I started talking with the foursquare people intesting their public API. After several days of fighting with Apache's HttpClient and oauth-signpost (Signpost, by the way, is awesome -- The problems were all in my implementation).
Now its mid-June and I've gotten back to working on UI features; I wrote a "UserActivity" yesterday that displays information about a user. At the moment this information is limited to their name, city and badges acquired. I've also started recovering (fixing bugs) from a lot of refactoring I've been doing in the UI to simplify the design.
I have to start work on figuring out what I want the app to do... In the short term and the medium term. The end state is pretty clear -- feature parity with the iPhone client and a pretty UI to boot. After that there are cool pie in the sky ideas but I'll probably be bored of working on the client by the time I get there.

Join the Foursquared Mailing List!