Dumpcatcher: Design Doc
As I do with any project at work, I want to put together a short doc describing the scope and scale at which I will write this app.
Summary
Dumpcatcher is a simple web service that takes authorized requests from remote clients and logs key-value pairs in its datastore for future analysis. These pairs are typically an aribitrary identifier and an exception/stack-trace.
Features
- Crash Stack Message storage
- Client libraries for Python, Java
- Data Aggregation
- Authenticated Client Requests
Clients must be able to submit stack traces (well, arbitrary strings) along with various bits of meta-data. Version, app name, etc.
I am targeting my <a href="http://joelapenna.com/git/foursquared.git">Foursquared Android ClientUnknown end tag for </a>
as well as any other pet projects I may use in the future.
I plan on allowing data aggregation by exception type, custom label and line.
All requests by clients must be sent by authorized clients to prevent the service from becoming a black hole for spam. Design:
Design
App Engine has a very simple data store and webapp framework that I intend to utiltize for the basic functionality of the app.
Users
Users represent a single Google Id and a particular developer using the system.
Products
A product is an application that uses the dumpcatcher to log crashes. A user may have multiple products.
Each product registered will have two values associated with it, a productKey which will be passed as a paramter in all HTTP requests to the server and a secret which will be used to HMAC sign a request.
Product secrets will be randomly generated UUIDs.
HTTP Request
All requests to the dumpcatcher service will be secured with an HMAC hash. The hash will be keyed by a unique identifier provided to the client
productKey
Each client -> server request will include a productKey, an identifier used to differentiate between different products using the service.
HMAC
All requests must be submitted with an HMAC-SHA1 hex digest of the request query paramters as well as an increasing "request" identifer. The message consists of a standard http "query", sorted by keyname and quoted, request
For: http://localhost/add?product_keyd=1234&some=pair&other=pair we would construct the digest like so:
TODO(jlapenna): Probably don't want to split on & if the contents of the request might contain one though, they should already be encoded. Something like that...
sorted_query = ''.join(sorted(request.query_string.split('&'))) hash = hmac.new('SOME KEY', sorted_query, hashlib.SHA1)
And, as such, the actual request made to the server will be:
'http://localhost/add?product_key=12345&some=pair&other=pair&hmac=%(hash)s'
On the backend the server will take the reverse steps and using the secret associated with the provided productKey, will verify the authenticity of the request by encoding the query paramters the same way it is done on the client, keying the result by secret associated with the provided productKey.
Datastore
Initially there will be three models, one corresponding to "crashes," another to "users" and the third to "products."
Each user will be associated with a specific Google ID but a single Google ID can have many products.
Security
Security and validity of client-> server requests will be handled via the usage of HTTPS for securing communications and for HMAC to verify authenticity of a client request.
Replay Attack
An attacker with access to the HTTP stream a client -> server request is sent over will be able to execute a replay attack by capturing the HTTP post made by the client and submitting it as its own, at any rate he so desires.
The solution as such is to only allow requests over HTTPS. This gains the added advantage of preventing any private data from leaking via a network observer packet sniffing.
Caveats
It is likely and highly reasonable that an app like this exists in a highly more polished and featureful way. I chose this project because I felt like it would be a good way to explore some new technologies and have a fun time; not because this is in any way "new" or "exciting"
