Sunday, November 8, 2009

Android Location APIs and why they anger me

The problem lies in the fact that as a end-developer I have to care about the various location inputs on a phone. I have to think about GPS vs Wifi vs Cell Tower LocationProviders in order to really take advantage of the phone's location capabilities. What basically happened was that my algorithm for picking a users location based on incoming location updates caused more accurate but out of date location updates to be preferred over slightly less accurate but much more recent updates.

This is the code that defines my "best location" strategy.  http://code.google.com/p/foursquared/source/browse/main/src/com/joelapenna/foursquared/location/BestLocationListener.java

I basically have to deal with the fact that I might get location-from-gps after I get location-from-wifi-tower OR i might  get location-from-wifi-tower after I get location-from-gps so that the Last Known location may in some cases return a more or less accurate location than what the phone is really capable of reporting just because I wanted to be able to get a quick-kind-of-accurate-lock while I was trying to accquire a accurate-as-possible sort of lock.

How would I make this better?

I could see an API where instead of registering with individual location providers, you would query the system for a set of location updates. You would define that you want FINE location, but will accept COARSE updates first. You would have a getBestKnownLocation instead of a getLastKnownLocation that would return a location based on heuristics you might define when registering your location provider.  Visually, the updates you'd receive would look something like the growing/shrinking circles you get in the gmaps app when honing in on a location. Maybe the developer would request a "HyperLocalWhileWalkingAround" strategy that could be swapped with "PassiveUpdatesWhenLocationIsSignificentlyChanged." both of which could have knobs for controlling specifics of the strategy.

In short, I want to abstract away the code I have shown as an example above. As a end-developer, I don't care where the location comes from, just that I get a location but I do care, what the data looks like.


Labels: ,

Tuesday, September 8, 2009

Dutch! Foursquare! Translation! Exclaimations! Are! Overdone!

I just submitted a change to the foursquare-for-android repository that enables a Dutch translation of our foursquare-for-android app! Neat! Thanks Marco!

Labels: ,

Monday, August 10, 2009

Foursquared Repo

Get the code or file bugs at: http://foursquared.googlecode.com

Labels: ,

Monday, July 20, 2009

Foursquare Screeny

Labels: ,

Thursday, July 2, 2009

Bookmarks Live Folder



Want a "live folder" for bookmarks on your android home screen?
http://market.android.com/search?q=pname:com.googlecode.livebookmarks

Labels:

New icon for Foursquared


Chris made me an awesome icon for the app. In return he got an alpha release! How lucky is that! Now for the low low cost of a line of code or cool menu icons, you too can have some buggy android software!

Labels: ,

Wednesday, July 1, 2009

Android, Foursquare and Push Notifications

How to get push notifications on Android.

As we all are aware its possible to run apps in the background in Android. The way push notifications work for the google apps like gmail and gtalk are by way of TCP connections with long TTLs. This allows the phone to wake up intermittently as the gsm radio will do (on the measure of microseconds) to see if new data has arrived on the wire. This implementation though, requires its own backend and for each service doing it, we'd be draining the battery even further...

Other work on Android Push:

XMPP/Jabber is even worse. Not only is xmpp/jabber a very chatty protocol, but as it works with one long xml stream, the disconnected nature of a cell phone does not jive well with it... holding an open xmpp connection is probably a very bad idea.

So where does this leave us with push notifications on Android if anything push we do is going to eat the battery and cause the user to uninstall the app?

One option would be to send periodic position updates based on some limiter (the API only has a radius accuracy of no less than a mile. We wouldn't have to wake up all that frequently for the common case of people don't drive around during night time activities). Knowing this, most of the work can be done on the backend to calculate "interesting things," and for notifications, we can send the response over gtalk. Specifically gtalk because most Android phones already have that previously mentioned long-lived tcp session that we can piggy back on. The message could be something along the lines of:

<a href="foursquare://com.playfoursquare.api.NotificationsAuthority/notification/12345?click=user">Mike M.</a> is hanging out at <a href="foursquare://com.playfoursquare.api.NotificationsAuthority/notification/notification/12345?click=venue">Mike's Bar and Grill</a>.

Users clicking on the link would be directed to the android foursquare app, where we could display awesome information.

I'd have to try this out to see if it actually works and how an android app would register as a global URI handler but it should be do-able.

Labels: ,

Foursquared Help

I'm looking for help on foursquared. I can start fleshing out some starter bugs and features so that anyone interested can just jump right in and code.

I've been hating git since I started using it and may end up migrating the SCM to HG on Google Code... I just feel more comfortable with both tools... Still not sure. If I have any external contributors it will really force my hand as I don't want to have to deal with emailed diffs and patches.

I'm not asking for testers because at the pace I'm going, the problems and features I'm trying to implement take up my allocated time plus a bit more. Increasing the water hose of bug reports and feature requests would just drown me. On the other hand, I did put together some crash reporting infrastructure while I was on the airplane so that when I do release, I can get automated bug reports: http://joelapenna.com/blog/labels/dumpcatcher (the code is at http://dumpcatcher.googlecode.com)

Right now I'm working on some troubling network connectivity issues (after some period of usage, all HTTP requests start hanging). After I resolve these issues I will be readed for limited testing... I'm content to release non-feature-complete if only to get some people looking at the app who might be developers.

I'm thinking that feature-wise the release will include the five screens I've sent before: http://joelapenna.com/foursquared/

Labels: ,

Tuesday, June 30, 2009

Fighting LiveFolders

Cupcake introduced "Live Folders" to the android world and since then I've wondered why the phone does not come with a LiveFolder implementation for bookmarks by default. In an effort to try and get code into core-android I set out to write it, but I've been struggling with all sorts of issues.

  1. ContentProvider.query returns a CursorWrapper, not a real cursor, which means if an existing content provider does not provide a live folders implementation and you want to write your own little content provider to proxy the requests with correctly formatted responses you'll have to implement your own cursor to delegate to the cursor wrapper. I resolved this by creating a MatrixCursor in my content provider into which I copy the columns from the BrowserProvider.
        public Cursor query(Uri uri, String[] projection, String selection, String[] selectionArgs,
                String sortOrder) {
            Cursor c = getContext().getContentResolver().query(Browser.BOOKMARKS_URI,
                    Browser.HISTORY_PROJECTION, "bookmark = 1", null, null);
    
            String[] liveFolderColumnNames = {
                    LiveFolders._ID, LiveFolders.NAME, LiveFolders.ICON_PACKAGE,
                    LiveFolders.ICON_RESOURCE,
            };
            MatrixCursor mc = new MatrixCursor(liveFolderColumnNames, c.getCount());
    
            int columnCount = c.getColumnNames().length;
            while (c.moveToNext()) {
                Object[] row = {
                        c.getString(Browser.HISTORY_PROJECTION_ID_INDEX),
                        c.getString(Browser.HISTORY_PROJECTION_TITLE_INDEX),
                        getContext().getPackageName(), R.drawable.ic_launcher_shortcut_browser_bookmark
                };
                mc.addRow(row);
            }
            return mc;
        }
    
  2. I can't figure out how in hell I'm supposed to use ICON_BITMAP to set an icon for each item in the LiveFolder -- Looking at the source code it looks like its expecting a byte[], which I am in fact providing, but then the provider is having difficult constructing a bitmap out of that data and NPE's trying to (and I don't know why it is doing so) thumbnail the icon!

Labels:

Monday, June 29, 2009

Dumpcatcher: Uncaught Exceptions and Client IDs

Alright, about halfway through my flight and I now have an Uncaught exception handler that will post messages to dumpcatcher.

It doesn't work with android at the momement... There is a thread I know I have to read on android-developers. I think I starred it because I knew I'd find it uuseful.

The first user of this class was going to be Foursquared, but it turns out the code I wrote doesn't work.

I quickly added the ability to create a client id attached to each crash in order to allow me track each individual client and in the process refactored a bit of the code here and there. I'll have to remeber to update the design doc as I have made some changes to the design after noticing flaws in my original work.

Labels: , ,

Monday, June 22, 2009

Dumpcatcher: Design Doc

As I do with any project at work, I want to put together a short doc describing the scope and scale at which I will write this app.

Summary

Dumpcatcher is a simple web service that takes authorized requests from remote clients and logs key-value pairs in its datastore for future analysis. These pairs are typically an aribitrary identifier and an exception/stack-trace.

Features

  • Crash Stack Message storage
  • Clients must be able to submit stack traces (well, arbitrary strings) along with various bits of meta-data. Version, app name, etc.
  • Client libraries for Python, Java
  • I am targeting my <a href="http://joelapenna.com/git/foursquared.git">Foursquared Android Client

    Unknown end tag for </a>

    as well as any other pet projects I may use in the future.
  • Data Aggregation
  • I plan on allowing data aggregation by exception type, custom label and line.
  • Authenticated Client Requests
  • All requests by clients must be sent by authorized clients to prevent the service from becoming a black hole for spam. Design:

Design

App Engine has a very simple data store and webapp framework that I intend to utiltize for the basic functionality of the app.

Users

Users represent a single Google Id and a particular developer using the system.

Products

A product is an application that uses the dumpcatcher to log crashes. A user may have multiple products.

Each product registered will have two values associated with it, a productKey which will be passed as a paramter in all HTTP requests to the server and a secret which will be used to HMAC sign a request.

Product secrets will be randomly generated UUIDs.

HTTP Request

All requests to the dumpcatcher service will be secured with an HMAC hash. The hash will be keyed by a unique identifier provided to the client

productKey

Each client -> server request will include a productKey, an identifier used to differentiate between different products using the service.

HMAC

All requests must be submitted with an HMAC-SHA1 hex digest of the request query paramters as well as an increasing "request" identifer. The message consists of a standard http "query", sorted by keyname and quoted, request

For: http://localhost/add?product_keyd=1234&some=pair&other=pair we would construct the digest like so:

TODO(jlapenna): Probably don't want to split on & if the contents of the request might contain one though, they should already be encoded. Something like that...

sorted_query = ''.join(sorted(request.query_string.split('&'))) hash = hmac.new('SOME KEY', sorted_query, hashlib.SHA1)

And, as such, the actual request made to the server will be:

'http://localhost/add?product_key=12345&some=pair&other=pair&hmac=%(hash)s'

On the backend the server will take the reverse steps and using the secret associated with the provided productKey, will verify the authenticity of the request by encoding the query paramters the same way it is done on the client, keying the result by secret associated with the provided productKey.

Datastore

Initially there will be three models, one corresponding to "crashes," another to "users" and the third to "products."

Each user will be associated with a specific Google ID but a single Google ID can have many products.

Security

Security and validity of client-> server requests will be handled via the usage of HTTPS for securing communications and for HMAC to verify authenticity of a client request.

Replay Attack

An attacker with access to the HTTP stream a client -> server request is sent over will be able to execute a replay attack by capturing the HTTP post made by the client and submitting it as its own, at any rate he so desires.

The solution as such is to only allow requests over HTTPS. This gains the added advantage of preventing any private data from leaking via a network observer packet sniffing.

Caveats

It is likely and highly reasonable that an app like this exists in a highly more polished and featureful way. I chose this project because I felt like it would be a good way to explore some new technologies and have a fun time; not because this is in any way "new" or "exciting"

Labels: , , , ,

Monday, June 15, 2009

Foursquared!

In early April I started working on an Android client for Foursquare I started out with an idea that I could screenscrape enough of the mobile app to make a worthy native api but I was quick to discover other nefarious methods to get at the API that Foursquare was using on the iPhone client. Armed with wireshark and a couple of python scripts, I was able to auto-generate a Java API based on data samples pulled from the TCP dump.
Over the next several weeks I started designing a UI for the app. First and foremost I developed a venue search + checkin activity. I then implmentented another screen to scratch an itch, the CheckinsActivty. This is neat because it displays in a list and in a map which each of your foursquare friends may be.
I had to take a break from the UI work when I started talking with the foursquare people intesting their public API. After several days of fighting with Apache's HttpClient and oauth-signpost (Signpost, by the way, is awesome -- The problems were all in my implementation).
Now its mid-June and I've gotten back to working on UI features; I wrote a "UserActivity" yesterday that displays information about a user. At the moment this information is limited to their name, city and badges acquired. I've also started recovering (fixing bugs) from a lot of refactoring I've been doing in the UI to simplify the design.
I have to start work on figuring out what I want the app to do... In the short term and the medium term. The end state is pretty clear -- feature parity with the iPhone client and a pretty UI to boot. After that there are cool pie in the sky ideas but I'll probably be bored of working on the client by the time I get there.

Join the Foursquared Mailing List!

Labels: ,

Thursday, April 23, 2009

Coming Soon!

Labels:

Monday, December 1, 2008

Security Learninatin.

Saturday I sent some time at Noisebridge talking about Android and its security model. In addition to learning a bunch about Android specific issues, I learned a lot about security in general. I think Saturday was the first time that I patched code I wrote with a specific security issue in mind. Never mind the fact that the "flaw" was really just me being dumb and at worst would result in an attacker DOS'ing my app by violating Facebook API rpc limits.

We discussed a couple of other flaws, one of which has an immediate fix though it may take a bit to implement. The other being one that I'm not willing to discuss yet. :D One thing I learned is that talking about security vulnerabilities is something that should be done with careful consideration of its impact: I'm not willing to expose an exploit in my app yet.

Labels:

Monday, October 27, 2008

Android Market is open!

Let the awesomeness begin!

Labels:

Tuesday, October 21, 2008

Statusinator

I've updated the first app I wrote for Android, Statusinator. It is a client that you can use to update your status and upload photos to Facebook. It will be up on the Market as soon as it is launched, but until then you can get it from the Project Homepage.

Labels: ,

Saturday, October 18, 2008

Losing Some Steam

Oh no! I just realized the other day that some of the core code in the Android side of my Top Secret Project is going to have to be re-written. I am very sad about this and have started to lose a little steam. It doesn't help that I've spent the past two weeks letting the feeling linger while I hung out with friends in Toronto. I have one week before I'm back in Chicago for a bit and while I don't think I'm going to be able to make my app launch-ready by Wedensday (The G1 Launch Date) I feel like the app *is* coming along to some capacity. I sometimes dream of having a team of Engineers working with me on this project, all as dedicated as I am. If that were true, the past two weeks, each and every one of them would have written a total of 0 code for the app. I guess its a good thing right now that I'm not running a startup or anything like that. Who ever heard of a startup stopped because some crazy country up North decided to abscond the dude in charge?

Labels: ,

Thursday, October 2, 2008

Catchup with two hands

its been six days and its time again to leave SF for a new place. I'm going to be in Toronto until the 15th. I'm in for work but ahould have plenty of time for fun. Heck, as I'm good at doing, I am in Toronto over a 3 day weekend too.

I have taken this whole week off of work, including Friday but excluding part of monday. In that time I have  been able to catch up on several months of life. I got a haircut, went clothes shopping and had several Hendricks (great gin, thanks for the recommendation josebiro) Martinis. I have also added a few more needed features to the top-secret prokect and am still looking for someone to do some UI design work, I could also use someone who is interested in mobile applications, python, java and/or javascript because I think I need more people to work on this if I'm going to get this completed in any reasonable amount of time.

On one hand, I see that I am making steady progress. Every day that I sit down and focus on it, I walk away at the end with something newer and better than before. On the otherhand, some of the ideas that propelled me to work on this project seem so far away in the future that I'm worried ill never get to the cool problems that will make this project stand out. Heck, I've no doubt pumped some people (including myself) up about this and I'm sure it won't meet anyone's expectations.

On the other hand, I'm still happy about the work that I have completed, I just wish it hadn't taken four months to get here. My thought is that in the next few days ill start using the app on my own and see if in its current state it is usable. If not, I'm going to have to take a hard look at what it will take to make my project get there. if not I will have several months of rewrite ahead of me and I will surely have lost the first to market edge.

Labels: , , , ,

Sunday, August 31, 2008

Android/App Engine prototyping problems

Almost a year ago, Google unleashed its Android SDK in a preview capacity. One of the pain points I experienced with it was the tedium in creating ContentProviders. Today, I'm feeling the same tedium. A ContentProvider is the sole mechanism for an android application to share information with external packages. ContentProviders can be backed by any kind of data, file, database or live server with custom backend. The popular method is creating a ContentProvider that is backed by a database. Creating such a beast is a time consuming operation and making it with a SQL database is quite contrary to the BigTable based datastore API in Google App Engine (GAE).
In my top secret project, I'm using the HttpClientService api that I've defined in my Missing SVN repository (the top-secret project is indeed different than Missing) to interface with my server. Currently, the client does no caching of data -- all information it needs is pulled at user request, from the interwebs. When testing on my local machine, this is not a problem. I have a fast workstation and there isn't much packet loss or latency on a loopback interface.

Unfortunately, this won't work in the "real world." I'm now at the point where I have to implement a sql-lite backed ContentProvider that tries to model a table-based GAE datastore api. Not only does this mean I have to keep track of two different schema, one for GAE and one for Android, but it means I have to implement the same "get my data" api twice, with a layer of abstraction between the android UI and the ContentProvider I'm resenting having to create.

What I'm considering doing is using something like Google's Protocol Buffers or Facebook's Thrift to define my data model and create stub interfaces for both GAE and Android. This seems like a bit of overkill for the current state of my project but even in the not-yet-ready-to-show-anyone stage of this project I'm having to consider these very-high-time-cost coding excersizes. This is going to consume time when I should be iterating on features and trying to get the app in a state where I can finally start to use it. I think I've proto-typed the android app as far as I can go but I don't want to start spending hordes of time on this when I have several core concepts not-yet-implemented.

I'm also feeling the pain of knowing that the stuff I prototyped at the beginning of the project are going to have to be fully re-written and if I don't provide a more featureful content provider.

One of the ways you can provide an abstracted interface to the ContentProvider is by wrapping it in a Service, which of course requires yet another interface declaration, this one with java primatives and simple classes using the AIDL interface spec provided by Android. I figure I'll have to cross that bridge some day, I'm just glad I don't have to do it right now.

Labels: , , , , , ,

Sunday, July 20, 2008

Design Patterns... BAH!

I started reading "Head First Design Patterns" last week, and taking advantage of my new-found knowledge I tried to refactor Missing's http client stack. Its taken me about eight hours and it works again. Its as ugly as it was before, maybe even more so.

The kookiness of the design stems from the fact that I want the ability to respond to an HTTP response, possibly from a different thread. Android has a cool thread/message-queuing class called a Handler, that allows you to post a Runnable to the handler and it will get executed on the thread in which the handler was instantiated.

Because I wanted to keep my http stack android-agnostic (not that I had a good reason for it, in fact I didn't even think of doing it at first.) I removed the knowledge of the Handler from the .http package.

Instead I created a decorator for the HttpResponseRunnable that does know about Android's handlers. For the sake of re-use on another project, I put it in the .http package... DOH!

On the positive side, I did manage to implement a design that allows me to dynamically create http requests and responses without having to subclass all the time. I don't know if in the long run that will be better.

Things I need to think about:
  1. What is the best way to chain http operations where each request depends on data from the previous request?
  2. How would I design the HttpService from the ground up? Where can I look at code that has similar features to the ones I'm coding up to see what design they used.
  3. Why am I constantly re-writing this stack?
  4. If I am going to have a large number of http callbacks, if there are many types of end points, do I want keep the ability to define them quickly, or do I go with something more verbose and more abstract?

Do you know Java? Want to discuss some of this with me? I'm a newb and could use insight from someone with more experience than I.

Labels: , , , ,

Friday, July 4, 2008

My n800 running android

Labels:

Monday, June 30, 2008

Missing

One advantage of being several thousand miles from home is that you have the chance to sit down and work on things that you're normally too distracted to deal with.

As such, out here I've had the chance to work a bit more on Missing. I've re-written the http client stack I was using to use HttpClient 4.x instead of 3.x, with better abstraction and a multithread-but-threadsafe client. I've also gotten the LocationManager code working and the phone now supports a location update round trip. The server component needed a few tweaks to make this work but the majority of my time has been spent working on the Android client.

I became more enthused about this when jgib pointed me at sf0.org where they're running with the whole play-a-game-but-in-real-life-and-get-levels-and-stuff.

Labels: , ,

Wednesday, June 4, 2008

How to run the hacking part of a cyberpunk game. Part I

This was taken from a recent email exchange I had with some friends in regards to running our own cyberpunk game.

First off, to dash Josh's hopes and dreams: This is a cool idea but of all the games we play, this is going to be one of the more expensive ones. Plastic tubes, foam and duct tape are cheap compared to any projectile driving device. Second, if you're going to represent technology, then it has to be to some degree realistic, what game are you playing if your "internets" is just a hand wavy alternate dimension/plane with the minor environmental effects? I think in both cases you'll require some sort of investment in enabling technology. GPS, airsoft guns, light or wireless access points. Something that will bring out the cyber part.

Making the interwebs is a very difficult prospect. I've tried games like "uplink" and have spent much time figuring out how to make "hacking" anything other than a drab and boring experience; mostly because its a drab boring experience. If life were more like the movie Hackers we'd be much better off but there would probably be more capsized oil tankers in the seas and far more dead artists threating the world for 25 million dollars. Row ... row ... row ...

I think one thing that could make this whole game more fun is distributed involvement. Because you have communication devices at your disposal, its possible to involve people who are not physically located at the game. For example. Joe schmoe is busy one weekend and cannot be onsite. He happens to be the puzzle solver sort of person so he knows a lot of what is going on. During game play something comes up, a player calls Joe schmoe and joe schmoe looks up on wikipedia the solution to the puzzle. Winner! Or, do you not want to consider the world wide web at your disposal?

I think another source of inspiration for this sort of game are ARGs. They do a good job of blending reality with gaming and since we're already in a derivative of the cyberpunk, dealing with real world tech in game would not be a bad thing. This game is also going to need much more preparation than another rpg because writing a website is not as easy as writing in script on a parchment.

Because we're not playing a tabletop game we can look at what shadowrun /current/ edition does with Decking. Because of the way networks have firewalled themselves from the internet a lot of decking is done inside. The running team gets the decker into the corp network, then they do their thing. Or they kidnap someone with approprate access and use their accounts. They did a good job of getting a character class that was essentially a solo gig and made it in integral part of the runner team. In otherwords even your hacker characters are going to have to get in there and be involved in modules. This also means that where people are in meatspace is relevant to where they are on the "interwebs."

Now, here is my pitch and Josh is in disagreement with me here. I think that you need a software framework to be easily able to build and interact with virtual puzzles. I think you'll need multiple gps enabled smart phones (iphone, android or s60) or wireless network + PCs, a server, wireless networking to get this game off the ground. Communications play a large part in the cyberpunk world -- especially when they go out and the players who have come to rely on their decker or their off-site drones lose some control. I also think you need some evolution of my software (http://missing.googlecode.com) or something like it (hopefully there is something better than the crap I have put together) to build any sort of technological layer on top of your game. Now the problem is you need this stuff to be accessible which means, like I mentioned earlier smart phones and laptops for the game.

Labels: , , ,

Monday, May 26, 2008

Statusinator

Last year after the Android SDK was released, I wrote a small app that allowed me to upload photos and update my facebook status with a native application. Here is the result. I re-implemented a subset of the official facebook API because I didn't understand it. (It wasn't "not invented here syndrome" it was more of a "I am not smart syndrome"). Anyways, I've posted the code at statusinator.googlecode.com and there is a Facebook app page as well.

Labels: , , ,

Tuesday, May 13, 2008

ADC Round 1 Winners! | Android Phone Fans

ADC Round 1 Winners! | Android Phone Fans: "City Slikkers - a Pervasive Game (alternatively Location Based Game) which takes place in the real-existing city. It is designed to connect a large number of players through-out the world and change the way the surroundings are seen. The central idea behind the concept is to give people the opportunity to symbolically interfere with the everyday urban environment and come into contact with previously unknown people. By PoroCity Media and Virtual Logic Systems."

I'm very excited about this Android contest winner. It'll be exciting to someone else's take on my idea for "Missing."

I think I am most interested in their statement that the game will "affect normal life as the everyday city is no longer the same due to the player's experience." I didn't take into account that masses playing this game will end up having an impact on the non players. Bringing a new landscape into a city, one that isn't people getting from home to work or from point A to point B has great potential to make each city this takes place in a bit more personal. Sure many people will have their own thing to do and many others wouldn't be interested in this fantasy but some people will latch on and every new player will make the game more persvasive.

I was not very keen on the description that "As the playing community will always be a minority they hereby form some kind of elitist, secret society which is based on knowledge, but not financial or political power." That attitude could definitely sour the experience, bringing a futher gap between gamers and the general public. To really make this work there has to be varying levels of involvement and complexity. As a weak game player I don't want the game to run to far above my head which this seems to imply.

Labels:

The views and opinions expressed in the blog are of Joe LaPenna. Google has nothing to do with these pages.
For information about Google please visit: Google Press Center