I recently started playing with Apache Pig.

Pig is a high-level platform for creating MapReduce programs used with Hadoop wikipedia

MapReduce is a programming model for processing large data sets wikipedia

I'm planning to run everything on my Macbook Air with 128GB SSD, so "large data set" will be relative.

I decided to search NYC Open Data for an interesting dataset and found this, the NYC Restaurant Inspection Results. (only 90MB)

As a quick recap for non US residents: Restaurants get points for violations. If a restaurant has an inspection score between 0 and 13 points it receives an A, those with 14 to 27 points a B and those with 28 or more a C. More info here. As a customer you like A, might tolerate B and want to stay away from C.

We have the data, but what information do we want to get out of it?

I figured it would be fun (and controversial!) to find out which cuisine is prone to be associated with bad ratings. For example, we might find out that cuban restaurants tend to get a lot of A ratings, whereas italian restaurants get more B's.

Here's the Pig Latin (with verbose comments):

show/hide

After running this script we end up with this file.

And here are the results, rendered with a short JS script (A - blue, B - green, C - red):

Now - while this exercise allowed me to produce pretty graphics the question I really wanted answered was:

Which cuisines are 'safe' and which should I stay away from?

I extended the Pig Latin script to produce the necessary result.

(You can look at the raw results here)

Based on these results we can create a a top-fifteen list of cuisines per grade:

Conclusion: The good ol' street cart seems safe (Hotdogs/Pretzels, Nuts/Confectionary), so does Armenian and Ethiopian. On the other end, try to stay away from chinese fusion (Chinese/Cuban, Chinese/Japanese) and Polynesian.

Disclaimer: Obviously, please take all of this with a grain of salt. The purpose of this exercise was to have some fun with Pig (yeah I said it), not to produce a 'proper' analysis of NYC restaurant ratings.

I, for one, enjoyed working with Pig quite a bit. The language is in it's early stages but mature enough to be fun to work with. it seems obvious to have an abstraction like this that keeps you from writing many many lines of boilerplate MapReduce code. That said, as with all abstractions, it's crucial to understand what happens under the hood. Especially when working with large data sets, execution time is super important (a simple group all or dump statement can cause a lot of trouble). For a good introduction to Hadoop and Pig I recommend this and this video.

My next step will be to use a bigger dataset and run the whole thing on a real Hadoop cluster instead of my local machine.

Dark and dirty track, accompanied by a fuzzy video by the29nov.

Just a short mix I recorded a while ago...

Objective-C Literals are a fabulous addition to the language. There's no doubt that this will make working with it a lot more fun and efficient.

However, I'd like to point out two things that struck me as a bit odd:

First, Boxed Expressions:

Follow this pattern:

NSLog(@"%@", [@"Hi There" class]] 
=> NSString (actually __NSCFConstantString)

NSLog(@"%@", [@[@"foo", @"bar"] class]) 
=> NSArray (actually __NSArrayI)

NSLog(@"%@", [@{@"key": @"value"} class]) 
=> NSDictionary (actually __NSDictionaryI)

NSLog(@"%@", [@3 class]); 
=> NSNumber (actually __NSCFNumber)

Basically @ followed by something will turn something into the NS version of something. Easy.

Now, here's one more:

NSLog(@"%@", [@YES class]);

Following our pattern, this should be something like NSBoolean, but there is no NSBoolean. What it actually turns out to be is [NSNumber numberWithBool:YES]. Yes, an NSNumber (actually __NSCFBoolean, a private subclass of NSNumber).

Now, this might make sense when looking at the NSNumber API. All +numberWith* constructors were given literal equivalents.

NSNumber *a = @3;
NSNumber *b = @4.0f;
NSNumber *c = @YES;
...

but it certainly isn't obvious when looking at this in a piece of code:

if (@NO)
    NSLog(@"Did you really expect to see this?");

I'm not sure what the motivation was to make __NSCFBoolean a subclass of NSNumber, instead of creating something like NSBoolean. Whatever the reasoning, the newest additions to the language hint at the possibility of this being a little flaw in the type hierarchy of the language. (I realize this is a ballsy statement coming from someone with limited experience in this particular field).

Second, Container Literals:

Here's how you create an array using the new Obj-C literals:

NSArray *array = @[ obj1, obj2, obj3 ];

Short, concise, a developers dream.

One might think this is expanded to

NSArray *array = [NSArray arrayWithObjects:obj1, obj2, obj3, nil];

but it's actually expanded to

int n = 3;
id foo[n];
foo[0] = obj1;
foo[1] = obj2;
foo[2] = obj3;

NSArray *array = [NSArray arrayWithObjects:foo count:n];

The difference is subtle and in most cases, the code won't behave any differently. To showcase this difference, let's assume one of the objects is nil.

[NSArray arrayWithObjects:obj1, obj2, obj3, nil] will terminate the list before any nil value. So, if obj2 were nil we would end up with an array only holding obj1. That's it.

@[ obj1, obj2, obj3 ] will throw an exception (or a compiler error if it can be determined at compile time). Boom.

Some might say the latter behavior is more intuitive, I would agree it's certainly more intuitive than silently cutting off the array when a nil value shows up. My feeling is, this change was made very consciously to get rid of a piece of legacy API they weren't very happy with.

The same is the case for dictionary literals.

NSDictionary *dict = @{ @"key" : @"value", @"key2": @"value2" };

will not expand to

[NSDictionary dictionaryWithObjectsAndKeys:@"value", @"key", @"value2", @"key2" nil]

but will instead use +[NSDictionary dictionaryWithObjects:forKeys:count:].

Again, the difference is subtle. If a key or value is nil, the former piece of code will terminate the list early. Which might cause a problem depending which item in the list it is. +[NSDictionary dictionaryWithObjects:forKeys:count:] will throw an exception or cause a compiler error.

All of this is clearly mentioned in the "Discussion" section of the LLVM docs on Objective C Literals.

It completely makes sense to choose the more secure expansion for both array and dictionary literals. But I'm curious to see if there will be any impact/outcry when people try to convert their code to use literals.

Then again, well tested and designed code will not try to use nil values in arrays or dictionaries. So in the worst case, this will just prompt some developers to clean up their act, and code.

What's a bit ironic, is that they added a piece of "discussable" behavior with Boxed Expressions as discussed in the first example (in particular @YES) while cleaning up some existing "discussable" behavior with container literals.

You win some, you lose some. At the end, we still have a great language that's the base for an incredible ecosystem of technologies. Especially given the fact it's really just a thin layer on top of it's parent.

http://www.digitalbond.com/2012/01/31/langners-stuxnet-deep-dive-s4-video/

Nils Frahm - Wintermusik by sonic pieces

He also has a new album out, called FELT:

Nils Frahm - Felt (teaser) by erasedtapes

"It is disingenuous at this point in time to equate race with mere color. The opportunity for that is long since gone. Initially, it was true that the only difference between blacks and whites was skin color, but the experience of centuries of racism has made that idea utopian."

http://cyberlog.wordpress.com/2007/11/18/fran-lebowitz-on-race/

Andy Stott - We Stay Together by modernlove

Follow up to the excellent "Passed Me By" EP. Not quite as good but still worthwhile, especially for the rainy days to come.

Favorite track from the great Passed Me By EP.