Delphix Hackathon June 2015

For the past few years the Delphix Engineering team has been holding hackathons where all the engineers get together and build fun and interesting projects. They are held during our biannual Engineering Kickoffs and have the additional advantage of having the engineering team co-located in the same office. Our last hackathon was such a rousing success that we decided to try out our first distributed hackathon and additionally we went beyond just the engineering team and opened it up to the entire company.

The event was held two weeks ago and featured a glorious 24 hours of hacking where people from around the company collaborated to build a huge variety of things ranging from internal tools to customer facing products. Hackathons are a great way to get to know and work with folks you dont normally interact with and this one was no exception with several new employees and interns experiencing it for the first time. Being a distributed event meant that communication was an additional challenge as several of the teams were split across offices and even time zones. Slack was an extremely effective communication tool to keep in touch and setup great teamwork to solve any tricky problems that people ran into. In fact two of the hacks ended up being improvements to how we use Slack internally.

After a day of furious coding, the hackers put the finishing touches on their projects and presented them over Webex to both the participating hackers and judging panel comprising our CEO and other senior execs at the company. The creativity on display was amazing and this is just a partial list of the hacks that were presented:

  • Interactive company org charts using D3.js
  • Microsecond accuracy monitoring of TCP/IP connection latency
  • Automatically surfacing CVEs to the Delphix security team
  • Auto hotlinking bug mentions in Slack to JIRA
  • Automatically flagging failed tests with known bugs
  • A mobile HTML GUI for Delphix

Our judges had some really tough decisions to make and in the end awarded the following prizes:

CVE MonitorBest Internal Hack went to Aaron Garvey for the Common Vulnerabilities and Exposures Monitor which streamlines the process for keeping up with newly public vulnerabilities in the software we use. A scheduled task periodically checks a feed of new vulnerabilities, searching them to determine if they reference products or libraries Delphix uses. When a match is found, a notification describing the vulnerability appears in the slack channel, allowing people to further investigate the issue, and optionally create a bug in JIRA with a single click.

Best Customer Facing Hack was awarded to Sebastien Roy, Brandon Baker and Peter Washington for their TCP Connection Latency tool which allows you to measure network latency between a Delphix engine and its target hosts with microsecond accuracy. This allows us to quickly and easily diagnose any network issues that may be present.

Rahul Nair, Venkat Krishnamani and Simon Persson won the Audience Choice Award for their Mobile GUI for Delphix. It is a mobile optimized GUI that allows customers to monitor their Delphix Engines and perform simple operations (snapshot, rollback, etc…) directly from their phones without requiring computer access. The intention is to allow Sysadmins and DBAs to do lightweight monitoring of the engines without requiring a computer.
Hackathon Mobile GUI

All in all everyone had a great time and we have some fantastic new tools and features that were built in just 24 hours. Hackathons are a great tradition for the engineering team at Delphix and I expect that several of the hacks will impact both internal processes and our external product in the coming weeks. It was a pleasure to open this hackathon up to the entire company and it led to some good collaboration across the company. People are already looking forward to our next hackathon and planning the new and amazing things that they will build.

How to bungle a product launch – my Dodge Charger Hellcat story so far

I’ve never been a fan of muscle cars!

Growing up in India I never got to see any in person and Top Gear was pretty much my only window to the automotive world. This meant that I skipped American cars and instead was always a fan of the european supercars and had posters of the Ferrari F40 and Lamborghini Diablo on my bedroom wall. After moving to the US I did begin to appreciate price to performance ratio of cars like the Mustang but was only ever interested in buying small lightweight sports cars (NB Miata and Elise) myself. All of that changed when I bought the Seven – while it weighs only 1400lbs it does have 240 hp which gives it a power to weight ratio in excess of 375 hp per ton. Driving around on the street and passing people with the barest touch of the throttle finally made me understand the attraction of insanely overpowered family cars. An AMG or M sedan became something I really wanted but in the end I just could not justify spending $100k+ on a sedan that would be worth less than $40k in 10 years. I did come close to pulling the trigger on the 556hp CTS-V station wagon but in the end felt it didnt have enough space for the dogs in the back and we ended up buying a Ford Flex Ecoboost instead.

This order of things was shaken up when Dodge announced the 2015 Charger SRT Hellcat in August 2014. Here was the chance to buy a 707hp 4 door sedan with a full factory warranty for just $65k. Plus looking at the existing prices on used Charger SRTs they appeared to hold their prices pretty well and I’d hopefully see a depreciation curve like the Elise where the prices drop for a while before they stabilize and then start inching up. The final push over the edge was provided by this wonderful review of the Challenger Hellcat by my buddy Jonny Lieberman:

Cruising through the rapidly gentrifying streets of downtown Los Angeles in a 2015 Dodge Challenger SRT Hellcat the other evening, I came upon a white Bentley. Some sort of Continental, and for the sake of rhetorical flourish, let’s assume she was a GT Speed. Fast car, no? What with 616 horsepower from a twin-turbo W-12, it ought to be. Yet all I could think while every single muscle and tendon in my right ankle suddenly went taught was, “Aw, poor Bentley! You’ve only got around 600 horsepower. Keep trying, little guy, you’ll get there.” A hair-trigger throttle and 707 hp do strange things to a man’s mind. Then I mashed the gas and, well friends, the smile’s still there. Strange things to a man’s face, too.

I started calling the various local Dodge dealers to see if they were taking deposits for the Charger Hellcat. I knew that the Hellcats (Challenger and Charger) were hotly anticipated cars but I was not expecting the level of slime I ran into at the dealers. Every singe dealer wanted me to “come in” and no one was willing to give me information over the phone. Request to be transferred to the sales manager were ignored while no one would disclose information about markups or the length of their waiting list. The dealers I physically visited wanted non-refundable deposits as well as a markup on a car whose price was yet to be announced.

After talking to all of these folks I felt like I had to take a shower and was ready to walk away from the Hellcat when someone on one of the Dodge forums mentioned Bob Fredericks as a fair dealer to talk to. He had a great reputation on the Challenger forums and even had Dodge make a custom run of 10 “Kowalski edition” Challengers to celebrate the movie Vanishing Point. I gave him a call and found that he was willing to answer my three main questions: is there a markup, how long is your waiting list and is the deposit refundable. He guaranteed me MSRP, I would be his first Charger HC order and the deposit was refundable till Dodge picked up the order. It was a pleasure to talk to someone who actually gave me straight answers and I ended up placing a deposit on Aug 28, 2014 and settled down to wait for Dodge to actually open the ordering process. Dodge CEO Tim Kuniskis on multiple occasions said that the Hellcats would not be limited production cars and that they would build as many orders as they received. I figured that even if it took a while, as the first person on Bob Fredericks Charger list I would eventually get the car I wanted at MSRP.

Dodge finally opened the Challenger Hellcat ordering process in the US on 9-Sept-14 and said that the car would start on an “allocation” basis with the initial spots decided based on the total number of Dodge cars sold by the dealer in the previous 180 days. This meant that the larger city dealers would be getting many more cars than the rural dealers and meant that people on lists at a smaller dealer might be in for a long wait. They also said that future allocations would then be based on the number of days the hellcats spent on the lot – this was supposed to help combat markups as the higher markup cars would sit on the lot for longer periods. In the end Dodge received over 5000 orders for the Challenger hellcat in the first month which was wildly over all their internal projections.

Since the Charger Hellcat ordering was not yet open, I kept an eye on the Challenger HC allocation process and while information on the process was thin on the ground, it looked like Bob Frederick was getting 1 Challenger HC per month which made me feel pretty good about being the first Charger HC on his list. After a couple of false start order day roumours in December orders for the US officially opened on 12-Jan-15 and Bob sent me a purchase order confirmation which showed my order config along with my build priority of #1 (sold order).

Now that my order was officially in I started spending much more time on the forums and found that a good number of the folks that ordered the Charger immediately got a VIN number as well as a scheduled build date while I was left dangling. That said I never expected to get the first car built so I settled down for what I figured would be a reasonably short wait as I was #1 on the list and a lot of people with Chargers appeared to be progressing quickly. Then on February 27th as the first Charger Hellcats were getting built, the official Chrysler blog had an interesting message that pissed of a huge number of people.

Gualberto Ranieri, Senior VP of Communications for the Fiat Chrysler Automobiles posted an insane message that actually blamed dealers for taking too many orders:

Overall, our independently owned and operated Dodge dealers have been doing a great job managing Hellcat orders. However, a small number of dealers are engaging in a practice that is causing a customer relations issue for FCA US LLC and all of its dealers. Specifically, this handful of dealers appears to have accepted large numbers of SRT Hellcat orders without regard to available supply and without advising their customers that orders may not be filled, if at all, for many months or longer. We believe such a practice may constitute a breach of the Dealer’s Sales and Service Agreement with FCA US LLC and a violation of other applicable laws.

So according to Mr Ranieri, dealers have been taking too many orders for a car that Dodge has repeatedly said will not be in limited production and that they will make as many as people ask for. Additionally rather than wondering why people are flocking to a small number of dealers and trying to replicate they successful model, he instead is threatening some of the most trustworthy and respected Mopar dealers with legal action for being too good at their job. FCA must be the only company on earth where they yell at salesmen for being too good at their job – “Sorry Bob, you are flunking this course for being too far ahead of your peers”. And if they really did have an upper limit on how many cars a dealer should order, why the hell did they let dealers keep inputing orders? I may not work in enterprise ordering systems but I do know enough about computers to know that its not very difficult to put in a maximum limit on the number of things someone can order.

So in order to “fix” this ordering situation Dodge released the following flow chart on how to find out if your dealer has available allocation for a Hellcat:

While making customers navigate a flow chart to buy a car is a bad idea to begin with, this entire process has a few additional problems. The key pieces of info you need to use the chart are:

  1. Does the Dealer have a Hellcat in X-Stock?
  2. Does the Dealer have a Hellcat in C-KZ status?
  3. Did the dealer have a Hellcat in stock for over 5 days last month?
  4. Dealer rank by total Dodge sales (90 days)

Now you may have noticed that none of this information is publicly available to the customer on the street. Instead you have to ask the dealer and take their word for anything they say. Now of course car salesmen are known paragons of virtue and will never lie. For example the 3 salesmen at Stewart Dodge who in December told me that Dodge had not yet built any Hellcats and that the number of cars to be produced would be decided in January based purely on CAFE numbers. These wonderful gentlemen also wanted a $10k non-refundable deposit and would not tell me how many orders they had already taken. Rather than rewarding dealers who have larger order books (primarily due to their transparency and lack of markup), Dodge is instead dinging them for it and is rewarding slimeball dealers with cars despite the fact that no one is ordering from them.

Additionally having a single combined allocation list for both the Charger and the Challenger does not make any sense when the Charger allocations opened several months after the Challenger. I have the #1 Charger spot at my dealer, sent in a refundable deposit in August and got a VON on Jan 12. However this dealer also has several people that placed Challenger orders in November and now you expect him to decide which person gets that single allocation? It would not be fair to the Challenger folks waiting since September to do my car but its also not fair to penalize me for Dodge only allowing Charger HC orders in January.

In the end the blog post created a huge uproar in the various blogs, was picked up by several automotive magazines and sites and generally made Dodge look even worse than before. The comments on the blog post were universally negative and while the post is still up, insiders say that Dodge quickly realized that they had added gasoline to the fire and are working on yet another way to allocate the cars. That said it has been a month since the post and there have been no official updates from Dodge.

At this point I have been on the dealers waiting list with cash down for 213 days of which 76 days have been spent as a sold order in the Dodge computer system. Dodge has delivered over 350 Charger HCs already but at this point I have no idea if and when I will get a car on my own. The fact that my dealer is widely assumed to be one of the ones that the blog post has been aimed at has not made me feel any better while the complete radio silence from Dodge corporate is only adding to the frustration. I am no stranger to waiting for a custom built car – I waited 155 days to get my Elise but in that case both Lotus and my dealer were up front about the process and the expected delay. I am going to hang on for the moment but at this rate I might be better of joining a waitlist for the new Ford Focus RS as there is a decent chance that it might actually get here before my Hellcat.

The only good thing about this whole idiotic situation is that I finally have a valid use case for the following meme.
null

An update and posting elsewhere

Its been 13 months since my last post and I figure its time to give all 3 of my loyal readers and update on the goings-on:

  • I spent 14 months at Syntertainment building a great new mobile experience with Will Wright and his merry band.
  • In a moment of madness I signed up to be the chair of the 2014 West Coast Lotus Meet which will be held in Tahoe in October. I have been posting a lot of information about it on the GGLC blog.
  • The Seven has been surprisingly reliable with only a fuel pump failure in roughly 4000 miles and a half dozen track days. That said it did develop a misfire/surging throttle last week which I hope can be fixed by cleaning the throttle position sensor.
  • I signed up for the Polar Bear Blat which is a 16 day, 3800 mile drive from Vancouver to Alaska in August 2015. This will be in the Seven with 17 other Sevens.
  • I’ve joined Delphix as the first engineer on a new Vanguard Engineering team. Today was the end of my third week there and I have already posted my first blog post on the company blog: Docker Replication using Delphix Application Data.
  • I put down a pre-order deposit for Dodge Charger Hellcat. Never owned a Dodge or a even a sedan but 707 horsepower should make up for any shortcomings.

While I have been posting elsewhere recently I do plan on posting here more frequently especially after the mad rush of the WCLM concludes.

Setting up a Seven – Road trip to WCLM 2013

One of things that I’ve always wanted to do in the Seven was go on a long road trip. However I’ve always been afraid of mechanical breakdowns which is why I took the Elise to LOG 31 in Vegas two years ago. Once the 2013 West Coast Lotus meet was announced for Seattle in July, I started toying with the idea of taking the Seven but kept prepping both cars so that I could make a last minute decision. As luck would have it I was changing the oil on the Elise a couple of days before the trip and found a transmission leak which meant that the Seven was now the only option for the 2000 mile trip.

I started checking weather forecasts along the route and thankfully it appeared to be little chance of rain which is a good thing in a roof/door/window/windshield-less car with potentially dodgy electrics. I did see that a heat wave was expected with temperatures in excess of 100F but having done a 98F trip down to Buttonwillow in the past I felt pretty confident that I would be able to handle it with my cool suit. In terms of mechanical reliability I had driven the car several hundred street miles in the past few months which made me feel good about making it to Seattle without an issue. The risky bit would be the track day at Pacific Raceway but if a terminal issue cropped up Rob @ DiestchWerks said he would be able to trailer it back on his race trailer if needed. With all this worked out I decided to take the Seven on an epic roadtrip.

Epic road trip begins

I started out with fellow Ultralite owner and GGLC member Jim R who was going to drive up with me till Shasta City. We started out at 7 am as I wanted to cover as much ground as possible before sunset and had set myself the goal of reaching Medford at the very least with Eugene as a stretch goal. Since the planed distance was only just over 400 miles, we even took a slightly longer route to get the obligatory pic with the Golden Gate Bridge just like my last trip tor WCLM 08.

Required photo op

The first couple of hours went quite well and we covered 100+ miles before it warmed up enough that I had to start up the cool shirt which is basically a shirt which pumps cold water through it to keep the wearer cool. They were originally designed for surgeons but are now used by firemen, military, hazmat and of course racing. I originally picked up a cool shirt to use it in Lemons racing but after I got the Seven I realised that it was the perfect way to stay cool in the car as well. Jim and I made it up to Shasta Lake before stopping for a bite to eat at the Basshole Bar & Grill in Lakeshore CA just after noon.

By this time the time the mercury was really rising and Jim decided to head home while I continued on to towards Oregon. As I entered the mountains below Mt Shasta the weather was quite pleasant (though assisted by the Cool Shirt :)) and I was feeling quite good about my progress for the day. That came to crashing halt as I got passed the mountains and entered Oregon where the weather really shot up. The temperatures in Ashland were well above 100F (Jim saw 108F at one point on his way back) and it was so hot that I literally just pulled off the highway and parked under a tree.

Heat break

At this point the ice cooler part in the Cool Shirt came in handy as it meant that I had some cold refreshments to help cool myself down. I then decided to try driving a little bit without my helmet on but the hot air hitting me in the face made it hard to breathe so I put the helmet back on and kept going in short 30 min stretches. By around 5 pm the temps started dropping and by the time I got to Eugene at 6 it was pleasant enough that I took the helmet off off and cruised up to Salem, OR where I spent the night. The next day I was up and running again and made it to Seattle despite some extreme heat and traffic in the Olympia area. My dash got so hot at one point that my phone overheated and shut down. In the end I did make it to Seattle with the car running like a champ with only the organic bit behind the wheel having issues with the heat (see below). BTW if you think I’m complaining too much about the heat, it was so bad that the WDOT had to shut down a bridge and water its deck to prevent excessive heat expansion.

This is what 850 miles in 100 degree weather looks like

The first official event at the West Coast Lotus meet was a track day at Pacific Raceways which was put on by the folks at ProFormance Racing School. We started out with some lead-follow laps as most of us had never driven the track before. Pacific Raceway is a fairly technical track with 350 feet of elevation change which means you have a lot of blind late apex corners. Plus the track is very different from my usual tracks with very little run-off or exit kerbing and plenty of surrounding greenery – it feels more like a hillclimb course instead of a closed circuit track.

After a few sighter laps and some conversations with the instructors I was able to work out a reasonable line and really got into the groove at the track. As you can see from the video below I was not pushing very hard as I did not want to break anything at the track. One thing to note is that I did discover that top speed on my car is 125 mph which I hit 2/3rd of the way down the looong straight – it was actually a bit disappointing as I thought that the Seven was a bit faster than that but I guess crappy aero performance really does take its toll. We did have a modified Elise (slicks, 300 hp, sequential gearbox, paddleshift) that was quite a bit faster and was hitting 140 mph down the same straight.

The most interesting moment of the video above was when I go off at the 14:20 mark. This was due to a stuck throttle though luckily it was stuck partially open and not at full throttle. I didnt realise this initially as the revs do go down when I get off the gas but the car did not slow down enough. I keep adding more brake pressure and end up locking the front wheels before I go off the course. At this point I go both feet in and the engine revs rise to ~4k rpm and I realise what the problem was. I was able to use partial clutch to get into the pits and killed the engine as soon as I got in. When I opened the hood I found that the bracket that the external throttle release spring attached to had broken off and the stock throttle spring was not strong enough to close it all the way which is why the engine was stuck at part throttle. I was able to ziptie the spring back together to get the car going again but called it quits for the track day as I didnt want to risk it happening again on course.

Went off with a broken throttle return supporting bracket. Back running with a quick ziptie fix

After the track day I headed over to the event hotel which had organized some excellent “Lotus Parking” for the duration of the event. This meant that we not only got to see the cars all together, but we also got to chat with all the owners as we came in and out of the lot. After the cocktails and reception many of us headed out into the parking lot to check out the cars and shoot the breeze with other Lotus fans.

Lotus everywhere

Day 2 started with an excellent tour of Paul Allens Flying Heritage Collection. Nothing to do with Lotus but it is an excellent collection of WW2 era machinery including airplanes, rockets and tanks. Many of the airplanes are still airworthy and are flown a few times a year. Its also right next to the Boeing Dreamliner factory where a bunch of planes were getting fitted for their customers. I highly recommend it for anyone who likes mechanical objects.

In the afternoon we headed over to the parking lot of Bellevue Community college for the WCLM autocross. It was a very tight course that you lapped multiple times to set a time. Most people were doing the course in first gear though due to my low gearing I was able to launch and do the entire run in second gear instead. The following video is of my 31.165 second run which ended up as the top time of the event. I was even able to avoid hitting “my cone” which folks were taking odds on whether I would hit it. Facebook users can see more photos from the autocross at the WCLM FB album

After the autocross we headed over to the Snoqualamie Casino where we had an excellent buffet dinner along with drinks on the roof right in the shadow of the mountains. Plus it was another great chance to check out the cars and we got a lot of regular casino guests coming by to ogle at the cars.

WCLM Dinner

I was feeling a bit under the weather on day 3 so I skipped the SOVREN Historic Races and the parade laps at Pacific Raceway. I directly went over to the LeMay museum in the afternoon where we were taking a group photograph before dinner and a private tour of the museum itself. The album below shows some of the cars on display in the museum, as well as some shots of the cars lined up for the group photo. Another wonderful place to attend and much better organized than the original warehouses that they used when I first visited the LeMay in 2010. After the LeMay trip I was hanging out with some Canadian attendees at the hotel who were quite surprised to learn that I had driven the Seven all the way from CA. Hopefully at the next WCLM we can have a some of them drive all the from Canada instead 😉

The final stop on the WLCM calendar was at the Griots Garage retail store in Tacoma where we got look at some of their cars and also got a demonstration of their car car products. Not very Lotus specific, but they are definitely car guys as evidenced by their McLaren display below. I have to say that Doug and the ELCC really put on a great WCLM and are going to get a lot of repeat business the next time they host a WCLM.

McLaren

After lunch I headed out around 2pm with the goal of making it back to Eugene, OR before nightfall. The return trip was going well until I stopped just before the WA-OR border and noticed that the left side of the car was covered with coolant. The upper radiator hose had sprung a leak and was dripping coolant under pressure.

And now we're dumping coolant

I filled it up with some water and Rob @ DiestchWerks a call to see if I could get him to trailer the car back. It ended up he was about 45 minutes ahead of me so I limped the car over there while he stopped at an auto parts to see what he could find. In the end he cut the hose at the point that it was leaking at and used a plastic coupler and some hose clamps to put it back together again. I then got back on the road and started driving with frequent stops to check the coolant levels and to watch for any further leaks. I was able to make it to Canyonville before nightfall and stayed at the Seven Feather Casino which was the site of the WCLM 08.

Spending the night at the home of WCLM 08

I decided to drop my original plan to drive straight down I-5 as the temperatures in the central valley were expected to be well over 100F and I did not want to put additional stress on my cooling system. Instead I took US-199 over from Grants Pass to Eureka and then took 101 all the way to SF. This turned out to be a great decision because though the trip would be longer, it was much much cooler plus US-199 is an fun road to drive in a sports car.

Lunch at the Samoa Cookhouse

After having lunch at the Samoa Cookhouse above, I took the famous Avenue of the Giants route through Humboldt Redwood Park which was another fun detour.

The Avenue of the Giants in a not very giant car

At this point I was far enough inland that it was starting to heat up again and I had to put my cool shirt on again. Plus the temps never went about 95 which meant that the cool suit was able to keep me quite comfortable. I then stopped off to meet a friend in Windsor, CA and was able to avoid much of the evening heat before making the final 90 minute drive back over the Golden Gate Bridge and through SF to get home again.

Home at last

All in all the car ran quite well and apart from the throttle and coolant issues it gave me no trouble at all in over 2000 miles in some blistering heat. I got lots of weird looks and several photos taken of me on my trip but I have to say that it was great fun and I would say that all Seven owners should consider doing at least one long road trip in their cars – it is an experience that you will not forget and you’d be surprised how reliable our cars can be. Now where is WLCM 2014 going to be? :-)


View Larger Map

Google Glass needs a Personal Video Loop

I’ve been doing a lot of thinking about Google Glass at the moment and so far I havent seen anything to make me want to rush out and buy a pair (although I reserve judgement till I actually play with one). One of the things getting a lot of attention is the ability to record hands-free video which in my opinion is over-rated. I’m basing this on in-car video recorders (and more recently dashcams) that people have been using to record hands free video while driving for years. I have been using a variety of different setups to capture in-car video over the years and have found that while its very cool at first, it’s quite rare to go back to look at the video. Of the hundreds of hours of video I have collected, I only look at it when there is something specific I needed to check (line, traffic, best lap, etc…) or to see if the video captured something spectacular (spins, passes, accidents, etc…). In general it is quite boring to look at hours of raw video even while doing something as interesting as racing cars on track. As for making edited videos it is a long, slow and difficult process and people wont be bothered to do it once.

What Google Glass really needs is to take a suggestion from the dashcam world and record a continuous loop of video that the user can save after the event to record the important/funny/interesting thing that they just happened to see. It would be a video version of the Personal Audio Loop (paper pdf). A Personal Video Loop if you will.

Imagine being able to immediately save some cute thing thing by your kids/dogs/cats/etc… Instead of remembering to tell Glass to record, it is always recording and you can just tell it to save the last X minutes. Apart from the cute overload videos, the number of dashcam videos on Autoblog and Jalopnik are proof that even the most average people will keep seeing some interesting things. Plus the sousveillance aspect of having cloud-stored video to back up you account of events would be pretty revolutionary as well.

I know there are a bunch of legal (and power) issues around recording video like this but someone should try building it. Glass explorers – start exploring :-)