Troubleshooting the Database

A customer sent me a short email about formatting in their new system. Simple enough, I created a sample in the system and sent them a specification for formatting, and told them “just delete the sample when you’re done”.

Alas. Ten minutes later, an email arrives “I deleted the sample… everything is gone and I can’t log in?!”

Shock.

Shock fades into denial – it must be the platform. Right? A database failure, or a virus. Glancing at the database, it appears to have rolled back 3 months. I sent off a support request to Heroku and they pulled a friendly face of “that’s very odd, our databases don’t do that kind of thing on their own”.

So, I took a closer look, and sure enough there are suggestions it isn’t just a rolled-back database – a few missing things were created before a few present things, so it isn’t just a plain rollback, more like a systematic deletion. And the automated database backup hasn’t run since last night, so the most recent work by the customer is lost.

Despite it apparently being a bug in the delete method, no matter how hard I tried, I wasn’t able to reproduce it in the CMS locally or on production. I created almost exact replicas of the structure prior to the error, and couldn’t get a repeat of the bug – maybe there was something malicious at play?

I played around for hours, loading up content, deleting it, renaming it, trying to be malicious, and nothing came up. I pored over the logs from the client working on the CMS, and as I got further back in time noticed that they were testing content in the “client” side application as they added it.

I connected the client side application and worked through some exercises, deleted a few from the CMS. Nothing. Edited a few more things, completed a few more exercises, hit delete.

Long pause… “Signed out of CMS”. Ok… How odd. I couldn’t sign back in, either – user not found. I’m a system administrator, how can I be not found! This sounds exactly like the reported problem. There’s a lot to be said for examining the logs carefully – not just the immediate problem, but the surrounding context. Get a feel for what the user was doing when they encountered the problem!

Another database dump, and sure enough, my user account is deleted, along with quite a few (but not all) of the slides. Mixed feelings at this point – on one hand, I’ve reproduced the bug and am making some headway. On the other hand, the bug and subsequent data loss must be my fault – depressing.

At this point I wrote down the facts as I knew them

The user needs to use the application as well as the CMS for the bug to occur
A few User(s) are deleted
Many Exercises are deleted

Not much links users to exercises in the system. Just one little table called “UserProgress” which has a many-1 relationship with users and exercises, which entries are created in as the user progresses through exercises. A quick check of the database, and indeed all UserProgress rows are deleted as well. A lot of fingers now point at this relation.

Opening up the rails models, the UserProgress model has dependent => destroy set on it’s relations to User and Slide. On the other end, that’s also set – so if one end is destroyed, so is the other, regardless of which end is destroyed. This is not good. If a slide is destroyed here, it cascades – all the Exercise’s Progress relations are deleted, then all the Progress’s Users, then all the User’s Progresses, then all the Progress’s Exercises. We have come full circle – and it doesn’t have to end here… This explains why almost all users, and absolutely all slides vanished from the production database.

I created a simple test case and deleted a slide the user has marked progress on. The Bug is reproduced! This is a vital step in any troubleshooting. Another two unrelated deletion-of-progress bugs raise their heads – unrelated in the sense they don’t help fix the initial bug, but they take an hour to fix (they get in the way of running test cases for the show-stopper bug).

A quick removal of the offending dependent => destroy association from the UserProgress model (http://guides.rubyonrails.org/association_basics.html – deleting a customer deletes orders, which is sensible. Not the other way round!) and I am ready to re-run my test case.

Fixed! Hurray – off to the autobuild it goes.

Lessons learned:

Bugs usually clump together around bad code
Bad code is usually the less tested code
“System” level bugs that don’t occur until everything comes together require more complex tests
Be humble, expect the mistake to be in your code, not others
Don’t just throw answers together until they work. (i.e. putting the association property on both sides “just to be sure”)

Uber Update – The secret Pyro Update

Again with the secret updates! Today, there is a hidden pyro update in addition to “World War Wednesday” Uber Update page. Right there as soon as you load the page. Yep, on the blackboard, a picture of a pyro with a great big arrow pointing to it – it’s a link to the pyro update.

Only one new weapon this time, “The Detonator”. Seems like a good name for it too, seeing as it’s a flare gun that detonates. Nice and simple! Details are scarce – “Alt fire detonates explosive flare” – but let’s take a guess – we shoot the flare, detonate it, and set nearby targets on fire. As benefits are rarely without drawbacks, I’ll take a guess at slightly lower damage, a slower moving flare, or increased reload time, though that’s pure speculation.

Sounds like a great weapon for setting distant enemy groups on fire, and a great addition for spy checking at a distance. Anyway, once again it’s exciting to find another secret. Makes me wonder if there’s a missed secret on page 1 – has anyone found one?

Team Fortress 2 – The Semi Secret Uber Update For Scout

Alongside the easily seen Mobster Monday and Timbuk Tuesday is a subtly (or not so) hidden scout update. Lurking at the bottom of the Timbuk Tuesday page is a can of what looks like bonk atomic punch. It’s a link! Click the can and you are led to a page detailing a new weapon set for the scout, “The #1 Fan”

Valve's scoutpack update — Awesome artwork as ever, courtesy of Valve.

“The Soda Popper” looks like another weapon rewarding you for killing effectively, like the Bazaar Bargain, building up ‘hype’ ready for a rain of minicrits. A smaller clip size is your downside, much like the Force-A-nature. A rain of minicrits from behind could cause total disarray to an unsuspecting team.

Next up is a bat – “The Atomizer”, which looks like an ‘access’ weapon – useful for getting behind enemy lines, grabbing a health pack, then wreaking havoc on the enemy team (or their teleporters). I imagine that third jump will also give an interesting edge in combat during the first few weeks when no one is expecting it! Combined with the force-a-nature, will this give the scout a quadruple jump… One can only imagine so, and the prospect is awesome.

The last new weapon, “The Winger” seems like a standard but nonetheless useful pistol replacement. The extra damage, presumably without scatter, looks like a great opportunity to sneak behind enemy lines and snipe their sentries/medics/etc when they least expect it.

Hidden updates somehow always excite me more than the obviously stated updates, it’s somewhat like a treasure trail…

x not a function or static data member, in CUDA

This error message misled me for a while. It was telling me that my templated CUDA kernel was “not a function or static data member” – this after I added a struct pointer as a parameter. If I changed the struct to an int parameter, it worked fine (except being semantically stupid). I didn’t see anything wrong with my declaration of the struct, but on closer inspection, I had mis-capitalised one letter. Not the error message I’d expect for this problem, so it took longer than expected to find.

Hope this helps if you have the same problem…

Reacquiring your ‘gaming mojo’

After reading this, I wanted to put some thoughts down. http://ask.slashdot.org/story/10/11/25/067219/Have-I-Lost-My-Gaming-Mojo?from=rss

I can think of two ways you might lose your gaming ‘mojo’. At least these are the two I’ve experienced myself.

The first case is when I find myself getting slaughtered in x game (or genre of game), when I remember that I used to win easily… I’ve had this with first person shooters, strategy games, and specific titles. Mostly this is a multi player phenomenon, but in some cases also single player. I’ve put this ‘sudden’ degradation in ability down to an unnoticed learning curve. In these games, I got good through practising, but without noticing an increase in talent.

When I come back to them I expect to be good, get beaten, and as a result feel, well, quite bad. Sometimes I just put the game away and resign it to the past. If I care, for whatever reason, I usually go to train in a harder game to bring my skill level back up (playing, for example, red orchestra for an hour or two before, say, team fortress gives me an epic boost in kills).

The second case is when I can’t find a game I actually want to play. I have plenty of choice (really, woe is me) but I just can’t get into anything. I flit off to other things, coding, walking,e eating, but can’t play. Sometimes I just go with this, and get on with non game life – but sometimes it feels like “I don’t want to play anything any more, maybe I should even sell my pc!”. Then I get worried that an important part of ‘me’ is missing, and desperately want to get into a game. What does it take for this? Well, something good usually.

And recently I noticed something remarkable – while I’ve lost interest in strategy gaming, I picked up total annihilation the other day, and as dragged in, despite being tired, cold, and uncomfortable… While first person shooters have recently been quite good, F.E.A.R. and Red Faction captivated me in a compelling way. And I can’t tell you why. But I will try to work it out. All I can really say is, many games recently are missing elements common to these addictive classics.

Calculating the FLOPS of your NVIDIA gpu

This is the formula for calculating the peak FLOPS on your cuda enabled gpu. (as uses by NVIDIA) This is useful as a comparison when evaluating how many FLOPS your cuda kernel is achieving, and how much you can hope to gain through optimization. You can get the numbers needed from the NVIDIA control panel’s system information option.

If your card is pre-fermi:
Processor clock*cuda cores*3

If your card is fermi based: (or higher?)
Processor clock*cuda cores*2

Original Source here.

Internal server error 500 using SVNKit (hudson problem)

Had trouble checking out from SVN on a Hudson client, the error rather cryptically just “internal server error: 500”. The exact same project, but checking out onto a different client, worked fine.
After some hunting around and trying various stuff in vain, I deleted the SVN authentication details stored on the client at %APPDATA%/subversion/auth/svn.simple and the checkout started worked properly.
I assume this is because the client in question had been previously used to check out from SVN with other applications and that the cached credentials were somehow not-quite-right, but that’s speculation.

Paste unformatted text in Word 2003

Annoyed that Word keeps trying to paste your copied text in the original format? Record a new macro to fix it!

Tools -> Macro -> Record New macro
Give it a name – I called mine “PasteUnformatted”. The name can’t have spaces.
Click keyboard shortcut and give it a shortcut, I used Ctrl+v (this means the default paste behaviour on ctrl+v is ignored, but I never want the default behaviour)
Click Ok. The macro starts recording.
Click Edit -> Paste Special -> Select ‘Unformatted text’, then stop recording the macro with the macro toolbar.
This doesn’t quite work, as it forgets the unformatted bit (don’t ask me why, this is Microsoft), so open tools->Macro->Macros.
Select your macro and pick edit. Change the line to this:

Selection.PasteAndFormat (wdFormatPlainText)

Save and exit the macro editer.

now when you press ctrl+v you should get unformatted text (so it should match your document’s current formatting)

There are better ways to do this in Word 2007.

‘Windows 8’

So, someone leaked windows 8 discussion documents and uploaded them, and you can take a look at some of them over at Microsoft Kitchen

I just found it amusing that in this slide: amusing slide about windows 8

They are clearly considering their target audience as ‘everyone’ and have a slightly humorous (I think) section

“Why humans matter”
-Substantial in size

Other than that the idea of logging in by face recognition sounds abysmal. But the increased speed and ubiquitous settings through a cloud is more interesting.

Chalk and cheese, and pork and fleas

Games I’m really enjoying at the moment!

Assassin’s creed 2 (and by proxy 1, as 2 has taught me how to enjoy 1 a lot more). You’re an Italian assassin, enacting revenge for your father’s death, and something altogether grander too. Such an improvement on assassin’s creed one it’s unbelievable.

Dirt 2 – a varied and relaxed rally game. (so often rally games are too hardcore and gritty) Has a great feel, and is one of the best racing games I’ve played. One of the best games – it hits so many spots. I’ve actually gone so far as 100% completing the single player.

Tropico 3 – A ‘Caribbean island sim’. I have Tropico 1 and 2 bought recently from a d2d sale and have briefly touched on them, but it’s 3 that I’ve been spending most of my time with. A great atmosphere, nice looking, and with a horrifically addictive gameplay, it’s been calling me back for a month or so now. And nothing gets your metal limbs ticking like being called El Presidente!