Log in

No account? Create an account


The Sarmatian Protopope

his desires inscrutable but surely base

Fail fast, recover fast: software and relationships
I write software for a living.

In the world of software, we're generally obsessed with not making mistakes (keeping our code bug-free). You wouldn't know it, given how many bugs there are in the code all around you. But we do try.

The thing is, mistakes happen. Bugs creep in. This is an ironclad, unassailable law of nature, right up there with death, taxes, and gravity.

Now, when mistakes in the code happen, it has various effects: people die, businesses lose money, or users are mildly annoyed for a fraction of a second on their random clickstream from Farmville to 4chan. (I'm very thankful to work in the industries of the second and third categories -- to date, nobody has died when my code crashes somebody's web page that is advertising extra-fancy baked beans or whatever.)

Anyways, as you can imagine, businesses, governments and universities spend a LOT of time and money studying how not to make mistakes (in the code). And there's generally two approaches in this quest.

One goes like this: *You try EXTRA HARD to not make any mistakes*.
To achieve this, there is a number of well-defined steps. First, you write *very careful* requirements. You set down a novel's length of rules and specifications, describing in great detail what your relationship, er, software, is going to do. You try to imagine all the ways in which the software can break, and make contingency plans for each one of them.

Then, after the requirements and specs are written, you *design* the software very carefully. With diagrams and plans, and lots of smart people involved. After that (and this is VERY important), you get EVERYBODY to sign off on the design. I mean, sign off in triplicate. The designers sign off on it, that they promise that it contains no mistakes. The engineers who will be building it sign off on it, that it looks reasonable and doable within time and budget. Your manager signs of on it, and her boss, and her boss's boss, all look over it carefully, and put their signature on it. A few key clients involved in the process also sign off on it, that this is exactly what they want, and that they surely will buy it in such and such quantities.

With so many smart and educated people involved (and I don't mean this sarcastically. They really are at the top of their fields), how could this plan possibly fail? And if it does fail, whose fault could it possibly be? Not the engineers -- their managers and clients put their signature that it's exactly what they want. Not the managers -- after all, *their* bosses looked it over and gave the go-ahead. And the designers and engineers should have foreseen all the potential pitfalls and contingencies!

Doesn't this sound reasonable and logical? It does to me (and to most national governments, and pretty much all businesses ever for several decades).

It also does not work. I mean fails utterly. (Rather, this process *does* work in a few rare cases -- but only in a completely predictable environment, where you're solving a known problem that is thoroughly understood, and which has been encountered before in exactly that form. But.. who lives in such a world? I don't, and you don't. None of the businesses alive today do.)

But usually? Either the software does not get written at all (because of unforseen complexities, unknowns and pitfalls), or it gets completed way past deadline and completely over budget. If it does get written, it still contains a lot of (sometimes killer) bugs in it. And even if it has relatively few bugs, by the time it gets finished, nobody buys it. It's not what the managers or the customers wanted (despite having read over the specs and design and put their signature and assurances on it).

Does such a consistent pattern of failure happen because of stupidity and incompetence? Because of a lack of effort? No. Usually, everybody is intelligent, conscientious, and works pretty hard. But nobody has the godlike intelligence to fully understand a new problem in a chaotic environment, to see all of its implications, failures and solutions. Nobody can design or create in a vacuum, with no customer feedback. And no customer knows themselves so fully that they can picture the finished product from your description, or can imagine what they want until they're actually using it.

However, over the years, the software industry has slowly been discovering another approach. Many of the bravest, happiest, and more importantly, richest and nimblest groups and businesses have stumbled on it.

It goes like this: Try a lot of things. Fail REALLY fast. And *recover even faster*, and put in course corrections. Put in clever safety nets and procedures to make sure that when you do fail, your mistakes don't do a lot of damage. Because you try them in a safe sandbox. You take tiny, bite-sized steps, in safe controlled environments. You learn very quickly, about yourselves, your team, your customers, your market.

This approach has many names (generally referred to as agile development, continuous integration, and, at the extreme, continuous delivery). And it is not for the faint of heart -- the teams have to be brave and resilient, the failsafes have to be cleverly designed, and everybody has to learn to forgive and reorient quickly. But holy shit, does it produce amazing results, and fast.

Now, as far as software goes, I believe in the second approach wholeheartedly. I can sing its praises, teach seminars, point to extensive studies that prove its effectiveness. At every place that I've worked, I've tried to apply the fail-fast/recover-fast tenets, to steer the dev team culture towards those practices. And have watched it do wonders, to speed up the business and make it more resilient.

But at home? I was completely blind to these patterns.

I love catvalente. A lot. I do not ever want to make mistakes. I don't want to do or say the wrong things. I don't want to have fights, and if we do fight, I don't want to overreact, and so on and so forth.
But when mistakes do get made (more often than not by me), and fights do happen... I feel like it's a disaster. I feel like the whole world stops (complete with a stop-the-record scratch noise from a movie). I get depressed for hours and days afterwards. How could it happen? Seriously, how can smart and competent people LET THIS HAPPEN? The specs and blueprints for our personalities and relationship? Must have been flawed to begin with! The teams? Incompetent! Ok, fine. But if we TALK about it a lot, and make the design a whole lot better, and TRY REALLY HARD not to make mistakes the next time, then they won't happen, right? (And then repeat, ad nauseum).

Yeah, I exaggerate. But only slightly. I feel really dumb about this, now that I see my own mental script.

And I've only fully realized all of this now, tonight. And even that much, I've only been able to do because, for just a few days (while distracted by something else), I gave the fail faster/recover method a try, unconsciously. We had a stupid fight (I snapped at her during a tense moment, overreacted, then blamed her for it).. except this time, through sheer luck and lust, I caught myself just a tiny bit faster. Backed off a bit sooner, apologized. And, most importantly, did not spend the rest of the evening (and the entire next day) depressed and blaming myself (and her) for being fundamentally badly designed for letting it happen in the first place.

Yes, I know, applying abstract concepts and analogies to messy human relationships is a ridiculous geek fallacy. But I am what I am. This is my mantra to myself, then: Beast, apply your beliefs from your daytime world to your home. Stop dreading mistakes and trying to hold the world together with your perfect will (ha), and stop blaming yourself and your girl for when they do happen, and thinking that it's the end of the world. You are not a godlike precise elegant virtuoso with perfect control and foresight. You're a sloppy, over-emotional beast. So, it's ok. Fail fast, recover even faster, and forgive like a pro.

On Åcon 5
Cheryl Morgan's writeup of Åcon 5 (a small yet fierce literary convention on the Finnish (Swedish-speaking) island of Åland, where catvalente was a guest of honor this year) explains it better, and in much more detail, than I could.

Real-time (?) Wind Map
For those of you fond of maps, real-time Earth representations, and wind, (like I am), check this out:

Real Time US Wind Map

(thanks to omnia_mutantur for showing this to me).

Sketching on the iPad
Question for my artist friends:

Do you use an iPad for sketching? If yes, what's the best iPad stylus? And what iOS software do you use to sketch or doodle?

If not, and you use some other tablet (Bamboo or whatnot), which one do you use? And what software (meaning, anything else besides Photoshop?)

Stuff on my mind
So much I want to write about! Not sure where to start. So, here, in random order:

* s00j (plus her boy) and stealthcello visited last week! Oh my god, it was so great to see them. I only caught a brief glimpse of them during Boskone, so it was really nice to have them over our house for several days.

* The house concert went off beautifully. (As hosts, there's always that anxiety of "well what if nobody comes"? But it was a ton of people; we ran out of chairs).

* I got to sing. With Sooj and Betsy. In the pitch dark of an abandoned WWII fort. Like.. we stood around in the dark and made music, all of us. This wasn't an official concert thing, just a private thing. I.. don't even know how to describe it, except wow.

* Cat is gone off to NYC for the week. I'm missing her crazy-intensely. And so many things make me miss her more - reading LJ, listening to music, reading books.. argh.

* So, a while ago, in spring or summer, a group of us (atheorist, @sarah_hines and I, (joined later by Cat)) started an informal meetup called Self-Inflicted Data. Where we met each week, and just reported various numbers to each other that we were tracking. The number change from person to person and from project to project. They're things like - weight, hours of sleep, number of pomodoros done on a particular project, number of pages of thesis written, etc, etc. I'd like to do a whole post on this group, so I'm just putting this in as a reminder. But guys.. it has helped me SO much. It's a huge motivation for me, from week to week. (In fact, the only reason I got the time/effort together to write this post is cause I promised I'd write at least 1 post for this week).

* I have been THOROUGHLY enjoying Cat's Girl Unlocked / Unfuck My Life project. I have, of course, been inspired to put together my own (overly?) ambitious Beast Unfucks His Life project, starting now (while Cat is away).

* (related to above) I've started running again! I'm FINALLY over the insane months-long cold I've had all winter. I cannot express how grateful I am for that, or how tired I was of being sick.

I go to a lot of conventions. (Some years more than others, but at least half a dozen per year, usually). Being a devil OCD a creature of order, I start trying to organize them in my head, and rate them against each other.

I usually consider a con pretty successful (for me personally -- judging to see if it was successful for catvalente, career-wise, happens on different axes), if it supplies a certain critical mass of the following things:

* I get to meet up with the people I wanted to see (usually friends we only get to see at those cons, locals, etc)
* I like the dealer room / art show. I manage to find some interesting book/art loot to haul (though hopefully not overspend).
* How's the programming look? How many of the panels would I like to go to? (I don't actually attend that many, since my main goal at cons is to help out Cat. But I still look to see how many are tempting).
* If I do make it to a panel, how was it?
* I meet interesting new people, get exposed to new ideas, have revelations.

So we went to Boskone this past weekend. And I really liked it, it was interesting and fairly devoid of stress. A success, by my parameters above!

Thoughts form Boskone:

* yagathai is amazing and super helpful, and has great friends. Cat and I had to organize a surprise SWFA party on no notice (ie, it was a surprise to us, that we had to run it, not that it was a surprise party). Mike stepped in (and also had a posse of awesome BwB friends who pitched in) and helped us out -- we threw together a pretty great party, considering the time limitations.

* Got to meet up with omnia_mutantur briefly, and chat

* Met a new favorite artist! Coralynn Rowell of robotlovenoises.com -- she was doing cute robot sketches for $5 a pop. I picked up a print of that tree painting that's on the front page, plus a couple others, and we also won an art auction original piece from her. I think I know who I'd like to ask to design my next large tattoo!

* OMG got to see s00j and Betsy Tinney again! Super briefly -- we had panels and meetings to set up during most of their concerts, but still! I missed them so much. They're coming to stay at our house in a few days, though, and we're having a house concert on the island (which means you should come!). In fact, we'll see them tonight at another house concert at Kyth's house in Concord.

* LOVED the dealer room / art room. They put the two areas together, plus the con suite, plus the gaming area, all in a huge open space in the basement. Loved it! You walk down there, and it's such a feeling of richness/plenty, loot and food and art as far as the eye can see. Plus it makes for a central social point of the con, very helpful. I wish more cons would do something similar (I understand not all hotels / spaces support that).

* Went to an interesting panel (on Writing for MMOs), recommended Neal Stephenson's REAMDE novel to everybody.

* Got a chance to hang out and talk more with Charlie Stross and Ian Tregillis.

* Got to see Lawrence Schimel again (haven't seen him in years, since that Wiscon in like 06)!

Anyways, it was a great con (my first Boskone, actually). Still much to think about and process.

This weekend, I:

* Was still lightly sick (stupid cough won't go away). Plus, got bonus food poisoning, for the first time in years and years. What the hell! That never happens to me. Also, I don't hork easily! (Which is a bad thing, when you need to). Boooo.

* Started a D&D campaign with Cat! Somewhat on short notice, so I've been ingesting massive amounts of Eberron sourcebooks (plus refreshing D&D 3.5 rules in my head). (I'm DMing).
I'm super excited about it (plus, there's rumours that we're starting a D&D campaign with friends in town, in addition), and of course very nervous (I still don't have much experience DMing).
Have any of you guys played in the Eberron setting? What do you think of it? (I'm really digging it so far, it has a Final Fantasy 7 sort of feel to it).

Under Milk Wood, Hot Tub
dance (like you mean it)
For various ridiculous reasons (that I can't remember now, since sleep dep causes short term memory loss), I haven't been sleeping much this past week. (A trend that has, blessedly, ended, today, finally a sweet 8 hrs). Here's what I do remember, though:

* We set up our Xmas New-Year's tree last weekend! It's enormous and beautiful and lurks in the library, portending good things.

* Our performance of Under Milk Wood (by Dylan Thomas) went off smashingly, this past Saturday. Hey, our goal was "as long as we have more audience members than cast members, we win". But we filled up the (beautiful slightly creepy New England) church quite nicely! The play itself (a reader's theater type thing, no sets) went off as perfect as we'd want. (I was blind Captain Cat, plus did sound effects on the ipad). The audience seemed to love it -- though the poetry of the play can be pretty dense at times, they laughed at all the right places, and raved about it afterwards.

* The cast party afterwards was really fun. In our hustle to get everything ready for the play, we forgot about food (will definitely order pizza, next time), so everybody was starving, but it turned out ok, cause I made massive choc-chip/butterscotch waffles for everybody.

* More importantly, we beta tested our hot tub! The electrics have been finally hooked up on Friday, we filled it up that night and set it heating. One, I was really worried if it was level enough (we dug the pit, filled gravel and sand, and put in concrete slabs, and leveled everything, all by ourselves). But even completely filled, it was perfectly level from every angle, whew. I mixed up the chemicals from the manual, cursing the complexity and my misspent youth of almost-failing out of AP Chemistry.

Anyways, we tried it out Saturday night! The night was really cold, and the hot tub cover had frost on it, but the water was quite toasty, and we could look at the stars. I still need to tweak the pH levels, but it's a promising start.

* Cat's mom visited over the weekend, for the play, and is flying out today. One of the places we took her to dinner was Grace, a restaurant built inside of a gorgeous church. Wow, I really admire the design (and the drinks were fantastic).

Minor dinner revelations
Food <i>slays</i> me sometimes.

You ever notice how, you're just walking along, you come home from work, lame dogs want to be walked, blah, blah, and then you sit down to eat, and WHOOMF. The world just stops for a second, and the camera rotates around 360 degrees to highlight your bewilderment, and possibly green Matrix sigils start bleeding from the corners of the room, just to highlight that you're <i>on</i> to something.

<i>It's so fucking good.</i> 

And your brain stops for a bit, because this is a culinary equivalent of <i>a sudden punch in the face when discussing the weather at the watercooler</i>. But in a good way.

It's SO out of proportion to what you expected, to the average intensity of experience of a Monday night. And your language centers blank out, and meanwhile, the inner gremlin of your mind calls up internal tech support and asks "Help. What is going on here?" and tech support says "Uh.. we have no idea. Just reboot and give it 20 minutes."

So last night was like that. And I'm still not quite sure how to properly convey to you how good that pasta was.

It involved.. well, pasta. That's the easy bit. But also these /perfect/ slices of portobello mushrooms in Marsala wine sauce (with some vanilla balsamic vinegar, I think). 

ALSO braised beef. In PUMPKIN BEER. Like.. I'm not sure you understand. Our pumpkin beer is really good BY ITSELF. And then there was this beef braised in it, in the oven, for hours. 

And the pasta, and the beef, and the mushrooms, and the sauce, all combined into something utterly ridiculously good. And I was hungry, of course, sure. But.. yeah. Out of proportion.

Handsome Spiders
needs more robots
The other day, jwz posted about Disposable 3d-printed spiderbots.

My immediate thoughts were:

* Ha. This very much reminds me of Patrick Farley's "Spiders" comics (about the alt-history US invasion of Afganistan with spider robots).
Original wayback machine version
New version of the comic

* The article mentions that the spiders were largely manufactured using 3D printing (specifically, selective laser sintering (SLS)). The question immediately begs itself --
How far is this from the capabilities of the current open-source 3D printing projects (like the RepRap / Thingiverse stuff that exceptionshift and matt_arnold (?) plays around with?  If far, how expensive are SLS capable machines?

Also, I love how cheap spiderbots follow the Apple design aesthetic.