However, in my experience, solution-oriented problem solving is not all that common. Since it is desirable and rare, one might conclude that it must be very difficult. However, I tried to formulate some recommended best practices for it and realized that it’s not really difficult at all. The core of solution-oriented problem solving consists of one set of questions TO ask and another set of questions NOT to ask.

The four questions of solution-oriented problem solving

Ask these in order whenever any problem comes up.

1. What happened/is happening?
2. Why did it happen?
3. How do we fix it?
4. How do we prevent similar problems from happening in the future?

Questions that should NOT be asked

Just as importantly, solution-oriented problem-solving requires that certain types of questions NOT be asked. The following questions make people defensive, are not helpful, and get in the way of actually solving the problem. Patterns in individual performance deficiencies, if recurring, may come up as answers to #3 and #4, but all such deficiencies should be looked at in context: were they due to impossible assignments, situational failings such as poor communication, or truly due to deficiencies of the person in the role?

• • Whose fault is this?
• • Why did that person screw up?
• • Why is that person so incompetent?

So, I said to myself, how could we save it? It certainly is silly, in this day and ago of Amazon, to expect to make money as a bookstore. But could we save the spiritual space? Needless to say, I never came up with a solution, but the idea I was most fervent about involved making the store a non-profit space, welcome to all, making some money off of assorted sales but mainly supported by tax-deductible subscriptions. “Because plenty of people would be willing to spend $20 or so a month to ensure the continued existence of such a valuable space!” — I asserted passionately any number of times.

Fast-forward much of a year, and it’s Wikipedia’s fund-drive time. Now, Wikipedia is also a valuable space. It’s the science fiction dream of accurate information available instantly online, only better, because it’s not related to any government, it strives to be unbiased, and it is truly available to anyone. If I have any sort of question, scientific, cultural, historical, you name it, I know that they are the most likely place to have the answer — or at least a good place to start — from the actual supernova that inspired Flashforward to a list of TV superheroes (go Buffy!) to a list of open-source configuration management tools. So, yeah, it’s valuable. And I’ve donated before because I believe in their value. And then Jimmy Wales sent me an email asking if, this year, I’d be willing to subscribe to support Wikipedia….

And I asked myself, was my belief that people will subscribe to support something valuable just bullshit?

To save you all any worry, it didn’t take long for me to decide that NO, it WASN’T, and I am now a proud subscription supporter of Wikipedia. And yeah, you all should donate something too. Just imagine if you had to wade through cdc.gov and the sites of religious nutcases to find out details about the first cloned dog.

And maybe then you can help me come up with ideas to promote, and support, an equivalent of the Bodhi Tree…

As with most of his assertions, this will attract a lot of comments. This time I’ll contribute some of them, because that assertion is a big fat cream puff of a target. Here’s a revision that I believe to be more accurate:

Anxiety is nothing… but repeatedly re-experiencing failure in
advance hypothetically in your mind, which may allow you to take
actions that prevent failure in reality.

That doesn’t mean that worry is the only thing you should be doing, or that incessant worry doesn’t have its costs that can prevent you from ever succeeding. However, worry can help prevent failure. Never worrying may result in heedless, repeated, or even unrecoverable failure. There are certainly many other strategies for achieving one’s goals that are more focused on success that are also essential, and these strategies may be better in terms of the comfort of the actor, but in terms of meeting goals, worry may be part of a well-balanced portfolio of strategies.

In other words, just thinking positive and keeping at it isn’t a guarantee of success.

Two recent pop-culture items I read yesterday have some bearing on this:

1. The LA Times review of “The Runaways” describes an early scene in which Joan Jett badgers Kim Fowley and “persuades him to handle her band, which at the moment consists of Joan, her guitar and a vague idea.This didn’t cause me to think, “Look, see, anyone can succeed if they are just willing to be pushy enough.” Instead, what I thought was “OF COURSE it succeeded. She’s JOAN JETT! Her talent is so great that with any drive at all it was inevitable that something would work out.”
2. The flip side of this is the “redundant Web site listing movie nude scenes” that Seth Rogen’s character was working on in “Knocked Up”, which received a passing reference in a Slate discussion of “Greenberg”. THAT effort was a pathetic failure. Not due to lack of enthusiasm, however. One of the more painful scenes in that movie is where Katherine Heigl tries to help Seth Rogen work on the site, and gets very excited about spotting some extreme nudity. It’s the same passion that exists in every startup, whether it succeeds or fails. It will not (and does not) help this startup at all. If the idea is dumb and derivative, no amount of passion will make it a success.

So, the moral seems to be: you have to work hard and have drive, but it needs to be of service of something that is actually worthwhile, a good idea or something you are actually good at. You can have all the passion and positive thinking in the world, but if you keep applying it to worthless things you will get no results. Much the same way that worrying, in rehearsing failure, has a risk of training your brain to expect it, enthusiastically doing dumb things without any thought, analysis, or attempt at improvement (activities that often are accompanied by, and may be improved by, worry) definitely trains you, even more so, to be very good at doing dumb, useless things, until eventually you might never be able to learn how to do smart, useful things.

So, what’s the answer? Figure out something to do in which I’m Joan Jett, I guess. It’s OK to fail and it’s necessary to do a lot of work, but it needs to be in service of something with potential, not just any old idea that comes my way.

Postscript: Yet another example to contemplate is Alex Chilton, who died on Wednesday of a heart attack. In the 1970’s he made some brilliant music in the band Big Star that was ahead of its time and completely failed to receive recognition. Unlike some artists, he didn’t have to die to get recognized: by the mid-80’s his music was being revered by the next generation of musicians and determinedly hip college students (including me) throughout the country. However, the bitterness of the earlier failure never seemed to leave him. He never did anything as brilliant again, and he had a distinctly ambivalent relationship with both the music of Big Star and with success itself. While he definitely made enough money to live as a musician, he was in no way even moderately well-off, and while he had a firm belief that he was being true to himself always, it was never clear that he was happy.

What does this have to do with worry? Well, many Big Star songs are overwhelmed with worry and pain: there are the obvious later ones such as “Holocaust” (about his mother’s death) and “Big Black Car” (featuring the achingly ironic lines “Nothing can hurt me/Nothing can touch me”), as well as earlier songs such as “What’s Going Ahn” (“I resigned everyone, ever since I was young”) and “Back of a Car” (“Thinking ’bout what to say/Can’t find a lot”). His later work was completely lacking in worry: there were snarky blues originals and cheesy Italian pop song covers, all devoid of concern (or much emotion at all). He had stopped worrying, and as a consequence, he had stopped making music that mattered.

Postscript #2: Much thanks to Arun Srinivasan, who called my attention to the original Seth Godin post and whose email thread with me about it inspired much of this post.

The study in question was fairly simple: people were asked to memorize either a 2-digit number or a 7-digit number, and then walked down a hall to another room where their memory of the number would be tested. On the walk, they passed a refreshment table that offered either fruit salad or chocolate cake. The point of the test was to see how the complexity of the memorization task would affect the subjects’ willpower (the proxy for which was whether they chose the healthy fruit salad snack or the unhealthy chocolate cake). 59% of the 7-digit subjects chose the cake, as opposed to only 37% of the 2-digit subjects. Having to memorize 7 digits as opposed to 2 significantly diminished one’s willpower/ability to resist self-indulgent choices.

The theory behind the experiment is that the frontal cortex is responsible for both memorization and willpower, and the frontal cortex has a very limited capacity (other studies have shown it to be 7±2 on average). So if all of your frontal cortex capacity is being used up holding memorized digits, there is little left over to resist the temptation of chocolate cake. So far, so interesting.

However, the obvious application to this study, to me at least, is to strategies for getting things done. Accomplishing hard, complex, energy-intensive tasks, such as delivering a software product, studying for finals, or starting a company requires a lot of willpower: you have to do a lot of things, all of which require energy and many of which aren’t fun, in a  timely manner, even when you’re discouraged because something didn’t work out, even when you just want to hang out with friends, surf the web, or go to sleep. Accomplishing these tasks also requires knowing what subtasks you are working on: you have to have a list of high-priority tasks that you keep in your mind at all times. Thus these tasks require both willpower and memorization.

If your list of high-priority tasks contains 7 or more tasks, then you’ve used up all of your frontal cortex capacity on the list and have none left over for the willpower you will need to execute the list. In fact, since you’re going to need a lot of willpower, it’s probably better to allocate at least 50% of your cortex capacity for that. You can’t allocate all of it because you do need to keep focused on some high-priority items at all time. Focusing on just one is risky, as you may get stuck on it for a while and if you can’t immediately switch to another one, you’ll be less efficient. Each additional item you can add to your priority list decreases the risk of getting stuck: you want to have your list be as long as possible while still leaving space for willpower.

Given all this, the optimal strategy is clear: have a high-priority list of size 3. That’s the maximum size that will still leave at least 50% of the average frontal cortex capacity available for willpower.

So, if you’re trying to do something hard and complex, something that has many moving parts and requires lots of willpower, you should make a list of your top three tasks, focus on those, and save the rest of your cortex to provide the willpower. If you have fewer than 3 items you run a higher risk of getting stuck; if you have more, you are likely to run out of willpower before you get any of them done. You need a “must-do” list so that you get things done, but a list of 7 “must-do” things is just too hard for your brain to handle. A list of three, however, leaves you plenty of brainspace for willpower, and makes it (almost) easy.

The overwhelming usefulness of named_scope is common knowledge in the Rails community. Named scopes are so useful that you want to use them everywhere, and that’s when you begin to encounter one of their frustrations: while they are quite easy to write and use, they are not always so easy to re-use.

Most of the time, reuse is provided by the (also overwhelmingly useful) chaining feature of named scopes. If you want to use a named scope in an association (has_one, has_many, or belongs_to), you can simply define the association and then chain the scope to it, optionally defining a method to access it if you will be using it often. For example, if you have a named scope published_in_month on the Post class that finds all posts published in a given month, and the User class has_many posts, then the code to find all posts for a given user written in a given month is simply

user.posts.published_in_month(month)

If you want to combine two named scopes together, again, you can just chain them (again optionally defining a new method — in this case a class method). If Post has another named scope with_comments that returns posts with comments, then the following code finds all posts published in a given month that have comments:

Post.published_in_month(month).with_comments

So far, so very good. But what if you want to define a named scope that reuses a named scope not for the given class, but for one of its associations? To be more precise, what if you have a named scope published_in_month on the Post class that finds all posts published in a given month, and the User class has_many posts, and you want to write a named scope on User to find all users who have posts published in a given month? Sure, you can do

(Post.published_in_month(month).collect {|post| post.user}).uniq

but that’s a little harder to read — and is not itself a named scope, and thus doesn’t get executed as a single query and doesn’t have the nice features of named scopes such as chaining.

This problem came up emphatically in some code I was writing recently. In our system, many objects belongs to a user, a user has many addresses that include a travel radius, and an address may be within the range of a given lat/lng point. The SQL to calculate whether an address is in-range is pretty messy, as you’d expect, and it was showing up all over the code. I refactored to put it in a named scope in_range on Address, but what I really needed was a named scope to find all objects whose addresses (through their user) were within range of a given specific address, and there wasn’t a clean way to reuse the named scope on Address. And from that need was born this plugin, associated_named_scope.

Note: While looking around online to see if there was already such a thing available, I discovered that this request had been made before [discussion now moved here]. This plugin doesn’t exactly provide the syntax as requested there, but it does provide the same functionality.

The plugin

associated_named_scope is a Rails plugin that, quite simply, allows (re)using named scopes on child associations to define named scopes in the parent classes. It is available at

http://github.com/Mechaferret/associated_named_scope

Usage

associated_named_scope adds a new option :association to named_scope, which expects two suboptions:

• :source — the source association for the child named scope. You can specify nested associations using standard hash syntax (e.g., {:user=>:addresses} for the addresses of the user of the current model).
• :scope — the named scope itself. You can specify arguments to the scope by using an array [:scope_name, arg1, ...]

These scopes chain and in every way behave just like normal named_scopes.

Examples

Here’s a simple example to find all users who have posts published in a given month:

class Post < ActiveRecord::Base
  belongs_to :user
  named_scope :published_in_month, lambda{|month|
    {:conditions=>["month(posts.publication_date)=?", month]}}
end

class User < ActiveRecord::Base
  has_many :posts
  named_scope :has_posts_for_month, lambda{|month|
    {:association=>{:source=>:posts, :scope=>[:published_in_month, month]}}
  }
end

And here’s a more complex example, using a nested association:

class Address < ActiveRecord::Base
  belongs_to :addressable, :polymorphic => true

  named_scope :in_range, lambda {|lat, lng, zip_code|
    {:conditions =>
      "(
        (POW(addresses.radius,2) >=
          (POW(69.1*(#{lat}-addresses.lat),2) +
            POW(69.1*(#{lng}-addresses.lng) * cos(#{lat} / 57.3), 2)
          )
        )
        OR (addresses.zip = '#{zip_code}')
      )"
    }}
end

class User < ActiveRecord::Base
  has_one :object_with_user
  has_many :addresses, :as => :addressable
end

class ObjectWithUser < ActiveRecord::Base
  belongs_to :user
  named_scope :within_range, lambda{|address|
   {:association=>{:source=>{:user=>:addresses},
     :scope=>[:in_range, address.lat, address.lng, address.zip]}}}
end

Some background: I’m writing code that needed to create an instance of NewModel for every existing instance of OldModel that was in one particular state (where OldModel implemented acts_as_state_machine). I check the code: sure enough, there are no named scopes to get only objects in that state. I could just write code to cover this case, but given how easy it should be to generate a named scope for each state, that seems like the wrong answer. Thus, I begin the following dialog:

Me: Does anyone know if acts_as_state_machine can generate named scopes for each state?
Other developer: No, I don’t think it does.
OD: That sure would be useful though.
M: And pretty easy to write…. But before I do that, let me check and see if anyone has updated it.
Other other developer: I made some modifications to acts_as_state_machine for that model, so that it could transition to multiple states.
M: (looking at the code in lib) Cool! It’s basically an implementation of a nondeterministic state machine.
OOD: I didn’t do it myself, I got it here. (IM’s me a link to http://justbarebones.blogspot.com/2007/11/actsasstatemachine-enhancements.html.)
M: Now I’m definitely looking to see what the latest acts_as_state_machine is.

To github I go, where I find AASM at http://github.com/rubyist/aasm/. Documentation is weak (I may try to contribute to that), but the code looks promising and has the multiple-state transitions OOD was using, as well as the ability to define specific actions on initial_state/final_state/transition triads. I download it, do a quick check to verify that it does in fact automatically generate named scopes for each state, and pop it in. A few syntax adjustments (everything is now prefixed with “aasm_”) and my new code works great.

Some of the old code, however, has started throwing errors. Why? Because it’s calling invalid state transitions! Apparently the old acts_as_state_machine just silently ignored them, while this one complains. Which is good, because now I can see that our mandatory email notifications weren’t getting generated sometimes, specifically in the cases in which the object was in an invalid state for the event that should generate them.

The new AASM allows transitions from a state to itself, and the aforementioned specific actions on initial_state/final_state/transition triads, though, so fixing the newly-discovered bug is as easy as defining a new transition and specific action. And now everything works, correctly, and AASM has helped me fix a bug I didn’t even know we had.

Summary of advantages of AASM over old acts_as_state_machine that I discovered in about half an hour of usage:

1. AASM generates named scopes for each state.
2. AASM allows nondeterministic transitions: that is, an object can transition from one state to a list of other states on an event, with the actual final state determined at runtime.
3. AASM allows specification of actions on initial_state/final_state/transition triads.
4. AASM actually throws errors on invalid state transitions, thus making debugging easier.

I’m sure there any many more, but that was more than enough to make my coding day much easier. And the actual code activity required to do the update is, as noted above, minimal.

I’m not sure if the authors think that the new version is completely ready for prime time. But in my experience it’s so much better than the old one (and also hasn’t displayed any bugginess), so I don’t think you need to wait.

There is one way, however, in which web applications offers a distinct advantage over other types of applications for agile development: measurement. Measurement is a key part of the feedback cycle for lean processes, but finding the right metrics to optimize value for your customers can be challenging. Web applications, however, can give you incredible amounts of data about what your users actually do: how they reach your site, what they do when they get there, whether they continue an activity or drop off. Everything they do is done through the web application, so everything they do can be retained for further study.

Given this wealth of data, it becomes much easier than in other forms of development to form and test hypotheses about what your users want. You can do “A/B” testing, randomly offering them different implementations of the same feature, and seeing which one does better in achieving your desired goal. Or you can do simple analytics: change a feature, and see how much your desired goal changes. You can also easily collect application-specific data, e.g., which products seem to get purchased together, what attributes of a customer make them likely to order the high-end version of a product, the relationship between followup contact with a customer and repeat purchases.

You get to use your entire user base to estimate the value of a given feature for your desired target audience. Because these users are part of your target audience and because of the effect of the “wisdom of crowds”, their collective evaluation will be far better than the a priori evaluation of your product team.

Of course, I’m glossing over the difficulty of figuring out in sufficient detail what your goals are and how to measure them. But often the goal is obvious and the results of the basic data can be compelling. We recently experienced this on a web application I’m currently working on. We were having a debate about exactly how much information to request on an initial inquiry form. Some of the team wanted to request some information on the form that would be of use for a possible future product (but which also could be collected elsewhere), while others wanted to keep the form’s size to a minimum. The application on production at the time of the debate was collecting the data. In the next release, we stopped collecting the data, and the inquiry completion rate immediately doubled and stayed there. After that, there was no more need for debate about collecting the extra data in the form.

But how do you decide what constitutes “wasted effort that does not add value”? The accepted non-software touchstone for agile development, “lean manufacturing”, defines “wasted effort” as “anything that does not create value for the end customer”. But how do you know what the end customer values?

Obviously (you say), you figure it out from the “earlier feedback”.

Yeah? How?

If your application is internal and your user base is well-known, collecting and acting on feedback is easy. You have your designated “onsite customer” and maybe a few others use the latest iteration of the product and tell you what they think. But what should you do if your application is a consumer-facing web application with anonymous customers, most of whom are far more likely to go away if they are not receiving value than to fill out a feedback form?

Yes, that’s when it gets interesting.

One clear way not to go is to assume that the internal product team, possibly augmented with IA/UX and usability experts, can figure out what your target users want by applying “usability standards” and “domain knowledge”. While all of these activities have their uses, the risk is high that the initial concept will not be what the users want. This will result in the following cycle:

1. Figure out concept of product.
2. Design product in conjunction with IA/UX experts, taking care that each page is easy-to-use and well-designed.
3. Take at least 2 months to build the product because each page has to be perfectly designed.
4. Since you’re trying to be “agile”, don’t take more than two months; put the team on death march to deliver the completed product on schedule.
5. Discover that the users don’t seem to value the new product enough, as measured by the lack of usage or revenue.
6. Repeat.

This is a loop, certainly, but it is hardly a feedback loop. Feedback is not obtained until step (5), and it’s not very precise: it’s either “yay” (we have users and are making money) or “boo” (we’re not), which gives you very little to go on for the next iteration.

Four of those cycles can easily take up a year without adding any value for the end customer. By then, you might as well have just used the year to develop the product using a waterfall development process: you’d have created just as much value for the user, and your team wouldn’t be frazzled with impossible deadlines and frustrated by doing the same thing over and over. In fact, often this process does devolve into a form of waterfall development: the coding process may be agile, but the visual design process of

Product feature–>wireframe–>Photoshop mockup–>approved page

is done strictly in sequence, i.e., waterfall.

Happily, this is not a problem without known solutions. The idea of “lean startups”, incorporating “customer development”, has some excellent strategies for avoiding the above trap: don’t make product decisions in a vacuum, develop customers in parallel with developing features, find early adopters for feedback and enlist them as your allies, make your first release as early as possible, make subsequent incremental releases near-continuously.

The lean startup method gives you real feedback for your web application as early as possible. It also has the nice side benefit of creating a cadre of enthusiastic evangelists for your product as it works. That is a feedback loop well worth implementing.

¹Assuming that the requirements are not well-understood, that costs of changing the system after release are not unacceptable, you know the drill….

http://github.com/Mechaferret/sql_crypt

This plugin provides two-way encryption of ActiveRecord attributes in a database using database encryption functions, instead of encrypting in code. This is useful when you are encrypting large numbers of model objects at once, for which Ruby encryption can be too slow, or if other applications written in languages besides Ruby are accessing the encrypted attributes, which would require keeping multiple implementations of the encryption algorithm synchronized.

Usage

Using sql_crypt is very simple. The following example will encrypt the “balance” attribute of an Account model:

class Account < ActiveRecord::Base
  sql_encrypted :balance, :key => 'my_secret_key'

You can specify multiple fields on the same line if they are using the same key; use separate lines for different keys, e.g.

  sql_encrypted :cc_number, :ssn, :key => 'fIn^nce-dan{}er'
  sql_encrypted :password, :key => 'password'

will encrypt “cc_number” and “ssn” using the key ‘fIn^nce-dan{}er’ and “password” using the key ‘password’.

History

I was working on a payroll application, which obviously needed to have two-way encryption for many different fields: salary, bonuses, reimbursements. The application has a summary page that would fetch data for all employees and display it on a single page. Originally, I implemented the encryption in Ruby using the crypt gem, but with over 100 employees the summary page was getting very slow to load. Thus, I switched to using MySQL’s aes_encrypt/aes_decrypt, which fixed that problem. Since I had many different attributes that needed encryption, it made sense to create a class method to specify that an attribute would be encrypted, and thus the code for sql_crypt was initially written.

Later, we had two separate applications that were accessing two-way encrypted fields using both Rails and PHP, and we were having some trouble getting the MD5 implementations to match. Reusing the sql_crypt code from the payroll app was the obvious solution. Once that was done, the code was general enough that, with a few tweaks, it seemed like it might be made useful for the general community. I made the tweaks, and sql_crypt is now publicly available.

All comments and suggestions are welcome.

I’m not talking about that kind of meeting.

Nor am I talking about all-hands meetings in which a bunch of people that work at geographically dispersed locations get together to meet in person and get to know each other (aka “culture-building” or “a party”) or the all-hands working group meeting, in which a team gets together offsite to do some focused work (aka “work, but in a more fun location”).

No, I am talking about the every-quarter-or-so, multi-hour to multi-day meeting with a detailed agenda and attendees sitting in chairs desperately trying not to fall asleep, the one where the people “up top” get together with the rank and file to “explain everything that’s going on right now”. The sort of meetings that truly give meetings a bad name and lead to manifestos like this one.

The main problem with these meetings is that they should never be necessary. The large-scale direction and upcoming products that are discussed in these meetings should be common knowledge to the people who work for you, part of their DNA, inhaled along with the air. I ran an organization of over 25 people for 3 years, and I never once held an all-hands meeting, and yet everyone was quite aware of where the company was going, and what the new products were, and which major customers and markets we were currently focusing on. They knew it because every day, in various scrums, I would mention that “this feature is for company X” or “we might be on a tight deadline next month because we need to release for trade show Y”. They knew it because I met with my direct reports once a week and planned out what we were doing, and then they told their reports, and so on. They knew it because any time someone was assigned a task, the explanation for the corporate need for the task would accompany the assignment, unless the person already knew it. It’s not that hard.

If you feel the need to have an all-hands meeting to explain “what’s going on”, you aren’t communicating adequately on a day-to-day basis. Thus your team is most likely confused and/or apathetic, and is definitely not working together with you towards your common goals, because they don’t know what the goals are. It is the absolute antithesis of the recipe for productive, high-performance teams.

The other possible explanation for big-bang all-hands meetings is that the team does know what you’re going to tell them, but you as a manager like to stand up in front of everyone and tell them what’s going on because you get something out of it, because it reinforces the management hierarchy and your position in it, because you feel like it earns you respect. In that case, instead of failure to communicate, what you have is straight-up poor management. Standing in front of a room full of people telling them things they already know, while they look bored and try not to roll their eyes, does nothing but reinforce belief in the Peter Principle.

Either way, when corporate management feels the need for periodic, large-scale all-hands meetings, the first question should be “In what way is company communication or management failing that there seems to be a need to hold this meeting?” One might consider having the team all get together for “work, but in a more fun location” or even “a party” to try to answer that question.