Over the past year, a fair amount has been written and tweeted about the need for "curated" crowdsourcing, a fancy way of saying that there needs to be management, methods and standards to guarantee the quality of crowd-powered outsourcing. An ambiguous crowd of various skill-levels can produce ambiguous results in much the same way that allowing everyone into a public pool virtually guarantees more splashing than swimming.
Finer Filtering: Employing an unfiltered crowd can be a bit of a crapshoot, yielding as much garbage as gold. We have found while managing projects on JobSpooler that curating a crowd involves both art and science. For example, good curation should include the ability to filter for credibility and competency. The most essential metric is reliability--a rating depicting the history of a worker to complete their task successfully. That said, a simple calculation of tasks attempted versus tasks completed doesn't tell the whole story. If the rating system isn't nuanced, it favors automatons that stick to doing simple tasks. A better algorithm balances the success rate with duration and difficulty.
Crowdslack: There is a significant difference between a worker who takes an assignment and then quickly abandons it for lack of time (or any other reason) compared to a worker that holds onto a task, letting it expire without submitting any work. The latter case negatively affects productivity-- another worker could have been completing the assignment and the worker ratings of the "slacker" should reflect this distinction. In the real world, there are more negative consequences when someone flakes out on a job rather than notifying an employer upfront that they aren't able to do it.
The skill scale: Beyond reliability ratings, specialized jobs benefit from specialized skills. Skill tests are useful tools but not always appropriate if the test takes longer to do than the assignment. If the barrier to entry is too draconian, many talented workers may decide that the juice is not worth the squeeze. We have found that skill tests are just one of multiple methods to find the cream of the crowd. Other times a simpler 'audition' job with a low barrier to entry will assist in developing a favorites list of people who will be invited back for more ambitious jobs not publicly listed on JobSpooler. This can also serve to train the crowd as to how you want material submitted and what your standards for approval are. A worker can learn a lot from a five minute task that gets rejected for good reason. They are more likely to lose patience with the job if they've worked for an hour and found out the work did not meet an obscure specification. JobSpooler also allows employers to send tasks back to workers for revisions if they are in the ballpark. This is efficient for productivity since it takes less time to make a small correction than for someone else to start from scratch. Revision mode also respects the time and effort of the freelancer, allowing them to take more pride in getting the job done right and encouraging them to take on more tasks.
A sidenote about skills: Education and experience can indicate aptitude but not outcome. Recently we had a domain-naming job for a large company that wanted to try tapping crowd-wisdom. The assignment was pretty straight-forward: come up with an unregistered domain that met a few specific criterion for length and content. We ran two identical jobs. One was open to all comers while the other, higher-paying, was sent out only to professional copywriters. Amongst several interesting findings, the most interesting result was that while the average submission from the copywriter group was better than the average submission by the crowd-at-large, the quantity of top-shelf suggestions were split equally between amateurs and pros. Though the testing sample was small, it does reinforce the notion that merit trumps pedigree.
Considering the Curator: Perhaps the most overlooked element of managed crowdsourcing rests not with the flock but with the shepherd. Obtaining good results depends largely on how well the job is described, how well it is divided and ultimately compensated. JobSpooler was designed to allow the employer to make adjustments while a job is in progress and it has been a helpful feature. Often, we believe a job is described clearly until we start getting results back and discover that instructions we thought were obvious were consistently misinterpreted. Other times, increasing the rate just a small amount or offering a bonus to those atop the leaderboard have made a big difference.
Employers taking their first flight into the moshpit would be wise to find a ready wrangler who can help navigate the teeming hordes. As some have discovered, labor Crowdsourcing gone wrong can become a tedious boondoggle; done right it is a boon to any organization.
