Crypto incentives for code reviews
Introduction
There is little debate on the matter, the Blockchain buzzword has stolen a chunk of the tech limelight from AI and machine learning in the past 6 months. A key driver of its success has been the recent boom in crypto currencies, where tech companies are able to source vast amounts of public investment via ICOs (Initial Coin Offerings).
The validity of cryptocurrencies as a store of value in the 21st century is very much up for economic discussion. However, the underlying blockchain technology that the majority of these coins employ seems to provide more tangible value to the tech community. That is; trust, decentralisation and security.
For an online supermarket like Picnic, trust and security are two highly coveted properties in supply chain management. The development of Blockchain technologies and protocols for such purposes have indeed caught our attention.
In addition to such development, we decided to turn the Blockchain to a novel task: our development process.
Motivation
At Picnic, our development process revolves around code reviews. We believe it to be a key influencer of code quality and knowledge sharing across the variety of systems that make up the Picnic tech ecosystem.
While we can promote knowledge sharing to a certain extent; via means of tech talks and impromptu coffee discussions, at the end of the day a true understanding of backend systems lies within the code.
The same can be said for the quality of our code. Having multiple keen eyes scan through your lines of code helps us all hold ourselves to a high development standard. It aids in catching bugs early, preventing critical issues, and it keeps the code clean, readable and architecturally sound.
We strive to increase the bus factor within the backend team, and share ownership of code and systems amongst us all. But how do you go about quantifying and incentivising this practice without using physical buses?
Idea
In an ideal world, everyone would be altruistic about code reviews. However we all have stakeholders to answer to and our time isn’t infinite, so we have to draw a line somewhere. On that note, what we promote in the tech team is simple:
Give back what you take. Better still, give back more if you can.
As backend developers at Picnic, we have a general policy that your code is reviewed by at least two other developers and preferably more for large impactful features and changes. Upon completing a ticket and submitting a PR (pull request) you are contracting two or more developers to spend time and effort reviewing your code. As developers ourselves, we know that this isn’t free, and even though we make our best efforts to create good quality PRs to reduce the overhead placed on our fellow colleagues, they don’t know what they are looking for until they find it.
To quantify the discrepancy between PRs and reviews, we make a parallel to the upload/download ratio metric implemented in a number of popular peer-to-peer file sharing protocols such as BitTorrent. The availability of the torrent being shared on the network is dependant upon people uploading at least what they download. The health of the network diminishes when:
In the realm of code reviews, we simply quantify our sharing metric as:
Given that we strive for two reviewers on every pull request, we should strive for a review network health of:
With this in mind, how do we encourage and incentivise developers to aim for this golden ratio, to keep the network healthy and keep developers happy?
Reviewer Leaderboard
To try and address this issue, Nathan Kooij and myself developed a leaderboard and fun incentive scheme based on this simple reviewing metric. Developers are ranked based on their reviewing ratio over all of our GitHub projects. For every hour that a developer has a ratio above one (and thus at least helping retain the network health) they receive a proportionate chunk of Picnic Coin in return. Effectively:
Developers mine coins based on their contributions to code reviews.
Of course, we should all be striving for a ratio of 2.0 or more, myself included. As you can see, we have some work to do to reach our desired network health.
The coin can be used to buy badges to flaunt your position on the leaderboard, and later used to organise team outings and other events.
We wanted to be sure to keep this a light-hearted yet informative affair, and to avoid shaming of colleagues we anonymise developers with a ratio below one. They can reveal their names by signing in using their GitHub account.
We understand that there can be negative implications with the introduction of gamification in the workplace. We aim to monitor the usage of this leaderboard carefully within our team and hope only to drive productivity and to avoid penalisation.
Gathering data from GitHub
The backend is powered by a lightweight Python application using the PyGithub API. We fetch pull requests from the Github API at a fixed rate and reduce them to our metrics which we serve to a Vue.js frontend app:
We have to filter the data where GitHub marks developers as reviewers on their own pull requests, and sidestep 0 division meaning that the number of authored pull requests is bounded below by 1.
Private Ethereum blockchain
Given our up to date code reviewing metrics we can use this to pay out our hard working developers in Picnic Coin.
Picnic Coin is powered by the popular contract based cryptocurrency Ethereum. We booted up a private Geth Ethereum node (a Go powered command line interface for Ethereum nodes), and initialised it with what is referred to as a genesis block. This is simply a single starting point for the blockchain to form all other transactions from. Think of it as the ground truth for the ledger. We give our main address a large sum of Picnic Coins in order to pay out our developers in the future:
and we initialise the node with the following command:
Once initialised we can run our Geth node with a specific configuration. We indicate to Geth that this is a private network, to conduct mining on a single thread (required in order to verify transactions) and to expose the RPC (Remote Procedure Call) interface so that our backend app can issue transactions directly to the node:
Paying developers
With our Ethereum node up and running, we can now issue transactions from our backend Python application based on our reviewing ratios.
Using the Ethereum web3 interface we can create accounts and generate wallet addresses for developers, and then proceed to pay them at a given going rate. We can fix the payment rate to any factor of wei (being Ethereum’s smallest denomination).
We then make the payment from the main account (etherbase), and we schedule this whole procedure at a rate of once an hour.
We expose the wallet addresses, account balances, ratios, and developer information to our frontend app.
Our developers can use an Ethereum wallet, such as Mist, and point the RPC requests to our private node. For example:
By doing this developers can manage their money, whether that be spending it on incentives or by donating it to their colleagues.
Developers can even deploy their own Picnic Coin Ethereum nodes to decentralise the consensus of the blockchain, thus better ensuring that their coins aren’t being tampered with.
What’s to come?
Open source project
We plan to open source this project, with a full setup guide and Docker files so that you can get your development team set up ASAP with your own coin! Keep an eye on our GitHub profile and this blog for further developments.
Analysis
What’s all this worth if we don’t gain anything from it? This is the next big question for this project. While it’s initially a fun concept, what we truly hope to gain from this leaderboard is an increased effort made towards reviewing code. We will be measuring the impact on the average review ratio for all developers since the introduction of this dashboard, and we will be reporting the findings in the next blog post.
We hope to then experiment with a number of other metrics, utilising PR metadata such as LoC, number of comments and time to review, in order to see how we can best incentivise developers to achieve desired reviewing productivity.
Ethereum Smart Contracts
For further integrations with Ethereum we aim to introduce the use of smart contracts into the Picnic Coin framework. We believe that the PR pattern follows that of the Ethereum smart contracts pattern. When a developer creates a PR and assigns reviewers, they are in effect issuing a contract to those reviewers to review the code. The Ethereum smart contracts framework allows us to write small programs (in Solidity) that can verify this transaction, acting as an escrow transaction service. On completion of a review the reviewer can then be paid the amount agreed upon in the contract. Otherwise if and when the contract expires (e.g. if the reviewer didn’t find the time to submit a review), the coin returns back to the author of the PR.
Summary
At Picnic we have gamified code reviews. We have devised a novel solution to address the inbalance in code production versus code reviewal in pursuit of knowledge sharing and a fairer code review process. In conjunction with this, we have also devised a unique incentivisation program via means of cryptocurrency compensation. In short:
We pay developers in Picnic Coin when they review more code than they produce.
We expect to make further improvements to the manner in which we rank and incentivise developers based on their code review contributions, so watch this space.
Update
Whilst a fun experiment to get familiar with cryptocurrencies and blockchain technologies, the project was decommissioned internally and healthy code review ratios are now pursued via other means.
