Category Archives: Bitcoin Tech

Proof of Solvency is a Big Deal


Leave it to programmers to look at a catastrophic $350M financial-company collapse, and immediately think:

“Why not just code that sort of inelegance out of the economy?”

That’s exactly what’s happening in response to the recent implosion of early bitcoin-exchange MtGox. On the day MtGox filed for bankruptcy protection, CoinSetter, a bitcoin trading-market startup based in NYC, began a public process to determine how to do cryptographic proof-of-solvency.

This idea has profound implications for all financial institutions. Cryptographic proof of solvency refers to a way to publicly prove, beyond any possible doubt, that an organization’s finances are in order. The idea is essentially to link all deposits/liabilities of an organization and use cryptography to prove that they sum to a certain amount. The company can then use the public bitcoin blockchain to provably demonstrate control of at least that quantity of funds. Critically, this proof would not rely on the company’s own statements, or statements by its auditors, but on cryptography; in other words, the laws of mathematics. As we’ve seen so many times before (Enron, Worldcom, MFGlobal, Madoff), it’s probably not a good idea to unilaterally trust what companies, or their auditors, say.

Bitcoin gives the world completely new financial possibilities through the combination of the Blockchain (bitcoin’s public ledger) and cryptography. Proof-of-solvency is just the obvious first-step in giving financial companies the ability to provide customers with unprecedented transparency and control of funds. This has very large potential benefits to society, given the financial-system costs associated with insurance, fraud prevention, auditing, etc. Not to mention multi-trillion dollar public bailouts when those measures fail to be enough anyway.

Do Bitcoiners Want a Bailout? No.


MtGox filed for bankruptcy protection today, and in so doing, disclosed that it has 127,000 customers who are owed on average the equivalent of $3500 each. One might expect this sort of thing to trigger angry cries from customers for government intervention and/or bail-outs.

Indeed, some do think this is the case:


While that tweet may have been mostly in jest, it did strike me that I’ve noticed ZERO calls for government intervention from MtGox customers. People are obviously angry; mostly at MtGox management, and somewhat at themselves. The various calls for action are mostly calls for new, better-run, more transparent businesses, better redundant storage of funds, and development of decentralized exchanges that cannot become insolvent. In short, even at the height of customer shock and anger, the calls from within the community are mostly for a change in consumer behavior, and development of better technology. Bailouts are not considered a desired or long-term viable solution.

On one hand, this doesn’t come as much of a surprise, with bitcoin’s roots and early userbase coming from a libertarian ethos. But usually when people lose money, real and/or irrational feelings come out. The lack of calls for intervention shows a deep belief in bitcoin’s potential to serve the greater good without explicit central management. Here are some quotes from the “Official Money Lost in Mt. GOX Thread”:

“417.8923 BTC…
Stupid me… Was going to withdraw and move elsewhere, but the price kept falling so… Greed.”

“8 BTC (0 EUR/USD)
I bought the BTC less than 24 hours before the trading stopped…
I don’t even regret it. Yes, I am a little said [sic], but I was aware what I am doing and I would do this again.”

“Looks like 38BTC gone. I knew better but trading was fun.”

“$0 and $0 BTC. The only person I trust with my private keys is myself.”

“2.08 BTC. 274 USD… Should’ve cashed out earlier, but I got greedy.”

“About 4k in USD…I should have seen the writing on the wall and converted back to BTC and pulled it.”

“they better go bankrupt and stay bankrupt, for bitcoin”

“I pretty much have my life savings on Gox/MtGox whatever. When I first joined gox and the bitcoin community as a whole back in 2011 I had £1000 to my name.

I took a lot of huge gambles over the last 2 years and in my opinion did fairly well in fact I had over 17k euros by the “end” sitting on gox. Not counting the money I withdrew to live on.

Easy come easy go and with the risks I was taking along the way this was always a very real possibility.

Bitcoin was still a great experience for me and I’m glad I was in for the ride (yes I’m still a believer).

I wouldn’t want a single donation from anyone and would refuse it respectfully as I was a risk taker from the get go Grin

I think the best thing this community can do is offer support in a form that would potentially prevent the very real chance of suicides.
Its ow so easy to forget there are real people behind the curtain of the internet and many of them are hurting and suffering.”

The above quotes are not cherry-picked. At 11-pages long, with some people claiming millions-of-dollars equivalent in losses, there’s not a single call for intervention in that thread. Bitcoin enthusiasts are a consistent bunch.

What MtGox Did Wrong


tl;dr: Everything.

A friend emailed me yesterday and asked “Why aren’t others vulnerable?” That’s a good question. For those of us who pay attention to bitcoin every day, it’s been clear for months (even years for some) that MtGox was a unique risk in the bitcoin ecosystem. It’s always been considered unwise to leave significant funds in *any* exchange for very long, but starting in summer 2013, it started looking downright insane to keep money in Gox.

The problems arguably started much earlier. A timeline of MtGox’s troubles:

May 20th, 2011 – Files incorrect banking application.

MtGox CEO Mark Karpeles opens a business banking account at Wells Fargo, and fails to declare MtGox as a “Money Transmitting Company”.

June 19th, 2011 – Customer database hacked. Market compromised.

MtGox gets hacked, and a single massive sell order is executed, causing the price to drop to pennies within minutes. I had the fortune of watching this flash-crash in real time; a fascinating lesson in market liquidity. MtGox’s failure here was tangential: as a result of this incident, it became clear that MtGox was hashing their customers’ passwords with MD5, a hashing algorithm long considered inappropriate for modern use by even novice security consultants. This was a very telling early insight into MtGox’s security and technology practices.

April, 11-12, 2013 – Trading halted due to bad technology.

MtGox suspends trading, calling it a necessary “market cooldown”. In reality, it was due to their inability to mitigate DDoS attacks and/or handle high-load on their systems due to high bitcoin trading volume. Either way, MtGox was a very profitable business and had months of warning (years, really) to realize they needed to upgrade their systems.

The trading halt on MtGox was the trigger for the end of the massive Spring 2013 bitcoin bull market. Prices crashed from over $200 to $50 in one day. This was the 2nd time failures at MtGox caused a market panic.

May 2nd, 2013 – Bad deal with CoinLab results in lawsuit

MtGox’s alleged failure to honor the terms of their merger deal with then pseudo-exchange CoinLab (now effectively defunct), is aggressively terminated by CoinLab with a $75M lawsuit. To be fair, it’s unclear who was at fault, but it’s likely both parties made serious faulty business decisions.

May 15th, 2013 – DHS seizes $5.5M from MtGox.

MtGox’s real trouble begins. The US Department of Homeland Security seizes MtGox’s Dwolla account, apparently containing $5.5M in funds. This is a direct result of Karpeles decision in 2011 to not check the “Money Transmitting Business” box on his Wells Fargo banking forms.

While clearly an egregious error, especially after FinCEN‘s guidance, to be fair, bitcoin was essentially a toy until 2013. Few took it seriously, and in that context, it was easy for many non-diligent early bitcoin business operators to dismiss existing money services regulations as not applicable.

June, July 2013 – Dollar withdrawals restricted.

MtGox suspended US dollar withdrawals on June 20th, and resumed them on July 4th. Unfortunately, despite the resumption of withdrawal processing, customers were unable to get funds out in a reasonable timeframe. Withdrawals usually took in excess of 4 weeks to complete. Rumor has it that MtGox’s tenuous banking partnerships (or the DHS) were imposing wire-transfer limitations on the company.

If the prior incidents were not sufficient warning, this was the huge red-flag. Naturally, this was also the point at which the price of bitcoin on MtGox started to diverge from the price on other exchanges. Due to the dollar withdrawal issues, traders had to buy bitcoin and transfer it out in order to withdraw funds from MtGox in a timely fashion. The MtGox price therefore started to steadily trade 10% (or more) higher than increasingly popular exchanges Bitstamp, Coinbase, and BTC China.

Many in the bitcoin community began more vocally advising traders to retain control of their own funds, and to specifically remove their funds from MtGox. The writing was on the wall.

February, 7-10 2014 – Bitcoin withdrawals suspended.

MtGox suspends bitcoin withdrawals, citing a known-since-2011 issue in the bitcoin protocol called “transaction malleability”. They claim that the issue is preventing them from reliably processing bitcoin withdrawals and that they have to freeze withdrawals while they sort it out. Not good.

The thing about transaction malleability is that it’s been a known issue/quirk of the bitcoin protocol since 2011. Briefly, there’s a several minute window between when a transaction is broadcast and when it’s confirmed in the bitcoin blockchain. It’s possible during that window to broadcast an identical transaction (same sender, same recipient, same quantity of bitcoin), but with a different transaction-hash. Only one of these transactions will make it into a block, with the other being considered a double-spend, and therefore dropped. Since this is a known issue, no diligent bitcoin service implementation uses the transaction-hash as a sole identifier for transactions in their internal accounting systems.

But MtGox apparently did. That meant that malicious individuals could withdraw bitcoin from MtGox, immediately issue a re-broadcast of the transaction with a different hash, and then if that re-broadcast transaction made it into the blockchain, the person could then contact MtGox support and say “Hey! My withdrawal never happened; see, the original transaction hash is not in a block! Send it to me again!”. Apparently MtGox even had an automated process for withdrawal resends!

While other exchanges did end up temporarily suspending withdrawals to evaluate their own code in this context, they all re-opened quickly and without issue. MtGox was the only exchange demonstrating such careless accounting and withdrawal processes.

February, 24th 2014 – MtGox finally dies.

MtGox suspends trading entirely, deletes their twitter history, and leaks documents alleging 744,000 missing bitcoin. What?!

We still don’t know the details, but CEO Mark Karpeles said today that the leaked documents are “more or less” legit.

Which begs the question: How on earth do you lose track of 744,000 bitcoin?! The document says the bitcoins “are missing due to malleability-related theft which went unnoticed for several years.” If true, that implies some unbelievably bad accounting practices, business operations, financial management, executive diligence, etc, etc. It’s not hard to check a bitcoin cold-wallet balance, and at least roughly reconcile accounts on a frequent basis.

The document also states: “The cold storage has been wiped out due to a leak in the hot wallet.” Again, What?! That can’t happen in a properly implemented system, and reeks of even more egregious technical incompetence.

Other possibilities, of course, include insider theft, or far more damage from the 2011 hack than has been admitted to date. UPDATE: Or MtGox may have simply lost the private keys to their coldest & oldest storage, or maybe the US Government has them. We may never know the truth, but one thing is for sure: Bitcoin is better off without such amateur-hour incompetent businesses as MtGox.

UPDATE: February, 28th 2014 – Bankruptcy.

MtGox declares bankruptcy, disclosing 127,000 customers owed an average of $3500 equivalent each.

In Summary

MtGox’s failures were many: regulatory, technical, business-strategy, accounting, management… The specific failures that made MtGox uniquely vulnerable to this kind of catastrophic implosion were:

1) Using a custom bitcoin implementation and not sufficiently updating it or handling long-known issues.
2) Not treating regulatory issues seriously.
3) Poor general security practices.
4) Poor business decisions/relationships.
5) Technology unable to handle predictable load and/or DDoS attacks.
6) Improper bitcoin funds management (cold/hot wallet).
7) Egregious accounting practices.

All these factors led to the situation MtGox is in today. No business should operate with this level of incompetence. MtGox was the last big holdover from early-bitcoin, where enthusiasts built initial services whose popularity quickly exceeded the innovators’ ability to manage the business. As Roger Ver said:

“Gox is the worst-run business in the history of the world.”

And that’s coming from “Bitcoin Jesus”.

Ultimately, the dramatic failure of MtGox marks the transition from early-adopters and a niche market, to seasoned professionals increasingly serving a mass-market. The current crop of bitcoin businesses is a different breed than the first generation: venture backed, run by proven talented entrepreneurs, and aggressively compliant with existing regulatory frameworks. These are the businesses that are driving bitcoin’s next phase of adoption.

Bitcoin Phase-1 is Over. On to Phase-2


It’s one of those days in bitcoin. MtGox internal documents allegedly leaked. 744,000 BTC potentially “missing” from MtGox’s books. NYTimes and Wired running articles with dramatic headlines and several FUD paragraphs…

Nobody ever said bootstrapping the world’s first truly global, ideal-for-our-times, decentralized currency was going to be easy. As the media circus surrounding MtGox’s likely demise begins in earnest tomorrow, let’s brush off the attacks on Bitcoin in general, and look to the next generation of bitcoin businesses.

MtGox was a holdover from early bitcoin, where clearly incompetent people/operations handled massive amounts of customer funds. That phase of bitcoin’s development has been ending, with the termination of MtGox being the final chapter.

The next phase will be driven by professional, larger-scale, audited, transparent companies. They will look to use the transparency, proof, and segmented control features of bitcoin to build trust, and demonstrate the unique power of bitcoin. Look soon for an exchange offering real-time proof-of-reserves, and multi-sig account control.

The next phase may not be as roller-coaster exciting as bitcoin’s birth, but it’ll change many more lives for the better. The next 100 million users await.

ASICS – Are They Evil?


As the arms race in bitcoin mining continues unabated, the flurry of misinformation and bad logic surrounding the issue is as intense as ever. Since litecoin originally launched and attempted to boast itself as having an “ASIC resistent” mining algorithm, people have been creating and touting various marginal alt-coins as the ultimate bitcoin successor because of supposed resistance to ASICs and mining centralization. This is flat out wrong. Any successful proof-of-work based crypto-coin that develops high enough value will experience the same kinds of mining dynamics we’re seeing with bitcoin. It’s all about the money.

The Arms Race

Now that each bitcoin is worth hundreds of dollars, the 25BTC block-reward distributed by the protocol to bitcoin “miners” every 10 minutes is very enticing. More and more serious, and well-funded, mining operations are appearing. You only need to glance briefly at a bitcoin-mining hashrate chart to understand that something explosive is going on here:



What are ASICS?

ASIC stands for Application Specific Integrated Circuit; ie, a specialized computer chip. Back in late 2012 when it became clear to some that bitcoin was likely to keep gaining in value, engineering teams started to work on developing ASIC chips for bitcoin mining. These chips could mine at 10-100x greater power-efficiency than the standard GPU (graphics-card) mining hardware at the time.

So what’s the controversy?

ASICs are expensive. A single top-of-the-line ASIC rig (the chip + supporting hardware) can cost $5,000-$10,000. That makes cutting-edge bitcoin-mining hardware out of reach for many people. No longer can someone spend a few hundred dollars on a fast GPU card, stick it in any old computer, and profitably mine bitcoins.

That has some people in the bitcoin community forecasting doom at the hands of big-money, centralized, corporate bitcoin mining operators. The fear is that, eventually, there will just be a few huge bitcoin miners that are susceptible to less-than-honest practices or government interference.

How is this a problem with ASICs?

It isn’t. People are just associating “big money” and centralized power structures with ASICs because ASICs cost “big money”. The problem is not ASICs; it’s the fact that when a coin achieves enough value, it suddenly becomes rational to throw significant money into mining it. This has already started happening to litecoin, with multi-thousand-dollar GPU rigs fairly common, and scrypt-ASICs around the corner (something that litecoin’s proponents originally said would never happen).

It’s simply the case that people are going to invest money into mining gear if there’s profit to be made. Whether the specific mining algorithm lends itself to ASICs, GPU farms, CPU farms, lots of memory, or something else, those with the ability to throw lots of money and compute power at the problem are going to get a greater share of the mining market. Eventually, this centralizes to those with the most power-efficient hardware, no matter what the algorithm.

So make no mistake: proof-of-work mining will be done at data-center scale in any mature and valuable coin.






This issue comes up repeatedly, sparking many heated discussions. Some people think issues surrounding the blocksize will be the ultimate downfall of bitcoin; others think the theoretical problems are largely overstated and/or completely resolvable. Count me in that latter camp.

What is it?

The number of  transactions per second that bitcoin can handle on-blockchain is directly tied to the data size of each block. Since bitcoin’s inception, the size of a block has been fixed in the bitcoin code to a maximum of 1MB. Since one new block is generated on the bitcoin network every 10 minutes on average, this works out to a theoretical maximum of 7 transactions per second. Compared to the thousands of transactions per second that the Visa and Mastercard networks handle, this is not a lot.

Why is the blocksize limited?

As with many things in life, there’s a tradeoff. Larger blocks would require more processing and bandwidth resources from bitcoin nodes (the global network of “volunteer” computers running the bitcoin software), which would eventually cause fewer people to be willing to run nodes. On the flipside, the network’s transaction processing capacity is clearly limited by the blocksize. To be a global currency handling everything from inter-bank settlement payments to online tipping and micro-transactions, clearly 7 transactions per second is far few. The current 1MB size was chosen by bitcoin’s creator, Satoshi Nakamoto, in the original codebase as a reasonable balance during bitcoin’s birth and initial growth stage.

What can be done about it?

Simple – change the code to raise the blocksize limit. While this is a one-line change to the code itself, it would require most nodes to incorporate the change at about the same time. Ever prescient, Satoshi even suggested how this can be done eventually, stating on the bitcointalk forum in 2010:

It can be phased in, like:
<br /><br />
if (blocknumber > 115000)<br />
     maxblocksize = largerlimit
<br /><br />
It can start being in versions way ahead, so by the time it reaches that block number and goes into effect, the older versions that don’t have it are already obsolete.

What *should* be done about it?

This is where people differ. Some are adamant that the 1MB limit stay in place forever, in order to keep it cheap for small-time operators (individuals with laptops) to run full bitcoin nodes. Others, myself included, prefer a more market-driven/organic approach. Acknowledging that Moore’s Law continually reduces the cost of computing power and bandwidth, I think a steady or market-defined increase in the blocksize is reasonable. It could even be eliminated entirely. In theory, bitcoin miners would be incented to strike a balance between including lots of transactions to gather fees, and the costs of processing and transmitting larger blocks. Like many dynamics in bitcoin, this seems like one where a natural market-driven optimum can efficiently emerge.