Proposed Rust community norm for unsafe code

Recently cessen asked people to write their thoughts on Rust community norm for unsafe code. So here it is.

I recommend reading the whole article, even if it is somewhat long. I will quote what I consider to be its essential point, but my quote will necessarily lose nuances.

Rust community's attitude towards unsafe code is something along the lines of, "Don't ever use it unless you're a wizard... and you're not a wizard."

I strongly reject this attitude. I believe this does not match Rust's design intent.

Different software has different needs, and I don't think a one-size-fits-all attitude towards unsafe code makes sense.

I also strongly reject this proposal. Shared community norm is valuable. I fear difference here can result in ecosystem split. Due to network effect, one-size-fits-all can make sense even if one size does not fit all, if suboptimality is compensated by community size.

Examples of shared community norm I noticed include expectation of portability in C, expectation of freeing memory in C, expectation of referential transparency in Haskell, expectation of non-blocking in Node.js, etc. For some software, it is entirely reasonable not to free memory in C (your program always allocates less than 1 GB), write side-effecting pure function in Haskell (for debug logging), block in your Node.js library for computation (you should wrap it in thread if you need it in non-blocking context), etc. but it is understood that this is just not done, for code intended to be shared and reused by others, because it breaks expectation and is egregious violation of principle of least surprise.

One important question is what this norm should be. I propose "Rust library should strive to have correct safety declaration in the interface". Let me explain word by word.

"Rust library". I think it is okay to hold applications, aka code not intended to be reused, to less standard.

"should strive". Making mistakes is okay and does not count against you, that is, not more than any other bugs. On the other hand, community norm is that these are bugs, and are expected to be fixed. Note that IT CAN ALWAYS BE FIXED, even if actually doing so is a bad idea because it is API breakage: you can change any such "fn" to "unsafe fn".

"to have correct safety declaration". This is to be fixed in stone by unsafe code guideline, but general outline, I think, is easy. Using your code, safe code should not be able to perform use-after-free, to produce aliased &mut, etc etc.

"in the interface". As long as the interface is safe, any use of unsafe in the implementation is fair game and does not count against you and lots and lots of uses of unsafe is not a reason to avoid the library. I mean, "std" is full of them. Go look at it.

Thanks for reading. Please comment on /r/rust subreddit.

Problem in Rust adoption

I observed and contributed to Rust programming language for four years now, and I have come to a conclusion: Rust's primary difficulty in adoption is mismatch of expected learning curve and actual learning curve. Actual learning curve itself is okay.

(Of course, there is general immaturity in tooling and ecosystem. Here I concentrate on conversion from trying Rust to getting basic Rust competency. Two important scenarios, 1. I am not trying Rust because I can't see how it would be useful to me right now, and 2. I finished learning Rust basics, but I still think it is not a good idea to do my real world project in Rust, are not discussed here.)

Going with a bit of caricature, consider Java, C++, Rust and "X programmer learns Y". Here "X programmer" excludes polyglot programmers, since they are minority. Since there are also too few Rust programmers to matter, there are four cases:

Case 1. Java programmers learn C++. People expect this to be hard, and it is actually hard.

Case 2. C++ programmers learn Java. People expect this to be easy, and it is actually easy.

Case 3. Java programmers learn Rust. People expect this to be hard, and it is actually hard.

Case 4. C++ programmers learn Rust. For some reason, people expect this to be easy, but it is actually hard. In my experience, it is equally hard for C++ programmers to learn Rust as for Java programmers to learn Rust. Mismatch in expectation causes frustration.

One theory I have is that Java programmers expect Rust to be hard because they think they don't know "memory management", and C++ programmers expect Rust to be easy because they think they know "memory management".

The problem is, C++ programmers know "C++ memory management", not "memory management". To learn Rust, you need to learn "Rust memory management". And knowing "C++ memory management" doesn't help you learning "Rust memory management", not as much as people expect. It helps a little, but if you expect, say, it to be as half as difficult as learning C++ for the first time, you'd be rudely surprised. In practice, people don't expect even half.

In other words, many people seem to have a linear model where Java is at one end, Rust is at the other end, and C++ is closer to Rust than Java. In reality, Rust is about equidistant from both C++ and Java.

The question is how to correct expectation. I really have no idea, but maybe explicitly talking about the problem will help.

Cohort analysis of Rust contributors

Cohort analysis is widely used in business analytics, but open source software projects do not seem to make use of it. The most interesting activity to analyze is code contribution, and for that all the data is readily available in the source code repository history. Data is even quite small: Rust is a big project, but there are less than 50,000 commits in total.

In 2014, Igor Steinmacher and co. published a paper titled Attracting, Onboarding, and Retaining Newcomer Developers in Open Source Software Projects. This is the most interesting paper presenting "developer joining model". Go read the paper, and other papers from Software Engineering and Collaborative Systems Research Group at Math and Statistics Institute, University of São Paulo.

According to the paper, "outsiders" are first "attracted" to be a "newcomer". Many factors affect this step, including software license, development infrastructure, project size, project complexity, project age, and specific events such as releases. Next, "newcomers" go through "onboarding" to become a regular "contributor". In this step, first impression matters a lot, and a timely response helps, and a rude response hinders. To "retain" "contributors", contributors need to be able to understand project process, and the project needs to keep a favorable atmosphere.

This is surprisingly (or not?) similar to user acquisition, conversion, and retention in business. So I wrote a quick Python script to do the analysis. Contributors are segmented to cohorts by month of first commit. Contributors are considered converted when they make commits in 2 different months. Contributors are considered retained when they keep making commits.

In 2 years period starting from January 2013, Rust acquired 711 new contributors in 23 months. (I excluded the first month.)

Rust releases every 3 months, and 3 months pattern is evident. Contributors are more likely to make their first commits in the month Rust released. Release 0.9 and 0.10 are especially prominent.

Rust converted 252 new contributors to repeat contributors, with conversion rate of 35%. Contributors took 2.3 months to convert in average. Following is plot of 19 months, excluding first 2 months (no new contributors can be converted there) and last 3 months (not enough time to observe conversion).

This is harder to interpret. From what I can tell, absolute conversion is increasing, but conversion rate is decreasing. This makes sense, since Rust had more self-selected contributors in its early life. Now there are more casual contributors.

Here is retention table of last 6 months of above period:

Cohort	0	+1	+2	+3	+4	+5
2014-04	14	6	2	1	0	2
2014-05	13	7	8	7	7
2014-06	17	5	6	6
2014-07	10	3	1
2014-08	22	10
2014-09	8

Data is quite noisy: after 2 months, retention rates are: 62%(8/13) for May, 35%(6/17) for June, 14%(2/14) for April, and 10%(1/10) for July. My impression is that Rust is retaining contributors well once they contribute in 3 different months, not 2. Maybe I need to change conversion criteria. Data is too noisy to say anything about time trend.

Plain Simple Stupid Language

People don't implement plain simple stupid language. It is an understandable but grave mistake. To implementers it may seem pointless and to users unattractive. But to students of language implementation plainness is a virtue, simplicity a must, and even stupidity can be helpful.

Modern programming languages are complex beasts. They are like cheetahs, tigers, lions, and even dragons. Some have wings like pegasus. When we speak of beasts we remark on their speed, their sharpness of teeth and claws, their unusual ability of breathing fire and winged flight. We rarely remark that they have eyes, mouth, and legs; hearts, lungs, brains, and kidneys. Above all we don't talk of the fact that they are animals -- and that they can do things we expect animals to be able to do. Those are implicitly understood.

Imagine that you are to explain what is a dog in words to an alien who never saw any animals. To people of Earth a particular animal is often described as an animal plus any difference to what I would call a plain simple stupid animal. But when you really try to describe it, you soon realize that the most of complexity is in that it is an animal, not that it is a dog. A dog may be simple, certainly simpler than dragons or pegasuses (pegasi?), but it still is an animal which is insanely complex in itself.

Programming a board game AI

This semester, the final project of the problem solving class, which belongs to the computer science department, was to program an AI for a board game called Halma, whose variant is often known by the name Chinese Checkers.

Typical board game AI consists of following parts: move generator, evaluation function, and tree search. As the name implies, move generator generates possible moves; it encodes the rule of the game, judging which moves are legal and which moves are not. If move generator encodes the rule, evalution function encodes the "intelligence"; it is a function from the state of the game to some number. Usually, larger number means that the game is favorable to the player. But evaluation function doesn't need to be perfect; that's where tree search comes in. Tree search can "amplify" the minimal intelligence of evaluation function.

Because of such amplification effects, programming a board game AI is an eerie experience. When you program an unfamiliar game, it is a routine experience for your program to outperform you. It is more surprising since even the dumbest evaluation function can beat you. Often evalution function as simple as "substract the number of moves I can make from the number of moves the opponent can make" works surprisingly well. Notice that this function is not even game-specific.

In tree search, the program considers all moves I and the opponent can make to certain "depth", and passes resulting positions to evaluation function. The program assumes that the opponent tries to minimize the value of evaluation function, and tries to maximize the value of evaluation function after its move. The secret is that the program can examine a lot of positions quickly. In my case, the program searched to depth of 5 moves and examined millions of positions for a single move. This sheer quantity of positions compensate for poor quality of evalution function.

There are rich literatures concerning the programming of a board game AI. After all, people programmed computers to play chess as early as 1956. At this dawn of the programming, valiant programmers of Los Alamos laboratory programmed their huge cranking machine, which was actually less capable than cheap graphing calculators of today, to play a simplified version of chess on 6 by 6 board (see Wikipedia, Los Alamos chess). It won a game against a human who just had learned the game. We came a long way since then.

Geology

Shen Kuo(沈括) was a Chinese scientist of the 11th century. His work, Meng Xi Bi Tan(夢溪筆談), contains his scientific writings.

In one particularly stunning passage, he describes a theory of land formation by sedimentation, and an evidence for coastline movement. Following is the Chinese text and my translation. I got the Chinese text of Meng Xi Bi Tan from the Project Gutenberg.

余奉使河北, 邊太行而北, 山崖之間, 往往銜螺蚌殼及石子如鳥卵者, 橫亙石壁如帶. 此乃昔之海濱, 今東距海已近千里. 所謂大陸者, 皆濁泥所湮耳. (卷二十四)

When I was in Hebei, I went north, Taihang Mountains on one side, and between cliffs, often seashells and stones like bird's eggs crossed stone walls like a belt. Once this was the seashore, now a thousand li east of the sea. So-called continents are all submerged mud. (Volume 24)

This insight is comparable to that of James Hutton, a Scottish geologist of the 18th century, who is considered the father of modern geology. In Concerning the System of the Earth, its Duration and Stability, printed in 1785, he wrote:

Hence we are led to conclude, that the greater part of our land, if not the whole had been produced by operations natural to this globe; but that in order to make this land a permanent body, resisting the operations of the waters, two things had been required; 1st, The consolidation of masses formed by collections of loose or incoherent materials; 2dly, The elevation of those consolidated masses from the bottom of the sea, the place where they were collected, to the stations in which they now remain above the level of the ocean.

James Hutton is often unappreciated, but he must be considered as an equal of Isaac Newton and Charles Darwin. He published one of the most profound idea in the history of science, that the Earth is old. Like Darwin's theory, his theory challenged Christian doctrine, since it was incompatible with the account in Genesis. And it was the idea that the Earth is old, a half century later, that let Darwin think there was enough time for all life to evolve.

Witch on a journey

Some think that it is futile to translate poems, or lyrics for that matter. Many people try anyway. Here is my take on translating one of my favorite Korean pop songs.

Witch on a journey
included in "Welcome To My Beach", Kona (1997).

Now listen to my words
at least once a day
clean your crystal balls
get up early in the morning
drink a cup of white milk
and three lemon candies a day
don't forget to brush your teeth
before you sleep

A new wind is blowing for your journey
now raise your little broomstick
and fly to the blue sky
don't worry
and remember

Our beginning more mysterious than magic
the night and the shining promise
I will believe and wait
for the day you will come back to me
and we will start again
new days
in the wonderful world

Black cat's name's Pippi
and black dress is ironed
have you forgotten anything?
check again!

(Repeat)