JPAllen.net

AOPP Oxford Software Carpentry Workshop Summary (14-15th Jan 2015)

2015-01-18T00:00:00+00:00

This week I taught my first two day Software Carpentry Workshop at the department of Atmospheric, Oceanic and Planetary Physics in Oxford. It was amazing to see people go from knowing little about git and Python to being competant and enthusiastic in under 10 hours of actual teaching time. I’ve taught before, but never seen a group of people so quick to pick up the material, or so interested in learning it. The SWC materials, the workshop format and the self-selecting nature of the attendees all made for a great experience for everyone.

The only thing left to do is a little reflection, so here are some things that worked well:

Everything technical worked as it should: wifi, power sockets, projector, installations.
The logistics worked well: coffee was on time, lunch was easy to get and close by, everyone arrived on time.
With the exception of the shell which was too slow, the material was taught at a good pace and was enjoyed by most.
The multi choice questions answered with post-its were a big hit.

The more interesting list is some things that could have been better:

Learners didn’t use the red post-its for help as much as they could have. A few people had problems that didn’t get fixed until well after they were no longer following. I think this is because we didn’t emphasise that they could be used anytime, and should be. Also, the first half of the first day was spent teaching material that everyone was mostly familiar with, so there wasn’t much need for red post-its and their use wasn’t engrained when it came to the next harder parts.
I didn’t make the example data for the shell course available. This worked well for the Python part, and I should have done the same for the shell.
People were asking for more multi-choice post-it questions, and harder ones. People mostly agreed on the answers each time, so I’d made them too easy and missed the chance for good discussion between learners as they worked out their different answers.
At least 3 of my questions had mistakes which made them either wrong, or ambiguous. Lesson learned: Always test your material on a few people before the lesson!
There were quite a few suggestions for something like a reference guide of bash and git commands. A summary of commands would be enough to recap rather than the wordy lession material.

Looking forward to my next workshop!

AOPP Oxford Software Carpentry Workshop Debrief

2015-01-14T00:00:00+00:00

Today I had my first experience teaching a Software Carpentry Workshop at the Atmospheric, Oceanic and Planetary Physics Department in Oxford. (There was another separate workshop running its second day in Oxford today as well.) The day went well and I feel that students left having enjoyed it and looking forward to coming back tomorrow!

I taught the shell in two halves in the morning, and my co-instructor Jens taught the basics of Python after lunch, also in two halves. We asked for feedback at the end of each one and half hour slot. I was worried this was excessive, but I’m glad we did because it allowed us to adjust the second half of each topic based on what people told us after the first. The shell in particular was a bit too basic for what the learners already knew, so I tried to go at a faster pace in the second half. I think the second half of the material was also naturally more engaging as it moved away from the absolute basics. The feedback suggested that pipes and grepping were the two favourite topics of the morning sessions.

A lot of learners suggested in their feedback that I should have had them download the same example data that I had, so they could also run my examples and commands as I typed them. This seems obvious in hindsight, and I’ll definitely do this next time.

There was also a lot of enthusiasm for the multiple choice questions that were answered by holding up a coloured sticky note and discussed between the learners afterwards. It was the first time I’d tried that technique, so I was hesitant to over use it, but based on the feedback I will try to increase the number of questions I ask and to make them more challenging next time.

The Etherpad didn’t work when we tried it in the first morning session, so we moved on without exploring it too much. I tried to reintroduce it in the second session when it was more stable, but I think we’d collectively lost enthusiasm for it. Everyone was already taking their own notes and no learners contributed to the Etherpad as we taught. Perhaps if I introduce this as the beginning of the day again tomorrow, with new topics, we will get more of an uptake. I don’t think the session particularly suffered from a lack of the Etherpad though.

There were minimal installation issues today and only took a few minutes to get things working for a few people. The Anaconda and Software Carpentry Windows installers are doing their job well. We’ll see if we have equal success with git and some more complicated Python libraries in tomorrow’s session.

Unit Tests vs Acceptance Tests

2014-11-26T00:00:00+00:00

There has been some discussion in the Software Carpentry community about teaching testing as part of their workshops. Actually the discussion was focused around why testing isn’t part of the standard curriculum, but I think a lot of the problem comes from confusing different types of tests and their applicableness. The tendency to lump everything together under the umbrella of ‘automated testing’ means that some very accessible and appropriate forms of testing get ignored because of problems with other types of tests. In my day to day work, I deal with two types of test, unit tests and acceptance tests, and this article is about understanding the differences.

Unit tests and acceptance tests need fundamentally different approaches in how we think about them, how we write them and how we teach them.

Unit tests

Unit tests check that your code does was you think it does, but not whether that is the correct thing to do. A good unit test checks that you have implemented the low level structure of your functions (or ‘units’) without any straight-forward errors. These types of errors are things that every programmer, in every discipline, is familiar with:

Mistakes in syntax or variable names;
Off-by-one errors in loops or array indices;
Edge cases like empty input, division by zero or handling NaNs;
Transcription errors when translating equations into code. It’s easy to make sign changes, miss out factors, use the wrong indices, etc.

The fundamental building blocks of a programming language are the same regardless of the context you’re using it in, so unit testing takes the same approach everywhere.

Unit testing is a replacement for the process of manually testing your function after writing it. Without unit testing, you might load your code into a console environment, and check that it behaves as expected for different input. If your language doesn’t have a console environment, you might try to call your program in a way that will trigger your function with the inputs you want to test. Testing edge cases like this can be very hard if not impossible. With unit testing, we replace this manual process with an automatic process. We still go through the process of thinking of good example input to test our function and its edge cases, but instead of testing it manually we write the tests in code. Automation allows us to run our tests quickly and repeatedly every time we make even a small change to the function.

Unit testing should be seen and taught as an integral part of the development process. If you write your functions first and then think about trying to test them afterwards, all you will discover is that you’ve written functions that are very hard to test. Instead we should think about how we’re going to test our code before we write a single line. This will force us to structure code in very small ‘units’ that do a single job. We can then easily test that each of these units do their single job as expected. These units can be built up into larger units, with tests written to check that the correct lower level functions (or ‘units’) are called as required.

With unit testing, the fundamental question of ‘how do I test this code?’ is the wrong question. Instead we should be asking (and teaching) ‘how do I write this code in a way that is testable?’

There are some quick heuristics to see if we’re dealing with unit tests:

They don’t read in any external data. Each test case is small enough that small sample inputs can be provided inline when testing the function, and the outputs of each individual function can be verified in a simple way.
Your entire test suite can run in less than 60 seconds. Each unit test will only test a tiny amount of code so will run very quickly. Even if you have thousands of them, a computer can crunch through them quickly.
You have the same order of magnitude of test code to program code. Since each function in your program code is small, and has a corresponding unit test, you’ll need to write similar amounts of test code to program code. In fact, in my experience I write 1-2x more test code than program code. However, the test code is much quicker to write since it’s ‘dumb’.

Acceptance tests

On the other hand, acceptance tests check that the combination of your little ‘units’ of code is doing something that is overall sensible and correct for your use case. You’ve verified that your code does what you expect it to with unit tests, but acceptance tests help to check that your mental model is correct, and that the code you’ve written actually solves the problem you set out to solve. For example, you could write a perfect implementation of a soil temperature model using Kelvin as your temperature units, only to find out that you input data is in Celsius and so your output is clearly physically wrong despite being the ‘correct’ output for your program.

What makes for a sensible acceptance test will depend on your discipline and the purpose of your code. Perhaps you need to check that the output of a simulation is physically reasonable and satisfies certain conversation laws. Perhaps you need to sanity check that your data cleaning code is not affecting the good parts of your data. It will completely depend on context, and that makes it hard to talk about in general. However, I think there are some universal points which can be said about acceptance testing.

Unlike unit testing, acceptance testing is not a replacement for manually verifying the result of running your code. Typically in a complex situation, you will be doing some manual checks to make sure that the result is sensible. Does it pass the ‘gut instinct’ test? Does it match what you understand about the system and expected result? Does it fit with anything you can check analytically? Does the data match observations in the expected way? And so on depending on your field.

Once you’re happy that your program produces the correct output for your inputs, you should pause and encode this in an automated acceptance test. The acceptance test will basically say ‘given this input, does my program produce this output’. The purpose of this test is to protect against unexpected changes in the behaviour of your program in the future. If you refactor your code to be structured differently, or use a different internal data format, or extend your algorithms from 2 dimensions to arbitrary dimensions, your acceptance tests allow you to quickly verify that the behaviour of your code hasn’t changed in a way that you didn’t expect.

Acceptance tests can be fragile, and that’s ok. If you program changes something small in its output, then it’s ok for your acceptance tests to start failing. This will make sure that your attention is drawn to any changes in the way your program behaves. If you expected the change to happen you can quickly fix up your acceptance tests to match the new output. If you didn’t expect the change then you can investigate further and see whether it’s an acceptable change, or whether it’s the result of an unexpected bug or problem.

While unit testing is a fundamental part of a good development workflow, acceptance testing is something that will typically be done after the main body of code has been written and you’re ready to start testing whether the whole thing behaves correctly.

Some quick heuristics to see if we’re dealing with acceptance tests:

Your example inputs and outputs are complex enough to be stored externally from your test code.
Your acceptance tests takes a while to run since they have to run a non-trivial amount of your program code to actually test whether it’s doing what you expect it to.
Your acceptance test code is an order of magnitude smaller in size than your project code. Verifying a correct solution is typically much easier than finding the correct solution in the first place.

(NB. Take everything below here with a pinch of salt because I don’t have much direct experience teaching these things).

It should be possible to teach unit testing to anyone from any discipline since the methodology and principles apply to any sort of programming. To go further, I don’t think we can separate ‘writing unit testable code’ from ‘writing reusable, modular, and readable code’. They are the same thing, and building up a mental model of programming that involves always asking ‘how can I write this code so that I can test it?’ will teach students how to write a higher quality of code, even if the tests are never written (but of course they will be!)

Teaching acceptance testing is harder, because it will depend on the specific code and discipline. However, I think acceptance testing doesn’t need to be taught in as much depth as unit testing because the general concepts are easier to state. Acceptance testing is an exercise in testing whether your program matches a certain input to a certain output. The definition of ‘input’, ‘output’ and ‘program’ will vary for everyone but the ingredients are all there once students are familiar with writing code, reading data, and automating tests.

Lifecycle Emails

2013-07-25T00:00:00+00:00

This post is based on a ten minute talk that I gave at Exeter Web in July. The slides are available here

Lifecycle emails are about sending your customers relevant emails depending on where they are in their relationship with your product or service. When someone first signs up they’re going to need help and advice with different things than someone who has been happily using your service for a few months but has recently stopped for some reason. Both of these users can be chased up with an appropriate lifecycle email and you’ll hopefully be able to improve their experience, and learn more about what your customers want and value.

When done correctly lifecycle emails are not at all spamy. Doing them correctly doesn’t mean following a certain process perfectly, or sticking to any “dos and don’ts”. Instead it means that you need to actually care about your users and genuinely mean what you say in your emails to them. When you ask for feedback you have to be ready to listen to what they say and embrace consider their suggestions when deciding how to go forward with your product or service. And when you send an email that looks like a personal conversation starter, you’d better be ready to actually have that conversation when the person on the other end replies.

When a user first signs up to your site you could send them an email like this:

Hi Joe, thanks a lot for signing up for AcmeWidgets! If you get lost, you can login again here.

If you ever have any problems or questions (or even just a simple suggestion about something we could improve) then please feel free to get in touch directly by replying to this email.

Regards,
Bob
Founder of AcmeWidgets

Almost every company sends out a ‘welcome to our service’ email, but you should use this email to give you customers a chance to engage you in a conversation immediately. Offer them your personal attention if they want it, and make it clear that a reply to even this basic email will get them talking to someone behind the scenes.

We tend to use services by imagining that they somehow just spring up and work. It’s easy to forget that there are people behind the products you use, and as the person behind a product you want to make sure that your customers are aware of you and understand that you care a lot about your users. After all, if your users aren’t happy and enjoying their experience then you’re failing in at least one way. If you can remind users that there are people just like them behind the product and give them an opportunity to vent their frustrations and share their ideas and wants with you then you’re in a great position. The sort of feedback that comes from your customers feeling empowered like that is incredible because they start to talk about what they actually want out your service, not just the incidental bugs or hiccups they are coming across.

Another email you might send is when it’s clear that a user is interested in your product and is using it day to day. Starting a free trial might be a good indicator of this, but I’ll leave it to you to decide what is appropriate for your business.

Hi Joe, I just noticed that you started a free trial and I wanted to reach out and see how you were getting on with AcmeWidgets. Has it helped to improve your Widget processes? Have you had a chance to try our Widget factory?

I head up the development of our Widget Platform, so if you have any suggestions for what we could improve then please let me know. We’re always open to feedback about what we can be doing better.

Regards,
Lucy
Chief Engineer of AcmeWidgets

The idea here is to find out what people like about your service, and what the people who are in the most likely position to start paying for it think. It’s not always nice to hear, but if people keep telling you that a certain thing just doesn’t work for them then it’s time to do something about it. But you need to give people a chance to tell you that first!

If things don’t go so well and a user abandons your site then it’s important to know why. Was it because something was missing, or was it because they stuck at a difficult or confusing part of using your product? Let’s find out!

Hi Joe, I noticed that you haven’t used AcmeWidgets for a while and I just wanted to check if you’d had any problems that had put you off? Is there anything we can do to make AcmeWidgets easier for you to use?

Please let me know if there’s anything I can do to help get you back on track, or why AcmeWidgets isn’t suitable for you.

Regards,
Bob
Founder of AcmeWidgets

This is another great chance for feedback, but it’s also proactive customer support. The bar is set so low by most companies when it comes to customer support that simply by showing up and caring about your users you’re going to delight them. And delighted users just love to share their stories with friends! It’s a very powerful thing to reach out and say ‘Hey, it looks like you had trouble. Let me help you with it.’ Most people just want to be productive, and so it’s impressive when you reach out and say you’re willing to invest some time that they didn’t even ask for into getting your them up and running. We’ve had more than one evangelical convert from this. Abandoned user to evangelical convert in just a few email exchanges? Not bad.

I’ll end with a final suggestion for an email, and undoubtably one of the most important. If a user stops paying for your product then find out why, and do everything in your power to negate this reason for the rest of your customers.

Hi Joe, I’m sorry to see that you cancelled your paid account. If you could spare a few minutes, I’d love to hear what we did wrong. Is AcmeWidgets missing something that you need? We’re always trying to improve our service so your feedback would be really valuable to us.

Regards,
Bob
Founder of AcmeWidgets

Tracking down a memory leak in Node.js and Socket.IO

2013-03-08T00:00:00+00:00

If you are running Node.js 0.8 and Socket.IO over HTTPS then you will be affected by this memory leak. See the bottom of the post for details of a fix.

Node.js is a great bit of kit, but it’s still a relatively young technology. I was bitten by this recently when I had to investigate a large memory leak in one of the Node.js apps I maintain. There are tools that can help to track down these sorts of problems, but it still took lots of trial and error to understand, and I thought it would be worthwhile to share my experience here.

Let me start by showing just how extreme the memory leak was. The graph below shows two very normal things that can happen in a Node.js app: connections coming in via WebSockets using the Socket.IO library, and uploading files. If we do either of these individually then there is no problem and the memory flattens off nicely, but when we send uploads and WebSocket connections together, BOOM, there is linear memory growth:

The Node.js version is 0.8.21 and the Socket.IO version is 0.9.13 here.

Getting a heapdump

Before doing any sort of debugging, I needed a snapshot of the state of the memory of the Node.js app. Since the memory leak was so extreme, I only had to wait a few minutes to be confident that a large proportion of the Node.js process’s memory was taken up by the leaking objects. To get a dump of the process’s memory, I used the heapdump module. To take a snapshot of the current state of the program’s memory, it’s as simple as:

heapdump = require("heapdump")
heapdump.writeSnapshot()

Of course, there’s no point doing this immediately when the application starts. A clever trick that we use for things like this is to allow telnet access directly to a REPL running in the context of the Node.js process (example in coffeescript):

require("net")
  .createServer (socket) ->
    repl = require('repl')
    repl.start("my-node-process>", socket)
  .listen 5000, "localhost"

I can then telnet in and take a snapshot after the process has grown it’s memory footprint large enough to be worth investigating:

$ telnet localhost 5000
my-node-process>require("heapdump").writeSnapshot()

This will write out a file called something like heapdump.1234567890.heapsnapshot to the same directory as the application’s code.

Inspecting the heap snapshot

Node.js is based on the V8 javascript engine used by the Chrome browser. This is very handy, because Chrome has a great debugging tool built in for inspecting heap snapshots from V8, and it works well for inspecting the heap snapshot of a Node.js process.

To load the heap snapshot, open up the Chrome developer tools, and go to the ‘Profiles’ tab. Right click in the left hand side bar, select ‘Load Heap Snapshot’, and load the snapshot that was taken previously by heapdump. The heap snapshot inspector lets one see all the objects in ones Node.js app’s memory at the time of the snapshot and shows how many of each object there are and how much memory they are using. After sorting by size, it was immediately clear that around 200Mb of my app’s memory was taken up by buffers. Expanding the Buffer list, I saw that there were many 10Mb Buffers hanging around. Whatever is creating and holding onto those is undoubtably the cause of the leak.

Clicking on an object in the heap inspector shows one the object’s retaining tree. That is, the parent objects that hold a reference to that object, and the objects that hold a reference to those, and so on right up to the top level of your application. The way that Node.js’s garbage collecting works means that an object will not be removed from memory until there are no references to it from elsewhere in the app. By examining a few of these buffers, it was clear that they were being held onto by a req.head object which was referenced by a WebSocket object. At this point, Socket.IO was starting to look suspicious.

The problem

A quick grep through the source code of Socket.IO quickly turned up the following lines as the likely origin of the leaking req.head object:

Manager.prototype.handleUpgrade = function (req, socket, head) {
  ...
  req.head = head
  ...
}

This is called when a regular HTTP connection requests to be upgraded to a WebSocket connection. From the Node.js docs for the upgrade event:

head is an instance of Buffer, the first packet of the upgraded stream, this may be empty.

So it’s a Buffer as expected. However, it’s only the first packet, so it certainly shouldn’t be taking up 10Mb by itself, and from the heap snapshot it looked like there were multiple req.head objects pointing to the same Buffer. At this point I must admit to being a bit stuck and flailing around on Google for a while. I did eventually find the answer though:

In Node.js version 0.8, the tls module, which is used by the https server in Node.js, allocates 10Mb SlabBuffer objects up front. It does this because allocating a buffer is relatively slow, and rather than allocating new buffers for each incoming request it can instead quickly allocate portions of this large buffer. The head buffer, which is passed to Socket.IO, is allocated from this 10Mb SlabBuffer. Crucially, the Node.js garbage collector will not remove the SlabBuffer until all of its children buffers are removed, but the head buffer is kept around for the whole time that the WebSocket connection is open. This means that every WebSocket connection can potentially be keeping 10Mb of space hanging around in memory.

This explains why our production process was seeing ever increasing memory usage. We had lots of traffic over HTTPS so lots of these 10Mb SlabBuffers were being created, and we also had people constantly connecting via WebSockets and hanging around for a while preventing the SlabBuffers from ever being freed from memory.

The fix

It’s hard to fault the design of Socket.IO here. There is no indication that keeping this small head buffer around should have such dire consequences. It’s also hard to fault the design in Node.js since in isolation the SlabBuffer implementation is great idea for improving the speed of Node.js. I think this is just one of those subtle bugs from an unexpected interaction between two systems which both seem to be acting sensibly on the surface.

There are a few patches out there which address this issue for Socket.IO: one here by jmatthewsr-ms and one here by me. These haven’t been merged yet since there is some ongoing discussion about whose fault this really is. There is some suggestion that this should actually be fixed in Node.js to avoid catching others out in the future. Either way, I think this is a crucial problem that the community at large should be more aware of.

For an immediate fix, you can monkey patch Socket.IO to include an additional last line in the handleUpgrade method in lib/manager.js which frees the req.head buffer:

Manager.prototype.handleUpgrade = function (req, socket, head) {
  var data = this.checkRequest(req)
    , self = this;

  if (!data) {
    if (this.enabled('destroy upgrade')) {
      socket.end();
      this.log.debug('destroying non-Socket.IO upgrade');
    }

    return;
  }

  req.head = head;
  this.handleClient(data, req);

  // Insert this line:
  delete req.head;
};

A fix which I haven’t tested but may help without solving the underlying cause is to shrink the size of the SlabBuffers allocated by the tls module:

require('tls').SLAB_BUFFER_SIZE = 100 * 1024 # 100Kb

You can only do this in Node 0.8.20 and above though. I haven’t tested this to see if makes much difference, or what affect it has on speed. Use at your own risk.

Conclusion

This is a subtle bug, yet I suspect it is responsible for many of the complaints about Socket.IO being a memory hog. The mix of circumstances are almost certainly found in a large number of production services. WebSockets are a common use case for Node.js and everyone should be using HTTPS wherever possible.

If this is something which has affected you and my post has helped out, please let me know. Contact details are in the side bar.