Super Fast CtrlP

CtrlP is my Vim fuzzy file/buffer/MRU/tag finder plugin of choice.

However, once you start working on projects with >100k source files it can get a bit slow. Here’s some tips to speed both indexing and searching by an order of magnitude.

Speed up indexing

By default, CtrlP uses globpath() to build its index. A faster alternative is to use ag (FYI: ag is just like ack, except faster).

let g:ctrlp_user_command = 'ag %s -i --nocolor --nogroup --hidden
      \ --ignore .git
      \ --ignore .svn
      \ --ignore .hg
      \ --ignore .DS_Store
      \ --ignore "**/*.pyc"
      \ -g ""'

For me, that speeds up indexing on a project with 100k files from ~10s to ~1s.

Note that other CtrlP options related to which files get included in the index (g:ctrlp_show_hidden, wildignore, g:ctrlp_custom_ignore, g:ctrlp_max_files, g:ctrlp_max_depth, g:ctrlp_follow_symlinks) do not apply when using g:ctrlp_user_command. Instead you need to pass similar options (if they exist) to your custom command.

Speed up matching
CtrlP’s default matcher is written in vimscript, which is not known for being fast. Luckily, it’s easy to switch in your own custom matcher. I had to experiment with a few before finding one that worked well for me:

Enable the plugin via:

let g:ctrlp_match_func = { 'match': 'pymatcher#PyMatch' }

For me, the resulting speed-up is also roughly an order of magnitude on long search strings (eg. 1s instead of 10s on a 100k index).

iOS Code Completion in Vim

I recently blogged about why I use Vim instead of Xcode for iOS development (why Vim is the future).

However, one thing that Xcode does really well is code completion. It’s not without a cost – you have to turn on indexing which can consume a lot of memory and can lock up the UI on load (for several minutes on one particularly large project I work on), but the functionality is really nice when it works.

So after moving back to Vim, the first thing I did was to see about getting code completion working on my iOS projects.


Lots of code completion plugins exist for Vim, but the most awesome (IMO) is YouCompleteMe (written by one of my colleagues). Check out the animated GIF at the top of that link to get a feel for it – it’s really cool. It does fuzzy matching and never blocks editing. I highly recommend enabling it as a language-agnostic identifier-based code completer (kind of like i_CTRL-X_CTRL-N on steroids).

YCM also has a Clang-based semantic completion engine which means that it works for iOS code! All you need to do is wire things up so that clang knows what compilation flags to use for your source code (where to find headers etc).

The trick is to generate a clang compilation database – basically a large JSON array stored in a file called compile_commands.json. Each item in the array contains the appropriate clang command (including all flags) for one of your source files:

    "directory" : "/path/to/your/src/",
    "command" : "clang ..."
    "file" : "foo.m"

So how do you generate a clang compilation database from your Xcode project? The best way that I’ve found is to build your project on the command line using xcodebuild and then use oclint-xcodebuild to parse the output (Oclint is an open source static analyser that also makes use of Clang’s compilation database feature).

Step 1: Build your project via xcodebuild and capture the log in a file called xcodebuild.log (I use xcpretty to make STDOUT more pretty):

xcodebuild -workspace $XCODE_WORKSPACE \
    -scheme $XCODE_SCHEME \
    -sdk iphonesimulator build \
    | tee -a xcodebuild.log \
    | xcpretty -c

Step 2: Run oclint-xcodebuild:

oclint-xcodebuild xcodebuild.log

You can use the resulting compile_commands.json for YCM, Oclint, and any other clang-based tools you like.

Wiring up YCM

YCM looks for a file in the directory of the opened file or in any directory above it. The quickest way to get YCM working is to copy the version that ships with YCM into your project’s top-level directory and set the compilation_database_folder variable in the script to point to the folder containing your generated compile_commands.json file.

Stripping out unwanted flags

There are some flags that need to be excluded from the clang compilation database in order to keep YCM happy. You can bake this logic into, or just use the following Perl one-liners in your build script:

perl -i -ple 's/-fmodules -fmodules\S* //g' compile_commands.json
perl -i -ple 's/--serialize-diagnostics \S* //g' compile_commands.json
perl -i -ple 's/(-MMD |-MT dependencies |-MF \S* |)//g' compile_commands.json
perl -i -ple 's/(-iquote|-I|-F)\s*\S*DerivedData\S* (?<!hmap )//g' compile_commands.json


Incremental Updates

Unless you do a clean build, xcodebuild.log only contains information about files that were actually built. Each time you run oclint-xcodebuild, it generates a new compile_commands.json, meaning that you’ll be missing information about most files unless you do a clean build every time. That kind of sucks.

Ideally, oclint-xcodebuild would incrementally apply new information it found in your latest xcodebuild log onto your existing clang compilation database (feature request here).

In the meantime, I’ve written a Node.js script that you can use to do the merging yourself:

# Install Node.js:
brew install nodejs

# Download merger script:
wget -O clang_compilation_db_merger.js


Update your build script to generate a temporary clang compilation db each time and then merge this information into your main JSON file:

oclint-xcodebuild -o compile_commands-tmp.json xcodebuild.log
node clang_compilation_db_merger.js compile_commands-tmp.json compile_commands.json

With this set up, you only need to do a clean build once, and henceforth your compilation database will be incrementally updated each time you build (once oclint-xcodebuild supports incremental updating, you can remove this manual merge step).

Future Work

It would be nice if compile_commands.json was completely portable – that way you could share the file between machines, avoid the need to do an initial clean build, and even get Objective-C code completion on linux! However due to things like precompiled headers (.pch) and header maps (.hmap) you’ll probably still have some references to DerivedData hanging around meaning that some files in your project may not work unless you build the project on that machine with xcodebuild. I’ll report back if I get around to figuring out how to solve that..

In the meantime, enjoy iOS code completion inside Vim! I’m using it successfully for projects that mix Objective-C, Objective-C++, C and C++ (basically, anything that Clang supports). Unlike Xcode, there’s no UI-blocking indexing step, and you still get all the sweet IntelliType features you’d expect from a semantic completion engine (jump to definition, compiler warnings in the margins, method signatures…). And of course, as YCM continues to rapidly improve you’ll get to enjoy all the new features that come down the pipeline.

Why Vim is the Future

Xcode is a great IDE. Unfortunately it’s a pretty crappy editor. At least, that’s how it feels coming from Vim.

I do a lot of iOS development these days. I really tried to like Xcode: I read the docs, I learnt the keyboard shortcuts, I watched screencasts.. but after 6 months I didn’t feel like I’d gotten much return on my investment. I was still wasting time reaching for my mouse, waiting for the UI to respond.. and every time I noticed something that was slowing me down there didn’t appear to be much I could do to speed up my workflow. A shiny graphical IDE doesn’t mean much to me if it doesn’t help me get my job done faster.

I’d settle for a tenth of the power of Vim. Just simple stuff like filtering the current selection through an external program, recording a macro to eliminate the tedium from a repetitive task, reflowing a multiline comment.. Those sorts of tasks happen much more regularly and have the potential to save me much more time than impressive IDE features such as language-aware refactoring or class creation dialogs.

So these days I’m back in an editor that actually feels like it was optimised for editing code. I fire up Xcode whenever I need to debug, but for everything else I’m back inside  the terminal.

The fundamental reason I try to encourage people to use editors like Vim or Emacs over IDEs is that any time you invest in learning power features repays you across every language you code in. I regularly switch between Objective-C(++), Java and Javascript. Who knows what languages I’ll be using next year. That’s why Vim really is the future for me.

Benchmarking Node.js websocket performance

I’ve been playing around with node.js and websockets, using Primus.IO which provides a common interface on top of various realtime frameworks (Engine.IO, Socket.IO, BrowserChannel, SockJS, ws etc..).

One of the primary things I’m interested in when looking at different frameworks is their performance characteristics. Primus makes switching between frameworks super easy (often it’s just a one-line change), but in order to actually compare performance you need some benchmarking code..

After some extended hunting I found a nice litle npm library called websocket-bench, written by the folks at M6Web. The current version only supports Socket.IO and Faye, but I’ve just sent out a pull request adding Primus support (meaning you can benchmark every realtime framework that Primus supports).

As a cool side-note, M6Web are based in France and I only stumbled across their benchmarking library as a result of reading this article they published in French (something I wouldn’t have been able to read six months ago).