Background

A few years ago, when I needed to add support to Vim for a new language, I simply searched for a Vim plugin for that language and installed it. It seemed everything was enabled by such a plugin, including syntax highlighting, autocompletion, outline extraction, and more.

However, things changed with the introduction of Language Server Protocol. I need to install a few plugins to support a single language.

How traditional language plugins work

In the early period of plain text processing, everything was simple and crude.

All we had in Vim was VimL, an interpreted script language, slow and not so powerful.

A language plugin worked roughly like this:

• File type detection: Language types are checked by file extensions, or shebang.
• Syntax highlighting: Source files are scanned for keywords, symbols, etc. and matching rules are generated based on them. Different groups apply different colors.
  • It hardly works with escaping, e.g. highlighting a complex regular expression in JavaScript.
• Autocompletion and Outlines: Source files are scanned for keywords, which are then fed into the autocompletion APIs.
  • Every single word that appears elsewhere is prompted for autocompletion, sometimes does not really help.
  • There is a famous tool for indexing C files and providing outlines, called ctags.

Sometimes a plugin is not actively maintained and there are successors overriding its matching rules. In such case we have multiple plugins but they still work as one, with one overriding another.

Language Server Protocol

Language Server Protocol is a game changer. It brings features from powerful IDEs to Vim.

With LSP, things become a lot more reliable in terms of syntax parsing, perhaps also a lot more complicated.

Here is how plugins work with LSP:

• File type detection: Same as before.
• Autocompletion and Outlines: Language server.
  • The language server parses the source files and feed the symbols back, as well as more information like types, references, diagnostics, etc.
  • Only valid values are prompted based on where the autocompletion happens.
  • Outlines are also generated by the language server based on syntax parsing.

Syntax Highlighting

Language servers are great. However, they are not responsible for syntax highlighting, even though they undertstand the syntax well.

Neovim, as a successor to Vim, introduces a native plugin called Treesitter for syntax highlighting.

It is AST based, i.e. it parses the source file into AST and provides accurate syntax information for highlighting. It is mucher lighter than a language server because it does not really care about the project, or the relationship between files. It just takes in one file and parses it immediately, and it is super fast.

There are several downsides for Treesitter:

• Binary needed.
  • Compilation is required for uncommon architectures like arm.
  • It took me quite a while when I first installed it on my Raspberry Pi.
• Limited language support.
  • Syntax highlighing for each language has to be implemented in treesitter.
  • The good news is that the project is actively maintained and it supports quite a few languages.
• Resource intensive.
  • Potentially posing challenges on low-profile devices.
• Neovim only.

Because of the reasons above, sometimes we still have to highlight syntax using the traditional way. There is a plugin maintaining highlighting rules for a lot of common languages: vim-polyglot.

How modern language plugins work

LSP is great, but Vim does not support it. Neovim supports LSP natively with limitations.

Luckily there are plugins that implement LSP for Vim, such as coc.nvim, which also allows us to write plugins with Node.js, and easily port VSCode extensions to Vim/Neovim thanks to the similar APIs.

This is another good reason why I use an LSP plugin despite it being natively supported by Neovim.

Here is an overview of how modern plugins work for a new language:

Plugins for Vim or Neovim?

Lua becomes a built-in language in Neovim as well as VimL. It is accepted by a lot of plugin authors since it is much simpler and more powerful than VimL.

So if a plugin is written in Lua, it only supports Neovim.

When Neovim came out, it aimed to be a faster Vim, with exactly the same APIs. In other words, most plugins written for Vim in VimL just work in Neovim.

However, a few new fancy APIs are introduced into Vim 8 to compete with Neovim while Neovim might have implemented them in different ways, such as floating windows, asynchronous jobs, etc. As a result, if a plugin uses the new APIs provided by one of them without an adapter, it probably does not work on the other.

Features	Vim	Neovim
VimL	✅	✅
New APIs in Vim 8+ / Neovim	❓	❓
Lua	❌	✅

So if you want to share configuration between Vim and Neovim, use plugins written in VimL. You may not be able to use fancy features like floating windows, but plugins like coc.nvim can still provide good experiences by connecting to a powerful extension system.

This is what I did for a long time.

Then I switched to Neovim only for lighter plugins and better syntax highlighting.