windows - Adding/displaying special characters in Vim

07
2014-07
  • ahmed

    I currently work on a project where I need to write the Unicode characters ɛ and ħ, which I just added to my keyboard.

    It works great almost everywhere, even in the command prompt, except for Vim who displays a question mark ? instead of ɛ and h instead of ħ.

    The encoding is set to utf-8 and using the :digraphs isn't much of a help since even there, a lot of characters are shown simply as ?.

    What can I do to fix this ?

    EDIT
    I use the Consolas font which has both the aforementioned characters.

    • Answers
  • Ben

    From the comment questions, even though the question states "encoding is set to utf-8", it appears Vim's encoding is NOT set to utf-8. It is still latin1 (ISO-8859-1). This means Vim does not know how to represent those special characters in memory, nor save them to a file.

    For Vim to correctly write "special" characters, it needs 2 things:

    1. An 'encoding' setting that can represent that character in memory, usually utf-8
    2. An empty value, or a setting that can represent that character on-disk for 'fileencoding'. Usually this is also utf-8, or empty.

    To properly read a file containing these characters, Vim again needs these two things. Vim can automatically detect the proper value for 'fileencoding', if your 'fileencodings' option is set properly.

    'fileencodings' is a comma-separated list that tells Vim (in order) which encodings to try. "ucs-bom" as in the comments on your question will detect unicode, but only if there is a BOM (byte-order mark). Additionally since your Vim's 'encoding' is still "latin1" it wouldn't do you any good anyway if Vim DID detect unicode.

    Note both 'encoding' and 'fileencodings' should be very close to the top of your .vimrc, before ANY mappings, menus, string options, or register settings. Otherwise existing text in those areas will be misinterpreted by Vim.

    See the help for each of these options, and the wiki page on configuring Vim for Unicode, for details.

    If Vim does't detect encoding properly on some files even with the correct settings, you can either:

    1. Set it manually, like :e ++enc=utf-8 myfile.txt
    2. Install a Vim plugin that will automatically detect file encoding

    • Related Question

    How to use UTF-8 in vim on Mac OS X?
  • Tadeusz A. Kadłubowski

    I want to edit UTF-8 documents with vim (7.2 installed via MacPorts, big feautre set, iconv support enabled, multi-byte support enabled) on Mac OS X 10.4 within terminal.app.

    Terminal.app is configured to use Monaco font (which has good Unicode coverage) and use UTF-8 as the character set encoding.

    Keyboard map is set up correctly. I can enter some localized characters like „zażółć” and even quotes around that… (yeah, and an elypsis).

    I've done my best to set up the environment:

    LC_ALL=pl_PL.UTF-8
LC_CTYPE=pl_PL.UTF-8
LANG=pl_PL.UTF-8
export LC_ALL
export LC_CTYPE
export LANG

    I have no encoding, fileencoding or termencoding set in .vimrc, so that it should default to what's set in the locale.

    What else have I missed? I can't enter non-ASCII UTF-8 characters in vim. It is interpreted as single-byte garbage rather than wider UTF-8 characters.


    • Related Answers
  • CoreSandello

    Check out this:

    (Thanks to Peter Vohmann for this Q&A.) In Terminal.app go the the Terminal (main) menu and choose Window Settings. Select Emulation from the popup menu, un-check the item "Escape non-ASCII characters". Then select Display from the popup menu, set Character Set Encoding to Unicode (UTF-8), if desired. Click on "Use settings as Default."

    (from MacVim Site)

    As far as I remember, 10.4 Terminal.app has some troubles, when dealing with UTF-8; checking setting above would, probably, help. As an alternative solution, consider using MacVim or iTerm as terminal application.

    Update: as Ben Stiglitz mentioned in comments, 10.4 Terminal is OK, but 10.4 bundled shells are not.

  • donut

    I don't know about Vim in the Terminal, but I have no troubles entering Korean characters in MacVim. This is with no extra setup, just as it came.