How to detect a character encoding?

I think googling "encoding heuristics" should help.

For example, https://gist.github.com/TaoK/945127

In the IHE context I onced faced the scenario that there was no given encoding in the XML declaration.
The rule was to use in such a case the character encoding which could be found in the HTTP HEADER.

For example: Content-Type: text/html; charset=utf-8

Is the file using BOM? If so you can check the header for the following signature: EF BB BF

This can be described as: $c(239, 187, 191)

Now keep in mind that most of editors abandoned the use of BOM in favor of digraphs and trigraphs detection heuristics as a fallback, yes, fallback. Because many assume you're already working with UTF-8 and won't work well with some charsets neither output BOM characters unless you tell it to use the desired charset.
 

You can try checking it against the US-ASCII table that goes from 0 to 127 code points, however that still wouldn't be 100% assertive about the stream containing UTF-8 characters.

We've got the same issue, but with an incoming HL7 feed with embedded, encoded characters - would be nice to be able to detect what's coming in, but I take it from this discussion that's not (reliably) possible. Don't really want to scan the whole text of every incoming message to heuristically look for possible encodings. Upstream say/think they are sending UTF-8, but we seem to be getting Window-1252, for the characters we've seen in the (limited) testing. Who knows what will come through the feed once it goes live!