如何执行 ISO-8859-1 到 UTF-8 文本文件的转换，同时不更改任何已经有效的 UTF-8 字符

Question 1

Python 的 UTF-8 解码器可以将非 UTF-8 字符作为特殊代码点 U+DC00 – U+DCFF 传递（这在 UTF-8 中通常是非法的）。之后可以找到它们并将其重新解码为其他内容：

#!/usr/bin/env python3
import argparse
import re
import sys

parser = argparse.ArgumentParser()
parser.add_argument("input")
args = parser.parse_args()

with open(args.input, "rb") as fh:
    buf = fh.read()
    buf = buf.decode("utf-8", errors="surrogateescape")
    buf = re.sub(r"[\udc00-\udcff]+",
                 lambda m: (m.group(0)
                             .encode("utf-8", errors="surrogateescape")
                             .decode("iso8859-1")),
                 buf)
    sys.stdout.write(buf)

您也可以手动完成：

#!/usr/bin/env python3
import argparse
import sys

parser = argparse.ArgumentParser()
parser.add_argument("input")
args = parser.parse_args()

def decipher_runes(fh):
    curr = None
    more = 0

    while buf := fh.read(1):
        ch = buf[0]
        if more == 0:
            # Expect a UTF-8 leading byte
            curr = bytearray([ch])
            if   ch & 0b10000000 == 0b00000000: more = 0
            elif ch & 0b11100000 == 0b11000000: more = 1
            elif ch & 0b11110000 == 0b11100000: more = 2
            elif ch & 0b11111000 == 0b11110000: more = 3
            elif ch & 0b11111100 == 0b11111000: more = 4
            elif ch & 0b11111110 == 0b11111100: more = 5
            else:                               more = -1
        else:
            # Expect a continuation byte
            curr.append(ch)
            if ch & 0b11000000 == 0b10000000: more -= 1
            else:                             more = -1

        if more < 0:
            more = 0
            yield curr.decode("iso8859-1")
        elif more == 0:
            yield curr.decode("utf-8")

    if more:
        yield curr.decode("iso8859-1")

with open(args.input, "rb") as fh:
    for ch in decipher_runes(fh):
        sys.stdout.write(ch)

Answer

Python 的 UTF-8 解码器可以将非 UTF-8 字符作为特殊代码点 U+DC00 – U+DCFF 传递（这在 UTF-8 中通常是非法的）。之后可以找到它们并将其重新解码为其他内容：

#!/usr/bin/env python3
import argparse
import re
import sys

parser = argparse.ArgumentParser()
parser.add_argument("input")
args = parser.parse_args()

with open(args.input, "rb") as fh:
    buf = fh.read()
    buf = buf.decode("utf-8", errors="surrogateescape")
    buf = re.sub(r"[\udc00-\udcff]+",
                 lambda m: (m.group(0)
                             .encode("utf-8", errors="surrogateescape")
                             .decode("iso8859-1")),
                 buf)
    sys.stdout.write(buf)

您也可以手动完成：

#!/usr/bin/env python3
import argparse
import sys

parser = argparse.ArgumentParser()
parser.add_argument("input")
args = parser.parse_args()

def decipher_runes(fh):
    curr = None
    more = 0

    while buf := fh.read(1):
        ch = buf[0]
        if more == 0:
            # Expect a UTF-8 leading byte
            curr = bytearray([ch])
            if   ch & 0b10000000 == 0b00000000: more = 0
            elif ch & 0b11100000 == 0b11000000: more = 1
            elif ch & 0b11110000 == 0b11100000: more = 2
            elif ch & 0b11111000 == 0b11110000: more = 3
            elif ch & 0b11111100 == 0b11111000: more = 4
            elif ch & 0b11111110 == 0b11111100: more = 5
            else:                               more = -1
        else:
            # Expect a continuation byte
            curr.append(ch)
            if ch & 0b11000000 == 0b10000000: more -= 1
            else:                             more = -1

        if more < 0:
            more = 0
            yield curr.decode("iso8859-1")
        elif more == 0:
            yield curr.decode("utf-8")

    if more:
        yield curr.decode("iso8859-1")

with open(args.input, "rb") as fh:
    for ch in decipher_runes(fh):
        sys.stdout.write(ch)

Question 2

.NET 允许您创建自定义编码器/解码器除了默认选项之外，还有无效字符（对无效字符抛出异常或用用户指定的字符串替换它们），因此您可以使用任何基于 .NET 的语言并编写自己的解码器将 ISO-8859-1 字符转换为 UTF-8。我编写了一个简单的 PowerShell 脚本来执行此操作。将 PowerShell 安装到 Linux如果你没有，请将以下脚本保存为convert.ps1

class Decoder88591FallbackBuffer : System.Text.DecoderFallbackBuffer {
    [char]$c; [int]$idx # Internal decoder state
    
    Decoder88591FallbackBuffer() { $this.Reset() }
    
    [bool] Fallback([byte[]]$bytesUnknown, [int]$index) {
        $this.idx = 1; $this.c = [char]::ConvertFromUtf32($bytesUnknown[0])
        return $true
    }

    [char] GetNextChar() {
        if ($this.idx -eq 1) {
            $this.idx = 2; return $this.c
        }
        return 0
    }

    [bool] MovePrevious() {
        if ($this.idx -eq 2) { $this.idx = 1; return $true }
        return $false
    }

    [int] get_Remaining() {
        if ($this.idx -eq 0) {
            if ($this.c -eq 0) { return 0 } else {return 1 }
        }
        return 0
    }

    [void] Reset() { $this.c = 0; $this.idx = 0 }
}

class Decoder88591Fallback : System.Text.DecoderFallback {
    Decoder88591Fallback() {}
    
    [Text.DecoderFallbackBuffer] CreateFallbackBuffer() {
        return [Decoder88591FallbackBuffer]::new();
    }

    [int] get_MaxCharCount() { return 1; }
}

$enc = [Text.Encoding]::GetEncoding(65001, `
    [Text.EncoderReplacementFallback]::new(), [Decoder88591Fallback]::new())
if ($PSVersionTable.PSVersion -ge [version]"6.0") {
    $content = Get-Content -AsByteStream -Raw $args[0]
} else {
    $content = Get-Content -Encoding Byte -Raw $args[0]
}
Set-Content -Path $args[1] -Encoding UTF8 -Value ($enc.GetString($content))

然后运行命令

./convert.ps1 testing.txt testing_out.txt

如果你想让它适用于 Windows-1252，那么只需更改[char]::ConvertFromUtf32($bytesUnknown[0])为[Text.Encoding]::GetEncoding(1252).GetString($bytesUnknown)[0]

示例输出：

$ cat -v 测试2.txt
这M-4s ISO-8859-1

Bx M-0 M-1 M-2 M-3 M-4 M-5 M-6 M-7 M-8 M-9 M-: M-; M- M-?
Cx M-@ MA MB MC MD ME MF MG MH MI MJ MK ML MM MN MO
Dx MP MQ MR MS MT MU MV MW MX MY MZ M-[ M-\ M-] M-^ M-_
Ex M-` Ma Mb Mc Md Me Mf Mg Mh Mi Mj Mk Ml Mm Mn Mo
Fx Mp Mq Mr Ms Mt Mu Mv Mw Mx My Mz M-{ M-| M-} M-~ M-^?

这M-bM-^@M-^Ys UTF-8

Bx M-BM-0 M-BM-1 M-BM-2 M-BM-3 M-BM-4 M-BM-5 M-BM-6 M-BM-7 M-BM-8 M-BM-9 M-BM-：M-BM-；M-BM- M-BM-？
Cx M-CM-^@ M-CM-^A M-CM-^B M-CM-^C M-CM-^D M-CM-^E M-CM-^F M-CM-^G M-CM-^H M-CM-^I M-CM-^J M-CM-^K M-CM-^L M-CM-^M M-CM-^N M-CM-^O
Dx M-CM-^P M-CM-^Q M-CM-^R M-CM-^S M-CM-^T M-CM-^U M-CM-^V M-CM-^W M-CM-^X M-CM-^Y M-CM-^Z M-CM-^[ M-CM-^\ M-CM-^] M-CM-^^ M-CM-^_
例如 M-CM- M-CM-! M-CM-" M-CM-# M-CM-$ M-CM-% M-CM-& M-CM-' M-CM-( M-CM-) M-CM-* M-CM-+ M-CM-, M-CM-- M-CM-. M-CM-/
Fx M-CM-0 M-CM-1 M-CM-2 M-CM-3 M-CM-4 M-CM-5 M-CM-6 M-CM-7 M-CM-8 M-CM-9 M-CM-: M-CM-; M-CM- M-CM-?

$ cat 测试2_out.txt
这是 ISO-8859-1

Bx ° ± ² ³ ´ µ ¶ · ¸ ¹ º » ¼ ½ ¾ ¿
Cx À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï
Dx Ð Ñ Ò Ó Ô Õ Ö × Ø Ù Ú Û Ü Ý Þ ß
Ex à á â ã ä å æ ç è é ê ë ì í î ï
Fx ð ñ ò ó ô õ ö ÷ ø ù ú û ü ý þ ÿ

这是 UTF-8

Bx ° ± ² ³ ´ µ ¶ · ¸ ¹ º » ¼ ½ ¾ ¿
Cx À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï
Dx Ð Ñ Ò Ó Ô Õ Ö × Ø Ù Ú Û Ü Ý Þ ß
Ex à á â ã ä å æ ç è é ê ë ì í î ï
Fx ð ñ ò ó ô õ ö ÷ ø ù ú û ü ý þ ÿ

注意od -c或者hd（大多数 Linux 发行版默认包含）会更好，cat -v因为它们可以更轻松地检查字节值

$ hd 测试.txt
00000000 54 68 69 73 b4 73 20 49 53 4f 2d 38 38 35 39 2d |这.s ISO-8859-|
00000010 31 0a 54 68 69 73 e2 80 99 73 20 55 54 46 2d 38 |1.这...是 UTF-8|
00000020 0a 0a |..|
00000022

$ od -c 测试.txt
0000000 这是 264 的 ISO - 8 8 5 9 -
0000020 1 \n 这是 342 200 231 s UTF-8
0000040 \n \n
0000042

更多信息请阅读

Answer