Perl version 5.6 introduced partial Unicode support. Perl 5.8 has much improved support for it, but still may be very cumbersome to use. I propose solutions to some problems, mostly for Perl 5.8 users.
In Perl 5.8 Unicode support is described in perluniintro, perlunicode, Encode, utf8, mentioned in encoding, -f open manpages (read them with perldoc tool, for instance).
The major problem with this documentation is its volume. Normal programmer won’t read it. (Cool programmers don’t read documentation, as we all know.) Most programmers don’t even need to read it all, because to work with Unicode you just need to know the basic facts and rules.
I somehow got into several different kinds of trouble with Unicode in Perl, both in 5.6 and 5.8, in several different projects. Always it was about processing and generating data in UTF8 encoding.
The two main problems I’ve seen are:
Having said the above, reading or at least browsing through the above mentioned manpages is still a good way to understand and solve your Unicode problems. If you don’t have time for that now, read on.
Perl版本5.6引入了部分Unicode支持。Perl 5.8對此有很大的改進性支持，但使用起來可能仍然很麻煩。我建議一些解決問題的方法，主要是針對Perl 5.8用戶。
在Perl 5.8中，Unicode的支持描述在Perluniintro, perlunicode, encode, utf8，在關於編碼(encoding)的內容當中。(例如，使用perldoc工具 -f manpage讀取它們)。
本文檔的主要問題是容量。普通程序員不會閱讀它。(眾所周知，很酷的程序員不會閱讀文檔。)，大多數程序員根本不需要讀完全部，所以為了要與Unicode一起工作，您只需要了解基本的事實和規則即可。
在幾個不同的專案中，我在Perl 5.6和5.8中遇到了幾種不同的Unicod 問題。始終是關於以UTF8編碼處理和產生的資料的問題。
我看到的兩個主要問題是：
話雖如此，閱讀或至少瀏覽上述手冊頁仍然是理解和解決Unicode問題的好方法。如果您現在沒有時間讀manpage，請繼續閱讀本文。
There is a distiction between bytes and characters.
There is a "utf8" flag on every scalar value, which might be "on" or "off".
(Attention! Here is the source of many many perl-unicode problems:) If you take a string with utf8 flag off and concatenate it with another string with utf8 flag on, perl will convert the first one to utf8.
This may sound ok and obvious. But then you think: how? Perl will need to know the encoding the string is, before converting it, and perl will try to guess it.
The algorithm perl uses in guessing is documented (uses some defaults and maybe checks your locale), but my suggestion is: never let perl do that, unless you have no choice. In my experience, this is the reason for double-encoded utf8 strings and it was the main source of headache.
If you process Unicode data in your scripts, always use utf8 to store and process your data and make sure perl knows you use it. (i.e. make sure all your unicode-strings have "utf8" flag on.)
Encode manpage will tell you about utf8on() and utf8off() functions, but don’t yet rush to use them.
There are better ways to do that.
字節和字符之間存在差異。
每個scalar值都有一個UTF-flag(UTF8標)，可能是"on"或"off"。
（注意！這是許多許多perl-unicode問題的來源:)如果您將UTF8標誌的字符串與另一個字符串與另一個帶有UTF8標誌的字符串相連，則Perl將將第一個轉換為UTF8 。 這聽起來還不錯，很明顯。但是您想：如何？ Perl將需要知道在將字符串轉換之前，Perl將嘗試猜測字符串。
記錄了Perl算法Perl在猜測中的使用（使用一些默認設備，也許可以檢查您的語言環境），但是我的建議是：除非您別無選擇，否則永遠不要讓Perl這樣做。 以我的經驗，這是進行雙重編碼UTF8字符串的原因，這是頭痛的主要來源。
如果您在腳本中處理Unicode數據，請始終使用UTF8來存儲和處理數據，並確保Perl知道您使用它。 （即，確保所有Unicode-string都有“ UTF8”標誌。） 編碼manpage會告訴您UTF8ON（）和UTF8OFF（）函數，但尚未急於使用它們。
有更好的方法可以做到這一點。
First of all, it is often already there, so you don’t need to worry. For instance, if you read your data in through an XML parser, you may assume that strings coming from it will be in UTF8 and will have utf8 flag on (unless you do something weird, like trying to get it in original form from the parser, which you shouldn’t anyway). If you read data from a file, there are several different ways to tell perl about it’s encoding.
In perl 5.6 there is a magic pack 'U', unpack ( 'U', …) construct, which can help. So if you open a file, which contains utf8 data, you read it into a variable and then you say:
$data = pack 'u<em>';
unpack('u</em>', $data);
And your $data now has "utf8" flag on.
In perl 5.8, you should use Encode::decode_utf8 in a similar way, but probably the same pack & unpack trick will do the job, I leave it for you to try. I use:
use Encode;
$data = Encode::decode_utf8($data);
(Do not forget though, that not every sequence of bytes is valid UTF8. So this operation may fail. See Encode manpage for error-handling.)
首先，它通常已經存在了，因此您不必擔心。例如，如果您通過XML parser(解析器)讀取數據，則可以假設來自UTF8的字符串將在UTF8中，並且會在上面有UTF8標誌（除非您做一些奇怪的事情，例如試圖以解析器的形式獲得原始形式，無論如何您都不應該）。
如果您從文件中讀取數據，則有幾種不同的方法來告訴Perl它的編碼。
在Perl 5.6中，有一個奇妙的 pack 'U', unpack ( 'U', …) construct (構造)，可以幫忙。開啟一個包含UTF8 data的檔案，將其讀入一個$data scalar，然後：
$data = pack 'u<em>';
unpack('u</em>', $data);
現在，您的$data就有了UTF8-flag。
在Perl 5.8中，類似的方式是使用Encode::decode_utf8。pack和unpack技巧應該也可以執行這項工作，所以我將其留給您嘗試。我呢則使用：
use Encode;
$data = Encode::decode_utf8($data);
（但是不要忘記，並非每個字節的每個序列都是有效的UTF8。因此，此操作可能會失敗。請參閱編碼MANPAGE以獲取錯誤處理。）
Let’s look at a real-life example. In ACIS we take HTML form input parameters through CGI in UTF8 encoding. We generate HTML in UTF8. To manipulate the user’s input, we need to tell perl that it is in UTF8:
use Encode;
my @parnames = $query -> param;
my $forminput = {};
foreach my $name ( @parnames ) {
my @val = $query -> param( $name );
foreach ( @val ) {
$ = Encode::decodeutf8( $ );
}
if( scalar( @val ) == 1 ) {
$forminput ->{$} = $val[0];
} else {
$forminput ->{$} = \@val;
}
}
|
An important thing is that the result of Encode::decodeutf8 doesn’t always have utf8 flag "on" and that is OK. If you decodeutf8 a pure-ASCII string, it won’t have the utf8 flag on. ASCII data is safe with regard to UTF-8 conversions: it doesn’t need any, so it is impossible to screw up.
例子 讓我們看一個現實生活中的例子。 在ACI中，我們通過UTF8編碼中的CGI採用HTML形式輸入參數。 我們在UTF8中生成HTML。 要操縱用戶的輸入，我們需要告訴Perl它在UTF8中：
使用Encode; 我的@parnames = $ query-> param; 我的$ forminput = {}; 對於我的$ name（@parnames）{ 我的@val = $ query-> param（$ name）; foreach（@val）{ $ _ = encode :: decodeutf8（$ ）; } if（標量（@val）== 1）{ $ forminput-> {$ } = $ val [0]; } 別的 { $ forminput-> {$ _} = \ @val; } }
重要的是，Eccode :: decodeutf8的結果並不總是在“ on”上具有utf8標誌。 如果您DECODEUTF8純ASCII字符串，則不會打開UTF8標誌。 對於UTF-8轉換，ASCII數據是安全的：它不需要，因此不可能搞砸。
The warning happens, when you output a Unicode-string (that means, a string with utf8 flag "on" and containing at least one unicode character outside ASCII range) on a non-unicode filehandle.
"What the f*ck 'Non-unicode filehandle?'" you could ask.
Perl 5.8 introduces PerlIO, a new Input/Output subsystem, which has the notion of a filehandle discipline layer. With a filehandle layer you can do on-the-fly transparent encoding conversions, or line-ending conversion. Say, if you open a file as:
open FILE, "<:encoding(iso-8859-7)", $filename;
it’s content will be assumed to be in iso-8859-7 encoding. Perl will use that to interprete file’s data correctly. (I.e. to convert it to internal UTF8). Basically, to get rid of the warning, you have two ways: one is wrong and the other is right. The wrong way is to turn off the utf8 flag on your data. Then the characters will turn into bytes, and it will print out smoothly.
The right way is to tell perl, that what your output is expected to be in UTF8. So, if you print to a file, open the file this way:
open FILE, ">:utf8", $filename;
If you print to standard output (or standard error), you can do this:
binmode( STDOUT, ":utf8" );
The Perl’s "There’s more than one way to do it" applies to Unicode support as much as to everything else. So if you do take a good look at the documentation, you’ll see that there are other ways, functions, tricks to fix (or break) your Unicode-aware script.
Ведущий: Иван Курманов, kurmanov /@/ openlib.org.
XHTML, CSS.
警告發生時，當您輸出一個UNicode-string（這意味著，一個帶有UTF8標誌的字符串“ ON”，並在非unicode fileHandle上輸出至少包含一個Unicode字符）。 “ f*ck'nonicode filehandle是什麼？”你可以問。
Perl 5.8介紹了一個新的輸入/輸出子系統Perlio，該系統具有FileHandle學科層的概念。 使用FileHandle層，您可以進行透明的編碼轉換或線路終止轉換。 說，如果您打開文件為：
打開文件，“ <：編碼（ISO-8859-7）”，$ filename;
它的內容將被認為是在ISO-8859-7編碼中。 Perl將使用它來解釋文件的數據正確
y。 （即將其轉換為內部UTF8）。 基本上，要擺脫警告，您有兩種方法：一個是錯誤的，另一個是正確的。 錯誤的方法是關閉數據上的UTF8標誌。 然後，字符將變成字節，並且將平穩打印出來。 正確的方法是告訴Perl，您的輸出預計將在UTF8中。 因此，如果您打印到文件，請以這種方式打開文件：
打開文件，“>：utf8”，$ filename;
如果您打印到標準輸出（或標準錯誤），則可以執行此操作：
binmode（stdout，“：utf8”）;
Perl的“有多種方法可以做到”，適用於Unicode的支持，以及其他所有方法。 因此，如果您確實對文檔進行了很好的了解，您會發現還有其他方法，功能，技巧可以修復（或破壞）Unicode-Awawawawawawawawawawawawawawawawawawawawawawawawe的腳本。
âcomouщ客：極 /@ / openlib.org。 XHTML，CSS。