Page encoding examples and encoding errors. HTML encoding

I hope I convinced you and you will use Unicode (by the way, "utf-8" and "Unicode" are synonyms or, to be more precise, utf-8 is one of the Unicode family encodings that has gained popularity among web developers) .

Now let's take a closer look at the file transcoding tools that I recommend you use, dear reader.

Tools for working with HTML file encodings

Actually, there are only three of them:

• PSPad. Free text editor, my favorite.
• . Another good text editor and also free.
• dreamweaver. Well, you are familiar with Dreamweaver from my .

Upload some HTML file to PSPad. And how can we understand what kind of encoding the loaded test subject has? Very simply in the status bar (below) everything is clearly written.

Encoding open html file windows-1251

And now, creating a new HTML document, let's take care of its encoding.

Going to the menu of my favorite PSPad. We are interested in the item Format. In it, we will put a checkmark in front of the utf-8 encoding.

And so the encoding of the future file is windows-1251

Now about how change html file encoding. Yes, it's very simple:

You need to click on the required encoding in the menu item Format and the encoding will change. After that, save the file, it is recoded, the job is done.

Concerning Notepad++ everything is very similar to the situation described above. Only to work with encodings you need to use the menu item Encodings.

The whole difference lies in the fact that in the case of Notepad ++ there are menu items specially designed for converting encodings. Convert... (Superfluous in my opinion, everything is simpler in PSPad and that's why I use it). Accordingly, it is on them that you need to click if you want to change the encodings of our HTML file.

Among other things, when saving in utf-8, we have a choice: without BOM or with BOM. We as webmasters need to use encoding UTF-8 (no BOM).

Here's what Wikipedia will answer us on the question " what is BOM»

To determine the Unicode representation format in a text file, a technique is used by which the U + FEFF character (zero-width non-breaking space), also called the Byte Order Mark, BOM, is written at the beginning of the text. This method makes it possible to distinguish between UTF-16LE and UTF-16BE because the U+FFFE character does not exist. It is also sometimes used to denote the UTF-8 format, although the concept of endianness does not apply to this format.

If you read the above text 10 times, scratch your head, it becomes clear: for utf-8 BOM we don't need. In addition, if you save the file with the php script in the encoding utf-8 with BOM, then it will not work, because the handler will not understand what kind of nonsense this is written at the beginning of the script file (I mean that same non-breaking space with zero width).

Well, well, it remains to take a close look at dreamweaver.

When creating a new file, pay attention to what encoding it will be created in. To do this, in the window for creating a new document File → New (Ctrl+N) use the button Preferences...

And see what is set as the default encoding:

The default encoding of the generated HTML file in Dreamweaver

Transcode open HTML file in Dreamweaver possible in dialogue Page Properties, which is launched from the menu Modify → Page Properties (Ctrl + J).

Select the required encoding, press OK and that's it, the task of transcoding is completed (but the BOM is still unnecessary, do not check the box).

Determination of the encoding by browsers

So, our HTML file is saved in the encoding we have chosen. Now let's deal with the question: How does the browser know about the encoding used in this HTML file?

There are three options here:

1. We ourselves tell the browser what encoding is set for this HTML file. This is done using the META tag.

In the above example, the browser is instructed that the downloaded HTML file is saved in the encoding utf-8.

If the HTML file is saved in windows-1251 encoding, then:

By the way, when transcoding files, do not forget to change the directives in the META tag to the actual ones. Dreamweaver, when changing the encoding, does this automatically, but in other text editors, you yourself need to match the applied encoding and the META tag directive.

The full HTML looks like this (I quote it to understand the question “where is the META tag with the encoding directive indicated” attention to the 4th line):

Well, etc.

2. Using the .htaccess file. Sometimes the server forcibly passes headers for downloaded HTML files and tells the browser the default encoding. In this case, the browser does not pay attention to the directives in the META tag, but displays the HTML file in the encoding reported by the server. In order for the file to be loaded in the encoding that you need (often the hosting forcibly specifies the windows-1251 encoding), a file called “.htaccess” is created in the root of the hosting directory.

This file is intended for additional server configuration. The effect of .htaccess directives applies to all files and subdirectories that are located in the directory where you saved the .htaccess file.

You can create this file, for example, in Total Commander by pressing the hot key combination Shift+F4 and specifying the name of the created file .htaccess. Next, in the text editor, directives for additional default encoding settings are specified.

For utf-8 encoded HTML files in .htaccess you need to write one line:

AddDefaultCharset UTF-8

For HTML files in Windows-1251 encoding:

AddDefaultCharsetWindows-1251

If your hosting is smart-wise and does not pay attention to these directives, then you can try:

Charsetdisable on AddDefaultCharset Off

If this does not work, then just ask your hoster what you should do to disable the default encoding :). It all depends on the specific server settings of the hosting provider.

3. PHP instruction specifying the default encoding. In the file that needs to be displayed in the desired encoding, regardless of the hosting provider's server settings, a directive with a php code is indicated at the very beginning:

This php code will send a server header with the browser's default encoding. In the above example, utf-8 encoding will be used to display the page.

Against such scrap, usually, there are no tricks in the server settings of the hosting provider.

I want to note that in order to process php instructions by the server, the html file must have the extension .php(for example index.php).

Do you have any other coding questions? Write in the comments. We need to solve these problems once and for all 🙂

Almost every newcomer to the field of web development, sooner or later, encounters coding problems in their projects. And then, as per the written script, the bombardment of forums begins with questions about how to defeat the hated " krakozyabry". The vast majority of problems have long been known and are treated quite easily, you just need to know " where does it hurt and what pill to take". Therefore, I propose to analyze the most popular errors due to which this problem appears and it is possible that my recommendations will save you from further collisions with them.

First, I strongly recommend that all documents were in the same encoding and the database, namely the fields with string data, had the same encoding. It is set when the database is created, or you can specify a comparison for each individual field. If you create a database using phpMyAdmin, then there should be no difficulties: the "Databases" tab > in the field under "Create a database" enter the name of your future database > next to the "Comparisons" drop-down list. If you create a database with an sql query, then write something like this:

CREATE DATABASE IF NOT EXISTS `my_db_name` CHARACTER SET utf8 COLLATE utf8_general_ci;

The choice of encoding is up to you, but I would advise you to choose for documents " UTF-8 without BOM" and comparison for base " utf8_general_ci" (unicode multilingual, case insensitive). Just do not forget to play it safe and make a dump before manipulating the database! I won’t describe here what BOM is, but if it’s very figurative and on the fingers, then this is such an invisible marker that was planned to distinguish between UTF-16LE and UTF-16BE encodings, but for some reason turned out to be unclaimed and now interferes with the web -developers live in peace;) BOM looks like a U+FEFF symbol and settles at the beginning of the document. Why is it still UTF-8? Here are at least a couple of reasons... You can easily display both Cyrillic and a quote from Al-Mutanabbi's poems or Chinese characters on the screen. This is because in the same windows-1251 encoding (cp1251) there are only 256 characters, while in UTF-8 there are about a hundred thousand of them, plus special characters, pictograms, icons, etc. If you are going to use ajax requests on your site, then this also adds a plus to the UTF-8 encoding, because the XMLHttpRequest object is friends with this encoding, and you will have to pervert with others and sometimes unsuccessfully. The same sitemap (sitemap.xml) that is used for indexing by search engines only works if this file is created with UTF-8 encoding. In addition, this encoding is the standard for many PHP functions to work, and is the standard recommended by the W3C.

When creating a new document, everything is clear, but what about an existing one in which it is desirable to change the encoding? One of the easiest ways is to open a document in Notepad++ , select " Encodings" and in the list " Convert to UTF-8 without BOM". Next, change the meta tag with the encoding definition:

And for php files, you can set the appropriate header, but only if the file is not included in another document, where such a header will already be sent earlier. This applies to both the header in the meta tag and the one sent by the header function:

Header("Content-Type: text/html; charset=utf-8");

We check the result in the browser. There may be several options here:

1. Everything is displayed fine and the issue is closed
2. Statically written data is displayed normally, but the data from the database is still the same "crazy"
3. Nothing has changed and the encoding remains crooked

Let's start with the last point. Happy owners of dedicated servers or VPS/VDS can change the encoding for the directive default_charset in the php.ini configuration file. For those who do not have access to php.ini or have, but need to change the encoding for only one site, you can use the .htaccess file by writing the following into it:

# in principle, the line below is enough: AddDefaultCharset UTF-8 # but sometimes, additional settings may be required: DefaultLanguage ru php_value default_charset "utf-8"

The .htaccess file is located at the root of your site. If you did not find it there, then we create it ourselves. In a regular notepad, create a document> " Save as"> select file type" All files"> in the "File name" field, write only the dot and the extension " .htaccess".

Let's move on to the second point - if the database was transferred to the desired encoding, but the data from it is displayed crookedly on the page. First, you need to make sure that the characters in the database itself are displayed normally. If the encoding did not "float" there, then you can either appeal to the configuration files again, or make a request immediately after connecting to the database:

SET NAMES utf8;

* I write the text of the request myself, but since I don’t know what extension you use to work with MySQL, I’ll show you several options:

// for legacy mysql_* $db = mysql_connect("localhost", "username", "password"); mysql_select_db("db_name", $db); mysql_query("SET NAMES utf8"); // for PDO and php versions below 5.3.6 $dbh = new PDO("mysql:host=localhost;dbname=db_name", "username", "password"); $dbh->exec("SET NAMES utf8"); // for PDO and php versions 5.3.6 and newer, can be specified directly when creating an object $dbh = new PDO("mysql:host=localhost;dbname=db_name;charset=utf8", "username", "password"); // or $db = new PDO("mysql:host=localhost;dbname=db_name", "username", "password", array(PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES utf8")); // for MySQLi $mysqli = new mysqli("localhost", "username", "password", "db_name"); $mysqli->set_charset("utf8");

Since I raised the issue of "outdated mysql_*", I want to draw your attention to the text highlighted in red in the php documentation. Worth thinking...
If you had one of the standard problems, then by following some or all of the above steps, the issue with the encoding will be resolved positively. But I would also like to mention some functions that may come in handy in non-standard situations. You can read more about them in the documentation, and I will just give a couple of examples without going into details:

Mb_internal_encoding() With this function, we can set or get the current script encoding: mb_internal_encoding("UTF-8"); // set echo mb_internal_encoding(); // no argument - get mb_http_input() and mb_http_output() Two functions that determine, set, or get the HTTP request or output character encoding: print_r(mb_http_input("I")); // determine the encoding of the http request input data mb_http_output("UTF-8"); // set encoding for http output echo mb_http_output(); // get the current character encoding of the http-output iconv() The function converts the characters of the string to the desired encoding: echo iconv("utf-8","cp1251","PџСЂРёРІРµС‚, РјРёСЂ!"); // Hello World! mb_convert_encoding() The function is similar to iconv(), but in my opinion it is better, because works more adequately. echo mb_convert_encoding("Привет, РјРёСЂ!","cp1251","utf-8"); // Hello World!

And in general, do not forget about the analogues of functions for working with multibyte strings. Most often, they have the same name, but with the prefix mb_. It's easy enough to feel the difference. Take, for example, the functions strlen() and mb_strlen() and conduct an experiment by measuring the length of the string:

// set internal encoding mb_internal_encoding("utf-8"); // no difference for latin characters echo strlen("incode"); // 6 echo mb_strlen("incode"); // 6 // But with Cyrillic it gives out - pichalka echo strlen("incode"); // 10 echo mb_strlen("incode"); // 5

Maybe someone does not need to explain this phenomenon, but for beginners I will explain: Cyrillic is encoded in two bytes, and strlen() counts exactly the number of bytes in a string, not the number of letters. So it turns out that five Cyrillic characters multiplied by two - we get 10. Chinese characters, if I'm not mistaken, are generally encoded in three bytes, so in the future for such cases, so that there are no misunderstandings, use the appropriate functions.

I repeat that these solutions are for common cases and in the vast majority, they solve the problem. But if you have a situation where all these methods did not work, then write here, we will try to figure it out together and supplement the article with a new "recipe for a headache";) Let me take my leave.