Wednesday, 23 July 2014


Mouse over the words to change the cursor.
auto

crosshair

default

e-resize

help

move

n-resize

ne-resize

nw-resize

pointer

progress

s-resize

se-resize

sw-resize

text

w-resize

wait

not-allowed

no-drop



Wednesday, 16 July 2014

7-Zip Command-Line

7z: 7-Zip icon
7-Zip is an effective compression program. The 7za.exe program is used to compress, extract and update files through the command line. It provides superior compression. It is open-source. This makes it easy to obtain and use.

Start

First you need to download the 7-Zip command line executable, 7za.exe. This is the exe you will use to run commands on archives. Please go to 7-zip.org and get the command line version.
Steps
Tip:For convenience and so you don't need to change environment paths, put the 7za.exe file in your user directory.
Next:Open the Windows console and test the 7za.exe program out with a few commands. Type in the exe name 7za and this will display.
7-Zip (A)  4.60 beta  Copyright (c) 1999-2008 Igor Pavlov  2008-08-19

Usage: 7za <command> [<switches>...] <archive_name> [<file_names>...]
       [<@listfiles...>]
That's the grammar of the commands you use on 7za.exe. The "command" is the main verb. Then you specify optional switches, the archive name (either source or destination archives) and files. My user directory is "C:\Users\Sam\".

Command a

Plus
You can use the "a" command with the single letter a. This command stands for "archive" or "add." Use it to put files in an archive. You have to specify the destination archive, and the source files (in that order).
On the test system, the directory C:\Users\Sam contains two files (file1.txt and file2.txt). The command puts those two files in an archive, and you need to type it into the command prompt.
C:\Users\Sam>7za a -t7z files.7z *.txt

7-Zip (A)  4.60 beta  Copyright (c) 1999-2008 Igor Pavlov  2008-08-19
Scanning

Creating archive files.7z

Compressing  file1.txt
Compressing  file2.txt

Everything is Ok

C:\Users\Sam>
7-Zip file manager in Microsoft Windows
Tip:To open your archive, right click on it and select 7-Zip -> Open archive. The screenshot shows the files compressed in files.7z.

Command d

Note
Here we see an example of the "d" command in 7-Zip command lines. This stands for 'delete' and is used much less often. It allows you to remove a certain file (or set of files) from inside an archive.
Note:You will need this if you use huge archives and need to save time. This is from the manual.
7z d archive.zip *.bak -r

7z:          use executable
d:           delete files
archive.zip: delete from this archive
*.bak:       only match bak files
-r:          traverse all subdirectories
Tip:You can also remove a single file from an archive with "d". This is more useful when you do not have a solid archive.

Command e

Mathematical constant e: Euler's number
Here we use the "e" command in your console window. The "e" stands for extract, and it means to unzip or expand an archive. You must specify the source archive always, and may also specify a destination.
Info:The "e" command extracts everything to a specified directory. Another command "x" can preserve directory structures in archives.
7z e archive.zip

7z:          executable
e:           use extract command
archive.zip: source archive you want to expand
Overwrite prompts:7-Zip will always prompt you if there is a file it needs to overwrite to extract the new file.
However:This can be problematic if you are scripting or embedding 7za.exe. In that case, see the -y switch.

Command l

List type
We next use the single-letter "l" (lowercase letter ell) command. The lowercase L is used to list the contents of archives and you probably will not need to use it often. I thought I would test it and show an example.
Next:This shows the listing of a solid archive.
The originals are 27216 bytes and 3888 bytes.
They compress down to 1030 bytes.
C:\Users\Sam>7za l files.7z

7-Zip (A)  4.60 beta  Copyright (c) 1999-2008 Igor Pavlov  2008-08-19

Listing archive: files.7z

Method = LZMA
Solid = +
Blocks = 1
Physical Size = 1202
Headers Size = 172

   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
2008-10-02 15:48:01 ....A        27216         1030  file1.txt
2008-10-02 15:47:45 ....A         3888               file2.txt
------------------- ----- ------------ ------------  ------------------------
     31104         1030  2 files, 0 folders
7-Zip command line output in Windows

Command t

Here we use the "t" command in the 7z program. This command allows you to test the integrity of archives. It stands for 'test' and is much less useful than the "-t" switch. Don't confuse the two. This one is used for diagnostics.
7z t archive.zip *.doc -r

7z:          use this executable
t:           test the specified archive
archive.zip: the archive you want to test
*.doc:       test all these files in the archive
-r:          recurse all child directories

Command u

The "u" command in 7-Zip stands for update. This is a useful command and is great when you want to replace old files in your archive with newer files. This prevents needing to decompress and recompress the entire archive.
7z u archive.zip *.doc

7z:          executable name
u:           update command
archive.zip: archive you want to update files in
*.doc:       only update these files (Word documents)
Warning:The "u" command doesn't work with solid archives. A solid archive is one where all the files are compressed together.
So:This means that you can't update specific files in solid archives with the "u" command.

Switch m

This section provides information
We can change the optimization settings in 7-Zip on the command line. This is the most important and useful option you can use. It specifies the method of compression. Here I will show a bunch of options, and also some examples.
Compression levels

Switch -mx0: Don't compress at all.
      This is called "copy mode."

Switch -mx1: Low compression.
      This is called "fastest" mode.

Switch -mx3: Fast compression mode.
      Will automatically set various parameters.

Switch -mx5: Same as above, but "normal."

Switch -mx7: This means "maximum" compression.

Switch -mx9: This means "ultra" compression.
      You probably want to use this.

Switch m, advanced

Squares
Here are advanced compression method (-m) switches. The first three are usually of limited use, but you might benefit from tweaking them. My experience is that manual optimizations to these options doesn't produce big benefits.
Switch -mfb:Specifies number of fast bytes.
Sometimes helps with "sparse" files.
Don't bother.
Switch -mpass:Number of passes for deflate compression. Don't bother with this. Automatically set with levels.
Switch -md:This specifies dictionary size.
It is automatically set, so don't bother.
Switch -mmt:Enable multithreading.
Use if you have quad-core and a huge archive.
Specify "on" or "off".
This may be enabled by default.

Command x

This command is like "e" except it preserves the full paths.
If you have an elaborate
or important directory structure,
use this option. This would be most useful for system backups or really big backups. Here's the example syntax.
7z x archive.zip

7z:          executable name
x:           use the extract command
archive.zip: the archive you want to extract all the files from

Switch t type

Here I show how to specify the precise archive type you want to create. Note that you can specify any filename you want for any type. But some extensions are recommended—they are standard.
Type switches

   Switch: -t7z
   Format: 7Z
Example filename: archive.7z (default option)

   Switch: -tgzip
   Format: GZIP
Example filename: archive.gzip, archive.gz

   Switch: -tzip
   Format: ZIP
Example filename: archive.zip (compatible)

   Switch: -tbzip2
   Format: BZIP2
Example filename: archive.bzip2

   Switch: -ttar
   Format: TAR
Example filename: tarball.tar (UNIX and Linux)

   Switch: -tiso
   Format: ISO
Example filename: image.iso (may not be supported)

   Switch: -tudf
   Format: UDF
Example filename: disk.udf
Also, the 7-Zip manual provides some useful examples for type switches. It shows the -tiso and -tudf switches. These are not the most common. Almost all of the examples in this document use -t switches.
7z a -tiso archive.iso
7z a -tudf archive.udf

7z:                         executable name
a:                          add to archive
-tiso or -tudf:             format of archive to create
archive.iso or archive.udf: name of archive to create

Solid archives

7z is the only file format in 7-Zip that you can specify whether the archive is solid or not. Solid means all the files are compressed as one. It makes it impossible to use the "u" command to update individual files.
Switch -ms=on:Enable solid mode.
This is the default so you won't often need to specify it.
Switch -ms=off:Disable solid mode.
Useful when you need to update individual files.
Will reduce compression ratios normally.

7z archives

Programming tip
You can change many values
and switches on 7z archives,
with endless permutations. Some things you can change are dictionary sizes, FastBytes values, MatchFinder values, and filters. Normally you don't need to deal with these.

PPMd

Page: lines of text
With the 7z format, you can specify the algorithm. PPMd is fast and effective for compressing plain text files. It is ideal for large collections of Word documents. PPMd does not perform as well on files containing binary data.
PPMd switch -mmem=24b, -mmem=24k, -mmem=24m:These control the amount of memory you use.
They are useful and higher is normally better.
PPMd switch -mo=2, -mo=32:These specify the model order in PPMd.
They are not normally useful.
Question and answer
When should I use PPMd? You should use PPMd when you have a large corpus (body) of text.
This could include HTML
or other formatting,
but plain text should dominate. It can improve ratios by around 30% on some datasets.

Example commands

Here I show the example compression commands from the 7-Zip manual. I demonstrated simple ones at the start of this document. These are more complex. We use more features of the 7-Zip command line.
7z a -tzip archive.zip *.jpg -mx0

7z:          name of executable
a:           add to archive command
-tzip:       specify a ZIP archive (useful for compatibility)
archive.zip: destination archive
*.jpg:       only add jpg files to archive
-mx0:        don't compress, just copy
      useful for already-compressed files
Example of 7z format. This next command line shows how to create a solid 7z archive of program files (executables). It uses multithreading mode, which means it will be fast on a dual core machine.
7z a -t7z archive.7z *.exe *.dll -ms -mmt

7z:         name of executable
a:          archive command specified
-t7z:       use 7z file type (less compatible and smaller results)
archive.7z: destination archive file
*.exe:      include all *.exe files in directory in new archive
*.dll:      include all *.dll files in new archive
-ms:        create solid archive (default)
-mmt:       multithread the operation (faster)

Create PPMd archive

PPMd is an extraordinary algorithm for compressing text and is relatively new. Here I show a command in the 7-Zip manual that compresses all the text files in the working directory. It creates a PPMd archive.
Tip:The command is useful because you will normally want to only compress text files with PPMd.
7z a -t7z archive.7z *.txt -m0=PPMd

7z:         executable name/path
a:          add command specified
-t7z:       use the 7z format (needed for PPMd)
archive.7z: destination archive file
*.txt:      select all text files
-mo=PPMd:   compress with this algorithm

Switch o

We show the "o" switch on the 7-Zip command line. Sometimes you do not want to extract to the current directory. This is where -o can come in handy. Use this to set the destination directory.
7z x archive.zip -oC:\Doc

7z:          executable name
x:           extract archive with paths intact
archive.zip: archive to extract files from
-oC:\Doc:    extract all files to the Doc folder on the C: drive

Switch p

We can use the "-p" switch, which refers to the word "password". This is really helpful when security and encryption is involved. You can specify a password on the command line. The syntax is a bit funky.
7za a pw.7z *.txt -pSECRET

7za:      name and path of 7-Zip executable
a:        add to archive
pw.7z:    name of destination archive
*.txt:    add all text files to destination archive
-pSECRET: specify the password "SECRET"
Opening password-protected archives. This next console output shows what happens when you try to open the password-protected archive. The password here is SECRET, which will allow the archive to be extracted.
C:\Users\Sam>7za x pw.7z

7-Zip (A)  4.60 beta  Copyright (c) 1999-2008 Igor Pavlov  2008-08-19

Processing archive: pw.7z

Enter password:
Header encryption:Add -mhe to encrypt headers. The password command will automatically deal with encrypted headers.
Tip:Remember, encrypted headers will hide the names of the files in your archive.

More switches

Here we take a closer look at more switches that are of limited use. They are useful to know, however, if you ever need to use them. Usually you can do better just by using the defaults that are slightly adjusted for your requirement.
Switch -ssc:Specify case-sensitive mode.
The default is -ssc- on Windows (insensitive).
The default is -scc on Linux (sensitive).
Switch -ssw:Compress locked files.
You can try this if you have problems opening files.
Switch -w:Set working directory.
You can use this when you want to specify temp folders.

Case-sensitive

Lowercase and uppercase words
We can use case-insensitive file names in the 7-Zip command line.
For those of you who use both Linux
and Windows,
the case-sensitive option is useful. I will show my own example here with some explanation.
7za.exe a archive.7z Z*.* -ssc

7za.exe:    7-Zip command-line executable path and name
a:          archive command
archive.7z: add files to this target archive
Z*.*:       select only files whose first letter is a capital Z

Switch v

Megabyte: MB
You can use the "v" switch on the command line. In data compression, a volume is a segment of a data set that is a certain number of bytes long. The volume switch specifies the exact size in bytes, kilobytes or megabytes.
Also:You can specify sequential volumes with the "v" switch on the 7za.exe command line.

Switch ao

The "ao" switch allows you to specify whether you want to overwrite old files. Be careful—you cannot restore an overwritten file normally. It takes another argument. Back up your data by copying the files in your file manager first.
Switch -aoa:This switch overwrites all destination files.
Use it when the new versions are preferred.
Switch -aos:Skip over existing files without overwriting. Use this for files where the earliest version is most important.
Switch -aou:Avoid name collisions. New files extracted will have a number appending to their names. You will have to deal with them later.
Switch -aot:Rename existing files. This will not rename the new files, just the old ones already there.
Example of the switches

7z x test.zip -aoa

7z:       use the 7-zip executable
x:        use the extract command
test.zip: extract files from this archive
-aoa:     overwrite all existing files. risky!

Multiple files

This section addresses adding multiple files to an archive. To add many files to one archive, please use the "a" command and the wildcard * symbol. Specify the name of the destination archive file and the source files afterwards.
Tip:Please read more in the section covering the "a" command, found in the previous part of this page.
How do I add many files with a specific extension? Use the "a" command and the wildcard * symbol, but specify the extension after the wildcard. For example, *.txt means all text files. You can use the wildcard anywhere.
How can I add many files from an entire subdirectory? Specify just the directory name. You do not need to use a wildcard. The 7-Zip manual helpfully shows this example. It specifies an entire directory called "subdir".
7z a -tzip archive.zip subdir\

7z:          use executable
a:           add to archive
-tzip:       use zip compression
archive.zip: create this archive
subdir\:     source directory

Formats

GZIP compression
This section answers questions about choosing formats. First, to use GZip compression, please specify the "-tgzip" option for the type switch. This makes a great way to compress files on your web server for HTTP compression.
How do I use BZip2? You can use BZip2 by specifying the "-tbzip2" switch. This can be combined with any compression level in the above charts. The different modes in 7-Zip automatically use many different settings.
How do I use 7z format? By specifying the "-t7z" switch for type. Or you can simply omit the type switch and that will default to 7z. This format offers the greatest compression ratios, but it does not work in all places.

Prompts

You can stop 7-Zip from displaying prompts. Please use the -y switch. This will assume a yes answer to all prompts. Use this only when you are confident that you are not going to lose any data.

Questions

There are many more possibilities and usages of the 7-Zip program on the command line in both Windows and Linux. This section answers some questions I had when doing this research, and also some questions that you may have.
Why can't I update my archive? It is probably a solid archive. 7z archives are by default solid archives—all the files are compressed together. Change the archive not to be solid if you want to update it. Search this page for "solid".
Can I specify the output directory? Yes—please use the "e" command and combine it with the -o switch. The syntax with -o is a bit funny so I will show the example from the 7-Zip help file. Here's how it works.
7z e archive.zip -oC:\soft *.cpp -r

7z:          executable
e:           use extract command
archive.zip: source archive you want to extract from
-oC:\soft:   the destination folder (-o is the switch and C:\soft is the argument)
*.cpp:       only extract cpp files (C++)
-r:          traverse all subdirectories
How can I see what's inside an archive? Use the "l" command as shown above. You might want to use "l" in a utility that you run from a command line to make sure your batch archiving properly works.
Warning: exclamation mark
How can I exclude certain files? Near the start we saw how to add files based on filters, but sometimes you want to manually exclude certain files. Use the -x switch, followed immediately with an exclamation mark and then the filename.
So:If you want to exclude "file1.txt", use the switch "-x!file1.txt". Please include the hyphen and exclamation.
How can I replace files already on disk with new files? By using the -ao switch, described above. There are other options, and it is usually a better idea to use one of the renaming options (-aou or -aot).
Can I ignore extracting files already on disk? Yes—please specify the -aos option, which means "skip overwriting files." This will cause 7za.exe to not copy the newer files out of the archive.
Note:Use -aos if your files don't change over time and overwriting would just be a waste.

Embed

Framework: NET
You can embed 7-Zip in a Windows .NET program using the tutorial in my article about .NET 7-Zip. This yields the same great compression but in your own GUI. The link shows some compression ratios.

Internal settings

You can change internal settings. You do not need to do this normally, as they are set automatically. I recommend just using the mx=0 (and 3, 5, 7, 9) settings. An in-depth study would be fascinating.
What values can I change in the internals? You can change compression filters, which change behaviors on executable files such as *.exe and *.dll. You can enable header compression and encryption (-mhc=on and -mhe=on).
Tip:Header compression is by default enabled.
Encryption must be explicitly enabled.

AdvanceCOMP

You can use AdvanceCOMP to improve compression ratios. The improvement is often small, less than 1%. 7-Zip and AdvanceCOMP use the same Deflate encoder, but AdvanceCOMP has more options and is more fine-grained.

Summary

We invoked 7-Zip on the command line. This provides superior compression with an open-source tool. We created new archives, added to existing archives, used different formats for compression, and used various strengths of compression.
So:I suggest you try the different options: experiment with balancing time required and strength of compression.

Tuesday, 15 July 2014

HAProxy to setup MySQL load balancing


HAProxy is an open source software which can load balance HTTP and TCP servers. In the 
previous article on HAProxy we configured load balancing for HTTP and in this one we'll do the same for MySQL. All your MySQL servers have to be configured to perform Master-Master replication as load balancing involves both reading and writing to all the backends.

The following three droplets will be used in this article:
Droplet 1 - Load Balancer
Hostname: haproxy
OS: Ubuntu Private IP: 10.0.0.100
Droplet 2 - Node 1
Hostname: mysql-1
OS: Debian 7 Private IP: 10.0.0.1
Droplet 2 - Node 2
Hostname: mysql-2
OS: Debian 7 Private IP: 10.0.0.2
Before proceeding, make sure all MySQL servers are up, running and are properly replicating database writes.

Prepare MySQL Servers

We need to prepare the MySQL servers by creating two additional users for HAProxy. The first user will be used by HAProxy to check the status of a server.
root@mysql-1# mysql -u root -p -e "INSERT INTO mysql.user (Host,User) values ('10.0.0.100','haproxy_check'); FLUSH PRIVILEGES;"
A MySQL user is needed with root privileges when accessing the MySQL cluster from HAProxy. The default root user on all the servers are allowed to login only locally. While this can be fixed by granting additional privileges to the root user, it is better to have a separate user with root privileges.
root@mysql-1# mysql -u root -p -e "GRANT ALL PRIVILEGES ON *.* TO 'haproxy_root'@'10.0.0.100' IDENTIFIED BY 'password' WITH GRANT OPTION; FLUSH PRIVILEGES"
Replace haproxy_root and password with your own secure values. It is enough to execute these queries on one MySQL master as changes will replicate to others.

Install MySQL Client

MySQL client has to be installed on the HAProxy droplet to test connectivity.
root@haproxy# apt-get install mysql-client
Now try executing a query on one of the masters as the haproxy_root user.
root@haproxy# mysql -h 10.0.0.1 -u haproxy_root -p -e "SHOW DATABASES"
This should display a list of MySQL databases.

Installing HAProxy

On the HAProxy server install the package.
root@haproxy# apt-get install haproxy
Enable HAProxy to be started by the init script.
root@haproxy# sed -i "s/ENABLED=0/ENABLED=1/" /etc/default/haproxy
To check if this change is done properly execute the init script of HAProxy without any parameters.
root@haproxy:~# service haproxy
Usage: /etc/init.d/haproxy {start|stop|reload|restart|status}

Configuring HAProxy

Rename the original configuration file
mv /etc/haproxy/haproxy.cfg{,.original}
Create and edit a new one
nano /etc/haproxy/haproxy.cfg
The first block is the global and defaults configuration block.
global
    log 127.0.0.1 local0 notice
    user haproxy
    group haproxy

defaults
    log global
    retries 2
    timeout connect 3000
    timeout server 5000
    timeout client 5000
More information about each of these options are covered in this article. Since we've told HAProxy to send log messages to 127.0.0.1 we have to configure rsyslog to listen on it. This has too been covered in the same article under Configure Logging for HAProxy.
Moving to the main configuration part.
listen mysql-cluster
    bind 127.0.0.1:3306
    mode tcp
    option mysql-check user haproxy_check
    balance roundrobin
    server mysql-1 10.0.0.1:3306 check
    server mysql-2 10.0.0.2:3306 check
Unlike HTTP load balancing HAProxy doesn't have a specific "mode" for MySQL so we usetcp. We've set HAProxy to listen only on the loopback address (assuming that application is on the same server) however if your application resides on a different droplet make it listen on 0.0.0.0 or the private IP address.
We need one more configuration block to see the statistics of load balancing. This is completely optional and can be omitted if you don't want stats.
listen 0.0.0.0:8080
    mode http
    stats enable
    stats uri /
    stats realm Strictly\ Private
    stats auth A_Username:YourPassword
    stats auth Another_User:passwd
Replace the usernames and passwords in "stats auth". This will make HAProxy listen on port 8080 for HTTP requests and the statistics will be protected with HTTP Basic Authentication. So you can access stats at
http://<Public IP of Load Balancer>:8080/
Once you're done configuring start the HAProxy service.
service haproxy start
Use the mysql client to query HAProxy.
root@haproxy# mysql -h 127.0.0.1 -u haproxy_root -p -e "SHOW DATABASES"
The "-h" option has to be present with the loopback IP address. Omitting it or using localhostwill make the MySQL client connect to the mysql.sock file which will fail.

Testing Load Balancing and Failover

To check if load balancing is working query the server_id variable twice or more.
root@haproxy# mysql -h 127.0.0.1 -u haproxy_root -p -e "show variables like 'server_id'"
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| server_id     | 1     |
+---------------+-------+
root@haproxy# mysql -h 127.0.0.1 -u haproxy_root -p -e "show variables like 'server_id'"
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| server_id     | 2     |
+---------------+-------+
This demonstrates roundrobin load balancing with equal weights, we'll now change the weight for mysql-2 and see the results.
nano /etc/haproxy/haproxy.cfg
server mysql-2 10.0.0.2:3306 check weight 2
Reload to apply this change.
service haproxy reload
Query for the server_id multiple times.
root@haproxy:~# for i in `seq 1 6`
do
mysql -h 127.0.0.1 -u haproxy_root -ppassword -e "show variables like 'server_id'"
done

+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| server_id     | 1     |
+---------------+-------+
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| server_id     | 2     |
+---------------+-------+
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| server_id     | 2     |
+---------------+-------+
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| server_id     | 1     |
+---------------+-------+
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| server_id     | 2     |
+---------------+-------+
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| server_id     | 2     |
+---------------+-------+
Now load balancing works in the ratio of 1:2 with one-thirds of the requests going to mysql-1 and two-thirds going to mysql-2.
Fail a MySQL server either by stopping the service
root@mysql-1# service mysql stop
or bringing the interface down.
root@mysql-1# ifconfig eth1 down
Try the "show variables" query now to see the result. The following log entries will indicate when and how HAProxy detected the failure.
tail /var/log/haproxy/haproxy.log
Nov 15 00:08:51 localhost haproxy[1671]: Server mysql-cluster/mysql-1 is DOWN, reason: Layer4 timeout, check duration: 2002ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

Reducing Failover Interval

When a MySQL server goes down HAProxy takes some time to detect this failure and remove it from the cluster. In this section we'll see how to control this time. First we'll see how to measure this value. One way is to block the MySQL port using iptables for a certain amount of time, then remove the rule and check the log.
root@mysql-1:~# ifconfig eth1 down &&
date &&
sleep 20 &&
ifconfig eth1 up &&
date

Fri Nov 15 00:37:09 IST 2013
Fri Nov 15 00:37:29 IST 2013
The port 3306 was blocked for 20 seconds, we'll look at the log file now.
root@haproxy:~# tail /var/log/haproxy.log
Nov 15 16:49:38 localhost haproxy[1275]: Server mysql-cluster/mysql-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Nov 15 16:49:56 localhost haproxy[1275]: Server mysql-cluster/mysql-1 is UP, reason: Layer7 check passed, code: 0, info: "5.5.31-0+wheezy1-log", check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
It took 6 seconds to detect a failure (difference between 16:49:38 and 16:49:32) and 4 seconds_to detect that the server can be reached (difference between 16:49:56 and 16:49:52). This is determined by the server parameters rise, fall and inter.
The rise parameter sets the number of checks a server must pass to be declaredoperational. Default is 2.
The fall parameter sets the number of checks a server must pass to be declared dead. Default is 3.
The inter parameter sets the interval between these checks. Default is 2000 milliseconds.
Putting this info together a server must fail 3 continuous checks which are performed at an interval of 2 seconds to be considered dead. So in our example above the following would've happened.
16:49:32 - Port 3306 on mysql-1 was blocked
16:49:34 - Check - Failed - Failure No. 1
16:49:36 - Check - Failed - Failure No. 2
16:49:38 - Check - Failed - Failure No. 3 (server removed and event logged)
And when the firewall rule was removed.
16:49:52 - Firewall rule removed port 3306 accessible
16:49:54 - Check - Passed - Success No. 1
16:49:56 - Check - Passed - Success No. 2 (server added to cluster and event logged)
The following settings will reduce the test interval to 1 second and also reduce the number of fall tests.
nano /etc/haproxy/haproxy.cfg
server mysql-1 10.0.0.1:3306 check fall 2 inter 1000
server mysql-2 10.0.0.2:3306 check fall 2 inter 1000
Sometimes you may not want to flood the private network with too many "test" packets especialy if you have a large amount of MySQL servers. In such cases the fastinter and downinter parameters will come handy.
The fastinter parameter sets the interval between checks while a server is transitioning UP or DOWN.
The downinter parameter sets the test interval when a server is DOWN.
That explanation might be confusing so we'll see it with an example.
nano /etc/haproxy/haproxy.cfg
server mysql-1 10.0.0.1:3306 check fastinter 1000
server mysql-2 10.0.0.2:3306 check fastinter 1000
Since we haven't specified the "inter" parameter it defaults to 2000ms. With this configuration we'll restart HAProxy and do the test again.
root@mysql-1:~# iptables -A INPUT -p tcp --dport 3306 -j REJECT &&
date &&
sleep 20 &&
iptables -D INPUT -p tcp --dport 3306 -j REJECT &&
date
Fri Nov 15 17:18:48 IST 2013
Fri Nov 15 17:19:08 IST 2013
Check the HAProxy log file.
root@haproxy:~# tail /var/log/haproxy.log
Nov 15 17:18:52 localhost haproxy[1353]: Server mysql-cluster/mysql-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Nov 15 17:19:11 localhost haproxy[1353]: Server mysql-cluster/mysql-1 is UP, reason: Layer7 check passed, code: 0, info: "5.5.31-0+wheezy1-log", check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Now it took only 4 seconds (compared to 6 earlier) to detect a failure and 3 seconds (compared to 4) to detect that the server was up. Behind the scenes this is what happened.
17:18:48 - Port 3306 blocked
17:18:50 - Check - Failed - Failure No. 1
17:18:51 - Check - Failed - Failure No. 2
17:18:52 - Check - Failed - Failure No. 3 (server removed and event logged)
And when the port was unblocked.
17:19:08 - Firewall rule removed
17:19:10 - Check - Passed - Success No. 1
17:19:11 - Check - Passed - Success No. 2 (server added to cluster and event logged)
First notice the interval between the port block event (17:18:48) and the first check (17:18:50), it is 2 seconds (the "inter" interval). Then notice the interval between Test 1 <-> Test 2 and Test 2 <-> Test 3, it is only 1 second (the "fastinter" interval). The same intervals can be noticed when the server moved from DOWN to UP. So "fastinter" controls the interval between these checks.
So what is downinter? When a server has been declared DOWN HAProxy continues checking it every 2 seconds (or the interval mentioned in inter). If you feel you're using up unnecessary network resources setting the downinter to say 5000 will make HAProxy check a DOWN server only once in 5 seconds.

Important

The tests we did previously REJECTed the packets which means when HAProxy initiated a connection by sending a SYN packet to mysql-1 it received a RST packet (instead of SYN + ACK). This is why the log entry mentioned "Connection refused". In this case only the fall, inter and fastinter values come into the scene.
Instead if HAProxy didn't receive anything after sending SYN the connection times out. In this case in addition to the above mentioned parameters the "timeout" duration comes into the scene. This situation can happen if
  • iptables is set to DROP
  • the private interface is down
  • there is a problem with the private networking infrastructure

Further Reading

Official documentation http://cbonte.github.io/haproxy-dconv/configuration-1.4.html

Monday, 14 July 2014

how to configure nginx as a reverse proxy for apache




How To Configure Nginx as a Reverse Proxy for Apache


Nginx as a reverse Proxy for Apache is one of the most usual ways to use Nginx, altough nowadays you can replace Apache with Nginx at 100%, there are still some people who need to setup Nginx as a Reverse Proxy for Apache. In this guide I will teach you how to gain a lot of server performance by serving your static files (jpg,gif,png,css,js,html) with Nginx, and the dynamic files with Apache on the backend.
Install Apache
yum install httpd httpd-devel -y
Configure the Reverse Proxy settings on Apache
Edit Listen directive at httpd.conf
nano -w /etc/httpd/conf/httpd.conf
Then find:
Listen 80
And change it to be:
Listen 81
Then, move at the end of the file and paste this code:
NameVirtualHost 127.0.0.1:81
 
# Define Server document root
   DocumentRoot /var/www/html/
 
# Define the virtual host
<VirtualHost 127.0.0.1:81>
   ServerName www.yourwebsite.com
   ServerAlias yourwebsite.com
   DocumentRoot /var/www/yourwebsite.com
       <Directory "/var/www/yourwebsite.com">
               Options FollowSymLinks -Includes
               AllowOverride All
               Order allow,deny
               Allow from all
       </Directory>
       RewriteEngine on
</VirtualHost>
Replace yourwebsite.com with your real domain name, and /var/www/yourwebsite.com with your real root directory.
Restart httpd to apply the changes:
service httpd restart
Install Nginx
mkdir /root/temp; cd /root/temp
wget http://nginx.org/packages/centos/6/noarch/RPMS/nginx-release-centos-6-0.el6.ngx.noarch.rpm
rpm -i nginx-release-centos-6-0.el6.ngx.noarch.rpm
yum install nginx -y
Configure Nginx main configuration
nano -w /etc/nginx/nginx.conf
Place this configuration inside:
user  nobody;
worker_processes  4;
error_log  logs/error.log crit;

worker_rlimit_nofile  8192;

events {
worker_connections  1024; # you might need to increase this setting for busy servers
use epoll; #  Linux kernels 2.6.x change to epoll
}

http {
server_names_hash_max_size 2048;
server_names_hash_bucket_size 512;

server_tokens off;

include    mime.types;
default_type  application/octet-stream;

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout  10;

# Gzip on
gzip on;
gzip_min_length  1100;
gzip_buffers  4 32k;
gzip_types    text/plain application/x-javascript text/xml text/css;

# Other configurations
ignore_invalid_headers on;
client_max_body_size    8m;
client_header_timeout  3m;
client_body_timeout 3m;
send_timeout     3m;
connection_pool_size  256;
client_header_buffer_size 4k;
large_client_header_buffers 4 32k;
request_pool_size  4k;
output_buffers   4 32k;
postpone_output  1460;

# Cache most accessed static files
open_file_cache          max=10000 inactive=10m;
open_file_cache_valid    2m;
open_file_cache_min_uses 1;
open_file_cache_errors   on;

# virtual hosts includes
include "/etc/nginx/conf.d/*.conf";

}
Configure Nginx Reverse Proxy settings
Now create this file:
nano -w /etc/nginx/conf.d/yourwebsite.com.conf
Then paste this inside:
server {
 listen    80;
 server_name  yourwebsite.com yourwebsite.com;
 access_log off;
 error_log  logs/yourwebsite.com-error_log crit; 

location ~* .(gif|jpg|jpeg|png|ico|wmv|3gp|avi|mpg|mpeg|mp4|flv|mp3|mid|js|css|html|htm|wml)$ {
 root /var/www/yourwebsite.com;
 expires 365d;
 }

location / {
 client_max_body_size    10m;
 client_body_buffer_size 128k;

 proxy_send_timeout   90;
 proxy_read_timeout   90;
 proxy_buffer_size    128k;
 proxy_buffers     4 256k;
 proxy_busy_buffers_size 256k;
 proxy_temp_file_write_size 256k;
 proxy_connect_timeout 30s;

 proxy_redirect  http://www.yourwebsite.com:81   http://www.yourwebsite.com;
 proxy_redirect  http://yourwebsite.com:81   http://yourwebsite.com;

 proxy_pass   http://127.0.0.1:81/;

 proxy_set_header   Host   $host;
 proxy_set_header   X-Real-IP  $remote_addr;
 proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
 }
}
Replace yourwebsite.com with your real domain name, and /var/www/yourwebsite.com with your real root directory.
What we have done here:
1.- Defined the location of static files, and let Nginx cache those files for 1 year.
2.- Then we added the reverse proxy configuration so Nginx can serve the information from 127.0.0.1:81, which is the port and IP where Apache is listening to.
Install mod_rpaf
mod_rpaf will help Apache to know who connects to it (otherwise the only IP address you will see in your logs is 127.0.0.1):
mkdir /root/temp; cd /root/temp
wget https://github.com/y-ken/mod_rpaf/archive/master.zip
unzip master.zip
cd mod_rpaf-master/
apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c
Once it’s installed, we need to configure it. Create a new file:
nano -w /etc/httpd/conf.d/rpaf.conf
And place this inside:
LoadModule rpaf_module modules/mod_rpaf-2.0.so

  RPAFenable On
  RPAFsethostname On
  RPAFproxy_ips 127.0.0.1 YOUR.SERVER.IP
Replace YOUR.SERVER.IP with your public server IP address.
Now restart httpd to apply the changes:
service httpd restart
Test your Nginx Reverse Proxy
curl -I http://www.yoursite.com/
You should see something like this:
HTTP/1.1 200 OK
Server: nginx
If you see Nginx in the Server: result, that means your website is being served using Nginx in the front end, and the backend dynamic files (php) are served using Apache. Need a better test? Try shutting down Apache:
service httpd stop
Then try to load your website on the browser or using curl, you will probably get a 502 bad gateway error, and that means your proxy is actually working. Now, get your httpd bad ass server back:
service httpd start
Other ways to test it: create a phpinfo file.
nano -w /var/www/yourwebsite.com/info.php
Paste the following lines into that file:
<? phpinfo( ); ?>
Save and exit.
Now visit www.yourwebsite.com/info.php and you should show see the php server information on your screen, then search for the server that is serving the phpinfo file, if you see Apache then your proxy configuration is working fine.