Syzygy tablebases: maximizing performance

by Albert Silver

2/10/2015 – Tablebases are wondrous tools to help analyze the endgame, and improve engine evaluation, but they do come at a cost. While the five-piece set will easily fit into almost any computer's memory without accessing the hard disk, the much larger files for six pieces can cause significant slow downs on some machines. Here are some tips to maximize performance.

The benefit of using the tablebases should be obvious to any. Even though computer chess experts argue that the precision of the knowledge is almost exactly offset by the fractional loss in speed when it analyzes endgames, anyone who has seen the engine produce wildly optimistic evaluations due to a misunderstanding of rook endgame realities, or material imbalance that has no hope of winning, will appreciate being able to trust the evaluation and not ask themselves whether this position is one of those exceptions.

Six piece positions can be even harder, such as rook versus bishop and two pawns, queen versus rook and two pawns (think of the numerous fortress situations), and the list goes on. The problem is that some of the files of the more difficult positions can take up one or even two gigabytes a piece. If the engine is accessing more than one due to the various situations it is considering, it can cause the computer to start reading the hard disk non-stop, which in turn can result in significant slowdowns. This is not related to ChessBase's DVD set, or any of the ChessBase programs, it is simply the nature of 20GB or more of files constantly being consulted to read the results as the engine looks something up.

Even the best hard drives, such as this Black Edition of Western Digital, may have
issues accessing all the information needed at the speeds the engine requests.

Not all computers will have any issues, so if you do not, then do not concern yourself any further. If you do, however, there are a variety of ways to resolve this issue, so that you need not forego on this great tool.

Use an SSD

The easiest solution is to install the files on an SSD. These are digital storage devices, much like a pendrive, that can be used like a hard disk, but are many many times faster. They have many benefits that exceed tablebases of course. Booting up a computer in which the operating system was installed will take a fraction of the time it would on a normal hard disk. If you have ever found yourself tired of waiting for a program to open or close, SSDs will more or less end that issue.

SSDs are incredibly fast, and can greatly improve your overall computer performance, not just
for tablebases as being suggested here

Finally, for the purposes of this article: installing the tablebases on the SSD will unquestionably solve any performance issues, if you had any.

There are two caveats that need to be mentioned: if you are running the engine on a laptop that doesn't already have an SSD, this solution may not be feasible, and the second is the cost. Although prices have plummeted since they first came out, a 256GB SSD can cost upwards of $100 in the US, which may not be acceptable to everyone.

Use a Pendrive

In many ways, this is the Goldilocks solution, providing enormous performance improvements, without the SSD's cost, and completely workable on a laptop as well. It also means you can use the tablebases on a laptop even if you do not have a DVD drive to install them from.  

For this solution, all you need is a pendrive that is big enough to fit them, which for the ChessBase
set means about 20GB of files, so a 32GB pendrive. It is also recommended that you use a USB 3.0
pendrive, as it will be considerably faster. Nowadays that will cost under $15.

Installing on a pendrive

There is nothing to it. Just find the Tablebases directory where you installed the files, in my case
my second hard disk (D:), and then copy them to your pendrive.

Configuring ChessBase to use a pendrive

Even if you were able to install the DVDs directly to a pendrive (I have not tested this, but imagine it is possible), if you take the pendrive to another computer, that copy of ChessBase or Fritz will still need to be configured. Thankfully, though there are a few steps to follow, it is not hard.

In ChessBase or Fritz click on File at the top left, and then Options

In the pane, click on the Tablebases tab at the top, and then the More... button

A new pane will open, specifically for configuring various tablebases. Choose Syzygy (Engine)
and then click on the [...] next to the three paths. Here you will need to show it where to find
the tablebases.

For each path, configure one of the three directories as above. Once done, click on Ok, and then Apply.

Now, whenever you turn on Komodo 8, or any other engine that supports the Syzygy tablebases such as Houdini 4 or Stockfish, it will use the Syzygy bases installed on the pendrive.

Using only five-pieces

Why would you want to do this? If you do not have an SSD or a pendrive, want the tablebases, but are suffering from slowdowns, you can use the following compromise: use only the five-piece set. Since they are under 1 GB, and can easily fit in the computer's RAM.

In the same configuration screen as above, click on Clear, and then set only the path for the
five-piece set. The directory is Syzygy345. Leave the others blank.

If you find yourself having to use this last scenario as a solution, I strongly suggest you get a pendrive for the six-piece set. The cost is small, but the information those six-piece tablebases add is truly not to be underestimated. It means that everytime its search leads to one of those situations with six pieces, it can stop searching as it knows exactly the result and evaluation. 

The Syzygy Tablebase DVDs can be found in the ChessBase Shop

Albert Silver
originally retrieved from http://en.chessbase.com/post/syzygy-tablebases-maximizing-performance

Syzygy Tablebases

Syzygy Tablebases: newest, fastest, smallest

by Albert Silver

2/8/2015 – The concept of tablebases is not new, and there have been several generations of them, such as the Thompson tablebases, the Nalimov tablebases and more. The latest engines such as Houdini 4 and Komodo 8 provide special support of the new Syzygy format. Discover what is so special about them, why you should not be without them, and learn just how easy they are to install.

Tablebases have been a boon to chess players and researchers. They represent the absolute indisputable truth of a position, that cannot be ignored. A complete five-piece tablebase set means that everytime the position is reduced to five pieces (these always include kings), such as king+rook against king+rook+pawn, there are no calculations to be made. There is an immediate and absolute result.

For over a decade now, the reference tablebase format has been the Nalimov set, created by Eugene Nalimov. The first five-man set came out in late 1998, and the six-man set was completed by 2005. They represented an important evolution over the previous standard Edwards tablebases, since the five-piece set took up a mere 7.1 GB, which was about eight times smaller than the predecessor.

That is why the new Syzygy Bases are such an important step up. Developed by Ronald de Man, and published in 2013 for the first time, the Syzygy Bases consist of two sets of files, WDL files storing win/draw/loss information considering the fifty-move rule for access during search, and DTZ files with distance-to-zero information for access at the root. The practical difference is that the full five-piece set of files takes up less than 1 GB, in other words they are over seven times smaller than the Nalimov tablebases.

Of course, while freeing up 6 GB on your hard disk is a nice gain, it is hardly cataclysmic in today's age of hard drives with thousands of GB. More importantly though, this advantage spreads to the six-piece set also, where this size difference is much more evident. The complete six piece Nalimov tablebases take up about one full terabyte, or 1000 GB. As a result, the Nalimov tablebase DVD set offered by ChessBase contains only a careful selection of the most useful six-piece combinations. What would a 'useless' one be? Examples might be King+Queen+Queen versus King+Bishop+Pawn, and so forth. The engine can win this easily, and adding these tablebases are not going to improve performance. However, even so, some potentially useful ones would be missed due to space considerations.

The Syzygy Bases come on four DVDs, and can easily be installed within minutes

Theoretically, you can track down and download all the Syzygy files on the internet (legally), and then install and configure ChessBase 12/13 or Fritz so that Komodo 8 can use them. Allow me to discourage you from this. I actually did this (Vive Le Geek) when the DVD set by ChessBase was not yet available and will say it is a tremendous hassle. It involves downloading each and every file, which number the hundreds, and unless you want to handpick the best ones as already done here, then the complete set will mean downloading over 140GB of files. You then need to configure the interface properly, which can be tricky sometimes. All in all, I don't recommend it, if only for the hideous bandwidth consumption.

The new Syzygy Base DVDs offered by ChessBase simplify this process so that even the least tech savvy user can enjoy them. You get the fullest experience with only the most superfluous piece combinations left out, and that most obvious advantage: the ease of use and installation. Just install the four DVDs, and you are done. ChessBase 12/13 will automatically be configured to access them, and as soon as you run an engine such as Houdini 4 or Komodo 8, they will consult them in their analysis. It is really that easy.

Installing  the Syzygy Bases

Installing them could not be easier. Starting with DVD 1, put it in the drive
and when the autoplayer asks you what to do, select Run Setup.exe

A new window will open asking you where you want to install them. The default is C:\Tablebases,
but feel free to change the location. Just be sure you have about 20GB free for this six-piece collection.If you own an SSD, then you should consider installing them there as it will greatly improve lookup speed and performance.

After click Install, just wait for it to finish copying the files. It will automatically configure ChessBase 12/13
or Fritz when it does

After it is done, just click on finish

After installing DVD 1, put in DVD 2 and repeat the process. Be sure to tell
it to install the in the same directory in case you chose a different one than
the default. You will need to do this for all four DVDs.

Testing the Syzygy Bases

Though there shouldn't be any need, if you want to be sure the process
worked, load up a simplified endgame

Start up the engine, and look at the analysis it produces. If you see tb= in it, then you are
good to go. The number means the number of positions in its analysis that it found in the
Syzygy table bases. Don't be surprised to see tens of thousand of finds as the position simplifies.

The Syzygy Tablebase DVDs can be found in the ChessBase Shop

Albert SilverBorn in the US, he grew up in Paris, France, where he completed his Baccalaureat, and after college moved to Rio de Janeiro, Brazil. He was champion of Rio de Janeiro with a peak rating of 2240, and was a key designer of Chess Assistant 6. In 2010 he joined the ChessBase family as an editor and writer at ChessBase News.

http://en.chessbase.com/post/syzygy-tablebases-newest-fastest-smallest

Perfect endgame analysis and a huge increase in engine performance: Get it with the new Endgame Turbo 4! Thanks to a new format the Endgame Turbo 4 is smaller, faster and yet has more scope. Four DVDs cover all five- and 27 of the most important six-piece-endgames and top engines such as Komodo Chess 8 and Houdini 4 play these endgames perfectly. But they also play other endgames much better because they can access the perfect knowledge of the Endgame Turbo! A must have for everyone wanting to improve in the endgame, correspondence players, endgame theoriticians, and friends of engine matches!

The Endgame Turbo 4 consists of 4 DVDs with endgame databases (Syzygy Tablebases). With the help of the Endgame Turbo, all five- and 27 six-piece endgames (including the sophisticated and practice-oriented endgame ¦, §, § - ¦) are played with absolute perfection. Likewise, top chess engines like Komodo Chess 8 and Houdini 4 handle endgames with more than six pieces much better since the programs can already access the endgame knowledge during the analysis. Definitely a must-have for correspondence players, endgame theoreticians and friends of engine matches! CONTENT The Endgame Turbo 4 contains all endgames with three pieces up to five pieces. Plus 27 of the most important endgames with 6 pieces. Six-piece endgames included on the DVDs:

System requirements

Minimum: Pentium III 1 GHz, 1 GB RAM, Windows Vista, XP (Service Pack 3), DVD-ROM drive, Deep Fritz 14, Komodo Chess 8, Houdini 4 or ChessBase 12/13, 20 GB free hard disk space. Recommended: PC Intel Core i7, 2.8 GHz, 4 GB RAM, Windows 7 (64 Bit) or Windows 8/8.1, DVDROM drive, Deep Fritz 14, Komodo Chess 8, Houdini 4 or ChessBase 12/13, 20 GB free hard disk space

End Game Table Bases

This egtb database is downloaded using utorrent and if u dont have it yet ,get it here

http://www.utorrent.com/

The Syzygy 6-men EGTB system is the recommended table base solution for Houdini.

For the Syzygy EGTB support you'll need to install the Syzygy EGTB files. At the time of writing all the Syzygy files are available for torrent download athttp://oics.olympuschess.com/tracker/index.php. The 3-4-5-men Syzygy are also available for direct download at http://olympuschess.com/egtb/sbases.

Update: the 6-men Syzygy are now also available as HTTP download, see http://tablebase.sesse.net/syzygy.

The total size of the 3-4-5-men files is 938 MB (290 files), the 6-men table bases require 149 GB (730 files).

For the Gaviota EGTB support you'll need to install the Gaviota EGTB files. At the time of writing they are available for download at the address http://www.olympuschess.com/egtb/gaviota. Download all 145 files and save them in a directory on your hard disk. The total disk space required is about 7 GB.

Alternatively you could skip the download and generate the tables directly on your own computer, please follow the instructions found at the Gaviota web site.

For the Nalimov EGTB support you'll need to install the Nalimov EGTB files. At the time of writing they are available for download at the address ftp://ftp.cis.uab.edu/pub/hyatt/TB orhttp://tablebase.sesse.net. Download all files and save them in a directory on your hard disk. The total disk space required for the 3-4-5-men table bases is about 7 GB (290 files in total).

Alternatively you can order the tables on DVD from several suppliers, this is probably the more efficient way of obtaining the 6-men Nalimov table bases.

SyzygyPath

Folder(s) containing the Syzygy EGTB files. If multiple folders are used, separate them by the ; (semi-column) character.

Example: C:\tb\Syzygy345;C:\tb\Syzygy6

GaviotaTbPath

Folder containing the Gaviota EGTB files.

GaviotaTbCache

Amount of Gaviota EGTB cache memory in MB.

Default 64, min 4, max 1024.

NalimovPath

Folder containing the Nalimov EGTB files.

NalimovCache

Amount of Nalimov EGTB cache memory in MB.

Default 32, min 4, max 1024.

EGTB Probe Depth

Minimum search depth for EGTB probing.

Default 10, min 2, max 99.

You can reduce this value if the table base files are installed on a very fast disk, or if you are running less than 4 threads.

For more detailed explanations, see the EGTB support topic.

Retrieved from url

http://www.cruxis.com/chess/manual/index.html?game_play.htm

What is my best Chess Engine Parameter

Playing over the chess server Infinity is a big challenge , People are really well prepared there and maximizing my engine parameter shall be the key to unlock many victory

So according to some research I did ,this is what I have found

-Log dir
-max cpus
-random
-main process priority
-child process priority
-use large pages
-multi pv

-Log dir
What directory to write the logs in.

-max cpus
How many CPUs to use during analysis and game play.  This number should be less than or equal to the number of CPUs on your computer.

-random
Don't remember

-main process priority
The Rybka chess engine has a main server that launches helper child processes.  The priority of the process controls how many time slices the process will get.  There is a single main process.  The higher the number the more compute power the process will get.  However, if you raise the priority of the processes too high, your computer will become unresponsive when the games are being played, especially if max cpus is set equal to your computer's resources.

-child process priority
The Rybka chess engine spawns child processes to help in the search.  The priority of these processes controls how many time slices the processes will get.  For all of these processes, I would not recommend elevating the priority above normal.

-use large pages
Large pages is a fiddly setting that will give your engine a small boost in compute power.  You have to run the processes with administrator rights and memory fragmentation makes this setting somewhat unreliable.  So use with care and read threads that talk about how to use it.

-multi pv 
Multi-pv is NOT good for game play.  It is sometimes useful for analysis of chess positions.  The idea of multi pv is that it has the chess engine study other lines besides the main line.  The problem with multi pv is that it slows down the engine a lot.


http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?tid=27663

Game Play

Top  Previous  Next

Tactical Mode (checkbox) When Tactical Mode is active, Houdini modifies its search process and uses some clever tricks to focus on tactical solutions in the root position. In Tactical Mode Houdini becomes a very skilled tactical problem solver. Below some results for two popular tactical test suites with Houdini running at 30 seconds per position using 2 threads on a Core i5-750 with 512 MB hash. 1) Arasan 14 Tactical Suite, comprising 197 medium to hard positions - Houdini 1.5a solves 148 positions, total solution time 34 minutes. - Houdini 2 solves 155 positions, total solution time 35 minutes. - Houdini 4 Tactical solves 177 positions, total solution time 22 minutes. 2) Die Tactical Suite, comprising 100 hard positions - Houdini 1.5a solves 29 positions, total solution time 40 minutes. - Houdini 2 solves 33 positions, total solution time 39 minutes. - Houdini 4 Tactical solves 56 positions, total solution time 30 minutes. Note that the focus on tactical moves will reduce the playing strength of the engine, the Tactical Mode is probably about 30 Elo weaker than the normal playing mode. One can compare the Tactical Mode to what a human player does when he/she tries to solve a test position in a chess magazine or website. He/she will scan for tactical, forcing moves, threats, sacrifices etc. That way a lot more tactical solutions can be found than what the player would actually see and decide to play over the board, during a normal game. The "this is a test position" message changes the search strategy. Houdini's Tactical Mode works exactly the same, very human-like: it will spend a lot more time looking for tactical moves, threats etc. In most positions this doesn't provide any benefit - most of the time there is no tactical solution available and all the focus on tactics is just wasted. But in test positions (selected because there IS a tactical solution) it works extremely well. Strength Limit strength from 0 (beginner) to 100 (full strength). Default is 100 (full strength). The strength reduction is mostly based on a combination of two techniques: • Limiting the number of positions searched - this reduces mostly the tactical strength of the engine; • Purposely picking a move the engine knows is not optimal - this reduces mostly the positional strength of the engine. The combination of the two produces a game with both tactical and strategic (positional) flaws. The following table shows the estimated Elo of Houdini 2 at reduced Strength levels. It is based on the feed-back of several users: Strength  0 => 1200 Elo Strength 10 => 1500 Elo Strength 20 => 1800 Elo Strength 30 => 2000 Elo Strength 50 => 2300 Elo This table is very approximate - we welcome any feedback from rated players that could improve these values. What is the highest strength level that you can consistently beat? UCI_Elo UCI-protocol compliant version of Strength parameter. Default 3000, min 1200, max 3000. Internally the UCI_Elo value will be converted to a Strength value according to the table given above. The UCI_Elo feature is controlled by the chess GUI, and usually doesn't appear in the configuration window. UCI_LimitStrength (checkbox) Activate the strength limit specified in the UCI_Elo parameter. This feature is controlled by the chess GUI, and usually doesn't appear in the configuration window. MultiPV Number of principal variations shown. Default 1, min 1, max 32. The MultiPV feature is controlled by the chess GUI, and usually doesn't appear in the configuration window. MultiPV_cp Limit the multi-PV analysis to moves within a range of the best move. Default 0, min 0, max 999. Values are in centipawn. Because of contempt and evaluation corrections in different stages of the game, this value is only approximate. A value of 0 means that this parameter will not be taken into account. Ponder (checkbox) Have the engine think during its opponent's time. The Ponder feature (sometimes called "Permanent Brain") is controlled by the chess GUI, and usually doesn't appear in the configuration window. Contempt Level of contempt to avoid draws in game play. Default 1, min 0 (none), max 2 (aggressive). The notion of "contempt" implies that Houdini will try to avoid draws by evaluating its own position slightly too optimistically. The Contempt level can be chosen between 0 (none) and 2 (aggressive), the default value of 1 should be a good compromise in most situations. • 0 = No Contempt The evaluations are accurate and identical for both sides. This is recommended for position analysis in which you analyze alternatively for White and Black. The starting position evaluates as approx. +0.15. • 1 = Default Contempt Contempt 1 is primarily based on piece value imbalance, Houdini will value its own pieces higher than the opponent pieces, so will only exchange them if there’s a clear positional advantage in doing so. This also means that the score is evaluated optimistically for the side to move (at most 0.15 pawn). For example, the starting position evaluates as approx. +0.30 when analyzing for White and +0.00 when viewed from Black. This is only recommended for position analysis if you always analyze for the same side. • 2 = Aggressive Contempt 2 adds some king safety imbalance, leading to a more attacking style. The contempt settings are fairly mild and have little impact on the objective strength of the engine. It’s hard to say which will give the best results against a given opponent, it may depend on the style and strength of the opponent. One could envisage more pronounced contempt but this would start to degrade the engine’s objective strength. By default the contempt is only activated during game play, not during infinite analysis. If you enable the Analysis Contempt checkbox, Houdini will also take into account the contempt for infinite analysis. Analysis Contempt (checkbox) Activate Contempt for position analysis. It is usually not recommended to activate the contempt for analyzing positions. When contempt is active, the score of the analysis will be optimistic (over-evaluated) for the side that is to move. That means that if you use Analysis Contempt the evaluations will change depending on whether White or Black has the move. For example, from the start position, when you do an analysis with Analysis Contempt (and Contempt value 1) you could find a best move e2-e4 scoring about +0.3 for White. If you then play e2-e4 and analyze for Black you could find a score close to +0.0. If you do the same without Analysis Contempt, you should find a consistent +0.15 score whether it’s White or Black to move. FiftyMoveDistance The number of moves after which the 50-move rule will kick in. Default 50, min 5, max 50. This setting defines the number of moves after which the 50-move rule will kick in - the default value is 50, i.e. the official 50-moves rule. Setting this option in the range of 10 to 15 moves can be useful to analyse more correctly blockade or fortress positions: - Closed positions in which no progress can be made without some sort of sacrifice (blockade); - End games with a material advantage that is insufficient for winning (fortress). By setting FiftyMoveDistance to 15, you're telling the engine that if it cannot make any progress in the next 15 moves, the game is a draw. It's a reasonably generic way to decide whether a material advantage can be converted or not. Mate Search Maximum search depth for mate search. Default 0, min 0, max 99. If set, this option will usually speed-up a mate search. If you know that a position is "mate in X", you can use X or a value slightly larger than X in the Mate Search option. This will prevent Houdini from going too deep in variations that don't lead to mate in the required number of moves. UCI_Chess960 (checkbox) Activate Fischer Random Chess a.k.a. Chess960 games. retrieved from url http://www.cruxis.com/chess/manual/index.html?game_play.htm

Reactivate a missing or offline dynamic disk

I had added a second hard driver few days ago and I just wanna share here how I have fixed it using Microsoft recommendation. it work fine with me using the first recomendation

Updated: January 21, 2005

Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2

To reactivate a missing or offline dynamic disk

• Using the Windows interface
• Using a command line

Using the Windows interface

1. Open Computer Management (Local).
2. In the console tree, click Computer Management (Local), click Storage, and then click Disk Management.
3. Right-click the disk marked Missing or Offline, and then click Reactivate Disk. (or click Online)
   
   The disk should be marked Online after the disk is reactivated.

Notes

• To perform this procedure on a local computer, you must be a member of the Backup Operators group or Administrators group on the local computer, or you must have been delegated the appropriate authority. To perform this procedure remotely, you must be a member of the Backup Operators group or Administrators group on the remote computer. If the computer is joined to a domain, members of the Domain Admins group might be able to perform this procedure. As a security best practice, consider using Run as to perform this procedure. For more information, see Default local groups, Default groups, and Using Run as.
• To open Computer Management, click Start, click Control Panel, double-click Administrative Tools, and then double-click Computer Management.
• A dynamic disk may become Missing when it is corrupted, powered down, or disconnected.
• A dynamic disk may become Offline if it is corrupted or intermittently unavailable. A dynamic disk may also become Offline if you attempt to import a foreign (dynamic) disk and the import fails. An error icon appears on the Offline disk. Only dynamic disks display theMissing or Offline status.
• Only dynamic disks can be reactivated.

Using a command line

1. Open Command Prompt.
2. Type:
   
   diskpart
3. At the DISKPART prompt, type:
   
   list disk
   
   Make note of the disk number of the missing or offline disk that you want to bring online. Missing disks are numbered M0, M1, M2, etc. and offline disks are numbered 0, 1, 2, etc.
4. At the DISKPART prompt, type:
   
   select diskn
5. At the DISKPART prompt, type:
   
   online

 

Value	Description
list disk	Displays a list of disks and information about them, such as their size, amount of available free space, whether the disk is a basic or dynamic disk, and whether the disk uses the master boot record (MBR) or GUID partition table (GPT) partition style. The disk marked with an asterisk (*) has focus.
select disk n	Sets the focus to the specified disk, where n is the disk number.
online	Brings an offline disk or volume with focus online.

Notes

• To perform this procedure on a local computer, you must be a member of the Backup Operators group, Administrators group, or you must have been delegated the appropriate authority. If the computer is joined to a domain, members of the Domain Admins group might be able to perform this procedure. As a security best practice, consider using Run as to perform this procedure. For more information, see Default local groups, Default groups, and Using Run as.
• To open a command prompt, click Start, point to All programs, point to Accessories, and then click Command prompt.
• A dynamic disk may become Missing when it is corrupted, powered down, or disconnected.
• A dynamic disk may become Offline if it is corrupted or intermittently unavailable. A dynamic disk may also become Offline if you attempt to import a foreign (dynamic) disk and the import fails. Only dynamic disks display the Offline status.
• Only dynamic disks can be reactivated.
• For more information about DiskPart, see Related Topics.

Information about functional differences

• Your server might function differently based on the version and edition of the operating system that is installed, your account permissions, and your menu settings. For more information, see Viewing Help on the Web.

See Also

Concepts

Partition styles
Disk status descriptions
Working with MMC console files
DiskPart