29 December 2014

Chess computing resources for 2015 - Part I: Database and Analysis Software



This starts a series of posts that will be, in part, an update of previous ones on computer-related resources.  In this post, I intend to review chess database and analysis software functionality in a more in-depth manner, from the point of view of an improving club player.  My primary aim is to provide a reasonably comprehensive guide on how improving players can use these computer tools in a practical manner.  I therefore focus on how they can assist game analysis and related studies, which I've made the centerpiece of my own improvement program.

Online chess game publishing is a separate but related topic that I think deserves its own future update, although the last post on the topic from 2013 remains useful as a starting point.

Companion posts in this series on other types of computing resources can be found here:
Part II - Chessplaying software and computer opponents
Part III - Game databases and e-books
ChessBase's list of web-based applications

Database and Analysis Software

This category of computing tools, which use game databases for training, study, research and analysis purposes, has evolved to the point where I believe it can and should be employed by all serious chessplayers, not just the pros.  The broad range of functions now included in software packages cover just about everything you would want to do in chess analysis and preparation.

Although some database programs offer options for computer opponents along with analysis functions, that is normally not their primary focus, so I've decided to break that category out (along with a more detailed discussion of computer engines) into a separate post on chessplaying software / computer opponents.  Finally, in a third post I'll include a list of some game database resources and discuss the use of chess e-books (that in reality are annotated databases) and related software.  Naturally, individual software packages can contain all of the above categories of chess resources.

Below we will explore various types of database functions and highlight their possible utility for the improving player.  In large part, I'm doing this in order to help expand my own training tools for the new year by making myself smarter on the available options.  That said, I hope readers may also find the walkthrough and function review helpful.

Because ChessBase 13 (CB13) was just recently released and is popularly viewed as the "gold standard" for database software, I'm using that as the primary reference point for examples.  Other packages (including the two listed below) may provide similar functionality in many respects and for less money (or even for free, in the case of SCID vs PC).  Because of the quality freeware available, there is now no barrier to entry for any serious player (of whatever grade) to have, at minimum, a database of their own collected games available for analysis, along with a large database of master-level games for study, comparison and research.
  • Chess Assistant 15 - this package from ChessOK is the main commercial database alternative to CB13, with its own proprietary format, although both it and CB13 can also work with PGN files.
  • SCID vs PC - this is a full-featured freeware database analysis package (including the Stockfish engine and pre-loaded opening books) that also has some extra features like a FICS client for online play.  Latest release version as of this post is October 2014, from the Sourceforge site linked above.
For organizational purposes, below I'll examine different database functions under the headings of opening, middlegame and endgame study, along with a final section on game analysis, which serves to put it all together.

A.  Opening Study

One of the fundamental tasks for a chessplayer is to keep a record of their opening repertoire, as it is created and evolved.  Although ChessBase has offered designated "repertoire" databases with the most recent versions, including two individual ones for White and Black with CB13, this type of database organization and use is not terribly intuitive, including for more advanced players.  The simple openings database system I use - SORDS - organizes different major variations into games, similar to the way openings books are divided into different chapters. This is something any database program can do for you.
  • Collect Openings is a CB13 function (see PDF manual p. 227) that is designed to be used on a database of your own games.  It will automatically create a new database that groups and merges games into different opening classifications, in effect showing you what your repertoire has been.  This seems like a powerful tool and could, for example, be used to generate your first repertoire database automatically.
  • Opening "books" are included in many database programs, can be generated from them, or downloaded as separate files.  CB13 allows you to generate them from a reference database (see PDF manual p. 191) or if you are online to access the "Live Book" which is always up-to-date.  A "book" gives you a tree of opening moves to follow, similar to printed references such as Modern Chess Openings (MCO), the Encyclopedia of Chess Openings (ECO, whose codes are now standard), Nunn's Chess Openings (NCO), etc.
Statistics are normally included along with book displays. For example, here is a screenshot of the LiveBook from CB13.  It shows the move (the opening move for White, in this case), number of games in the book with that move, the score (result) in percentage terms, average Elo of players playing that move, the date, and the book evaluation.  This is typical across database programs.  The CB13 Livebook also has "visits" and a percentage next to it, which shows what moves people have been looking at in the online book.


The "Reference" tab in CB13 provides a deeper look at the position on the board and is primarily used for study of the opening phase, as it automatically retrieves and compares all games in the database with the same position.  As an example, below shows Gelfand-Tomashevsky from the 2014 Baku Grand Prix commentary after Black's 6th move.  As with the book format, we can see the move list (starting with White's move 7), number of games in the database and score (from White's perspective).  It also includes the last time it was played in the database by a titled player, the "Hot" indicator (a bar that reflects more recent play by top players), and a list of the best players who have played the move.  (move list, % result from White's perspective, "Hot" games, best players)


In the second window, the database program provides the most popular and highest-scoring lines (showing number of games and percentage score) along with the line fragment.  This shows at a glance how the game can evolve from the position and points of departure.  The third window contains all of the database games (obviously with a scroll bar) and can be sorted by all of the column headers - White player, White Elo rating, Black player, Black Elo rating, result, date, and notation.  These functions are all quite helpful and will be discussed further in the Game Analysis section below.  All of the games shown can be brought up in a separate window by double-clicking them.

"Report - Opening Report" is a CB13 function that scans the reference database and sorts information into a new database with HTML text and links, essentially providing an automatically-generated document detailing the reference pane information.  While this can be quite helpful to have all laid out for you in once place, as the main moves and continuations are included in the output, along with links to high-level sample games, the function is not fully reliable.  For example, running the opening report on the game position above, from October 2014, does not include it or any of the other 2014 games in the database output.  I believe this is a quirk in how CB13 is querying the date field of the games, as I've noticed the 2014 games in the database (Mega 2014) have included a month in that field; the software appears not to be able to recognize that with this particular function.  This same quirk can affect column sorting by date, although the games themselves at least appear.

Search functions allow you to look up games by ECO code and whatever other criteria you would like to add.  For example, as part of my "Stonewall Hero" exercise, I pulled all of Artur Yusupov's games in the Stonewall (ECO code A90) up via the search function and copied them into a new database.
  • In the reverse process, most databases should allow you to look up the ECO code of the game - in CB13 this can be done via "Report - Opening Classification".  ECO codes can also be searched online at various chess sites (or even just via standard internet search such as Google, for example "ECO code Classical Caro-Kann").
Identifying the game novelty (the move when it first goes out of the database) can be done various ways, including simply stepping through the game moves until there are no other database games showing in the Reference section.  The shortcut for this in CB13 is to use the Report - Novelty annotation function, which accesses the Live database for comparison purposes.  In the example above, Gelfand-Tomashevsky, the novelty is flagged as 12...c5 and the software automatically includes a predecessor game as a variation in the annotations, as well as a game in what it considered a relevant major variation for White on move 9.  More on the role of variations will be discussed in the below Game Analysis section.


B.  Middlegame Study

While openings are an obvious choice for database research, since there are normally so many other examples to draw on for each game analyzed, the middlegame should not be ignored either.  Looking at similar structures, tactics and maneuvers across different games is an excellent way to identify and better understand position-types that are common in your games.

In CB13 the user has to manually activate the "themes" for database sorting, but once that is done (File - Options - Misc - Activate theme keys), they will automatically appear for databases in ChessBase format.  There are four tabs with different theme categories: a general one, tactics, strategy and endgames.  What appears there will depend on how (or if) the games have been modified or marked.
  • "Themes" tab contains a number of different miscellaneous categories, the most useful of which (if you have an annotated database like Mega 2014) is probably "Commented games", which contains games with comment symbols and/or text broken down into a number of subcategories such as "Initiative", "Attack", etc.  Individual historically important players have their own subcategories, for example in Mega 2014 Fischer has 162 commented games.
  • "Tactics" tab provides broad categories ("Attack", "Defense", etc.) rather than examples of specific tactics - see below for a more specific search function - but also includes games with training questions.  In Mega 2014 there are 3732 games with training questions (2088 commented) included.  Because CB13 allows you to edit any game to include training questions, this could be a very useful tool for assembling your own computer-based "flashcards" for tactics, using your own games or those of others as a base.  Training questions can also be used in an opening repertoire database, as a means of testing your recall and understanding of positions.  The questions can also be toggled on or off in the main menu, so if you prefer to see the whole game at once it's not an obstacle to have the questions included in database games.
  • "Maneuver search" (PDF manual p. 88) is the way you can look for a particular tactic across different games using CB13.  You need to follow the manual's instructions for the search, but it allows you to do some powerful things, for example look for all bishop sacrifices on h7 while automatically mirroring tactics for the other side (in this case Black bishop sacrifices on h2).
  • "Strategy" tab contains a large number of categories and is much more developed than the corresponding tactical themes.  For example, there is a detailed breakdown of pawn structure types and different material imbalances.  This offers an easy way of researching both classic and recent games with particular middlegame features, for example an isolated White pawn on d4.
  •  "Report - similar move" (PDF manual p.168) - "With one click the program finds games with similar pawn structures where similar moves were made."  When analyzing games (see below), this is a particularly interesting feature.
  • "Report - similar structures" (PDF manual p.167) - "With one click the program loads all the games with similar pawn structures from the database and sorts them by similarity to the current game. The program also takes account of the positioning of the pieces (rooks on open rows, queen and bishop on the same diagonal, the position of the kings)."  This is also a particularly powerful comparative feature when analyzing games and can benefit a serious student, one who wants to emulate Benedict of Amber and observe how similar battles are fought.  Both this and "similar move" do appear to work properly, in my limited testing.

C.  Endgame Study

Endgame themes lend themselves well to database analysis and comparison.  The "Endgames" tab contains a lot of different subcategories, for the most part broken down by material, already populated.  Below I've expanded the Pawn Endings and Rook vs. Other Pieces endings as examples.

  • "Report - Similar endgames" is a CB13 function that can be called from a specific game (PDF manual p.167) - "The games are sorted by similarity. The program takes account of pawn structures (passed pawns, blocked pawns, connected pawns, chains, isolanis, backward pawns) and the relevant positioning of the pieces (rooks behind passed pawns, rook is cutting the king off, king in a square, wrong bishop, etc)."  Again, this type of function is useful for your own game analysis, especially if you want to take it to a deeper level or to get a quick idea of how other players have handled similar situations.  This can, among other things, help identify winning ideas and plans.
  • Searching for material distribution in an endgame is a more basic function, but if your database software lacks automatic report generation, then it can serve a similar purpose.

D.  Game Analysis: putting it all together

For the type of game analysis I normally do for training purposes, I take a balanced approach between doing only a superficial review and a deep probing of my games and selected master-level ones of interest.  For a more detailed discussion of the benefits of the analysis process, you can follow the above link; here I'll focus on the utility of using database and analysis software to enable the process.
  • Commentary / "your thoughts here" - any software that allows you to create and maintain a database of your own commented games is an invaluable study tool.  Putting your thoughts down (in terms of text commentary, symbols, and variations) shortly after each serious game you play is fundamental to better understanding your strengths and your areas needing improvement.  Normally I do this on a fairly superficial level (taking 15-20 minutes) as a first pass on analysis of one of my games, to identify my thinking process, evaluations during the game, and key variations that were considered but not played.
  • Openings / "what would Grandmasters do?" - the next step in analysis is stepping through the opening moves with the "Reference" tab open.  This tells me how far the game follows the main line (as indicated by the majority of database games at the master level), if there are major move branches or only one overwhelming preference by professional players, and helps capture transpositions that only looking at the move sequence would ignore.  Some of the most useful additional functions are looking at what the strongest players for your side do (sorting by Elo) and also sorting by date, to see what is currently considered best (or at least most popular).  Other games can easily be copied into your game as variations, either for temporary or permanent reference.
  • Variations / "choices choices choices" - using the software, you have a free hand to put down all of the options that occur to you at each point and then explore them as much as you want.  While the same type of thing can be done using a physical board - and this has its uses for training - the virtual board gives you much greater efficiency in moving between the stem position and all of the various branches, as well as recording your thoughts.  It's important to be curious about what works and what doesn't in a position (and why).  Something that made an impression on me was a quote from GM Nigel Davies, as blogger Robert Pearson used to cite, "...move the pieces around...Grandmasters do this, amateurs don't."  In other words, set things up for yourself and work it out.
  • Engine analysis / "evaluate" - there has been a good deal of silliness written about the horrible evils of using chess engines in training, on the one hand, and their infallibility, on the other hand.  "Pitfalls of Computer Analysis" contains some more detailed thoughts on this topic.  To my mind, the best use of engines, when loaded into database software, is to provide a very strong objective evaluation of the position in front of you.  One trick even top grandmasters like Kramnik use is to load an engine and have it display its current evaluation, but not the moves under consideration, so they can work it out for themselves.  The major caveat for improving players using an engine for assistance is that, if you cannot understand its evaluation in the absence of concrete tactics, it will not be of much use to you from a learning perspective.  However, if you can at least describe the major positional factors at play, a top engine can help guide you in better understanding things like the value of compensation, space advantages and structural strengths/weaknesses, especially as you step through variations to see their final outcomes.  Engine analysis normally is of significant assistance in identifying key turning points in a game, along with highlighting missed opportunities - especially ones which would simply never occur to you.
For more in-depth computer analysis, CB13 includes the "deep analysis" function, which is best left running for a lengthy period of time on a game and then saved as a different game from the original one.  It also goes a step further with programmed multiple "analysis jobs" which let you queue up position analyses with various advanced parameters for your engine(s).  This type of approach was first included a few years ago in the Aquarium software package (which will be referenced in the next post in the series) as "IDeA" projects.  These types of functions in practical terms will be of interest more for the professional or correspondence player, or people who are more interested in very deep analysis of individual games; I have not done more than give the functions a cursory look.

In Part II of this series, I will take a closer look at chessplaying software and engines, including some additional thoughts on the practical value of computer opponents and analysis for the improving player.

Below are some additional references for CB13 functions:
All of the above are from the ChessBase site and of course one must keep in mind that they are designed to help sell the product.  The individual authors' assessments of the product nevertheless seem to be valid; I would consider that the first article is the most candid in that respect.

8 comments:

  1. Ha, i did not know what interesting features my software has, thank you for that.

    ReplyDelete
    Replies
    1. Glad it was helpful. I think this is the first time I've actually gone through the entire ChessBase manual and tinkered with (or at least looked at) all the functions.

      Delete
  2. Great post!

    Chessbase IS awesome but for some reasons I have had struggles with it on my computer. Not struggles understanding it but struggles with it just going kablooey on my computer. Not sure why?

    SCID vs PC is also nice.

    I have moved onto Hiarcs Chess Explorer for database stuff. Some less functions but much more user friendly and streamlined. I still analyze (for the most part) with the Shredder GUI because it is the only GUI that EASILY allows one to show only the Engine evaluation and nothing else.

    Also have begun messing with Chess Position Trainer for repertoire (Am finally creating a repertoire) LOVE this program. I have also begun to care less about size of database and just begun using Chessgames.com for finding good player and opening collections.

    ReplyDelete
    Replies
    1. Thanks for bringing up the Shredder GUI, I mentioned that in the second post (just put it up). Will also take a look at Hiarcs Chess Explorer. Finally, I'll plan to mention Chessgames.com in the third and final post on database resources. There are some really good online ones like that available now.

      I posted a while back on the difficulties of getting ChessBase 11 to install on a Windows 7 machine. ChessBase 13 (a native 64-bit program) however has installed and run without any issues so far.

      Delete
  3. Really great post as usual!

    I like to speak up for Chess Assistant and Aquarium because I see them as the underdogs a bit. So I just wanted to add a couple points about those though I am by no means a power user. Also, I bought Chessbase 12 and use that as well. I'm not really trying to do a compare and contrast here, nor to convince anybody one is better than another, just to mention a couple of points.

    As we discussed before Chess Assistant seems more geared toward the tree approach. I find this suits my needs and interests. Many times what I am interested in doing with an opening or database research is to look through different option and navigate down different options and then back up and try something else (like with the opening explorer function you find on chessgames.com). This is true both when I am wanting to look at pro games and when I am looking at my own. This is definitely possible in Chessbase but I find it is not nearly as convenient. In Chess Assistant the tree is a separate window and the info is compiled before hand so there is no delay when you are navigating. My experience with Chessbase is that it is a bit fussy to use (a small inconvenient window) and it seems to build up the statistics for every position as you navigate to it, and does it in weird way where the numbers jump up then up again and keep jumping till they reach the full value and you have to wait a bit to see if it is done jumping. Of course the flipside of the coin here is significant. In Chess Assistant you have to rebuild the tree when you update the database. For small databases this happens so quick as to instantaneous, but of course you have to remember. Also for larger databases it can take a while.

    I also find Chess Assistant preferable for maintaining the database of my own games. I break my own games databases into separate black and white databases. The cool thing about this is that the trees for those two databases just reflect my own choices on the respective color for that database, so when I am looking at the choices I can easily see what choices I have made and how often. I certainly believe this is also possible for Chessbase, but in my experience the little details of selecting the games and copying them and adding them to the relevant bases is significantly easier in Chess Assistant. I don't remember all the details of my troubles but I remember that the straw that broke the camel's back for me with Chessbase was when I copy pasted some games and Chessbase randomly changed game info. I lost my faith in it there.

    I will say that in general Chessbase looks much nicer and more professional. One other point, I find it quite convenient that Chess Assistant saves your working layout when you close the program and everything is there as you left in when you open it up.

    ReplyDelete
    Replies
    1. Thanks for the comments and it's good to have some up-to-date practical info on Chess Assistant - which is in fact difficult to track down via searches. There are very few neutral/objective reviews of both products that can be found, although there's a lot more description of and hints on how to use ChessBase, due to the ChessBase site articles and promotions. Functionally they're both very similar, although I can't speak to whether Chess Assistant has quite the full range of automatic sorting and searching functions (what ChessBase has under its tabs). Perhaps if you did a similar walkthrough for Chess Assistant, that would be a bonus for its users.

      Past ChessBase versions have been somewhat buggy, but starting with 11 and now with 13 as a 64-bit program, I haven't seen any issues with the standard functions; the new analysis features with CB13 may still need a bit of work, as I know they weren't quite fully functional on release. Building and copying games and databases are very easy, just drag and drop.

      Delete
  4. i came across this page whilst researching the differences between Chessbase and Chess Assistant. I'm looking to make the change from Chessbase, fed up with almost every update stripping out live parts of the program which were promised until the end of this calendar year. A few weeks after my purchase of CB12 (i've had almost every version down the years and spent well over £2000 on various CB products) i asked for my money back. The promised speed increases over CB11 turned out not to be true. Searching my reference database (many times the size of Mega Database) actually takes up to 15x LONGER on CB 12 as it did with CB11.
    Unfortunately, this article doesn't compare the two, it seems more like an advert for ChessBase...

    ReplyDelete
    Replies
    1. Hello Unknown,

      I too would be happy to find more reviews of Chess Assistant, since I couldn't locate any detailed ones when I was researching my last database purchase (and ended up going with CB13). I certainly had no intention of shelling out the money to purchase Chess Assistant as well just for comparison's sake - this isn't a review site, it's my training blog, so is not funded by anyone else. Perhaps in the future you could contribute a review of Chess Assistant when you purchase it, and post a link? Globulon above also posted some useful things about his experience with CA, although you didn't mention if that was helpful in your search; I hope it was.

      Re: the reported computer performance with CB12 vs. CB11, YMMV depending on your computer setup and the software's overhead and processor load. In 2012 I finally switched out of CB10 to CB11 and found that to be a significant upgrade in performance. I didn't see the point in purchasing CB12 when it came out, so didn't. CB13 is 64-bit and with some added/improved functions, so I made the jump there. Completely agree that CB12 is probably not worth the money to upgrade if you have CB11. And for anyone who wants to save money, use the free DB software and/or websites (a number of which are listed on this blog in the articles and comments).

      Re: the "advert for ChessBase" bit, if you look at the ChessBase site, it doesn't encourage mention of its commercial and freeware competitors and provide links to them.

      Delete