All site contents ©2018 Casey Connor unless otherwise noted.

Lacinato ABX/Shootout-er blind testing audio software

Last Updated: Wednesday, 18 April 2018

Interested in a web application instead?
See the Lacinato WebABX/Shootouter.

Current version: 2.40
changelist is at the bottom of this page

Lacinato ABX is the only cross-platform ABX and shootout blind testing software for audio. It allows double-blind ABX testing of audio files (seeing if you can tell which file is being played), as well as "shootouts" (picking your favorite from a shuffled, anonymized list). Or you can just use it as a convenient way to switch between files (maintaining the current play position, if you like.) It is free to download and use (see license.). Although it's Java-based, you do not need to install Java to use it. Runs on Linux/Windows/Mac.

Donations Welcome! - support the cause of accurate listening!

Contact me if you have questions/corrections/suggestions.

Features:

  • ABX testing
  • ABCD..X testing (more than two files)
  • Shootouts, including fine-grained scoring of files for each shootout
  • Jump from one file to the next while maintaining play position
  • Automatically detects changes in audio files and prompts to refresh or ignore -- very handy for rapid cycling of versions that you want to re-test. (Unfortunately this doesn't work on Windows due to a Java bug.)
  • Drag-and-drop audio files or groups of audio files or *.abx files into the program (see README.txt for the simple .abx file syntax and other details)
  • Automatically loads .abx file it finds in startup directory (see README.txt)
  • Per-file gain adjustment
  • Per-file time offset (to synchronize files with each other)
  • Configurable play range / looping
  • Extensive keyboard shortcuts, and optional audio cues for success/failure in ABX choices, making it relatively accessible for screen readers (see README.txt file for some more details.)
  • Save and recall sets of files/gains/offsets/loop points/etc. for recall or sharing with others
  • Save test results as a text file (use shift-s shortcut; alternately you can just copy/paste)
  • Supports a fair variety of file types, including MP3, WAV, AAC, FLAC, AIFF, and can even play audio streams from many video files. (FLAC will be converted in the background to WAV.)
  • Optionally checks for new versions on startup (disable in config.txt if you are offline for faster startup).
  • Calculates the p-value (which leads to the "confidence that results are better than chance")
  • Will play high sample rates and bit-depths (limited by your audio system only) but make sure your operating system isn't downsampling. E.g. in Windows you must set your fancy sound interface as the default interface, and you need to choose the default sample rate / bit depth to use for the device. Lacinato ABX/Shootouter will not change the sample rate or bit depth as it plays: it is at the mercy of your operating system. You can set your OS to the highest rate you will be testing, but if you play files of a lower sampling rate, upsampling will occur.

Homebrewed Music made a tutorial walkthrough and some videos about the ABX/Shootouter (version 2.2).

Simply download one of the all-in-one packages below and it will run self-contained as a normal app (no installation or setup of anything is required -- super easy). Extract the contents of the archive into a normal folder before running the program, though!

If you do have Java 8 installed (with the JavaFX component that is included in the Oracle version) you can just download the much smaller .jar file and run that ("java -jar ABX.jar").


DOWNLOADS:

YOU MUST EXTRACT THE CONTENTS OF THE ARCHIVE BEFORE RUNNING. SIMPLY OPENING THE ARCHIVE AND RUNNING "ABX" WILL NOT WORK.

All-in-one packages (57 to 71MB):

Linux: 32-bit64-bit
(see notes on linux, below!)
Mac: (no 32-bit)64-bit
(OS X 10.7.3 Lion or newer is required)
Windows: 32-bit64-bit

Or download just the .jar file (64k), but that requires Oracle's version of Java 8.


Keyboard shortcuts:

  • 1 through 0: play file in slots 1 through 10; hold shift to choose the file during a test; hold control to choose the audio file to use in the slot
  • a through j: play file in slots 1 through 10; hold shift to choose the file during a test; hold control to choose the audio file to use in the slot
  • (note that on some platforms the numbers will work better than the letters for file choosing and playing)
  • w: add a row (used to add a new file to the list after the first couple slots are filled). For visually-impaired users: in some program states this won't work; in those cases use the numbers/letters described above with the control key.
  • space bar: stop any playing audio
  • ; and ': move left right 5 seconds; hold control to move 30 seconds
  • r: show/hide results
  • shift-r: Reset the ABX/shootout test status
  • x: Start ABX, or play "X", or go to next "X"
  • s: start shootout, or go to next shootout
  • shift-s: Save the test results to a text file
  • k: toggle "keep pos." -- the toggle that keeps the playing position as files are switched
  • l: toggle loop
  • n: go to next "X" or next shootout, if in either
  • shift-n: toggle the status of the Notification sound indicating success or failure during an ABX test
  • [ and ]: set the start/end point for the loop, shift-[ and shift-] to reset start and end positions to start/end of file
  • o: open .abx fileset
  • up/down arrow keys: (when gain/offset fields are focused) increase/decrease gain or time offset; hold ctrl for fine adjust

Shootout scoring:

Generally shootouts are for choosing which file is your favorite. In that case, you can just click "choose" for each shootout, and ignore the scoring sliders. Sometimes you may want to score the files as you go: for each shootout, you can apply a score from 0 to 10, which you can also think of as awarding "points". When you view the results later, you can see what the average score was for each file, the total "points" allocated to each file, as well what percentage of the total allocated points each file received.

Important notes on ABX testing:

  • Pick two or more files that vary only in the way you want to test yourself for (except for time offset and levels -- those can be adjusted.)
  • Click "Start AB*X" or "Start Shootout" to start the tests, and click "choose" to make your choice.
  • You can use the per-file "play" buttons while ABX testing to play A, B, etc., but to do a more proper test, that could be considered cheating. (You must use them for a Shootout, of course.)
  • Using the notification bell also violates the principals of serious ABX testing, since you're not supposed to get feedback as you go along, but it can be handy in some cases.
  • After 15 or so tests, click "Show/Hide Results" to see how you did.
  • For ABX testing, the "Confidence" value is the percentage chance that your results are better than chance. E.g. if you got a 95% accuracy result but the Confidence value is 60%, that means that the accuracy result is not statistically significant. A common standard is to require 95% or better confidence in a result before considering it meaningful. For example, choosing 12 out of 16 correctly results in an accuracy of 75% with a a confidence of 96% -- this is a much more meaningful result, implying strongly that you can identify 75% correctly and that that result is not due to chance.
  • Some recommend not to do more than 25 or so tests for a given pair of files, because listening fatigue will start to bias your results.
  • Taking sets of tests over and over for the same pair of files until you get the result you are looking for renders your results meaningless (after all, even a 95% confidence result is wrong 5% of the time, so if you run sets of trials 20 times, guessing completely at random, one of them will give you that impressive-seeming result.) You should be able to reliably repeat a result for it to count. Pulling up a couple files and getting a 12 out of 15 doesn't mean much unless you can repeat it. Try doing multiple tests over several days
  • On that theme: the internet is a big place, so if you post two totally identical files and 300 people download them and ABX test them in a trial of 20 comparisons, 15 of those people will likely get 15 or more correct with a 95%+ confidence result, purely by chance, and will proudly post their results. This obviously does not mean that a small group of people have golden ears that can hear the difference between two digitally-identical files, so beware the impressions you get from reading forum posts.
  • Lacinato ABX uses an exact binomial distribution algorithm to calculate the p-value.

Notes on Linux version:

Lacinato ABX/Shootouter should "just work" on most linux distributions, but some releases may need some additional configuration. You can see Oracle's official supported systems list, but any distribution with pulseaudio installed, and most without it installed, should work.

If Lacinato ABX can not open an MP3 file because the codecs are not installed on your system, it will losslessly convert it to WAV using a third-party library. If the codecs are installed on your system, it will not need to do a conversion. Lacinato ABX uses Oracle's Java 8, which requires libavcodec, libavformat, and libavutil to be installed for direct MP3 support -- more recent Linux distributions use more recent versions of those packages than are supported by Java, which may cause MP3 files to require the automatic conversion despite the codecs being present. This issue comes and goes with new releases of the Java framework. Test things first before bothering to install anything -- it might just work fine for you. I'd strongly recommend not worrying about any of this. If for some reason you want to install the older library versions in order to achieve "no-conversion" MP3 playback, the older versions of libavcodec/libavformat/libavutils can co-exist peacefully with the newer versions just fine and shouldn't cause any problems. Lacinato ABX will remove the temporary files it creates during on-the-fly conversion. They are stored in /tmp by default (configurable in config.txt).

How to install those libraries will depend on your distribution, but for distros based on Ubuntu, the packages you want to install are libavutil51, libavcodec53, and libavformat53.

If you're not using an Ubuntu-based distribution, see if you can find a source of repositories for older versions of your distribution that used the correct package versions. Failing that, you can seek out .deb package files for them and install them (e.g. with "dpkg -i") but I have not had much luck with that. If you find a solution for a particular distribution, or can't figure one out, please get in touch.

If you don't have have pulseaudio installed -- i.e. you're just using ALSA -- then things should work fine if you are using the default sound interface. If you have configured a sound interface in .asoundrc, e.g. for an external sound interface, then you might run in to issues if you don't configure it to be able to handle multiple audio streams at the same time.

To test if your interface can handle multiple streams at the same time, try playing a video in YouTube while you play an audio file in a music player. If both streams play through the interface simultaneously, then Lacinato ABX/Shootouter should work fine.

For example, Debian Wheezy 7.7 uses ALSA. This ABX software creates a different audio stream for each media file you are comparing. If you configure an external interface and don't set up multi-stream mixing, you will only be able to have one audio file open in the ABX program at a time, making it useless. You could read up on dmix -- I believe that enabling it in .asoundrc will fix that problem. Let me know if it does. Installing pulseaudio should also solve the issue, but read up on the subject before jumping in.

Change list:

2.40 fixes a decoding bug with some FLAC files (24bit depth, in one case)
2.39 fixes a few seeking/looping bugs, enables check for new version on startup (optionally), adds FLAC support via conversion to WAV, as well as MP3 support for linux distros that lack the needed system libraries, and adds a config.txt to set configuration variables
2.38 accessibility release: adds a few keyboard shortcuts, the ability to save results to a text file, a simpler alternate .abx syntax, a bell/buzzer for ABX guess success/failure, and fixes a few very minor bugs
2.37 fixes MP3 playback in some recent operating system versions
2.36 adds optional scoring of files during shootouts, auto-detection of changed audio files (except on Win due to Java bug), keyboard shortcuts, various UI improvements including protection against loss of test results
2.35 prevents freezes on Mac OS X (please report if you discover otherwise)
2.34 improves a lot of UI stuf
2.33 fixes some number parsing bugs
2.32 supports opening .abx files directly when associated in OS
2.31 adds drag/drop for *.abx files

License: Lacinato ABX/Shootouter is (c) 2014-2018 Casey Connor. It is free to download and use, but should not be distributed.