Lacinato ABX/Shootout-er blind testing audio software

Lacinato ABX is the only cross-platform ABX and shootout blind testing software for audio. It allows double-blind ABX testing of audio files (seeing if you can tell which file is being played), as well as "shootouts" (picking your favorite from a shuffled, anonymized list). Or you can just use it as a convenient way to switch between files (maintaining the current play position, if you like.) It is free to download and use (see license.). Although it's Java-based, you do not need to install Java to use it. Runs on Linux/Windows/Mac.

Features:

• ABX testing
• ABCD..X testing (more than two files)
• Shootouts, including fine-grained scoring of files for each shootout
• Jump from one file to the next while maintaining play position
• Automatically detects changes in audio files and prompts to refresh or ignore -- very handy for rapid cycling of versions that you want to re-test. (Unfortunately this doesn't work on Windows due to a Java bug.)
• Drag-and-drop audio files or groups of audio files or *.abx files into the program (see README.txt for the simple .abx file syntax and other details)
• Automatically loads .abx file it finds in startup directory (see README.txt)
• Per-file gain adjustment
• Per-file time offset (to synchronize files with each other)
• Configurable play range / looping
• Extensive keyboard shortcuts, and optional audio cues for success/failure in ABX choices, making it relatively accessible for screen readers (see README.txt file for some more details.)
• Save and recall sets of files/gains/offsets/loop points/etc. for recall or sharing with others
• Save test results as a text file (use shift-s shortcut; alternately you can just copy/paste)
• Supports a fair variety of file types, including MP3, WAV, AAC inside .mp4 files (not .aac files directly), FLAC, AIFF, and can even play audio streams from many video files. (FLAC will be losslessly converted in the background to WAV.) Other supported formats listed here.
• Optionally checks for new versions on startup (disable in config.txt if you are offline for faster startup).
• Calculates the p-value (which leads to the "confidence that results are better than chance")
• Will play high sample rates and bit-depths (limited by your audio system only) but make sure your operating system isn't downsampling. E.g. in Windows you must set your fancy sound interface as the default interface, and you need to choose the default sample rate / bit depth to use for the device. Lacinato ABX/Shootouter will not change the sample rate or bit depth as it plays: it is at the mercy of your operating system. You can set your OS to the highest rate you will be testing, but if you play files of a lower sampling rate, upsampling will occur.

Homebrewed Music made a tutorial walkthrough and some videos about the ABX/Shootouter (version 2.2).

Simply download one of the all-in-one packages below and it will run self-contained as a normal app (no installation or setup of anything is required -- super easy). Extract the contents of the archive into a normal folder before running the program, though!

If you do have Java 11 installed, along with the required JavaFX component, you can just download the much smaller .jar file and run that ("java -jar ABX.jar"), but there are some potential difficulties with paths and modules and so forth that make this complicated (and I won't be able to help if it doesn't work for you). If it doesn't work, you might try running with the following CLI options (potentially with modified jfx path for your system if necessary):
--module-path "$DIR"/lib/jfx/lib --add-modules javafx.controls,javafx.swing,javafx.media

DOWNLOADS:

YOU MUST EXTRACT THE CONTENTS OF THE ARCHIVE BEFORE RUNNING. SIMPLY OPENING THE ARCHIVE AND RUNNING "ABX" WILL NOT WORK.

All-in-one packages (57 to 71MB):

Or download just the .jar file (64k), but that requires Java 11, JavaFX, and some technical expertise (see above).

Using a Mac and can't open "because the developer cannot be verified"? Try these tips.

Keyboard shortcuts:

• 1 through 0: play file in slots 1 through 10; hold shift to choose the file during a test; hold control to choose the audio file to use in the slot
• a through j: play file in slots 1 through 10; hold shift to choose the file during a test; hold control to choose the audio file to use in the slot
(note that on some platforms the numbers will work better than the letters for file choosing and playing)

• w: add a row (used to add a new file to the list after the first couple slots are filled). For visually-impaired users: in some program states this won't work; in those cases use the numbers/letters described above with the control key.
• space bar: stop any playing audio
• ; and ': move left right 5 seconds; hold control to move 30 seconds
• r: show/hide results
• shift-r: Reset the ABX/shootout test status
• x: Start ABX, or play "X", or go to next "X"
• s: start shootout, or go to next shootout
• shift-s: Save the test results to a text file
• k: toggle "keep pos." -- the toggle that keeps the playing position as files are switched
• l: toggle loop
• n: go to next "X" or next shootout, if in either
• shift-n: toggle the status of the Notification sound indicating success or failure during an ABX test
• [ and ]: set the start/end point for the loop, shift-[ and shift-] to reset start and end positions to start/end of file
• o: open .abx fileset
• up/down arrow keys: (when gain/offset fields are focused) increase/decrease gain or time offset; hold ctrl for fine adjust

Shootout scoring:

Generally shootouts are for choosing which file is your favorite. In that case, you can just click "choose" for each shootout, and ignore the scoring sliders. Sometimes you may want to score the files as you go: for each shootout, you can apply a score from 0 to 10, which you can also think of as awarding "points". When you view the results later, you can see what the average score was for each file, the total "points" allocated to each file, as well what percentage of the total allocated points each file received.

Important notes on ABX testing:

• Pick two or more files that vary only in the way you want to test yourself for (except for time offset and levels -- those can be adjusted.)
• Click "Start AB*X" or "Start Shootout" to start the tests, and click "choose" to make your choice.
• You can use the per-file "play" buttons while ABX testing to play A, B, etc., but to do a more proper test, that could be considered cheating. (You must use them for a Shootout, of course.)
• Using the notification bell also violates the principals of serious ABX testing, since you're not supposed to get feedback as you go along, but it can be handy in some cases.
• After 15 or so tests, click "Show/Hide Results" to see how you did.
• For ABX testing, the "Confidence" value is the percentage chance that your results are better than chance. E.g. if you got a 95% accuracy result but the Confidence value is 60%, that means that the accuracy result is not statistically significant. A common standard is to require 95% or better confidence in a result before considering it meaningful. For example, choosing 12 out of 16 correctly results in an accuracy of 75% with a a confidence of 96% -- this is a much more meaningful result, implying strongly that you can identify 75% correctly and that that result is not due to chance.
• Some recommend not to do more than 25 or so tests for a given pair of files, because listening fatigue will start to bias your results.
• Taking sets of tests over and over for the same pair of files until you get the result you are looking for renders your results meaningless (after all, even a 95% confidence result is wrong 5% of the time, so if you run sets of trials 20 times, guessing completely at random, one of them will give you that impressive-seeming result.) You should be able to reliably repeat a result for it to count. Pulling up a couple files and getting a 12 out of 15 doesn't mean much unless you can repeat it. Try doing multiple tests over several days
• On that theme: the internet is a big place, so if you post two totally identical files and 300 people download them and ABX test them in a trial of 20 comparisons, 15 of those people will likely get 15 or more correct with a 95%+ confidence result, purely by chance, and will proudly post their results. This obviously does not mean that a small group of people have golden ears that can hear the difference between two digitally-identical files, so beware the impressions you get from reading forum posts.
• Lacinato ABX uses an exact binomial distribution algorithm to calculate the p-value.

Use/citations in academia:

Lacinato ABX has been cited in a few papers.

Let me know if you are aware of any others, or if you yourself are using it for research!

• Βότσκος, Σωκράτης. Acoustical analysis of Pan's caves across Greece, and correlation with the acoustics of the woodwinds. ("Ακουστική ανάλυση σπηλαίων του Πάνα ανά την Ελλάδα, και συσχετισμός με τη φυσιολογία των ξύλινων πνευστών μουσικών οργάνων.") (2020).
• Raivio, Matti-Mikael. Subjective and objective differences between analog compressors and software plugins that model them: 1176 and LA-2A compressors as case examples. ("Analogisten kompressorien ja niitä mallintavien ohjelmistoliitännäisten subjektiiviset ja objektiiviset eroavaisuudet: tapausesimerkkeinä") 1176-ja LA-2A-kompressorit." (2019).
• Feller, F. (2018). Audio Production and Critical Listening: Technical Ear Training, in Computer Music Journal, vol. 42, no. 1, pp. 82-86.
• Corey, J. (2016). Audio production and critical listening: Technical ear training. Routledge.
• Dhorje, R. S., & Badadapure, P. P. R. (2015). High Quality Voice Morphing based on Residual Prediction Technique. 5(11), pp. 378–382.
• Shenhuy, Kaan. Archaeacoustics: A Perceptual Study (2014)

Notes on Linux version:

Lacinato ABX/Shootouter should "just work" on most linux distributions, but some releases may need some additional configuration. Any distribution with pulseaudio installed, and most without it installed, should work.

If Lacinato ABX can not open an MP3 file because the codecs are not installed on your system, it will losslessly convert it to WAV using a third-party library. If the codecs are installed on your system, it will not need to do a conversion. Lacinato ABX uses Java 11, which requires libavcodec, libavformat, and libavutil to be installed for direct MP3 support -- more recent Linux distributions use more recent versions of those packages than are supported by Java, which may cause MP3 files to require the automatic conversion despite the codecs being present. This issue comes and goes with new releases of the Java framework. Test things first before bothering to install anything -- it might just work fine for you. I'd strongly recommend not worrying about any of this. If for some reason you want to install the older library versions in order to achieve "no-conversion" MP3 playback, the older versions of libavcodec/libavformat/libavutils can co-exist peacefully with the newer versions just fine and shouldn't cause any problems. Lacinato ABX will remove the temporary files it creates during on-the-fly conversion. They are stored in /tmp by default (configurable in config.txt).

How to install those libraries will depend on your distribution, but for distros based on Ubuntu, the packages you want to install are libavutil51, libavcodec53, and libavformat53.

If you're not using an Ubuntu-based distribution, see if you can find a source of repositories for older versions of your distribution that used the correct package versions. Failing that, you can seek out .deb package files for them and install them (e.g. with "dpkg -i") but I have not had much luck with that. If you find a solution for a particular distribution, or can't figure one out, please get in touch.

If you don't have have pulseaudio installed -- i.e. you're just using ALSA -- then things should work fine if you are using the default sound interface. If you have configured a sound interface in .asoundrc, e.g. for an external sound interface, then you might run in to issues if you don't configure it to be able to handle multiple audio streams at the same time.

To test if your interface can handle multiple streams at the same time, try playing a video in YouTube while you play an audio file in a music player. If both streams play through the interface simultaneously, then Lacinato ABX/Shootouter should work fine.

For example, Debian Wheezy 7.7 uses ALSA. This ABX software creates a different audio stream for each media file you are comparing. If you configure an external interface and don't set up multi-stream mixing, you will only be able to have one audio file open in the ABX program at a time, making it useless. You could read up on dmix -- I believe that enabling it in .asoundrc will fix that problem. Let me know if it does. Installing pulseaudio should also solve the issue, but read up on the subject before jumping in.

Change log:

2.43 update to JDK 11 and fix Mac version again; add ARM version for Mac; retire Linux 32bit version
2.42 fixes a bug in the Mac version that prevented it from running
2.41 fixes a decoding bug with some MP3 and WAV files on Windows (and updates the underlying media framework for all platforms)
2.40 fixes a decoding bug with some FLAC files (24bit depth, in one case)
2.39 fixes a few seeking/looping bugs, enables check for new version on startup (optionally), adds FLAC support via conversion to WAV, as well as MP3 support for linux distros that lack the needed system libraries, and adds a config.txt to set configuration variables
2.38 accessibility release: adds a few keyboard shortcuts, the ability to save results to a text file, a simpler alternate .abx syntax, a bell/buzzer for ABX guess success/failure, and fixes a few very minor bugs
2.37 fixes MP3 playback in some recent operating system versions
2.36 adds optional scoring of files during shootouts, auto-detection of changed audio files (except on Win due to Java bug), keyboard shortcuts, various UI improvements including protection against loss of test results
2.35 prevents freezes on Mac OS X (please report if you discover otherwise)
2.34 improves a lot of UI stuf
2.33 fixes some number parsing bugs
2.32 supports opening .abx files directly when associated in OS
2.31 adds drag/drop for *.abx files

License: Lacinato ABX/Shootouter is (c) 2014-2021 Casey Connor. It is free to download and use, but should not be distributed.