In LibreOffice we've long supported Microsoft Office's "Office Binary Document RC4 Encryption" for decrypting xls, doc and ppt. But somewhere along the line the Microsoft Office encryption scheme was replaced by a new one, "Office Binary Document RC4 CryptoAPI Encryption", which we didn't support. This is what the error dialog of...
"The encryption method used in this document is not supported. Only Microsoft Office 97/2000 compatible password encryption is supported."
...from LibreOffice is telling you when you open, for example, an encrypted xls saved by a contemporary Microsoft Excel version.
I got the newer scheme working this morning for xls, so from LibreOffice 5-3 onwards (I may backport to upstream 5-2 and Fedora 5-1) these variants can be successfully decrypted and viewed in LibreOffice.