Audacious

Disregard the previous URLs.

They should be:

$ wget https://ws.audioscrobbler.com/2.0/
--2013-09-05 22:32:56--  https://ws.audioscrobbler.com/2.0/
Resolving ws.audioscrobbler.com (ws.audioscrobbler.com)... 195.24.232.205
Connecting to ws.audioscrobbler.com (ws.audioscrobbler.com)|195.24.232.205|:443... connected.
ERROR: The certificate of `ws.audioscrobbler.com' is not trusted.
ERROR: The certificate of `ws.audioscrobbler.com' hasn't got a known issuer.

$ openssl s_client -showcerts -connect ws.audioscrobbler.com:443
CONNECTED(00000003)
depth=2 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
 0 s:/OU=Domain Control Validated/CN=*.audioscrobbler.com
   i:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA - G2
(...)
 1 s:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
(...)
 2 s:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA - G2
   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
(...)
Server certificate
subject=/OU=Domain Control Validated/CN=*.audioscrobbler.com
issuer=/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA - G2
(...)
    Verify return code: 19 (self signed certificate in certificate chain)

Currently I built the scrobbler plugin with libcurl4-openssl-dev, instead of libcurl4-gnutls-dev, to avoid this problem.

It works on my Debian Wheezy machine. :)

Thanks, Ruey-Yuan.

After investigating this, it seems that the issue may be related with the order the certificates are sent by last.fm.
OpenSSL seems to be more resilient to server (mis?)configurations. After checking if it really solves this issue, a strong dependency on OpenSSL to avoid compiling against GnuTLS may solve this.

Relating to the above openssl command, if CApath is given on the command line, OpenSSL will validate the certificate by looking at the installed CA certs:

$ openssl s_client -connect ws.audioscrobbler.com:443 -CApath /etc/ssl/certs/
(...)
    Verify return code: 0 (ok)

Doing the same with GnuTLS will not:

$ gnutls-cli -p 443 ws.audioscrobbler.com --x509cafile /etc/ssl/certs/ca-certificates.crt 
Processed 159 CA certificate(s).
Resolving 'ws.audioscrobbler.com'...
Connecting to '195.24.232.205:443'...
*** Verifying server certificate failed...
*** Fatal error: Error in the certificate.
*** Handshake has failed
GnuTLS error: Error in the certificate.

But I'll need further confirmation on this, to get rid of any biases.

Trying to load https://last.fm in Firefox on Windows 7 also complains about a bad certificate. I've notified last.fm of the problem.
http://www.last.fm/forum/21713/_/2215292

I couldn't reproduce that with an up to date Firefox (23.0.1) and Windows 7 (64-bit) with all updates installed.
Did you try with all updates?

Anyway, the address in question is ws.audioscrobbler.com, not last.fm. Can you reproduce that on Windows with ws.audioscrobbler.com?

I just now updated Firefox to 24.0 (it was 23.0.1) and that fixed the warning on https://last.fm. Neither 23.0.1 nor 24.0 had a problem with https://ws.audioscrobbler.com.

I hate these problems where none of the symptoms make any sense ...

I'll try to look at this with some more time during the next weekend. No promises as usual, though.

John: on #321 you said that you also see this problem on Windows, with OpenSSL. Does that still happen? Windows updates frequently update certificates. Some needed certificate could been missing (?) but now isn't, coincidentally with you updating Firefox.

• scrobbler built with libcurl-gnutls, doesn't recognise the certificate.
• scrobbler built with libcurl-openssl, trusting Ruei-Yua's comment (#343-2), recognises the certificate

• scrobbler built with libcurl-openssl, doesn't recognise the certificate.

This is starting to look more and more a last.fm-side problem.

Here are the results from compiling and using the scrobbler on a Debian testing machine, with all updates as of when I performed the tests (Sep 21st, 2013):

libcurl4-gnutls-dev 7.32.0-1:

scrobbler_communication.c:151 [send_message_to_lastfm]: Could not communicate with last.fm: Peer certificate cannot be authenticated with given CA certificates.

libcurl4-nss-dev 7.32.0-1:

scrobbler_communication.c:151 [send_message_to_lastfm]: Could not communicate with last.fm: Problem with the SSL CA cert (path? access rights?).

libcurl4-openssl-dev 7.32.0-1:
Everything works.

I also tested changing the URL to https://google.com and the connection was well established with both GnuTLS and OpenSSL. Obviously, Google didn't like the subsequent requests, but it's cert was recognised, that's all.
Using the NSS backend gave the same message as before.

• The last.fm certificate is not OK;
• NSS support is not well configured on my machine. Apparently some additional configuration is needed to build a certificate DB, which could be built by the distro¹[2] but for some reason it isn't. It makes no sense to force users/installers to do this kind of magic just to scrobble from Audacious.

1 - http://curl.haxx.se/docs/sslcerts.html (see Peer SSL Verification with NSS)
2 - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=537866

I still get this on Windows:

scrobbler_communication.c:151 [send_message_to_lastfm]: Could not communicate with last.fm: Peer certificate cannot be authenticated with given CA certificates.

I have finally received a confirmation from last.fm that the issue is on their side and that they are working to solve it.

Nevertheless, please leave this issue open until the problem is definitely solved.

Luís Picciochi wrote:

Nevertheless, please leave this issue open until the problem is definitely solved.

Sure.

Luís, what is known about the correct certificate?

I still have no answer from last.fm.

However, yesterday I updated my wget installation, and that came with a dependency on libgnutls28. The previous version was libgnutls26.
The libgnutls26 is version 2.12.23-7, libgnutls28 is 3.2.4-4.

wget is now able to connect to https://ws.audioscrobbler.com , although it gets a 400 error subsequently (but that's not an issue):

$ wget https://ws.audioscrobbler.com/2.0/
--2013-10-08 20:02:34--  https://ws.audioscrobbler.com/2.0/
Resolving ws.audioscrobbler.com (ws.audioscrobbler.com)... 195.24.233.55
Connecting to ws.audioscrobbler.com (ws.audioscrobbler.com)|195.24.233.55|:443... connected.
HTTP request sent, awaiting response... 400 Bad Request
2013-10-08 20:02:36 ERROR 400: Bad Request.
$

gnutls-bin now also depends on libgnutls28:

$ gnutls-cli -p 443 ws.audioscrobbler.com --x509cafile /etc/ssl/certs/ca-certificates.crt 
Processed 164 CA certificate(s).
Resolving 'ws.audioscrobbler.com'...
Connecting to '195.24.233.55:443'...

- Certificate type: X.509
- Got a certificate list of 3 certificates.
- Certificate[0] info:
(...)
- Status: The certificate is trusted. 
(...)
- Handshake was completed

This is leading me to think that quite likely, libgnutls28 (v3.2) is capable of dealing with last.fm's apparently crapped-up certificates. I'm planning to test this further by compiling libcurl against this more recent libgnutls, and see how the scrobbler goes. I'll probably do this over the weekend.

If there's anyone on the audience with more free time than me to do this, feel free to reply with your experience.

I just tested the scrobbler with libcurl 7.33.0, which was compiled against GnuTLS 3.2.4.
The scrobbler was able to contact last.fm, and submitted the most recent tracks. Due to a restriction on last.fm's side, older plays were lost. I could only submit tracks played on the last week. The others were at least 3 weeks old, and last.fm happily discarded them.

The solution is then to enforce to build against libcurl, which has to have been built either against libopenssl, or against libgnutls >=3.2.4 (this might be trickier than it seems).

jlindgren: when you tested on Windows, what was libcurl built against? libcurl? libopenssl? Which version?

I'll leave this here so it gets noticed:
Users will lose their oldest scrobbles due to this issue. If users want to re-scrobble their tracks, they should make a backup of ~/.config/audacious/scrobbler.log before upgrading libgnutls and prepare to re-scrobble them after this is solved. The backed-up scrobbler.log can be opened with any text editor.

Of course, that should read: what was libcurl built against? libgnutls? libopenssl?

I am using OpenSSL 1.0.1e on Windows. The first problem was that OpenSSL didn't know where to look for CA certificates. After copying some certificates from my Linux partition, I can connect to google.com:443 successfully, but last.fm:443 is giving me the same output as in the original bug description:

depth=0 OU = Domain Control Validated, CN = *.last.fm
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 OU = Domain Control Validated, CN = *.last.fm
verify error:num=27:certificate not trusted
verify return:1
depth=0 OU = Domain Control Validated, CN = *.last.fm
verify error:num=21:unable to verify the first certificate
verify return:1

Edit: I get the same output when I run openssl from the command line in Arch Linux, but in spite of that, the scrobbler can connect. So maybe libcurl+OpenSSL is more tolerant than OpenSSL on its own. I will try some more tests on Windows when I can find the time.

Please don't test against last.fm:443 but ws.audioscrobbler.com:443 .

Luís Picciochi wrote:

I still have no answer from last.fm.

Have you (or anyone else) tried to contact them one more time?

Due to a restriction on last.fm's side, older plays were lost. I could only submit tracks played on the last week. The others were at least 3 weeks old, and last.fm happily discarded them.
...
If users want to re-scrobble their tracks, they should make a backup of <config> before upgrading libgnutls and prepare to re-scrobble them after this is solved

And this looks like separate issue. Do you have any more info on it?

I still have no idea on how to portably ensure libcurl was built against gnutls >= 3.2.4. We could check libcurl's version string, but that seems too fragile for the long term. Suggestions are welcome.

Bob Bib wrote:

Luís Picciochi wrote:

I still have no answer from last.fm.

Have you (or anyone else) tried to contact them one more time?

I was able to communicate with a last.fm person again today. There's still no progress on their side. The issue is still "in the backlog".

Due to a restriction on last.fm's side, older plays were lost. I could only submit tracks played on the last week. The others were at least 3 weeks old, and last.fm happily discarded them.
...
If users want to re-scrobble their tracks, they should make a backup of <config> before upgrading libgnutls and prepare to re-scrobble them after this is solved

And this looks like separate issue. Do you have any more info on it?

Yes and no.
I got confirmation from them that scrobbles older than 2 weeks are bluntly discarded by last.fm.

The Audacious scrobbler implements their API quite strictly, so where it says:

http://www.last.fm/api/scrobbling wrote:

Lfm error codes that indicate a scrobble request should be retried are:

• 11. Service Offline - This service is temporarily offline, try again later.
• 16. The service is temporarily unavailable, please try again.

Additionally this lfm error code indicates that the client should reauthenticate to get a new session key before retrying the request:

• 9. Invalid session key - Please re-authenticate

All other error codes indicate the scrobble request was incorrectly formed in some way and should not be retried.

I have interpreted this as to only retry attempts that got those errors. last.fm returns an "OK" status for discarded tracks, so the scrobbler happily goes on with that and doesn't look back.
A workaround can be implemented to keep retrying refused, old tracks. I don't like this solution as it goes against last.fm's spec, but I agree that this is better than losing scrobbles. Once again, I'll try to implement that over the weekend.

Luís Picciochi wrote:

I got confirmation from them that scrobbles older than 2 weeks are bluntly discarded by last.fm.

The Audacious scrobbler implements their API quite strictly, so where it says:
...

I just wonder if it is an old bad behavior of Last.fm or a new "bug feature";
if it's new, it obviously can be not reflected in documentation...

Bob Bib wrote:

I just wonder if it is an old bad behavior of Last.fm or a new "bug feature";
if it's new, it obviously can be not reflected in documentation...

Last.fm has always had a threshold for dropping old scrobbles, and they always documented it. If that threshold was reduced to 2 weeks or 2 days, it's indifferent for the correctness of the specification & implementations.

I'm not seeing a clean way out of this until either last.fm fix their certificates and/or libcurl packagers increase their dependency on libgnutls to libgnutls28.

Until then, I'll probably end up implementing a dirty, version string-based hack to prevent audacious to compile against libcurl with libgnutls < 28.

Luís Picciochi wrote:

Until then, I'll probably end up implementing a dirty, version string-based hack to prevent audacious to compile against libcurl with libgnutls < 28.

I don't think it's necessary to add a compile-time check, especially since GnuTLS isn't even a direct dependency. The issue should be documented, though. If you email me a few sentences summarizing the issue, I'll add them to http://audacious-media-player.org/problems.

I started working on a quick fix so no scrobbles are lost. See http://redmine.audacious-media-player.org/issues/260#note-1

Once this is commited, users will have to update their Audacious before last.fm fixes their servers. If not, they will still have to back-up their scrobbler.log file (which I recommend doing ASAP anyway).

jlindgren: I'll hand you a description of these problems + workarounds once I have the code ready for integrating with the main branch (hopefully tomorrow or on Wednesday).

Luís, whether correctly I understand that I will have to update a Audacious with your decision before there will be a communication with last.fm?

Linking back to http://audacious-media-player.org/news/24-scrobbler-issue for reference.

What progress now?

Hi from Last.hq,

Sorry this has taken so long, but we've finally addressed the ordering of the certificates and this problem should now be resolved. If it isn't, please let me know (best to send a pm to my last.fm account) and I'll ask the team to take another look.

Cheers,
-Jon

Well, better late than never. Thanks for the update, Jon.

I confirm that at least some communication is possible with audacious 3.4.1 (with libcurl built against an old libgnutls). I didn't test all of the interactions yet (including scrobbling), though.

Can someone confirm if the Windows version is also working?

I was looking into the Windows version a while back, and it seems that OpenSSL doesn't know how to access the client-side certificates that Windows provides. There are various bits of code posted online that will fetch the certificates and feed them into OpenSSL, but I haven't implemented such a workaround in Audacious (nor will I have time in the near future).

Audacious 3.4.2 communicate with Last.fm (I checked by logging into another profile).

Will I lose my listening when I'm connected to Last.fm into base profile?

Виктор Пономарёв wrote:

Audacious 3.4.2 communicate with Last.fm (I checked by logging into another profile).

Will I lose my listening when I'm connected to Last.fm into base profile?

I meant Linux version.

Copy the ~/.config/audacious/scrobbler.log file to a backup location.

Then start Audacious configured with your profile. Look at your last.fm profile to see all your old plays start appearing.

You didn't happen to start audacious with -V and save the output to a file, did you?

Luís Picciochi wrote:

You didn't happen to start audacious with -V and save the output to a file, did you?

Yes, of course.

I plan to clear the list of listenings and reload sсrobbler.log. Please remind me the command with -V attribute and output parameter.

I deleted the log from this issue because it contained your full submissions, including signature - this can allow others to re-scrobble those plays. I should remove that output from the scrobbler's debug mode.

Anyway, thanks for all the input. I'll try to look into this on Sunday. These are busy days around here. :-)

Can we close this since the certificate ordering problem is resolved? Further support questions can be asked and answered on the forums.