Can I obtain TLS secrets from an HTTP client to decrypt my own HTTPS conversation?

0 投票
最新提问 用户: (660 分)

I am building a website recorder that acts like a proxy, in order to test web scrapers on an ongoing basis. It is split into three Docker containers, all on GNU/Linux: (1) a proxy, (2) an API and request queue, and (3) a simple web app.

It works fine for HTTP sites: I click a button in the web app, this makes a request to the API container, and that adds something to an internal request queue, which then requests the site via the proxy. The proxy records the site as it passes through.

However, I'd forgotten that one cannot record HTTPS site traffic, and now I've come to implement this, I've found that proxies just use the CONNECT verb, and then act as a data exchanger between the client and the target. I believe I cannot replay the same data chunks as part of the encryption uses a randomised, throwaway, symmetric key (however I have a script suitable for testing this, so I will do so just for the educational value!).

So, I was wondering if my fetching client could give up enough secrets for the proxy system to decode the byte-stream? I am using Wget to do the fetch, which I guess would be using OpenSSL. It does not need to be Wget though: if I were using a PHP script with file_get_contents with a stream context, can I ask the openssl module for the decryption keys?

(To be fair, I will probably not solve the problem in this fashion even if it is possible, I just thought it would be really interesting to learn a bit more about TLS. In practice, I will record a "null" entry against all secure websites in the proxy, and require the requesting service to notify the proxy of header/body data via an API call, so it can be later played back. They will of course have plaintext copies of these items).

发表于 用户: (660 分)
@jww: regarding your edit, isn't my question explicitly about the OpenSSL library, insofar as a client such as Wget would need to have access to encryption secrets from OpenSSL in order to be able to pass them onto a decryption program?
发表于 用户: (660 分)
No response to the above, so restoring tag - please get in touch before modifying it again.
发表于 用户: (140 分)
If this were possible, I'd be very concerned.
发表于 用户: (660 分)
Why @miken32? Encryption is, for most practical purposes, unbreakable because the client does not emit secrets to crackers who would want them. However, in a closed system, if it can be asked to give up those secrets with permission, surely that can be used for decryption without worrying that the encryption system is flawed in general?
发表于 用户: (660 分)
I've added a bounty, to tempt people into answering. It's a fun question without a real purpose, but I think it's interesting. If this is possible, I expect answers will say that one would have to write/modify ones own client, so as to be able to fish out the necessary secrets from OpenSSL. Of course, I am not looking for working code, as that would be rather broad - just the techniques involved. Does OpenSSL permit all secrets to be obtained by a client, without recompiling OpenSSL?
发表于 用户: (600 分)
By the way, if this is for a commercial project, I'd recommend using one of the existing HTTPS debugging proxies to do this, rather than attempting to write your own. I know of at least and that do exactly this, including TLS unwrapping. If this is for a hobby or educational project to learn about HTTPS then good on you and carry on!
发表于 用户: (660 分)
Good advice, thanks @Rich. The purpose is for a commercial project, but it's just for testing rather than production, and in any case I'm avoiding the need to handle TLS connections in the proxy by modifying them in the client and correcting that in the proxy later. I did try a ready-made solution (Wiremock) but it didn't work as I intended, and I had to back out a fair bit of work. At least if I write it, I will be able to modify it :-).

2 个回答

0 投票
最新回答 用户: (180 分)

Might be Referer header, will work Or try using CSRF tokens

What is a CSRF token ? What is its importance and how does it work?

发表于 用户: (660 分)
I think you've misunderstood the question. I'm asking if TLS secrets can be extracted out of an encryption library such as OpenSSL. This data certainly would not be made available to websites running inside a browser, so it could not be accessed via JS or by reading HTTP headers.
0 投票
最新回答 用户: (600 分)

Yes, I think you have a couple of options here.

HTTPS is specifically designed to thwart "Man in the Middle" attacks and eavesdroppers, which is essentially what you are trying to achieve. You can break some of its assumptions though, and defeat it.

At the start of the SSL connection, 1. the remote server presents its public key and its cert, 2. the client verifies the cert and 3. sends a session key encrypted with the server's public key. For more details, see e.g. An overview of the SSL or TLS handshake

You have two possible ways to circumvent this protection, in the scenario you describe:

1. Rewrite the TLS data, replacing the server's cert and key with your own

Since you control the communications channel, you could substitute the server's public key & cert with one that you control, at step (1). If you then ask the client to skip step (2) using the --no-check-certificate argument to wget, you can then have full access to the encrypted data.

This is how the Fiddler debugging proxy allows access to HTTPS traffic, see

2. Retrieve the session key from the client application

Since the client app knows the session key, if you could extract it, you could then decrypt the stream. I think this is what you had in mind in the question.

wget itself has no options to allow logging the session key (see "HTTPS (SSL/TLS) Options"), but it does look like its TLS library, "GnuTLS" has a debugging option that will do what you want, see "Debugging and auditing" in the GnuTLS docs:

SSLKEYLOGFILE When set to a filename, GnuTLS will append to it the session keys in the NSS Key Log format. That format can be read by wireshark and will allow decryption of the session for debugging.

Try setting the SSLKEYLOGFILE environment variable to a filename, and see if wget will then log your TLS session keys to that file? You might need to recompile wget with a debug build of GnuTLS. I haven't tried this myself.

发表于 用户: (660 分)
Excellent, number 2 looks very close to what I had in mind, but number 1 is very interesting - thank you! I need to do some reading here...
发表于 用户: (660 分)
I'm happy to mark this as the answer for now, but will leave the bounty open for the remainder of its run, in case anyone else wants a shot at it!
发表于 用户: (660 分)
Thanks again for your help on this. I am tied up with other work at present, but will come back when I can to try the environment variable you suggest. The bounty is yours!
欢迎来到 Security Q&A ,有什么不懂的可以尽管在这里提问,你将会收到社区其他成员的回答。