HTTPS cURL request fails at 340th request when used in a recursive function

[tschoonen]

Description

Recently a consumer of an API I had made ran into some strange issues, upon inspection of his code I discovered that the problem was because he used cURL in a recursive function, once put in a regular loop the problems disappeared.

Further inspection revealed this only seemed to occur with HTTPS requests, not plain HTTP. The server that the requests are made to does not make a difference as I've tested several differen hosts.

The included code uses the default snakeoil Apache configuration, the extra cert opts do not affect the outcome if left out on a properly configured server with TLS.

The following code:

<?php

/**
 * This will end at request php/php-src#340
 */
function recursiveFunctionTest($request_no = 1)
{
	echo "Request no #$request_no...\n";

	$ch = curl_init();
	curl_setopt($ch, CURLOPT_URL, 'https://localhost/');
	curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

	curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
	curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
	curl_setopt($ch, CURLOPT_CAINFO, '/etc/ssl/certs/ssl-cert-snakeoil.pem');

	$response = curl_exec($ch);

	curl_close($ch);

	if ($response === false) {
		echo 'Request returned false', PHP_EOL;
		exit(1);
	}

	if ($request_no < 400) {
		recursiveFunctionTest(++$request_no);
	}

	echo "Done!\n";
}

recursiveFunctionTest();

Resulted in this output:

Request no php/php-src#1...
Request no php/php-src#2...
Request no php/php-src#3...
[...]
Request no php/php-src#338...
Request no php/php-src#339...
Request no php/php-src#340...
Request returned false

But I expected this output instead:

Request no php/php-src#1...
Request no php/php-src#2...
Request no php/php-src#3...
[...]
Request no php/php-src#398...
Request no php/php-src#399...
Request no php/php-src#400...
Done!

This code will run just fine:

<?php

/**
 * This will comfortably come to request php/php-src#400
 */
function whileLoopTest()
{
	$request_no = 0;

	do {
		$request_no++;

		echo "Request no #$request_no...\n";

		$ch = curl_init();
		curl_setopt($ch, CURLOPT_URL, 'https://localhost/');
		curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
		curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

		curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
		curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
		curl_setopt($ch, CURLOPT_CAINFO, '/etc/ssl/certs/ssl-cert-snakeoil.pem');

		$response = curl_exec($ch);

		curl_close($ch);

		if ($response === false) {
			echo 'Request returned false', PHP_EOL;
			exit(1);
		}
	} while($request_no < 400);

	echo "Done!\n";
}

whileLoopTest();

PHP Version

8.1.8

Operating System

Ubuntu 22.04 LTS

[KapitanOczywisty]

If you can, please show error before curl_close:

if(curl_errno($ch)){
    var_dump(curl_errno($ch), curl_error($ch));
}

Can you also check if adding unset($ch); after curl_close changes the behavior?

[cmb69]

Can you also check if adding unset($ch); after curl_close changes the behavior?

Very good point, because:

This function has no effect. Prior to PHP 8.0.0, this function was used to close the resource.

Also, it may help to add curl_setopt($ch, CURLOPT_VERBOSE, true);, to see cURLs debug diagnosis.

[tschoonen]

Using unset does seem to resolve the issue, this seems rather unintuitive since it's within a function so one would expect it to be unset once the function has run its course. And even so failing at the 340th request is at least a little strange.

I added both the verbose option and the curl_error() to the script to see what it says without the unset in the code. It says it has a problem setting the certificate file, however if I try it on a server with a regular certificate it complains about not being able to resolve the host (see output 2).

<?php

function recursiveFunctionTest($request_no = 1)
{
	echo "Request no #$request_no...\n";

	$ch = curl_init();
	curl_setopt($ch, CURLOPT_URL, 'https://localhost/');
	curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
	curl_setopt($ch, CURLOPT_VERBOSE, true);

	curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
	curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
	curl_setopt($ch, CURLOPT_CAINFO, '/etc/ssl/certs/ssl-cert-snakeoil.pem');

	$response = curl_exec($ch);

	if (curl_errno($ch)){
		var_dump(curl_errno($ch), curl_error($ch));
	}

	curl_close($ch);

	if ($response === false) {
		echo 'Request returned false', PHP_EOL;
		exit(1);
	}

	if ($request_no < 400) {
		recursiveFunctionTest(++$request_no);
	} else {
		echo "Done!\n";
	}
}

recursiveFunctionTest();

Output 1 (snakeoil cert):

Request no php/php-src#340...
*   Trying 127.0.0.1:443...
* Connected to localhost (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* error setting certificate file: /etc/ssl/certs/ssl-cert-snakeoil.pem
* Closing connection 0
int(77)
string(68) "error setting certificate file: /etc/ssl/certs/ssl-cert-snakeoil.pem"
Request returned false

Output 2 (regular cert):

Request no php/php-src#340...
* getaddrinfo() thread failed to start
* Could not resolve host: [MY_HOST]
* Closing connection 0
int(6)
string(36) "getaddrinfo() thread failed to start"
Request returned false

[cmb69]

Using unset does seem to resolve the issue, this seems rather unintuitive since it's within a function so one would expect it to be unset once the function has run its course.

It is, but in the recursive case, the function only terminates after all requests have been made.

I assume there is an issue with too many open files, in which case we can't do anything about it. Either unset the handle manually, or use the iterative version; or, even better, reuse the curl handle, instead of creating new handles for each request.

[tschoonen]

Using unset does seem to resolve the issue, this seems rather unintuitive since it's within a function so one would expect it to be unset once the function has run its course.

It is, but in the recursive case, the function only terminates after all requests have been made.

I assume there is an issue with too many open files, in which case we can't do anything about it. Either unset the handle manually, or use the iterative version; or, even better, reuse the curl handle, instead of creating new handles for each request.

Personally I always use a loop, not recursive functions (also because it used to cause problems in <7.0). But a client of mine did.

It does make sense, because if the cURL request is put in a separate function this problem no crops up. The cURL handle is then destroyed by the garbage collector once the request function has been completed.

However, I wouldn't know how a programmer could be made aware of this without having intricate knowledge of this particular problem.

<?php

function doRequest()
{
	$ch = curl_init();
	curl_setopt($ch, CURLOPT_URL, 'https://localhost/');
	curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

	curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
	curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
	curl_setopt($ch, CURLOPT_CAINFO, '/etc/ssl/certs/ssl-cert-snakeoil.pem');

	$response = curl_exec($ch);

	if ($response === false) {
		echo 'Request returned false', PHP_EOL;
		exit(1);
	}
}

function recursiveFunctionTest($request_no = 1)
{
	echo "Request no #$request_no...\n";

	doRequest();

	if ($request_no < 400) {
		recursiveFunctionTest(++$request_no);
	} else {
		echo "Done!\n";
	}
}

recursiveFunctionTest();

[cmb69]

However, I wouldn't know how a programmer could be made aware of this without having intricate knowledge of this particular problem.

Well, the best we can do is to improve the documentation. Particularly, the curl_close() page should mention how to actually release a CurlHandle (not only that the function no longer has any effect). And it seems prudent to mention there, that reusing of handles is preferable over creating and destroying the handles repeatedly.

[tschoonen]

However, I wouldn't know how a programmer could be made aware of this without having intricate knowledge of this particular problem.

Well, the best we can do is to improve the documentation. Particularly, the curl_close() page should mention how to actually release a CurlHandle (not only that the function no longer has any effect). And it seems prudent to mention there, that reusing of handles is preferable over creating and destroying the handles repeatedly.

That does seem like the best approach. However, shouldn't the cURL extension maybe give more a more informative error message when this occurs? Such as 'too many open files', assuming that is the case. As it stands now it would be quite hard to deduce why cURL failed to perform the request.

[cmb69]

However, shouldn't the cURL extension maybe give more a more informative error message when this occurs?

From what I can tell, we cannot influence that; the error is raised by libcurl. Trying to guess why libcurl failed doesn't look like a sensible approach. And after all, this is very much an edge-case; 400 requests are probably uncommon per se, but keeping the handles open is even more so.

[tschoonen]

Fair enough. Thanks for the helpful comments, I will relay this to my client.