Fix the issue in getOrQueueForIdleConn where connectionPerished conn.…

[linfeip]

…isAlive involves network requests, which can cause the p.idleMu.Lock() when obtaining conn from the pool to be blocked for too long.

Summary

the lock in getOrQueueForIdleConn will cause connection acquisition to block under high concurrency due to the IO request in c.nc.Read(b[:]) inside isAlive.

func (p *pool) getOrQueueForIdleConn(w *wantConn) bool {
        p.idleMu.Lock()
	defer p.idleMu.Unlock()

	// Try to deliver an idle connection from the idleConns stack first.
	for len(p.idleConns) > 0 {
		conn := p.idleConns[len(p.idleConns)-1]
		p.idleConns = p.idleConns[:len(p.idleConns)-1]

		if conn == nil {
			continue
		}

		if reason, perished := connectionPerished(conn); perished {
     ...
}

func (c *connection) isAlive() bool {
	if c.nc == nil {
		return false
	}
      ...
	err := c.nc.SetReadDeadline(time.Now().Add(1 * time.Millisecond))
	if err != nil {
		return false
	}
	var b [1]byte
	_, err = c.nc.Read(b[:])
	return errors.Is(err, os.ErrDeadlineExceeded)
}

Background & Motivation

Before modifying isAlive

After

[linfeip]

I think it is unnecessary to verify whether this conn is alive here, because even if the connection is alive when it is obtained, there is no guarantee that the conn will still be alive when it is passed to the caller. The connection may have been disconnected between these two steps.