Skip to content

transport: Prevent sending negative timeouts #8312

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 14, 2025

Conversation

arjan-bal
Copy link
Contributor

@arjan-bal arjan-bal commented May 12, 2025

After #8290 is merged, negative timeout values fail the request with an internal status instead of a deadline exceeded. This test uses negative deadlines and can fail if the server sends an internal error before the client sees the deadline expire.

Sample failure: https://github.com/grpc/grpc-go/actions/runs/14979137997/job/42078795928?pr=8308

The client checks the deadline expiry before creating a stream.

grpc-go/stream.go

Lines 410 to 413 in 709023d

func (cs *clientStream) newAttemptLocked(isTransparent bool) (*csAttempt, error) {
if err := cs.ctx.Err(); err != nil {
return nil, toRPCErr(err)
}

However, the deadline may expire after this check, but before the grpc-timeout header is created.

RELEASE NOTES:

  • transport: Fail RPCs on the client when using extremely short contexts that expire before the grpc-timeout header is created.

@arjan-bal arjan-bal added this to the 1.73 Release milestone May 12, 2025
@arjan-bal arjan-bal added Type: Testing Area: Testing Includes tests and testing utilities that we have for unit and e2e tests within our repo. labels May 12, 2025
@arjan-bal arjan-bal changed the title test: Avoid using negative deadline in test test: Avoid using negative timeout in test May 12, 2025
@arjan-bal arjan-bal requested a review from dfawley May 12, 2025 18:35
Copy link

codecov bot commented May 12, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.22%. Comparing base (d36b02e) to head (e1bcc17).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8312      +/-   ##
==========================================
+ Coverage   82.19%   82.22%   +0.03%     
==========================================
  Files         419      419              
  Lines       41992    41992              
==========================================
+ Hits        34516    34530      +14     
+ Misses       6008     5997      -11     
+ Partials     1468     1465       -3     
Files with missing lines Coverage Δ
internal/transport/http2_client.go 92.15% <100.00%> (-0.52%) ⬇️

... and 18 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@@ -5342,7 +5342,7 @@ func testRPCTimeout(t *testing.T, e env) {
ResponseSize: respSize,
Payload: payload,
}
for i := -1; i <= 10; i++ {
for i := 1; i <= 10; i++ {
ctx, cancel := context.WithTimeout(context.Background(), time.Duration(i)*time.Millisecond)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is even worrysome. If it takes >1ms before the timeout is computed by grpc, then it could fail. It seems like there's probably something in our code that we should change that prevents the client from sending a negative deadline instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made the fix to validate the timeout on the client side and return a DeadlineExceeded error.

@dfawley dfawley assigned arjan-bal and unassigned dfawley May 12, 2025
@arjan-bal arjan-bal force-pushed the fix-negative-deadline-in-test branch from 87ad73f to bcc45b0 Compare May 13, 2025 05:08
@arjan-bal arjan-bal changed the title test: Avoid using negative timeout in test transport: Prevent sending negative timeouts May 13, 2025
@arjan-bal arjan-bal assigned dfawley and unassigned arjan-bal May 13, 2025
@arjan-bal arjan-bal force-pushed the fix-negative-deadline-in-test branch from bcc45b0 to e1bcc17 Compare May 13, 2025 05:16
Copy link
Member

@dfawley dfawley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thank you for fixing this!

defer server.Stop()

if err := server.StartClient(); err != nil {
t.Fatal(err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should say what it's doing in the error message, to make debugging a failure easier.

dfawley

This comment was marked as duplicate.

@dfawley dfawley assigned arjan-bal and unassigned arjan-bal and dfawley May 13, 2025
@arjan-bal arjan-bal merged commit e3f13e7 into grpc:master May 14, 2025
12 of 14 checks passed
@arjan-bal arjan-bal deleted the fix-negative-deadline-in-test branch May 14, 2025 03:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Testing Includes tests and testing utilities that we have for unit and e2e tests within our repo. Type: Testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants