[FEATURE] Direct web app (ASGI/WSGI) testing from python tests

[DanLipsitt]

It seems that python tests using schemathesis.from_asgi are run using Requests, and expect a server to be running. I would like to run the tests directly as happens with schemathesis run --app on the command line. If this is already possible I was not able to figure out how by reading the docs and searching the code.

[Stranger6667]

Hi @DanLipsitt

Thank you for opening the issue :)

Here are some details of how things are composed for the ASGI case (WSGI is similar):

CLI imports the app, then automatically uses from_asgi for ASGI apps, and makes calls via case.call_asgi, which uses starlette's TestClient, which is a requests session with ASGI adapters.

Python tests are intended to use the same case.call_asgi. Though, it does not invoke FastAPI's events as reported in #1305 (there is a workaround for it)

It should be the same code that communicates with ASGI/WSGI apps both in Python tests & CLI. So, it does not expect the server to be running in an HTTP sense (e.g. urllib3 response is built manually from what is received through ASGI).

Though I might be missing something, or there could be some bug - what kind of behavior do you observe with Python tests that does not correspond to the "direct" model of execution?

[DanLipsitt]

Thanks for your reply, @Stranger6667!

My setup is very similar to the one in the #1305 bug you linked. I am getting the following error:

>           raise MissingSchema(error)
E           requests.exceptions.MissingSchema: Invalid URL '/health': No schema supplied. Perhaps you meant http:///health?

../__pypackages__/3.9/lib/requests/models.py:390: MissingSchema

This error from requests was deeply confusing at first because the library has a typo and means "scheme" (eg http) when it says "schema." This doesn't appear to be a schema error. All my endpoints produce this same error.

If this isn't enough info, I will try to provide a minimized example for you. Thanks again for your help and for this useful library.

[Stranger6667]

Indeed! I think I also had this error when I was checking one other issue, as far as I remember it was related to base_url.
So, it would be nice to prevent this kind of errors altogether - e.g. catch that requests error and reraise something more appropriate (or change the loaders API to avoid such combination of inputs).

If this isn't enough info, I will try to provide a minimized example for you. Thanks again for your help and for this useful library.

You're very welcome! It would be great if you could provide a code sample, as it will be easier for me to avoid solving the problem only for my particular setup :)

[DanLipsitt]

I've opened awtkns/fastapi-crudrouter#141 to add Schemathesis tests to that project. Those tests trigger the same issue described here, so hopefully you can use it as a reproduction and kill two birds with one stone.

[Stranger6667]

Awesome! I'll try to take a look at it this week :)

[DanLipsitt]

CI for awtkns/fastapi-crudrouter#141 appears to be waiting on maintainer approval, but you can see the issue if you check out my branch.

[DanLipsitt]

Tests have been approved and run in fastapi-crudrouter.

[Stranger6667]

Thank you for pinging me @DanLipsitt! I added a comment there with a suggestion that makes those tests properly executed at least on my machine. Please, take a look :)

[Stranger6667]

I think it would be nice to add call_asgi_and_validate and call_wsgi_and_validate methods for consistency to avoid using call_and_validate with ASGI / WSGI apps + some docs on that.

UPDATE: Not sure about that :(

[Stranger6667]

@DanLipsitt I added some validation & docs to #1399 - let me know if there is anything else I can do.