Skip to content

Architecture overview #603

Open
Open
@honzajavorek

Description

@honzajavorek

It would be awesome if Crawlee had an architecture overview like this https://docs.scrapy.org/en/latest/topics/architecture.html and explanations of all the concepts. As a beginner Crawlee user, I use it as a black box, but then I'm surprised by certain limitations or certain behavior. I don't know how the storage folder works, or the integration with the platform. I can read the docs about how to use it and what's supposed to happen, but otherwise, it's all just magic. I'd like to understand better how it works internally, at least in terms of a simple flow chart, which breaks it down to several black boxes.

Not sure if Docusaurus supports Mermaid, but even if not, you can use Mermaid to draft the chart here in GitHub Issues, before it makes it to the docs. See apify/apify-docs#1015 how it can be used for iterative collaboration.

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentation.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions