Preserve HTML entities in headers

[josevalim]

Closes #2114.
Closes #2120.

[josevalim]

cc @garazdawi

[github-actions]

📦 Docs artifacts are ready: https://github.com/elixir-lang/ex_doc/actions/runs/14911198110/artifacts/3087777715

[josevalim]

Btw, I think this is going to have some subtle issues, as we are using two different ways to compute headers (one using regexes and another without), so we may need to look into that before shipping this.

[garazdawi]

Tested it for us and it solves the issue and I haven't found any regressions so far :)

[josevalim]

@garazdawi please check again! I made a big refactoring that unified how we handle link headings, so we only do it once in a single place, using DocAST instead of relying too much on regexes.

[garazdawi]

Seems to work fine except for the small detail in map_tags.

[garazdawi]

Suggested change

	def map_tags({tag, attrs, inner, meta}, fun),
	do: fun.({tag, attrs, Enum.map(inner, &map_tags(&1, fun)), meta})
	def map_tags({tag, attrs, inner, meta}, fun) when is_list(inner),
	do: fun.({tag, attrs, Enum.map(inner, &map_tags(&1, fun)), meta})
	def map_tags({tag, attrs, inner, meta}, fun),
	do: fun.({tag, attrs, inner, meta})

When edoc is used to generate the doc ast, inner can be a string without being wrapped in a list.

[josevalim]

I basically wrote all of the traversal functions assuming that the contents is a list. :( Any chance we could normalize edoc in the long term? I will also improve the traversal functions here meanwhile.

[garazdawi]

I basically wrote all of the traversal functions assuming that the contents is a list.

Maybe you could normalize the input in the retriever?

Any chance we could normalize edoc in the long term?

yes, I think that it is a mistake that it is like that today. I'm surprised that our own validators do not catch it...