Sitemap Loader, adding header to request or session to use.

[hebrod77]

Is there a way to add a request session or a request header to the SitemapLoader call.

[dosubot[bot]]

🤖

Based on the current implementation of the SitemapLoader class in the LangChain framework, there isn't a direct way to add a request session or a request header to the SitemapLoader call. The SitemapLoader class does not have a method for adding request headers or sessions to the HTTP requests it makes. The class is primarily used for loading and parsing sitemaps, and it does not contain any methods or attributes for setting request headers or sessions.

However, the WebBaseLoader class, from which SitemapLoader inherits, does provide functionality for adding request headers and sessions to the HTTP requests it makes. This is done in the __init__ method of the class. The header_template parameter of the __init__ method is used to set the headers for the HTTP requests. If no header_template is provided, a default header template is used. The headers are then set on the session object, which is an instance of requests.Session(). The session parameter of the __init__ method is used to set the session for the HTTP requests. If no session is provided, a new requests.Session() is created and used.

The scrape_all method in the WebBaseLoader class, which is used by the SitemapLoader class to make HTTP requests, uses the session object created during the initialization of the WebBaseLoader instance to set the headers for the HTTP requests.

So, while there isn't a direct way to add a request session or a request header to the SitemapLoader call, you can do so indirectly by providing a session object with the desired headers when creating an instance of the SitemapLoader class.

Here is an example of how you can do this:

from requests import Session
from langchain.document_loaders import SitemapLoader

# Create a session with the desired headers
session = Session()
session.headers.update({'header-name': 'header-value'})

# Create a SitemapLoader instance with the session
loader = SitemapLoader(web_path='http://example.com/sitemap.xml', session=session)

# Now the SitemapLoader instance will use the session with the desired headers for its HTTP requests

Please note that this is a workaround and may not work in all cases, depending on the specifics of your use case and the implementation of the LangChain framework. If this does not meet your needs, you may need to modify the LangChain framework to add the desired functionality.

Sources

• libs/langchain/langchain/document_loaders/sitemap.py
• libs/langchain/langchain/document_loaders/web_base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[GuillermoGarciaF]

I confirmed that WebBaseLoader(<url>, session=session) works fine.
WebBaseLoader uses the requests.Session and the defined session headers to make the request.

However, SitemapLoader(<url>, session=session) is not working.
SitemapLoader on the same URL and session returns and empty response.

The SitemapLoader() init method has argument **kwargs, which are passed to the WebBaseLoader base class.
However, something is missing in the SitemapLoader implementation, as the session is not correctly used in its logic.

[eric-burel]

Relevant line of code in sitemap loader implementation
The SitemapLoader will instantiate its parent class, so a WebBaseLoader, with the kwargs passed up. It seems the issue is still there right now, it might be worth opening an issue with a reproduction, I couldn't dig further at this point as I am not familiar with auth and seesions with requests.
I guess when the actual HTTP calls are made, the passed args are not used anymore, perhaps they are applied only to fetching the sitemap and not the content of it.