BUG: validate_input function doesnt comes with helper func param anymore ?

[mdisec]

Description

Hey,

I would like to validate the API tokens during the configuration phase, thus I need to access helper instance. All the examples that I've seen on the existing add-ons has following code from the _helper.py file generated by ucc.

def validate_input(helper, definition):
    """Implement your own validation logic to validate the input stanza configurations"""
    # This example accesses the modular input variable
    # notes = definition.parameters.get('notes', None)
    pass

But what I've seen on my _helper.py file is following one.

def validate_input(definition: smi.ValidationDefinition):
    pass

I would like to have an access to the helper instance because of it's proxy aware http client function etc.

Otherwise I guess I have to access to the metadata fields of the ValidationDefinition and fetch the sessionid and use conf_manager.ConfManager like it's being used at get_account_api_key function that is auto generated ?

I would like to simply build a proxy aware (I mean I wanna use proxy details defined under the Configuration > Proxy of the add-on instead of global one) http client in order to validate the API token before saving the "input" so that I can make sure our SaaS is accesible before Splunk try to ingest the data. What is the best practice for this ?

I also would like to use the helper within the def stream_events for get_check_point etc

What UCC version are you using?

Latest

Additional System Info

WSL

[artemrys]

Hey @mdisec

Sorry for the late response

I think you are referring to Add-on Builder-generated add-ons, they indeed have a helper instance available for the developers. There is currently nothing like this in the UCC framework.

I totally understand your use case, let me talk to the team and I'll circle back to you here.

[mdisec]

Thank you for the reply @artemrys ! Let me share what I've ended up to use is as follow. They might help the team to evaluate why I kindly ask for these helpers :D

I've implemented following helper functions in order to create a proxy aware http client.

def get_proxy_uri(proxy: dict) -> str:
    uri = None
    if proxy and proxy.get("proxy_url") and proxy.get("proxy_type"):
        uri = proxy["proxy_url"]
        if proxy.get("proxy_port"):
            uri = "{}:{}".format(uri, proxy.get("proxy_port"))
        if proxy.get("proxy_username") and proxy.get("proxy_password"):
            uri = "{}://{}:{}@{}/".format(
                proxy["proxy_type"],
                proxy["proxy_username"],
                proxy["proxy_password"],
                uri,
            )
        else:
            uri = "{}://{}".format(proxy["proxy_type"], uri)
    return uri

def get_proxy_config_from_manager(session_key: str) -> dict:
    cfm = conf_manager.ConfManager(
        session_key,
        ADDON_NAME,
        realm=f"__REST_CREDENTIAL__#{ADDON_NAME}#configs/conf-{ADDON_NAME}_settings",
    )
    account_conf_file = cfm.get_conf(f'{ADDON_NAME}_settings')
    return account_conf_file.get("proxy")

def proxy_aware_http_client(session_key: str) -> requests.Session:
    session = requests.Session()
    proxy = get_proxy_config_from_manager(session_key)
    if proxy.get("proxy_enabled", 0) == "1":        
        session.proxies = {
            "http": get_proxy_uri(proxy),
            "https": get_proxy_uri(proxy),
        }
    return session

I use http_client = proxy_aware_http_client(session_key) within the stream_events function as follow.

def stream_events(inputs: smi.InputDefinition, event_writer: smi.EventWriter):
   """
   This function auto generated by UCC
   """
    for input_name, input_item in inputs.inputs.items():
        normalized_input_name = input_name.split("/")[-1]
        logger = logger_for_input(normalized_input_name)
        try:
            session_key = inputs.metadata["session_key"]
            log_level = conf_manager.get_log_level(
                logger=logger,
                session_key=session_key,
                app_name=ADDON_NAME,
                conf_name=f"{ADDON_NAME}_settings",
            )
            logger.setLevel(log_level)
            log.modular_input_start(logger, normalized_input_name)
            api_key = get_account_api_key(session_key, input_item.get("account"))
            http_client = proxy_aware_http_client(session_key)
        except Exception as e:
            log.log_exception(logger, e, "my custom error type", msg_before="Exception raised while ingesting data for demo_input: ")

I'm literally using python's requests module... But the problem is I think that I also need to find a way to get the Splunk global proxy configuration and use it if there is no app specific proxy defined within my app, right ? I'm kind a lost here.. Maybe behind-the-scene Splunk assigning the export=http_proxy and http_proxy enviroment variables and python automatically use them ?... I dunno..

About the checkpoint stuff

In terms of checkpoint helper replication, I ended up directly using KVStoreCheckpointer from from solnlib.modular_input import checkpointer.

    checkpointer.KVStoreCheckpointer(
        collection_name=collection_name,
        session_key=session_key,
        app=ADDON_NAME,
    )

In general, I believe UCC can be easily used by developers to develop their own add-ons to ingest data from their SaaS solution without even touching to the "Splunk Add-on Builder". Having old helpers within the UCC would be lovely since people like me who are new to the splunk app development can easily develop their app via UCC without the hustle I've been going through <3