Dynamic and Hybrid Libraries, Remote Libraries, and Extension Architecture: Diagnostics, Failure Modes, and Production Practices

1. Evidence-first diagnostic sequence

1. Preserve the first output.xml, log/report, server log, command, and version manifest.
2. Confirm Python/Robot/library/server versions and executable paths.
3. Confirm the exact suite/test selection, variables, URI, timeout, cwd, and --pythonpath.
4. Validate imports and then inspect keyword discovery with Libdoc/list before business execution.
5. Compare advertised arguments/types/docs with the call Robot attempted.
6. For local dynamic dispatch, inspect whether run_keyword was reached.
7. For Remote, confirm the server process/listener and loopback/private address before analyzing keyword logic.
8. Inspect serialization shapes and server-side result dictionaries.
9. Check concurrency/CI/container networking only if those layers exist.
10. Apply the smallest correction and rerun the smallest controlled slice.

This order prevents a common waste: debugging implementation code when Robot rejected the call at metadata validation, or debugging Robot syntax when the server was never listening.

2. Intentionally broken dynamic metadata: Robot rejects a valid implementation call

Replace only the first Normalize Token argument specification with the broken version below:

from __future__ import annotations

from robot.api import logger

from .domain import normalize_token


class DynamicCatalog:
    """Runtime-discovered provider used only to demonstrate the dynamic API."""

    def get_keyword_names(self):
        return ["Normalize Token", "Describe Provider"]

    def get_keyword_arguments(self, name):
        specs = {
            "Normalize Token": ["token"],  # BROKEN: metadata omits supported prefix argument
            "Describe Provider": [],
        }
        return specs[name]

    def get_keyword_types(self, name):
        types = {
            "Normalize Token": {"token": "str", "prefix": "str"},
            "Describe Provider": {},
        }
        return types[name]

    def get_keyword_tags(self, name):
        return ["rf21", "dynamic"]

    def get_keyword_documentation(self, name):
        docs = {
            "__intro__": "Synthetic dynamic provider for Chapter 21.",
            "__init__": "No import arguments.",
            "Normalize Token": "Normalize a synthetic token and return prefix:value.",
            "Describe Provider": "Return the provider architecture name.",
        }
        return docs.get(name, "")

    def run_keyword(self, name, args, kwargs):
        logger.info(f"dynamic dispatch keyword={name}")
        if name == "Normalize Token":
            values = {"token": None, "prefix": "ID"}
            if len(args) > 2:
                raise AssertionError("Normalize Token accepts at most two positional arguments")
            for key, value in zip(("token", "prefix"), args):
                values[key] = value
            values.update(kwargs)
            return normalize_token(values["token"], values["prefix"])
        if name == "Describe Provider":
            return "dynamic"
        raise AssertionError(f"Unknown dynamic keyword: {name}")

*** Settings ***
Library    rf21_ext.dynamic_catalog.DynamicCatalog

*** Test Cases ***
Metadata Mismatch Is Visible
    ${value}=    Normalize Token    ab 12    REF
    Should Be Equal    ${value}    REF:AB-12

python -m robot --pythonpath src \
  --outputdir evidence/broken-metadata \
  suites/broken_metadata.robot

Expected failure: Robot sees metadata declaring only one argument and fails the call before normal dispatch. The repair is not to catch the error inside run_keyword; that method was not the failing layer. Restore ["token", "prefix=ID"], regenerate Libdoc, and rerun the same single test.

3. Missing metadata creates late and vague failures

Removing get_keyword_arguments can make a keyword appear to accept arbitrary arguments. A malformed call may then enter run_keyword and fail with an implementation-specific index/key error. That is not “flexibility”; it is a weaker public contract when the signature was knowable.

Repair by restoring authoritative metadata, not by surrounding the dispatcher with a broad except Exception: return None path that would turn contract failures into false greens.

4. Remote connection failure: preserve the outage evidence

Start the loopback server, record its URI, run the Remote test successfully, then stop the server. Without changing the URI, run only the Remote test again into a new evidence directory.

# After server shutdown, REMOTE_URI still points to the old loopback port.
python -m robot \
  --variable "REMOTE_URI:${REMOTE_URI}" \
  --outputdir evidence/remote-down \
  --test "Remote Provider Matches Local Contract" \
  suites/remote_provider.robot

Expected evidence is a connection/import/keyword-discovery failure tied to the unavailable endpoint. Preserve it. Do not “fix” the incident with a giant timeout or blanket retry. First confirm whether the server is supposed to exist, then restart the owned disposable server and rerun into evidence/repaired-remote.

5. Unsafe endpoint exposure is an architecture defect

# DO NOT USE IN THIS LAB OR AS A PRODUCTION DEFAULT.
# SimpleXMLRPCServer(("0.0.0.0", 8270), ...)

Binding to all interfaces is not a troubleshooting shortcut. A Remote endpoint can represent executable automation capability and the protocol does not inherently authenticate callers. If cross-host use is required, design the service boundary explicitly: private addressing, network allowlists, authenticated/authorized front door, protected transport, least privilege, audit logs, and a constrained keyword surface.

6. Serialization/type surprises: diagnose the boundary, not the assertion syntax

7. Orphaned remote process

A passing Robot suite does not prove the server was stopped. CI should retain the PID/port record and have a teardown/finally step owned by the process launcher. If teardown itself fails, record that as a separate operational failure; do not delete the first Robot result.

When Pabot is introduced later, a fixed port becomes an immediate collision risk. Worker-unique ports and ownership are required.

8. “Dynamic because we can” is a maintainability failure mode

If every keyword name, signature, and implementation is known in source, a dynamic registry duplicates Python's own method table. It adds metadata drift, dispatch code, test surface, and poorer static tooling without buying runtime capability. The production repair may be architectural simplification back to the static API, not another helper around run_keyword.

9. Logs, traces, and metadata can leak sensitive values

Remote output, error text, tracebacks, server logs, URIs, and metadata are all evidence channels. Never place real credentials in a keyword name, URI query string, server-ready line, exception message, or serialized sample. Use synthetic values in labs and redact production evidence before broad retention.

Do not respond to leakage by disabling all diagnostics. Reduce and classify what is logged while keeping enough non-sensitive correlation data to diagnose failures.

10. Performance diagnosis by layer

Do not flatten logs, remove evidence, or enlarge timeouts before measuring where the cost actually occurs.

11. Minimal repair runbook

• Import fails: verify interpreter and explicit search path.
• Keyword absent: inspect get_keyword_names/Libdoc.
• Argument rejected: compare advertised spec/types before dispatcher code.
• Dispatcher fails: reproduce with one local dynamic call.
• Remote discovery fails: check server PID, bind address, port, URI/path, and timeout.
• Remote execution fails: correlate Robot output with server result/error and server log.
• Value differs: compare documented serialization shape.
• Cleanup fails: stop only the owned process and preserve evidence.

12. Explicit anti-patterns

• Do not expose a Remote server publicly to “test connectivity.”
• Do not disable TLS/SSH verification or bypass authentication in unrelated infrastructure.
• Do not add broad retries around non-idempotent remote operations.
• Do not catch every dynamic dispatch error and return PASS/empty data.
• Do not use arbitrary PYTHONPATH mutation instead of a reproducible package/search-path contract.
• Do not delete original Robot/server evidence before the repaired run succeeds.
• Do not keep mutable global registries as accidental cross-test or cross-worker coordination.

Knowledge check

References and version anchors

• Robot Framework 7.4.2 — Dynamic API metadata — validation and dispatch
• Robot Framework 7.4.2 — Remote supported types — serialization semantics
• Robot Framework 7.4.2 Remote library — connection/timeout behavior