Allow setting headers (e.g., Content-Type)

[hbssessa]

There’s a clear way to modify the simWS C++ plugin to allow setting custom HTTP headers (like Content-Type) for served resources. The main idea is to extend server_meta and the onWSHTTP method.

1. Add a header map to server_meta

struct server_meta : public _meta
{
    my_server *srv{nullptr};
    optional<std::string> httpHandler;
    std::map<std::string, std::string> responseHeaders; // <-- new
    virtual ~server_meta() { if(srv) delete srv; }
};

2. Modify onWSHTTP to apply headers

void onWSHTTP(server_meta *meta, websocketpp::connection_hdl hdl)
{
    my_server::connection_ptr con = meta->srv->get_con_from_hdl(hdl);

    sim::addLog(sim_verbosity_debug, "onWSHTTP: %s", con->get_resource());

    if(!meta->httpHandler)
    {
        con->set_status(websocketpp::http::status_code::not_found);
        return;
    }

    if(auto h = hdl.lock())
    {
        httpCallback_in in;
        in.serverHandle = srvHandles.add(meta, meta->scriptID);
        in.connectionHandle = connHandles.add(hdl, meta->scriptID);
        in.resource = con->get_resource();
        in.data = con->get_request_body();
        httpCallback_out out;

        if(httpCallback(meta->scriptID, meta->httpHandler->c_str(), &in, &out))
        {
            // Set status and body
            con->set_status(static_cast<websocketpp::http::status_code::value>(out.status));
            con->set_body(out.data);

            // Apply custom headers
            for(const auto &kv : meta->responseHeaders)
            {
                con->replace_header(kv.first, kv.second);
            }
        }
        else
        {
            con->set_status(websocketpp::http::status_code::internal_server_error);
        }
    }
    else
    {
        throw std::runtime_error("connection_hdl weak_ptr expired");
    }
}

3. Expose a function:

void setResponseHeader(setResponseHeader_in *in, setResponseHeader_out *out)
{
    auto meta = srvHandles.get(in->serverHandle);
    meta->responseHeaders[in->headerName] = in->headerValue;
}

So from the lua code it is possible to serve js files calling:

simWS.setResponseHeader(server, "Content-Type", "application/javascript")

[fferri]

What's the need to manually set Content-type header?

[hbssessa]

There are two main reasons:

1. If this feature is allowed, the browser can correctly recognize and serve files. Modern browsers don’t serve files themselves, but they rely on the server to provide the correct MIME type. Without it, the file may not be recognized or may be blocked for security reasons.

2. This approach removes the need to import every script manually in the HTML. You can serve JavaScript files as type="module", which enables automatic dependency resolution inside JavaScript.

[fferri]

The proposed solution could be a bit confusing, as after calling simWS.setResponseHeader(s, "Content-Type", "text/javascript") once, all files (images, html documents, etc) will be served with a text/javascript content-type.

I suggest you two solutions instead:

1: determine content-type automatically based on resource suffix (i.e. file extension)

In onWSHTTP look at the requested resource (con->get_resource()) and set the content-type header based on simple rules, e.g.:

• ends with ".js" -> content-type = text/javascript
• ends with ".htm" or ".html" or "/" -> content-type = text/html
• ends with ".json" -> content-type = application/json
  and so on...

This would already cover common file types without requiring intervention from the user of simWS.

(optional) 2: allow overriding content-type on a per-resource basis

To override the content-type for a particular resource, you can have the Lua HTTP callback return an appropriate third value, i.e. some raw HTTP headers (string) by adding a third return value to the HTTP callback return statement: return <status>, <data>, <headers>

To do so, add a return value in simWS/sourceCode/callbacks.xml <script-function> name="httpCallback" e.g. <param name="headers" type="string" default='""' /> then in onWSHTTP after the call to httpCallback(...) you will find the overridden headers in httpCallback_out::headers, where you would split by newline, then split lines by ": " and apply the custom headers.

[hbssessa]

Thank you for your reply.

I tried the first solution, but it doesn’t work, the browser still does not recognize the MIME type for JavaScript files of type module. I see this approach was used in the CoppeliaSim Three.js plugin, which is the original reason for the proposal (but that does not use javascript dependencies).

The second solution might work and is worth trying, though I’m not sure if it’s the simplest approach.

I only know that the proposed solution works, but it may be suboptimal, and the implementation often depends on personal preference. As soon as it is working and the implementation is clean and maintainable I am fine with that ;)

[fferri]

I tried the first solution, but it doesn’t work, the browser still does not recognize the MIME type for JavaScript files of type module. I see this approach was used in the CoppeliaSim Three.js plugin, which is the original reason for the proposal.

I think you are mistaken: not simWS or threejsFrontend are setting the "Content-type" header currently, as there is no way to do it.

So, solution 1 of my previous message is the obvious (and easiest) route to go for.

[hbssessa]

Yes, I forked threejsFrontend to separate threejsFrontend.js in smaller and manageable modules.
If you write a js file with for example "import { ClassTest } from './ClassTest.js'" the first solution will not work and will not allow to serve it. Although, I agree with you that is easy solution for single js file with no dependencies

[fferri]

Can you show the implementation?

[hbssessa]

Of course!!! Sure, we will have the meeting organized with Marc next week so you can evaluate and take the best route. Glad to share with you!

[fferri]

The strategy I am suggesting is also used by common web servers.
For example, see Apache's mod_mime where MIME types are mapped to file extensions via directives such as AddType text/javascript .js. The same principle applies to Nginx, IIS, Lighttpd, and Node.js: MIME types are determined by file extensions.
So this is not a matter of personal preference, but rather of choosing a correct, reasonable, and widely adopted solution.

In the codebase, we want correct logic that avoids:
1. Confusing behavior of the plugin (see one of my previous comments).
2. Code duplication, which occurs when the MIME type has to be set manually. (You didn’t show the Lua part of your proposed solution, but I assume you would need to duplicate the MIME type detection logic in every server’s Lua script. Why not perform this operation inside the plugin itself, avoiding duplication?)

I will not review your code in a meeting 😉 — GitHub is a much better place for that. If you want to share the relevant part of the implementation you said didn’t work, I’ll be happy to take a look (time permitting — I’m still on vacation).

[hbssessa]

Try to run it, but please enjoy your holidays!!! no rush!!! ;)

Test.js
"import { ClassTest } from './ClassTest.js'

const test = new ClassTest('MyTest')
test.greet() // Should log: "Hello from MyTest!"
"

ClassTest.js
"export class ClassTest {
constructor(name) {
this.name = name
}

greet() {
console.log(Hello from ${this.name}!)
}
}"

threejsFrontend.html

<script type="module" src="core/Test.js"></script> <--add this

You just obtain error from the browser which does not allow you to load the class with the current simWS implementation unless you change the Visualizationstream.lua inside onWSHTTP
from this:

function onWSHTTP(server, connection, resource, data)
local url = require 'socket.url'
resource = url.unescape(resource)
local status, data = 404, nil
if resource == '/' then resource = '/' .. custom.rootResource end
if any(function(ext) return string.endswith(resource, '.' .. ext) end, {'html', 'htm', 'css', 'js', 'js.map'}) then
status, data = getFileContents(resourcesDir .. resource)
if status == 200 and string.endswith(resource, '.html') then
data = string.gsub(data, 'const wsPort = 23020;', 'const wsPort = ' .. wsPort .. ';')
data = string.gsub(data, string.escpat'', custom.extraHTML or '')
data = string.gsub(data, string.escpat'', custom.extraHelp or '')
end
end
if status == 404 and resource ~= '/favicon.ico' then
sim.addLog(sim.verbosity_errors, 'resource not found: ' .. resource)
end
return status, data
end

to something like this :

if any(function(ext) return string.endswith(resource, '.' .. ext) end,
       {'html','htm','css','js','js.map','ico'}) then
    status, content = getFileContents(resourcesDir .. resource)

    local mime = "text/plain"
    if string.endswith(resource, '.html') or string.endswith(resource, '.htm') then
        mime = "text/html; charset=UTF-8"
    elseif string.endswith(resource, '.js')  then
        mime = "application/javascript; charset=UTF-8"
    elseif string.endswith(resource, '.css') then
        mime = "text/css; charset=UTF-8"
    elseif string.endswith(resource, '.js.map') then
        mime = "application/json; charset=UTF-8"
    elseif string.endswith(resource, 'favicon.ico') then
        mime = "image/x-icon"
    end

    if type(mime) == "string" then <---NEED THIS
        simWS.setResponseHeader(server, "Content-Type", mime) <---NEED THIS
    end <---NEED THIS
    if status == 200 and string.endswith(resource, '.html') then
        content = string.gsub(content, 'const wsPort = 23020;', 'const wsPort = ' .. wsPort .. ';')
        content = string.gsub(content, string.escpat'<!--[[CUSTOM_HTML]]-->', custom.extraHTML or '')
        content = string.gsub(content, string.escpat'<!--[[CUSTOM_HELP]]-->', custom.extraHelp or '')
    end
end

[fferri]

I think you can easily push the MIME-type <=> extension mapping logic up to the C++ plugin, like I explained previously.

The Lua-based approach could be a nice-to-have for exceptionally overriding the content-type for one particular file; but it should not be required to state that "a .js file is a javascript file". However, even for this use-case, I think simWS.setResponseHeader is misleading, as it changes the content-type for all future requests. I would return the content type in the HTTP callback (see my comment).

[hbssessa]

Good that you understood my requests for the needed changes. Let’s go over the implementation details in the meeting to make sure we’re fully aligned. I’d like to avoid opening a PR based on a possible misunderstanding, especially since this could affect multiple plugins. That said, I’ve already started working on it, so I can show you something tomorrow that should make more sense. ;)