Defining the Retrieval Source for Webmentions

[jalcine]

The Webmention spec provides a means of resolving the receiver. This issue is meant to define where said Webmentions can be requested from for the later rendering and presentation of them by a party.

• Is another endpoint needed?
• What should the request format look like?
• Should this endpoint support IndieAuth or some sort of authorization?

Though related, this can inform how #1 works.

[jalcine]

There's some discussion on a similar concept for OAuth2: https://mailarchive.ietf.org/arch/msg/oauth/6lfqd3n8O196a3Om9HxVfCFLLQk/

[jalcine]

I'm in the favor of making this something that lives in the same endpoint following the concept of q=source in Micropub. This would provide current implementators who are familiar with Micropub a common interface and also reduce the need to have an endpoint for each action/concept.

[manton]

I wonder if q=source is really even needed, because there is no behavior defined for GET requests on the Webmention endpoint. Maybe a simple GET with a target parameter (to specify the post to retrieve mentions for) is enough of a default. If we're borrowing from Microsub, an action parameter could allow extensibility for other types of requests.

[dshanske]

Some people use the GET behavior to display a webmention form.

[manton]

Interesting, I hadn't seen that. @dshanske: Does the WordPress plugin do anything for GET?

[dshanske]

We do that. If you do a GET, no parameters on the endpoint you get a webmention form. But that's with no query parameters, so this could still work.

[jalcine]

I like that flow @manton, definitely more backcompat!

[barnabywalters]

Between the wordpress plugin, webmention.io (example) and various independent implementations (example), the webmention endpoint being a human-readable form is a common pattern which we shouldn’t break.

[jalcine]

From what's proposed so far:

• Maintaining the existing POST behavior
• Extending the endpoint such that:
  • When it's called as a GET with no parameters, that it can do whatever the endpoint chooses to do (in my case, it just 404s, in others, they render a form)
  • When it's called as a GET with a query parameter of target, it'll render a list of the Webmentions received by this endpoint for the URL defined at target. I'm guessing the initial validation flow in the spec should be followed here too

Does that sound good so far?

[manton]

That sounds good to me!

[aaronpk]

One reason I'm not super excited about the idea of using the same endpoint for retrieving webmentions is it will likely be very common to require authentication to retrieve the webmentions, whereas the webmention endpoint has to not require authentication to accept webmentions. It's usually harder to make a single endpoint do different things depending on whether a request is authenticated.

[manton]

If we don't use the same endpoint, I assume we'll want either another link tag or some version of a q=config like Micropub's discovery of the media endpoint? It would be nice to avoid an extra link tag for this, especially if it becomes very common for most Webmention implementations to support retrieval.

[jalcine]

Hm, I figure that optional authorization for an endpoint could be done similarly to the presence of a parameter in a query or in a header. And to get more Webmentions that are potentially gated (for moderation reasons) that providing a token would allow those entries to be added into the list.

[jalcine]

@aaronpk What ideas for fetching webmentions do you have? I remember you mentioning from the IETF?

[jalcine]

So currently, I'm toying with my local setup having a tag like the following on pages that accept Webmentions:

...
<link rel="webmention" href="https://webmention.service/endpoint" />
<link rel="webmention feed" href="https://webmention.service/endpoint/feed.html" type="text/html" title="Mentions on this Page" />
<link rel="webmention feed" href="https://webmention.service/endpoint/feed.jf2" type="application/json+jf2" />
...

After seeing how Wordpress sites tend to explicitly expose a feed for subscribing to comments on a particular page, this felt like a good form of prior art to follow and something that works with existing implementations (because it can simply fail with a 400 if one attempts to send a Webmention using the Webmention feed's URL and because we can specify the usable mime types for it in there)

[jalcine]

I opted for the above because this would allow my endpoint to just focus on handling CRUD'ing Webmentions as they come in and allowing another dedicated endpoint to handle rendering and do conditional authorization if needed.

[manton]

That's an interesting approach. I think I'd still prefer something like q=config because then it's just a single link tag if someone is configuring their site to point to a Webmention service hosted somewhere else. That service could even add different feed formats without requiring any changes in the post or site theme HTML.

[jalcine]

I've been thinking more about Aaron's earlier comment, to keep it simpler for existing implementations, perhaps using the rel "webmention-endpoint" that does what Manton suggests? That way, I can keep having my implementation use a singular endpoint for both but if another implementation chooses to have a different endpoint, it could be specific accordingly.

The biggest win with this approach is that if it's all dependent on changing a rel value for dual-function endpoints and adding a new link rel for additional methods.

[manton]

@jalcine Not sure I'm understanding this suggestion… So, keep the existing rel="webmention" the same, but add "webmention-endpoint" which returns JSON to describe the different feeds? I think I'm confused. 🙂

[jalcine]

Close! I'm proposing something like this:

Single Endpoint Setups

<link rel="webmention webmention-endpoint" href="https://my.webmention.endpoint/the-url" />

This would allow for people like me who'd want to use a singular endpoint to both do the existing work of receiving Webmentions and sending them as well as any kind of query work we could put with a Micropub-esque query (or something else - Micropub comes to mind but it could be something else).

Multiple Endpoint Setups

<link rel="webmention-endpoint" href="https://my.webmention.endpoint/the-url" />
<link rel="webmention" href="https://my.webmention.endpoint/the-other-url" />

This allows for the case that @aaronpk advocates and keeps those concerns separately.

---

It's up to the one handling fetching, sending, etc to discern the difference of these endpoints and that's already done by the spec.

[manton]

If we go with something like this, I think we need a rel name other than "webmention-endpoint"… Something more specific so it won't be confused with regular "webmention".

I also think the default behavior (if there is only a single rel="webmention") should be that that endpoint can do everything: sending Webmentions and also retrieving the source. I feel like splitting this out into separate endpoints will be the exception.

[jalcine]

Coming back to this while working on Lighthouse, I've decided to work to maintain as much of the existing functionality of Webmention while making it extensible. Here's what I have (and have planned):

Existing Functionality

Sending a Webmention from the standard (at the time of writing):

POST /webmention-endpoint HTTP/1.1
Host: lighthouse.example
Content-Type: application/x-www-form-urlencoded

source=https://waterpigs.example/post-by-barnaby&
target=https://aaronpk.example/post-by-aaron


HTTP/1.1 201 Accepted
Location: http://lighthouse.example/status/DEhB9Jme

New Functionality

Fetching Webmentions for a particular URL (using a Micropub-styled query):

GET /webmention-endpoint?q=source&target=https://aaronpk.example/post-by-aaron&format=mf2+json HTTP/1.1
Host: lighthouse.example
Accept: application/mf2+json

HTTP/1.1 200 OK
Content-Type: application/mf2+json

{
  "items": [
    {
       "id": "wm-mention-id-403",
       "type": ["h-entry"],
       "properties": {
          "url": ["https://waterpigs.example/post-by-barnaby"],
       }
    }
  ],
}

In this example, I put the expected format in both the Accept header and as a query parameter. The point of that is to highlight that either CAN be expected but aren't required (an endpoint can choose to default to rendering it as HTML or JF2). I think defaulting it to JF2 would make sense since things like https://webmention.io/ already does so.

Fetching Webmention counts for a particular URL (using a Micropub-styled query):

GET /webmention-endpoint?q=count&target=https://aaronpk.example/post-by-aaron&format=json HTTP/1.1
Host: lighthouse.example
Accept: application/json

HTTP/1.1 200 OK
Content-Type: application/json

{
  "total": 40,
  "type": {
    "mention": 10,
    "protected": 7,
    "like": 13,
    "comment": 10
  }
}

The type key doesn't have to have children that map to post types, that can be implementation specific.

Sending a Webmention (this follows the Micropub flow for updating posts and is implicitly how the act of creation of entries are represented in Koype):

POST /webmention-endpoint HTTP/1.1
Host: lighthouse.example
Content-Type: application/x-www-form-urlencoded

action=submit&
source=https://waterpigs.example/post-by-barnaby&
target=https://aaronpk.example/post-by-aaron


HTTP/1.1 201 Accepted
Location: http://lighthouse.example/status/DEhB9Jme

This approach still works with existing formats (if the server is capable of either defaulting to the classic approach or defaulting the action to be submit) and allows for other operations on a Webmention endpoint to occur. I have planned features like pausing acceptance of new Webmentions that I could represent with action=pause.

Pros and Cons

• Pro 1: This makes Webmention a bit more like Micropub while maintaining the same functionality
• Pro 2: This allows for capability discovery in newer implementations (similar to Micropub's configuration query)
• Con 1: This introduces complexity into the standard and makes it difficult to determine the capability of a Webmention endpoint
• Con 2: This introduces content negotiation when attempting to retrieve Webmentions for a URL.

I think Con 1 can be fixed by clients by attempting to query for configuration. If it fails, they can that it's operating with an older version of Webmention that doesn't support queries. The failure case would be if it doesn't return a JSON body, that is, some places might return HTML on a GET request and that's OK.

[jalcine]

Ah, other things I wanted to add into a querying endpoint was the ability to look up URLs that can be sent a Webmention (similar to what https://webmention.app/ does). That request would look like:

GET /webmention-endpoint?q=discover&url=https://aaronpk.example/post-by-aaron&format=json HTTP/1.1
Host: lighthouse.example
Accept: application/json

HTTP/1.1 200 OK
Content-Type: application/json

[
   "https://aaronpk.example/page/1",
   "https://aaronpk.example/page/2",
   "https://aaronpk.example/page/3",
   "https://aaronpk.example/page/4",
   "https://aaronpk.example/page/5",
   "https://aaronpk.example"
]

[jalcine]

I know that authentication was a potential issue here. In my implementation, having conditional authentication isn't too difficult, so I'm curious about the case here where it would be (like is the authentication token checked before anything occurs in a request?).

[manton]

Cool. I'm going to add support for target and format params in a GET to Micro.blog. Is there any precedent in other specs for the value of format? I'm currently using format=jf2 and I see you have format=mf2+json.

[manton]

I also wonder about consistency with Webmention.io's JSON output. It has a children field, you have items. Not really sure where the children came from as items seems more in line with other JF2 output.

[jalcine]

Cool. I'm going to add support for target and format params in a GET to Micro.blog. Is there any precedent in other specs for the value of format? I'm currently using format=jf2 and I see you have format=mf2+json.

I only used mf2+json to mimic application/mf2+json (using the suffix)!

I also wonder about consistency with Webmention.io's JSON output. It has a children field, you have items. Not really sure where the children came from, as items seems more in line with other JF2 output.

It's probably because it returns a top-level item (an h-feed) and to describe its descendants, one would use children. I think I might use that instead as well.

[jalcine]

I think it'd be okay to do jf2 and jf2+json (I figure with jf2, the +json part SHOULD be implied).

[jalcine]

An upside of this approach with the format parameter in the query is that one could implement an RSS or JSON Feed for Webmentions and attach that to a static site!

[manton]

I've added an initial version of this GET to Micro.blog. I support "jf2", "mf2+json" (same thing), or "jsonfeed" in the format for now. I also have a few "wm-" fields that Webmention.io uses.

Here's what it looks like.

GET /webmention?target=https%3A%2F%2Fwww.manton.org%2F2022%2F03%2F12%2Fspeaking-of-cold.html&format=jf2

{
  "type": "feed",
  "name": "Conversation",
  "children": [
    {
      "type": "entry",
      "author": {
        "type": "card",
        "name": "Pete Moore",
        "url": "https://pimoore.ca",
        "photo": "https://micro.blog/pimoore/avatar.jpg"
      },
      "url": "https://micro.blog/pimoore/12577850",
      "published": "2022-03-12T17:00:16+00:00",
      "wm-received": "2022-03-12T17:00:16+00:00",
      "wm-id": 12577850,
      "content": {
        "text": "@manton wow, Utah really is gorgeous.",
        "html": "<p><a href=\"https://micro.blog/manton\">@manton</a> wow, Utah really is gorgeous.</p>\n"
      },
      "mention-of": "https://www.manton.org/2022/03/12/speaking-of-cold.html",
      "wm-property": "mention-of",
      "wm-private": false
    }
  ]
}

GET /webmention?target=https%3A%2F%2Fwww.manton.org%2F2022%2F03%2F12%2Fspeaking-of-cold.html&format=jsonfeed

{
  "version": "https://jsonfeed.org/version/1.1",
  "title": "Conversation",
  "home_page_url": "https://micro.blog/manton/12577803",
  "feed_url": "https://micro.blog/webmention?target=https%3A%2F%2Fwww.manton.org%2F2022%2F03%2F12%2Fspeaking-of-cold.html&format=jsonfeed",
  "items": [
    {
      "id": "12577850",
      "content_html": "<p><a href=\"https://micro.blog/manton\">@manton</a> wow, Utah really is gorgeous.</p>\n",
      "url": "https://micro.blog/pimoore/12577850",
      "date_published": "2022-03-12T17:00:16+00:00",
      "authors": [
        {
          "name": "Pete Moore",
          "url": "https://pimoore.ca",
          "avatar": "https://micro.blog/pimoore/avatar.jpg",
          "_microblog": {
            "username": "pimoore"
          }
        }
      ]
    }
  ]
}

[jalcine]

Amazing! I can imagine this will be great for those who want to show mentions on their sites now. I'm still working to utilize these feeds on my site, but this is a great first step!

[jalcine]

Stealing the format of webmention.io's query Looking at prior art, there's a few things that can be added here (Lighthouse will most likely support all of these):

• Allowing target to be a list or a string (via: https://github.com/aaronpk/webmention.io#find-links-to-multiple-pages)
• Providing a domain to fetch mentions from (via: https://github.com/aaronpk/webmention.io#find-all-links-to-your-domain)
• Listing everything (or anything) (via: https://github.com/aaronpk/webmention.io#find-all-links-to-all-sites-in-your-account)

Because I'm reusing some logic from my Micropub server for these endpoints, it'll have all of the querying options as well in Micropub (filtering, sorting, range fields, etc).

Hoping to have a public alpha of this by end of the month (if not 10 Apr)